Skip to content

XLSX/XLS: keep text cells that look numeric as text (02134 stays 02134) - #2708

Open
Cindy (cindykrafft) wants to merge 2 commits into
microsoft:mainfrom
cindykrafft:fix/xlsx-keep-text-cells
Open

Cindy (cindykrafft) wants to merge 2 commits into
microsoft:mainfrom
cindykrafft:fix/xlsx-keep-text-cells

Conversation

@cindykrafft

Copy link
Copy Markdown

XlsxConverter / XlsConverter read sheets with pd.read_excel defaults, so cells the workbook stores as text are parsed as numbers. With default settings, including the CLI:

| zip   | phone      | sku  |        | zip  | phone     | sku    |
| 02134 | +15551234  | 007  |   →    | 2134 | 15551234  | 7.0    |
| 00501 | 0612345678 | 1e3  |        | 501  | 612345678 | 1000.0 |

ZIP codes, phone numbers, IDs and part numbers lose their leading zeros or signs, silently.

This reads with dtype=object and then calls infer_objects(), so text cells stay text and genuinely numeric columns render byte-identically to before.

Related: open PR #2516 uses plain dtype=object for a different problem and would fix this too, but (per its description) also changes how float columns render. This is the narrower change.

Tests

  • test_xlsx_text_cells_that_look_numeric_stay_text fails on main and passes with the change; test_xlsx_numeric_cells_render_as_before guards the numeric path.
  • packages/markitdown: all tests pass (1095 on main plus the new ones). packages/markitdown-ocr: 108 passed after updating one assertion that pins the exact read_excel keyword arguments.
  • black 23.7.0 (the pre-commit pin) clean.
  • The .xls path shares the code but was not exercised (creating a test file needs xlwt).

Disclosure: this was found, reproduced and patched by an AI agent (Claude Code), which ran the reproduction and tests quoted above. It has not been separately reviewed by a human.

pd.read_excel parsed cells stored as text into numbers, so 02134 became
2134, +15551234 became 15551234 and 007 became 7.0. Read with
dtype=object and infer_objects() so text stays text and numeric columns
render as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VVn4SeAXPsUajXT2rtHQfo
@carlitogwapito

Copy link
Copy Markdown

I checked the XLSX read helper at 0e905d9 with pandas 3.0.2 and openpyxl 3.1.5. Eight small baseline/candidate checks confirm that leading-zero text survives with this patch, real numbers stay numeric, and an interior blank stays missing. Nice, focused fix.

One separate boundary worth keeping explicit: cells stored as the literal strings NA, N/A, NULL, and nan still become missing values. This happens on both main and this branch, so it's not a regression introduced here. dtype=object doesn't turn off pandas' default NA recognition.

Two small follow-ups, if useful:

  • Add a literal-marker row plus a nonmissing anchor to the text-preservation tests, so this boundary doesn't get confused with numeric inference.
  • Keep the scope clear: numeric-looking text is preserved; the existing missing-value policy is unchanged. A future change to that policy should also test real blanks rather than just disabling NA parsing across the board.

These were in-memory XLSX helper checks, not a full Markdown conversion or .xls test run. No need to broaden this PR to address the separate policy question.

Keeping numeric-looking text as text does not change how pandas reads
"NA", "N/A", "NULL" or "nan" data cells: they were missing before and
still are. Say so in the helper docstring and add a test with a kept text
cell next to them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VVn4SeAXPsUajXT2rtHQfo
@cindykrafft

Copy link
Copy Markdown
Author

Thanks for testing it, and for the clear scoping. I took both suggestions in 36422a3:

  • New test_xlsx_missing_value_markers_are_unchanged: NA, N/A, NULL and nan data cells next to leading-zero ZIP codes, plus a kept text cell. It pins that those markers are still read as missing while the numeric-looking text is kept. (It passes on main as well; it's there to keep the two behaviours from being confused.)
  • The _read_excel_sheets docstring now says that pandas' missing-value markers are unchanged.

Agreed that changing the missing-value policy is a separate question, and that it would need tests with real blank cells. This PR doesn't touch it. #2709 carries the same commit.

Disclosure: this change and reply were produced by an AI agent (Claude Code), which ran the tests mentioned.

@VANDRANKI PRABHU KIRAN VANDRANKI (VANDRANKI) left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Community review, so this does not clear the merge gate. I read the whole diff and ran it.

What I ran, on Windows with Python 3.13 and pandas 3.0.6, against markitdown main (4cc9fa1) and this PR head (36422a3):

  • Bug on main: a sheet with text cells 02134, +15551234, 007, 0042, 00501, 0612345678 converts to 2134, 15551234, 7, 42, 501, 612345678. On the PR the same sheet keeps every one of those as written. A text column of 1e5 and a 22-digit string also stay as written (main gives 1.000000e+05 and 1.234568e+21).
  • test_xlsx.py passes on the PR (28 passed). With the PR's test_xlsx.py run against main's source, the two text-preservation tests (..._stay_text and ..._missing_value_markers_are_unchanged) fail and the rest pass.
  • Regression check: I converted a set of columns on both trees and diffed the output: date columns, date plus time, a date column with a blank, int columns with a blank, mixed int and float, bool, mixed text and number, a percent-formatted cell, a time cell, a timedelta cell, and an error cell. All identical on main and the PR.
  • The one difference I found is an edge case: a numeric column holding a number too large for int64 (I used 1e20 next to 0.1+0.2). Main renders 1.000000e+20 and 3.000000e-01; the PR renders 100000000000000000000 and 0.3. I think the PR output is fine, but it is a visible change that the description does not mention.
  • Timing on a 30,000 row by 5 column sheet: 6.3 s on main and 6.6 s on the PR, with byte-identical Markdown. tests/test_files/test.xls converts to identical output on both.

Not run: the .xls path with numeric-looking text (I had no such .xls file and did not build one), and markitdown-ocr, which is not installed in my environment. The only change there is the test's expected kwargs ("dtype": object), which I read but did not execute.

One note, not a blocker: NA-like text such as NA, N/A, NULL and nan still becomes NaN, as the new test documents. That is existing behavior and the PR is clear about it.

The change does what it says and I found no problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants