Repository navigation
XLSX/XLS: keep text cells that look numeric as text (02134 stays 02134) - #2708
Cindy (cindykrafft) wants to merge 2 commits into
Conversation
pd.read_excel parsed cells stored as text into numbers, so 02134 became 2134, +15551234 became 15551234 and 007 became 7.0. Read with dtype=object and infer_objects() so text stays text and numeric columns render as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VVn4SeAXPsUajXT2rtHQfo
|
I checked the XLSX read helper at One separate boundary worth keeping explicit: cells stored as the literal strings Two small follow-ups, if useful:
These were in-memory XLSX helper checks, not a full Markdown conversion or |
Keeping numeric-looking text as text does not change how pandas reads "NA", "N/A", "NULL" or "nan" data cells: they were missing before and still are. Say so in the helper docstring and add a test with a kept text cell next to them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VVn4SeAXPsUajXT2rtHQfo
|
Thanks for testing it, and for the clear scoping. I took both suggestions in 36422a3:
Agreed that changing the missing-value policy is a separate question, and that it would need tests with real blank cells. This PR doesn't touch it. #2709 carries the same commit. Disclosure: this change and reply were produced by an AI agent (Claude Code), which ran the tests mentioned. |
There was a problem hiding this comment.
Community review, so this does not clear the merge gate. I read the whole diff and ran it.
What I ran, on Windows with Python 3.13 and pandas 3.0.6, against markitdown main (4cc9fa1) and this PR head (36422a3):
- Bug on main: a sheet with text cells
02134,+15551234,007,0042,00501,0612345678converts to2134,15551234,7,42,501,612345678. On the PR the same sheet keeps every one of those as written. A text column of1e5and a 22-digit string also stay as written (main gives1.000000e+05and1.234568e+21). test_xlsx.pypasses on the PR (28 passed). With the PR'stest_xlsx.pyrun against main's source, the two text-preservation tests (..._stay_textand..._missing_value_markers_are_unchanged) fail and the rest pass.- Regression check: I converted a set of columns on both trees and diffed the output: date columns, date plus time, a date column with a blank, int columns with a blank, mixed int and float, bool, mixed text and number, a percent-formatted cell, a time cell, a timedelta cell, and an error cell. All identical on main and the PR.
- The one difference I found is an edge case: a numeric column holding a number too large for int64 (I used 1e20 next to 0.1+0.2). Main renders
1.000000e+20and3.000000e-01; the PR renders100000000000000000000and0.3. I think the PR output is fine, but it is a visible change that the description does not mention. - Timing on a 30,000 row by 5 column sheet: 6.3 s on main and 6.6 s on the PR, with byte-identical Markdown.
tests/test_files/test.xlsconverts to identical output on both.
Not run: the .xls path with numeric-looking text (I had no such .xls file and did not build one), and markitdown-ocr, which is not installed in my environment. The only change there is the test's expected kwargs ("dtype": object), which I read but did not execute.
One note, not a blocker: NA-like text such as NA, N/A, NULL and nan still becomes NaN, as the new test documents. That is existing behavior and the PR is clear about it.
The change does what it says and I found no problems.
XlsxConverter/XlsConverterread sheets withpd.read_exceldefaults, so cells the workbook stores as text are parsed as numbers. With default settings, including the CLI:ZIP codes, phone numbers, IDs and part numbers lose their leading zeros or signs, silently.
This reads with
dtype=objectand then callsinfer_objects(), so text cells stay text and genuinely numeric columns render byte-identically to before.Related: open PR #2516 uses plain
dtype=objectfor a different problem and would fix this too, but (per its description) also changes how float columns render. This is the narrower change.Tests
test_xlsx_text_cells_that_look_numeric_stay_textfails on main and passes with the change;test_xlsx_numeric_cells_render_as_beforeguards the numeric path.packages/markitdown: all tests pass (1095 on main plus the new ones).packages/markitdown-ocr: 108 passed after updating one assertion that pins the exactread_excelkeyword arguments..xlspath shares the code but was not exercised (creating a test file needsxlwt).Disclosure: this was found, reproduced and patched by an AI agent (Claude Code), which ran the reproduction and tests quoted above. It has not been separately reviewed by a human.