fix(documents): render tables in TOC-path section text instead of fusing cells

[dgunning]

What was wrong

Text for sections found through the table of contents glued each table row's cells together. For example, Regions' 10-K Item 5 read November 1-30, 20217,294,800: the year 2021 fused with a 7,294,800 share count. Across the 70 tracked 10-K/10-Q/20-F fixtures there were 144 such tokens, all on the TOC path; doc.text() had none. This turned up while fixing GH #1347, whose fix moved ExxonMobil's 10-Q onto the TOC path.

Root cause

SECSectionExtractor._extract_section_content builds TOC section text by walking the raw lxml tree and adds a break only after block elements. td and th aren't block elements, so neighbouring cells ran together. doc.text() renders tables through TableProcessor → TextExtractor/FastTableRenderer instead, so the same job had two implementations.

Fix

• The TOC walk hands each whole <table> in range to TableProcessor and to a new TextExtractor.render_table, the rule doc.text() uses. A table that contains the section's end boundary keeps the element-wise walk, so the section still stops inside it.
• Two layout differences from doc.text(), both deliberate:
  • Column width is unbounded. The default truncates cells over 500 chars.
  • Alignment padding collapses to two spaces and rule lines are dropped. The padding alone made table-heavy sections 1.5–2.6× longer.
• Routing section text through the shared renderer exposed two data losses in that renderer. Both are fixed at the source, so doc.text() gains the content too:
  • FastTableRenderer._identify_meaningful_columns dropped columns of one- or two-character values, such as Regions' "$85" and Morgan Stanley's "WM".
  • TableProcessor._extract_cell_content kept only a cell's <div> text, which lost labels like Netflix's "Derivatives not designated as hedging instruments:".

The TOC-path SIGNATURES cutoff mentioned in the bead already landed in #1355.

Evidence for the re-pinned values

Checked across all 70 tracked 10-K/10-Q/20-F fixtures by counting every letter and digit in doc.text() and in each section, on main and on this branch:

Check	Result
TOC-path fused tokens	144 → 0
Documents losing any letter/digit	0
Sections losing any letter/digit	0
Sections added or removed	0
doc.text()	+2,474 letters/digits across 58 documents
Largest per-section gain	0.65%; by share, HD Item 4 +17 chars (a page-footer cell, "25 / Table of Contents")

So every changed pin is either collapsed table padding or recovered content. The 20-F's Item 18 going from 181,349 to 151,598 chars is padding. Content assertions were updated to the separated form ("Item 5. Other Information") and none were loosened. The FilingSummary baseline for AAPL reports R8, R13 and R14 changes only because cells now keep their own text ahead of their <div>s.

Verification

• -m fast -n 4: 7,691 passed, 0 failed.
• New test test_wzgu_toc_section_table_cells.py: 6 passed.
• Affected tests: TOC: with one anchor per page, consecutive items share a span — obj['Item 7A'] returns Item 7 #1345 13 passed, TOC: a label cell that carries its own title ("Item 1. Business") yields no item label — FirstEnergy's obj['Item 7'] returns Item 8 #1347 12 passed.
• check_regression_provenance.py and check_regression_skips.py: OK on 395 files.

Bead: edgartools-wzgu

🤖 Generated with Claude Code