Skip to content

fix(html): recover from deeply nested Wikipedia and Bing pages - #2444

Open
kevin (kevin9327) wants to merge 1 commit into
microsoft:mainfrom
kevin9327:fix/deeply-nested-site-html
Open

fix(html): recover from deeply nested Wikipedia and Bing pages#2444
kevin (kevin9327) wants to merge 1 commit into
microsoft:mainfrom
kevin9327:fix/deeply-nested-site-html

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

Converting a deeply nested Wikipedia page produces the whole document — site notice, page chrome, footer — instead of the article:

Site notice chrome
Nested
Deep article content
Footer chrome

expected the Wikipedia extraction it gets for any other page:

# Nested

Deep article content

A Bing results page loses its header the same way: Deep result content instead of ## A Bing search for 'nested' found the following results: followed by the result.

markdownify walks the DOM recursively, so deeply nested markup raises RecursionError. HtmlConverter catches it and falls back to BeautifulSoup's iterative get_text() (#1644, issue #1636), and RssConverter does the same for item content (#2333). WikipediaConverter and BingSerpConverter read exactly that kind of markup and have no such guard, so they raise. MarkItDown._convert records the failed attempt and keeps going, and HtmlConverter — registered at PRIORITY_GENERIC_FILE_FORMAT, so it is tried after them — converts the page as generic HTML. Nothing is raised to the caller: the page just quietly stops being read as a Wikipedia article or a search-results page, and title becomes the raw <title> ("Nested - Wikipedia") rather than the page title.

This gives both converters the same guard, written the same way, including the strict=True escape hatch the other two accept.

Note that strict is now popped from kwargs in these two converters rather than being forwarded into markdownify's options, matching HtmlConverter.convert and RssConverter.convert.

Reproduction

New test file, on unmodified main (packages/markitdown). It lowers the recursion limit around each conversion, as the two existing tests for this failure in test_module_misc.py do, so the depth needed does not depend on the host:

$ python -m pytest tests/test_deeply_nested_site_html.py -q
FAILED tests/test_deeply_nested_site_html.py::test_deeply_nested_wikipedia_page_keeps_the_article_extraction
FAILED tests/test_deeply_nested_site_html.py::test_deeply_nested_bing_serp_keeps_the_results_extraction
2 failed, 2 passed in 3.63s

The failures are the fallthrough itself:

>       assert markdown.startswith("# Nested")
E       AssertionError: assert False
E        +    where ... = 'Site notice chrome\nNested\nDeep article content\nFooter chrome'.startswith

>       assert markdown.startswith("## A Bing search for 'nested' found")
E       assert False
E        +    where ... = 'Deep result content'.startswith

The two tests that pass before and after are the pins: strict=True still raises RecursionError out of both converters, and a shallow Wikipedia page still converts to # Shallow / Plain **article** text.

With the fix:

$ python -m pytest tests/test_deeply_nested_site_html.py -q
4 passed in 2.49s

Verification

$ python -m pytest tests/test_module_vectors.py tests/test_module_misc.py \
    tests/test_html_converter.py tests/test_rss_converter.py -q \
    -k "not file_uris and not case_insensitive and not speech"
301 passed, 2 skipped, 3 deselected in 94.63s    # origin/main
301 passed, 2 skipped, 3 deselected in 116.37s   # this branch

test_module_vectors.py covers the test_wikipedia.html and test_serp.html end-to-end vectors, and test_module_misc.py covers the existing HTML and RSS deep-nesting fallbacks.

(The three deselected tests fail on main on this Windows machine for unrelated reasons: console encoding, file-URI paths, and a missing ffmpeg.)

black reports all three touched files unchanged.

HtmlConverter (microsoft#1644) and RssConverter (microsoft#2333) fall back to plain-text
extraction when markdownify's recursive traversal exceeds Python's
recursion limit. WikipediaConverter and BingSerpConverter read the same
kind of markup and did not, so a deeply nested page threw the
site-specific extraction away and came back as the whole document.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant