fix(html): recover from deeply nested Wikipedia and Bing pages - #2444
Open
kevin (kevin9327) wants to merge 1 commit into
Open
fix(html): recover from deeply nested Wikipedia and Bing pages#2444kevin (kevin9327) wants to merge 1 commit into
kevin (kevin9327) wants to merge 1 commit into
Conversation
HtmlConverter (microsoft#1644) and RssConverter (microsoft#2333) fall back to plain-text extraction when markdownify's recursive traversal exceeds Python's recursion limit. WikipediaConverter and BingSerpConverter read the same kind of markup and did not, so a deeply nested page threw the site-specific extraction away and came back as the whole document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Converting a deeply nested Wikipedia page produces the whole document — site notice, page chrome, footer — instead of the article:
expected the Wikipedia extraction it gets for any other page:
A Bing results page loses its header the same way:
Deep result contentinstead of## A Bing search for 'nested' found the following results:followed by the result.markdownify walks the DOM recursively, so deeply nested markup raises
RecursionError.HtmlConvertercatches it and falls back to BeautifulSoup's iterativeget_text()(#1644, issue #1636), andRssConverterdoes the same for item content (#2333).WikipediaConverterandBingSerpConverterread exactly that kind of markup and have no such guard, so they raise.MarkItDown._convertrecords the failed attempt and keeps going, andHtmlConverter— registered atPRIORITY_GENERIC_FILE_FORMAT, so it is tried after them — converts the page as generic HTML. Nothing is raised to the caller: the page just quietly stops being read as a Wikipedia article or a search-results page, andtitlebecomes the raw<title>("Nested - Wikipedia") rather than the page title.This gives both converters the same guard, written the same way, including the
strict=Trueescape hatch the other two accept.Note that
strictis now popped fromkwargsin these two converters rather than being forwarded into markdownify's options, matchingHtmlConverter.convertandRssConverter.convert.Reproduction
New test file, on unmodified
main(packages/markitdown). It lowers the recursion limit around each conversion, as the two existing tests for this failure intest_module_misc.pydo, so the depth needed does not depend on the host:The failures are the fallthrough itself:
The two tests that pass before and after are the pins:
strict=Truestill raisesRecursionErrorout of both converters, and a shallow Wikipedia page still converts to# Shallow/Plain **article** text.With the fix:
Verification
test_module_vectors.pycovers thetest_wikipedia.htmlandtest_serp.htmlend-to-end vectors, andtest_module_misc.pycovers the existing HTML and RSS deep-nesting fallbacks.(The three deselected tests fail on
mainon this Windows machine for unrelated reasons: console encoding, file-URI paths, and a missingffmpeg.)blackreports all three touched files unchanged.