Skip to content

fix: keep blank lines inside fenced code blocks - #2446

Open
kevin (kevin9327) wants to merge 1 commit into
microsoft:mainfrom
kevin9327:fix/blank-lines-in-code-blocks
Open

fix: keep blank lines inside fenced code blocks#2446
kevin (kevin9327) wants to merge 1 commit into
microsoft:mainfrom
kevin9327:fix/blank-lines-in-code-blocks

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

Converting a notebook whose code cell holds

def first():
    return 1


def second():
    return 2

produces the same code with the two blank lines between the definitions reduced to one:

'```python\ndef first():\n    return 1\n\ndef second():\n    return 2\n```'

Two blank lines between top-level definitions is what PEP 8 asks for, so this is the common case, not an odd one. The same happens to any <pre> block routed through the HTML converter, and so to DOCX, PPTX, EPUB and RSS content that carries one.

MarkItDown._convert normalizes a converter's result with re.sub(r"\n{3,}", "\n\n", ...). That is right for the ragged whitespace converters leave between blocks, but a fenced code block is the document's own content: its blank lines are data, and rewriting them means the code in the Markdown is not the code in the source.

The collapsing now steps over fenced code blocks. Only a closed fence is treated as a block, so an unterminated one is still normalized, and everything outside a fence collapses exactly as before.

Reproduction

New test file, on unmodified main (packages/markitdown):

$ python -m pytest tests/test_code_block_blank_lines.py -q
FAILED tests/test_code_block_blank_lines.py::test_notebook_code_cell_keeps_its_blank_lines
FAILED tests/test_code_block_blank_lines.py::test_html_code_block_keeps_its_blank_lines
2 failed, 3 passed in 1.56s

with the failure being the rewritten code:

>       assert CODE in markdown
E       AssertionError: assert 'def first():\n    return 1\n\n\ndef second():\n    return 2'
E         in '```\ndef first():\n    return 1\n\ndef second():\n    return 2\n```'

The three tests that pass before and after are the pins: two paragraphs still separate with exactly one blank line, the blank lines around a code block still collapse, and an unterminated fence is still normalized.

With the fix:

$ python -m pytest tests/test_code_block_blank_lines.py -q
5 passed in 1.29s

Verification

Whole suite, before and after — the five new tests are the whole difference:

$ python -m pytest tests -q -k "not test_output_to_stdout and not test_input_from_stdin_without_hints \
    and not test_convert_url and not file_uris and not case_insensitive and not speech"
607 passed, 4 skipped, 39 deselected in 82.30s   # origin/main
612 passed, 4 skipped, 39 deselected in 82.64s   # this branch

(The deselected tests fail on main on this Windows machine for reasons unrelated to the change: console encoding, file-URI paths, and a missing ffmpeg.)

black reports both touched files unchanged.

The output normalization collapsed every run of blank lines, including
the ones inside a code fence, so a notebook cell or a <pre> block that
separated two definitions with two blank lines came back with one. The
collapsing now steps over closed fenced blocks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants