Skip to content

fix(ipynb): label code fences with the notebook's own language - #2443

Open
kevin (kevin9327) wants to merge 1 commit into
microsoft:mainfrom
kevin9327:fix/ipynb-code-language
Open

fix(ipynb): label code fences with the notebook's own language#2443
kevin (kevin9327) wants to merge 1 commit into
microsoft:mainfrom
kevin9327:fix/ipynb-code-language

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

Converting an R notebook produces

```R
plot(cars)
```

as

```python
plot(cars)
```

The same happens to Julia, Scala, C#, Bash and every other kernel: IpynbConverter._convert writes the fence with a hard-coded python info string, whatever the notebook says it is. The output then tells a reader — and any downstream renderer or model — that the code is Python.

nbformat records the notebook's language in metadata.language_info.name, which is the field nbconvert reads; metadata.kernelspec.language is the kernel's own declaration and is all some producers write. This reads the first, falls back to the second, and falls back to python when a notebook declares neither, which is what the converter has always emitted.

Only the first word of the declaration is used, since a fence's info string ends at the first whitespace.

Reproduction

New test file, on unmodified main (packages/markitdown):

$ python -m pytest tests/test_ipynb_code_language.py -q
FAILED tests/test_ipynb_code_language.py::test_language_info_names_the_fence
FAILED tests/test_ipynb_code_language.py::test_kernelspec_language_is_used_when_language_info_is_absent
2 failed, 2 passed in 2.61s

with the failure showing the wrong label:

>       assert "```R\n" + SOURCE in markdown
E       AssertionError: assert ('```R\n' + 'plot(cars)\n') in '```python\nplot(cars)\n\n```'

The two tests that pass before and after are the ones that pin the existing behaviour: a notebook declaring python, and a notebook declaring no language at all, both still get python.

With the fix:

$ python -m pytest tests/test_ipynb_code_language.py -q
4 passed in 2.22s

Verification

tests/test_module_vectors.py includes the end-to-end test_notebook.ipynb vector, which requires ```python — that notebook declares language_info.name: "python", so it keeps its label.

$ python -m pytest tests/test_module_vectors.py tests/test_module_misc.py -q \
    -k "not file_uris and not case_insensitive and not speech"
183 passed, 2 skipped, 3 deselected in 63.85s   # origin/main
183 passed, 2 skipped, 3 deselected in 61.26s   # this branch

(The three deselected tests fail on main on this Windows machine for reasons unrelated to notebooks: console encoding, file-URI paths, and a missing ffmpeg.)

black reports both touched files unchanged.

Code cells were always fenced as ```python, so an R, Julia or C#
notebook came out claiming to be Python. Read the language nbformat
records in metadata.language_info.name, falling back to the kernel's
own metadata.kernelspec.language and then to python.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant