Allow plain text `.qmd` files as source notebooks #1521

bhoov · 2025-06-03T21:28:24Z

Addresses #1461

Using quarto and its VS code extension, I find that writing .qmd files to be a smoother interactive alternative to .ipynb files. That .qmd files are plain text comes with several advantages:

.qmd seamlessly integrates with Cursor AI/other AI copilots.
.qmd is fully compatible with standard git tooling
.qmd works better with VIM keybindings
.qmd files don't need a special nbdev_clean step to remove cell metadata and outputs, meaning your source files are not altered in any way by nbdev's transpilation process (something that bothers me immensely when developing in .ipynb)

Turns out, nbdev doesn't need many changes to implement this feature.

Allow export globbing functions to search for .qmd in addition to .ipynb
Implement a read_qmd/write_qmd function for converting the .qmd to/from nbdev's AttrDict format. This means two-way sync (via nbdev_update) also works for .qmd and its corresponding .py files.
Because outputs are not stored inside .qmd files, I use execnb's run_all to generate outputs for the docs inside _proc/-cached .ipynb files.
The custom frontmatter parser needed some tweaking to allow cells to include general markdown after the custom frontmatter.

It looks like there have been other attempts to allow .qmd support for nbdev (see this quarto issue) or allow plain-text support (see #1499). However, .qmd support is still missing in the current version of nbdev, and the latter seems to introduce jupytext as an additional dependency which uses the slow quarto convert command to pair a .ipynb and .qmd (this PR introduces a faster .qmd <-> .ipynb parser). Now you can seamlessly develop using a mix of .qmd and .ipynb, whichever you prefer, with no additional dependencies.

I've written up a small tutorial for setting good VSCode defaults in nbs/tutorials/develop_in_plain_text.qmd

A few notes of caution and room for improvement:

Ensure that all files under nbs/ have distinct names: no 00_core.ipynb and 00_core.qmd, as both of these will create the intermediate _proc/00_core.ipynb
Currently, the nbdev_prepare will run executable cells in .qmd documents twice: 1x when testing and, because outputs aren't saved, 1x when generating the docs.

The PR is in a pretty stable position already (see this fork of nbdev rewritten entirely using .qmd files). There may be edge cases that I haven't considered, but in all I hope this is nearing a good shape to distribute.

…not key: vals from lists

…on cells)

- Adopt Quarto for documentation and notebooks making use of [this nbdev PR](AnswerDotAI/nbdev#1521) that allows full `.qmd` driven packages - Convert all `ipynb` files to `.qmd` format - Use nbdev_docs to generate the documentation website - Adopt logger that solves #3 (#3)

football-kowshik · 2025-09-13T14:52:37Z

What are the next steps here?

TinasheMTapera · 2025-09-20T16:50:35Z

I may be having challenges with this, but just wanted to check to see if you've seen this before or if it's something external to your fork:

nbdev_proc_nbs:

"""
Traceback (most recent call last):
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/process.py", line 210, in _process_chunk
    return [fn(*args) for args in chunk]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/process.py", line 210, in <listcomp>
    return [fn(*args) for args in chunk]
            ^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/fastcore/parallel.py", line 63, in _call
    return g(item)
           ^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/serve_drv.py", line 35, in main
    elif src.suffix=='.qmd': exec_qmd(src, dst, x)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/serve_drv.py", line 23, in exec_qmd
    cb()(nb)
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/processors.py", line 292, in __call__
    def __call__(self, nb): return self.nb_proc(nb).process()
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/process.py", line 130, in process
    for proc in self.procs: self._proc(proc)
                            ^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/process.py", line 122, in _proc
    if hasattr(proc,'begin'): proc.begin()
                              ^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/processors.py", line 108, in begin
    if getattr(cells[idx+1], 'has_sd', 0):
               ~~~~~^^^^^^^
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/bin/nbdev_proc_nbs", line 8, in <module>
    sys.exit(nbdev_proc_nbs())
             ^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/fastcore/script.py", line 125, in _f
    return tfunc(**merge(args, args_from_prog(func, xtra)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/quarto.py", line 217, in nbdev_proc_nbs
    _pre_docs(**kwargs)[0]
    ^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/quarto.py", line 209, in _pre_docs
    cache = proc_nbs(path, n_workers=n_workers, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/nbdev/serve.py", line 82, in proc_nbs
    parallel(nbdev.serve_drv.main, files, n_workers=n_workers, pause=0.01, **kw)
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/fastcore/parallel.py", line 134, in parallel
    return L(r)
           ^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/fastcore/foundation.py", line 100, in __call__
    return super().__call__(x, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/fastcore/foundation.py", line 108, in __init__
    items = listify(items, *rest, use_list=use_list, match=match)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/site-packages/fastcore/basics.py", line 79, in listify
    elif is_iter(o): res = list(o)
                           ^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/process.py", line 620, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/n/home03/ttapera/.conda/envs/era5_sandbox/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
IndexError: list index out of range

Any thoughts? What else would you like to see to help debug?

bhoov · 2025-09-22T02:11:39Z

I got this PR to work for my personal use cases and didn't see much initial interest on this PR to bring it into the main branch. Seems like there's gotten to be a bit more traction since I first made the PR, and I'm happy to push this forward.

What are the next steps here? @football-kowshik

From my side, it has been awhile since I've rebased with the main. I will do that and see what bugs/clashes have come up since then and try to resolve those. Beyond that it's up to the maintainers to see if this is worth incorporating into the main branch (I think it definitely is, but I am biased. The .qmd workflow has proven much smoother for my use cases and it is fully backward compatible with .ipynbs.)

@TinasheMTapera I am not positive, but this bug looks a lot like the weird edge cases I encountered when trying to parse .qmd files as valid nbdev source. Could you share a minimal .qmd file that reproduces this bug? I'm a bit new at contributing to larger OSS projects on github, but I feel that this bug doesn't need its own issue since it is pertinent only to this PR.

kurianbenoy-sarvam · 2025-10-15T09:45:55Z

@bhoov you haven't requested Jeremy Howard or any of the maintainers to review the PR.

I recently asked in discord, why this PR was not reviewed and the answer I received from Jeremy was:

No one requested a review so i didn't see it 🙂

TinasheMTapera · 2025-10-16T16:15:12Z

@bhoov I was able to figure out the problem, and it was not related to your PR, but rather to an edge case of nbdev itself, so please disregard!

bhoov · 2025-10-20T21:28:37Z

I recently asked in discord, why this PR was not reviewed and the answer I received from Jeremy was:

No one requested a review so i didn't see it 🙂

Github actually does not allow me to request reviewers or assign people to this repository, otherwise I would have.

@jph00 can you review this PR? :)

jph00 · 2025-10-20T21:33:33Z

Will do!

jph00

Thanks for this. Tbh it's a long way from being something that we could merge at this stage. Rather than try to get this PR into shape, my suggestion would be to gradually add a series of PRs which have the minimal code necessary for each piece. Start with something that adds the most useful bit you need with the least amount of code.

jph00 · 2025-10-29T05:44:40Z

nbdev/clean.py

    if stdin: return _write(f_in=sys.stdin, f_out=sys.stdout)
    if fname is None: fname = get_config().nbs_path
-    for f in globtastic(fname, file_glob='*.ipynb', skip_folder_re='^[_.]'): _write(f_in=f, disp=disp)
+    for f in globtastic(fname, file_re=r'.*\.ipynb$', skip_folder_re='^[_.]'): _write(f_in=f, disp=disp) # Don't clean .qmd files


can we leave the glob as it was then?

jph00 · 2025-10-29T05:49:14Z

nbdev/frontmatter.py

    def cell(self, cell):
        if cell.cell_type=='raw': self._update(_fm2dict, cell)
-        elif cell.cell_type=='markdown' and 'title' not in self.fm: self._update(_md2dict, cell)
+        elif (cell.cell_type=='markdown' and 'title' not in self.fm): self._update(_md2dict, cell)


not sure this needs changing?

jph00 · 2025-10-29T05:51:54Z

nbdev/frontmatter.py

 _re_fm_nb = re.compile(_RE_FM_BASE+'$', flags=re.DOTALL)
 _re_fm_md = re.compile(_RE_FM_BASE, flags=re.DOTALL)

+_re_fm_title_desc = re.compile(r'^#\s+(\S.*?)(?:\n|$)(?:\s*\n)*(?:>\s+(\S.*?)(?:\n|$)(?:\s*\n)*)?', flags=re.MULTILINE)


This has gotten far too complicated. Perhaps reduce the scope to only support frontmatter titles etc in qmd? Try to get to a PR that adds as little code as possible, and keeps the code no more complex than what we had before.

jph00 · 2025-10-29T05:54:55Z

nbdev/clean.py

    check_fname = path/".last_checked"
    last_checked = os.path.getmtime(check_fname) if check_fname.exists() else None
-    nbs = globtastic(fname, file_glob='*.ipynb', skip_folder_re='^[_.]') if fname.is_dir() else [fname]
+    nbs = globtastic(fname, file_re=r'.*\.ipynb$|.*\.qmd$', skip_folder_re='^[_.]') if fname.is_dir() else [fname]


We could simplify this in the multiple places it occurs, eg:

Suggested change

nbs = globtastic(fname, file_re=r'.*\.ipynb$|.*\.qmd$', skip_folder_re='^[_.]') if fname.is_dir() else [fname]

nbs = globtastic(fname, file_re=r'.*\.(ipynb|qmd)$', skip_folder_re='^[_.]') if fname.is_dir() else [fname]

Although it feels like a glob would be nicer. What if we added a file_exts param to globtastic which took a comma separated list of file extensions? Then we could just have file_exts='ipynb,qmd'.

jph00 · 2025-10-29T05:58:46Z

nbdev/process.py

    "Process cells and nbdev comments in a notebook"
    def __init__(self, path=None, procs=None, nb=None, debug=False, rm_directives=True, process=False):
-        self.nb = read_nb(path) if nb is None else nb
+        self.nb = read_nb_or_qmd(path) if nb is None else nb


Probably not worth breaking backwards incompatibility to change the name of this function. It's OK if read_nb is slightly misleadingly named :)

jph00 · 2025-10-29T05:59:08Z

nbdev/qmd.py

+import re
+
+import sys,os,inspect,shutil


Suggested change

import re

import sys,os,inspect,shutil

import sys,os,inspect,shutil,re

jph00 · 2025-10-29T19:07:10Z

nbdev/qmd.py

+            if intermediate_md_source: raw_cells.append(_qmd_to_raw_cell(intermediate_md_source, 'markdown'))
+
+    # Construct the final notebook dictionary
+    notebook_dict = {


Don't hard code dicts like this. Use the functions we have.

jph00 · 2025-10-29T19:08:03Z

nbdev/qmd.py

+
+
+# %% ../nbs/api/15_qmd.ipynb
+def _get_fence_ticks(source:str):


Use a proper well-tested parser - don't do this kind of thing manually.

jph00 · 2025-10-29T19:08:50Z

nbdev/qmd.py

+
+# %% ../nbs/api/15_qmd.ipynb
+@call_parse
+def ipynb_to_qmd(


Too much code here - and it shouldn't be printing on success. Keep things much simpler! :)

bhoov · 2025-10-29T19:44:45Z

Thanks for the review Jeremy. I definitely prioritized feature completeness at the expense of tasteful code and minimal changes 🙃. Still new to contributing to larger existing projects, I'll get there

Do you suggest closing this mammoth PR and instead introduce bite-sized PRs to the main branch? Or should I make smaller PRs to a dedicated "qmd_support" branch?

jph00 · 2025-10-29T21:17:20Z

I'd suggest closing this and do a single small PR, and work through that together first. Thanks for your patience! :)

…

Message ID: ***@***.***>

bhoov added 30 commits May 29, 2025 11:51

Add .qmd read functionality to processor

c93ea72

Fix file glob for nbdev to detect .qmd files as source nbs

bf8c091

Don't clean .qmd files

fceff8f

Simplify read_qmd in style of nbdev code

f113f14

Allow custom directives in .qmd files

3a8d61b

Attempt .qmd to generate good looking docs

c7389eb

Update sidebar.yml to look for .ipynb instead of .qmd in cache folder

c477249

Execute .qmd files when rendering

82d3205

Cleanup function in succinct style of nbdev

b98f5ee

Cleanup debugging statements

6109227

Enable back-syncing qmd from changes in .py files

8a32180

Only use official frontmatter for .qmd documents

98f118d

Add documentation for no custom YAML for qmd

8b98a61

Move qmd processing to 15_qmd.ipynb

d43cf36

Allow .qmd frontmatter to still auto-calc title and description, but …

adf7039

…not key: vals from lists

Create script to convert .ipynb to .qmd

e025097

Add script to convert ipynb to qmd

19788db

Allow markdown after custom frontmatter

33ad5f1

Pre-compile fm regex

b124c76

Improve converting from .ipynb to .qmd (account for backticks in pyth…

33a10cb

…on cells)

.qmd files now read custom KV yaml

bdfb1db

Custom frontmatter now parsed correctly for docs

86354d9

Cleanup frontmatter code

04c87d5

Copy .qmd for docs

deebd53

Update nbs

f49bd4b

Allow frontmatter keys to contain hyphens

90db296

Fix index.qmd not found

c503fc4

Initial draft of .qmd file tutorial

cb5a410

Add annotations to custom functions

f27e408

Update tutorial

e510a85

Cleanup

4f23ca8

bhoov mentioned this pull request Jun 6, 2025

use .qmd as as full replacement of .ipynb #1461

Open

bhoov added 4 commits July 3, 2025 14:59

Merge branch 'main' into qmd_support

f893026

Allow qmd to ipynb conversion

f10b4e9

Add support for ipynb in qmd directory

2e425db

Merge branch 'main' into qmd_support

f96f3f8

jph00 self-requested a review October 20, 2025 21:34

jph00 requested changes Oct 29, 2025

View reviewed changes

Merge branch 'main' into qmd_support

bc2043c

	nbs = globtastic(fname, file_re=r'.\.ipynb$\|.\.qmd$', skip_folder_re='^[_.]') if fname.is_dir() else [fname]
	nbs = globtastic(fname, file_re=r'.*\.(ipynb\|qmd)$', skip_folder_re='^[_.]') if fname.is_dir() else [fname]



		# %% ../nbs/api/15_qmd.ipynb
		def _get_fence_ticks(source:str):

Allow plain text .qmd files as source notebooks #1521

Are you sure you want to change the base?

Allow plain text .qmd files as source notebooks #1521

Uh oh!

Conversation

bhoov commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

football-kowshik commented Sep 13, 2025

Uh oh!

TinasheMTapera commented Sep 20, 2025

Uh oh!

bhoov commented Sep 22, 2025

Uh oh!

kurianbenoy-sarvam commented Oct 15, 2025

Uh oh!

TinasheMTapera commented Oct 16, 2025

Uh oh!

bhoov commented Oct 20, 2025

Uh oh!

jph00 commented Oct 20, 2025

Uh oh!

jph00 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bhoov commented Oct 29, 2025

Uh oh!

jph00 commented Oct 29, 2025 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Allow plain text `.qmd` files as source notebooks #1521

Allow plain text `.qmd` files as source notebooks #1521

bhoov commented Jun 3, 2025 •

edited

Loading