Make QuerySet.explain() return parsable JSON #340

Jibola · 2025-07-16T17:49:30Z

Context

Calling explain() is extremely useful in debugging a MongoDB query. However, parsing the output of the explain call is a nightmare. This is because we take each line and format them using pprint to accommodate Django's native explain functionality, which joins all information line by line.

Solution

Rather than split key/values into multiple lines, we should just dump the json as one string blob in the list. This way .explain can easily leverage json.load or json_util.load.

Confirm the fix
Create a test_explain test case
Update the changelog

Changes in this PR

import PyMongo library json_util
call json_util.dumps(..., indent=4) and return that in a list of length 1.

Change

Before

>>> exp = Author.objects.filter().explain()
>>> json.loads(exp)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
...
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

After

aclark4life

LGTM, less pprint too!

timgraham · 2025-07-16T18:52:46Z

To me, it's not an improvement in readability.

Before:

>>> print(Question.objects.explain())
explainVersion: '1'
queryPlanner: {   'indexFilterSet': False,
    'maxIndexedAndSolutionsReached': False,
    'maxIndexedOrSolutionsReached': False,
    'maxScansToExplodeReached': False,
    'namespace': 'mysite.polls_question',
    'optimizationTimeMillis': 0,
    'optimizedPipeline': True,
    'parsedQuery': {},
    'planCacheKey': '7DF350EE',
    'planCacheShapeHash': '8F2383EE',
    'prunedSimilarIndexes': False,
    'queryHash': '8F2383EE',
    'rejectedPlans': [],
    'winningPlan': {   'direction': 'forward',
                       'isCached': False,
                       'stage': 'COLLSCAN'}}
executionStats: {   'allPlansExecution': [],
    'executionStages': {   'advanced': 0,
                           'direction': 'forward',
                           'docsExamined': 0,
                           'executionTimeMillisEstimate': 0,
                           'isCached': False,
                           'isEOF': 1,
                           'nReturned': 0,
                           'needTime': 0,
                           'needYield': 0,
                           'restoreState': 0,
                           'saveState': 0,
                           'stage': 'COLLSCAN',
                           'works': 1},
    'executionSuccess': True,
    'executionTimeMillis': 0,
    'nReturned': 0,
    'totalDocsExamined': 0,
    'totalKeysExamined': 0}
queryShapeHash: '7229101CA7C854EFFD9939CFFED9E674B0B07394314E0D9379C20096DE409F8A'
command: {   '$db': 'mysite',
    'aggregate': 'polls_question',
    'cursor': {},
    'pipeline': [{'$match': {'$expr': {}}}]}
serverInfo: {   'gitVersion': 'bed99f699da6cb2b74262aa6d473446c41476643',
    'host': 'barkley',
    'port': 27017,
    'version': '8.0.11'}
serverParameters: {   'internalDocumentSourceGroupMaxMemoryBytes': 104857600,
    'internalDocumentSourceSetWindowFieldsMaxMemoryBytes': 104857600,
    'internalLookupStageIntermediateDocumentMaxSizeBytes': 104857600,
    'internalQueryFacetBufferSizeBytes': 104857600,
    'internalQueryFacetMaxOutputDocSizeBytes': 104857600,
    'internalQueryFrameworkControl': 'trySbeRestricted',
    'internalQueryMaxAddToSetBytes': 104857600,
    'internalQueryMaxBlockingSortMemoryUsageBytes': 104857600,
    'internalQueryPlannerIgnoreIndexWithCollationForRegex': 1,
    'internalQueryProhibitBlockingMergeOnMongoS': 0}

After:

>>> print(Question.objects.explain())
{"explainVersion": "1", "queryPlanner": {"namespace": "mysite.polls_question", "parsedQuery": {}, "indexFilterSet": false, "queryHash": "8F2383EE", "planCacheShapeHash": "8F2383EE", "planCacheKey": "7DF350EE", "optimizationTimeMillis": 0, "optimizedPipeline": true, "maxIndexedOrSolutionsReached": false, "maxIndexedAndSolutionsReached": false, "maxScansToExplodeReached": false, "prunedSimilarIndexes": false, "winningPlan": {"isCached": false, "stage": "COLLSCAN", "direction": "forward"}, "rejectedPlans": []}, "executionStats": {"executionSuccess": true, "nReturned": 0, "executionTimeMillis": 1, "totalKeysExamined": 0, "totalDocsExamined": 0, "executionStages": {"isCached": false, "stage": "COLLSCAN", "nReturned": 0, "executionTimeMillisEstimate": 0, "works": 1, "advanced": 0, "needTime": 0, "needYield": 0, "saveState": 0, "restoreState": 0, "isEOF": 1, "direction": "forward", "docsExamined": 0}, "allPlansExecution": []}, "queryShapeHash": "7229101CA7C854EFFD9939CFFED9E674B0B07394314E0D9379C20096DE409F8A", "command": {"aggregate": "polls_question", "pipeline": [{"$match": {"$expr": {}}}], "cursor": {}, "$db": "mysite"}, "serverInfo": {"host": "barkley", "port": 27017, "version": "8.0.11", "gitVersion": "bed99f699da6cb2b74262aa6d473446c41476643"}, "serverParameters": {"internalQueryFacetBufferSizeBytes": 104857600, "internalQueryFacetMaxOutputDocSizeBytes": 104857600, "internalLookupStageIntermediateDocumentMaxSizeBytes": 104857600, "internalDocumentSourceGroupMaxMemoryBytes": 104857600, "internalQueryMaxBlockingSortMemoryUsageBytes": 104857600, "internalQueryProhibitBlockingMergeOnMongoS": 0, "internalQueryMaxAddToSetBytes": 104857600, "internalDocumentSourceSetWindowFieldsMaxMemoryBytes": 104857600, "internalQueryFrameworkControl": "trySbeRestricted", "internalQueryPlannerIgnoreIndexWithCollationForRegex": 1}, "ok": 1.0}

WaVEV · 2025-07-17T02:42:24Z

django_mongodb_backend/compiler.py

-            result.append(f"{key}: {formatted_value}")
-        return result
+        # explain() expects a list and joins on a newline. Concatenate no lines
+        return [json_util.dumps(explain)]


I think you can use

Suggested change

return [json_util.dumps(explain)]

return [json_util.dumps(explain, indent=4, ensure_ascii=False)]

Since json_util.dumps() is a pymongo specific json parsing function, I don't think we'll need the ensure_ascii=False override.

Thoughts?

I'm not sure about ensure_ascii, but I'm wondering if you expect some difference in the output by using json_util.dumps() instead of json.dumps()?

We didn't have tests in this repo when I originally implemented this, but it would be useful to now have at least one test of the output in tests/queries_/test_explain.py (new file).

I'm fine with ident=4 but just to be clear, that introduces the same "nightmare to parse" issue of newlines. Basically, your original usage mistake was not using print(Model.objects.explain()) so the newlines weren't rendered nicely. I think it's fine to make this change anyway. Incidentally, is there a use case for calling json.loads() on the result of explain() or was that just an attempt at making the output more readable?

I'm not sure about ensure_ascii, but I'm wondering if you expect some difference in the output by using json_util.dumps() instead of json.dumps()?

json_util properly handles the case of non-json-serializable BSON types (I.e. ObjectId()). We know we're getting a dictionary from a mongodb query and we want it to be parseable. This allows everything to be viewed, and if it ever fails, that's a bug against PyMongo rather than this library.

We didn't have tests in this repo when I originally implemented this, but it would be useful to now have at least one test of the output in tests/queries_/test_explain.py (new file).

Sure. I can add that.

I'm fine with ident=4 but just to be clear, that introduces the same "nightmare to parse" issue of newlines. ...

It actually doesn't. It does keep the \n ticks in pprint, but in a much more readable way.
For me it's also a QOL issue. For exceptionally large queries, it gets nauseating to read an entire rendered print output so my standard workflow is adding it to a dictionary and then iterate through the query each piece. So in this new world all three paths become viable:

print continues to work as designed

loads can now give a manageable dictionary

pprint gives an arguably easier to parse JSON blob. .

ensure_ascii option was to handling some letters, like Spanish letter ñ, or ó, and so on. If this flag was in true, those letter got broken.

WaVEV

json dump can be prettified

Jibola · 2025-07-17T14:42:17Z

To me, it's not an improvement in readability.

Before:

>>> print(Question.objects.explain())
explainVersion: '1'
queryPlanner: {   'indexFilterSet': False,
    'maxIndexedAndSolutionsReached': False,
    'maxIndexedOrSolutionsReached': False,
    'maxScansToExplodeReached': False,
    'namespace': 'mysite.polls_question',
    'optimizationTimeMillis': 0,
    'optimizedPipeline': True,
    'parsedQuery': {},
    'planCacheKey': '7DF350EE',
    'planCacheShapeHash': '8F2383EE',
    'prunedSimilarIndexes': False,
    'queryHash': '8F2383EE',
    'rejectedPlans': [],
    'winningPlan': {   'direction': 'forward',
                       'isCached': False,
                       'stage': 'COLLSCAN'}}
executionStats: {   'allPlansExecution': [],
    'executionStages': {   'advanced': 0,
                           'direction': 'forward',
                           'docsExamined': 0,
                           'executionTimeMillisEstimate': 0,
                           'isCached': False,
                           'isEOF': 1,
                           'nReturned': 0,
                           'needTime': 0,
                           'needYield': 0,
                           'restoreState': 0,
                           'saveState': 0,
                           'stage': 'COLLSCAN',
                           'works': 1},
    'executionSuccess': True,
    'executionTimeMillis': 0,
    'nReturned': 0,
    'totalDocsExamined': 0,
    'totalKeysExamined': 0}
queryShapeHash: '7229101CA7C854EFFD9939CFFED9E674B0B07394314E0D9379C20096DE409F8A'
command: {   '$db': 'mysite',
    'aggregate': 'polls_question',
    'cursor': {},
    'pipeline': [{'$match': {'$expr': {}}}]}
serverInfo: {   'gitVersion': 'bed99f699da6cb2b74262aa6d473446c41476643',
    'host': 'barkley',
    'port': 27017,
    'version': '8.0.11'}
serverParameters: {   'internalDocumentSourceGroupMaxMemoryBytes': 104857600,
    'internalDocumentSourceSetWindowFieldsMaxMemoryBytes': 104857600,
    'internalLookupStageIntermediateDocumentMaxSizeBytes': 104857600,
    'internalQueryFacetBufferSizeBytes': 104857600,
    'internalQueryFacetMaxOutputDocSizeBytes': 104857600,
    'internalQueryFrameworkControl': 'trySbeRestricted',
    'internalQueryMaxAddToSetBytes': 104857600,
    'internalQueryMaxBlockingSortMemoryUsageBytes': 104857600,
    'internalQueryPlannerIgnoreIndexWithCollationForRegex': 1,
    'internalQueryProhibitBlockingMergeOnMongoS': 0}

After:

>>> print(Question.objects.explain())
{"explainVersion": "1", "queryPlanner": {"namespace": "mysite.polls_question", "parsedQuery": {}, "indexFilterSet": false, "queryHash": "8F2383EE", "planCacheShapeHash": "8F2383EE", "planCacheKey": "7DF350EE", "optimizationTimeMillis": 0, "optimizedPipeline": true, "maxIndexedOrSolutionsReached": false, "maxIndexedAndSolutionsReached": false, "maxScansToExplodeReached": false, "prunedSimilarIndexes": false, "winningPlan": {"isCached": false, "stage": "COLLSCAN", "direction": "forward"}, "rejectedPlans": []}, "executionStats": {"executionSuccess": true, "nReturned": 0, "executionTimeMillis": 1, "totalKeysExamined": 0, "totalDocsExamined": 0, "executionStages": {"isCached": false, "stage": "COLLSCAN", "nReturned": 0, "executionTimeMillisEstimate": 0, "works": 1, "advanced": 0, "needTime": 0, "needYield": 0, "saveState": 0, "restoreState": 0, "isEOF": 1, "direction": "forward", "docsExamined": 0}, "allPlansExecution": []}, "queryShapeHash": "7229101CA7C854EFFD9939CFFED9E674B0B07394314E0D9379C20096DE409F8A", "command": {"aggregate": "polls_question", "pipeline": [{"$match": {"$expr": {}}}], "cursor": {}, "$db": "mysite"}, "serverInfo": {"host": "barkley", "port": 27017, "version": "8.0.11", "gitVersion": "bed99f699da6cb2b74262aa6d473446c41476643"}, "serverParameters": {"internalQueryFacetBufferSizeBytes": 104857600, "internalQueryFacetMaxOutputDocSizeBytes": 104857600, "internalLookupStageIntermediateDocumentMaxSizeBytes": 104857600, "internalDocumentSourceGroupMaxMemoryBytes": 104857600, "internalQueryMaxBlockingSortMemoryUsageBytes": 104857600, "internalQueryProhibitBlockingMergeOnMongoS": 0, "internalQueryMaxAddToSetBytes": 104857600, "internalDocumentSourceSetWindowFieldsMaxMemoryBytes": 104857600, "internalQueryFrameworkControl": "trySbeRestricted", "internalQueryPlannerIgnoreIndexWithCollationForRegex": 1}, "ok": 1.0}

Ah, to @WaVEV 's point, that should be mitigated with indent=4 I'll update the before & after to reflect.

WaVEV

LGTM, the only thing: what if we have a character like ñ

Co-authored-by: Tim Graham <[email protected]>

Jibola requested review from timgraham, aclark4life and WaVEV July 16, 2025 17:49

aclark4life approved these changes Jul 16, 2025

View reviewed changes

WaVEV approved these changes Jul 17, 2025

View reviewed changes

WaVEV reviewed Jul 17, 2025

View reviewed changes

WaVEV requested changes Jul 17, 2025

View reviewed changes

timgraham changed the title ~~Make explain() yield a mongodb-compatible dumped json~~ Make QuerySet.explain() return JSON Jul 18, 2025

Jibola requested a review from WaVEV July 18, 2025 16:53

WaVEV approved these changes Jul 18, 2025

View reviewed changes

timgraham changed the title ~~Make QuerySet.explain() return JSON~~ Make QuerySet.explain() return parsable JSON Jul 19, 2025

timgraham force-pushed the simplify-explain branch 2 times, most recently from 6706112 to abc335f Compare July 19, 2025 22:59

Make QuerySet.explain() return parsable JSON

ced5649

Co-authored-by: Tim Graham <[email protected]>

timgraham force-pushed the simplify-explain branch from abc335f to ced5649 Compare July 19, 2025 23:16

Jibola merged commit 6b5d00c into main Jul 21, 2025
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make QuerySet.explain() return parsable JSON #340

Make QuerySet.explain() return parsable JSON #340

Uh oh!

Jibola commented Jul 16, 2025 •

edited by timgraham

Loading

Uh oh!

aclark4life left a comment

Uh oh!

timgraham commented Jul 16, 2025

Uh oh!

WaVEV Jul 17, 2025 •

edited

Loading

Uh oh!

Jibola Jul 17, 2025

Uh oh!

timgraham Jul 17, 2025

Uh oh!

Jibola Jul 17, 2025 •

edited

Loading

Uh oh!

WaVEV Jul 18, 2025

Uh oh!

WaVEV left a comment •

edited

Loading

Uh oh!

Jibola commented Jul 17, 2025

Uh oh!

WaVEV left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

	return [json_util.dumps(explain)]
	return [json_util.dumps(explain, indent=4, ensure_ascii=False)]

Make QuerySet.explain() return parsable JSON #340

Make QuerySet.explain() return parsable JSON #340

Uh oh!

Conversation

Jibola commented Jul 16, 2025 • edited by timgraham Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Solution

Changes in this PR

Change

Before

After

Uh oh!

aclark4life left a comment

Choose a reason for hiding this comment

Uh oh!

timgraham commented Jul 16, 2025

Uh oh!

WaVEV Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jibola Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

timgraham Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Jibola Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WaVEV Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

WaVEV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jibola commented Jul 17, 2025

Uh oh!

WaVEV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jibola commented Jul 16, 2025 •

edited by timgraham

Loading

WaVEV Jul 17, 2025 •

edited

Loading

Jibola Jul 17, 2025 •

edited

Loading

WaVEV left a comment •

edited

Loading

WaVEV left a comment •

edited

Loading