INTPYTHON-676: Adding security and optimization to cache collections #343

linted · 2025-07-17T19:38:36Z

Add HMAC signing of pickled cache data. This implementation uses Blake2b from hashlib to avoid introducing new 3rd party dependencies.

HMAC introduces some overhead to performance, but for cache entries less then 32kb the impact is less then 100ns. For cache entries larger then 1MB, signing can introduce up to 2ms of latency. For BSON serializable types (int, str, bytes), pickling and signing are skipped, and the values are stored in the cache collection directly.

The feature is easily disabled by setting "ENABLE_SIGNING" = False within the CACHE configuration.

Introduced three new cache config options:

ENABLE_SIGNING - boolean value to turn HMAC signing on or off. Defaults to True (on)
SALT - optional 16 character string to salt HMAC signatures with
KEY - optional string to use when performing HMAC operations. This can be up to 64 charcers, and if not provided, SECRET_KEY is used instead.

If the cache fails to validate a signature SuspiciousOperation will be thrown.

…nd loading with greater security.

aclark4life · 2025-07-17T19:43:55Z

Thanks! OK to close #336 ?

timgraham

There's some consensus on the Django forum thread that it could be useful to have pluggable serializers (i.e. CACHES["alias"]["SERIALIZER"] = "path.to.Serializer"). If feasible, this seems like it would lead to a better separation of concerns. Currently the MongoSerializer andMongoDBCache classes seem rather intertwined. That would mean only one additional CACHES configuration value, with the other values like key and salt set on the custom serializer.

I guess I can understand that django.utils.crypto was insufficient for this task. Do you think we could propose some API modernization there? Even if we can't use some Django API right away for th is, I'd be more comfortable in the long run if we weren't "rolling our own crypto" in a way that doesn't get the eyes of Django's security team. (Basically, if Django adopts this sort of thing too, then this project doesn't look so strange.)

timgraham · 2025-07-21T19:32:48Z

docs/source/topics/cache.rst

+Cache entries include a HMAC signature to ensure data integrity by default.
+You can disable this by setting ``ENABLE_SIGNING`` to ``False``.  
+Signatures can also include an optional key and salt parameter by setting 
+``KEY`` and ``SALT`` repectively. Signatures are provided by the Blake2 hash
+function, making Key sizes limited to 64 bytes, and salt sizes limited to 16 
+bytes. If a key is not provided, cache entries will be signed using the 
+``SECRET_KEY``.


I think it's important to be explicit about what "data integrity" means. (i.e. If your database is compromised, it can lead to RCE.)

It should also be made clear that this behavior differs from Django's built-in database backends.

It would be helpful to give some guidance about the performance implications.

Do you have any examples of where performance implications are documented in django? I would love to copy the format. I think it would be hard for me to invent a new format for describing a 2000ns slow down due to signing while also showing that to be only a 3% difference from the original without simply including a large list of numbers.

django_mongodb_backend/cache.py

timgraham · 2025-07-21T19:47:17Z

django_mongodb_backend/cache.py

-        if type(obj) is int:  # noqa: E721
-            return obj
-        return pickle.dumps(obj, self.protocol)
+    def _get_signature(self, data) -> Optional[bytes]:


We currently don't have any type hints in this project (Django hasn't adopted them), so it's a bit out of place to add them in this PR.

I understand that, Django was primarily written prior to type hinting's standardization. I personally feel like it is required here since I am breaking from the standard APIs for some functions. It felt more correct to add them to everything then to only add them to certain functions. I can remove them, but I am worried it will reduce readability and maintainability.

timgraham · 2025-07-21T19:49:13Z

docs/source/topics/cache.rst

@@ -32,6 +32,25 @@ In addition, the cache is culled based on ``CULL_FREQUENCY`` when  ``add()``
 or ``set()`` is called, if ``MAX_ENTRIES`` is exceeded. See
 :ref:`django:cache_arguments` for an explanation of these two options.

+Cache entries include a HMAC signature to ensure data integrity by default.
+You can disable this by setting ``ENABLE_SIGNING`` to ``False``.  
+Signatures can also include an optional key and salt parameter by setting 


What is the use case for custom key and salt? Probably it should be explained that SECRET_KEY is the default.

As suggested by another reviewer, I removed the SALT option since it wasn't providing what I expected it to. I also added the documentation of SECRET_KEY as requested.
I don't have a valid use case for changing the key, but I feel like we would be limiting developers by arbitrarily removing the option when it doesn't really cost anything to keep it. Do you think it's fine as is, or should we brainstorm some reasons to keep/remove it?

timgraham · 2025-07-21T19:52:49Z

django_mongodb_backend/cache.py

+        if pickled:
+            try:
+                if self.signer is not None:
+                    # constant time compare is not required due to how data is retrieved


I'm familiar with the usage of django.utils.crypto.constant_time_compare() but I'm not sure what "how data is retrieved" means in this context.

The threat model for signing data is based on the assumption that a malicious actor has gained write access to the cache collection, but no access (or very very low privilege) to the server Django is running on.
A constant time comparison would prevent the actor from determining the correct hmac hash by measuring the length of time before an error is thrown in a side channel attack. It is my personal opinion that a constant time comparison here is not required due to a minimum of two network requests (client -> server -> database) which should introduce enough entropy to make a short circuit comparison sufficient.
During my testing and profiling of potential solutions, I found that use of constant_time_compare introduced a significant amount of latency, which is what prompted the removal.

If you feel like a side channel attack is a potential issue for this threat model, I am willing to reintroduce the constant time comparison.

timgraham · 2025-07-21T19:54:42Z

django_mongodb_backend/cache.py

+                    "pickled": pickled,
+                    "signature": signature,


I think it's not good to add "pickled" and "signature" keys to all cache data when signing is disabled.

The pickled entry would be required regardless of the the signing option due to the optimizations of allowing unpicked byte strings.
I need to reevaluate if removing the signature field would introduce any issues or vulnerabilites, but I think that is fine.

timgraham · 2025-07-21T23:25:49Z

django_mongodb_backend/cache.py

+            case int() | str() | bytes():
+                return (obj, False, None)


I think this is what you refer to as "optimization". I think any optimization should be done separate from security hardening to make that change more clear. However, I wonder if the benefit of not serializing str/bytes is mostly for the signing case. While it helps with CPU, it adds extra the "pickled" attribute for all keys in the cache. It might be better to limit this optimization a new MongoSignedSerializer class and leave MongoSerializer unchanged. (As our code strays more and more from tried and tested patterns in Django, I feel less confident in its robustness. In this case, the MongoSerializer is copied from django.core.cache.backends.redis.RedisSerializer. We have to ask why Django didn't make this decision and whether it is really better.)

In my testing, unpickled str and bytes are 4x faster during dumps and 20x faster during loads. We would see similar performance increases for dict, list and tuple, but I found it to be slower to check that each element is JSON/BSON serializable then to simply pickle and sign them.

This is one of the optimizations that I made, however, it is the least impactful overall. The majority of the speed increases come from switching to directly using python's blake2b algorithm over Django's Django.core.signing implementation. Django.core.signing is slow by comparison in part due to the algorithm implementation and because of string concatenation between the HMAC hash and the algorithm choice.

The pickled field adds 10 bytes to every cache request. I personally do not believe it to be a significant overhead for the performance difference, but if we change the field name to simply p we can reduce it to a mere 4 bytes if that is more palatable.

While I would normally agree that copying existing, tested code is better, I believe that the redis code is severely limited by a traditional relational database thought process. I do not believe it is taking advantage of the inherent performance boosts we can gain by using Mongodb's document model. While there are other performance limitations we are forced into due to the Django API (colocation of cached headers and page data), this is one that is easy to remediate due to all code paths being within our own code.

ZacharyEspiritu

Some crypto review comments attached. Let me know if you have any questions!

ZacharyEspiritu · 2025-07-22T23:27:08Z

django_mongodb_backend/cache.py

+        if is_pickled:
+            try:
+                if self.signer is not None:
+                    # constant time compare is not required due to how data is retrieved


I would suggest (if only to avoid having to reason about these subtleties in the future) that you use the constant time comparison always when comparing cryptographic hashes. It is theoretically possible that using a non-constant time comparison for equality might allow someone to more easily craft a valid HMAC for a maliciously crafted payload.

Originally, I opted not to use Django.core.Signing.constant_time_compare since it was roughly 4.5x slower then the short circuit comparison ==. I just reran the tests and found The short circuit comparison to have roughly a 5ns difference between matches/non-matches. Personally, I don't think that is a measurable speed difference when there is multiple network requests occurring and adding entropy, but I understand that I could be missing a potential threat model for this attack.

That said, I did also benchmark hmac.compare_digest which came out as 3x faster then Django's implementation. Since the main concern about these changes are performance related, I think that hmac's implementation is a good alternative and provides the side channel attack safety that is currently lacking.

On a related note, Django's constant_time_compare() is an alias of secrets.compare_digest() (which is an alias of hmac.compare_digest()) except that Django's version coerces the arguments using force_bytes(). This might be artifact of the Python 2/3 transition. No Django tests fail when removing those force_bytes() calls. I'll suggest that Django deprecates constant_time_compare() in favor of hmac.compare_digest().

I implemented constant time compare using hmac.compare_digest. @timgraham, would you rather it be done with Django's implementation?

No, my point was that Django's implementation is just an alias hmac.compare_digest(). (Frankly, if the database is compromised, I think timing attacks would be the least of the problems.)

django_mongodb_backend/cache.py

ZacharyEspiritu · 2025-07-23T13:01:33Z

django_mongodb_backend/cache.py

+
+        self._key = params.get("KEY", settings.SECRET_KEY[:64])
+        if len(self._key) == 0:
+            self._key = settings.SECRET_KEY[:64]


A bit of a subtle thing here. If I'm understanding correctly, SECRET_KEY comes from Django and is a value that is used in various places around Django for cryptographic operations. Since it is used in potentially other cryptographic operations, there are (admittedly, theoretical) "key-reuse" vulnerabilities when you use it as the key for different cryptographic operations (e.g. HMAC and whatever other usecase Django is already using it for).

The most correct thing to do is to derive a key from SECRET_KEY using a key derivation function. Python's hashlib contains pbkdf2_hmac which can be configured to do this for you.

PBKDF2 asks for a "iteration" parameter—if you look it up online, the conventional wisdom for PBKDF2 is to use a very large number of "iterations" because generally PBKDF2 is used for converting human-chosen passwords (which have low entropy) into keys. However, if I'm right in assuming that SECRET_KEY is already a cryptographically-secure random string (e.g. has high entropy), then a single iteration = 1 is sufficient to derive a strong key.

This might look something like this:

from hashlib import pbkdf2_hmac purpose = b'mongodbcachekey' iterations = 1 self._key = pbkdf2_hmac('sha256', settings.SECRET_KEY[:64], purpose, iterations)

@ZacharyEspiritu Interesting suggestion about SECRET_KEY! That's probably something we can do in the project template.

Interesting, I thought this might be solved by setting Blake2b's person field during the hasher initialization. Do you think it would still be useful to have the person field set if we transition to pbkdf2_hmac?

…umentation

INTPYTHON-676: Optimized cache collection to provide faster storing a…

d750611

…nd loading with greater security.

linted mentioned this pull request Jul 17, 2025

INTPYTHON-676 Add optional signing of cache data #336

Closed

linted added 2 commits July 17, 2025 15:53

Fixed missing code block notation in cache docs

ae3986a

fixed missing newline before code block

c874952

timgraham reviewed Jul 21, 2025

View reviewed changes

Renamed unclear variable

0d4cf8b

ZacharyEspiritu reviewed Jul 23, 2025

View reviewed changes

linted added 2 commits July 23, 2025 11:52

Added hmac constant time comparision to hash checking

29080ba

Added pbkdf2 hashing of the cache collection HMAC key and updated doc…

d158b13

…umentation

INTPYTHON-676: Adding security and optimization to cache collections #343

Are you sure you want to change the base?

INTPYTHON-676: Adding security and optimization to cache collections #343

Uh oh!

Conversation

linted commented Jul 17, 2025

Uh oh!

aclark4life commented Jul 17, 2025

Uh oh!

timgraham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZacharyEspiritu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!