FakeRedis as a base for an SQL-db backed Redis Replacement?

[gabriels1234]

I'd like to gather thoughts on this Idea: use fakeredis as a base to make a DB powered version compatible with redis.
(cc/ @cunla )
Motivation: Databases are quite fast. Many projects don't actually need Redis (For cache).

Also, Redis is the preferred (only!) backend for RQ (where R stands for Redis!). But Database Speed is more than enough for most rq use-cases. Also, there are unaddressed concerns about robustness/reliability of RQ (a.k.a lost jobs) that made developers move away from RQ. This should be solved but the Atomic-style of Transactional Databases.

Last but not least: You can get practically the same results with webserver+DB( Postgres!) ($x2) than webserver+DB+Redis ($x3). I believe we should allow the small guy to play in the same league at a fraction of the price, whenever possible.

Is it something worth pursuing? is FakeRedis the perfect match for this project? How to achieve it, by forking the repo or by augmenting FakeRedis to support a DB Backend? I'm sure this will become the new default for RQ setups.

[cunla]

I think you are looking for the pq package.

[gabriels1234]

@cunla Well, since I work mostly within the RQ ecosystem/stack, it makes a lot of sense to try to make a "drop-in" replacement for Redis() that allows to keep using everything else the same way.
Worker, etc and the peace of mind that "It just works", I think it's worth the persuit.

Do you see any limitations/problems of this approach?
Thanks!

[cunla]

I can't see any issue implementing it, though it requires a deeper design in order to answer it better.
You can definitely pursue it. Call it redis-in-postgres or something similar :)

[gabriels1234]

Hi @cunla !
I've managed to create a Custom Database Class (with a few other tweaks to fakeredis) and have it (apparently) working with SQLite (with transactional atomicity).
Question (perhaps you know)
for a simple

r1.set('foo', 'bar')
r1.get('foo')  # <--- 

the __getitem__ for Database (also for my Sub-class) gets called 4 consecutive times(found it via logging) . It might be a redis thing, but Do you think this could be avoided somehow?

Thanks!
PD: I see it as a way to run django-tasks-scheduler (for synchonization tasks, no need for latency) without redis.

[cunla]

Not sure 100%.
Is it calling it with the same param?
I suspect it might check for expiration or something of that sort.
Add a print(traceback.format_exc()) in the __getitem__ to see the stacktrace on each call

[bra-fsn]

Looking for something similar: I would like to expose certain data to redis clients using a custom backend.
For the very first sight, hooking into

fakeredis-py/fakeredis/_helpers.py

Line 129 in d9db2d4

class Database(MutableMapping): # type: ignore

seemed to be the way to do it, but after a few minutes it has turned out, that this object is just used to store key->fakeredis._commands.Item objects, so it can't easily used as the abstraction layer for implementing a custom backend.

I will look into this more, but asking anyways: is there a point in the code where the in-memory dict database could be easily swapped out by something, which offers CRUD-like operations (coupled with expiration handling)?

[cunla]

Probably CommandItem https://github.com/cunla/fakeredis-py/blob/master/fakeredis/_commands.py#L41C7-L68