Skip to content

Rework shared memory #54

@pietroalbini

Description

@pietroalbini

The current status of shared memory is a real mess, and I really need to solve it:

First of all, you can't easily use sub-dictionaries because they're not synchronized, and you need to do something like this, which is really ugly:

mydata = shared["mydata"]
mydata["test"] = "abc"
shared["mydata"] = mydata

Other than that, shared memory is directly available via the shared argument of any hook: this might seem convenient, but prevents adding more shared things, like a global memory (as requested in #22) and locks, which are currently implemented by adding the method at runtime.

In order to solve this problems, and allowing to create custom drivers more easily, a big rewrite of the whole thing is needed. The new API I want for it is:

# For a component's memory
shared.memory["mydata"] = shared.object("dict")
shared.memory["mydata"]["test"] = "yay"
with shared.lock("hello"):
    pass

# A custom bucket
bucket = bot.shared.bucket("test")
bucket.memory["mydata"] = "test

  • Move the functionality of shared into shared.memory
  • Use a custom proxy instead of the multiprocessing one, so it's easier to write custom drivers and the API is consistent between them
    • Add basic support for objects management
    • Add ability to export/import objects and switch drivers
    • Add ability to add memory preparers
    • Add support for the runner driver
  • Add some sort of garbage collection

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions