What would an implementation of this framework look like?

[mdubinko]

I'm interested an anyone's thoughts on this. What would an implementation look like? Is this purely conceptual a la OSI layers, or could all or parts of it get implemented as software+template components? (And in that case, what would a reference implementation look like?)

[daveshap]

We're working on several demo implementations.

[Qriscuit]

Do you have a clear idea of what the architecture of this software looks like ? if so a UML diagram will be really helpful. And when do you think will we have the very first implementation by ?

[daveshap]

This is a framework like OSI. You should read the framework or the paper first.

[daveshap]

I'm interested an anyone's thoughts on this. What would an implementation look like? Is this purely conceptual a la OSI layers, or could all or parts of it get implemented as software+template components? (And in that case, what would a reference implementation look like?)

Looks like the team wants to build HER first (from the movie)

[DataBassGit]

@jjtolton It can be either depending on your usecase. Remember, this is just the framework. In most cases, full state logging is probably the best bet, so long as you are categorizing datapoints properly. However you choose to store your data, you can return on the data needed for whichever layer agent it goes to. This is also good for human verification of data.

However, there needs to be infrastructure to support such a system. Since we are potentially talking about hundreds of messages per second, monitoring and maintaining such a system would be heavy on the host. For structured data queries like you would want for verification, document stores are the most light weight. (Think MongoDB) But, querying these kinds of datastores against unindexed values is also very heavy. So a query like:

db.collection.find({
  "metadata.tags": "layer=1", 
  "timestamp": {
    $gte: new Date("September 1, 2023 00:00:00"),
    $lt: new Date("September 1, 2023 00:10:00")
  }
})

would be relatively lightweight, as we would index against the layer tag and timestamps are just math operations. However,

db.logs.find({
 "metadata.tags": "layer=1",
  message: { 
    $regex: /murder/
  }
})

is going to be a much more intense task because this is a string operation. But, that can be alleviated by offloading some of that work to the GPU. This is where vector databases shine. Because searches like this (Semantic Similarity) are done mathematically, they can actually be performed by the graphics card.

Now, you don't want to be using precious GPU on every single call, and for the most part, the AI doesn't need to always access this data, it doesn't really matter what system your bus runs on. You just devise your bus to always dump new messages into your logging system. The bus would only need to keep the very most recent messages from each layer. Therefore, your bus has plenty of room for full state, and you can let the database handle separating deltas when you scale up to thousands of inferences per second.

[jjtolton]

Yeah, very interesting. Great discussion. I think part of my confusion initially was around the terminology "bus". In CE/EE terminology, bus has the connotation of something where the signal is the same at all points, and so I was initially thinking some sort of shared buffer between the layers. The message broker discussion was very clarifying, but depending on how this is implemented, now we have a consistency issue. That being said, this isn't necessarily a problem. In nature we have several examples of an organism that experiences high (or infinite) internal latency between processing centers. The long-neck dinosaurs were examples of the multi-brain (unless that story changed since I was in kindergarten). Perhaps a more salient example is the patient who has undergone a corpus callosotomy and is still able to function mostly normally despite both halves of the brain being unable to communicate at all (except indirectly -- message passing, via sight, sound, touch, etc).

Taking a step back, it's probably better to dial in on some assumptions before charging ahead with the implementation discussion. For instance, @DataBassGit, you've mentioned external state -- MongoDB, in this case, or a Vector database. @samgriek you mentioned Redis. Zookeeper/etcd also come to mind, here.

Some other assumptions are probably worth digging out. Are the LLMs in charge of these layers at GPT-4 level? Are they fine-tuned? Or could these also be GPT-3.5 or GPT-3 equivalent, with prompt-tuning only?

As I see it, one of the major open problems with any agent system right now is keeping it on track, due to the incredible short-term memory. One thing that concerns me about sending, for instance, full state to a model that hasn't been fine tuned is that the model may forget who it is and assume the identity of a higher (or lower) layer by accident. Whereas, if all models were GPT-4, this would be less likely to happen.

I think my hope for this framework is that a collection of well selected open source LLMs runnable on commodity hardware driving the layers, with minimal instruct fine-tuning, and some degree of retrieval augmented generation (to stay on track), could out-perform more expensive/exclusive models.

[DataBassGit]

@jjtolton No, you're getting it. The bus can literally be any system that holds information. In my example, I may not have been clear, the database stores logs of the bus state. It can also act as the bus, however. Hell, the bus can be any file on the computer, or just a variable in the app.

Memory is actually not really a problem. The browser chatbots out there right now are essentially just filling the context window with the chat history. AutoAgents often employ a database, but the implementation isn't always the best. That being said, any system with a database has infinite memory, it's just a matter of remembering the right things at the right time.

When we're talking about full state vs delta, we're talking about what's stored on the bus, not what goes to the model. Each layer agent can pull the relevant data needed from the bus. That data presumably gets injected into a prompt template. This reduces token usage, saves context window space, and facilitates better attention from the model.

[jjtolton]

@DataBassGit right on. And this is where we get into the really nitty-gritty.

That being said, any system with a database has infinite memory, it's just a matter of remembering the right things at the right time.

When we're talking about full state vs delta, we're talking about what's stored on the bus, not what goes to the model. Each layer agent can pull the relevant data needed from the bus. That data presumably gets injected into a prompt template. This reduces token usage, saves context window space, and facilitates better attention from the model.

I think one of the major challenges here is getting the agent, or sub-agent-layer-boss, to be aware it can do that -- and to do that reliably. I've been playing with Open Interpreter -- they have a very interesting take on the problem, but you can tell from the code how much they are struggling to extract possible code from non-GPT-4 models. So turning these things into tool users -- well, let me ask -- do you think that's a significant part of this puzzle?

I actually think it's so important that it nearly begs the question. If we can get agents to reliably use (and discover, build, reuse, etc) tools, that may obviate at least a part of the need for this framework (although the morality part of it is nice).

[DataBassGit]

@jjtolton Check out my other project, AgentForge. It's pinned on my profile. We have a pretty nifty tool use solution, but it's not been put through the ringer yet. Essentially, we select tools by semantic search. The agent deploys tools in a very similar way to how GPT-4's function usage works. And, most importantly, it's set up to work for any (instruct trained) model and can use any python library.

Like I said, hasn't been put through the ringer yet, but preliminary tests work.