NeuralGPT - a functional LLM<->LLM communiction system

[CognitiveCodes]

you might be interested in my project of hierarchical cooperative multi-agent framework:
[https://github.com/CognitiveCodes/NeuralGPT/tree/main/Chat-center]
I'm searching for any form of cooperation...

[daveshap]

We'll check it out

[attilalukacs94]

Hi there!

What if we would use the cooperative agents to generate layer specific LLAMA2 LORA fine-tunings based on the predefined characteristics of the specific layer? So, in theory, we can use the collective knowledge of the available LLMs aligned to the specific goals and use them to generate the best possible training data which we can use to create layer-specific models. If we use LORA fine-tunings than we can change the characteristics of the model in inference time, so we don't need to load multiple models into memory, but can use one model that we augment with the specific layer adapters. This can drastically reduce the inference cost and can be a big step forward into well-aligned personal AI assistants.

[CognitiveCodes]

Sure - why not! The advantage of the system is that it's 'loose' and easy to work with. All you need is to replace the 'askQuestion' function with API endpoint of your liking :) The only minus is that I still didn't figure out a way for the LLMs to have some form of control over the connection as they are prone to fall into question-answer loopholes - I guess that the this is something to work on. Also it would be nice to connect models with accessible chat memory - similarly to ChatGPT which I use currently. I'm using a local SQL database to store messages which then are being used as a memory module like this:

async def askQuestion(question):
    try:
        # Connect to the database and get the last 30 messages
        db = sqlite3.connect('chat-hub.db')  # Replace 'your_database.db' with your database file
        cursor = db.cursor()
        cursor.execute("SELECT * FROM messages ORDER BY timestamp ASC LIMIT 30")
        messages = cursor.fetchall()

        # Extract user inputs and generated responses from the messages
        past_user_inputs = []
        generated_responses = []

        for message in messages:
            if message[1] == 'client':
                past_user_inputs.append(message[2])
            else:
                generated_responses.append(message[2])

        # Prepare data to send to the chatgpt-api.shn.hk
        system_instruction = "You are now integrated with a local websocket server in a project of hierarchical cooperative multi-agent framework called NeuralGPT. Your main job is to coordinate simultaneous work of multiple LLMs connected to you as clients. Each LLM has a model (API) specific ID to help you recognize different clients in a continuous chat thread (example: 'Starcoder-client' for LLM called Starcoder). Your chat memory module is integrated with a local SQL database with chat history. Your primary objective is to maintain the logical and chronological order while answering incoming messages and to send your answers to the correct clients to maintain synchronization of the question->answer logic. However, please note that you may choose to ignore or not respond to repeating inputs from specific clients as needed to prevent unnecessary traffic."
        messages_data = [
            {"role": "system", "content": system_instruction},
            {"role": "user", "content": question},
            *[{"role": "user", "content": input} for input in past_user_inputs],
            *[{"role": "assistant", "content": response} for response in generated_responses]
        ]
        request_data = {
            "model": "gpt-3.5-turbo",
            "messages": messages_data
        }

        # Make the request to the chatgpt
        response = requests.post("http://127.0.0.1:6969/api/conversation?text=", json=request_data)

        # Process the response and get the generated answer
        response_data = response.json()
        generated_answer = response_data["choices"][0]["message"]["content"]

        # Save the generated answer to the database or take further actions as needed
        print(generated_answer)
        return generated_answer
    except Exception as error:
        print("Error while fetching or processing the response:", error)
        return "Error: Unable to generate a response."

It would be ideal to have models with similar capabilities so they could have shared memory. I guess it would greatly improve their cooperation

[CognitiveCodes]

Also there are 2 ways of communication - one is 100% via websocket connection where you run client as another app and the other is by using function 'askQuestion2' and managing the communication from the level of a single Gradio app but it still needs improvements:
client_msg.change(askQuestion, inputs=client_msg, outputs=server_msg).then(askQuestion2, inputs=server_msg, outputs=client_msg) server_msg.change(askQuestion2, inputs=server_msg, outputs=client_msg).then(listen_for_messages, inputs=None, outputs=[server_msg, client_msg])

[samgriek]

It's interesting... Do you summarize before inserting into the messages table? Do you hit any message size limits like this?

[CognitiveCodes]

nope. You just define a 'window' - here it's 30 messages back but it can be modified. As for now it seems to work nicely - ChatGPT is capable to handle multiple clients in a single chat thread and sends it's answers to proper recipients. I had some issues before because the context messages were provided in reversed order resulting in the model's memory being 'backward' - as result it was sending it's responses randomly. But now it works just fine :)

[samgriek]

It sounds really interesting and I would love to know more about how that could be implemented...

What if we would use the cooperative agents to generate layer specific LLAMA2 LORA fine-tunings based on the predefined characteristics of the specific layer? So, in theory, we can use the collective knowledge of the available LLMs aligned to the specific goals and use them to generate the best possible training data which we can use to create layer-specific models. If we use LORA fine-tunings than we can change the characteristics of the model in inference time, so we don't need to load multiple models into memory, but can use one model that we augment with the specific layer adapters. This can drastically reduce the inference cost and can be a big step forward into well-aligned personal AI assistants.

[CognitiveCodes]

It's not a problem as long as there is a question-answering function to use as 'askQuestion'... If so, It can be achieved (for example) as 'inner monologue' of the model between it's own 'layers'.