Feature Request: Support Reading Transcript Entries and/or Modifications

[james-333i]

Current Transcript.entries is private and not accessible to read and/or modify. I imagine there must have been a good reason for this and possibly should just expose a read helper.

Reading at the very least is important so that we can see how the system prompt / instructions is modified as tools or any other behind the scene instructions are incorporated along user / assistant messages.

If we could read a full transcript entries we could:
a) Easily count tokens to ensure we don't overrun in usage.
b) Build up new transcripts with either removed or modified entries to allow chat editing / branching.
c) Better understand any build up that isn't fully exposed to the user.

I think this ties in nicely with:
#103

[mattt]

@james-333i Thanks for calling this out. AnyLanguageModel strives for 1:1 API compatibility Apple Foundation Models, for better or for worse.

This tripped me up originally, but Transcript does actually expose its list of entries, albeit not how you might expect. Rather than an entries property, Transcript conforms to RandomAccessCollection. So you can enumerate over it in a for...in, or access individual elements by position.

Does that meet your needs? Or is there anything still missing?

[james-333i]

Hi @mattt

Thank you for the quick reply (and for your work on this project!).

Yes, I've been an Objective-C developers for close to 20 years now (using many of your older libraries over the years including AFNetworking on numerous projects) and this is one of those aspects of Swift that trips me up. Thank you for the explanation.

With that said, I did a test to print the full entries at the end of an exchange and I see Instructions (those that I have manually supplied) and Prompt (User) and Response (Assistant) entries.

                        for entry in llmSession.transcript {
                            switch entry {

                            case .prompt(let prompt):
                                extract(prompt.segments)

                            case .response(let response):
                                extract(response.segments)

                            case .instructions(let instructions):
                                extract(instructions.segments)

                            case .toolCalls(let toolCalls):
                                dump(toolCalls)

                            case .toolOutput(let toolOutput):
                                dump(toolOutput)

                            }
                        }

                        func extract(_ segments: [Transcript.Segment]) {
                            for segment in segments {
                                switch segment {
                                case .text(let text):
                                    print(text)

                                default:
                                    break
                                }
                            }
                        }

What I'm not seeing is information related to the tool calls that are executed and/or simply loaded in to the model to know how they affect token usage.

1. I'm not seeing things like the tool description, guided instructions, etc. being populated as part of the instructions.
2. I'm not seeing toolsCalls or toolOutput getting triggered.

Is there another way I can get this information?

EDIT: To be clear. I tested a case where I attached 10 tools to a Foundation Model. I tried to chat and it immediately told me I exceeded the 4096 context window limit. This is understandable but I need a way to be able to:
a) Understand the number of tokens each tool impacts the model by seeing what the model receives.
b) See input/output of the tool that could affect token usage.

[mattt]

@james-333i Thanks for sharing that context. A couple suggestions:

Apple has this article, which spends some time walking you through how to instrument Foundation Models. It also links to TN3193, which (in my experience) like most tech notes, has a higher signal-to-noise ratio than the standard docs.

In particular, that tech note makes suggestions like "Give the model a maximum of 3–5 tools to choose from" and "If you’re reaching the context window limit, consider breaking up tool calls across multiple language model sessions". 4096 is orders of magnitude lower than what we're used to now with frontier models context windows and what we can get with models running on MLX / Core ML / llama.cpp / Ollama (at least on desktop hardware), and it's a difficult constraint for a lot of workflows.

The second recommendation I can make is to check out the new ToolExecutionDelegate API that was added in the 0.7.0 release. That won't help as much to account for token counts, but it should give you an easier way to understand when and how tools are being called (and control their execution)

[james-333i]

Thanks @mattt for the feedback and references.

I think the issue I'm noting is that before a model starts inference, it loads connected tools schema in to Instructions since by default each tool has:
includesSchemaInInstructions = true

Yet when I read the Instructions object, I not seeing the schema when I print the segments. There is also a "toolDefinitions" property but that is not populated. Could this be a missing part of implementation?

I am seeing that you have functions that appear to mimic the creation of the JSON Schema for other models so it would be helpful to have a helper to get the JSON Schema for a tool on demand so that tokens can be estimated so users could be notified how many tokens the schema will take up when a tool is enabled.

I'm sure I could come up with a function in my own code that estimates this but would ideally get the true value.