feat: Add eve first party eve instrumentation

[Luca Forstner (lforst)]

Ok this instrumentation is a bit interesting:

• Eve already has docs for braintrust but the traces it generates are not great. They're otel traces with wild disconnection issues.
• Eve is most likely deployed to Vercel workflows very often which means each step runs in it's own "function invocation", meaning we cannot rely on ALS.
• To combat the "can't use ALS" situation, we use the session/turn-IDs that eve gives us to produce hashes that we then use for our trace and span IDs.
• Also, because the turns are detached runtime-wise, we have a hard time creating a trace for each turn, which is arguably not what we want in any case (?) - so instead we attach all turns, tools, subagents, llm calls to a singular trace as they happen.

The way someone instruments their app is via:

// agent/instrumentation.ts
import { braintrustEveInstrumentation, initLogger } from "braintrust";
import { defineState } from "eve/context";
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation(
  braintrustEveInstrumentation({
    defineState,
    setup: ({ agentName }) => {
      initLogger({
        projectName: 'project',
      });
    },
  }),
);

plus

// agent/hooks/braintrust.ts
import { braintrustEveHook } from "braintrust";
import { defineHook } from "eve/hooks";

export default defineHook(braintrustEveHook());

[Abhijeet Prasad (AbhiPrasad)]

no orchestrion config? Or iitm setup?

[Abhijeet Prasad (AbhiPrasad)]

why are these false? Should we make them user configurable?

[Abhijeet Prasad (AbhiPrasad)]

should we bound? Like use an LRU to manage memory?

[Abhijeet Prasad (AbhiPrasad)]

should we use current here? Ditto with some other places where we reference state instead of current.

[Abhijeet Prasad (AbhiPrasad)]

should we only call flushInstrumentation for events we handle?

[Abhijeet Prasad (AbhiPrasad)]

should be a minor!

[Abhijeet Prasad (AbhiPrasad)]

kind feels weird there is no createTurn and endTurn methods on the bridge here.

[Abhijeet Prasad (AbhiPrasad)]

I think we can have

type EveEntityKind = "session" | "step" | "subagent" | "tool" | "turn";

async function generateEveIds(
  kind: EveEntityKind,
  ...parts: string[]
): Promise<{ rowId: string; spanId: string }> {
  const [rowId, spanId] = await Promise.all([
    deterministicEveId(`eve:row:${kind}`, ...parts),
    deterministicEveId(`eve:${kind}`, ...parts),
  ]);
  return { rowId, spanId };
}

and take care of most of the use cases, and then we just need rootSpanIdForSession and spanIdForSubagent?

[Abhijeet Prasad (AbhiPrasad)]

clanker tells me ToolState.kind and ToolState.stepIndex were written at every creation site but never read.