Trace LLM call spans for both create and create_stream

[ekzhu]

Confirmation

• I confirm that I am a maintainer and so can use this template. If I am not, I understand this issue will be closed and I will be asked to use a different template.

Issue body

Give user a clear time line of LLM usage and understanding of model usage and performance.

[EItanya]

As commented on the original PR:

So I thought about that and I'm actually not sure. Instrumentation libraries already exist for many of these and they allow the user to more easily control it.

For example OpenAI: https://pypi.org/project/opentelemetry-instrumentation-openai/

I created a demo using the instrumentation for OpenAI and HTTPX and it worked super well!

In the long term it might actually make sense to take this approach with Autogen as well, but that's a whole separate discussion.

In terms of telemetry I'm actually fairly interested in metrics as well. Does Autogen emit any metrics, or is there any plan to do so?

[ekzhu]

Instrumentation libraries already exist for many of these and they allow the user to more easily control it.

That's true. Makes me think differently as well.

In terms of telemetry I'm actually fairly interested in metrics as well. Does Autogen emit any metrics, or is there any plan to do so?

Certainly, we do. There are several attempts to redesign how metrics are tracked. E.g., for token cost and latency #5790.

[ekzhu]

@EItanya Thinking about this again. I think it is still a good idea to trace the LLM calls because: 1. it is one of the most time-consuming span, and 2. not all client SDKs we depend on have instrumentation.

[victordibia]

Great conversation so far.
I agree that we should support instrumentation from other libraries - our docs already shows how to do this - https://microsoft.github.io/autogen/dev/user-guide/core-user-guide/cookbook/instrumenting.html#observing-llm-calls-using-open-ai

I also agree with @ekzhu on having our own developer experience path (compatible with other libs) for telemetry. Like @ekzhu mentions, this is handy as not all libraries have otel sdks like oai. Integrating with ChatCompletion client provides a path to log things once for all.

I think it will be important that we enable some flag e.g., AUTOGEN_TRACE_CONTENT to help users enable/disable content of LLM spans (provide more control on privacy)

[EItanya]

Agreed, I think controls will be the solution, but I also think that env vars can become very magical very fast, so I think it's potentially worth thinking about a more robust configuration strategy. Definitely not blocking but something I wanted to mention.