feat: integrate Langfuse #2467

fengju0213 · 2025-05-23T05:07:22Z

Description

fixed:#2245

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have read the CONTRIBUTION guide (required)
I have linked this PR to an issue using the Development section on the right sidebar or by adding Fixes #issue-number in the PR description (required)
I have checked if any dependencies need to be added or updated in pyproject.toml and uv lock
I have updated the tests accordingly (required for a bug fix or a new feature)
I have updated the documentation if needed:
I have added examples if this is a new feature

If you are unsure about any of these, don't hesitate to ask. We are here to help!

…gfuse

lightaime

Can we have an instruction on how to self-host langfuse for observability? Also the current implementation does not abstract the same way as AgentOps. Is is possible to abstract them in the same way?

fengju0213 · 2025-05-26T09:25:21Z

Can we have an instruction on how to self-host langfuse for observability? Also the current implementation does not abstract the same way as AgentOps. Is is possible to abstract them in the same way?

If necessary, we could add a self-hosted mode. However, essentially, this would just involve running Langfuse inside Docker. Should we add a parameter to let users launch Langfuse with a one-click Docker build, or leave it to them to start the service manually
According to Langfuse's documentation, the current approach would be more convenient.

Saedbhati · 2025-05-27T15:01:07Z

camel/utils/langfuse.py

+        public_key: Langfuse public key. Can be set via LANGFUSE_PUBLIC_KEY.
+        secret_key: Langfuse secret key. Can be set via LANGFUSE_SECRET_KEY.


Suggested change

public_key: Langfuse public key. Can be set via LANGFUSE_PUBLIC_KEY.

secret_key: Langfuse secret key. Can be set via LANGFUSE_SECRET_KEY.

public_key (Optional[str]): Langfuse public key. Can be set via

LANGFUSE_PUBLIC_KEY environment variable.

(default: None)

lightaime · 2025-05-28T01:01:17Z

Can we have an instruction on how to self-host langfuse for observability? Also the current implementation does not abstract the same way as AgentOps. Is is possible to abstract them in the same way?

If necessary, we could add a self-hosted mode. However, essentially, this would just involve running Langfuse inside Docker. Should we add a parameter to let users launch Langfuse with a one-click Docker build, or leave it to them to start the service manually According to Langfuse's documentation, the current approach would be more convenient.

we can show how to setup langfuse locally with a README
I think we need to abstract the observability tools into a unified interfece. Otherwise, it would be difficult to maintain. @Wendong-Fan what do you think?

Wendong-Fan · 2025-06-01T18:25:32Z

pending to wait for #2536 finish

Wendong-Fan

Thanks, @fengju0213!

Regarding the observability feature, after reviewing several existing solutions, I believe it would be more beneficial for us to develop a foundational observability tool in-house rather than integrating various third-party solutions. Here's why:

Unified Solution: We only need a single, cohesive observability solution to monitor our system's operations.
Native Integration & Better Compatibility: As a framework, building this feature ourselves from the ground up will allow for more native integration. This approach will ensure the tool is better suited to track data generated by our own framework.
OpenTelemetry Compatibility: Some third-party solutions like Langfuse support OpenTelemetry, we can also leverage OpenTelemetry-compatible libraries (e.g., OpenLIT, OpenLLMetry, Arize, MLflow). By building our own native support, users who prefer other OpenTelemetry-based tools (like Langfuse) could still integrate with and monitor our system effectively.

For the next steps, I propose the following:

Define Key Performance Indicators (KPIs): We need to identify and define the critical KPIs for tracking agent performance. This will involve studying existing solutions to compile a comprehensive list of metrics for our observability system.
Select an OpenTelemetry-based Framework: Choose the most suitable framework that aligns with OpenTelemetry standards to build upon. Some libraries we could check:
OpenLIT
OpenLLMetry
Arize
MLflow
Develop the Tracking System: Construct the system to monitor and track agent operations based on the defined KPIs and selected framework

cc @lightaime

lightaime · 2025-06-03T01:06:06Z

Thanks @Wendong-Fan.

Just to understand why we want observability tool in-house at the moment: what kinds of problem that exisiting observability tools cannot solve? Developing an observability tool in-house could be a pretty big project, not quite sure if that is the priority at the moment.

coderabbitai · 2025-06-04T03:21:32Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

fengju0213 and others added 15 commits April 30, 2025 13:42

update

1e20d5a

Merge branch 'master' of https://github.com/camel-ai/camel

8ba430f

Merge branch 'master' of https://github.com/camel-ai/camel

9b2a26c

Merge branch 'master' of https://github.com/camel-ai/camel

d4edd32

Merge branch 'master' of https://github.com/camel-ai/camel

1840fa2

Merge branch 'master' of https://github.com/camel-ai/camel

e9ff111

Merge branch 'master' of https://github.com/camel-ai/camel

217d199

Merge branch 'master' of https://github.com/camel-ai/camel

866da49

integrate langfuse

ece794b

Update .env

21ccbc8

Merge branch 'master' into langfuse

3fc0162

fix pre-commit

4de1ae6

Merge branch 'langfuse' of https://github.com/camel-ai/camel into lan…

b3822b5

…gfuse

Update langfuse.py

937450f

Merge branch 'master' of https://github.com/camel-ai/camel

eb54886

lightaime reviewed May 25, 2025

View reviewed changes

Merge branch 'master' of https://github.com/camel-ai/camel

2572658

Wendong-Fan requested review from nitpicker55555 and Saedbhati May 26, 2025 12:46

fengju0213 added 2 commits May 27, 2025 12:51

Merge branch 'master' of https://github.com/camel-ai/camel

bfcff0e

update

b1d91de

Saedbhati reviewed May 27, 2025

View reviewed changes

fengju0213 added 2 commits May 29, 2025 14:12

Merge branch 'master' of https://github.com/camel-ai/camel

64ca6e2

Merge branch 'master' of https://github.com/camel-ai/camel

01a4d7a

Wendong-Fan mentioned this pull request Jun 1, 2025

[Feature Request] Develop a foundational observability tool in-house #2536

Open

2 tasks

Wendong-Fan assigned fengju0213 Jun 1, 2025

Wendong-Fan added the Pending label Jun 1, 2025

Wendong-Fan added this to Project Camel Jun 1, 2025

Wendong-Fan added this to the Sprint 30 milestone Jun 1, 2025

Wendong-Fan reviewed Jun 1, 2025

View reviewed changes

fengju0213 added 2 commits June 2, 2025 11:21

Merge branch 'master' of https://github.com/camel-ai/camel

2dbf536

Merge branch 'master' of https://github.com/camel-ai/camel

216af8b

fengju0213 added 2 commits June 3, 2025 23:36

Merge branch 'master' of https://github.com/camel-ai/camel

a9a3cef

Merge branch 'master' into langfuse

d6ac196

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: integrate Langfuse #2467

feat: integrate Langfuse #2467

Uh oh!

fengju0213 commented May 23, 2025 •

edited

Loading

Uh oh!

lightaime left a comment

Uh oh!

fengju0213 commented May 26, 2025 •

edited

Loading

Uh oh!

Saedbhati May 27, 2025

Uh oh!

lightaime commented May 28, 2025

Uh oh!

Wendong-Fan commented Jun 1, 2025

Uh oh!

Wendong-Fan left a comment •

edited

Loading

Uh oh!

lightaime commented Jun 3, 2025

Uh oh!

coderabbitai bot commented Jun 4, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

Uh oh!

Uh oh!

		public_key: Langfuse public key. Can be set via LANGFUSE_PUBLIC_KEY.
		secret_key: Langfuse secret key. Can be set via LANGFUSE_SECRET_KEY.

-        public_key: Langfuse public key. Can be set via LANGFUSE_PUBLIC_KEY.
-        secret_key: Langfuse secret key. Can be set via LANGFUSE_SECRET_KEY.
+        public_key (Optional[str]): Langfuse public key. Can be set via
+            LANGFUSE_PUBLIC_KEY environment variable.
+            (default: None)

feat: integrate Langfuse #2467

Are you sure you want to change the base?

feat: integrate Langfuse #2467

Uh oh!

Conversation

fengju0213 commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

lightaime left a comment

Choose a reason for hiding this comment

Uh oh!

fengju0213 commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Saedbhati May 27, 2025

Choose a reason for hiding this comment

Uh oh!

lightaime commented May 28, 2025

Uh oh!

Wendong-Fan commented Jun 1, 2025

Uh oh!

Wendong-Fan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightaime commented Jun 3, 2025

Uh oh!

coderabbitai bot commented Jun 4, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

Uh oh!

fengju0213 commented May 23, 2025 •

edited

Loading

fengju0213 commented May 26, 2025 •

edited

Loading

Wendong-Fan left a comment •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)