Skip to content

Conversation

@abhishekg999
Copy link
Contributor

No description provided.

@abhishekg999 abhishekg999 changed the base branch from main to staging November 7, 2025 02:33
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @abhishekg999, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request marks a significant step towards a more structured and extensible API for the judgeval library by introducing a new v1 client. It refactors the API generation process, integrates a base_url into client initialization for greater flexibility, and lays the groundwork for advanced features like comprehensive evaluation, diverse scoring mechanisms (built-in, custom, and prompt-based), and OpenTelemetry-driven tracing. These changes aim to enhance the library's modularity and prepare it for future feature development.

Highlights

  • New V1 API Client Structure: Introduced a new versioned API client (v1) with a dedicated directory structure (src/judgeval/v1), encompassing data models, evaluation, scorers, and tracing functionalities.
  • Base URL Integration: Modified the API client generation logic and existing client classes (JudgmentSyncClient, JudgmentAsyncClient) to accept and utilize a base_url parameter, making API requests more flexible.
  • Enhanced API Generation Script: Added a new script (scripts/api_generator_v1.py) specifically for generating the v1 internal API client and its corresponding TypedDict types from an OpenAPI specification, including schema filtering capabilities.
  • Comprehensive Scorer Framework: Implemented a robust framework for various scorer types, including built-in scorers (e.g., AnswerCorrectnessScorer, FaithfulnessScorer), custom scorers, and prompt-based scorers, each with dedicated factories.
  • OpenTelemetry-based Tracing: Integrated OpenTelemetry for tracing, providing BaseTracer and Tracer classes to enable span management, attribute setting, and asynchronous evaluation within traced operations.
  • Flexible Error Handling: Updated the JudgmentAPIError class to allow the response attribute to be optional, improving error handling robustness.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant amount of work towards a new v1 SDK, including a new API client generator, data models, and an OpenTelemetry-based tracing system. The overall structure is well-designed, using factories and clear separation of concerns. My review focuses on the new v1 implementation, where I've identified a couple of high-severity bugs related to caching and argument inspection in the tracer, along with some medium-severity style suggestions for the new API generator script to improve maintainability.

Comment on lines 444 to 465
def _format_inputs(
self, f: Callable[..., Any], args: Tuple[Any, ...], kwargs: Dict[str, Any]
) -> Dict[str, Any]:
try:
params = list(inspect.signature(f).parameters.values())
inputs: Dict[str, Any] = {}
arg_i = 0
for param in params:
if param.kind == inspect.Parameter.POSITIONAL_OR_KEYWORD:
if arg_i < len(args):
inputs[param.name] = args[arg_i]
arg_i += 1
elif param.name in kwargs:
inputs[param.name] = kwargs[param.name]
elif param.kind == inspect.Parameter.VAR_POSITIONAL:
inputs[param.name] = args[arg_i:]
arg_i = len(args)
elif param.kind == inspect.Parameter.VAR_KEYWORD:
inputs[param.name] = kwargs
return inputs
except Exception:
return {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current implementation of _format_inputs is complex and has a bug where it includes self in the captured arguments for instance methods. You can simplify this significantly and make it more robust by using inspect.signature().bind(). This will correctly handle all argument passing styles and allow you to easily remove self or cls from the result.

    def _format_inputs(
        self, f: Callable[..., Any], args: Tuple[Any, ...], kwargs: Dict[str, Any]
    ) -> Dict[str, Any]:
        try:
            bound_args = inspect.signature(f).bind(*args, **kwargs)
            bound_args.apply_defaults()
            arguments = bound_args.arguments
            if "self" in arguments:
                del arguments["self"]
            if "cls" in arguments:
                del arguments["cls"]
            return arguments
        except Exception:
            return {}



def filter_schemas() -> Dict[str, Any]:
from typing import Generator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This local import of Generator can be moved to the top of the file (line 7) with the other imports from the typing module. This improves code readability and follows standard Python style guidelines.

check=True,
)
finally:
import os
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The import os statement should be at the top of the file with other imports, following Python's standard conventions. Placing it inside a function or a finally block is unconventional and can be less efficient.

@propel-code-bot
Copy link

✔️ Propel has finished reviewing this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants