fix: Upgrade Hugging Face Inference API to support Inference Providers #5454
Conversation
- Upgrade @huggingface/inference from v2.6.1 to v4.13.2 - Update ChatHuggingFace to use InferenceClient with chatCompletion API - Update HuggingFaceInference (LLM) to use v4 HfInference with Inference Providers - Update HuggingFaceInferenceEmbedding to use v4 HfInference - Add endpoint handling logic to ignore custom endpoints for provider-based models - Add improved error handling and validation for API keys - Update UI descriptions to guide users on proper configuration Fixes FlowiseAI#5161
Summary of ChangesHello @aibysid, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the integration with Hugging Face by upgrading the underlying inference library to its latest major version. This upgrade enables the use of Hugging Face's Inference Providers, streamlines the interaction with chat models through a new client, and introduces more intelligent endpoint management. Additionally, it improves the overall user experience by providing clearer UI instructions and more robust error handling for API key and connection issues. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request upgrades the Hugging Face Inference API to a newer version, enabling support for Inference Providers. The changes are well-implemented, including updates to ChatHuggingFace, HuggingFaceInference, and HuggingFaceInferenceEmbedding to use the new InferenceClient and chatCompletion APIs. The logic to handle custom endpoints versus provider-based models is sound, and the error handling has been significantly improved. My feedback includes a couple of suggestions to improve the robustness of API key validation and to ensure consistent handling of optional parameters in API calls.
| if (!huggingFaceApiKey) { | ||
| console.error('[ChatHuggingFace] API key validation failed: No API key found') |
There was a problem hiding this comment.
The API key validation should also check for empty or whitespace-only strings for robustness. This is also consistent with the validation performed in core.ts.
| if (!huggingFaceApiKey) { | |
| console.error('[ChatHuggingFace] API key validation failed: No API key found') | |
| if (!huggingFaceApiKey || huggingFaceApiKey.trim() === '') { | |
| console.error('[ChatHuggingFace] API key validation failed: Empty or undefined') |
| const params: any = { | ||
| temperature: this.temperature, | ||
| max_tokens: this.maxTokens, | ||
| stop: options?.stop ?? this.stopSequences, | ||
| top_p: this.topP | ||
| } | ||
| // Include optional parameters if they are defined | ||
| if (this.topK !== undefined) { | ||
| params.top_k = this.topK | ||
| } | ||
| if (this.frequencyPenalty !== undefined) { | ||
| params.frequency_penalty = this.frequencyPenalty | ||
| } | ||
| return params |
There was a problem hiding this comment.
The handling of optional parameters is inconsistent. temperature, maxTokens, and topP are added to the params object unconditionally, which means they could be passed as undefined to the API. While many libraries filter out undefined properties, it's safer and more explicit to only add properties that have a defined value. This would also make the implementation consistent with how topK and frequencyPenalty are handled.
const params: any = {};
if (this.temperature !== undefined) params.temperature = this.temperature;
if (this.maxTokens !== undefined) params.max_tokens = this.maxTokens;
if (this.topP !== undefined) params.top_p = this.topP;
if (this.topK !== undefined) params.top_k = this.topK;
if (this.frequencyPenalty !== undefined) params.frequency_penalty = this.frequencyPenalty;
const stop = options?.stop ?? this.stopSequences;
if (stop) params.stop = stop;
return params;
1 participant
Fixes #5161