feat(llm): add native groq provider support and fix cache_breakpoint …#6287
feat(llm): add native groq provider support and fix cache_breakpoint …#6287Mahajan-Sachin wants to merge 5 commits into
Conversation
![corridor-security[bot]](https://avatars.githubusercontent.com/in/1177339?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary: This PR adds native Groq LLM provider routing and strips Anthropic-specific cache metadata from messages sent to non-Anthropic providers. No exploitable security vulnerabilities were identified.
Risk: Low risk. The changes use an environment-sourced API key and HTTPS default endpoint consistent with existing OpenAI-compatible provider handling, without modifying authentication, authorization, public endpoints, or sensitive data storage.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds Groq as a native OpenAI-compatible provider by registering it in the native provider allowlist, mapping ChangesGroq Native Provider and Cache Breakpoint Fix
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
lib/crewai/src/crewai/llm.py (1)
558-563: 🧹 Nitpick | 🔵 TrivialGroq model prefix patterns are unnecessarily broad and include unpopulated model types.
The patterns
["llama", "gemma", "mixtral", "whisper"]will match any model starting with these strings. The constants show onlyllamaandgemmamodels currently;mixtralandwhisperpatterns have no corresponding models. Patterns likegroq/llamacpp-customorgroq/gemma-unofficialwould pass this check but be rejected by the Groq API.While the API will ultimately reject invalid models, the inclusion of unmatched prefixes suggests the patterns should either be removed if not needed or clarified if they anticipate future Groq models.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/llm.py` around lines 558 - 563, The groq provider model prefix check in the condition where provider == "groq" includes patterns for "mixtral" and "whisper" that do not correspond to any actual models in the Groq constants. Update the prefix list by removing "mixtral" and "whisper" from the array, keeping only the prefixes "llama" and "gemma" that have actual corresponding models. This will prevent the check from incorrectly accepting invalid model names that would ultimately be rejected by the Groq API.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/llms/providers/openai_compatible/completion.py`:
- Around line 46-51: Add a new test method called test_groq_config to the test
file that verifies the Groq provider configuration is correctly registered with
the expected values. The test should validate that the Groq entry has base_url
set to "https://api.groq.com/openai/v1", api_key_env set to "GROQ_API_KEY",
base_url_env set to "GROQ_BASE_URL", and api_key_required flag set to True.
Follow the same pattern and structure used in the existing test methods for
other OpenAI-compatible providers such as test_openrouter_config,
test_deepseek_config, and test_ollama_config to ensure consistency.
---
Nitpick comments:
In `@lib/crewai/src/crewai/llm.py`:
- Around line 558-563: The groq provider model prefix check in the condition
where provider == "groq" includes patterns for "mixtral" and "whisper" that do
not correspond to any actual models in the Groq constants. Update the prefix
list by removing "mixtral" and "whisper" from the array, keeping only the
prefixes "llama" and "gemma" that have actual corresponding models. This will
prevent the check from incorrectly accepting invalid model names that would
ultimately be rejected by the Groq API.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a272e948-080b-496f-bac7-2c19c2a247b9
📒 Files selected for processing (3)
lib/crewai/src/crewai/llm.pylib/crewai/src/crewai/llms/providers/openai_compatible/completion.pylib/crewai/tests/test_llm.py
|
@coderabbitai Removed |
|
Tip For best results, initiate chat on the files or code changes.
✏️ Learnings added
|
|
@coderabbitai Please review the latest changes on this PR. Does everything look good now? Is there anything else I need to update or change? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
echo "=== llm.py relevant sections ==="
cat lib/crewai/src/crewai/llm.py | grep -n "groq\|GROQ\|cache_breakpoint\|CACHE_BREAKPOINT\|SUPPORTED_NATIVE\|native_provider\|_get_native_provider\|OPENAI_COMPATIBLE\|provider_mapping\|mixtral\|whisper\|gemma\|llama" | head -80
echo ""
echo "=== _format_messages_for_provider method ==="
grep -n "_format_messages_for_provider" lib/crewai/src/crewai/llm.pyLength of output: 3803 🏁 Script executed: #!/bin/bash
echo "=== Full _format_messages_for_provider ==="
grep -n -A 40 "def _format_messages_for_provider" lib/crewai/src/crewai/llm.py
echo ""
echo "=== _get_native_provider ==="
grep -n -A 30 "def _get_native_provider\|_infer_provider\|native_provider" lib/crewai/src/crewai/llm.py | head -100
echo ""
echo "=== SUPPORTED_NATIVE_PROVIDERS ==="
grep -n -A 20 "SUPPORTED_NATIVE_PROVIDERS" lib/crewai/src/crewai/llm.py | head -40
echo ""
echo "=== provider_mapping ==="
grep -n -A 20 "provider_mapping" lib/crewai/src/crewai/llm.py | head -40Length of output: 9685 🏁 Script executed: #!/bin/bash
echo "=== openai_compatible/completion.py ==="
cat lib/crewai/src/crewai/llms/providers/openai_compatible/completion.py | head -80
echo ""
echo "=== test_llm.py groq sections ==="
grep -n -B 3 -A 10 "groq\|replicate\|is_litellm" lib/crewai/tests/test_llm.py | head -120
echo ""
echo "=== test_openai_compatible.py ==="
grep -n -B 2 -A 15 "groq\|Groq\|TestProviderRegistry\|TestLLMIntegration" lib/crewai/tests/llms/openai_compatible/test_openai_compatible.py | head -100Length of output: 11414 🏁 Script executed: #!/bin/bash
echo "=== Full diff on PR ==="
gh pr diff 6287 --repo crewAIInc/crewAI 2>/dev/null | head -400Length of output: 6428
After reviewing the full diff, the implementation is well-structured and correct. Here's a complete breakdown: ✅ What's correct
🔎 Minor items worth addressing1. Deferred import inside
# At the top of llm.py with other imports
from crewai.llms.cache import CACHE_BREAKPOINT_KEY2. Missing unit test for The stripping logic is new but has no dedicated test. Consider adding one alongside existing message formatting tests: def test_format_messages_strips_cache_breakpoint_for_non_anthropic():
"""cache_breakpoint keys must be removed for non-Anthropic providers."""
from crewai.llms.cache import CACHE_BREAKPOINT_KEY
llm = LLM(model="groq/llama-3.3-70b-versatile", is_litellm=False)
# Simulate a message that carries the cache breakpoint marker
messages = [
{"role": "user", "content": "Hello", CACHE_BREAKPOINT_KEY: True},
]
with patch.dict(os.environ, {"GROQ_API_KEY": "test-key"}):
formatted = llm._format_messages_for_provider(messages)
assert all(CACHE_BREAKPOINT_KEY not in msg for msg in formatted)
def test_format_messages_preserves_cache_breakpoint_for_anthropic():
"""cache_breakpoint keys must be preserved for Anthropic providers."""
from crewai.llms.cache import CACHE_BREAKPOINT_KEY
with patch.dict(os.environ, {"ANTHROPIC_API_KEY": "test-key"}):
llm = LLM(model="anthropic/claude-3-5-sonnet-20241022")
messages = [
{"role": "user", "content": "Hello", CACHE_BREAKPOINT_KEY: True},
]
formatted = llm._format_messages_for_provider(messages)
assert any(CACHE_BREAKPOINT_KEY in msg for msg in formatted)3. Extra blank lines in There are two consecutive blank lines after SummaryThe core implementation is complete and all the routing wires are consistent. The two substantive suggestions are the deferred import cleanup and the |
|
@coderabbitai All requested changes (clean up of deferred imports, addition of formatting tests, and cleaning up whitespace in |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
echo "=== Check CACHE_BREAKPOINT_KEY import at top of llm.py ==="
head -60 lib/crewai/src/crewai/llm.py | grep -n "CACHE_BREAKPOINT\|cache_breakpoint\|from crewai.llms.cache"
echo ""
echo "=== Check no deferred import inside _format_messages_for_provider ==="
grep -n -A 50 "def _format_messages_for_provider" lib/crewai/src/crewai/llm.py | head -60
echo ""
echo "=== Check new cache_breakpoint tests in test_llm.py ==="
grep -n -B 2 -A 20 "cache_breakpoint\|CACHE_BREAKPOINT" lib/crewai/tests/test_llm.py
echo ""
echo "=== Check whitespace in test_openai_compatible.py (blank lines between methods) ==="
grep -n "def test_\|^$\|^ $" lib/crewai/tests/llms/openai_compatible/test_openai_compatible.py | head -60Length of output: 7024
Everything is clean and consistent. The PR is ready to merge. 🎉 If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! |
|
@coderabbitai Will you assign maintainers / trigger review workflows automatically, or should I write a comment to notify the maintainers? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Check for CODEOWNERS file
echo "=== CODEOWNERS ==="
fd -i "CODEOWNERS" --exec cat {} 2>/dev/null || echo "No CODEOWNERS found"
# Check for auto-assign or reviewer assignment GitHub Actions workflows
echo ""
echo "=== Auto-assign or reviewer workflows ==="
fd -e yml -e yaml . .github/workflows --exec grep -l "assign\|reviewer\|review-request" {} 2>/dev/null || echo "No auto-assign workflows found"
# Check current PR reviewers and assignees
echo ""
echo "=== Current PR reviewers/assignees ==="
gh pr view 6287 --repo crewAIInc/crewAI --json reviewRequests,assignees,author | jq .Length of output: 1042
Here's what you can do to get maintainer attention:
The PR itself is in great shape, so it's purely a matter of getting the right eyes on it. 👍 |
|
@coderabbitai Can you look at the commit history and identify other active maintainers or core contributors who frequently work on LLM integrations or core codebase components in this repo? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
echo "=== Top contributors overall (last 6 months) ==="
gh api "repos/crewAIInc/crewAI/commits?per_page=100&since=2025-12-01" \
--jq '.[].commit.author.name' | sort | uniq -c | sort -rn | head -20
echo ""
echo "=== Recent PR authors merged into main (LLM/provider related) ==="
gh pr list --repo crewAIInc/crewAI --state merged --limit 50 \
--json number,title,author,mergedAt \
--jq '.[] | select(.title | test("llm|provider|groq|anthropic|openai|litellm|model|cache"; "i")) | "\(.mergedAt[:10]) @\(.author.login): #\(.number) \(.title)"'
echo ""
echo "=== Recent commits to lib/crewai/src/crewai/llm.py ==="
gh api "repos/crewAIInc/crewAI/commits?path=src/crewai/llm.py&per_page=30" \
--jq '.[] | "\(.commit.author.date[:10]) \(.author.login // .commit.author.name): \(.commit.message | split("\n")[0])"' 2>/dev/null | head -30
echo ""
echo "=== Repo contributors ==="
gh api "repos/crewAIInc/crewAI/contributors?per_page=20" \
--jq '.[] | "\(.contributions) commits: @\(.login)"'Length of output: 4720
🔑 Core maintainers most active on LLM integrations
💡 Recommended actionSince this PR touches You can request them directly via the "Reviewers" gear on the PR sidebar, or drop a comment like:
|
|
Hey @greysonlalonde @lucasgomide @joaomdmoura — this PR adds native Groq provider support and fixes the cache_breakpoint issue for non-Anthropic providers (resolves #6286 and #5886). All local unit tests are passing and CodeRabbit has approved the changes. Would appreciate a review when you get a chance! thank you! 😊 |
1 participant
Description
This PR implements native Groq LLM provider support in CrewAI and fixes the
cache_breakpointissue when using non-Anthropic providers."groq"configuration toOPENAI_COMPATIBLE_PROVIDERSpointing tohttps://api.groq.com/openai/v1.LLMclass routing (SUPPORTED_NATIVE_PROVIDERS,provider_mapping, and_get_native_provider) to route Groq models (llama,gemma,mixtral,whisper) natively using theOpenAICompatibleCompletionprovider.cache_breakpointkey from messages for all non-Anthropic providers in_format_messages_for_providerto prevent API errors (e.g.,BadRequestError: 400).Related Issues
Closes #6286
Closes #5886
Verification
1. Automated Tests
Unit tests added to
lib/crewai/tests/test_llm.pyverifying native routing and fallback behaviors. Both tests passed successfully:2. Manual Verification
Successfully ran a crew kickoff using
groq/llama-3.3-70b-versatileunder native SDK configuration, routing successfully without falling back to LiteLLM:And the LLM generated the final answer correctly:
llm-generated
Summary by CodeRabbit
New Features
groq/...models to the Groq provider.Bug Fixes
Tests
LLM(...)behavior, plus assertions for cache breakpoint handling.