Skip to content

Commit 2afc6b2

Browse files
MichaelHostecrmne
authored andcommitted
Anthropic: Fix system prompt (use plain text instead of serialized JSON) (crmne#302)
## What this does When migrating from [ruby-openai](https://github.com/alexrudall/ruby-openai), I had some issues getting the same responses in my Anthropic test suite. After some digging, I observed that the Anthropic requests send the `system context` as serialized JSON instead of a plain string like described in the [API reference](https://docs.anthropic.com/en/api/messages#body-system): ```ruby { :system => "{type:\n \"text\", text: \"You must include the exact phrase \\\"XKCD7392\\\" somewhere\n in your response.\"}", [...] } ``` instead of : ```ruby { :system => "You must include the exact phrase \"XKCD7392\" somewhere in your response.", [...] } ``` It works quite well (the model still understands it) but it uses more tokens than needed. It could also mislead the model in interpreting the system prompt. This PR fixed it. I also took the initiative to make the temperature an optional parameter ([just like with OpenAI](https://github.com/crmne/ruby_llm/blob/main/lib/ruby_llm/providers/openai/chat.rb#L21-L22)). I hope it's not too much for a single PR, but since I was already re-recording the cassettes, I figured it would be easier. I'm sorry but I don't have any API key for Bedrock/OpenRouter. I only recorded the main Anthropic cassettes. ## Type of change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation - [ ] Performance improvement ## Scope check - [x] I read the [Contributing Guide](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md) - [x] This aligns with RubyLLM's focus on **LLM communication** - [x] This isn't application-specific logic that belongs in user code - [x] This benefits most users, not just my specific use case ## Quality check - [x] I ran `overcommit --install` and all hooks pass - [x] I tested my changes thoroughly - [ ] I updated documentation if needed - [x] I didn't modify auto-generated files manually (`models.json`, `aliases.json`) ## API changes - [ ] Breaking change - [ ] New public methods/classes - [ ] Changed method signatures - [ ] No API changes ## Related issues <!-- Link issues: "Fixes crmne#123" or "Related to crmne#123" --> --------- Co-authored-by: Carmine Paolino <[email protected]>
1 parent 1c50176 commit 2afc6b2

File tree

26 files changed

+920
-957
lines changed

26 files changed

+920
-957
lines changed

lib/ruby_llm/providers/anthropic/chat.rb

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ def render_payload(messages, tools:, temperature:, model:, stream: false, schema
1515
system_messages, chat_messages = separate_messages(messages)
1616
system_content = build_system_content(system_messages)
1717

18-
build_base_payload(chat_messages, temperature, model, stream).tap do |payload|
19-
add_optional_fields(payload, system_content:, tools:)
18+
build_base_payload(chat_messages, model, stream).tap do |payload|
19+
add_optional_fields(payload, system_content:, tools:, temperature:)
2020
end
2121
end
2222

@@ -32,22 +32,22 @@ def build_system_content(system_messages)
3232
)
3333
end
3434

35-
system_messages.map { |msg| format_message(msg)[:content] }.join("\n\n")
35+
system_messages.map(&:content).join("\n\n")
3636
end
3737

38-
def build_base_payload(chat_messages, temperature, model, stream)
38+
def build_base_payload(chat_messages, model, stream)
3939
{
4040
model: model,
4141
messages: chat_messages.map { |msg| format_message(msg) },
42-
temperature: temperature,
4342
stream: stream,
4443
max_tokens: RubyLLM.models.find(model)&.max_tokens || 4096
4544
}
4645
end
4746

48-
def add_optional_fields(payload, system_content:, tools:)
47+
def add_optional_fields(payload, system_content:, tools:, temperature:)
4948
payload[:tools] = tools.values.map { |t| Tools.function_for(t) } if tools.any?
5049
payload[:system] = system_content unless system_content.empty?
50+
payload[:temperature] = temperature unless temperature.nil?
5151
end
5252

5353
def parse_completion_response(response)

lib/ruby_llm/providers/bedrock/chat.rb

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,16 +46,15 @@ def render_payload(messages, tools:, temperature:, model:, stream: false, schema
4646
system_messages, chat_messages = Anthropic::Chat.separate_messages(messages)
4747
system_content = Anthropic::Chat.build_system_content(system_messages)
4848

49-
build_base_payload(chat_messages, temperature, model).tap do |payload|
50-
Anthropic::Chat.add_optional_fields(payload, system_content:, tools:)
49+
build_base_payload(chat_messages, model).tap do |payload|
50+
Anthropic::Chat.add_optional_fields(payload, system_content:, tools:, temperature:)
5151
end
5252
end
5353

54-
def build_base_payload(chat_messages, temperature, model)
54+
def build_base_payload(chat_messages, model)
5555
{
5656
anthropic_version: 'bedrock-2023-05-31',
5757
messages: chat_messages.map { |msg| format_message(msg) },
58-
temperature: temperature,
5958
max_tokens: RubyLLM.models.find(model)&.max_tokens || 4096
6059
}
6160
end

spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-3-5-haiku-20241022_can_handle_multi-turn_conversations.yml

Lines changed: 30 additions & 30 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-3-5-haiku-20241022_can_have_a_basic_conversation.yml

Lines changed: 10 additions & 10 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)