Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Notebook assistant unreliable #1008

Closed
AlgebraicAmin01 opened this issue Jan 4, 2025 · 2 comments
Closed

Notebook assistant unreliable #1008

AlgebraicAmin01 opened this issue Jan 4, 2025 · 2 comments

Comments

@AlgebraicAmin01
Copy link

Describe the issue in detail here. Attach any relevant screenshots or files. The section below was automatically generated. Remove any information that you do not wish to include in the report.

Chatbook/Source/Chatbook/LLMUtilities.wl

Line 37 in 92802b1

ConfirmMatch[ llmSynthesize0[ prompt, evaluator, 1 ], Except[ "", _String ], "Result" ],

Debug Data
Property Value
Name "Wolfram/Chatbook"
Version "2.0.1"
ReleaseID "92802b1ef8f559233ce0b012c36984092f64df5b"
EvaluationEnvironment "Session"
FrontEndVersion "Desktop: 14.1 for Mac OS X ARM (64-bit) (July 16, 2024)"
KernelVersion "14.1.0 for Mac OS X ARM (64-bit) (July 16, 2024)"
SystemID "MacOSX-ARM64"
Notebooks True
DynamicEvaluation True
SynchronousEvaluation False
TaskEvaluation False

Settings

Property Value
InheritanceTest True
SetCellDingbat False
TabbedOutput False
WorkspaceChat True
AllowSelectionContext True
AppName "NotebookAssistance"
LLMEvaluator "NotebookAssistant"
MaxContextTokens 32768
MaxToolResponses 3
Model <|"Service" -> "LLMKit", "Name" -> Automatic|>
PromptGenerators {"RelatedDocumentation"}
ServiceCaller "NotebookAssistance"
Tools {"NotebookEditor"}
ToolSelectionType <|"DocumentationLookup" -> None, "DocumentationSearcher" -> None|>
ToolOptions <|"WolframLanguageEvaluator" -> <|"AppendURIPrompt" -> True, "Method" -> "Session"|>|>
AutoGenerateTitle True
AutoSaveConversations True
ConversationUUID "eb198463-ae3f-4e0d-9b92-c5a25d9be3e9"
StyleSheetVersion "1.5.2.3.3941177587"

Failure Data

<|
  "Evaluation" :> Wolfram`Chatbook`Common`llmSynthesize["Your task is to read a chat transcript between a user and assistant, and then select any relevant Wolfram Language documentation snippets that could help the assistant answer the user's latest message.\n\nEach snippet is uniquely identified by a URI (always starts with 'paclet:' or 'https://resources.wolframcloud.com').\n\nChoose up to 5 documentation snippets that would help answer the user's MOST RECENT message.\n\nRespond with JSON in the following forma...,
  "Information" -> "Result@@Source/Chatbook/LLMUtilities.wl:37,5-37,92",
  "ConfirmationType" -> "ConfirmMatch",
  "Expression" :> Failure["InvalidResponse", <|"Data" -> {<|"BodyChunk" -> "<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n", "BodyChunkProcessed" -> {}, "StatusCode" -> Missing["NotAvailable"], "EventName" -> "BodyChunkReceived"|>, <|"BodyChunk" -> "", "BodyChunkProcessed" -> {}, "StatusCode" -> 504, "EventName" -> "BodyChunkReceived"|>}|>],
  "Pattern" -> Except["", _String],
  "Stack" :> Wolfram`Chatbook`Common`Private`$stack$55484,
  "Failure" -> Failure["ConfirmationFailed", <|"MessageTemplate" -> "Result@@Source/Chatbook/LLMUtilities.wl:37,5-37,92", "MessageParameters" -> {}, "ConfirmationType" -> "ConfirmMatch", "Expression" :> Failure["InvalidResponse", <|"Data" -> {<|"BodyChunk" -> "<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n", "BodyChunkProcessed" -> {}, "StatusCode" -> Missing["NotAvailable"], "EventName" -> "BodyChunkReceived"|>, <|"BodyChunk" -> "", "BodyChunkProcessed" -> {}, ...,
  "Arguments" -> {}
|>

Stack Data

trimURL
throwFailure
catchTop
formatGeneratedPrompt
catchTopAs
withChatState
catchTop
@AlgebraicAmin01
Copy link
Author

AlgebraicAmin01 commented Jan 4, 2025 •
edited
Loading

pretty much every time i ask for some basic help like asking for a function's expression in WL i get this:
General::ChatbookInternal
•••_^ General: An unexpected error occurred. Report this issue »

@rhennigan
Copy link
Member

Duplicate of #989

@rhennigan rhennigan marked this as a duplicate of #989 Jan 6, 2025
@rhennigan rhennigan closed this as not planned Won't fix, can't repro, duplicate, stale Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rhennigan @AlgebraicAmin01