how to use non-json output parsers with SmartScraperGraph?

[prusswan]

Right now, due to the logic in GenerateAnswerNode (used by SmartScraperGraph), JsonOutputParser is always added when llm_model is an instance of ChatOpenAI. As I am using models that may not always return valid JSON, I prefer to use StrOutputParser to handle the raw llm response content, before trying to parse JSON directly.

I could use JsonOutputParser too if there is way to add/perform error handling for invalid JSON.

[dosubot]

Hi @prusswan! I'm Dosu and I’m helping the Scrapegraph-ai team.

Right now, SmartScraperGraph (via GenerateAnswerNode) always uses JsonOutputParser for most LLMs, including ChatOpenAI, unless you provide a schema or use Bedrock. There’s no config option to switch to StrOutputParser or another custom parser—output parser selection is hardcoded based on model type and schema presence. Error handling for invalid JSON is limited: if the LLM returns invalid JSON, the node catches the JSONDecodeError and puts an error message and the raw response in the output, but there’s no fallback or retry logic, and no way to inject your own parser or error handling without modifying or subclassing GenerateAnswerNode itself (source).

If you want to use StrOutputParser or implement more robust error handling, you’d need to subclass GenerateAnswerNode and override the output parser logic. This isn’t currently documented or supported via configuration, so code changes are required for advanced use cases (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[prusswan]

@dosu that is helpful, can you show me how I can modify smart_scraper_graph.py to handle responses with invalid JSON?

Another idea I have is to define and use a schema that accepts raw string, including non-JSON data. Is this something possible?

[dosubot]

To handle invalid JSON responses in SmartScraperGraph, you’ll need to modify or subclass GenerateAnswerNode to use StrOutputParser or implement fallback logic, since the current implementation always uses JsonOutputParser for most LLMs unless a schema is provided, and there’s no config option for a custom parser. The relevant logic is in generate_answer_node.py, where the output parser is selected based on the presence of a schema and the LLM type. You can look at how StrOutputParser is used in GenerateCodeNode as a reference for handling raw string output (example).

Here’s a sketch of what you could do:

• Subclass GenerateAnswerNode and override the execute/process method to use StrOutputParser instead of JsonOutputParser, or to try JSON parsing first and fall back to string parsing if it fails.
• In your custom node, you can catch json.JSONDecodeError and then process the raw string as needed.

As for defining a schema that accepts raw string (non-JSON) data: this isn’t supported in the current SmartScraperGraph implementation. The schema logic expects a pydantic model and always routes through a JSON parser (source), so you can’t use a schema to accept arbitrary strings.

If you want robust handling of non-JSON or malformed outputs, customizing GenerateAnswerNode to use StrOutputParser (or a fallback strategy) is the way to go. Let me know if you want a concrete code example for this!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[prusswan]

@dosu Can you recommend alternatives for web scraping that can handle both JSON and non-JSON responses from the LLM?

[dosubot]

If you need to handle both JSON and non-JSON (raw string) responses from LLMs, you have a few options:

Within Scrapegraph-ai, several nodes natively support StrOutputParser and can process raw string outputs, including GenerateScraperNode, GenerateCodeNode, HtmlAnalyzerNode, PromptRefinerNode, ReasoningNode, and MergeGeneratedScriptsNode. You can build a custom graph using these nodes for more flexible output parsing, instead of relying on SmartScraperGraph, which is hardwired to expect JSON and JsonOutputParser in its main answer node and doesn’t support config-based parser switching or fallback logic out of the box (source, source).

ScriptCreatorGraph is another practical alternative: it uses BeautifulSoup for scraping, which can handle both structured (JSON) and unstructured (raw string) data, making it suitable for mixed-format LLM outputs (docs).

If you want even more control, frameworks like LangChain or LlamaIndex let you define custom output parsers or post-processing steps—so you can try JSON parsing first and fall back to raw string handling if needed.

In summary: for flexible output parsing in Scrapegraph-ai, use or compose graphs with nodes that leverage StrOutputParser, or consider ScriptCreatorGraph with BeautifulSoup. For maximum flexibility, look at general-purpose LLM frameworks that let you define your own parsing logic.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}