Adding GraphDocuments to Neo4J sometimes fails when they're generated via LLMGraphTransformer

[gvasilei]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import fs from 'fs';
import "neo4j-driver";
import { Effect, Config, Redacted, Console, Schema, ParseResult, Either, Data } from "effect"
import { Neo4jGraph } from "@langchain/community/graphs/neo4j_graph";
import { LLMGraphTransformer } from "@langchain/community/experimental/graph_transformers/llm";
import { ChatOpenAI } from "@langchain/openai";
import { ConfluenceDocument } from "./documents/Confluence";


class Neo4jError extends Data.TaggedError("Neo4jError")<{
  message: string
}> {}

export const loadDataEffect = (filePath: string) => {

  return Effect.gen(function* () {
    const neo4j_uri = yield* Config.string("NEO4J_URI").pipe(Config.withDefault("bolt://localhost:7687"))
    const neo4j_username = yield* Config.redacted("NEO4J_USERNAME")
    const neo4j_password = yield* Config.redacted("NEO4J_PASSWORD")
    
    const graph = yield* Effect.tryPromise(() => Neo4jGraph.initialize({ url: neo4j_uri, username: Redacted.value(neo4j_username), password: Redacted.value(neo4j_password) }));
    yield* Effect.tryPromise(() => graph.refreshSchema())

    const llm = new ChatOpenAI({ model: "gpt-4o-mini", temperature: 0 });
    const llmGraphTransformer = new LLMGraphTransformer({
      llm: llm
    });

    const text = fs.readFileSync(filePath, 'utf8');
    const result = Schema.decodeUnknownEither(Schema.mutable(Schema.Array(ConfluenceDocument)))(JSON.parse(text))
    if (Either.isLeft(result)) {
      yield* Console.error(ParseResult.TreeFormatter.formatErrorSync(result.left));
    } else {
      const documents = result.right;
     
      for (const doc of documents) {
        yield* Console.log(`${doc.metadata.id} - ${doc.metadata.title}`);
        const result2 = yield* Effect.promise(() => llmGraphTransformer.convertToGraphDocuments([doc]));
        yield* Console.log(`Nodes: ${result2[0].nodes.length} - Relationships:${result2[0].relationships.length}`);

        const eitherRes = yield* Effect.either(Effect.tryPromise({
          try: () => graph.addGraphDocuments(result2),
          catch: (e) => {
            console.error(e);
            return new Neo4jError({ message: `something went wrong ${e}`})
          }
        }));

        const a = Either.match(eitherRes, {
          onLeft: (error) => `Recovering from ${error._tag}`,
          onRight: (value) => value // Do nothing in case of success
        })
        
      }
    }
    
    yield* Console.log("Done");
  })
}

Effect.runPromise(loadDataEffect("PATH_TO_FILE");

Error Message and Stack Trace (if applicable)

Neo4jError: Failed to invoke procedure `apoc.merge.node`: Caused by: java.lang.IllegalArgumentException: The list of label names may not contain any `NULL` or empty `STRING` values. If you wish to merge a `NODE` without a label, pass an empty list instead.
 code: "Neo.ClientError.Procedure.ProcedureCallFailed"

      at captureStacktrace (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/result.js:624:21)
      at new Result (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/result.js:51:53)
      at newCompletedResult (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/transaction.js:528:25)
      at run (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/transaction.js:344:28)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/transaction.js:109:35
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/transaction-managed.js:54:21
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:45:60
      at step (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:30:41)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:22:97
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:21:94
      at new Promise (1:11)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:21:41
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:44:70
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:148:37
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:136:47
      at step (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:17:47)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:9:16
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:5:27
      at new Promise (1:11)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:283:12
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:109:5
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:54:17
      at new Promise (1:11)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/transaction-executor.js:53:9
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/session.js:591:16
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/session.js:570:9
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:44:70
      at step (/Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:30:41)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:22:97
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:21:94
      at new Promise (1:11)
      at /Users/gvasilei/git_src/GraphRag/node_modules/neo4j-driver-core/lib/internal/query-executor.js:21:41

Description

I'm trying to generate a Knowledge Graph from text documents via langchain and Neo4j. While most of the times, nodes and relationships are identified and persisted, occasionally calling addGraphDocuments fails with an internal neo4j exception.

I'm using the default prompt and I'm not providing any allowedNodes and allowedRelationships to LLMGraphTransformer

System Info

version: '0.3.5'
mac
bun version 1.1.22

[dosubot]

Hey @gvasilei! I'm here to help you with any bugs, questions, or contributions you have regarding LangChain.js. Let's squash that bug together!

I found a similar issue that might be related to the problem you're experiencing. There was an issue with the LLMGraphTransformer.convertToGraphDocuments function failing when the prompt chain to fetch nodes and relationships returned null or missing data. This issue was closed with a patch that addressed the problem ^[1].

The error message you're encountering suggests that the apoc.merge.node procedure in Neo4j is being called with a list of label names that contains null or empty string values. This could be related to the data being passed to the addGraphDocuments function, possibly due to the LLMGraphTransformer generating nodes with missing or null labels.

To address this, you might want to ensure that the nodes being generated by the LLMGraphTransformer have valid labels before attempting to add them to the Neo4j graph. You can add a check in your code to filter out or handle nodes with null or empty labels before calling addGraphDocuments.

To continue talking to Dosu, mention @dosu.

[gvasilei]

I don't believe that this is the right way to fix the issue.

By the way, when passing baseEntityLabel: true in addGraphDocuments the issue stops manifesting. Internally it seems that a different cypher query is executed, that calls apoc.create.addLabels instead of apoc.merge.node

[dosubot]

Hi, @gvasilei. I'm Dosu, and I'm helping the LangChain JS team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The issue involves a bug in LangChain.js when adding GraphDocuments to Neo4J using the LLMGraphTransformer.
• It was suggested that the problem might be due to nodes with null or empty labels.
• You noted that using baseEntityLabel: true in addGraphDocuments resolves the issue by triggering a different internal query.

Next Steps:

• Could you confirm if this issue is still relevant to the latest version of the LangChain JS repository? If so, please comment to keep the discussion open.
• If there are no further updates, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!