Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not able to deploy to a cloud VM behind a firewall #1024

Open
icejean opened this issue Jan 23, 2025 · 4 comments
Open

Not able to deploy to a cloud VM behind a firewall #1024

icejean opened this issue Jan 23, 2025 · 4 comments

Comments

@icejean
Copy link

icejean commented Jan 23, 2025 •
edited
Loading

Hi,all,
I just try to deploy Neo4j Knowledge Graph Builder to a public cloud VM behind a firewall for a demo & research purpose, but I'm not able to work out a solution yet, here's the problem details.
As shown in the following network graph, the backend is serving at 172.18.0.2:4000, the frontend is serving at 172.18.0.3:4040, they are running with docker on a Ubuntu 22 host with a list of IPs，the IP 172.18.0.1 is for connetion with docker networks, the IP 172.17.0.1 is for accessing Ubuntu host services from the backend and frontend container, and the IP 10.60.136.78 is for public service to Windows ReAct Client, it's mapped to IP 117.50.174.65 by firewall to the public.

Image

As the CORS rule of browser requires that ReAct Javascript client need to request backend within the same domain, I need to deploy a Nginx server and setup reverse proxy for frontend and backend. But the Nginx server can only access backend service at 172.17.0.1:4000, not at 172.18.0.2:4000, frontend is the same also:

(base) root@10-60-136-78:~# curl http://172.18.0.2:4000
(base) root@10-60-136-78:~# curl http://172.17.0.1:4000
{"detail":"Not Found"}(base)
 root@10-60-136-78:~# 
(base) root@10-60-136-78:~# curl http://172.18.0.3:4040
(base) root@10-60-136-78:~# curl http://172.17.0.1:4040
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="icon" type="image/png" sizes="32x32" href="/kgbuilder/favicons/favicon-32x32.png">
    <link rel="icon" type="image/png" sizes="194x194" href="/kgbuilder/favicons/favicon-194x194.png">
    <link rel="icon" type="image/png" sizes="16x16" href="/kgbuilder/favicons/favicon-16x16.png">
    <link rel="shortcut icon" href="/kgbuilder/favicons/favicon.ico">
    <title>Neo4j graph builder</title>
    <script type="module" crossorigin src="/kgbuilder/assets/index-1f639c8d.js"></script>
    <link rel="stylesheet" href="/kgbuilder/assets/index-03e5dd9e.css">
  </head>
  <body>
    <div id="root"></div>
    
  </body>
</html>

So I config Nginx reverse proxy as follow to put frontend and backend into the same domain:

# frontend
location /kgbuilder/ {
    rewrite ^/kgbuilder/(.*)$ /$1 break;

    proxy_pass http://172.17.0.1:4040;  
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    sub_filter 'href="/assets/' 'href="/kgbuilder/assets/';
    sub_filter 'src="/assets/' 'src="/kgbuilder/assets/';
    sub_filter 'url("/assets/' 'url("/kgbuilder/assets/';
    sub_filter_once off;

    # WebSocket 
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_read_timeout 20d;

    proxy_buffering off;
}

# backend
location /kgbuilderapi/ {
    rewrite ^/kgbuilderapi/(.*)$ /$1 break;

    proxy_pass http://172.17.0.1:4000;  
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    sub_filter_once off;

    # WebSocket 
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_read_timeout 20d;

    proxy_buffering off;
}

I add an environment variable to .env to tell the frontend base and point the backend URI to the reverse proxy public URI :

# Optional Backend

# Added by Jean 2025/01/20
LANGUAGE="chinese"

EMBEDDING_MODEL="all-MiniLM-L6-v2"
IS_EMBEDDING="true"
KNN_MIN_SCORE="0.94"
# Enable Gemini (default is False) | Can be False or True
GEMINI_ENABLED=False
# LLM_MODEL_CONFIG_ollama_llama3="llama3,http://host.docker.internal:11434"

# Added by Jean 2024/11/17
LLM_MODEL_CONFIG_openai_gpt_4o="gpt-4o,sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
LLM_MODEL_CONFIG_openai_gpt_4o_mini="gpt-4o-mini,sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
LLM_MODEL_CONFIG_openai_gpt_3_5="gpt-3.5-turbo,sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
HTTP_PROXY="http://172.17.0.1:7890"
HTTPS_PROXY="http://172.17.0.1:7890"

# Enable Google Cloud logs (default is False) | Can be False or True
GCP_LOG_METRICS_ENABLED=False
NUMBER_OF_CHUNKS_TO_COMBINE=6
UPDATE_GRAPH_CHUNKS_PROCESSED=20

NEO4J_URI="bolt://host.docker.internal:7687"
NEO4J_USERNAME="neo4j"
NEO4J_PASSWORD="xxxxxxxxxxxx"

# Comment out by Jean, 2024/11/7
LANGCHAIN_API_KEY="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
LANGCHAIN_PROJECT=""
LANGCHAIN_TRACING_V2="false"
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"

GCS_FILE_CACHE=False
ENTITY_EMBEDDING=True

# Optional Frontend
VITE_BACKEND_API_URL="http://117.50.174.65/kgbuilderapi"

# Added by Jean 2025/01/23
VITE_FRONTEND_BASE_PATH="/kgbuilder/"

VITE_BLOOM_URL="https://workspace-preview.neo4j.io/workspace/explore?connectURL={CONNECT_URL}&search=Show+me+a+graph&featureGenAISuggestions=true&featureGenAISuggestionsInternal=true"
VITE_REACT_APP_SOURCES="local,youtube,wiki,s3,web"
VITE_ENV="DEV"
VITE_TIME_PER_PAGE=50
VITE_CHUNK_SIZE=5242880
VITE_GOOGLE_CLIENT_ID=""
VITE_CHAT_MODES=""
VITE_BATCH_SIZE=2

#VITE_LLM_MODELS="diffbot,openai_gpt_3_5,openai_gpt_4o,openai_gpt_4o_mini" # ",ollama_llama3"
VITE_LLM_MODELS_PROD="openai_gpt_4o,openai_gpt_4o_mini,openai_gpt_3_5,diffbot"

And modify ~/frontend/vite.config.ts as follow to set a frontend base path:

import { defineConfig, loadEnv } from 'vite';
import react from '@vitejs/plugin-react';

// see https://stackoverflow.com/questions/73834404/react-uncaught-referenceerror-process-is-not-defined
// otherwise use import.meta.env.VITE_BACKEND_API_URL and expose it as such with the VITE_ prefix
export default defineConfig(({ mode }) => {
  const env = loadEnv(mode, process.cwd(), 'VITE_');
  const frontendBasePath = env.VITE_FRONTEND_BASE_PATH || '/';  // default root path
  return {
    base: frontendBasePath,  //  base
    define: {
      'process.env': env,
    },
    plugins: [react()],
    optimizeDeps: { esbuildOptions: { target: 'es2020' } },
  };
});

Then modify ~/frontend/Dockerfile to point the the backend to the reverse proxy URI too.

# ARG VITE_BACKEND_API_URL="http://172.18.0.2:4000"
ARG VITE_BACKEND_API_URL="http://117.50.174.65/kgbuilderapi"

And here is the file docker-compose.yml, export extra host to activate 171.17.0.1 and map frontend to 4040, backend to 4000:

version: "3"

services:
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    volumes:
      - ./backend:/code
    environment:
      - NEO4J_URI=${NEO4J_URI-neo4j://database:7687}
      - NEO4J_PASSWORD=${NEO4J_PASSWORD-password}
      - NEO4J_USERNAME=${NEO4J_USERNAME-neo4j}
      - OPENAI_API_KEY=${OPENAI_API_KEY-}
      - DIFFBOT_API_KEY=${DIFFBOT_API_KEY-}
      - EMBEDDING_MODEL=${EMBEDDING_MODEL-all-MiniLM-L6-v2}
      - LANGCHAIN_ENDPOINT=${LANGCHAIN_ENDPOINT-}
      - LANGCHAIN_TRACING_V2=${LANGCHAIN_TRACING_V2-}
      - LANGCHAIN_PROJECT=${LANGCHAIN_PROJECT-}
      - LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY-}
      - KNN_MIN_SCORE=${KNN_MIN_SCORE-0.94}
      - IS_EMBEDDING=${IS_EMBEDDING-true}
      - GEMINI_ENABLED=${GEMINI_ENABLED-False}
      - GCP_LOG_METRICS_ENABLED=${GCP_LOG_METRICS_ENABLED-False}
      - UPDATE_GRAPH_CHUNKS_PROCESSED=${UPDATE_GRAPH_CHUNKS_PROCESSED-20}
      - NUMBER_OF_CHUNKS_TO_COMBINE=${NUMBER_OF_CHUNKS_TO_COMBINE-6}
      - ENTITY_EMBEDDING=${ENTITY_EMBEDDING-False}
      - GCS_FILE_CACHE=${GCS_FILE_CACHE-False}
      # - LLM_MODEL_CONFIG_ollama_llama3=${LLM_MODEL_CONFIG_ollama_llama3-}
      # Added by Jean, 2024/11/7
      - LLM_MODEL_CONFIG_openai_gpt_4o=${LLM_MODEL_CONFIG_openai_gpt_4o-}
      - LLM_MODEL_CONFIG_openai_gpt_4o_mini=${LLM_MODEL_CONFIG_openai_gpt_4o_mini-}
      - LLM_MODEL_CONFIG_openai_gpt_3_5=${LLM_MODEL_CONFIG_openai_gpt_3_5-}
      - HTTP_PROXY=${HTTP_PROXY}
      - HTTPS_PROXY=${HTTPS_PROXY}
      - http_proxy=${HTTP_PROXY}
      - https_proxy=${HTTPS_PROXY}
      # Added by Jean 2025/1/20
      - LANGUAGE=${LANGUAGE-english}

    # env_file:
    #   - ./backend/.env
    container_name: backend
    extra_hosts:
      - host.docker.internal:host-gateway
    ports:
      - "4000:8000"
    networks:
      - net

  frontend:
    depends_on:
      - backend
    build:
      context: ./frontend
      dockerfile: Dockerfile
      args:
        - VITE_BACKEND_API_URL=${VITE_BACKEND_API_URL-http://localhost:8000}
        - VITE_REACT_APP_SOURCES=${VITE_REACT_APP_SOURCES-local,wiki,s3}
        - VITE_GOOGLE_CLIENT_ID=${VITE_GOOGLE_CLIENT_ID-}
        - VITE_BLOOM_URL=${VITE_BLOOM_URL-https://workspace-preview.neo4j.io/workspace/explore?connectURL={CONNECT_URL}&search=Show+me+a+graph&featureGenAISuggestions=true&featureGenAISuggestionsInternal=true}
        - VITE_TIME_PER_PAGE=${VITE_TIME_PER_PAGE-50}
        - VITE_CHUNK_SIZE=${VITE_CHUNK_SIZE-5242880}
        - VITE_LARGE_FILE_SIZE=${VITE_LARGE_FILE_SIZE-5242880}
        - VITE_ENV=${VITE_ENV-DEV}
        - VITE_CHAT_MODES=${VITE_CHAT_MODES-}
        - VITE_BATCH_SIZE=${VITE_BATCH_SIZE-2}
        - VITE_LLM_MODELS=${VITE_LLM_MODELS-}
        - VITE_LLM_MODELS_PROD=${VITE_LLM_MODELS_PROD-openai_gpt_4o,openai_gpt_4o_mini,diffbot,gemini_1.5_flash}
        - DEPLOYMENT_ENV=local

        - HTTP_PROXY=${HTTP_PROXY}
        - HTTPS_PROXY=${HTTPS_PROXY}
        - http_proxy=${HTTP_PROXY}
        - https_proxy=${HTTPS_PROXY}


    volumes:
      - ./frontend:/app
      - /app/node_modules

    # Added by Jean, 2024/11/7
    environment:
      - HTTP_PROXY=${HTTP_PROXY}
      - HTTPS_PROXY=${HTTPS_PROXY}
      - http_proxy=${HTTP_PROXY}
      - https_proxy=${HTTPS_PROXY}

    env_file:
      - ./frontend/.env
    container_name: frontend
    extra_hosts:
      - host.docker.internal:host-gateway      
    ports:
      - "4040:8080"
    networks:
      - net

networks:
  net:

The running result is:

Unexpected Application Error!
404 Not Found

Image

The source of index.html seems to be O.K..

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="icon" type="image/png" sizes="32x32" href="/kgbuilder/favicons/favicon-32x32.png">
    <link rel="icon" type="image/png" sizes="194x194" href="/kgbuilder/favicons/favicon-194x194.png">
    <link rel="icon" type="image/png" sizes="16x16" href="/kgbuilder/favicons/favicon-16x16.png">
    <link rel="shortcut icon" href="/kgbuilder/favicons/favicon.ico">
    <title>Neo4j graph builder</title>
    <script type="module" crossorigin src="/kgbuilder/assets/index-1f639c8d.js"></script>
    <link rel="stylesheet" href="/kgbuilder/assets/index-03e5dd9e.css">
  </head>
  <body>
    <div id="root"></div>
    
  </body>
</html>

Frontend logs:

frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:11 +0000] "GET / HTTP/1.1" 200 785 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"
frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:11 +0000] "GET /assets/index-1f639c8d.js HTTP/1.1" 200 3602835 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"
frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:11 +0000] "GET /assets/index-03e5dd9e.css HTTP/1.1" 200 3572661 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"
frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:29 +0000] "GET /favicons/favicon-16x16.png HTTP/1.1" 200 545 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"
frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:29 +0000] "GET /favicons/favicon-194x194.png HTTP/1.1" 200 3352 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"
frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:29 +0000] "GET /assets/public-sans-700-bold-normal-39dc2fce.woff2 HTTP/1.1" 200 14752 "http://117.50.174.65/kgbuilder/assets/index-03e5dd9e.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"
frontend  | 172.18.0.1 - - [23/Jan/2025:19:52:29 +0000] "GET /assets/syne-neo-500-medium-normal-5d4117ae.woff2 HTTP/1.1" 200 25580 "http://117.50.174.65/kgbuilder/assets/index-03e5dd9e.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0" "14.26.197.74"

Nginx logs:

14.26.197.74 - - [24/Jan/2025:03:52:11 +0800] "GET /kgbuilder/ HTTP/1.1" 200 388 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
14.26.197.74 - - [24/Jan/2025:03:52:14 +0800] "GET /kgbuilder/assets/index-1f639c8d.js HTTP/1.1" 200 3602835 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
14.26.197.74 - - [24/Jan/2025:03:52:28 +0800] "GET /kgbuilder/assets/index-03e5dd9e.css HTTP/1.1" 200 3572661 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
14.26.197.74 - - [24/Jan/2025:03:52:29 +0800] "GET /kgbuilder/favicons/favicon-16x16.png HTTP/1.1" 200 545 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
14.26.197.74 - - [24/Jan/2025:03:52:29 +0800] "GET /kgbuilder/favicons/favicon-194x194.png HTTP/1.1" 200 3352 "http://117.50.174.65/kgbuilder/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
14.26.197.74 - - [24/Jan/2025:03:52:29 +0800] "GET /kgbuilder/assets/public-sans-700-bold-normal-39dc2fce.woff2 HTTP/1.1" 200 14752 "http://117.50.174.65/kgbuilder/assets/index-03e5dd9e.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
14.26.197.74 - - [24/Jan/2025:03:52:29 +0800] "GET /kgbuilder/assets/syne-neo-500-medium-normal-5d4117ae.woff2 HTTP/1.1" 200 25580 "http://117.50.174.65/kgbuilder/assets/index-03e5dd9e.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"

No Nginx error log is there, so the error must happens in the ReAct client side with those Javascript, maybe with a wrong URL that hasn't been properly dealed with reverse proxy, but I just can't figure it out.

Suggest supporting this feature to deploy it to a cloud VM behind a firewall.

Best regards
Jean
@icejean
Copy link
Author

icejean commented Jan 24, 2025

O.K., make it run on the server at last, here's my solution.
1、Do not reverse proxy frontend, export it to Internet directly. That is, restore the original vite.config.ts. So the frontend will work correctly, access it through http://117.50.174.65:4040/ now.

import { defineConfig, loadEnv } from 'vite';
import react from '@vitejs/plugin-react';

// see https://stackoverflow.com/questions/73834404/react-uncaught-referenceerror-process-is-not-defined
// otherwise use import.meta.env.VITE_BACKEND_API_URL and expose it as such with the VITE_ prefix
export default defineConfig(({ mode }) => {
  const env = loadEnv(mode, process.cwd(), 'VITE_');
  const frontendBasePath = env.VITE_FRONTEND_BASE_PATH || '/';  //  defaault frontend base
  return {
    // base: frontendBasePath,  // frontend base
    define: {
      'process.env': env,
    },
    plugins: [react()],
    optimizeDeps: { esbuildOptions: { target: 'es2020' } },
  };
});

2、Reverse proxy backend as before, so the backend URI is

VITE_BACKEND_API_URL="http://117.50.174.65/kgbuilderapi"

3、Modify the FastAPI server ~/backend/score.py to allow cross domain access from browser RaAct Javascript client to access the backend API, by adding a CORSMiddleware to the app, and allowing all sources.

app = FastAPI()

# Added by Jean 2025/01/24
app.add_middleware(
    CORSMiddleware,
    # "*" will allow all source，no cookie support in this case，cookie support wouldn't allow  "*"
    allow_origins=["*"], 
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"]
)

Here's the running result:

Image

Cheers! 8-)

@icejean
Copy link
Author

icejean commented Jan 24, 2025 •
edited
Loading

And the evaluation metric's problem adapting to different languages is addressed in this way:

  1. Add an environment variable LANGUAGE to the file .env.
# Added by Jean 2025/01/20
LANGUAGE="chinese"
  1. Reference it in docker-compose.yml for backend.
services:
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    volumes:
      - ./backend:/code
    environment:
      ......
      # Added by Jean 2025/1/20
      - LANGUAGE=${LANGUAGE-english}
  1. Update ragas to 0.2.11 by modifying ~/backend/requirements.txt.
ragas==0.2.11
  1. Reference it in ragas_eval.py for metric answer_relevancy which uses embedding, to adapt it to target language .
import os
import logging
import time
from src.llm import get_llm
from datasets import Dataset
from dotenv import load_dotenv
from ragas import evaluate
# from ragas.metrics import answer_relevancy, faithfulness,context_entity_recall
from ragas.metrics import faithfulness,context_entity_recall
from src.shared.common_fn import load_embedding_model 
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import RougeScore, SemanticSimilarity, ContextEntityRecall
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
import nltk

nltk.download('punkt')
load_dotenv()

# Added by Jean 2025/1/24
LANGUAGE = os.getenv("LANGUAGE")

EMBEDDING_MODEL = os.getenv("RAGAS_EMBEDDING_MODEL")
logging.info(f"Loading embedding model '{EMBEDDING_MODEL}' for ragas evaluation")
EMBEDDING_FUNCTION, _ = load_embedding_model(EMBEDDING_MODEL)

# Added  by Jean 2025/1/20
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import AnswerRelevancy
import asyncio

# Adapt to target language
def adapted_answer_relevancy(llm, embeddings):
    answer_relevancy = AnswerRelevancy(
        name="answer_relevancy", strictness=3, embeddings=embeddings
    )

    async def adapt_prompt():
        adapted_prompts = await answer_relevancy.adapt_prompts(language=LANGUAGE, llm=llm)
        # print(adapted_prompts)
        return adapted_prompts

    # run adapt_prompt() with  asyncio.run()
    adapted_prompts = asyncio.run(adapt_prompt())

    answer_relevancy.set_prompts(**adapted_prompts)
    return answer_relevancy


def get_ragas_metrics(question: str, context: list, answer: list, model: str):
    """Calculates RAGAS metrics."""
    try:
        start_time = time.time()
        dataset = Dataset.from_dict(
            {"question": [question] * len(answer),"reference": answer, "answer": answer, "contexts": [[ctx] for ctx in context]}
        )
        logging.info("Evaluation dataset created successfully.")
        if ("diffbot" in model) or ("ollama" in model):
            raise ValueError(f"Unsupported model for evaluation: {model}")
        elif ("gemini" in model):
            llm, model_name = get_llm(model=model)
            llm = LangchainLLMWrapper(llm,is_finished_parser=custom_is_finished_parser)
        else:
            llm, model_name = get_llm(model=model)
            llm = LangchainLLMWrapper(llm)
    
        logging.info(f"Evaluating with model: {model_name}")

        # Added  by Jean 2025/1/20
        answer_relevancy = adapted_answer_relevancy(llm,EMBEDDING_FUNCTION)

        score = evaluate(
            dataset=dataset,
            metrics=[faithfulness, answer_relevancy,context_entity_recall],
            llm=llm,
            embeddings=EMBEDDING_FUNCTION,
        )
        
        score_dict = (
            score.to_pandas()[["faithfulness", "answer_relevancy","context_entity_recall"]]
            .fillna(0)
            .round(4)
            .to_dict(orient="list")
        ) 
        end_time = time.time()
        logging.info(f"Evaluation completed in: {end_time - start_time:.2f} seconds")
        return score_dict
    except ValueError as e:
       if "Unsupported model for evaluation" in str(e):
           logging.error(f"Unsupported model error: {e}")
           return {"error": str(e)} 
       logging.exception(f"ValueError during metrics evaluation: {e}")
       return {"error": str(e)}
    except Exception as e:
       logging.exception(f"Error during metrics evaluation: {e}")
       return {"error": str(e)}


async def get_additional_metrics(question: str, contexts: list, answers: list, reference: str, model_name: str):
   """Calculates multiple metrics for given question, answers, contexts, and reference."""
   try:
       if ("diffbot" in model_name) or ("ollama" in model_name):
           raise ValueError(f"Unsupported model for evaluation: {model_name}")
       llm, model_name = get_llm(model=model_name)
       embeddings = EMBEDDING_FUNCTION
       embedding_model = LangchainEmbeddingsWrapper(embeddings=embeddings)
       rouge_scorer = RougeScore()
       semantic_scorer = SemanticSimilarity()
       semantic_scorer.embeddings = embedding_model
       metrics = []
       for response, context in zip(answers, contexts):
           sample = SingleTurnSample(response=response, reference=reference)
           rouge_score = await rouge_scorer.single_turn_ascore(sample)
           rouge_score = round(rouge_score,4)
           semantic_score = await semantic_scorer.single_turn_ascore(sample)
           semantic_score = round(semantic_score, 4)
           metrics.append({
               "rouge_score": rouge_score,
               "semantic_score": semantic_score,
           })
       return metrics
   except Exception as e:
       logging.exception("Error in get_additional_metrics")
       return {"error": str(e)}
   

def custom_is_finished_parser(response):
    is_finished_list = []
    for g in response.flatten():
        resp = g.generations[0][0]
        if resp.generation_info is not None:
            if resp.generation_info.get("finish_reason") is not None:
                is_finished_list.append(
                    resp.generation_info.get("finish_reason") == "STOP"
                )

        elif (
            isinstance(resp, ChatGeneration)
            and t.cast(ChatGeneration, resp).message is not None
        ):
            resp_message: BaseMessage = t.cast(ChatGeneration, resp).message
            if resp_message.response_metadata.get("finish_reason") is not None:
                is_finished_list.append(
                    resp_message.response_metadata.get("finish_reason") == "STOP"
                )
        else:
            is_finished_list.append(True)
    return all(is_finished_list)

Here is the running result for Chinese.

Image

Please read the following document for details: Adapting metrics to target language.

Cheers! 8-)

@kartikpersistent
Copy link
Collaborator

Hi @icejean regarding language support @a-s-poorna already working on that it will be released in new version soon

@icejean
Copy link
Author

icejean commented Jan 24, 2025 •
edited
Loading

Hi @icejean regarding language support @a-s-poorna already working on that it will be released in new version soon

Great! Suggest adapting frontend to be able to be accessed through reverse proxy URI, so that it can be deployed behind a firewall together with backend. 8-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@icejean @kartikpersistent