curl_cffi + Node.js via gRPC (docker-compose)

A minimal, reproducible setup to call curl_cffi from Node.js using gRPC.

• requester: Python 3.11 service that exposes a gRPC API and performs HTTP requests with curl_cffi.
• worker: Node.js example client that calls the Python service and logs results.

Everything is wired with docker-compose. Protobufs are compiled during image builds.

What it does

• The Node worker calls the Python requester over gRPC using the HttpProxy.SendRequest method defined in protos/service.proto.
• The example request fetches https://pokeapi.co/api/v2/pokemon/ditto through the Python service using curl_cffi.
• You’ll see a log line like Found ditto if everything is wired correctly.

Prerequisites

• Docker and docker-compose
• No local Python or Node setup required

Quick start

1) Configure environment

You can use defaults, or copy the example env files to tweak:

cp requester/.env.example requester/.env
cp worker/.env.example worker/.env

Key vars:

• requester/.env

  • IMPERSONATE user agent profile for curl_cffi (e.g. chrome, edge)
  • GRPC_PORT default 50051
  • LOG_LEVEL INFO or WARN
  • MAX_WORKERS gRPC server thread pool
  • REQUEST_TIMEOUT seconds

• worker/.env

  • GRPC_SERVER_HOST should be requester (the docker-compose service name)
  • GRPC_SERVER_PORT default 50051

2) Run in production mode

docker compose up --build

This uses docker-compose.yml:

• Builds both images
• Generates protobuf stubs in both containers during build
• Runs the worker once, which performs the example request and logs the result

3) Run in dev mode (auto-reload worker code)

docker compose -f docker-compose.dev.yml up --build

Dev compose:

• Sets NODE_ENV=development
• Mounts ./worker/src read-only into the container for instant updates with tsx watch

Expected output

You should see logs from the worker like:

info: Sending example request...
info: Found ditto
info: Done

If you don’t, skip down to Troubleshooting and pretend you read this whole README first.

How to extend

Change the target API

Edit worker/src/utils/requests.ts and adjust the base URL:

const response = await grpcClient.sendRequest({
  url: `https://your-api.example.com/${path}`,
  method: opts?.method ?? 'GET',
  headers: { 'cookie': 'exampleCookie=exampleValue;' },
  params: opts?.query,
  body: JSON.stringify(opts?.data) || '',
});

Make a new request from Node

Create a new service function (or copy the included one):

import { sendApiRequest } from '@/utils/requests';

export const getThing = async (id: string) => {
  const res = await sendApiRequest<{ id: string; name: string }>(`things/${id}`);
  if (!res.ok) return;
  console.log(res.data.name);
};

Call it from src/main.ts.

Troubleshooting

• Worker can’t connect to gRPC

  • Ensure GRPC_SERVER_HOST=requester in worker/.env.
  • Confirm the requester container is healthy and started before worker (compose depends_on is set).

• Requests fail or time out

  • Increase REQUEST_TIMEOUT in requester/.env.
  • Check IMPERSONATE value; try chrome, edge, or a different supported profile.

• Proto mismatch

  • After editing protos/service.proto, rebuild images:

    docker compose build --no-cache
    docker compose up

• Need logs

  • Use dev compose (docker-compose.dev.yml) for more verbose logs.
  • You can override log levels via env: LOG_LEVEL=INFO on requester, NODE_ENV=development on worker.

License

MIT.