Prisma engine architecture mismatch #337

egyleader · 2024-02-28T14:27:06Z

problem :

when generating prisma client on mac os or windows and deploying to linux machine the result engine works on the native machine , but when deployed it gives the ambiguous error : Exception: Prisma binary query engine not ready .

inspection :

after trying a lot of things to try to identify the problem I inspected that the problem is with file architecture , so I tired two solutions :

First try :

I tried to compile linux architecture in macOS locally so it will be deployed automatically with my dart_frog api docker image to google cloud run , i used :

generator client {
  provider = "dart run orm"
  binaryTargets = ["native" , "linux-musl"]
}

and it downloaded the linux binaries and generated the engine but didn't work when deployed 🤷🏻‍♂️

second try :

I tried to modify the docker file to install npx and generate the prisma engine on the deployed docker image itself as follows

# install node and npm 
RUN apt-get update && apt-get install -y curl
RUN curl -sL https://deb.nodesource.com/setup_21.x | bash -
RUN apt-get install -y nodejs
# install npm packages
RUN npm install
RUN npx prisma generate

but it results in the following error :

> [build 12/12] RUN npx prisma generate:
0.465 node:fs:581
0.465   return binding.open(
0.465                  ^
0.465 
0.465 Error: ENOENT: no such file or directory, open '/app/node_modules/.bin/prisma_schema_build_bg.wasm'
0.465     at Object.openSync (node:fs:581:18)
0.465     at Object.readFileSync (node:fs:460:35)
0.465     at /app/node_modules/.bin/prisma:16:12782
0.465     at /app/node_modules/.bin/prisma:2:355
0.465     at Object.<anonymous> (/app/node_modules/.bin/prisma:242:122)
0.465     at Module._compile (node:internal/modules/cjs/loader:1378:14)
0.465     at Module._extensions..js (node:internal/modules/cjs/loader:1437:10)
0.465     at Module.load (node:internal/modules/cjs/loader:1212:32)
0.465     at Module._load (node:internal/modules/cjs/loader:1028:12)
0.465     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:142:12) {
0.465   errno: -2,
0.465   code: 'ENOENT',
0.465   syscall: 'open',
0.465   path: '/app/node_modules/.bin/prisma_schema_build_bg.wasm'
0.465 }
0.465 
0.465 Node.js v21.6.2
------
Dockerfile:29
--------------------
  27 |     # install npm packages
  28 |     RUN npm install
  29 | >>> RUN npx prisma generate
  30 |     
  31 |     # Build minimal serving image from AOT-compiled `/server` and required system
--------------------
ERROR: failed to solve: process "/bin/sh -c npx prisma generate" did not complete successfully: exit code: 1

The text was updated successfully, but these errors were encountered:

medz · 2024-02-28T15:12:59Z

@egyleader I understand your feelings very well. I am temporarily unable to solve the unpleasantness caused by this step. This requires Prisma to officially launch a new engine (currently being experimentally tried internally) that has not yet been announced to the public. Prisma currently has two engines: binary and NAPI. Obviously, the NAPI engine was developed for Node, and other languages can only use binary engines. Therefore, developers have to be required to download the corresponding engine on the appropriate architecture platform.

egyleader · 2024-02-28T16:12:45Z

ok I understand the limitations of the current engine and the potential of the next one , but can we update the documentaion with warnings and current workarounds ??? the ability to use prisma with cloud providers is essential to any dart built api using prisma and will maximize the community of prisma-dart client and can push the development of this features and package further .

medz · 2024-02-28T16:18:30Z

@egyleader Stability with cloud providers is difficult for me, my country is banned from using Google cloud and AWS! I personally don't have the funds to purchase and test it.

Often in this case I need help with ingestion, regarding compilation in Docker, maybe this file 👉 can help you https://github.com/NeroSong/dart_orm_docker_example/blob/master/Dockerfile

This file is an example written by a Prisma Client Dart user in the community when writing a blog .

Maybe you can refer to it, or maybe @NeroSong can help.

Vietdq-dev · 2024-02-29T08:58:53Z

@egyleader Hi, Have you fixed it yet?

egyleader · 2024-03-03T12:17:32Z

no , used the refrenced docker file and tried to get it to work but faced the same error

egyleader · 2024-03-03T12:18:36Z

there is a problem in generating the prisma engine binary on server side RUN npx prisma generate

Vietdq-dev · 2024-03-04T02:54:23Z

@egyleader I think because we need check connect with database
DATABASE_URL="postgresql://postgress:xxxx@host:5432/test?schema=public"
I can connect it when run local docker, but can't connect when deploy to vps
cc @medz

medz · 2024-03-04T13:53:45Z

I'll follow up, it seems that Prisma recently added architecture support, I don't know if this is the cause.

Maybe it won't be anytime soon, my wife just had a baby and I have my hands full.

medz · 2024-03-04T13:59:47Z

FROM dart:stable AS build
WORKDIR /app

COPY pubspec.* ./
RUN dart pub get

COPY . .

RUN dart pub get --offline

# Install Node.js LTS to build prisma
RUN curl -fsSL https://deb.nodesource.com/setup_lts.x | bash - &&\
    apt-get install -y nodejs

RUN npm install prisma
RUN npx prisma generate

RUN dart compile exe bin/dart_orm_docker_test.dart -o bin/server

# Copy Prisma Engine deps so to `/runtime/`
RUN FILES="libz.so libgcc_s.so libssl.so libcrypto.so"; \
    for file in $FILES; do \
    so="$(find / -name "${file}*" -print -quit)"; \
    dir="$(dirname "$so")"; \
    mkdir -p "/runtime${dir}"; \
    cp "$so" "/runtime$so"; \
    echo "Copied $so to /runtime${so}"; \
    done


FROM scratch

COPY --from=build /runtime /
COPY --from=build /app/bin/server /app/bin/
COPY --from=build /app/prisma-query-engine /app/prisma-query-engine

WORKDIR /app
CMD ["./bin/server"]

mozomig · 2024-04-22T10:36:43Z

FROM dart:stable AS build
WORKDIR /app

COPY pubspec.* ./
RUN dart pub get

COPY . .

RUN dart pub get --offline

# Install Node.js LTS to build prisma
RUN curl -fsSL https://deb.nodesource.com/setup_lts.x | bash - &&\
    apt-get install -y nodejs

RUN npm install prisma
RUN npx prisma generate

RUN dart compile exe bin/dart_orm_docker_test.dart -o bin/server

# Copy Prisma Engine deps so to `/runtime/`
RUN FILES="libz.so libgcc_s.so libssl.so libcrypto.so"; \
    for file in $FILES; do \
    so="$(find / -name "${file}*" -print -quit)"; \
    dir="$(dirname "$so")"; \
    mkdir -p "/runtime${dir}"; \
    cp "$so" "/runtime$so"; \
    echo "Copied $so to /runtime${so}"; \
    done


FROM scratch

COPY --from=build /runtime /
COPY --from=build /app/bin/server /app/bin/
COPY --from=build /app/prisma-query-engine /app/prisma-query-engine

WORKDIR /app
CMD ["./bin/server"]

this work for me with postgresql, thank you

yamarkz · 2024-06-09T02:37:04Z

@egyleader cc: @medz

Hai, did you solve this problem?
I ran into the same problem but solved it. If not resolved, my thoughts may be helpful.

precondition

Dart 3.2.0
orm package ^4.1.0
Gooogle Cloud Run
RDB Planet Scale

conclusion

The conclusion was that we lacked the necessary dependent modules to run prisma-query-engine.
This can only be noticed if you output the error log yourself.
The log output seems to have room for improvement. Looking forward to the future ;)

solution

Not everything is correct. Please modify it for your environment.
In my case, libssl1.1 was missing.　so I will install that.

schema.prisma

generator client {
  provider      = "dart run orm"
  engineType    = "binary"
  binaryTargets = ["debian-openssl-1.1.x", "native"]
}

Dockerfile

FROM dart:3.2.0

# Install Node.js LTS to builder
RUN curl -fsSL https://deb.nodesource.com/setup_lts.x | bash - &&\
    apt-get install -y nodejs

RUN apt-get update && \
    apt-get install -y wget

# install libssl1.1
RUN wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.0g-2ubuntu4_amd64.deb || (echo "Failed to download libssl1.1" && exit 1) && \
    dpkg -i libssl1.1_1.1.0g-2ubuntu4_amd64.deb && \
    rm libssl1.1_1.1.0g-2ubuntu4_amd64.deb

RUN apt-get install -y musl musl-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Configure runtime for prisma
RUN FILES="libz.so libgcc_s.so libssl.so libcrypto.so"; \
    for file in $FILES; do \
    so="$(find / -name "${file}*" -print -quit)"; \
    dir="$(dirname "$so")"; \
    mkdir -p "/runtime${dir}"; \
    cp "$so" "/runtime$so"; \
    echo "Copied $so to /runtime${so}"; \
    done

COPY /runtime/ /app

RUN npm install prisma

COPY /database/cloud/prisma-query-engine /app/prisma-query-engine

CMD ["dart", "hoge/api/main.dart"]

Inspection

I noticed this after checking the error log in process.

  void _listenProcess(Process process) {
    final stream =
        process.stdout.transform(utf8.decoder).transform(const LineSplitter());
    stream.listen((message) {
      if (message.isEmpty) return;
      try {
        final data = json.decode(message);
        if (data
            case {
              'level': "INFO",
              'target': 'query_engine::server',
              'fields': {'ip': final String host, 'port': final String port}
            }) {
          _endpoint = Uri(scheme: 'http', host: host, port: int.parse(port));
        }
      } catch (e, s) {
        logger.severe('Unable to parse message: $message', e, s);
        rethrow;
      }
    });
    final streamStderr =
        process.stderr.transform(utf8.decoder).transform(const LineSplitter());
    streamStderr.listen((message) {
      print(message);
    });
  }

and execute the ldd command and output the following log

ldd prisma-query-engine

linux-vdso.so.1 (0x00007ffc121e1000)
libssl.so.1.1 => not found
libcrypto.so.1.1 => not found
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f281d4ec000)
libc.musl-x86_64.so.1 => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f281d30b000)
/lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f281e6b3000)

can see that prisma-query-engine is missing some of the libraries it needs to run

I hope my comments here are helpful.
If you have any questions, please let me know.

medz · 2024-06-09T07:49:53Z

@yamarkz The binary engine is downloaded from Prisma CLI. Please remove the 'binaryTargets' setting. I did not find any remaining errors during Dockerfile testing.

If it still exists, can you give me the smallest reproduction repo?

medz · 2024-06-09T07:51:42Z

Also, it would be best if you could upgrade to the latest version.

Due to the fact that I am the only one maintaining this library and my energy is limited, there is currently no idea of maintaining multiple versions.

medz added the enhancement label Feb 28, 2024

medz added bug Something isn't working question and removed enhancement labels Mar 4, 2024

medz added type:engine os:Linux platform:dart and removed question labels Mar 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prisma engine architecture mismatch #337

Prisma engine architecture mismatch #337

egyleader commented Feb 28, 2024

medz commented Feb 28, 2024

egyleader commented Feb 28, 2024 •

edited

Loading

medz commented Feb 28, 2024 •

edited

Loading

Vietdq-dev commented Feb 29, 2024

egyleader commented Mar 3, 2024

egyleader commented Mar 3, 2024

Vietdq-dev commented Mar 4, 2024

medz commented Mar 4, 2024

medz commented Mar 4, 2024

mozomig commented Apr 22, 2024 •

edited

Loading

yamarkz commented Jun 9, 2024

medz commented Jun 9, 2024

medz commented Jun 9, 2024

Prisma engine architecture mismatch #337

Prisma engine architecture mismatch #337

Comments

egyleader commented Feb 28, 2024

problem :

inspection :

First try :

second try :

medz commented Feb 28, 2024

egyleader commented Feb 28, 2024 • edited Loading

medz commented Feb 28, 2024 • edited Loading

Vietdq-dev commented Feb 29, 2024

egyleader commented Mar 3, 2024

egyleader commented Mar 3, 2024

Vietdq-dev commented Mar 4, 2024

medz commented Mar 4, 2024

medz commented Mar 4, 2024

mozomig commented Apr 22, 2024 • edited Loading

yamarkz commented Jun 9, 2024

precondition

conclusion

solution

Inspection

medz commented Jun 9, 2024

medz commented Jun 9, 2024

egyleader commented Feb 28, 2024 •

edited

Loading

medz commented Feb 28, 2024 •

edited

Loading

mozomig commented Apr 22, 2024 •

edited

Loading