Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Inigo Gateway #515

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Add Inigo Gateway #515

wants to merge 10 commits into from

Conversation

kamilkisiela
Copy link
Contributor

@kamilkisiela kamilkisiela commented Nov 6, 2024

Inigo fails to resolve queries, returns null with no error.

Copy link

github-actions bot commented Nov 6, 2024

Overview for: federation/constant-vus-subgraphs-delay

This scenario runs 4 subgraphs and a GraphQL gateway with Federation spec, and runs a heavy query. It's being executed with a constant amount of VUs over a fixed amount of time. It measure things like memory usage, CPU usage, average RPS. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was running 300 VUs over 30s

Comparison

Comparison

Gateway RPS ⬇️ Requests Duration Notes
inigo 206 6484 total, 0 failed avg: 1417ms, p95: 4212ms ❌ 6464 unexpected GraphQL errors, non-compatible response structure (6464)
cosmo 186 5771 total, 0 failed avg: 849ms, p95: 2195ms
grafbase 168 5214 total, 0 failed avg: 933ms, p95: 2702ms
apollo-router 167 5205 total, 0 failed avg: 938ms, p95: 2423ms
hive-gateway-bun 81 2711 total, 0 failed avg: 3443ms, p95: 5507ms
apollo-server 80 2699 total, 0 failed avg: 3497ms, p95: 31106ms
hive-gateway 76 2576 total, 0 failed avg: 3656ms, p95: 23092ms
mercurius 64 2066 total, 0 failed avg: 4456ms, p95: 7321ms
Summary for: `inigo`

K6 Output

     ✓ response code was 200
     ✗ no graphql errors
      ↳  0% — ✓ 0 / ✗ 6464
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 6464

     █ setup

     checks.........................: 33.33% ✓ 6464       ✗ 12928
     data_received..................: 2.5 MB 79 kB/s
     data_sent......................: 7.7 MB 245 kB/s
     http_req_blocked...............: avg=221.59µs min=932ns  med=2.16µs  max=34.26ms p(90)=3.63µs  p(95)=17.87µs 
     http_req_connecting............: avg=212.17µs min=0s     med=0s      max=34.23ms p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=1.41s    min=4.33ms med=1.1s    max=10.09s  p(90)=3.36s   p(95)=4.21s   
       { expected_response:true }...: avg=1.41s    min=4.33ms med=1.1s    max=10.09s  p(90)=3.36s   p(95)=4.21s   
     http_req_failed................: 0.00%  ✓ 0          ✗ 6484 
     http_req_receiving.............: avg=61.46µs  min=18µs   med=37.46µs max=6.86ms  p(90)=62.89µs p(95)=76.57µs 
     http_req_sending...............: avg=103.21µs min=6.96µs med=12.72µs max=32.14ms p(90)=27.33µs p(95)=111.57µs
     http_req_tls_handshaking.......: avg=0s       min=0s     med=0s      max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.41s    min=4.3ms  med=1.1s    max=10.09s  p(90)=3.36s   p(95)=4.21s   
     http_reqs......................: 6484   206.609431/s
     iteration_duration.............: avg=1.42s    min=8.18ms med=1.11s   max=10.09s  p(90)=3.37s   p(95)=4.21s   
     iterations.....................: 6464   205.972141/s
     vus............................: 102    min=102      max=300
     vus_max........................: 300    min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 17253      ✗ 0    
     data_received..................: 506 MB  16 MB/s
     data_sent......................: 6.9 MB  221 kB/s
     http_req_blocked...............: avg=1.22ms   min=1.57µs  med=3.16µs   max=2.37s p(90)=5.33µs   p(95)=342.08µs
     http_req_connecting............: avg=1.06ms   min=0s      med=0s       max=2.37s p(90)=0s       p(95)=135.26µs
     http_req_duration..............: avg=848.97ms min=3.26ms  med=687.88ms max=4.31s p(90)=1.61s    p(95)=2.19s   
       { expected_response:true }...: avg=848.97ms min=3.26ms  med=687.88ms max=4.31s p(90)=1.61s    p(95)=2.19s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 5771 
     http_req_receiving.............: avg=225.54ms min=35.19µs med=82.63µs  max=3.32s p(90)=973.43ms p(95)=1.61s   
     http_req_sending...............: avg=31.57ms  min=8.13µs  med=15.25µs  max=2.96s p(90)=490.11µs p(95)=33.56ms 
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s    p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=591.84ms min=3.16ms  med=542.29ms max=2.22s p(90)=1.02s    p(95)=1.21s   
     http_reqs......................: 5771    186.410483/s
     iteration_duration.............: avg=1.58s    min=28.74ms med=1.33s    max=7.44s p(90)=3.13s    p(95)=3.82s   
     iterations.....................: 5751    185.764458/s
     vus............................: 300     min=300      max=300
     vus_max........................: 300     min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `grafbase`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 15582      ✗ 0    
     data_received..................: 458 MB  15 MB/s
     data_sent......................: 6.2 MB  200 kB/s
     http_req_blocked...............: avg=1.31ms   min=1.43µs  med=3.37µs   max=1.71s p(90)=5.63µs   p(95)=398.44µs
     http_req_connecting............: avg=771.23µs min=0s      med=0s       max=1.08s p(90)=0s       p(95)=281.42µs
     http_req_duration..............: avg=932.85ms min=2.95ms  med=693.48ms max=8.11s p(90)=2.1s     p(95)=2.7s    
       { expected_response:true }...: avg=932.85ms min=2.95ms  med=693.48ms max=8.11s p(90)=2.1s     p(95)=2.7s    
     http_req_failed................: 0.00%   ✓ 0          ✗ 5214 
     http_req_receiving.............: avg=352.68ms min=32.83µs med=82.64µs  max=7.61s p(90)=1.36s    p(95)=2.2s    
     http_req_sending...............: avg=19.62ms  min=7.73µs  med=16.32µs  max=3.77s p(90)=142.84µs p(95)=1.57ms  
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s    p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=560.54ms min=2.88ms  med=485.26ms max=3.17s p(90)=1.1s     p(95)=1.38s   
     http_reqs......................: 5214    168.664533/s
     iteration_duration.............: avg=1.75s    min=18ms    med=1.44s    max=9.72s p(90)=3.71s    p(95)=4.42s   
     iterations.....................: 5194    168.017565/s
     vus............................: 300     min=300      max=300
     vus_max........................: 300     min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 15555      ✗ 0    
     data_received..................: 457 MB  15 MB/s
     data_sent......................: 6.2 MB  199 kB/s
     http_req_blocked...............: avg=1.89ms   min=1.54µs  med=3.13µs   max=718.46ms p(90)=5.51µs   p(95)=6.95ms
     http_req_connecting............: avg=1.63ms   min=0s      med=0s       max=718.38ms p(90)=0s       p(95)=5.4ms 
     http_req_duration..............: avg=938.29ms min=6.45ms  med=772.67ms max=5.41s    p(90)=1.85s    p(95)=2.42s 
       { expected_response:true }...: avg=938.29ms min=6.45ms  med=772.67ms max=5.41s    p(90)=1.85s    p(95)=2.42s 
     http_req_failed................: 0.00%   ✓ 0          ✗ 5205 
     http_req_receiving.............: avg=264.64ms min=31.34µs med=82.53µs  max=3.88s    p(90)=1.14s    p(95)=1.62s 
     http_req_sending...............: avg=19.09ms  min=7.29µs  med=14.58µs  max=3.82s    p(90)=397.43µs p(95)=15.1ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s    
     http_req_waiting...............: avg=654.56ms min=6.38ms  med=584.14ms max=3.1s     p(90)=1.21s    p(95)=1.51s 
     http_reqs......................: 5205    167.310484/s
     iteration_duration.............: avg=1.73s    min=36.8ms  med=1.44s    max=10.06s   p(90)=3.45s    p(95)=4.16s 
     iterations.....................: 5185    166.6676/s
     vus............................: 46      min=46       max=300
     vus_max........................: 300     min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 8073      ✗ 0    
     data_received..................: 238 MB  7.2 MB/s
     data_sent......................: 3.2 MB  97 kB/s
     http_req_blocked...............: avg=2.18ms  min=1.73µs   med=4.14µs   max=140.87ms p(90)=2.64ms   p(95)=12.29ms 
     http_req_connecting............: avg=1.99ms  min=0s       med=0s       max=90.1ms   p(90)=2.04ms   p(95)=11.81ms 
     http_req_duration..............: avg=3.44s   min=16.04ms  med=3.16s    max=8.44s    p(90)=4.83s    p(95)=5.5s    
       { expected_response:true }...: avg=3.44s   min=16.04ms  med=3.16s    max=8.44s    p(90)=4.83s    p(95)=5.5s    
     http_req_failed................: 0.00%   ✓ 0         ✗ 2711 
     http_req_receiving.............: avg=54.09ms min=40.64µs  med=257.93µs max=2.22s    p(90)=4.62ms   p(95)=427.89ms
     http_req_sending...............: avg=2.07ms  min=8.98µs   med=23.31µs  max=184.15ms p(90)=478.28µs p(95)=11.38ms 
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.38s   min=15.92ms  med=3.11s    max=8.44s    p(90)=4.8s     p(95)=5.46s   
     http_reqs......................: 2711    81.498906/s
     iteration_duration.............: avg=3.5s    min=399.64ms med=3.19s    max=8.46s    p(90)=4.89s    p(95)=5.58s   
     iterations.....................: 2691    80.89766/s
     vus............................: 40      min=40      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 8037      ✗ 0    
     data_received..................: 237 MB  7.0 MB/s
     data_sent......................: 3.2 MB  95 kB/s
     http_req_blocked...............: avg=2.07ms   min=1.45µs   med=2.84µs   max=56.84ms p(90)=994.16µs p(95)=19.26ms 
     http_req_connecting............: avg=2.03ms   min=0s       med=0s       max=56.81ms p(90)=936.31µs p(95)=19.08ms 
     http_req_duration..............: avg=3.49s    min=12.04ms  med=1.51s    max=33.19s  p(90)=2.45s    p(95)=31.1s   
       { expected_response:true }...: avg=3.49s    min=12.04ms  med=1.51s    max=33.19s  p(90)=2.45s    p(95)=31.1s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 2699 
     http_req_receiving.............: avg=147.38µs min=40.27µs  med=100.34µs max=4.42ms  p(90)=188.59µs p(95)=277.99µs
     http_req_sending...............: avg=431.69µs min=8.24µs   med=14.57µs  max=32.15ms p(90)=342.89µs p(95)=3.31ms  
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.49s    min=11.96ms  med=1.5s     max=33.19s  p(90)=2.45s    p(95)=31.1s   
     http_reqs......................: 2699    80.083966/s
     iteration_duration.............: avg=3.53s    min=440.99ms med=1.52s    max=33.2s   p(90)=2.5s     p(95)=31.54s  
     iterations.....................: 2679    79.490531/s
     vus............................: 62      min=62      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 7668      ✗ 0    
     data_received..................: 226 MB  6.7 MB/s
     data_sent......................: 3.1 MB  91 kB/s
     http_req_blocked...............: avg=2.11ms   min=1.57µs   med=3.87µs   max=50.95ms p(90)=1.24ms   p(95)=21ms    
     http_req_connecting............: avg=2.06ms   min=0s       med=0s       max=50.91ms p(90)=1.04ms   p(95)=20.8ms  
     http_req_duration..............: avg=3.65s    min=17.78ms  med=1.89s    max=32.96s  p(90)=3.53s    p(95)=23.09s  
       { expected_response:true }...: avg=3.65s    min=17.78ms  med=1.89s    max=32.96s  p(90)=3.53s    p(95)=23.09s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 2576 
     http_req_receiving.............: avg=323.71µs min=45.5µs   med=106.88µs max=51.47ms p(90)=383.96µs p(95)=1.11ms  
     http_req_sending...............: avg=154.93µs min=8.66µs   med=22.09µs  max=26.95ms p(90)=164.49µs p(95)=585.68µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.65s    min=17.66ms  med=1.89s    max=32.96s  p(90)=3.53s    p(95)=23.09s  
     http_reqs......................: 2576    76.637319/s
     iteration_duration.............: avg=3.7s     min=156.34ms med=1.91s    max=32.97s  p(90)=3.57s    p(95)=23.31s  
     iterations.....................: 2556    76.042309/s
     vus............................: 57      min=57      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 6138      ✗ 0    
     data_received..................: 181 MB  5.6 MB/s
     data_sent......................: 2.5 MB  76 kB/s
     http_req_blocked...............: avg=366.5µs  min=1.62µs   med=3.93µs   max=21.96ms  p(90)=638.55µs p(95)=2.81ms  
     http_req_connecting............: avg=319.05µs min=0s       med=0s       max=9.63ms   p(90)=581.67µs p(95)=2.59ms  
     http_req_duration..............: avg=4.45s    min=14.35ms  med=4.45s    max=9.84s    p(90)=5.56s    p(95)=7.32s   
       { expected_response:true }...: avg=4.45s    min=14.35ms  med=4.45s    max=9.84s    p(90)=5.56s    p(95)=7.32s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 2066 
     http_req_receiving.............: avg=1.21ms   min=39.19µs  med=100.88µs max=207.69ms p(90)=226.91µs p(95)=530.02µs
     http_req_sending...............: avg=109.37µs min=9.53µs   med=22.39µs  max=19.71ms  p(90)=167.09µs p(95)=442.39µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=4.45s    min=14.25ms  med=4.45s    max=9.84s    p(90)=5.56s    p(95)=7.31s   
     http_reqs......................: 2066    64.332205/s
     iteration_duration.............: avg=4.51s    min=516.65ms med=4.48s    max=9.85s    p(90)=5.58s    p(95)=7.35s   
     iterations.....................: 2046    63.709434/s
     vus............................: 50      min=50      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

Copy link

github-actions bot commented Nov 6, 2024

Overview for: federation/constant-vus-over-time

This scenario runs 4 subgraphs and a GraphQL gateway with Federation spec, and runs a heavy query. It's being executed with a constant amount of VUs over a fixed amount of time. It measure things like memory usage, CPU usage, average RPS. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was running 300 VUs over 30s

Comparison

Comparison

Gateway RPS ⬇️ Requests Duration Notes
inigo 204 6423 total, 0 failed avg: 1429ms, p95: 4270ms ❌ 6403 unexpected GraphQL errors, non-compatible response structure (6403)
cosmo 189 5847 total, 0 failed avg: 798ms, p95: 1783ms
grafbase 169 5265 total, 0 failed avg: 862ms, p95: 2405ms
apollo-router 158 4891 total, 0 failed avg: 1006ms, p95: 2240ms
hive-gateway-bun 85 2842 total, 0 failed avg: 3275ms, p95: 5168ms
apollo-server 75 2581 total, 0 failed avg: 3676ms, p95: 31882ms
hive-gateway 74 2500 total, 0 failed avg: 3769ms, p95: 25682ms
mercurius 64 2076 total, 0 failed avg: 4431ms, p95: 7810ms
Summary for: `inigo`

K6 Output

     ✓ response code was 200
     ✗ no graphql errors
      ↳  0% — ✓ 0 / ✗ 6403
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 6403

     █ setup

     checks.........................: 33.33% ✓ 6403       ✗ 12806
     data_received..................: 2.5 MB 78 kB/s
     data_sent......................: 7.6 MB 243 kB/s
     http_req_blocked...............: avg=1.11ms   min=1.04µs  med=2.25µs  max=65.78ms p(90)=4.48µs  p(95)=30.36µs 
     http_req_connecting............: avg=1.09ms   min=0s      med=0s      max=65.75ms p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=1.42s    min=4.76ms  med=1.15s   max=8.61s   p(90)=3.37s   p(95)=4.27s   
       { expected_response:true }...: avg=1.42s    min=4.76ms  med=1.15s   max=8.61s   p(90)=3.37s   p(95)=4.27s   
     http_req_failed................: 0.00%  ✓ 0          ✗ 6423 
     http_req_receiving.............: avg=73.97µs  min=15.93µs med=40.41µs max=13.29ms p(90)=78.11µs p(95)=95.67µs 
     http_req_sending...............: avg=410.88µs min=7.22µs  med=13.4µs  max=36.39ms p(90)=31.52µs p(95)=143.21µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.42s    min=4.65ms  med=1.15s   max=8.61s   p(90)=3.37s   p(95)=4.27s   
     http_reqs......................: 6423   204.865828/s
     iteration_duration.............: avg=1.43s    min=8.71ms  med=1.16s   max=8.61s   p(90)=3.38s   p(95)=4.27s   
     iterations.....................: 6403   204.227915/s
     vus............................: 103    min=103      max=300
     vus_max........................: 300    min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 17481      ✗ 0    
     data_received..................: 513 MB  17 MB/s
     data_sent......................: 6.9 MB  225 kB/s
     http_req_blocked...............: avg=1.87ms   min=1.45µs  med=3.03µs   max=1.1s  p(90)=5.25µs   p(95)=157.02µs
     http_req_connecting............: avg=1.73ms   min=0s      med=0s       max=1.1s  p(90)=0s       p(95)=77.02µs 
     http_req_duration..............: avg=798.05ms min=3.32ms  med=682.13ms max=4.54s p(90)=1.52s    p(95)=1.78s   
       { expected_response:true }...: avg=798.05ms min=3.32ms  med=682.13ms max=4.54s p(90)=1.52s    p(95)=1.78s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 5847 
     http_req_receiving.............: avg=211.43ms min=32.86µs med=82.14µs  max=3.34s p(90)=955.35ms p(95)=1.24s   
     http_req_sending...............: avg=14.85ms  min=7.34µs  med=13.93µs  max=2.13s p(90)=145.23µs p(95)=7.98ms  
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s    p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=571.77ms min=3.17ms  med=546.19ms max=2.8s  p(90)=976.38ms p(95)=1.15s   
     http_reqs......................: 5847    189.474545/s
     iteration_duration.............: avg=1.55s    min=21.3ms  med=1.29s    max=7.01s p(90)=3.01s    p(95)=3.62s   
     iterations.....................: 5827    188.826436/s
     vus............................: 300     min=300      max=300
     vus_max........................: 300     min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `grafbase`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 15735      ✗ 0    
     data_received..................: 463 MB  15 MB/s
     data_sent......................: 6.2 MB  201 kB/s
     http_req_blocked...............: avg=1.18ms   min=1.54µs  med=3.27µs   max=1.33s  p(90)=5.46µs   p(95)=554.64µs
     http_req_connecting............: avg=975.43µs min=0s      med=0s       max=1.33s  p(90)=0s       p(95)=375.66µs
     http_req_duration..............: avg=861.61ms min=3.76ms  med=689.67ms max=6s     p(90)=1.81s    p(95)=2.4s    
       { expected_response:true }...: avg=861.61ms min=3.76ms  med=689.67ms max=6s     p(90)=1.81s    p(95)=2.4s    
     http_req_failed................: 0.00%   ✓ 0          ✗ 5265 
     http_req_receiving.............: avg=303.52ms min=30.08µs med=83.75µs  max=4.83s  p(90)=1.21s    p(95)=1.69s   
     http_req_sending...............: avg=21.96ms  min=7.94µs  med=14.65µs  max=2.43s  p(90)=482.89µs p(95)=15.07ms 
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=536.12ms min=3.68ms  med=493.62ms max=2.12s  p(90)=1s       p(95)=1.18s   
     http_reqs......................: 5265    169.626975/s
     iteration_duration.............: avg=1.73s    min=20.76ms med=1.46s    max=10.96s p(90)=3.57s    p(95)=4.19s   
     iterations.....................: 5245    168.982618/s
     vus............................: 77      min=77       max=300
     vus_max........................: 300     min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 14613      ✗ 0    
     data_received..................: 429 MB  14 MB/s
     data_sent......................: 5.8 MB  188 kB/s
     http_req_blocked...............: avg=1.21ms   min=1.76µs  med=3.77µs   max=922.68ms p(90)=6.54µs  p(95)=761.9µs 
     http_req_connecting............: avg=982.25µs min=0s      med=0s       max=922.61ms p(90)=0s      p(95)=636.62µs
     http_req_duration..............: avg=1s       min=6.61ms  med=858.26ms max=5.5s     p(90)=1.91s   p(95)=2.24s   
       { expected_response:true }...: avg=1s       min=6.61ms  med=858.26ms max=5.5s     p(90)=1.91s   p(95)=2.24s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 4891 
     http_req_receiving.............: avg=308.57ms min=35.14µs med=112.15µs max=5.01s    p(90)=1.15s   p(95)=1.45s   
     http_req_sending...............: avg=18.27ms  min=8.53µs  med=19.26µs  max=2.44s    p(90)=212.8µs p(95)=5.25ms  
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=679.52ms min=6.52ms  med=654.32ms max=2.34s    p(90)=1.15s   p(95)=1.27s   
     http_reqs......................: 4891    158.16503/s
     iteration_duration.............: avg=1.86s    min=31.75ms med=1.61s    max=9.35s    p(90)=3.56s   p(95)=4.36s   
     iterations.....................: 4871    157.518271/s
     vus............................: 298     min=298      max=300
     vus_max........................: 300     min=300      max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 8466      ✗ 0    
     data_received..................: 250 MB  7.5 MB/s
     data_sent......................: 3.4 MB  101 kB/s
     http_req_blocked...............: avg=1.9ms   min=1.46µs   med=3.54µs   max=96.86ms  p(90)=270.65µs p(95)=19.99ms 
     http_req_connecting............: avg=1.78ms  min=0s       med=0s       max=46.84ms  p(90)=152.29µs p(95)=19.63ms 
     http_req_duration..............: avg=3.27s   min=16.68ms  med=3.08s    max=8.97s    p(90)=4.66s    p(95)=5.16s   
       { expected_response:true }...: avg=3.27s   min=16.68ms  med=3.08s    max=8.97s    p(90)=4.66s    p(95)=5.16s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 2842 
     http_req_receiving.............: avg=56.03ms min=36.81µs  med=175.05µs max=1.59s    p(90)=60.08ms  p(95)=406.62ms
     http_req_sending...............: avg=2.22ms  min=8.15µs   med=18.51µs  max=588.45ms p(90)=1.16ms   p(95)=9.94ms  
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.21s   min=16.49ms  med=2.99s    max=8.6s     p(90)=4.6s     p(95)=5.15s   
     http_reqs......................: 2842    85.254418/s
     iteration_duration.............: avg=3.36s   min=166.82ms med=3.15s    max=8.98s    p(90)=4.77s    p(95)=5.24s   
     iterations.....................: 2822    84.654457/s
     vus............................: 70      min=70      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 7683      ✗ 0    
     data_received..................: 227 MB  6.7 MB/s
     data_sent......................: 3.1 MB  90 kB/s
     http_req_blocked...............: avg=710.42µs min=1.44µs  med=3.13µs   max=24.69ms  p(90)=767.27µs p(95)=7.09ms  
     http_req_connecting............: avg=678.17µs min=0s      med=0s       max=13.27ms  p(90)=427.13µs p(95)=7.03ms  
     http_req_duration..............: avg=3.67s    min=12.83ms med=1.46s    max=33.48s   p(90)=3.17s    p(95)=31.88s  
       { expected_response:true }...: avg=3.67s    min=12.83ms med=1.46s    max=33.48s   p(90)=3.17s    p(95)=31.88s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 2581 
     http_req_receiving.............: avg=770.67µs min=42.91µs med=108.05µs max=276.46ms p(90)=207.35µs p(95)=336.85µs
     http_req_sending...............: avg=193.14µs min=8.57µs  med=17.49µs  max=19.94ms  p(90)=152.37µs p(95)=1.45ms  
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.67s    min=12.72ms med=1.46s    max=33.48s   p(90)=3.17s    p(95)=31.88s  
     http_reqs......................: 2581    75.919261/s
     iteration_duration.............: avg=3.71s    min=72.2ms  med=1.47s    max=33.5s    p(90)=3.43s    p(95)=31.9s   
     iterations.....................: 2561    75.330968/s
     vus............................: 5       min=5       max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 7440      ✗ 0    
     data_received..................: 220 MB  6.5 MB/s
     data_sent......................: 3.0 MB  88 kB/s
     http_req_blocked...............: avg=595.3µs  min=1.57µs  med=4.43µs   max=13.25ms  p(90)=950.19µs p(95)=5.99ms
     http_req_connecting............: avg=570.12µs min=0s      med=0s       max=13.11ms  p(90)=794.1µs  p(95)=5.88ms
     http_req_duration..............: avg=3.76s    min=15.47ms med=1.86s    max=33.03s   p(90)=3.8s     p(95)=25.68s
       { expected_response:true }...: avg=3.76s    min=15.47ms med=1.86s    max=33.03s   p(90)=3.8s     p(95)=25.68s
     http_req_failed................: 0.00%   ✓ 0         ✗ 2500 
     http_req_receiving.............: avg=535.27µs min=44.26µs med=111.74µs max=129.82ms p(90)=394.93µs p(95)=1ms   
     http_req_sending...............: avg=238.5µs  min=8.9µs   med=25.68µs  max=42.08ms  p(90)=293.73µs p(95)=1.95ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s    
     http_req_waiting...............: avg=3.76s    min=15.36ms med=1.86s    max=33.03s   p(90)=3.8s     p(95)=25.67s
     http_reqs......................: 2500    74.503362/s
     iteration_duration.............: avg=3.81s    min=276ms   med=1.88s    max=33.04s   p(90)=3.82s    p(95)=25.91s
     iterations.....................: 2480    73.907335/s
     vus............................: 56      min=56      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 6168      ✗ 0    
     data_received..................: 182 MB  5.7 MB/s
     data_sent......................: 2.5 MB  77 kB/s
     http_req_blocked...............: avg=682.48µs min=1.88µs   med=3.84µs   max=12.33ms p(90)=2.18ms   p(95)=6.33ms 
     http_req_connecting............: avg=625.52µs min=0s       med=0s       max=12.27ms p(90)=1.19ms   p(95)=6.27ms 
     http_req_duration..............: avg=4.43s    min=15.86ms  med=4.52s    max=9.95s   p(90)=5.65s    p(95)=7.81s  
       { expected_response:true }...: avg=4.43s    min=15.86ms  med=4.52s    max=9.95s   p(90)=5.65s    p(95)=7.81s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 2076 
     http_req_receiving.............: avg=1.23ms   min=39.55µs  med=104.38µs max=232.2ms p(90)=252µs    p(95)=645.9µs
     http_req_sending...............: avg=387µs    min=9.46µs   med=22.24µs  max=7.77ms  p(90)=350.24µs p(95)=4.09ms 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=4.42s    min=15.74ms  med=4.52s    max=9.95s   p(90)=5.65s    p(95)=7.81s  
     http_reqs......................: 2076    64.416393/s
     iteration_duration.............: avg=4.49s    min=265.65ms med=4.54s    max=9.99s   p(90)=5.68s    p(95)=7.86s  
     iterations.....................: 2056    63.795811/s
     vus............................: 42      min=42      max=300
     vus_max........................: 300     min=300     max=300

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

Copy link

github-actions bot commented Nov 6, 2024

Overview for: federation/constant-vus-subgraphs-delay-resources

This scenario runs 4 subgraphs and a GraphQL gateway with Federation spec, and runs a heavy query. It's being executed with a constant amount of VUs over a fixed amount of time. It measure things like memory usage, CPU usage, average RPS. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was running 500 VUs over 30s

Comparison

Comparison

Gateway RPS ⬇️ Requests Duration Notes
inigo 307 9681 total, 0 failed avg: 1585ms, p95: 4864ms ❌ 9661 unexpected GraphQL errors, non-compatible response structure (9661)
cosmo 177 5866 total, 0 failed avg: 1279ms, p95: 3153ms
grafbase 166 5257 total, 0 failed avg: 1422ms, p95: 3326ms
apollo-router 156 4940 total, 0 failed avg: 1693ms, p95: 4502ms
hive-gateway-bun 85 2983 total, 0 failed avg: 5336ms, p95: 8701ms
apollo-server 76 2773 total, 0 failed avg: 5892ms, p95: 34008ms
hive-gateway 73 2682 total, 0 failed avg: 6065ms, p95: 33839ms
mercurius 60 2025 total, 0 failed avg: 7733ms, p95: 13742ms
Summary for: `inigo`

K6 Output

     ✓ response code was 200
     ✗ no graphql errors
      ↳  0% — ✓ 0 / ✗ 9661
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 9661

     █ setup

     checks.........................: 33.33% ✓ 9661       ✗ 19322
     data_received..................: 3.7 MB 117 kB/s
     data_sent......................: 12 MB  365 kB/s
     http_req_blocked...............: avg=970.92µs min=911ns   med=2.12µs  max=46.59ms p(90)=3.31µs  p(95)=1.77ms  
     http_req_connecting............: avg=947.52µs min=0s      med=0s      max=41.28ms p(90)=0s      p(95)=1.22ms  
     http_req_duration..............: avg=1.58s    min=4.3ms   med=1.19s   max=11.86s  p(90)=3.86s   p(95)=4.86s   
       { expected_response:true }...: avg=1.58s    min=4.3ms   med=1.19s   max=11.86s  p(90)=3.86s   p(95)=4.86s   
     http_req_failed................: 0.00%  ✓ 0          ✗ 9681 
     http_req_receiving.............: avg=59.09µs  min=16.24µs med=36.12µs max=7.87ms  p(90)=54.73µs p(95)=66.52µs 
     http_req_sending...............: avg=96.47µs  min=6.65µs  med=12.5µs  max=37.08ms p(90)=25.2µs  p(95)=132.78µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.58s    min=4.25ms  med=1.19s   max=11.86s  p(90)=3.86s   p(95)=4.86s   
     http_reqs......................: 9681   307.45695/s
     iteration_duration.............: avg=1.58s    min=8.3ms   med=1.2s    max=11.86s  p(90)=3.86s   p(95)=4.86s   
     iterations.....................: 9661   306.821774/s
     vus............................: 228    min=228      max=500
     vus_max........................: 500    min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 17538      ✗ 0    
     data_received..................: 515 MB  16 MB/s
     data_sent......................: 7.0 MB  211 kB/s
     http_req_blocked...............: avg=3.57ms   min=1.54µs  med=3.89µs   max=1.84s  p(90)=13.22µs p(95)=24.08ms
     http_req_connecting............: avg=3.05ms   min=0s      med=0s       max=1.84s  p(90)=0s      p(95)=23.88ms
     http_req_duration..............: avg=1.27s    min=3.22ms  med=1.15s    max=5.15s  p(90)=2.36s   p(95)=3.15s  
       { expected_response:true }...: avg=1.27s    min=3.22ms  med=1.15s    max=5.15s  p(90)=2.36s   p(95)=3.15s  
     http_req_failed................: 0.00%   ✓ 0          ✗ 5866 
     http_req_receiving.............: avg=252.56ms min=34µs    med=90.42µs  max=4.36s  p(90)=1.14s   p(95)=1.81s  
     http_req_sending...............: avg=33.37ms  min=8.24µs  med=20.1µs   max=3.43s  p(90)=1.35ms  p(95)=14.34ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s     p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=993.48ms min=3.06ms  med=977.07ms max=4.71s  p(90)=1.69s   p(95)=2s     
     http_reqs......................: 5866    177.849414/s
     iteration_duration.............: avg=2.68s    min=20.37ms med=2.39s    max=11.66s p(90)=5.07s   p(95)=6.03s  
     iterations.....................: 5846    177.24304/s
     vus............................: 26      min=26       max=500
     vus_max........................: 500     min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `grafbase`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 15711      ✗ 0    
     data_received..................: 462 MB  15 MB/s
     data_sent......................: 6.2 MB  198 kB/s
     http_req_blocked...............: avg=3.66ms   min=1.42µs  med=3.34µs  max=2.2s   p(90)=29.81µs  p(95)=18.93ms
     http_req_connecting............: avg=3.06ms   min=0s      med=0s      max=1.67s  p(90)=0s       p(95)=18.77ms
     http_req_duration..............: avg=1.42s    min=3.11ms  med=1.26s   max=6.77s  p(90)=2.45s    p(95)=3.32s  
       { expected_response:true }...: avg=1.42s    min=3.11ms  med=1.26s   max=6.77s  p(90)=2.45s    p(95)=3.32s  
     http_req_failed................: 0.00%   ✓ 0          ✗ 5257 
     http_req_receiving.............: avg=240.45ms min=32.61µs med=74.03µs max=5.16s  p(90)=1.06s    p(95)=1.49s  
     http_req_sending...............: avg=25.83ms  min=7.9µs   med=15.27µs max=3.34s  p(90)=913.33µs p(95)=10.82ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s     p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=1.15s    min=3.03ms  med=1.08s   max=4.75s  p(90)=1.98s    p(95)=2.6s   
     http_reqs......................: 5257    166.725015/s
     iteration_duration.............: avg=2.91s    min=48.52ms med=2.64s   max=10.84s p(90)=5.32s    p(95)=6.26s  
     iterations.....................: 5237    166.090718/s
     vus............................: 315     min=315      max=500
     vus_max........................: 500     min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 14760      ✗ 0    
     data_received..................: 434 MB  14 MB/s
     data_sent......................: 5.9 MB  186 kB/s
     http_req_blocked...............: avg=3.84ms   min=1.52µs  med=3.66µs   max=3.96s  p(90)=284.98µs p(95)=22.79ms
     http_req_connecting............: avg=2.56ms   min=0s      med=0s       max=1.31s  p(90)=104.21µs p(95)=22.19ms
     http_req_duration..............: avg=1.69s    min=7.56ms  med=1.27s    max=12.94s p(90)=3.63s    p(95)=4.5s   
       { expected_response:true }...: avg=1.69s    min=7.56ms  med=1.27s    max=12.94s p(90)=3.63s    p(95)=4.5s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 4940 
     http_req_receiving.............: avg=652.93ms min=34.84µs med=103.63µs max=10.27s p(90)=2.33s    p(95)=3.35s  
     http_req_sending...............: avg=29.45ms  min=8.36µs  med=17.68µs  max=4.33s  p(90)=1.76ms   p(95)=12.6ms 
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=1.01s    min=7.39ms  med=937.07ms max=3.52s  p(90)=1.87s    p(95)=2.33s  
     http_reqs......................: 4940    156.399956/s
     iteration_duration.............: avg=3.1s     min=46.62ms med=2.6s     max=16.07s p(90)=6.12s    p(95)=7.24s  
     iterations.....................: 4920    155.766758/s
     vus............................: 268     min=268      max=500
     vus_max........................: 500     min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 8889      ✗ 0    
     data_received..................: 262 MB  7.5 MB/s
     data_sent......................: 3.5 MB  102 kB/s
     http_req_blocked...............: avg=2.72ms  min=1.87µs   med=3.86µs   max=51.31ms p(90)=15.11ms p(95)=23.9ms  
     http_req_connecting............: avg=2.64ms  min=0s       med=0s       max=33.97ms p(90)=14.49ms p(95)=23.69ms 
     http_req_duration..............: avg=5.33s   min=17.83ms  med=5.12s    max=11.43s  p(90)=7.52s   p(95)=8.7s    
       { expected_response:true }...: avg=5.33s   min=17.83ms  med=5.12s    max=11.43s  p(90)=7.52s   p(95)=8.7s    
     http_req_failed................: 0.00%   ✓ 0         ✗ 2983 
     http_req_receiving.............: avg=64.18ms min=36.94µs  med=114.39µs max=3.54s   p(90)=8.49ms  p(95)=323.94ms
     http_req_sending...............: avg=2.07ms  min=7.92µs   med=19.97µs  max=1.48s   p(90)=1.81ms  p(95)=3.36ms  
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s       max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=5.27s   min=17.59ms  med=5.1s     max=11.43s  p(90)=7.25s   p(95)=8.67s   
     http_reqs......................: 2983    85.565341/s
     iteration_duration.............: avg=5.49s   min=490.85ms med=5.19s    max=11.47s  p(90)=8.09s   p(95)=8.83s   
     iterations.....................: 2963    84.991654/s
     vus............................: 163     min=163     max=500
     vus_max........................: 500     min=500     max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 8259      ✗ 0    
     data_received..................: 244 MB  6.7 MB/s
     data_sent......................: 3.3 MB  91 kB/s
     http_req_blocked...............: avg=2.79ms   min=1.51µs   med=3.13µs   max=87.82ms p(90)=12.34ms  p(95)=25.22ms
     http_req_connecting............: avg=2.7ms    min=0s       med=0s       max=31ms    p(90)=11.95ms  p(95)=24.92ms
     http_req_duration..............: avg=5.89s    min=12.5ms   med=1.97s    max=35.75s  p(90)=32.28s   p(95)=34s    
       { expected_response:true }...: avg=5.89s    min=12.5ms   med=1.97s    max=35.75s  p(90)=32.28s   p(95)=34s    
     http_req_failed................: 0.00%   ✓ 0         ✗ 2773 
     http_req_receiving.............: avg=443.4µs  min=43.17µs  med=103.93µs max=57ms    p(90)=228.18µs p(95)=460µs  
     http_req_sending...............: avg=299.86µs min=8.35µs   med=15.98µs  max=34.89ms p(90)=735.09µs p(95)=1.58ms 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=5.89s    min=12.41ms  med=1.97s    max=35.75s  p(90)=32.28s   p(95)=34s    
     http_reqs......................: 2773    76.472964/s
     iteration_duration.............: avg=5.95s    min=460.91ms med=1.99s    max=35.75s  p(90)=32.34s   p(95)=34.03s 
     iterations.....................: 2753    75.92141/s
     vus............................: 23      min=23      max=500
     vus_max........................: 500     min=500     max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 7986      ✗ 0    
     data_received..................: 236 MB  6.5 MB/s
     data_sent......................: 3.2 MB  88 kB/s
     http_req_blocked...............: avg=3.92ms   min=2.04µs   med=4.88µs   max=62.23ms  p(90)=18.83ms  p(95)=28.4ms 
     http_req_connecting............: avg=3.83ms   min=0s       med=0s       max=62.2ms   p(90)=18.63ms  p(95)=28.09ms
     http_req_duration..............: avg=6.06s    min=14.66ms  med=2.61s    max=35.68s   p(90)=24.35s   p(95)=33.83s 
       { expected_response:true }...: avg=6.06s    min=14.66ms  med=2.61s    max=35.68s   p(90)=24.35s   p(95)=33.83s 
     http_req_failed................: 0.00%   ✓ 0         ✗ 2682 
     http_req_receiving.............: avg=807.28µs min=45.87µs  med=119.21µs max=168.48ms p(90)=718.98µs p(95)=2.66ms 
     http_req_sending...............: avg=441.65µs min=9µs      med=30.89µs  max=47.63ms  p(90)=855.26µs p(95)=2.45ms 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=6.06s    min=14.52ms  med=2.61s    max=35.68s   p(90)=24.35s   p(95)=33.83s 
     http_reqs......................: 2682    73.903541/s
     iteration_duration.............: avg=6.13s    min=504.85ms med=2.65s    max=35.69s   p(90)=24.89s   p(95)=33.86s 
     iterations.....................: 2662    73.352433/s
     vus............................: 29      min=29      max=500
     vus_max........................: 500     min=500     max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 6015      ✗ 0    
     data_received..................: 178 MB  5.3 MB/s
     data_sent......................: 2.4 MB  72 kB/s
     http_req_blocked...............: avg=10.17ms min=1.95µs   med=4.97µs   max=115.89ms p(90)=51.84ms  p(95)=59.3ms 
     http_req_connecting............: avg=9.93ms  min=0s       med=0s       max=105.37ms p(90)=51.24ms  p(95)=58.9ms 
     http_req_duration..............: avg=7.73s   min=14.4ms   med=8.58s    max=15.83s   p(90)=11.18s   p(95)=13.74s 
       { expected_response:true }...: avg=7.73s   min=14.4ms   med=8.58s    max=15.83s   p(90)=11.18s   p(95)=13.74s 
     http_req_failed................: 0.00%   ✓ 0         ✗ 2025 
     http_req_receiving.............: avg=8.01ms  min=44.61µs  med=121.46µs max=417ms    p(90)=573.11µs p(95)=10.2ms 
     http_req_sending...............: avg=3.05ms  min=8.92µs   med=31.24µs  max=76.22ms  p(90)=8.25ms   p(95)=24.41ms
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=7.72s   min=14.3ms   med=8.58s    max=15.83s   p(90)=11.17s   p(95)=13.74s 
     http_reqs......................: 2025    60.622433/s
     iteration_duration.............: avg=7.86s   min=596.51ms med=8.6s     max=15.84s   p(90)=11.27s   p(95)=13.77s 
     iterations.....................: 2005    60.023693/s
     vus............................: 131     min=131     max=500
     vus_max........................: 500     min=500     max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

Copy link

github-actions bot commented Nov 6, 2024

Overview for: federation/ramping-vus

This scenario runs 4 subgraphs and a GraphQL gateway with Federation spec, and runs a heavy query. We are running a heavy load of concurrent VUs to measure response time and other stats, during stress. It measure things like memory usage, CPU usage, response times. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was trying to reach 2000 concurrent VUs over 60s

Comparison

Comparison

Gateway duration(p95)⬇️ RPS Requests Durations Notes
cosmo 5332ms 178 12545 total, 0 failed avg: 2083ms, p95: 5332ms, max: 10445ms, med: 1622ms
apollo-router 6357ms 160 11315 total, 0 failed avg: 2621ms, p95: 6357ms, max: 15257ms, med: 2429ms
grafbase 7584ms 154 10842 total, 0 failed avg: 3166ms, p95: 7585ms, max: 18391ms, med: 2909ms
inigo 13377ms 290 20490 total, 0 failed avg: 3650ms, p95: 13378ms, max: 35776ms, med: 1727ms ❌ 20470 unexpected GraphQL errors, non-compatible response structure (20470)
hive-gateway-bun 27975ms 89 7346 total, 0 failed avg: 11868ms, p95: 27976ms, max: 35876ms, med: 10312ms
mercurius 39373ms 53 4816 total, 0 failed avg: 22069ms, p95: 39373ms, max: 40351ms, med: 23006ms
hive-gateway 54153ms 75 6891 total, 0 failed avg: 13876ms, p95: 54153ms, max: 59083ms, med: 3637ms
apollo-server 60000ms 76 7097 total, 562 failed avg: 13173ms, p95: 60000ms, max: 60129ms, med: 2349ms ❌ 562 failed requests, 562 non-200 responses, 562 unexpected GraphQL errors
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 37575      ✗ 0     
     data_received..................: 1.1 GB  16 MB/s
     data_sent......................: 15 MB   212 kB/s
     http_req_blocked...............: avg=201.69ms min=1.72µs  med=3.66µs  max=8.58s  p(90)=574.14ms p(95)=1.23s   
     http_req_connecting............: avg=185.44ms min=0s      med=0s      max=8.58s  p(90)=448.82ms p(95)=1.14s   
     http_req_duration..............: avg=2.08s    min=3.27ms  med=1.62s   max=10.44s p(90)=4.47s    p(95)=5.33s   
       { expected_response:true }...: avg=2.08s    min=3.27ms  med=1.62s   max=10.44s p(90)=4.47s    p(95)=5.33s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 12545 
     http_req_receiving.............: avg=416.63ms min=34.13µs med=84.81µs max=8s     p(90)=1.42s    p(95)=3.08s   
     http_req_sending...............: avg=101.19ms min=7.92µs  med=17.65µs max=5.8s   p(90)=160.79ms p(95)=653.34ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s     p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=1.56s    min=3.13ms  med=1.28s   max=8.68s  p(90)=3.89s    p(95)=4.36s   
     http_reqs......................: 12545   178.964825/s
     iteration_duration.............: avg=5.43s    min=8.84ms  med=4.56s   max=22.32s p(90)=11.53s   p(95)=13.18s  
     iterations.....................: 12525   178.679509/s
     vus............................: 4       min=4        max=1909
     vus_max........................: 2000    min=2000     max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 33885      ✗ 0     
     data_received..................: 993 MB  14 MB/s
     data_sent......................: 13 MB   190 kB/s
     http_req_blocked...............: avg=297.11ms min=1.77µs  med=4.15µs  max=8.48s  p(90)=929.6ms  p(95)=2.5s    
     http_req_connecting............: avg=295.11ms min=0s      med=0s      max=8.48s  p(90)=897.52ms p(95)=2.49s   
     http_req_duration..............: avg=2.62s    min=7.43ms  med=2.42s   max=15.25s p(90)=4.99s    p(95)=6.35s   
       { expected_response:true }...: avg=2.62s    min=7.43ms  med=2.42s   max=15.25s p(90)=4.99s    p(95)=6.35s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 11315 
     http_req_receiving.............: avg=532.9ms  min=29.71µs med=96.33µs max=10.76s p(90)=1.93s    p(95)=3.56s   
     http_req_sending...............: avg=126.53ms min=8.36µs  med=21.54µs max=7.05s  p(90)=182.46ms p(95)=766.22ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s     p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=1.96s    min=7.3ms   med=1.81s   max=8.96s  p(90)=4.05s    p(95)=4.72s   
     http_reqs......................: 11315   160.062342/s
     iteration_duration.............: avg=6.4s     min=43.27ms med=5.67s   max=26.41s p(90)=12.84s   p(95)=15.58s  
     iterations.....................: 11295   159.779422/s
     vus............................: 587     min=76       max=2000
     vus_max........................: 2000    min=2000     max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `grafbase`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 32466      ✗ 0     
     data_received..................: 953 MB  14 MB/s
     data_sent......................: 13 MB   184 kB/s
     http_req_blocked...............: avg=375.46ms min=1.58µs  med=4.88µs   max=12.16s p(90)=981.98ms p(95)=3.01s   
     http_req_connecting............: avg=369.6ms  min=0s      med=0s       max=12.16s p(90)=978.55ms p(95)=3.01s   
     http_req_duration..............: avg=3.16s    min=3.13ms  med=2.9s     max=18.39s p(90)=6.2s     p(95)=7.58s   
       { expected_response:true }...: avg=3.16s    min=3.13ms  med=2.9s     max=18.39s p(90)=6.2s     p(95)=7.58s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 10842 
     http_req_receiving.............: avg=539.74ms min=33.62µs med=103.56µs max=13.68s p(90)=1.79s    p(95)=2.66s   
     http_req_sending...............: avg=153.77ms min=8.22µs  med=25.95µs  max=9.11s  p(90)=251.57ms p(95)=858.83ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=2.47s    min=3.05ms  med=2.06s    max=11.33s p(90)=5.4s     p(95)=5.92s   
     http_reqs......................: 10842   154.685387/s
     iteration_duration.............: avg=6.75s    min=23.25ms med=6.05s    max=26.26s p(90)=13.43s   p(95)=15.4s   
     iterations.....................: 10822   154.400042/s
     vus............................: 50      min=50       max=1925
     vus_max........................: 2000    min=2000     max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `inigo`

K6 Output

     ✓ response code was 200
     ✗ no graphql errors
      ↳  0% — ✓ 0 / ✗ 20470
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 20470

     █ setup

     checks.........................: 33.33% ✓ 20470      ✗ 40940 
     data_received..................: 7.8 MB 111 kB/s
     data_sent......................: 24 MB  345 kB/s
     http_req_blocked...............: avg=58.32µs min=1.13µs  med=2.88µs  max=13.59ms p(90)=30.42µs p(95)=245.12µs
     http_req_connecting............: avg=45.41µs min=0s      med=0s      max=13.37ms p(90)=0s      p(95)=174.11µs
     http_req_duration..............: avg=3.64s   min=4.63ms  med=1.72s   max=35.77s  p(90)=10.26s  p(95)=13.37s  
       { expected_response:true }...: avg=3.64s   min=4.63ms  med=1.72s   max=35.77s  p(90)=10.26s  p(95)=13.37s  
     http_req_failed................: 0.00%  ✓ 0          ✗ 20490 
     http_req_receiving.............: avg=71µs    min=17.27µs med=44.82µs max=14.29ms p(90)=70.58µs p(95)=84.02µs 
     http_req_sending...............: avg=41.46µs min=7.92µs  med=15.15µs max=13.82ms p(90)=40.47µs p(95)=59.1µs  
     http_req_tls_handshaking.......: avg=0s      min=0s      med=0s      max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=3.64s   min=4.56ms  med=1.72s   max=35.77s  p(90)=10.26s  p(95)=13.37s  
     http_reqs......................: 20490  290.722624/s
     iteration_duration.............: avg=3.65s   min=8.59ms  med=1.73s   max=35.77s  p(90)=10.27s  p(95)=13.38s  
     iterations.....................: 20470  290.438853/s
     vus............................: 290    min=70       max=1991
     vus_max........................: 2000   min=2000     max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 21978     ✗ 0     
     data_received..................: 645 MB  7.9 MB/s
     data_sent......................: 8.7 MB  106 kB/s
     http_req_blocked...............: avg=18.32ms min=1.97µs  med=4.42µs   max=861.44ms p(90)=23.95ms p(95)=131.14ms
     http_req_connecting............: avg=17.63ms min=0s      med=0s       max=811.52ms p(90)=23.28ms p(95)=128.51ms
     http_req_duration..............: avg=11.86s  min=19.25ms med=10.31s   max=35.87s   p(90)=25.46s  p(95)=27.97s  
       { expected_response:true }...: avg=11.86s  min=19.25ms med=10.31s   max=35.87s   p(90)=25.46s  p(95)=27.97s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 7346  
     http_req_receiving.............: avg=99.16ms min=35.19µs med=112.09µs max=7.2s     p(90)=7.9ms   p(95)=448.26ms
     http_req_sending...............: avg=12.88ms min=8.94µs  med=26.83µs  max=1.94s    p(90)=15.53ms p(95)=53.4ms  
     http_req_tls_handshaking.......: avg=0s      min=0s      med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=11.75s  min=19.01ms med=10.13s   max=35.87s   p(90)=25.36s  p(95)=27.96s  
     http_reqs......................: 7346    89.555879/s
     iteration_duration.............: avg=12.1s   min=84.67ms med=10.44s   max=35.89s   p(90)=25.86s  p(95)=28.42s  
     iterations.....................: 7326    89.312057/s
     vus............................: 109     min=57      max=1999
     vus_max........................: 2000    min=2000    max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 14388     ✗ 0     
     data_received..................: 423 MB  4.7 MB/s
     data_sent......................: 5.7 MB  64 kB/s
     http_req_blocked...............: avg=208.3µs  min=1.67µs   med=4.06µs   max=24.46ms p(90)=422.56µs p(95)=657.33µs
     http_req_connecting............: avg=174.92µs min=0s       med=0s       max=24.36ms p(90)=352.87µs p(95)=514.94µs
     http_req_duration..............: avg=22.06s   min=14.24ms  med=23s      max=40.35s  p(90)=38.32s   p(95)=39.37s  
       { expected_response:true }...: avg=22.06s   min=14.24ms  med=23s      max=40.35s  p(90)=38.32s   p(95)=39.37s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 4816  
     http_req_receiving.............: avg=19.29ms  min=37.57µs  med=100.37µs max=2.59s   p(90)=343.98µs p(95)=1.01ms  
     http_req_sending...............: avg=50.97µs  min=9.34µs   med=25.85µs  max=15.36ms p(90)=63.46µs  p(95)=83.81µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=22.04s   min=14.13ms  med=23s      max=40.35s  p(90)=38.32s   p(95)=39.36s  
     http_reqs......................: 4816    53.496992/s
     iteration_duration.............: avg=22.25s   min=342.71ms med=23.12s   max=40.36s  p(90)=38.47s   p(95)=39.5s   
     iterations.....................: 4796    53.274828/s
     vus............................: 138     min=54      max=2000
     vus_max........................: 2000    min=2000    max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `hive-gateway`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     █ setup

     checks.........................: 100.00% ✓ 20609     ✗ 0     
     data_received..................: 605 MB  6.7 MB/s
     data_sent......................: 8.2 MB  91 kB/s
     http_req_blocked...............: avg=989.84µs min=2.03µs  med=5.31µs   max=172.31ms p(90)=572.7µs  p(95)=2.49ms  
     http_req_connecting............: avg=948.38µs min=0s      med=0s       max=172.24ms p(90)=485.29µs p(95)=2.26ms  
     http_req_duration..............: avg=13.87s   min=17.36ms med=3.63s    max=59.08s   p(90)=48.71s   p(95)=54.15s  
       { expected_response:true }...: avg=13.87s   min=17.36ms med=3.63s    max=59.08s   p(90)=48.71s   p(95)=54.15s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 6891  
     http_req_receiving.............: avg=1.33ms   min=39.72µs med=126.22µs max=384.04ms p(90)=987.72µs p(95)=2.68ms  
     http_req_sending...............: avg=444.19µs min=9.04µs  med=32.9µs   max=115.88ms p(90)=82.32µs  p(95)=330.13µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=13.87s   min=17.24ms med=3.63s    max=58.89s   p(90)=48.71s   p(95)=54.15s  
     http_reqs......................: 6891    75.999677/s
     iteration_duration.............: avg=13.93s   min=118ms   med=3.67s    max=59.1s    p(90)=48.78s   p(95)=54.22s  
     iterations.....................: 6867    75.734985/s
     vus............................: 111     min=56      max=1999
     vus_max........................: 2000    min=2000    max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✗ response code was 200
      ↳  92% — ✓ 6515 / ✗ 562
     ✗ no graphql errors
      ↳  92% — ✓ 6515 / ✗ 562
     ✓ valid response structure

     █ setup

     checks.........................: 94.56% ✓ 19545     ✗ 1124  
     data_received..................: 574 MB 6.2 MB/s
     data_sent......................: 8.5 MB 91 kB/s
     http_req_blocked...............: avg=432.48µs min=1.5µs   med=3.71µs   max=126.7ms  p(90)=351.89µs p(95)=548.23µs
     http_req_connecting............: avg=403.01µs min=0s      med=0s       max=126.64ms p(90)=284.12µs p(95)=435.39µs
     http_req_duration..............: avg=13.17s   min=12.41ms med=2.34s    max=1m0s     p(90)=57.55s   p(95)=1m0s    
       { expected_response:true }...: avg=9.14s    min=12.41ms med=2.26s    max=59.99s   p(90)=38.43s   p(95)=48.05s  
     http_req_failed................: 7.91%  ✓ 562       ✗ 6535  
     http_req_receiving.............: avg=313.21µs min=0s      med=110.85µs max=222.59ms p(90)=198.08µs p(95)=295.83µs
     http_req_sending...............: avg=170.22µs min=8.45µs  med=19.59µs  max=42.25ms  p(90)=60.18µs  p(95)=85.37µs 
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=13.17s   min=12.31ms med=2.34s    max=1m0s     p(90)=57.55s   p(95)=1m0s    
     http_reqs......................: 7097   76.461731/s
     iteration_duration.............: avg=13.22s   min=55.53ms med=2.37s    max=1m0s     p(90)=57.61s   p(95)=1m0s    
     iterations.....................: 7077   76.246255/s
     vus............................: 22     min=22      max=2000
     vus_max........................: 2000   min=2000    max=2000

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

@ardatan ardatan force-pushed the kamil-inigo branch 3 times, most recently from f9e83fd to 4b560f3 Compare March 20, 2025 16:16
@ardatan ardatan requested a review from dotansimha March 20, 2025 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants