Block tracking & Auto Pagination transform (graphprotocol#37)

Co-authored-by: Dotan Simha <[email protected]>

Author: ardatan

.changeset/orange-toys-hope copy.md

@@ -0,0 +1,26 @@
+---
+'@graphprotocol/client-auto-pagination': patch
+---
+
+#### Auto Pagination Transform
+
+`graph-client` implements automatic pagination using `first:` and `after:` filters of `graph-node`.
+
+At the moment, `graph-node` allow fetching only 1000 records per query. This transfomer allow you to run queries with any limit, and the breaks it automatically to multiple concurrent requests, then merges the responses into a single response.
+
+This feature is implemented in `@graphprotocol/client-auto-pagination` and installed automatically with the `graph-client` CLI package.
+
+### Usage Example
+
+```yaml
+# .graphclientrc.yml
+sources:
+  - name: uniswap
+    handler:
+      graphql:
+        endpoint: https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2
+    transforms:
+      - autoPagination:
+          validateSchema: true # Validates that the schema source actually contains the required input filters.
+          limitOfRecords: 1000 # Default is 1000, you can change if you indexer has different configuration in GRAPH_GRAPHQL_MAX_FIRST var.
+```

.changeset/orange-toys-hope.md

@@ -0,0 +1,23 @@
+---
+'@graphprotocol/client-block-tracking': patch
+---
+
+#### Block Tracking Transform
+
+`graph-client` implements automatic block tracking using `number_gte` filter of `graph-node`. This automates the process [of fetching and tracking the block number of entites](https://thegraph.com/docs/en/developer/distributed-systems/#polling-for-updated-data).
+
+This feature is implemented in `@graphprotocol/client-block-tracking` and installed automatically with the `graph-client` CLI package.
+
+### Usage Example
+
+```yaml
+# .graphclientrc.yml
+sources:
+  - name: uniswap
+    handler:
+      graphql:
+        endpoint: https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2
+    transforms: # The following section will make sure to automatically fetch the block information, and then use it for tracking in future queries.
+      - blockTracking:
+          validateSchema: true # Validates that the schema source actually contains _meta and input block filters.
+```

.vscode/settings.json

@@ -13,6 +13,6 @@
   "files.exclude": {
     "**/.git": true,
     "**/.DS_Store": true,
-    "**/node_modules": true
+    "**/node_modules": false
   }
 }

README.md

@@ -14,17 +14,19 @@ This library is intended to simplify the network aspect of data consumption for
 
 > The tools provided in this repo can be used as standalone, but you can also use it with any existing GraphQL Client!
 
-| Status | Feature                                | Notes                                                   |
-| ------ | -------------------------------------- | ------------------------------------------------------- |
-| ✅     | Multiple indexers                      | based on fetch strategies                               |
-| ✅     | Fetch Strategies                       | timeout, retry, fallback, race                          |
-| ✅     | Build time validations & optimizations |                                                         |
-| ✅     | Client-Side Composition                | with improved execution planner (based on GraphQL-Mesh) |
-| ✅     | Raw Execution (standalone mode)        | without a wrapping GraphQL client                       |
-| ✅     | Local (client-side) Mutations          |                                                         |
-| ✅     | Integration with `@apollo/client`      |                                                         |
-| ✅     | Integration with `urql`                |                                                         |
-| ✅     | TypeScript support                     | with built-in GraphQL Codegen and `TypedDocumentNode`   |
+| Status | Feature                                                         | Notes                                                                                                                            |
+| ------ | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| ✅     | Multiple indexers                                               | based on fetch strategies                                                                                                        |
+| ✅     | Fetch Strategies                                                | timeout, retry, fallback, race                                                                                                   |
+| ✅     | Build time validations & optimizations                          |                                                                                                                                  |
+| ✅     | Client-Side Composition                                         | with improved execution planner (based on GraphQL-Mesh)                                                                          |
+| ✅     | Raw Execution (standalone mode)                                 | without a wrapping GraphQL client                                                                                                |
+| ✅     | Local (client-side) Mutations                                   |                                                                                                                                  |
+| ✅     | [Automatic Block Tracking](./packages/block-tracking/README.md) | tracking block numbers [as described here](https://thegraph.com/docs/en/developer/distributed-systems/#polling-for-updated-data) |
+| ✅     | [Automatic Pagination](./packages/auto-pagination/README.md)    | doing multiple requests in a single call to fetch more than the indexer limit                                                    |
+| ✅     | Integration with `@apollo/client`                               |                                                                                                                                  |
+| ✅     | Integration with `urql`                                         |                                                                                                                                  |
+| ✅     | TypeScript support                                              | with built-in GraphQL Codegen and `TypedDocumentNode`                                                                            |
 
 > You can find an [extended architecture design here](./docs/architecture.md)
 
@@ -170,7 +172,7 @@ client.execute(myQuery, myVariables, {
 
 > You can find the [complete documentation for the `graphql` handler here](https://www.graphql-mesh.com/docs/handlers/graphql#config-api-reference).
 
-#### Environment Variables Inteporlation
+#### Environment Variables Interpolation
 
 If you wish to use environment variables in your The Graph Client configuration file, you can use interpolation with `env` helper:
 
@@ -282,6 +284,84 @@ sources:
 
 </details>
 
+#### Block Tracking
+
+The Graph Client can track block numbers and do the following queries by following [this pattern](https://thegraph.com/docs/en/developer/distributed-systems/#polling-for-updated-data) with `blockTracking` transform;
+
+```yaml
+sources:
+  - name: uniswapv2
+    handler:
+      graphql:
+        endpoint: https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2
+    transforms:
+      - blockTracking:
+          # You might want to disable schema validation for faster startup
+          validateSchema: true
+          # Ignore the fields that you don't want to be tracked
+          ignoreFieldNames: [users, prices]
+          # Exclude the operation with the following names
+          ignoreOperationNames: [NotFollowed]
+```
+
+[You can try a working example here](./examples/transforms)
+
+#### Automatic Pagination
+
+With most subgraphs, the number of records you can fetch is limited. In this case, you have to send multiple requests with pagination.
+
+```graphql
+query {
+  # Will throw an error if the limit is 1000
+  users(first: 2000) {
+    id
+    name
+  }
+}
+```
+
+So you have to send the following operations one after the other:
+
+```graphql
+query {
+  # Will throw an error if the limit is 1000
+  users(first: 1000) {
+    id
+    name
+  }
+}
+```
+
+Then after the first response;
+
+```graphql
+query {
+  # Will throw an error if the limit is 1000
+  users(first: 1000, skip: 1000) {
+    id
+    name
+  }
+}
+```
+
+After the second response, you have to merge the results manually. But instead The Graph Client allows you to do the first one and automatically does those multiple requests for you under the hood.
+
+All you have to do is;
+
+```yaml
+sources:
+  - name: uniswapv2
+    handler:
+      graphql:
+        endpoint: https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2
+    transforms:
+      - autoPagination:
+          # You might want to disable schema validation for faster startup
+          validateSchema: true
+```
+
+[You can try a working example here](./examples/transforms)
+
 #### Client-side Composition
 
 The Graph Client has built-in support for client-side GraphQL Composition (powered by [GraphQL-Tools Schema-Stitching](https://www.graphql-tools.com/docs/schema-stitching/stitch-combining-schemas)).

examples/transforms/.gitignore

@@ -0,0 +1 @@
+.graphclient

examples/transforms/.graphclientrc.yml

@@ -0,0 +1,11 @@
+sources:
+  - name: uniswap
+    handler:
+      graphql:
+        endpoint: https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2
+    transforms:
+      # Enable Automatic Block Tracking
+      - autoPagination:
+          validateSchema: true
+      - blockTracking:
+          validateSchema: true

examples/transforms/package.json

@@ -0,0 +1,12 @@
+{
+  "name": "transforms-example",
+  "private": true,
+  "version": "0.0.0",
+  "scripts": {
+    "start": "graphclient serve-dev"
+  },
+  "dependencies": {
+    "@graphprotocol/client-cli": "0.0.3",
+    "graphql": "16.3.0"
+  }
+}

jest.config.js

@@ -1,18 +1,29 @@
 const { resolve } = require('path')
+const { pathsToModuleNameMapper } = require('ts-jest')
 const CI = !!process.env.CI
 
 const ROOT_DIR = __dirname
 const TSCONFIG = resolve(ROOT_DIR, 'tsconfig.json')
 const tsconfig = require(TSCONFIG)
 
+process.env.LC_ALL = 'en_US'
+
 module.exports = {
   testEnvironment: 'node',
   rootDir: ROOT_DIR,
   restoreMocks: true,
   reporters: ['default'],
   modulePathIgnorePatterns: ['dist'],
-  moduleNameMapper: {},
+  moduleNameMapper: pathsToModuleNameMapper(tsconfig.compilerOptions.paths, { prefix: `${ROOT_DIR}/` }),
   collectCoverage: true,
   cacheDirectory: resolve(ROOT_DIR, `${CI ? '' : 'node_modules/'}.cache/jest`),
-  testMatch: ['**/?(*.)+(spec|test).[jt]s?(x)'],
+  extensionsToTreatAsEsm: ['.ts'],
+  coverageThreshold: {
+    global: {
+      branches: 80,
+      functions: 95,
+      lines: 90,
+      statements: 90,
+    },
+  },
 }

package.json

@@ -8,7 +8,7 @@
     "prebuild": "rimraf packages/*/dist",
     "check": "yarn workspaces run check",
     "build": "tsc --project tsconfig.build.json && bob build",
-    "test": "jest --passWithNoTests",
+    "test": "jest --passWithNoTests --detectLeaks --detectOpenHandles",
     "release": "yarn build && changeset publish",
     "release:canary": "(node scripts/canary-release.js && yarn build && yarn changeset publish --tag canary) || echo Skipping Canary...",
     "fix-bin": "node scripts/fix-bin.js",
@@ -26,7 +26,8 @@
     "tools"
   ],
   "contributors": [
-    "Dotan Simha <[email protected]>"
+    "Dotan Simha <[email protected]>",
+    "Arda Tanrikulu <[email protected]>"
   ],
   "license": "MIT",
   "bugs": {
@@ -58,7 +59,9 @@
     "prettier": "2.6.2",
     "pretty-quick": "3.1.3",
     "rimraf": "3.0.2",
-    "typescript": "4.6.3"
+    "typescript": "4.6.3",
+    "weak-napi": "2.0.2",
+    "ts-jest": "27.1.4"
   },
   "resolutions": {
     "@changesets/apply-release-plan": "6.0.0",

packages/auto-pagination/README.md

@@ -0,0 +1,22 @@
+## Automatic Unlimited Pagination
+
+`graph-client` implements automatic pagination using `first:` and `after:` filters of `graph-node`.
+
+At the moment, `graph-node` allow fetching only 1000 records per query. This transfomer allow you to run queries with any limit, and the breaks it automatically to multiple concurrent requests, then merges the responses into a single response.
+
+This feature is implemented in `@graphprotocol/client-auto-pagination` and installed automatically with the `graph-client` CLI package.
+
+### Usage Example
+
+```yaml
+# .graphclientrc.yml
+sources:
+  - name: uniswap
+    handler:
+      graphql:
+        endpoint: https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2
+    transforms:
+      - autoPagination:
+          validateSchema: true # Validates that the schema source actually contains the required input filters.
+          limitOfRecords: 1000 # Default is 1000, you can change if you indexer has different configuration in GRAPH_GRAPHQL_MAX_FIRST var.
+```

packages/auto-pagination/__tests__/auto-pagination.test.ts

@@ -0,0 +1,78 @@
+import { makeExecutableSchema } from '@graphql-tools/schema'
+import { wrapSchema } from '@graphql-tools/wrap'
+import { execute, graphql, parse } from 'graphql'
+import AutoPaginationTransform from '../src'
+
+describe('Auto Pagination', () => {
+  const users = new Array(20).fill({}).map((_, i) => ({ id: (i + 1).toString(), name: `User ${i + 1}` }))
+  const LIMIT = 3
+  const schema = makeExecutableSchema({
+    typeDefs: /* GraphQL */ `
+            type Query {
+                users(first: Int = ${LIMIT}, skip: Int = 0): [User!]!
+            }
+            type User {
+                id: ID!
+                name: String!
+            }
+        `,
+    resolvers: {
+      Query: {
+        users: (_, { first = LIMIT, skip = 0 }) => {
+          if (first > LIMIT) {
+            throw new Error(`You cannot request more than ${LIMIT} users; you requested ${first}`)
+          }
+          return users.slice(skip, skip + first)
+        },
+      },
+    },
+  })
+  const wrappedSchema = wrapSchema({
+    schema,
+    transforms: [
+      new AutoPaginationTransform({
+        config: {
+          limitOfRecords: LIMIT,
+        },
+      }),
+    ],
+  })
+  it('should give correct numbers of results if first arg are higher than given limit', async () => {
+    const query = /* GraphQL */ `
+      query {
+        users(first: 10) {
+          id
+          name
+        }
+      }
+    `
+    const result = await execute({
+      schema: wrappedSchema,
+      document: parse(query),
+    })
+    expect(result).toEqual({
+      data: {
+        users: users.slice(0, 10),
+      },
+    })
+  })
+  it('should respect skip argument', async () => {
+    const query = /* GraphQL */ `
+      query {
+        users(first: 10, skip: 1) {
+          id
+          name
+        }
+      }
+    `
+    const result = await execute({
+      schema: wrappedSchema,
+      document: parse(query),
+    })
+    expect(result).toEqual({
+      data: {
+        users: users.slice(1, 10),
+      },
+    })
+  })
+})