protocol · rvagg · May 10, 2021 · Apr 7, 2021 · Apr 8, 2021 · Apr 8, 2021
diff --git a/home/olizilla/2021-04-exploration-report-lotus-openrpc.md b/home/olizilla/2021-04-exploration-report-lotus-openrpc.md
@@ -0,0 +1,306 @@
+# Lotus OpenRPC support (2021-04)
+
+> Exploration report by @olizilla
+
+Lotus provides OpenRPC compliant, discoverable, JSON based definition of it's v0 api, that _should_ allow API clients and documentation to be generated automatically. There are some roadblocks in the way.
+
+## Key findings
+
+- You can't directly use the Typescript API client generated by the recommended [open-rpc/generator](https://github.com/open-rpc/generator) tool, as Lotus uses a custom json-rpc implementation that does not yet support sending the request id field as a string. See: https://github.com/filecoin-project/go-jsonrpc/issues/28
+- There does not appear to be a way to pass auth credentails or headers to the generated client. The Transport implementations support it, but the client does not pass the options through. Requires a small PR to [open-rpc/generator] to fix.
+- The [open-rpc/generator](https://github.com/open-rpc/generator) is under-maintained. 
+  - The Typescript api client it generates for Lotus produces 500+ non-fatal errors when compiled with `[email protected]`
+  - The generated api docs fails to build with a gatsby webpack compilation error under npm@6 see: https://github.com/open-rpc/generator/issues/621
+  - under npm@7 the deps fail to install.
+- It's difficult to explore the Lotus api definition in the OpenRPC Playground, which should be part of the magic.
+  - if you don't have https set up for your lotus api, your browser prevents the playground from requesting the definition doc due to mixed content warnings. https://playground.open-rpc.org/?url=http://44.236.109.158:28000/rpc/v0
+  - you can manually switch to using http for the playground, but the requet still fails with a CORS error. http://playground.open-rpc.org/?url=http://44.236.109.158:28000/rpc/v0
+- The Lotus OpenRPC definitons need improving; we things like `name: p1, type: number, desctiption: number is a Number`
+- OpenRPC can't define the file import api as it is a REST endpoint that does not conform to json-rpc. See: https://docs.filecoin.io/reference/lotus-api/#features This means we can only ever use OpenRPC as part of a customised API client generation tool.
+
+## Recommendations
+
+- Improve the Lotus OpenRPC with meaningful definitions for all methods, parameters, and return types.
+- Fix issues where [filecoin-project/go-jsonrpc](https://github.com/filecoin-project/go-jsonrpc) does not comply with JSON-RPC 2.0 spec.
+- Use the Lotus OpenRPC definitions in place of custom go-code introspection to generate [js-lotus-client-schema](https://github.com/filecoin-shipyard/js-lotus-client-schema#updating-the-schemas)
+- Use GitHub actions to automatically publish updates to the `js-lotus-client`.
+- Upgrade `api.chain.love` to lotus >= 1.17 and ensure it's OpenRPC discovery method `rpc.discover` is publically readable without an api key, and publish links to the OpenRPC playground and inspector for it, to make the OpenRPC support easy to explore and share.
+- Contribute fixes to docs and Typescript generation back to https://github.com/open-rpc/generator
+
+## Inspecting the OpenRPC definiton
+
+You can fetch if via the standard `rpc.discver` method
+
+```console
+ curl -H "Content-Type: application/json" \
+  --data '{ "jsonrpc": "2.0", "method": "rpc.discover", "params": [], "id": 1 }' \
+  'http://44.236.109.158:28000/rpc/v0' | jq
+```
+
+<details>
+  <summary>Show response snippet</summary>
+
+```json
+{
+  "jsonrpc": "2.0",
+  "result": {
+    "info": {
+      "title": "Lotus RPC API",
+      "version": "1.6.0-dev"
+    },
+    "methods": [
+      {
+        "deprecated": false,
+        "description": "```go\nfunc (s *FullNodeStruct) BeaconGetEntry(p0 context.Context, p1 abi.ChainEpoch) (*types.BeaconEntry, error) {\n\treturn s.Internal.BeaconGetEntry(p0, p1)\n}\n```",
+        "externalDocs": {
+          "description": "Github remote link",
+          "url": "https://github.com/filecoin-project/lotus/blob/master/api/apistruct/struct.go#L885"
+        },
+        "name": "Filecoin.BeaconGetEntry",
+        "paramStructure": "by-position",
+        "params": [
+          {
+            "deprecated": false,
+            "description": "abi.ChainEpoch",
+            "name": "p1",
+            "required": true,
+            "schema": {
+              "description": "Number is a number",
+              "examples": [
+                10101
+              ],
+              "title": "number",
+              "type": [
+                "number"
+              ]
+            },
+            "summary": ""
+          }
+        ],
+        "result": {
+          "deprecated": false,
+          "description": "*types.BeaconEntry",
+          "name": "*types.BeaconEntry",
+          "required": true,
+          "schema": {
+            "additionalProperties": false,
+            "examples": [
+              {
+                "Data": "Ynl0ZSBhcnJheQ==",
+                "Round": 42
+              }
+            ],
+            "properties": {
+              "Data": {
+                "media": {
+                  "binaryEncoding": "base64"
+                },
+                "type": "string"
+              },
+              "Round": {
+                "title": "number",
+                "type": "number"
+              }
+            },
+            "type": [
+              "object"
+            ]
+          },
+          "summary": ""
+        },
+        "summary": "BeaconGetEntry returns the beacon entry for the given filecoin epoch. If\nthe entry has not yet been produced, the call will block until the entry\nbecomes available\n"
+      },
+```
+
+</details>
+
+
+A copy of the uncompressed and prettfied openrpc definitons for lotus can be found in this gist: https://gist.github.com/olizilla/fc6abf836023b6ee662955695f1cfd21
+
+You can preview the generated OpenRPC docs for the Lotus API by passing the files in that gist to the OpenRPC playground. See here for the docs for the Lotus full node api: https://playground.open-rpc.org/?url=https://gist.githubusercontent.com/olizilla/fc6abf836023b6ee662955695f1cfd21/raw/a4cb5aa3c6fbbc47bcf4b92d2370b8b012a810cb/full.json
+
+## Generate an API client
+
+Get the openrc definitions from the lotus repo, ungzip them, and pass them to the [open-rpc/generator].
+
+```shell
+# ungzip the lotus full node openrpc definition
+curl "https://raw.githubusercontent.com/filecoin-project/lotus/f4a2c8fa49dfc456090f5a5ea1b4ae62d33b4f85/build/openrpc/full.json.gz" | zcat > full.json 
+
+# create the typescript rpc client
+npx @open-rpc/generator generate -t client -l typescript -n "lotus-openrpc-client" -d full.json 
+```
+
+A copy of the generated API client is available at: https://github.com/olizilla/lotus-openrpc-client
+
+You can [open-rpc/generator] to generate a rust client, but this is untested.
+
+## Use the API client
+
+Make a client instances and ask it for the `ChainHead`
+
+```js
+const LotusRPCAPI = require('../client/typescript/build/index.js').LotusRPCAPI
+
+const lotus = new LotusRPCAPI({ 
+  transport: {
+    type: 'http',
+    host: '44.236.109.158',
+    port: '28000',
+    path: '/rpc/v0'
+  }
+})
+
+async function main () {
+  console.log('Fetching ChainHead')
+  try {
+    const head = await lotus.ChainHead()
+    console.log('ChainHead', head)
+  } catch (err) {
+    console.error(err)
+    process.exit(-1)
+  }
+}
+
+main()
+```
+
+Running it fails with an error as the client sends the `id` parameter as a String (allowed per the JSON-RPC 2.0 spec) but unsupported at present by [filecoin-project/go-jsonrpc](https://github.com/filecoin-project/go-jsonrpc/issues/28)
+
+```console
+$ node index.js
+Fetching ChainHead
+JSONRPCError: unmarshaling request: json: cannot unmarshal string into Go struct field request.id of type int64
+    at new JSONRPCError (/Users/oli/Code/olizilla/lotus-openrpc-client/client/typescript/node_modules/@open-rpc/client-js/build/Error.js:24:28)
+```
+
+## Generate API Docs
+
+```console
+# ungzip the lotus full node openrpc definition
+curl "https://raw.githubusercontent.com/filecoin-project/lotus/f4a2c8fa49dfc456090f5a5ea1b4ae62d33b4f85/build/openrpc/full.json.gz" | zcat > full.json 
+
+# create the api docs
+npx @open-rpc/generator generate -t docs -l gatsby -n "lotus-openrpc-client" -d full.json 
+```
+
+At time of writing the generated docs do not build locally, but deps fail to install under npm@7 and the gatsby project fails to build under npm@6
+
+<details>
+
+<summary>show build output</summary>
+
+```console
+$ nave use 14
+$ npm -v
+6.14.12
+$ npm start
+$ npm start
+
+> [email protected] start /Users/oli/Code/tmp/lotus/docs/gatsby
+> npm run develop
+
+
+> [email protected] develop /Users/oli/Code/tmp/lotus/docs/gatsby
+> gatsby develop
+
+success open and validate gatsby-configs - 0.127s
+success load plugins - 2.064s
+success onPreInit - 0.047s
+success initialize cache - 0.009s
+success copy gatsby files - 0.135s
+success onPreBootstrap - 0.027s
+success createSchemaCustomization - 0.007s
+success Checking for changed pages - 0.001s
+success source and transform nodes - 0.283s
+success building schema - 0.385s
+info Total nodes: 52, SitePage nodes: 2 (use --verbose for breakdown)
+success createPages - 0.042s
+success Checking for changed pages - 0.001s
+success createPagesStatefully - 0.082s
+success update schema - 0.028s
+success write out redirect data - 0.002s
+success Build manifest and related icons - 0.123s
+success onPostBootstrap - 0.135s
+info bootstrap finished - 6.660s
+success onPreExtractQueries - 0.001s
+success extract queries from components - 0.605s
+success write out requires - 0.005s
+success run static queries - 0.065s - 5/5 76.39/s
+success run page queries - 0.012s - 3/3 256.93/s
+⠸ Building development bundle
+TypeError: (0 , _createSvgIcon.default) is not a function
+    at Object../node_modules/@xops.net/gatsby-openrpc-theme/node_modules/@material-ui/icons/Brightness3.js (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:42051:43)
+    at __webpack_require__ (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:36:30)
+    at Module../node_modules/@xops.net/gatsby-openrpc-theme/src/layouts/index.tsx (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:42289:88)
+    at __webpack_require__ (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:36:30)
+    at Object../node_modules/gatsby-plugin-layout/wrap-page.js (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:83637:26)
+    at __webpack_require__ (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:36:30)
+    at Object../node_modules/gatsby-plugin-layout/gatsby-ssr.js (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:83614:27)
+    at __webpack_require__ (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:36:30)
+    at Object../.cache/api-runner-ssr.js (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:163:11)
+    at __webpack_require__ (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:36:30)
+    at Module../.cache/develop-static-entry.js (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:559:73)
+    at __webpack_require__ (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:36:30)
+    at /Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:127:18
+    at /Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:130:10
+    at webpackUniversalModuleDefinition (/Users/oli/Code/tmp/lotus/docs/gatsby/public/render-page.js:3:20)
+
+ ERROR 
+
+There was an error compiling the html.js component for the development server.
+See our docs page on debugging HTML builds for help https://gatsby.dev/debug-html TypeError: (0 , _createSvgIcon.default) is not a function
+
+
+  14 | var _createSvgIcon = _interopRequireDefault(require("./utils/createSvgIcon"));
+  15 |
+> 16 | var _default = (0, _createSvgIcon.default)( /*#__PURE__*/React.createElement("path", {
+     |                ^
+  17 |   d: "M9 2c-1.05 0-2.05.16-3 .46 4.06 1.27 7 5.06 7 9.54 0 4.48-2.94 8.27-7 9.54.95.3 1.95.46 3 .46 5.52 0 10-4.48 10-10S14.52 2 9 2z"
+  18 | }), 'Brightness3');
+  19 |
+
+
+  WebpackError: TypeError: (0 , _createSvgIcon.default) is not a function
+
+  - Brightness3.js:16 
+    [gatsby-openrpc-theme]/[@material-ui]/icons/Brightness3.js:16:16
+
+  - wrap-page.js:12 
+    node_modules/gatsby-plugin-layout/wrap-page.js:12:26
+
+  - gatsby-ssr.js:3 
+    node_modules/gatsby-plugin-layout/gatsby-ssr.js:3:27
+
+
+not finished Building development bundle - 10.991s
+
+npm ERR! code ELIFECYCLE
+npm ERR! errno 1
+npm ERR! [email protected] develop: `gatsby develop`
+npm ERR! Exit status 1
+npm ERR! 
+npm ERR! Failed at the [email protected] develop script.
+npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
+```
+
+</details>
+
+
+## Background
+
+[OpenRPC] is a spec for defining / documenting your [JSON-RPC] compatibale api. With an OpenRPC definitinon for you API, you can automatically generate API clients and documentation with [open-rpc/generator]
+
+The Lotus Filecoin implementation landed OpenRPC definintions for it's API in https://github.com/filecoin-project/lotus/pull/5843
+
+The rationale for OpenRPC (as opposed to the widely deployed OpenAPI) is given here 
+> This project [openrpc] is a fork of openapi. It was modified to accommodate JSON-RPC APIs.
+>
+> While you could get some things to work, using openapi on anything but path & http based api is fitting a round peg to a square hole.
+>  https://github.com/open-rpc/spec/issues/112
+
+Lotus uses a custom go json-rpc client https://github.com/filecoin-project/go-jsonrpc
+
+[OpenRPC]: https://open-rpc.org/ "defines a standard, programming language-agnostic interface description for JSON-RPC 2.0 APIs."
+[JSON-RPC]: https://www.jsonrpc.org/specification "a stateless, light-weight remote procedure call (RPC) protocol."
+[open-rpc/generator]: https://github.com/open-rpc/generator "Multi-Component & Multi-Language Generators for OpenRPC"