Proposal to add E2E and Integration testing on 1.x branch #3098
Description
(This proposal is continuation of #2682, a companion to #3070, and a tracking issue for all integration testing related topics for the 1.x branch)
E2E
Web3 1.x's stability might benefit from following ethereum/solidity's example and E2E testing its latest state against large projects that already use the package. Solidity does this nightly in CI - they build a new solc, swap it into Zeppelin, Gnosis and Colony and run those projects' own tests using the new version.
Some targets that seem promising for this purpose here are:
Technical
Web3 is a Lerna monorepo and so are two of the potential targets listed above. Because its repo structure and published formats are quite different it won't just npm install from a github url.
However, there's a nice tool called verdaccio that lets you set up a local private npm registry in CI. Web3 could "publish" a candidate branch to this ephemeral registry, clone an E2E target (ex: embark) and install the new web3 via npm proxy. Then run the target's tests.
It's also possible to chain publications together. For example, openzeppelin-solidity uses truffle so you could:
- publish a new Web3,
- install it in Truffle,
- publish a new Truffle,
- swap the new Truffle into Zeppelin and run Zeppelin's tests.
Additionally:
- could be used to verify that built assets are located correctly and usable.
- ethereum-js may migrate some things to monorepo in future. Any solution developed here might pay dividends there as well.
Integration
Clients
To recap #2682 - Web3 1.x's unit tests rely primarily on mocked server responses. They're super fast but some things are hard to validate without testing against a node. There's also variance in behavior between the three most widely used clients. It would be good to create a small suite of tests for stuff like:
- simple send and call
- events
- subscriptions
- errors
- transaction signing
..and make sure everything executes as expected across this matrix:
| Client | instamine | automine 2s |
|---|---|---|
| geth --dev | x | x |
| parity --dev | x | x |
| ganache-cli | x |
Client Helpers
To simplify things in CI and make this work portable, Web3 could also publish small Node cli wrappers for the dockerized Parity and Geth dev clients.
These would:
- pull down any client version from docker hub
- abstract away most of command config flags,
- open all APIs,
- have many well-funded accounts
- make it easy for people to generate reproduction cases for client specific problems.
Something that launches like this:
$ web3-geth-dev --period 2 --tag latest
See also: #2688 and #2693: I think access to these clients can be provided in a flexible way with JS wrappers alone - e.g without building any containers or using docker compose.
- nomiclabs/builder-docker provides a nice API for the general case of managing docker published resources from JS.
- A basic geth dev wrapper I've used in CI for other projects. Not perfect at all but works.
- Parity dev Node cli wrapper
- Geth dev Node cli wrapper
Network Emulators
- Dropped connections are frequently mentioned in the issues. Emulate with comcast?
Installation
- Windows: see Can't install web3 version 1.2 on Windows 10 #2971, npm install succeeds on latest version of Node on Windows but does show errors #3033
- Mac OSX
The EF has an account with CircleCI. (EthereumJS uses it). They let you specify which OS a job is run with pretty easily.
cf: #3034 (Azure Pipelines)
Other stuff 🐘
There is definitely other stuff to test. Would be grateful for any and all community suggestions about this topic.