Web3.js - 1.3.0

[nivida]

Description

Because we have soon finished all of the stabilization improvements listed here are we able to plan the tasks until 1.3.0.

Improvements

• Number overflows (will be done later with 2.0)
  • PR: Integer overflow fixed in formatters #3234
  • Issue: Formatter has a large number of overflow bugs #3227 & block timestamp - "Error: Number can only safely store up to 53 bits" #1905
  • This possibility can get provided with a configuration option
• Insecure credential storage (not possible in a non-breaking manner)
  • Issue: Feature Discussion - Let's Upgrade Security On the Wallet #2739
  • This will only be implemented if possible in a non-breaking manner
• Enforce 0x prefix for any given hex value
  • PR: Extend _txInputFormatter with hex prefix check #3317
  • Issue: web3.eth.accounts.privateKeyToAccount accepts private key without "0x" prefix but returns incorrect result #3041 & Request: ensure '0x' prefixing for all data fields #1082
• Transaction revert handling
  • PR: Adds revert instruction handling #3248
  • Issues: Incorrect return message for require() in public/external pure/view-type functions #1903 Support reason string  #1707 Unstructured errors returned from reverted transactions #2197 Implement new error codes and error reason passing/throwing, when implemented in nodes #176
• New build pipeline and bundles (we kept the current pipeline and improved it)
  • Issues: web3.min.js v1.2.2: Uncaught ReferenceError: regeneratorRuntime is not defined #3155 CDN broken #3074 Why is web3.min.js not provided anymore? #2623 Compiled web3 bundle is HUGE #1178
• WebsocketProvider reconnecting
  • PR: Provider related improvements #3190
  • Issues: MaxListenersExceededWarning when sending contract (v1.0.0-beta.33) #1648 Possible closure leak in web3.eth.Contract 1.x ? #3042 [v1.2.1] Connection not Open. #3092 Provider does not reconnect after failure #1085 Question: Keepalive for ws subscriptions? #1391 Reconnect on Websocket Drop #1558 Reconnect feature is missing when websocket connection is closed. #1852 issue with websocket connection to infura #1646 WebsocketProvider: Incorrect response when duplicate IDs #1510 Can't assign more than one handler to connection events #1933
  • Improves the listener handling at all and adds the possibility to activate auto-reconnect
• Allow custom registry for ENS
  • PR: Adds the possibility to define a custom registry for the ENS module #3301
  • Issue: ENS: option to use custom registry addresses #3010

Feel free to give your feedback about the lists above @alcuadrado @cgewecke @gnidan @michaelsbradleyjr @sohkai @caleteeter

[nivida]

As addition:
The correct way to release those breaking changes would be to increment the major version. But we sadly have to do those breaking changes within a minor release because 2.0.0-alpha.x with a complete refactored architecture is already released.

[michaelsbradleyjr]

I'm excited for the improvements. The breaking changes situation is pretty awkward.

Users who have e.g. "web3": "^1.2.1" would get 1.3.0 on a fresh install and may be frustrated by the breaking changes.

On the other hand, releasing as 3.0.0 could be confusing for a lot of people. But semver.satisfies('3.0.0', '^1.2.1 || ^2.0.0-alpha') // => false, which is a good thing.

Another option would be to move publishing of the 2.x architecture into a scoped package, e.g. @web3-next/web3. Instead of 1.3.0 you could then release as 3.0.0 or 2.0.0 or 2.1.0, with 3.0.0 probably being the cleanest option owing to semver behavior : semver.satisfies('2.1.0', '^2.0.0-alpha') // => true and semver.satisfies('2.0.0', '^2.0.0-alpha') // => true. When the new architecture is ready to become the stable release, it could be published as the unscoped package as 4.0.0 or whatever major version bump is applicable at that time.

All these approaches have downsides. I'm not sure what's the best way.

[alcuadrado]

I think this way of planning releases is wrong. Project goals and major tasks should be defined, and the releases should be derived from there, not the other way round.

For example, fixing the loss of numeric precision in some operations/data structures is a major communication and coordination task that can't be scoped to a single release.

[lautarodragan]

I think this way of planning releases is wrong. Project goals and major tasks should be defined, and the releases should be derived from there, not the other way round.

For example, fixing the loss of numeric precision in some operations/data structures is a major communication and coordination task that can't be scoped to a single release.

My 2 cents: using semantic-release and actually following semver really helps with this. At Po.et we have followed this approach, plus only allowing squash-merging, so an entire PR becomes a single commit and gets released depending on the commit's description, and it has worked wonders.

semantic-release automates the whole package release workflow including: determining the next version number, generating the release notes and publishing the package.

This removes the immediate connection between human emotions and version numbers, strictly following the Semantic Versioning specification.

Personally I'd love for this project to follow the same approach after 1.x branch is finished and the first 2.x is done.

[nivida]

@alcuadrado

I think this way of planning releases is wrong.

It would be great to have sometimes more constructive inputs from your side.

Project goals and major tasks should be defined

Many of those issues are listed in the stabilization issue but we couldn't release them because they would break stuff. This means there was first the goal and the major tasks defined and if I'm not wrong did you wrote those issues down.

I've created this issue to inform @gnidan etc. about the next steps and to get feedback because there is mostly no other way to reach out to the biggest consumers on another communication channel in the same constructive way.

For example, fixing the loss of numeric precision in some operations/data structures is a major communication and coordination task that can't be scoped to a single release.

Does this mean you would break one method after the other?

@lautarodragan Thanks for your feedback!

[nivida]

@michaelsbradleyjr

I'm excited for the improvements. The breaking changes situation is pretty awkward.

Oh yes, it is.

On the other hand, releasing as 3.0.0 could be confusing for a lot of people.

Yes, this will definitely be confusing especially after the confusion we already had with releasing beta.37 as 1.0 and beta.55 as 2.0-alpha.

Another option would be to move publishing of the 2.x architecture into a scoped package, e.g. @web3-next/web3. Instead of 1.3.0 you could then release as 3.0.0 or 2.0.0 or 2.1.0, with 3.0.0 probably being the cleanest option owing to semver behavior : semver.satisfies('2.1.0', '^2.0.0-alpha') // => true and semver.satisfies('2.0.0', '^2.0.0-alpha') // => true. When the new architecture is ready to become the stable release, it could be published as the unscoped package as 4.0.0 or whatever major version bump is applicable at that time.

This would probably be a solution to not break those DApps which already do use the web3.js lib with the fully refactored architecture (2.0).
Another option would be to back-port the newly added modules of 2.0 to 1.0 and all other added features and to synchronize the number handling (returning numbers as strings and not BN objects). This would give us the possibility to release those changes as 2.0 but would break the number handling and wallet interface for the 1.0 and 2.0-alpha users.

All these approaches have downsides. I'm not sure what's the best way.

Same here, but I think we will find the best solution here in this issue:)
Thanks for your feedback, Michael!

[holgerd77]

Maybe I am seeing this to simple, but I think the best and also least confusing way is just to break with semantic versioning on the 1.x branch and do breaking releases as necessary in the minor releases. I think people will very much understand that, since this is just the most "natural" solution coming from the circumstances. Using devs are likely very much aware of the existence of the 2.x branch and can "feel" the need for a clean solution here.

I think key here is just very much communication and to put the word out on (various) occasion(s) (@nivida: eventually even decide quickly and put the info in the upcoming Dev update branch?) so that people can prepare. And if this breaks some things, this is still better to once have this breaking occurence within a clear cut and softly force people into once do another release on their libraries and e.g. narrowing down the Web3.js 1.x dependency - than many of the other solutions (go on with v3.0.0) which will likely remain some unnatural explain-me-thing for the foreseeable future and will cause somewhat unpredictable hazzle.

Just my 2 cents.

[michaelsbradleyjr]

@nivida do you presently have a sense/goal of when the 2.x architecture will be ready to become the stable release +/- 3 months, e.g. Q2 2020?

@holgerd77

Maybe I am seeing this to simple, but I think the best and also least confusing way is just to break with semantic versioning on the 1.x branch and do breaking releases as necessary in the minor releases.

I personally feel this is acceptable, despite downsides. But, if that's the way it goes, I also think the README for the 1.x branch (currently this repo's default branch) should prominently feature (above the Installation section) a notice to the effect that:

This project's 1.x releases will not necessarily adhere to semver in its minor releases and therefore downstream projects should either specify an exact version or a patch .x range, e.g. 1.3.0 or 1.3.x, until they are ready to migrate to ^2.0.0. Starting with 2.0.0 this project will strictly adhere to semver.

However, if active development of 1.x is anticipated to stretch into late 2020 or beyond, and if there is any sense or "gut feeling" that 1.x might see 2+ rounds of breaking changes in its minor versions, I'd be more in favor of "communication" (see below) plus releasing 1.3.0 as 3.0.0 and simultaneously doing future 2.x architecture ("new architecture") releases in a scoped package, with the plan to release the new architecture, when it's ready, as a future major version of the unscoped package. To @lautarodragan's point above, take a look at the releases for ipfs-http-client (notwithstanding ipfs-inactive/js-ipfs-http-client#1175). Without necessarily praising that project's rather large number of major releases (@alcuadrado's point is not lost on me), I think what's evident is the flexibility of iterative development (@iurimatias) that adheres to semver: when the project hits a breaking change, its maintainers simply release a major version, and likewise for patch and minor versions.

I think key here is just very much communication...

👏 I agree 100% — blog posts, prominent notices in READMEs, etc. Whatever path is chosen, this is key.

I think people will very much understand...

Some users will be upset / confused either way, but as I once remarked to a team member: some days us Open Source devs are cutting beautiful gemstones 💎 and other days it's more like we're making sausage 🌭.

[gnidan]

To lend my perspective, I strongly discourage breaking changes without respecting semver. It might seem like a trivial issue from the perspective of this project, but substantial breaking changes destroy development velocity downstream. Take it from someone who lives downstream.

Now, looking at these breaking changes, I don't have a sense for the severity, but they don't seem to be that big a deal. At the surface, they appear to be changes to which I wouldn't be opposed. In absence of time to evaluate each change for breakages, has anyone collected this data? Not just the list of issues, as @nivida includes, but a full and up-to-date description of the impact on the external interface (in terms of what the interface is now vs. what the interface will be). That would help me anticipate the required changes on Truffle's codebase, as well as the impact on Truffle's users.

In general, though, I am really into @michaelsbradleyjr's idea of making a web3-next package. The original plan for 1.x was to avoid non-patch releases, but since that now seems infeasible, it's really looking like the number space between 1.0.0 and 2.0.0 isn't as wide as we thought. I can see the downsides, certainly, but please: it's really hard to abide severe breakages.

I'll try to dig into the specifics more next week, to test my hypothesis that the listed breakages are minor enough to accept. I hope my grave warnings here are not necessary 😄

[joshstevens19]

Hey nice to see this discussion going on. So my view on this is quite simple and some of you have kinda said the same thing. Angular made this mistake, they released the router package on version 3 but their core packages was version 2 so when they went to bump packages the router package was out of sync. Instead of really overthinking everything they thought "users won't care here really as long as they can install the new package" they just bumped it to version 4, and guess what nobody cared and it solved their issue.

My proposal would be:

1. Leave 1.x alone now, only major bug fixes, no breaking changes. The whole aim of coming back to 1.x was to make it stable and tick off all the issues on the list. Once that is done it should in my view be closed and only merged in for major bugs. If we keep working on the 1.x branch as we are its going to easily bring in breaking changes and start being a beta unstable version again which was what we're trying to get away from.

2. I am afraid once you release a package in the wild you have to honour people may be using that package in production code. We at funfair actually do use the 2.x-alpha package for some stuff. The tools have to admit and take responsibility if the versions get messed up but like i said they're out in the wild and people are going to be using them, so ability to patch any critical fixes is a must, we can't just assume people may not be using it and put breaking changes in a version branch which then highlights a critical fix needed. This would then cause the developers software which had the old alpha package to either not upgrade due to breaking changes and keep the big security issue on their app or take the hit and rework their app to resolve the breaking changes introduced (all of that is a must no in my view).

3. So that leads to.... Leave 2.x alone as well same reasons as above.

4. The whole new architecture should be worked on within a 3.x branch, this then means its very clear where the new code is and protects us from all areas. This is then easy to communicate to the community and solves all our issues.

5. I would not start doing 1.3.0 > 3.0.0 as then again you added another dependency version which you have to honour. You would then start making the complete new web3 rewrite in that branch your then hurting people who installed 1.3.0 and stopping them from being able to fix any major security issues. oh and then you start jumping to 4.x, 5.x etc etc its a nasty cycle.

6. As long as we then mark/agree 1.x is stable enough so dapps can work good enough, people will get on with it and be fine. This then allows @nivida to be able to crack on with the entire rewrite of the new web3 and we all can be happy.

7. All of these fixes we are doing with the lib are great but i'm guessing they have lived on this library for a very long time? even from the start? So nothing new and dapps have managed to get around it. Not saying that the overflow issue is not important i'm saying web3 is 3+years old and if this existed from the get go i'm sure it can wait and be fixed in the new web3 when that is completed? I am also assuming the overflow only affected a few probably "unpopular" calls. Again i am just looking at the bigger picture here.

key benefits:

• Locking 1.x and marking as stable so dapps don't break and let it start getting the view of being a 1.x stable web3 version. We need stability here with web3 and this would be a really good step forward. Also remember this was the sole reason 1.x got touched again... for stability.
• If a major security issue is found on 1.x or 2.x-alpha we can publish a fix without having the complications of maybe having breaking changes, again honouring the package has been published and doing the correct thing for anyone who has that in production.
• It's then very clear that all the new stuff lives under the 3.x branch (once its ready) and we avoid any confusion going forward.

Anyway just my view, open to any discussions around it 👍

[nivida]

@michaelsbradleyjr @alcuadrado @holgerd77 @lautarodragan @joshstevens19 @gnidan

I'm impressed by all the feedback we got together here and I think we have definitely listed all possible solutions. ;)

@cgewecke and I had a call yesterday evening to discuss the coming release and especially all the inputs we got here. We were both agreeing in the end that we won't publish any breaking changes within the 1.x version scope. This doesn't mean that we will remove those issues above but we will definitely only implement those additions or fixes (mostly edge-cases) who don't introduce a breaking change.

Is there any missing feature or non-breaking fix who should be a part of the 1.3.0 release and isn't listed above?

Edit:
I've updated the list above.