plugins and back compat

[Raynos]

When we break back compat on levelup all the plugins may or may not break.

A lot of level-* modules in the ecosystem use level-sublevel to extend a LevelUP instance with new functionality.

This works great up and until we change levelup in a back compat breaking way like removing createWriteStream() or changing the createReadStream() range options API.

The core problem is that level-* modules have an invisible peer dependency on levelup at a certain version. When levelup changes those modules break.

It also means that as the ecosystem grows you won't be able to use certain level-* modules together because you have module A using the old range query API and module B using the new range query API.

Core problem

plugins do not depend on something that can be versioned statically and instead depend on something passed in at run time with a certain interface.

Possible solutions

1. Never break back compat. This may sound silly but it's genuinely the solution.
2. Peer depend on things. I highly recommend against this, my experience with peer deps is a disaster. @shama also has opinions about peer deps being a "solution" for plugins.
3. statically require and depend on levelup. This is the solution that would work the best.

How do you require levelup in a plugin?

If we were to combine 1 & 3 together we could in theory choose to never break back compat on leveldown and break levelup up so that its a set of functions that take a leveldown instance as an argument.

The solution is to break levelup into two modules. One module is levelup, the interface you use in your app because you want the nice interface and the streams and everything. The second is a set of functions that are versioned independently and can be used by level- plugin authors

In that regard a level- plugin author would a leveldown instance (probably wrapped by levelup or passed directly) and it would call functions from the levelup module on the leveldown instance. This means that if we remove the createWriteStream from the nice levelup interface it means nothing for level- authors because they always depended on the write-stream.js module directly and called it as a function.

What does this look like for level- authors?

Imagine I had an imaginary copy-users level- plugin that would currently look like

function copyUsers(db) {
  db.copyUsers = function (targetDb) {
    db.createReadStream({
      start: "users~",
      end: "users~~"
    }).write(targetDb.createWriteStream())
  }
  return db
}

If I wanted to use the new functional equivelant it would look like

var readStream = require("levelup/fns/read-stream")
var writeStream = require("maxes-write-stream")

function CopyUsers(db) {
  return function copyUsers(targetDb) {
    readStream(db, { gt: "users~", lt: "users~~" })
      .pipe(writeStream(targetDb))
  }
}

Notice how this second version uses the new gt & lt api for read stream because it uses read-stream at a specific version that has that new api. It also uses maxes write stream from npm.

Any breaking changes to the levelup interface for db do not break this module (as long as abstract leveldown doesn't change)

What does this look like for levelup

The public levelup interface doesn't need to change. The internals would be refactored so all methods re-use the functions.

Those functions could live as seperate versioned modules or as files in a folder inside levelup.

Benefits.

As long as the functions are individually versioned we can break back compat on the public db interface of levelup as much as we want without breaking anything in the level- eco system.

In the long run this means that we will never run into the "this plugin only works with levelup1 not levelup2" problem. We still run into a "this plugin only works with abstract leveldown1 not abstract leveldown2" problem.

[Raynos]

I forgot to mention disadvantages of this process

Disadvantages

• if we have each levelup method individually versioned we need a module (maybe git repo) for each one. This is a pain to manage, it also splits out communication. The split of communication is good / bad dependending on who you ask.
• exposing leveldown instances as those are passed to plugins. Maybe not breaking back compat on leveldown is harder then not breaking back compat on levelup. We have to ask a question about which is better, should level plugins invisibly peer depend on the leveldown or levelup interface.

I'm not actually sure whether this is a good idea, but it's an interesting way to avoid breaking plugins with levelup changes.

[shama]

Just throwing this in... @mikolalysenko's ndarray project is a good example that I think will better avoid the plugin problem. ndarray modules operate on a data object format rather than an instance of ndarray with the APIs attached. So each ndarray module can lock to a specific version of ndarray. ndarray "plugins" will only run into trouble if the format needs to change dramatically. The format is fairly simple though so not likely to happen very often, if at all anymore.

[juliangruber]

When this happened to me it wasn't because of a breaking change but because of a feature addition, that wasn't supported by modules that were patching levelup.

To prevent this from happening again, we either need to

• officially support modules that necessarily overwrite levelup methods
• OR expose the mechanisms to extend / overwrite without patching and thus staying compatible

However I think that this is not levelup's responsibility, as levelup is the lowest-posible-level js api for accessing levelup backends and shouldn't be concerned with those extension needs.

So, either

• support modules like hooks and sublevel
• OR write a plugin layer that is officially supported
• OR write a higher level version of levelup with certain features already built in

[Raynos]

@juliangruber

in theory if your module uses sublevel and if the database has been wrapped by the latest version of sublevel by the application user then even if your module uses a really old version of sublevel then new features shouldn't break.

But this raises a good point, I'm not sure how sublevel would be implemented with the seperate functions approach as it would have to monkey patch a leveldown interface or be baked into levelup.

[juliangruber]

@Raynos see dominictarr/level-sublevel#41 it doesn't support the new ranges.

[juliangruber]

Another common issue is the chained batch that for example multilevel and sub level don't yet support.