p5.js 2.0 Compatibility

[limzykenneth]

Hi @ziyuan-linn @shiffman @lindapaiste, we had briefly gone through the potential changes to preload to use promises in p5.js 2.0 before (@shiffman you have reviewed it in the proposal list as well) and we have now released p5.js 2.0 betas that uses an async setup function instead of the preload function.

I went ahead and tested an example from every model and find they generally don't need major changes as your existing implementation already uses promises anyway and preload is more of a patch for you. I've created a collection for you to checkout what the updated examples look like (except for SoundClassifier which I couldn't get to work in the regular version), but here is the ImageClassifier one as an example:

// Initialize the Image Classifier method with MobileNet. A callback needs to be passed.
let classifier;

// A variable to hold the image we want to classify
let img;

// Variables for displaying the results on the canvas
let label = "";
let confidence = "";

async function setup() {
  img = await loadImage("images/bird.jpg");
  classifier = ml5.imageClassifier("MobileNet");
  createCanvas(400, 400);
  let results = await classifier.classify(img);
  
  // The results are in an array ordered by confidence
  console.log(results);

  image(img, 0, 0, width, height);
  // Display the results on the canvas
  fill(255);
  stroke(0);
  textSize(18);
  label = "Label: " + results[0].label;
  confidence = "Confidence: " + nf(results[0].confidence, 0, 2);
  text(label, 10, 360);
  text(confidence, 10, 380);
}

There are a few considerations for ml5 and I'm very happy to help work through them however you wish.

---

In terms of transition, we are happy to work with whatever plan you'd like and based on p5.js' timeline we have a few suggestion that we are putting forward for addon libraries. The rough timeline is described here: processing/p5.js#7488 but essentially the p5.js web editor will still default to using 1.x, after the planned end of March 2025 release of p5.js 2.0, until August 2026 to provide ample time for transition and to reduce mid academic year disruption. Your release timeline don't need to match ours so this is just for reference.

For addon libraries, I see three options.

1. Release new version that only support p5.js 2.0 async/await while user needing preload function use the previous released version. This is potentially quite disruptive for your user.
2. Release parallel versions until August 2026, one supporting preload function, another supporting just promises.
3. Release version that is compatible with both promises and preload. In a way this is where ml5.js is at so I would recommend this, however there are a few things I noticed while working on the examples that I'll explain below.

For documentation and examples, we are planning to follow the same timeline roughly while working out a good way to make 1.x and 2.0 reference available for whoever needs either versions.

---

I noticed that the factory functions used to create the model object returns the created object directly instead of a promise, which I guess is to enable usage in preload since it doesn't work with returning promises. For most model it still works with async setup without issue but there are a couple model that require an async operation to complete before it can start working, the async operation is guarded behind the model.ready promise. While async setup can await the model.ready promise, which is what I did in those cases, it is likely not fully intended to be used this way. It adds an extra line to the user code and not all model needs it. I think this affect ml5.js usage outside of p5.js as well.

The ideal suggestion from me would be to have the factory function return a function that resolved to the model object that has all asynchronous setup task completed, similar to the pattern often seen in WASM backed libraries. This technically changes the API of your factory functions and may be a breaking change that you don't want to make.

The alternative is to propogate that model.ready promise down the chain to where it is async already and always await it before any operation. For example in BodyPose:

async detectLoop() {
  await this.ready;
  // Do everything else
}

This is a pattern I used in some obscure projects I did before and the downside is mainly additional boilerplate.

Let me know if my analysis above is incorrect and/or there's a better way to resolve this. I'm happy to go with whatever plan ml5 wants to go with in terms of implementing compatibility with p5.js 2.0, I can file PR towards them once there's a decision towards any particular solution as well.

[shiffman]

Hi @limzykenneth! First, many apologies for letting this issue linger for so long! Your thoughtfulness, thorough explanation, and consideration for the ml5.js project and other p5.js libraries is so appreciated! Thank you for the time and care you’ve put into explaining everything for us.

@ksen0 visited ITP this past week and we had a lovely discussion about p5.js and the changes related to async and await.

I know we may be a little bit late in terms of the 2.0 timeline, but the ml5.js team is planning some meetings on March 13 and 14th. Talking about p5.js 2.0 is on the the top of our agenda and we can share back more details here and begin any implementation details after our discussion!

My personal preference, if it's not too difficult, is to continue to support both preload() and await in the next 1.x release of ml5.js. If we release an ml5.js 2.0 someday, that would probably be a good time to remove support for preload alongside p5.js. I'm also inclined to update the ml5.js examples to use async/await by default but b/c there is a fairly extensive library of video tutorials and curriculum materials that rely on preload() I believe it's important to continue to support preload out of the box (whether with the p5.js compatibility add-on or just referencing p5.js 1.0 instead). We'll likely want to write up an FAQ about this on the ml5.js website as well!

Thanks again and feel free to let us know any further thoughts or updates here in this thread!

[limzykenneth]

In a way ml5.js already supports both preload() and async/await so I believe that is the best way forward for ml5.js as well. Let me know if you need any support or clarification on implementation as I'm planning to be working with other addon libraries for their transitions as well.