Overhaul How Images Are Generated

[ToxicPine]

Hey @CamDavidsonPilon, I enjoyed your work with the Pioreactor so much that I felt like porting it to the Zymbit!

In my attempts to reproduce a working Pioreactor image on a more locked-down, secure operating system, I've become somewhat familiar with the interlocking pieces of software that make the image work, and thus I've come to share my feedback!

I appreciate your efforts very much, and I hope you can forgive me if I am too direct, here's what I think we need to address as soon as possible:

Firstly, I think we can do better regarding software updates. This build system seems to work with Arm64 as well as Arm32, but the latest upgrade script from the last 24.x.x to 25.1.12 takes installations out-of-step with the images produced by this tool when on Arm64. These updates fail because the script attempts to install a 32-bit ArmV7 Python wheel. This is just an example of what makes the current architecture is fragile. Generally, it's far to easy for new updates to result in Pioreactor installations that substantially differ from the default images produced by this tooling, especially if the files pulled from external servers by these scripts change, or if some error during an update leaves a previously working device in a half-working state. I suggest addressing this at the root by building these images on a system such as NixOS. It's an operating system which allows you to reproduce bit-for-bit identical set-ups across many devices, since the entire operating system is managed as a set of declarative configurations. By pushing a new configuration, every device reaches an identical state which is also identical with the default images. I suggest NixOS. Alternatively, consider an A/B partition layout for the root directory.

Secondly, I think we can all agree that build system for these images is far more fragile than we'd like. I've had to work through many silent failures without helpful error messages whilst trying to reproduce this set-up on a different Linux platform for the sake of Zymbit support, particularly around PioreactorUI itself. There are endless 403 errors if there's a missing file somewhere, or if a database is misconfigured, or anything else. Sometimes, these errors end up in inappropriate places within the WebUI, such as an XML-formatted HTTP Error 50x appearing in the pioreactor.local/config instead of the actual .ini data. There's a general lack of helpful error messages and documentation for those of us who wish to dig deeper into some of these codebases.

I think there's a lot of room for simplification, which could be achieved by relying less on this disjointed web of co-dependent services and script-driven set-ups. In particular, it appears to me that much of the logic concerning the creation of SQLite databases and configuration files could and should be moved out of these bash scripts, so that there may be clearer and more specific messages when any individual file or database is misconfigured, produced by the daemon which is actually concerned with those files. This also creates an opportunity for these files to be automatically re-populated if they are ever accidentally deleted.

[CamDavidsonPilon]

Hi @ToxicPine

Short answer: yes!

Medium answer: there's lots to discuss here, and I'll post a larger comment in the days ahead!