[Feature]: Update Ubuntu base image and MongoDB version #443
Comments
|
So there are certainly a number of reasons why I have held off upgrading to at least 22.04 - the first and the biggest elephant in the room being that MongoDB 3.x is no longer supported by MongoDB. There are at least still some level of security updates from Canonical with it being packaged in the official repos for 20.04 (focal) so it's basically at a dead end support wise. As for making the jump to noble/24.04 - that's not supported according to the FAQ so 22.04 would be where the base OS upgrade would need to end up. The good news about JDK is that focal, jammy, and noble all have OpenJDK 8, 11, 17, and 21 available. Comparing the versions from focal to noble, they're both running As for MongoDB 7, I don't see any reference to MongoDB 7 being the suggested version, at least not on the US version of the FAQ (https://www.tp-link.com/us/support/faq/3272/). It mentions: "Omada SDN Controller supports MongoDB v3 and v4. Here we will show how to install v4.4." The bad news is that even from MongoDB directly, MongoDB 4.4, which is the oldest version that TP-Link supports, went EOL on February 2024.. I will have to see if I can get someone on the TP-Link forums to provide some guidance around future MongoDB versions and see if they have any info on future support of newer versions. |
|
100% with you on all the above. jammy looks like the go-to base image for now.. Similarly, OpenJDK 17 seems fine as a choice, and the upgrade gap is not big... |
|
Ah I see - so it's starting with The thing that really sucks is that I do not want to continually ship all of the MongoDB binaries forever until people upgrade so I would be leaning toward doing a specific migration image that the user would have to run manually. The MongoDB binaries and dependencies are about 1 GB which is just a deal breaker for me. I am working on updating the |
|
OK, so https://github.com/mbentley/docker-omada-controller/tree/mongodb-upgrade-to-7.0/mongodb_upgrade At least the migrations run for both amd64 and arm64. I haven't yet tested to make sure that I can install MongoDB 7.x and have the controller start up but this is all I can probably work on for today. The basic run commands are: I didn't publish any of the images yet so they have to be built. |
|
Another thing would be that moving to a newer version of MongoDB would completely drop support for armv7l (which I would NOT be mad about at all). |
|
Here is a WIP branch: https://github.com/mbentley/docker-omada-controller/tree/update-base-and-mongo
docker build \
--pull \
--build-arg INSTALL_VER="5.14" \
--build-arg ARCH="amd64" \
--build-arg BASE=mbentley/ubuntu:22.04 \
--progress plain \
-f Dockerfile.v5.x \
-t mbentley/omada-controller:5.14-22.04-mongodb7-amd64 \
.
docker build \
--pull \
--build-arg INSTALL_VER="5.14" \
--build-arg ARCH="arm64" \
--build-arg BASE=mbentley/ubuntu:22.04 \
--progress plain \
-f Dockerfile.v5.x \
-t mbentley/omada-controller:5.14-22.04-mongodb7-arm64 \
.And two images published to Docker Hub:
At least I will be able to use these to test if the migration scripts worked. |
|
https://github.com/mbentley/docker-omada-controller/tree/mongodb-migrate-to-7.0/mongodb_upgrade Tested this with my current Omada 5.13 / MongoDB 3.68 on both amd64 and arm64. Both upgrades went fine:
I did try |
|
🤦 bummer. I will see if I can test it myself soon. *edit: oh who am I kidding - just tested it and it looks like I can start the controller for both the arm64 and amd64 images after running a migration. One thing I am not sure about that I will need to validate is if there is any good reason to actually run a repair before migrating. I'm guessing no and that could speed up the migration by a good bit. Another thing to add from a feature standpoint would be somehow making the entrypoint intelligent enough to detect if you are running the 22.04 based image with MongoDB 7 when you have not run a migration. I know that MongoDB will just fail to start but I would like to just spit out an error message that tells people to follow instructions somewhere to perform the migration. |
|
OK, I just made a pretty big update to the migration script in the branch to have cleaner output - all of the MongoDB logs are being put to a file. Repair should only be needed if the db was shut down uncleanly. Maybe that should be a first step before doing the first upgrade, just to make sure but this output is much more easy to understand if the upgrade was successful: |
|
So just thinking about the migration/upgrade script further, there are some additional things I would like to do:
And then in the actual controller image once it's updated:
I am sure I am forgetting about all sorts of scenarios that need to be addressed but it is good to think through earlier than later. |
|
Just as an FYI in case some one else runs into this. Trying out the new image with MongoDB7 with no luck. mongod process would crash. Turns out my CPU does not support the avx instruction set. This seems to be a requirement for mongodb 7. https://www.mongodb.com/community/forums/t/fresh-install-result-in-crash-at-startup/264498 |
|
Oh yikes. Good catch on that. I will have to see if I can add a check pre-upgrade to make sure that AVX is present although I imagine that the rollback process I'm working on should work if it fails to start. |
|
OK, I just pushed some changes to the upgrade branch:
|
|
Should the AVX requirement be treated as a requirement for running the controller? As described in the README vs scripted requirement? |
|
According to https://www.mongodb.com/docs/manual/administration/production-notes/#x86_64, the AVX requirement was added in MongoDB 5.x so anyone who can't run AVX basically can't run a supported MongoDB. AVX became available in 2011 and I am not sure how long it really took before it really became broadly adopted but that's almost 13 years ago at this point where both Intel and AMD had it first available in a released CPU. Regardless, MongoDB 6.x goes EOL July 2025 so that's not really a long runway. I would say that AVX will then become a requirement for running the controller so it should be both in the README and scripted to prevent a ton of issues being filed because it doesn't start.
|
In my case, it happens to be an Atom C3758 (Denverton family) so not too old. I'm guessing for the Atom format they removed some lesser used features to make it smaller/cooler/more power efficient/etc. |
I tried building a Docker image locally from this folder and running the migration script. It didn't work, output below. I was unable to locate the logs as you mentioned. The previous version of the script worked. |
|
Ah so this is part of the tricky part at the moment. The MongoDB logs are now being sent to I will update it to go to *edit: I pushed a commit that moved the logs to the persistent data location as mentioned above. |
Ah yeah, that's unfortunate. I have an Intel Atom C3558 which is my pfsense firewall and it also does not have AVX support but obviously it being my pfsense, I don't use it to run an Omada Controller. This makes me really wish that I had analytics about the controllers that run the image to better understand about them but that gets into an area I haven't been in before - having to deal with privacy of collecting data. |
If this is a container with the sole purpose of upgrading the database, I would bring the your treated output to the docker logs, and have the full logs into the file. Also, I figured the mapping of the data folder by looking into the script - I assume you are using the same data bindings as if it was a controller container. |
|
Sorry, I will add a README with better, consistent instructions as yes, the data is mapped the same as in the controller container.
I also just realized that another thing I didn't ever do was verify that there was actually data in the |
|
On another note, I was looking at the requirements for MongoDB incompatibilities are going to be a huge pain. I feel like I will end up maintaining the 20.04 based images for a lot longer than I was hoping along side updated 22.04 + MongoDB 7 😞 Pretty sure I have a Raspberry Pi 4 around here somewhere that I can take and do some testing on. My guess is that if it isn't supported, it will have a failure to start 5.x so hopefully the rollback logic still handles that use case. |
|
There is now a README with instructions. The TODO list is at the top of the |
|
Awesome, will give it another try on my Orange Pi 5 arm64 and on Intel Pentium Gold G7400 amd64 architecture. |
Ah yeah, it looks like the RK3588S is a Cortex-A76 which has arm 8.1 & 8.2 extensions so that could explain why it works fine. I also have a few Raspberry Pi 3 B+ boards that I can test on as well since the Cortex A53 (rpi 3b+) and Cortex A72 (rpi 4) have the same instruction set ( |
|
Followed README. I got the repair error, may be because that my omada-controller was not gracefully shutdown: 2024-07-29T13:51:51.891+0000 F STORAGE [initandlisten] An incomplete repair has been detected! This is likely because a repair operation unexpectedly failed before completing. MongoDB will not start up again without --repair. Maybe you could add back the repair to prevent this issue, at least from the first step of migration? This would prevent this from plain out stopping. |
|
Thanks for the feedback. I think it's definitely worth the delay of having to run the repair to ensure this will work for most people out of the box. I just pushed a change to the branch so that it will perform a repair before the first upgrade to make sure it won't run into this issue. |
|
I see in the latest beta community forum thread that there is a note about them adding support for MongoDB 8.0. I'm going to have to watch out in the change logs to make sure that they don't deprecate the current version of MongoDB that's used in the image since that would force a lot of people to perform a migration. |
|
I've created a new branch to go to 8.0: https://github.com/mbentley/docker-omada-controller/tree/update-base-and-mongo/mongodb_upgrade and done some basic testing: # build migration image
docker build \
--pull \
--progress plain \
-f Dockerfile.upgrade-amd64 \
-t mbentley/omada-controller:mongodb-upgrade-amd64 \
.
# run controller
docker run -d \
--name omada-controller \
--stop-timeout 60 \
--restart unless-stopped \
--ulimit nofile=4096:8192 \
--net=host \
-v omada-data:/opt/tplink/EAPController/data \
-v omada-logs:/opt/tplink/EAPController/logs \
mbentley/omada-controller:5.15
# configure controller in the interface and then stop
docker stop omada-controller
# create my own backup to be able to roll back to test multiple times
docker run -it --rm -v omada-data:/data -w /data alpine \
sh -c 'tar cvf db.tar db && ls -l /data/db.tar'
# run the upgrade
docker run -it --rm \
-v omada-data:/opt/tplink/EAPController/data \
mbentley/omada-controller:mongodb-upgrade-amd64
# roll back upgrade to test multiple times
docker run -it --rm -v omada-data:/data -w /data alpine \
sh -c 'rm -rfv /data/db/ && tar xvf db.tar && ls -l /data/db/' |
|
Would be willing to test this thing, since we might need to at some point, due to cluster mode and all. So i think it would be nice to get this thing done to the mainline containers. Who knows, maybe we get some speed improvements also. :) |
|
Putting so thoughts down...with May 2025 being the end of standard support for 20.04/focal, this is something that I probably realistically need to finalize the plan of action on this month. There are a few tricky bits:
I was really hoping that TP-Link would have a new major version release of 6.x at some point before the EOS for 20.04/focal which would be the trigger for the breaking change seeing as this migration needs a separate image due to the OS dependencies to run so many versions of MongoDB. That means that I will most likely have to push this change somewhat randomly. To reduce the support burden and user pain, I can probably just create a set of image tags for the last version of 20.04, like with
There are a few more things left in the To Do list that would be good to have as there will likely be failures on arm64 systems that don't have the right chip level support I'll also need to do the logic in the entrypoint to block trying to start for the new OS versions but that should be pretty easy as I'll just be able to check the OS version like I am doing here but checking for not a match and know from that. |
|
OK, sorry if anyone has been trying to do anything with any of the branches testing wise or whatever but I merged mongodb-upgrade-to-8.0 into update-base-and-mongo just to simplify any ongoing work. I would really like to see if there is some way to test if a system is running ARMv8.2-A or later for arm64 to add a check, otherwise it will try to run the migration and fail. This not working on any Raspberry Pi 4 and older is really going to suck too! |
|
Hi If you'd like to test on a Pi 4, this might help: I'm using it on another device running a Cortex-A72 without any issues. |
|
Unfortunately those would add a good bit of technical debt, introducing significant inconsistency between amd64 and arm64 as well as having amd64 special builds for just raspberry pi devices which isn't something I would like to do. I did that for 32 bit armv7l for a while and it was less than ideal. - plus as far as I am aware, there is no way to detect naturally through Docker a difference between At this point, I would rather just continue to build images that are As for detecting gcc -g atomic.c -o atomic -march=armv8.2-a -lpthread -static#include <stdio.h>
#include <threads.h>
#include <stdatomic.h>
atomic_int acnt;
int cnt;
int f(void* thr_data)
{
for(int n = 0; n < 1000; ++n) {
++cnt;
++acnt;
}
return 0;
}
int main(void)
{
thrd_t thr[10];
for(int n = 0; n < 10; ++n)
thrd_create(&thr[n], f, NULL);
for(int n = 0; n < 10; ++n)
thrd_join(thr[n], NULL);
printf("The atomic counter is %u\n", acnt);
printf("The non-atomic counter is %u\n", cnt);
}It will error out on systems that don't have support: And execute properly on those that do: It's only |
|
I do suppose that if someone wants to still use their raspberry pi and use their own MongoDB using the external MongoDB capability, that would work since they could run a version 4.4 that would work on their Raspberry Pi. That also means I need to think about more complex checks for blocking starting because I hadn’t taken external MongoDB support into account. |
What problem are you looking to solve?
TP-Link FAQ for Omanda SDN Controller install on Linux suggests Ubuntu 22.04 as the base system, with MongoDB 7 as suggested for the database.
The image is using Ubuntu 20.04 with the built-in MongoDB version 3.68. I found this while investigating a fix for
It would be interesting to provide a more current tech stack with either jammy or noble as the default, and with a more modern and currently supported MongoDB version. Reasons for this change:
Steps to provide a solution
ARG BASE=ubuntu:22.04) - currently just overriding to build-legacytag-legacyimage build for the previous versionsAdditional Context
It would be important to start this upgrade with MongoDB upgrade, given that you have to walk version to version to get to a more modern version. I'm facing challenges running 4.0 on an Ubuntu noble install due to outdated libraries.
The text was updated successfully, but these errors were encountered: