Upcoming rewrite of git history

[facchinm]

Hi everyone
Since we are approaching a beta2 stage, we'd like to cleanup the repo before it's too late 🙂
To keep things "easy", we used to commit llext-edk and firmwares folders, and this creates a couple of problems, namely:

• need to keep in sync the code and the artifacts
• the repo gets bigger and bigger, with a lot of useless "mass rebuilds"

So we overhauled the CI and use it as an artifact provider! When #102 will be merged, every time we push to main, the compiled files will be uploaded to our servers, ready to be retrieved by a fancy utility in the exact expected status.

This will primarily benefit users who don't want to install the whole zephyr toolchain to just modify the core/library files, while the "hardcore" users will not see any change in their workflow (except a much cleaner git history 😉 )

The only drawback (@KurtE @mjs513) is that git pull will not sync anymore after the merge (if coming from an "uncleaned" core) so the core will need to be redownloaded 😞

[KurtE]

Maybe something like git stash will still allow you to get around this. I usually have run into this with the current stuff anyway as I
usually have stuff from the previous build_all

But I can also imagine we might need to do a fresh reload after this PR is done!

[KurtE]

Looks like: maybe yesterday you updated the main project here with this change?

Any more updates to our work flow.

That is if I sync up to this, and then do:
west update
extras/build_all.sh

Can we then still simply choose which board we want in the IDE and do the flash? Or do we need to run some other script to then
update it within the IDE area?

I did see in the readme that you can do the ./extra/build.sh on a specific board
and then simply do west flash

Looks like most of the other PRs have been pulled in and closed. Note On portenta H7, IO pins. Currently the only ones that
can be used for pinMode, digitalWrite, etc are those that are on the outside connector. So none of the ones on the high
density connectors that are used for GPIO on the breakboards ... But that is a different subject

[mjs513]

@KurtE
Had the same question on how to build for individual boards with the latest changes to the build process. Was looking the other day.

Also noticed looks like there is going to be 2 build options for GIga - one with and one without the display - or am I reading that wrong?

As for the Portenta H7 - kind of limited with out the high density pins defined - or are you plaining on separate builds for the different carrier boards? Inquiring minds want to know - otherwise will probably add them locally.

Also any idea when the next official release will be?

[pillo79]

Yes, this is now merged. The outputs from the build procedure are now not committed and ignored, so a git pull (or any other Git operation) will not modify them.

The pros of not having binary / build output files in the repository:

• the repo is significantly smaller;
• there are way less commit conflicts;
• there is no possibility of having out-of-date files in Git;
• there is only one source of truth: the GitHub build infrastructure.

The cons:

• Doing a git pull and using the new core in a directory with previously built cores will result in confusion 😵‍💫 You now need to always update the artifacts after a pull!

How to update the artifacts after a git command changes the source files:

• Building the core for all targets (or individual boards) exactly as before, or
• run the new zephyr-sync-artifacts tool that is there but we still haven't fully released 😇

We also still need to clear the repository of the old files - that will be a "ground zero" day where all commits are redone; we may even move to a different repository. I think the next release will have this and the tool properly packaged for everyone to use!

Hope this clears a few questions 🙂

[KurtE]

Thanks @pillo79,

Yes it helps! It is great to not have all of those conflicts show up, let along the time it takes to upload/download all of these files!

But I am guessing that this tool will only help sync up, if/when I sync up to the https://www.github.com/arduino/ArduinoCore-zephyr
arduino branch? That is I am guessing that if I check in something to my own fork/branch, like WIP changes for a PR, and then
go to a different machine of mine (or my Windows copy), and sync up, to my current changes, the zephyr-sync-artifacts tools
will not grab new binaries for my fork/branch.

Likewise, I assume, it will not work, if I sync up to the USB fixes PR, it won't bring down the artifacts for this?

git fetch upstream pull/96/head:usb_fixes
git checkout usb_fixes

Although, maybe it does? if the github checks builds the variants and keeps the artifacts.

Side note: I appreciate how the build_all.sh now automatically builds all of the current variant!
But at times I miss the old simple scripts in that, it was easy to look up the arguments to pass to build.sh to build for
a specific board. Some reason I have a hard time remembering to build for the giga it is:

./extra/build.sh arduino_giga_r1//m7 --shield giga_display_shield
Each time I end up needing to look into boards.txt for the two lines:

giga.build.zephyr_target=arduino_giga_r1//m7
giga.build.zephyr_args=--shield giga_display_shield

Although it is probably just me 😆

[pillo79]

You are correct, this will no longer work for the two cases you mentioned:

• when moving locally from devel machine to test machine
• when pushing to forks and PRs in Github

However,

• in the first case you can manually create a tarball after the build with extra/package_core.sh and copy/extract that to the test machine's core folder. Bonus: there are no extra files on the test machine, so no confusion of what is used where.
• in the second case the artifacts get built but they are not downloaded - we will need to look into supporting this situation.

Re/ the side note, I agree! Just pushed an update allowing build.sh to understand Arduino board names. So you can simply run extra/build.sh giga and it will pick the proper command as defined in the boards.txt entry with that name 😉

[KurtE]

Thanks @pillo79 - Yep, the side note makes it a lot easier to remember... Will try out the stuff that you mentioned that @mjs513
is trying out.

Currently I am doing it with a set of rsync commands between the ubuntu FS and windows FS.

As a different side note, trying to figure out why some of our test sketches where we are trying out things on the display, work on the
0.3.1 release, but not on the current stuff. And in fact just changing the bootloader shows it...
I included it here in case anyone is interested, should probably instead add it to the issue about the display...

zephyr_GIGA_shield_touchpaint-250429a.zip

I added simple fill the screen with 4 colors directly into our SDRAM buffer and then tell it display. This was just a check to make
sure our quick and dirty GFX wrapper was not totally screwing up.

And it pauses waiting for input from user to continue...
On current code, the full screen display look like they might work, but this display, sort of momentarily shows something like our image and then goes to full red screen...

Debugged it into stm32_ltdc_write call: where I added prints for passed in stuff:

static int stm32_ltdc_write(const struct device *dev, const uint16_t x,
				const uint16_t y,
				const struct display_buffer_descriptor *desc,
				const void *buf)
{
...
	printk("stm32_ltdc_write(%p %u %u %p %p):", dev, x, y, desc, buf);
	if ((x == 0) && (y == 0) &&
...
	} else {
...
		pend_buf = dst;
		printk("pend_buf: %p\n", pend_buf);

And the prints show:

Display::write8(0 0 0xc000000c) 0x806351c 804d6e5
stm32_ltdc_write(0x8062e1c 0 0 0xa 0xc000000c):pend_buf: 0xc0000080
Display::write8(0 0 0xc000000c) 0x806351c 804d6e5
stm32_ltdc_write(0x8062e1c 0 0 0xa 0xc000000c):pend_buf: 0xc0000080
Display::write8(0 0 0xc000000c) 0x806351c 804d6e5
stm32_ltdc_write(0x8062e1c 0 0 0xa 0xc000000c):pend_buf: 0xc0000080

So looks like the memory we allocated for our frame buffer in SDRAM
0xc000000c from above:

void GigaDisplay_GFX::begin() {
    display = new Display();
    display->begin();

    SDRAM.begin();
    buffer = (uint16_t*)SDRAM.malloc(this->width() * this-> height() * sizeof(uint16_t));
    sizeof_framebuffer = this->width() * this-> height();

Overlaps the memory for the pending buffer; 0xc0000080

So something not in place for the shared memory pool or ??? ... Investigating

[KurtE]

quick update: if I hack it up to use the shared memory allocator, it appears to fix it:

#ifdef CONFIG_SHARED_MULTI_HEAP
#include <zephyr/kernel.h>
#include <zephyr/devicetree.h>
#include <zephyr/multi_heap/shared_multi_heap.h>
#endif

void GigaDisplay_GFX::begin() {
    display = new Display();
    display->begin();
#ifdef CONFIG_SHARED_MULTI_HEAP
    buffer = (uint16_t*)shared_multi_heap_aligned_alloc(SMH_REG_ATTR_EXTERNAL, 16, (this->width() * this-> height() * sizeof(uint16_t)));
#else
    SDRAM.begin();
    buffer = (uint16_t*)SDRAM.malloc(this->width() * this-> height() * sizeof(uint16_t));
#endif    
    sizeof_framebuffer = this->width() * this-> height();
    this->display->setFrameDesc(this->width(), this-> height(), this-> height(), sizeof_framebuffer);
    Serial.print("Buffer: 0x"); Serial.println((uint32_t)buffer, HEX);

Not sure why the difference between builds... but..
Or if this is the needed fix

[mjs513]

@KurtE
Just updated the couple of test sketches I have as well - graphicstest etc. and using SMH seems to have fixed the issue.
Looks like new PR issued #117

as you said not sure why specifing using sdram for full screen does not work.

[mjs513]

Re/ the side note, I agree! Just pushed an update allowing build.sh to understand Arduino board names. So you can simply run extra/build.sh giga and it will pick the proper command as defined in the boards.txt entry with that name 😉

Makes life a lot simpler when we are only working with one board

Have to get use to the changes you are making. Argh

EDIT: Hope you update the readme with the these changes so they are all in one place

[mjs513]

@pillo79

tried running synch artifacts and got

./sync-zephyr-artifacts
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:

??? what am I missing :)

[pillo79]

Error handling can definitely be improved 🤣 the current syntax is:

sync-zephyr-artifacts <path-to-repo> [<sha-id>]

so the simplest call is sync-zephyr-artifacts . (with the dot as "current directory") and it will figure out the sha from the Git information. For debug purposes you can also give a sha and it will download that (if it is available).
Note that only branch "heads" are built, so when multiple commits are pushed at once, only the most recent one is available. In practice it should not be an issue because you either just want "the latest" (which is always available) or you are interested in developing with the core further and thus will install Zephyr and build your own cores.

[mjs513]

Just tried it with a fresh copy of the core:

root@DESKTOP-5R9LQSU:/home/my_new_zephyr_folder/ArduinoCore-zephyr# ./extra/sync-zephyr-artifacts/syn
c-zephyr-artifacts.sh ./home/my_new_zephyr_folder/ArduinoCore-zephyr
Git SHA: 9bfd899
Download URL: http://downloads.arduino.cc/cores/zephyr/ArduinoCore-zephyr-9bfd899.tar.bz2
Error downloading archive: bad status: 404 Not Found

learning process for me - @KurtE is more familar with this.

[KurtE]

@facchinm @pillo79 Wondering if this one should be closed and/or also updated the more recent issue
#163