Save each partition as individual binaries

[chris-subtlebytes]

Fixes #714

[SergioGasquez]

I guess we need to decide a solution, I like what @ivmarkov suggested here: #714 (comment)

I would try to avoid having major breaking changes as we recently released 3.0

[chris-subtlebytes]

What are our options?

1. Add new commands save-firmware, save-part-table, and save-bootloader.

   From a user point of view, it becomes unwieldy to juggle three different commands especially because the partition table and partition table offset needs to be specified for each one. I already needed to juggle a handful while signing and encrypting. Remembering to update three different commands would have caused many more errors on my end. I was modifying the partition table to add nvs-keys, modifying sdkconfig.defaults to change bootloader flags, and messing about with the actual firmware image all at the same time. Having many commands that all need the same flags (bootloader and partition table) increases the odds that one of them has an incorrect value when the user is running each one separately.

2. Break compatibility by making --merge work as per docs and flag.

   Most users are going to use espflash flash, which isn't affected. I didn't need to save-image until I needed to sign and encrypt the bootloader and app image. Nobody can do that today because the functionality is broken. Every use case for save-image I can think of usually requires the individual parts, such as OTAs or inspecting the ELF.

   In my opinion, this still beta software that should be expected to break for bugfixes. If anybody has strong opinions on stability, I would hope they use esptool. The most impact a user would have is to add --merge to their command.

3. Update the default for --merge to true, but that's a bit tricky according to Add support for automatic negation flags clap-rs/clap#815

   This keeps the default behavior and docs the same.

4. Rename the --merge flag to --separate-bins
   This is similar to reversing the default value for --merge but still causes a minor breakage. I don't think anyone is using --merge because it's broken.

5. Is there any inspiration we can draw from esptool? I haven't used it much.

[ivmarkov]

What are our options?

1. Add new commands save-firmware, save-part-table, and save-bootloader.
   From a user point of view, it becomes unwieldy to juggle three different commands especially because the partition table and partition table offset needs to be specified for each one.

I don't think this statement is correct.
(Please check my understanding below, and if you think I'm wrong on any of those, if you can show the relevant code in espflash or ESP-IDF, really important to get this straight. :-) )

1. save-bootloader would not need neither the partition table, nor the partition table offset.

While save-bootloader can patch the built-in bootloader binary image to support a different flash size, turns out - just checked this morning - it cannot patch the built-in bootloader to support a different partition table offset than the default 0x8000 (@SergioGasquez right?).

Also, if you need a bootloader with a different partition table offset than 0x8000 OR a bootloader with secure boot capabilities (as I have doubts the built-in one is compiled with secure boot capabilities), save-bootloader is useless to you. I - for one - am instead using the custom-built bootloader that you always get with every single esp-idf-*-based binary crate, out of the box.

save-bootloader might use a user-supplied partition table offset to only check whether the built-in bootloader fits before the parrtition table start, but that's it.

2. save-firmware does not need the partition table itself or its offset either for the same reasons: it cannot patch an already compiled app image to a different partition table offset. The app image needs to know the partition table offset (but not the partition table CSV or BIN!) during OTA (updated: and also for all esp_partition_t-based APIs of course). The only way to do it is to compile the app image from the beginning with a new value for CONFIG_PARTITION_TABLE_OFFSET (if you use the esp-idf-* crates) or whatever ad-hoc method you use for baremetal, as OTA updates are not yet "officially" supported there, with esp-hal.

save-firmware might take advantage of a user-provided partition-table to check if the app image being converted to .BIN fits in the factory or one of the OTA partitions of the table, but that's it. And the current check for this is anyway a bit heuristical, as save-image can't actually know against which partition in the partition table to check the size of the app image anyway.

3. Obviously, save-part-table does not need to know the partition table offset either.

I already needed to juggle a handful while signing and encrypting. Remembering to update three different commands would have caused many more errors on my end. I was modifying the partition table to add nvs-keys, modifying sdkconfig.defaults to change bootloader flags, and messing about with the actual firmware image all at the same time. Having many commands that all need the same flags (bootloader and partition table) increases the odds that one of them has an incorrect value when the user is running each one separately.

I get it that it is difficult (I'm suffering myself through this), but we also have to be crystal clear among the three of us (@SergioGasquez included) what we need and what we don't / won't need/do w.r.t. support for Secure Boot in espflash.

[chris-subtlebytes]

Apologies for taking so long to respond; I'll try to be more active. Do we want this discussion in the PR or the issue?

save-bootloader would not need neither the partition table, nor the partition table offset.

If you want to support encryption, which is looks like #718 is going to do, then you will need the offset of the partition you're saving. See python -m espsecure encrypt_flash_data, which requires --address. If omitted: $ espsecure encrypt_flash_data: error: the following arguments are required: --address/-a because of how the encryption works.

Also, if you need a bootloader with a different partition table offset than 0x8000 OR a bootloader with secure boot capabilities (as I have doubts the built-in one is compiled with secure boot capabilities), save-bootloader is useless to you

This was annoying to encounter because I was setting bootloader-changing sdkconfig.defaults settings and scratching my head wondering why they're not working. I'm using --bootloader to get the one that's built anyway: espflash save-image --merge --chip esp32c3 --bootloader target/riscv32imc-esp-espidf/release/bootloader.bin --partition-table partitions.csv --partition-table-offset 0x000c000 target/riscv32imc-esp-espidf/release/my-project /tmp/espflash/my-image.bin

So I imagine espflash save-bootloader would need to be espflash save-bootloader --bootloader /target/*/*/bootloader.bin for anyone to use properly. It's still necessary to save the image like this because it corrects the header and SHA.

we also have to be crystal clear among the three of us what we need and what we don't / won't need/do w.r.t. support for Secure Boot in espflash.

Here's my full set of commands (using my current proposed commit), which I think touches most aspects aside from secure boot v1:

# step 1 build and save
$ cargo build --release
# save all image partitions as per proposed patch
$ espflash save-image --chip esp32c3 target/riscv32imc-esp-espidf/release/my-project path/to/my-project.bin

# step 2 generate signing keys
# do three times because there are three key efuse slots and three key revocation efuses
# you could add this to espflash, but would people really trust it over openssl? I suppose it may be more convenient but we're talking about security.
# for ESP32, ESP32-S2, ESP32-S3, ESP32-C3
$ openssl genrsa -out my-project_rsa3072_secure_boot_signing_key_0.pem 3072
# for ESP32-C2, ESP32-C6, ESP32-H2, ESP32-P4
$ openssl ecparam -name prime256v1 -genkey -noout -out my-project_p256_secure_boot_signing_key_0.pem

# step 3 sign binaries
# sign the image and bootloader from step 1 with the key generated in step 2 using espsecure.py.
# imo this could be folded into save-image with either `--sign0 path --sign1 path --sign2 path` or `--sign path0 --sign path1 --sign path2`
$ python -m espsecure sign_data FILE_FROM_ESPFLASH_SAVEIMAGE --version 2 --keyfile my-project_KEYTYPE_secure_boot_signing_key_0.pem
$ python -m espsecure sign_data path/to/bootloader.bin --version 2 --keyfile my-project_KEYTYPE_secure_boot_signing_key_0.pemmy-project_KEYTYPE_secure_boot_signing_key_1.pem my-project_KEYTYPE_secure_boot_signing_key_2.pem

# step 4 generate encryption key and burn it into efuse
# (usually only for development purposes as the ESP should do this internally for release devices)
$ openssl rand -out path/to/my-project_aes128_development_flash_encryption_key.key 32
$ python -m espefuse --port PORT --chip CHIP burn_key BLOCK_KEY3 path/to/my-project_aes128_development_flash_encryption_key.key XTS_AES_128_KEY

# step 4.5 encrypt binaries
# espflash could add `--encrypt` to `save-image`. Sometimes other partitions (nvs-keys, FAT, etc) need to be encrypted as well, so there needs to be 2 ways of doing it.
$ python -m espsecure encrypt_flash_data --keyfile ~/.secrets/my-project/my-project_aes128_development_flash_encryption_key.key --aes_xts --address 0x0010000 --out images/0x0010000_my-project.bin.encrypted images/0x0010000_my-project.bin

# step 5 generate hash of signing keys
# needs to be done once per signing key
# this could be added to espflash as a new command
$ python -m espsecure extract_public_key --version 2 --keyfile path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pem path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.pem
$ python -m espsecure digest_sbv2_public_key --keyfile path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.pem -o path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.digest
$ python -m espefuse --port PORT --chip CHIP burn_key BLOCK_KEYX path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.digest SECURE_BOOT_DIGESTX

# step 6 upload all partitions to device
# this step could definitely be improved by espflash as we have to run the command per each partition (bootloader, partition table, app). Care needs to be taken to still handle the bespoke partitions too.
$ espflash write-bin 0xADDRESS path/to/0x0000000_my-project.bin # bootloader
$ espflash write-bin 0xADDRESS path/to/0x000c000_my-project.bin # partition table
$ espflash write-bin 0xADDRESS path/to/0x0010000_my-project.bin # app
# extra flashes sometimes needed
# note that if the flash is encrypted, reading an erased region will look like valid (albeit random) data so encrypted devices always need to write encrypted zeroes after erasing for some features to work.
$ espflash write-bin 0xADDRESS path/to/0x000d000_my-project.bin # ota data, which sometimes needs to be written with zeroes to boot into the newly flashed factory partition instead of ota2
$ espflash write-bin 0xADDRESS path/to/0x03ed000_my-project.bin # nvs keys, which needs to be written with zeroes for it to internally generate a new key for an nvs partition.

# Wait about a minute for the plaintext flash image to self-encrypt. Interrupting before it's finished can cause corruption.
# If the device does not support RTS/DTR, manually enter download mode and run espflash with --before no-reset.

# verify that secure boot and flash encryption are enabled
$ python -m esptool get_security_info

#718 comments:

we are currently discussing a v4 release

Is there a chat room or new issue open for this? I still need to upload a lot of changes / bug reports / feature requests for esp-idf-[sys, svc] and if espflash v4 is related to those then maybe I can at least offload my ideas there. I think at this point I may have one of the most complex esp-idf projects so I've hit a lot of rough areas (which is expected).

[ivmarkov]

Thanks a lot for your feedback.

I've put comments below so that we can compare with the way I do these steps.

Apologies for taking so long to respond; I'll try to be more active. Do we want this discussion in the PR or the issue?

save-bootloader would not need neither the partition table, nor the partition table offset.

If you want to support encryption, which is looks like #718 is going to do, then you will need the offset of the partition you're saving. See python -m espsecure encrypt_flash_data, which requires --address. If omitted: $ espsecure encrypt_flash_data: error: the following arguments are required: --address/-a because of how the encryption works.

The --address parameter of the encrypt_flash_data command however is not the address of the partition table, but the starting address of the partition you are encrypting. This is so because encryption is position-dependent. For most chips the BL start address is 0x0; for some - it is 0x1000. In any case, espflash is well aware of the chip type and knows the bootloader address.

With that said, I don't think we should be having save-bootloader do encryption. If we do this, we should also make it do signing, as signing must happen before encryption.

Where do we stop?

I strongly believe each command should do just one simple thing. save-bootloader should just generate an unencrypted binary image of a bootloader so that you don't have to do it yourself. Not that I believe once encryption and secure boot are in play, this is not very realistic (that is, to use a pre-generated bootloader image) but anyway.

If you need to sign and/or encrypt it, you might use different commands (which - btw - and for the most part - wouldn't even care if they are encrypting a bootloader, an app image or some data partition). Just like the existing commands from Esapressif's Python tooling.

Also, if you need a bootloader with a different partition table offset than 0x8000 OR a bootloader with secure boot capabilities (as I have doubts the built-in one is compiled with secure boot capabilities), save-bootloader is useless to you

This was annoying to encounter because I was setting bootloader-changing sdkconfig.defaults settings and scratching my head wondering why they're not working. I'm using --bootloader to get the one that's built anyway: espflash save-image --merge --chip esp32c3 --bootloader target/riscv32imc-esp-espidf/release/bootloader.bin --partition-table partitions.csv --partition-table-offset 0x000c000 target/riscv32imc-esp-espidf/release/my-project /tmp/espflash/my-image.bin

So I imagine espflash save-bootloader would need to be espflash save-bootloader --bootloader /target/*/*/bootloader.bin for anyone to use properly. It's still necessary to save the image like this because it corrects the header and SHA.

The header and the SHA would only need to be corrected if the flash-size is changed inside the bootloader header.
Given that you use your custom-built bootloader, I do not understand why espflash should do that for you in the first place?
Just put CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y in your sdkconfig.defaults and then you don't have to deal with save-bootloader at all.

# step 2 generate signing keys
# do three times because there are three key efuse slots and three key revocation efuses
# you could add this to espflash, but would people really trust it over openssl? I suppose it may be more convenient but we're talking about security.
# for ESP32, ESP32-S2, ESP32-S3, ESP32-C3
$ openssl genrsa -out my-project_rsa3072_secure_boot_signing_key_0.pem 3072
# for ESP32-C2, ESP32-C6, ESP32-H2, ESP32-P4
$ openssl ecparam -name prime256v1 -genkey -noout -out my-project_p256_secure_boot_signing_key_0.pem

1. You need just one signing key
2. This signing key should be global and generated once only. Having a signing key per device is a bad idea. How do you send a signed OTA image to the device after that? By keeping all signed device-specific keys? So this is more like Step 0 or Step -1.

Step 2 is to only extract the public key from the signing key as it is needed during efuse further below.

UPDATE: Forgot to say that an alternative to 3 signing keys (too much hassle) is that you just close the remaining two slots. If I'm not mistaken, the efuse command would even do it if you allow it to... or was it the bootloader...

# step 3 sign binaries
# sign the image and bootloader from step 1 with the key generated in step 2 using espsecure.py.
# imo this could be folded into save-image with either `--sign0 path --sign1 path --sign2 path` or `--sign path0 --sign path1 --sign path2`
$ python -m espsecure sign_data FILE_FROM_ESPFLASH_SAVEIMAGE --version 2 --keyfile my-project_KEYTYPE_secure_boot_signing_key_0.pem
$ python -m espsecure sign_data path/to/bootloader.bin --version 2 --keyfile my-project_KEYTYPE_secure_boot_signing_key_0.pemmy-project_KEYTYPE_secure_boot_signing_key_1.pem my-project_KEYTYPE_secure_boot_signing_key_2.pem

Agreed

# step 4 generate encryption key and burn it into efuse
# (usually only for development purposes as the ESP should do this internally for release devices)
$ openssl rand -out path/to/my-project_aes128_development_flash_encryption_key.key 32
$ python -m espefuse --port PORT --chip CHIP burn_key BLOCK_KEY3 path/to/my-project_aes128_development_flash_encryption_key.key XTS_AES_128_KEY

Agreed. But not sure it is only necessary for development purposes. If you want to preserve the DL mode with security enabled and be capable of uploading new images via UART, you need to keep all encryption keys for all devices you have ever created (as I'll do). One never knows.

# step 4.5 encrypt binaries
# espflash could add `--encrypt` to `save-image`. Sometimes other partitions (nvs-keys, FAT, etc) need to be encrypted as well, so there needs to be 2 ways of doing it.
$ python -m espsecure encrypt_flash_data --keyfile ~/.secrets/my-project/my-project_aes128_development_flash_encryption_key.key --aes_xts --address 0x0010000 --out images/0x0010000_my-project.bin.encrypted images/0x0010000_my-project.bin

Agreed, but you already see how your save-image idea starts to break? encryption is after signing and does not care what is being encrypted. I don't think adding magic flags to save-image/save-bootloader/whatever-elf-to-bin cmd we come up with will make it clearer.

I think we should just follow the division of work (and general command layout) that the Espressif Python tooling did. It had stand the test of time at least.

# step 5 generate hash of signing keys
# needs to be done once per signing key
# this could be added to espflash as a new command
$ python -m espsecure extract_public_key --version 2 --keyfile path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pem path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.pem
$ python -m espsecure digest_sbv2_public_key --keyfile path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.pem -o path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.digest
$ python -m espefuse --port PORT --chip CHIP burn_key BLOCK_KEYX path/to/my-project_KEYTYPE_secure_boot_signing_key_X.pub.digest SECURE_BOOT_DIGESTX

An easier way is to use burn_key_digest instead of burn_key and skip the whole "generate hash from public key" thing, because burn_key_digest tales the public key rather than its sha-256 hash in the first place.

# step 6 upload all partitions to device
# this step could definitely be improved by espflash as we have to run the command per each partition (bootloader, partition table, app). Care needs to be taken to still handle the bespoke partitions too.
$ espflash write-bin 0xADDRESS path/to/0x0000000_my-project.bin # bootloader
$ espflash write-bin 0xADDRESS path/to/0x000c000_my-project.bin # partition table
$ espflash write-bin 0xADDRESS path/to/0x0010000_my-project.bin # app
# extra flashes sometimes needed
# note that if the flash is encrypted, reading an erased region will look like valid (albeit random) data so encrypted devices always need to write encrypted zeroes after erasing for some features to work.

Agreed, but you can also just merge all binaries together, then encrypt the whole thing, and then just tell espflash to write-bin the big elephant from address 0x0. Or help develop espfactory which does that and much more.

$ espflash write-bin 0xADDRESS path/to/0x000d000_my-project.bin # ota data, which sometimes needs to be written with zeroes to boot into the newly flashed factory partition instead of ota2

Agreed. An alternative is to just do espflash erase-flash before flashing, however "erase-flash" does not work with secure download mode enabled - neither with espflash which does not support secure DL mode at all, nor with esptool.

$ espflash write-bin 0xADDRESS path/to/0x03ed000_my-project.bin # nvs keys, which needs to be written with zeroes for it to internally generate a new key for an nvs partition.

Agreed if you start with an empty NVS partition. For suffciently-complex projects that do have device-specific data this might not be the case (it is not for me). In these cases you need to pre-generate and pre-encrypt both the NVS partition table contents, as well as the NVS keys partition.

# Wait about a minute for the plaintext flash image to self-encrypt. Interrupting before it's finished can cause corruption.

I don't think relying on the bootloader to do it is ideal. It is best to use CONFIG_SECURE_FLASH_REQUIRE_ALREADY_ENABLED=y and pre-encrypt all data yourself. This way the bootloader does not have to do the pre-encrypt process itself.
And - if you leave secure DL mode enabled you can continue re-flashing your pre-encrypted images, given that you keep the encryption key around. Because this bootloader-induced encryption works the first time only.

# If the device does not support RTS/DTR, manually enter download mode and run espflash with --before no-reset.

Why is this necessary?

# verify that secure boot and flash encryption are enabled
$ python -m esptool get_security_info

Agreed!

Is there a chat room or new issue open for this? I still need to upload a lot of changes / bug reports / feature requests for esp-idf-[sys, svc] and if espflash v4 is related to those then maybe I can at least offload my ideas there. I think at this point I may have one of the most complex esp-idf projects so I've hit a lot of rough areas (which is expected).

The chat is called esp-rs and is a Matrix chat (linked from most projects).

Feel free to open issues (and PRs) at the esp-idf-* projects.

[chris-subtlebytes]

The --address parameter of the encrypt_flash_data command however is not the address of the partition table, but the starting address of the partition you are encrypting.

If we have a save command for each (bootloader, partition table, app image), those offsets are all described in the partition table, then you'd need to manually supply the offset for each partition or use the partition table. That was the point I was trying to make if there was a need to support encryption. And yes, the bootloader won't need a specified offset.

I don't think we should be having save-bootloader do encryption. If we do this, we should also make it do signing, as signing must happen before encryption.

Sure, but I don't see why espflash couldn't sign as well. Of course these may not be features for a while or ever, but I don't think we should design the save-image commands to limit us from the opportunity. I'm not saying having separate commands for each partition is still compatible with signing and encryption, but it would get unwieldy passing the same flags in for each one.

put CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y in your sdkconfig.defaults and then you don't have to deal with save-bootloader

Fair, but I don't want to have a separate sdkconfig.defaults for every device I support. I'm sure I could come up with a workaround but using the same commands for each partition type was convenient.

You need just one signing key. This signing key should be global and generated once only. Having a signing key per device is a bad idea.

I think you misunderstand. Most ESPs support more than one signature key slot and a corresponding set of efuses for key revocation.

This is a mitigation in case the private signing key is leaked. Say my first signing key gets compromised, I can make a new image that revokes it and secure boot will burn the efuse and only use slots 2 and 3 for verification going forward. It happens :/ https://arstechnica.com/gadgets/2022/12/samsungs-android-app-signing-key-has-leaked-is-being-used-to-sign-malware/

Agreed, but you already see how your save-image idea starts to break? encryption is after signing and does not care what is being encrypted.

I still maintain that we could add a --sign flag. And yes I know how encryption and signing work; note the order of step 3 (sign) and step 4 (encrypt). I don't see how my idea is starting to break - please expand.

An easier way is to use burn_key_digest instead of burn_key

neat. maybe I'll use that moving forward.

Agreed, but you can also just merge all binaries together, then encrypt the whole thing, and then just tell espflash to write-bin

If espflash can do the whole thing (pad, sign parts, merge, encrypt, etc) all in one shot without any intermediate commands, that would be amazing.

Agreed. An alternative is to just do espflash erase-flash

If you erase-flash on an encrypted partition, the regular reads it will be reading encrypted raw flash zeroes which will be essentially noise to the reader. You need to generate a file of zeroes, encrypt it, then write the encrypted section. Because this is only a scenario for development, I don't see how secure download mode matters because you could just disable it.

I don't think relying on the bootloader to do it is ideal. It is best to use CONFIG_SECURE_FLASH_REQUIRE_ALREADY_ENABLED=y and pre-encrypt all data yourself.

You have a different threat model than me. If I distribute the software, why do I also need to be able to extract the consumer's data? I'm relying on the ESP to internally generate it's own unique encryption key so that the consumer can feel confident that it's nearly impossible (without breaking ESP's security model) to exfiltrate the data. In fact, the APIs I have for my project explicitly do not expose the data generated on device.

My instructions are tailored for building a release build.

# If the device does not support RTS/DTR, manually enter download mode and run espflash with --before no-reset.
Why is this necessary?

This wasn't too relevant to the discussion here but it made its way in from my copy/paste. It's a nice little tip for anyone that stumbles across this though.

---

I feel like your response was written from the perspective that I don't know what I'm doing. I provided the encryption instructions so we could look over what could/should be added to espflash and the merits of the different approaches for commands and flags.

It almost sounds like you're arguing FOR a single command instead of separate save-bootloader, save-parition-table, save-app-image in case espflash can support generating a signed, (sometimes encrypted) single fat binary.

Imagine a future like this:

espflash flash \
--chip chip \
--partition-table path/to/table --partition-table-offset 0x1234 \
--sign0 path/to/key0 --sign1 path/to/key1 --sign2 path/to/key2 \
--encrypt path/to/encryption-key \
--partition 'partition_name:path/to/contents' # in case you have custom otadata, nvs, nvs-keys, fat, etc., matched up with part name in the partition table.

And nearly identical save-image

espflash save-image \
--chip chip \
--partition-table path/to/table --partition-table-offset 0x1234 \
--sign0 path/to/key0 --sign1 path/to/key1 --sign2 path/to/key2 \
--encrypt path/to/encryption-key \
--partition 'partition_name:path/to/contents' \ # in case you have custom otadata, nvs, nvs-keys, fat, etc., matched up with part name in the partition table.
path/to/out/ # the output directory of the image, merged or otherwise.

(--partition just as a potential mechanism to specify the data necessary to build a merged image. The flag format is illustrative and not a serious proposal)

With #727 reading env and config for the missing flags, it could even be simplified to espflash flash and espflash save-image [--merged] path/to/out.

My understanding is that the equivalent with separate commands would look like this. Please correct me if you have different ideas.

espflash save-bootloader \
--chip chip \
--sign0 path/to/key0 --sign1 path/to/key1 --sign2 path/to/key2 \
--encrypt path/to/encryption-key
path/to/out/

espflash save-partition-table \
--chip chip \
--partition-table path/to/table --partition-table-offset 0x1234 \
--sign0 path/to/key0 --sign1 path/to/key1 --sign2 path/to/key2 \
--encrypt path/to/encryption-key
path/to/out/

espflash save-app-image \
--chip chip \
--offset 0x1234 \ # or --partition-table?
--sign0 path/to/key0 --sign1 path/to/key1 --sign2 path/to/key2 \
--encrypt path/to/encryption-key
path/to/out/

And without signing or encryption:

espflash save-bootloader \
--chip chip \
path/to/out/

espflash save-partition-table \
--chip chip \
--partition-table path/to/table --partition-table-offset 0x1234 \
path/to/out/

espflash save-app-image \
--chip chip \
--offset 0x1234 \ # or --partition-table?
path/to/out/

[SergioGasquez]

Hi! Sorry for the late reply! And thanks for the discussion, I will try to catch up/do some investigation and provide some input.

it cannot patch the built-in bootloader to support a different partition table offset than the default 0x8000 (@SergioGasquez right?).

It cant at the moment, but if we know where this happens in the bootloader we may be able to patch it.

Is there a chat room or new issue open for this? I still need to upload a lot of changes / bug reports / feature requests for esp-idf-[sys, svc] and if espflash v4 is related to those then maybe I can at least offload my ideas there. I think at this point I may have one of the most complex esp-idf projects so I've hit a lot of rough areas (which is expected).

As Ivan mentioned, most of the things are discussed in the Matrix chat or in the community meetings. We have a v4 gh milestone with the current issues we want to tackle for [email protected] and we are already starting to work towards it, so we can already merge breaking changes, like this one.

I still cant decide if we want to go the 3 commands or the 1 command road, to be honest, but we want to make sure this is also compatible with esp-hal, where we dont have sdkconfig.

[ivmarkov]

Hi! Sorry for the late reply! And thanks for the discussion, I will try to catch up/do some investigation and provide some input.

it cannot patch the built-in bootloader to support a different partition table offset than the default 0x8000 (@SergioGasquez right?).

It cant at the moment, but if we know where this happens in the bootloader we may be able to patch it.

I think this is pretty much impossible, as the partition offset is just a C #define and as such is hard-coded in gazillion of places in both the bootloader, as well as the app image. There is just no single, predictable place in these where this offset is recorded, that you can patch "post-factum".

Is there a chat room or new issue open for this? I still need to upload a lot of changes / bug reports / feature requests for esp-idf-[sys, svc] and if espflash v4 is related to those then maybe I can at least offload my ideas there. I think at this point I may have one of the most complex esp-idf projects so I've hit a lot of rough areas (which is expected).

As Ivan mentioned, most of the things are discussed in the Matrix chat or in the community meetings. We have a v4 gh milestone with the current issues we want to tackle for [email protected] and we are already starting to work towards it, so we can already merge breaking changes, like this one.

I still cant decide if we want to go the 3 commands or the 1 command road, to be honest, but we want to make sure this is also compatible with esp-hal, where we dont have sdkconfig.

I think the topic is deep enough that I'll bring it for the next ESP-RS mtg/discussion on Feb 13. Will try to do a short summary writeup on all issues, not just this one (the list keeps piling up a bit).

[ivmarkov]

@chris-subtlebytes If you can attend the next mtg I would really appreciate that. This stuff is so complex, that we need every person with experience in that we can get. :)

Also, I haven't addressed your last comment yet because I think a live meeting might actually be the easier way to resolve these issues, as I think they are also getting a bit stylistic or philosophical (as in whether we should follow the command layout / overall approach of the tried-and-tested esptool, or whether espflash should risk it a bit and invent something a bit on its own).

[SergioGasquez]

Just for reference, the meeting we are talking about is esp-rs/rust#251, happening 13th of February 2025 - 5 pm CET/GMT+1, does that work for you @chris-subtlebytes? If it doesnt work for you we should probably come up with another way to properly discuss this

[chris-subtlebytes]

I can do that

[SergioGasquez]

Hi @ivmarkov! During the meeting we spoke about having two commands:

• save-image which will have --merge by default
• save-firmware which will only include the app in the binary
  We didn't though of useful scenarios for save-bootloader and save-partition-table and we already have partition-table subcommand which converts partition tables from csv to bin.

Should you have any input, it will be more than welcome!

[ivmarkov]

@SergioGasquez

Hi @ivmarkov! During the meeting we spoke about having two commands:

• save-image which will have --merge by default
• save-firmware which will only include the app in the binary

We really have to look at these two also in light of potential future Secure Boot V2 and pre-encryption support in espflash. Read on.

We didn't though of useful scenarios for save-bootloader and save-partition-table and we already have partition-table subcommand which converts partition tables from csv to bin.

For save-bootloader - we might actually need it, read on.

As per above, I think additionally we have to think forward a bit how the above two commands will play with Secure Boot V2 and Flash Ecryption (where encryption is done the "pre-encrypt" way):

1. Secure Boot V2

• At a minimum, both save-image and save-firmware should support Secure Padding via an (optional?) --pad parameter.
• Will signing of the firmware image be supported with a separate command - say - sign-firmware, which expects an already generated .bin file (the esptool approach) or will it be done with an additional parameter (say, --sign`)?

If espflash supports signing of firmware images, then it should support signing (and padding) "a" bootloader too. Also options here:

• save-bootloader exists and pads and signs either the internal bootloader (but then it has to be pre-compiled with Secure Boot V2 caps enabled and possibly with flash encryption caps enabled??)
• or there is a separate command, sign-bootloader that pads the bootloader and then signs it

2. Flash Pre-encryption

Ditto here: a separate command (esptool-like encrypt) that takes a (possibly signed) bin (and an image offset, as the encryption is position independent) or somehow part of save-whatever.

Philosophical

The long commenting thread in this PR is exactly because with @chris-subtlebytes we are a bit diverging as to how save-firmware and save-image would work when the user wants to also sign or pre-encrypt or both the app image and the bootloader. So this has to be resolved. Either separate signing and encryption commands, or "save-firmware" and "save-image" with signing and encryption superpowers.

BUT then you need a "raw" sign and encrypt commands still so that you could

• sign and encrypt an externally-supplied bootloader,
• encrypt a possibly externally-supplied binary partition table,
• encrypt an NVS "keys" partition
• encrypt an "empty" partition containing "0xff" (like the coredump one and not only)

... and you probably see how we start turning in circles here (from my POV) in that having save-blahblah with superpowers would not cover all use cases where you want ot sign and encrypt "stuff" so if we have generic "sign" and "encrypt" commands, then why should save-blahblah be doing signing and encryption in the first place?

[ivmarkov]

@SergioGasquez
Oh, one more thing!

save-image --merge is also not perfect!

Having a single flash "merged" binary for bootloader + partition table + app image (and bootloader and app-image -pre-signed already!) sounds good BUT in practice you also want to add to the "merged thing":

• The NVS "keys" partition (which must be encrypted)
• The "NVS" partition (NOT encrypted)
• The coredump empty partition (encrypted 0xff)
• ... and so on

See where I'm going is... how would I even pass these things to save-image --merge? And then because part of the stuff needs to be encrypted while the other - not, and also because some things need to be signed (and padded before that) while others - not - save-image needs to know about encryption and signing and thus becomes a bit "super-powerful"... we might or might not want that...

[SergioGasquez]

Again, sorry for the late reply and thanks for all the input! I guess this would need some discussion on the next community meeting (yet to be scheduled), but looks like the most "safe" bet is to have save-image --merge, save-firmware, save-bootloader and maybe some tweaks to partition-table subcomand

[chris-subtlebytes]

What's the ideal end goal here?

In my ideal world, aside from some initial setup like generating keys, I want to have a tight single-command iteration loop. Right now we sorta have this with cargo espflash flash. I think that, assuming in the future we have extra partition data, padding, signing, and encryption, it should all be possible with that one command. The motivations are primarily having a faster iteration loop and fewer duplicate flags to keep track of between commands.

I did suggest a way we could pass partition data to the with a new repeated flag --partition partition_name_or_address:partition_path. As for whether to encrypt each partition, espflash already reads the partition table which has the flag for whether to encrypt or not. The way esptool merge_bin works is by having repeated pairs of address and image file: esptool merge_bin <address> <filename> [<address> <filename> ...].

Now aside from flashing, I see a few more use cases (please suggest more if I forgot any):

• Save full image for:
  • Flashing
• Save bootloader for:
  • Flashing the bootloader only
  • Merging into a full image
  • Manually signing and pre-encrypting
  • esptool image_info
  • Updating header and SHA
  • ???
  • Merge into a full image
• Save app image for:
  • OTA
  • Flashing the app image only
  • Manually signing and pre-encrypting
  • esptool image_info
  • Updating header and SHA
  • ???
  • Merge into a full image

---

I'm going to try to see what both schemas look like with all of their applicable flags.

espflash save-part-table <FILE> (already implemented today as espflash partition-table?)

• --output <FILE>
espflash save-bootloader <FILE>
• --pad If the user doesn't pad before signing, it could be confusing. Maybe it's fine because iirc espidf will pad for us if the secure boot sdkconfig is enabled.
  espflash save-appimage <FILE> or save-factory?
• --pad
espflash sign-data <FILE> (bootloader and app image have the same format)
• See espsecure sign_data
• User needs to ensure that the bootloader and app image is padded
• --keyfile <SIGNING_KEY_FILE>
• --append-signatures append signatures to an already signed image
• --signature precalculated signature
• --output
espflash encrypt-flash-data
• See espsecure encrypt_flash_data
• --keyfile
• --address
• --output
espflash merge-bin
• See esptool merge_bin
• repeated
• all the other flags for setting header values

At this point it feels like we would be reimplementing esptool and espsecure with limited real-world benefit because a user would have to run though each command to pad, sign, encrypt, and merge each partition for every development iteration loop. The main advantage I see with espflash here is the integration with cargo so imo we should take a more streamlined approach.

Here's the development loop with the above assuming that the developer is also adjusting sdkconfig values that change the bootloader:

1. cargo espflash save-bootloader

• assuming it's already padded by espidf build
• getting bootloader path from cargo integration

2. cargo espflash save-factory

• assuming it's already padded by espidf build
• getting app image path from cargo integration

3. cargo partition-table --to-binary --output myout/myproject-parttable.bin
4. cargo espflash sign-data target/*/*/bootloader.bin --keyfile mykeyfile --output myout/myproject-bootloader.bin

• espflash can't know if we want to sign the bootloader or app image so we need to give the target path unless we make sign-bootloader and sign-factory commands. Note also that they would make sense for cargo espflash but non-cargo espflash would still require the path which defeats the purpose of having it be opinionated. Also, someone familiar to the signing process might get confused as to why there's a command to sign the bootloader vs app image when it's implemented as a single function in espsecure sign_data.

5. cargo espflash sign-data target/*/*/myproject --keyfile mykeyfile --output myout/myproject-app.bin
6. cargo espflash encrypt-flash-data --address 0x0 myout/myproject-bootloader.bin --output myout/myproject-bootloader.bin.encrypted
7. cargo espflash encrypt-flash-data --address 0x111 myout/myproject-parttable.bin --output myout/myproject-parttable.bin.encrypted
8. cargo espflash encrypt-flash-data --address 0x222 myout/myproject-app.bin --output myout/myproject-app.bin.encrypted
9. cargo espflash encrypt-flash-data --address 0x333 myproject-nvskeys.bin --output myout/myproject-nvskeys.bin.encrypted
10. cargo espflash encrypt-flash-data --address 0x444 myproject-fat.bin --output myout/myproject-fat.bin.encrypted
11. espflash merge-bin 0x0 myout/myproject-bootloader.bin.encrypted 0x111 myout/myproject-parttable.bin.encrypted 0x222 myout/myproject-app.bin.encrypted 0x333 myproject-nvskeys.bin.encrypted 0x444 myproject-fat.bin 0x555 myproject-nvs.bin --output myout/myproject-image

• I suppose this could be simplified if we pass the partition table instead, and the address+filepath.

12. [cargo?] espflash flash myout/myproject-image

• can't use cargo espflash flash without the image path because it won't be able to merge properly unless we combine the merge step into flash but then how would it know to use the custom signed and encrypted parts?

Please share if you have any suggestions on how to simplify the above or if I'm misunderstanding what your ideas are.

I think the same process could be done like so:
cargo espflash flash

• partition table collected from espconfig.toml
• whether or not to sign collected from espconfig.toml, or --sign <KEYFILE>
• whether or not to encrypt collected from espconfig.toml or --encrypt <KEYFILE>
  • conditionally signing partitions can be looked up from the partition table to avoid signing nvs
• additional parts to merge specified either by --partition partition_name_or_address:partition_path, repeated <ADDRESS> <FILE>, or in espconfig.toml. In fact, I think the majority of use cases will have static or well-known paths to the partition content so putting it in espconfig.toml could work really well.

The same would apply to cargo espflash save-image [--as-parts] (I think --as-parts is a better name than --no-merged if we want merged as default).

---

Regarding the discussion about the partition table offset, I don't think we need to patch anything and as you've seen it's hard/impossible. The best approach is to set CONFIG_PARTITION_TABLE_OFFSET=0x1234 in sdkconfig.defaults and possibly read that value from cargo espflash otherwise the user will have to specify it to espflash manually for encrypting and merging.

---

BUT then you need a "raw" sign and encrypt commands still so that you could

sign and encrypt an externally-supplied bootloader,

espflash save-image already has a functional --bootloader <FILE> flag that does this. I agree it starts to break down if the external bootloader is already signed. How often will it already be signed though, and can we pass this rare use case back to esptool? And I sure hope a signed and encrypted bootloader is exceedingly rare. Lastly, we could just have signing and encryption commands to avoid using the esptool.

encrypt a possibly externally-supplied binary partition table,

What's to stop the user from converting the binary table back to csv with espflash partition-table --to-csv my-external-parttable.bin --output my-external-parttable.csv then passing that to espflash save-image --partition-table my-external-parttable.csv ...?

encrypt an NVS "keys" partition

Lookup the encrypted flag from the supplied (either by cargo espflash or by flag) partition table for each partition.

encrypt an "empty" partition containing "0xff" (like the coredump one and not only)

Specify the address/partname and partition data to espflash via --partition partition_name_or_address:partition_path or use esptool merge_bin semantics with repeated <ADDRESS> <FILE>. Then lookup the encrypted flag in the part table.

... and you probably see how we start turning in circles here (from my POV) in that having save-blahblah with superpowers would not cover all use cases where you want ot sign and encrypt "stuff" so if we have generic "sign" and "encrypt" commands, then why should save-blahblah be doing signing and encryption in the first place?

I'm not sure I follow 100% here. Can you specify which scenarios you expect to be problematic? My thoughts are that users with odd requirements can use more awkward tools. We could either tell them to use the existing esptool/espsecure or expose sign and encrypt as commands. I don't see how having both a superpower save-image and separate sign/encrypt commands would really interfere with each other or get all that messy.

---

If you don't want to wait until the next community meeting, I'm good with scheduling something earlier.