SFTP-OTA: Does it recover from a bad update?

[jubeormk1]

Rationale

As pointed out in Radically Open Security Audit #7, we should guarantee that a bad OTA would not brick the system.

This has been partially addressed in the sftp-ota feature for Espressif devices, but I am afraid that I am missing the last step to guarantee it. Enable the Rollback mechanism! We need to check the current state and fix it:

Tasks

• Apply a faulty OTA update and assess what is the result: Rollback or brick?
• If bricked add configuration to enable Espressif Rollback mechanism
• Document the findings and modifications

[brainstorm]

AFAICT this requires compiling a custom bootloader (via esp-idf) and enabling the CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE? I cannot find any similar variable in https://github.com/esp-rs/esp-hal/tree/main/esp-bootloader-esp-idf, so that probably means we've to create our own BLOBs for that to work :-S

Not against it, just noting that it might get tricky, although mostly on the CD (Continuous Deployment) side, when and if we provide pre-compiled firmware images users can flash directly to their devices to get started.

[jubeormk1]

That is a fair point. I will do some exploration work to see what would it require.

[jubeormk1]

Finally I got to run a test for this point. I run an sftp-ota on the current main HEAD (commit d8116f4)

TL;DR:

The test was satisfactory for the Espressif platform. Here is a summary of the results:

• If the packed ota has a bad magic number, esp32c6 will reject the partition and rolls back
• If is is a different esp32 target (esp32c3) it will issue a mismatched chip error and roll back
• If the application does not validate the ota it will still run after ota and reboots (bootloader change required to modify this).

If the users pack another application or non ota application, they can prevent the sftp-ota mechanism from work... but it is their own generic hardware, that they access with their private key. Maybe that is exactly what they want!

Tests

Correct OTA

Pack an binary and the pack it again (the repacked binary has the extra metadata and this makes it a bad ota):

cargo run packer -- ssh-stamp.bin
cargo run packer -- ssh-stamp.ota
mv ssh-stamp.ota{,.repacked}         # <- Produces an incorrect ota
cargo run packer -- ssh-stamp.bin  # <- Produces a correct ota for validation

Clean flash and run a sftp-ota enable application:

espflash erase-flash -p ~/devs/prog_esp32c6
ESP_LOG=WARN,ssh_stamp=debug cargo run-esp32c6 --features sftp-ota -- -p ~/devs/prog_esp32c6

In another terminal, I connect to the wifi hotspot and put the correct ota update.

echo "put ssh-stamp.ota" | sftp ssh-stamp-pkey

After the fact I observe the restart of the esp32c6 in the debugger terminal:

DEBUG - ServEvent::SessionSubsystem                                                                                                                                                                                                                                       
DEBUG - We got SFTP subsystem                                                                                                                                                                                                                                             
DEBUG - Checking bridge session type                                                                                                                                                                                                                                      
DEBUG - Handling SFTP session                                                                                                                                                                                                                                             
ESP-ROM:esp32c6-20220919                                                                                                                                                                                                                                                  
Build:Sep 19 2022                                                                                                                                                                                                                                                         
rst:0x3 (LP_SW_HPSYS),boot:0xc (SPI_FAST_FLASH_BOOT)                                                                                                                                                                                                                      
Saved PC:0x4001974a                                                                                                                                                                                                                                                       
SPIWP:0xee                                                                                                                                                                                                                                                                
mode:DIO, clock div:2                                                                                                                                                                                                                                                     
load:0x40875730,len:0x175c                                                                                                                                                                                                                                                
load:0x4086b910,len:0xec8                                                                                                                                                                                                                                                 
load:0x4086e610,len:0x31c4                                                                                                                                                                                                                                                
entry 0x4086b91a                                                                                                                                                                                                                                                          
I (22) boot: ESP-IDF v5.5.1-838-gd66ebb86d2e 2nd stage bootloader                                                                                                                                                                                                         
I (23) boot: compile time Nov 27 2025 09:46:10                                                                                                                                                                                                                            
I (24) boot: chip revision: v0.1                                                                                                                                                                                                                                          
I (24) boot: efuse block revision: v0.3                                                                                                                                                                                                                                   
I (28) boot.esp32c6: SPI Speed      : 80MHz                                                                                                                                                                                                                               
I (31) boot.esp32c6: SPI Mode       : DIO                                                                                                                                                                                                                                 
I (35) boot.esp32c6: SPI Flash Size : 8MB                                                                                                                                                                                                                                 
I (39) boot: Enabling RNG early entropy source...                                                                                                                                                                                                                         
I (43) boot: Partition Table:                                                                                                                                                                                                                                             
I (46) boot: ## Label            Usage          Type ST Offset   Length                                                                                                                                                                                                   
I (52) boot:  0 app_config       WiFi data        01 02 00009000 00002000                                                                                                                                                                                                 
I (59) boot:  1 otadata          OTA data         01 00 0000d000 00002000                                                                                                                                                                                                 
I (65) boot:  2 phy_init         RF data          01 01 0000f000 00001000                                                                                                                                                                                                 
I (72) boot:  3 ota_0            OTA app          00 10 00010000 001e0000                                                                                                                                                                                                 
I (78) boot:  4 ota_1            OTA app          00 11 001f0000 001e0000                                                                                                                                                                                                 
I (85) boot:  5 extra_data       WiFi data        01 02 003d0000 00010000                                                                                                                                                                                                 
I (92) boot: End of partition table                                                                                                                                                                                                                                       
I (95) esp_image: segment 0: paddr=001f0020 vaddr=42000020 size=1a6b0h (108208) map                                                                                                                                                                                       
I (123) esp_image: segment 1: paddr=0020a6d8 vaddr=40800000 size=00014h (    20) load                                                                                                                                                                                     
I (124) esp_image: segment 2: paddr=0020a6f4 vaddr=4201a6f4 size=e44d4h (935124) map                                                                                                                                                                                      
I (305) esp_image: segment 3: paddr=002eebd0 vaddr=40800014 size=10278h ( 66168) load                                                                                                                                                                                     
I (320) esp_image: segment 4: paddr=002fee50 vaddr=40810290 size=0270ch (  9996) load                                                                                                                                                                                     
I (323) esp_image: segment 5: paddr=00301564 vaddr=408129a0 size=001e0h (   480) load                                                                                                                                                                                     
I (329) boot: Loaded app from partition at offset 0x1f0000                    <------------------------ Booting from partition ota_1                                                      
I (329) boot: Disabling RNG early entropy source...                         

Incorrect OTA (repacked)

Repeating the process with the repacked ota (clean, flash, sftp...) rejects the OTA on the base of a bad magic number:

DEBUG - ServEvent::SessionSubsystem                                                                                                                                                                                                                                       
DEBUG - We got SFTP subsystem                                                                                                                                                                                                                                             
DEBUG - Checking bridge session type                                                                                                                                                                                                                                      
DEBUG - Handling SFTP session                                                                                                                                                                                                                                             
ESP-ROM:esp32c6-20220919                                                                                                             
Build:Sep 19 2022                                                                                                                    
rst:0x3 (LP_SW_HPSYS),boot:0xc (SPI_FAST_FLASH_BOOT)                                                                                 
Saved PC:0x4001974a                                                                                                                  
SPIWP:0xee                                                                                                                                                                                                                                                                
mode:DIO, clock div:2                                                                                                                                                                                                                                                     
load:0x40875730,len:0x175c                                        
load:0x4086b910,len:0xec8                                                                                                            
load:0x4086e610,len:0x31c4                                                                                                           
entry 0x4086b91a                                                                                                                                                                                                                                                          
I (22) boot: ESP-IDF v5.5.1-838-gd66ebb86d2e 2nd stage bootloader                                                                    
I (23) boot: compile time Nov 27 2025 09:46:10                                                                                                                                                                                                                            
I (24) boot: chip revision: v0.1                                                                                                                                                                                                                                          
I (24) boot: efuse block revision: v0.3                                                                                              
I (28) boot.esp32c6: SPI Speed      : 80MHz                                                                                          
I (31) boot.esp32c6: SPI Mode       : DIO                                                                                            
I (35) boot.esp32c6: SPI Flash Size : 8MB                                                                                            
I (39) boot: Enabling RNG early entropy source...                                                                                    
I (43) boot: Partition Table:                                     
I (46) boot: ## Label            Usage          Type ST Offset   Length                                                                                                                                                                                                   
I (52) boot:  0 app_config       WiFi data        01 02 00009000 00002000                                                                                                                                                                                                 
I (59) boot:  1 otadata          OTA data         01 00 0000d000 00002000                                                                                                                                                                                                 
I (65) boot:  2 phy_init         RF data          01 01 0000f000 00001000                                                                                                                                                                                                 
I (72) boot:  3 ota_0            OTA app          00 10 00010000 001e0000                                                                                                                                                                                                 
I (78) boot:  4 ota_1            OTA app          00 11 001f0000 001e0000                                                                                                                                                                                                 
I (85) boot:  5 extra_data       WiFi data        01 02 003d0000 00010000                                                                                                                                                                                                 
I (92) boot: End of partition table                                                                                                  
E (95) esp_image: image at 0x1f0000 has invalid magic byte (nothing flashed here?)   <------------ Invalid magic byte                                                                                                                                                                                     
E (102) boot: OTA app partition slot 1 is not bootable                                                                               
I (107) esp_image: segment 0: paddr=00010020 vaddr=42000020 size=1d610h (120336) map                                                                                                                                                                                      
I (137) esp_image: segment 1: paddr=0002d638 vaddr=40800000 size=00014h (    20) load                                                                                                                                                                                     
I (138) esp_image: segment 2: paddr=0002d654 vaddr=4201d654 size=fcd60h (1035616) map                                                                                                                                                                                     
I (339) esp_image: segment 3: paddr=0012a3bc vaddr=40800014 size=10270h ( 66160) load                                                                                                                                                                                     
I (354) esp_image: segment 4: paddr=0013a634 vaddr=40810288 size=0296ch ( 10604) load                                                                                                                                                                                     
I (357) esp_image: segment 5: paddr=0013cfa8 vaddr=40812bf8 size=001e0h (   480) load                                                                                                                                                                                     
I (362) boot: Loaded app from partition at offset 0x10000                                       <----------------- Booting from ota_0                                     
I (363) boot: Disabling RNG early entropy source...                                                   

Incorrect OTA (Valid ota wrong target)

I repeated the process using cargo build-esp32c3 to obtain a different target. Will it have a valid magic number? Mismatched chip

DEBUG - ServEvent::SessionSubsystem                                                                                                   │Connected to ssh-stamp-pkey.
DEBUG - We got SFTP subsystem                                                                                                         │sftp> put ssh-stamp.esp32c3.ota
DEBUG - Checking bridge session type                                                                                                  │Uploading ssh-stamp.esp32c3.ota to /ssh-stamp.esp32c3.ota
DEBUG - Handling SFTP session                                                                                                         │ssh-stamp.esp32c3.ota                                                                              0%   32KB 445.0KB/s   00:30 ETA
ESP-ROM:esp32c6-20220919                                                                                                              │write remote "/ssh-stamp.esp32c3.ota": Failure
Build:Sep 19 2022                                                                                                                     │Interrupt  
rst:0x3 (LP_SW_HPSYS),boot:0xc (SPI_FAST_FLASH_BOOT)                                                                                  │Interrupt  
Saved PC:0x4001974a                                                                                                                   │Interrupt  
SPIWP:0xee                                                                                                                            │Interrupt  
mode:DIO, clock div:2                                                                                                                 │worker@hil-ssh-stamp:~/ssh-stamp$ target/riscv32imc-unknown-none-elf/release/ssh-stamp                  
load:0x40875730,len:0x175c                                                                                                            │-bash: target/riscv32imc-unknown-none-elf/release/ssh-stamp: cannot execute binary file: Exec format error
load:0x4086b910,len:0xec8                                                                                                             │worker@hil-ssh-stamp:~/ssh-stamp$ vim ota/README.md 
load:0x4086e610,len:0x31c4                                                                                                            │worker@hil-ssh-stamp:~/ssh-stamp$ espflash save-image --chip=esp32c3 target/riscv32imac-unknown-none-elf/release/ssh-stamp ssh-stam
entry 0x4086b91a                                                                                                                      │p.esp32c3.bin
I (22) boot: ESP-IDF v5.5.1-838-gd66ebb86d2e 2nd stage bootloader                                                                     │[2026-05-24T12:03:17Z INFO ] __ A new version of espflash is available: v4.4.0
I (23) boot: compile time Nov 27 2025 09:46:10                                                                                        │Error:   x Failed to open image target/riscv32imac-unknown-none-elf/release/ssh-stamp
I (24) boot: chip revision: v0.1                                                                                                      │  `-> No such file or directory (os error 2)
I (24) boot: efuse block revision: v0.3                                                                                               │
I (28) boot.esp32c6: SPI Speed      : 80MHz                                                                                           │worker@hil-ssh-stamp:~/ssh-stamp$ espflash save-image --chip=esp32c3 target/riscv32imc-unknown-none-elf/ ssh-stamp.esp32c3.bin
I (31) boot.esp32c6: SPI Mode       : DIO                                                                                             │CACHEDIR.TAG  release/      
I (35) boot.esp32c6: SPI Flash Size : 8MB                                                                                             │worker@hil-ssh-stamp:~/ssh-stamp$ espflash save-image --chip=esp32c3 target/riscv32imc-unknown-none-elf/release/ ssh-stamp.esp32c3.
I (39) boot: Enabling RNG early entropy source...                                                                                     │bin
I (43) boot: Partition Table:                                                                                                         │.cargo-lock        build/             examples/          libssh_stamp.d     ssh-stamp          
I (46) boot: ## Label            Usage          Type ST Offset   Length                                                               │.fingerprint/      deps/              incremental/       libssh_stamp.rlib  ssh-stamp.d        
I (52) boot:  0 app_config       WiFi data        01 02 00009000 00002000                                                             │worker@hil-ssh-stamp:~/ssh-stamp$ espflash save-image --chip=esp32c3 target/riscv32imc-unknown-none-elf/release/ssh-stamp ssh-stamp
I (59) boot:  1 otadata          OTA data         01 00 0000d000 00002000                                                             │.esp32c3.bin
I (65) boot:  2 phy_init         RF data          01 01 0000f000 00001000                                                             │[2026-05-24T12:03:39Z INFO ] __ A new version of espflash is available: v4.4.0
I (72) boot:  3 ota_0            OTA app          00 10 00010000 001e0000                                                             │Chip type:         esp32c3
I (78) boot:  4 ota_1            OTA app          00 11 001f0000 001e0000                                                             │Merge:             false
I (85) boot:  5 extra_data       WiFi data        01 02 003d0000 00010000                                                             │Skip padding:      false
I (92) boot: End of partition table                                                                                                   │App/part. size:    878,816/4,128,768 bytes, 21.29%
E (95) boot_comm: mismatch chip ID, expected 13, found 5           # < -------- Mismatch chip                                                                   │[2026-05-24T12:03:39Z INFO ] Image successfully saved!
E (100) boot: OTA app partition slot 1 is not bootable                                                                                │worker@hil-ssh-stamp:~/ssh-stamp$ cargo ota-packer -- ssh-stamp.esp32c3 
I (105) esp_image: segment 0: paddr=00010020 vaddr=42000020 size=1d610h (120336) map                                                  │ssh-stamp.esp32c3      ssh-stamp.esp32c3.bin  ssh-stamp.esp32c3.ota  
I (135) esp_image: segment 1: paddr=0002d638 vaddr=40800000 size=00014h (    20) load                                                 │worker@hil-ssh-stamp:~/ssh-stamp$ cargo ota-packer -- ssh-stamp.esp32c3. 
I (136) esp_image: segment 2: paddr=0002d654 vaddr=4201d654 size=fcd60h (1035616) map                                                 │ssh-stamp.esp32c3.bin  ssh-stamp.esp32c3.ota  
I (337) esp_image: segment 3: paddr=0012a3bc vaddr=40800014 size=10270h ( 66160) load                                                 │worker@hil-ssh-stamp:~/ssh-stamp$ cargo ota-packer -- ssh-stamp.esp32c3.bin 
I (352) esp_image: segment 4: paddr=0013a634 vaddr=40810288 size=0296ch ( 10604) load                                                 │    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.06s
I (355) esp_image: segment 5: paddr=0013cfa8 vaddr=40812bf8 size=001e0h (   480) load                                                 │     Running `target/x86_64-unknown-linux-gnu/debug/ota-packer ssh-stamp.esp32c3.bin`
I (360) boot: Loaded app from partition at offset 0x10000                                                                             │Packing ssh-stamp.esp32c3.bin as OTA...
I (361) boot: Disabling RNG early entropy source...  

Good OTA but not partition validation

Ok, but what if the application does not validate the new partition (Not OTA application)? It does run from the new application and it will keep running it after reboots. To modify this behavior a custom base bootloader is required, enforcing partition validation. Doable but not there.