instruction abort with FAR: 0x2000000 usually inside mmu_setup_secondary

[svenpeter42]

What we know

• The exception is triggered when the secondary cores are brought up from inside a m1n1 guest
• Most of the time it happens on the first additional core that is brought up but occasionally it only appears on the second or even third additional core.
• The bug is somewhere in the host m1n1
• Usually FAR is 0x2000000 and the exception happens when returning from mmu_secondary_setup or possibly when returning to mmu_secondary_setup and comes from EL2 (i.e. host m1n1)
• Moving code around inside mmu_secondary_setup can make the exception disappear, using a different compiler as well
• Bisecting hasn't been helpful for me

See also #462

[jannau]

The issue is reproducible when using clang from Fedora 42. It is breaking on the return from mmu_secondary_setup(). An added __asm__ volatile("ldr x12, [sp, #8]" : : : "r12"); at the end of the function results in 0x2000000 replicated in x12.
What I find suspicious is that the frame (x29) is off from the reported SP:

TTY> x28-x30: 0000000000000000 00000008028d7884 0000000002000000
TTY> SP:       0x804437ed0

0x8028d7884 is inside the m1n1 binary so most likely dummy_stack. I do not understand how that's possible. The call should come from smp_secondary_entry() and use the dynamically allocated per CPU stack.

Other ways to mask the issue:

• increasing DUMMY_STACK_SIZE to 0x2000 or 0x4000
• adding __asm__ volatile("ldr x12, [sp, #8]" : : : "r12"), sysop("isb") or sysop("dmb sy") after

    if (supports_pan())
        msr(PAN, 0);
  

[svenpeter42]

Can confirm that these changes also mask the issue for me

[svenpeter42]

@jannau have you checked what 0x8028d7884 actually points to in your binary? I just did with my local build and x29 points to the instruction just after the branch to deep_wfi here

   2d11c:	54ffff40 	b.eq	2d104 <smp_secondary_entry+0x9c>  // b.none
   2d120:	94001945 	bl	33634 <supports_arch_retention>
   2d124:	36000060 	tbz	w0, #0, 2d130 <smp_secondary_entry+0xc8>
   2d128:	97ff78bb 	bl	b414 <deep_wfi>
   2d12c:	14000002 	b	2d134 <smp_secondary_entry+0xcc>  // <--- x29 points here
   2d130:	d503207f 	wfi

[svenpeter42]

Okay, so the cores are started long before hv_start_secondary is called so they're all idling in smp_secondary_entry -> deep_wfi probably. If >=2 cores for some reason shared the same stack and one core was just in mmu_secondary_setup while the other entered deep_wfi the other core might just overwrite the first core's LR on the stack.

this also explains 0x2000000 which is the original value of SYS_IMP_APL_CYC_OVRD stored on the stack inside deep_wfi.

I don't see any evidence of invalid SP when they are brought up though but this makes too much sense. I'm probably just messing up because it's late here.

[svenpeter42]

nope, the stacks seem to be correct:

TTY> Starting secondary CPUs...
TTY> Starting CPU 1 (0:0:1)... Stack for CPU: 0x803900000
TTY>   Started. stack = 0x8038fffd0
TTY> Starting CPU 2 (0:0:2)... Stack for CPU: 0x803914000
TTY>   Started. stack = 0x803913fd0
TTY> Starting CPU 3 (0:0:3)... Stack for CPU: 0x803928000
TTY>   Started. stack = 0x803927fd0
TTY> Starting CPU 4 (0:1:0)... Stack for CPU: 0x80393c000
TTY>   Started. stack = 0x80393bfd0
TTY> Starting CPU 5 (0:1:1)... Stack for CPU: 0x803950000
TTY>   Started. stack = 0x80394ffd0
TTY> Starting CPU 6 (0:1:2)... Stack for CPU: 0x803964000
TTY>   Started. stack = 0x803963fd0
TTY> Starting CPU 7 (0:1:3)... Stack for CPU: 0x803978000
TTY>   Started. stack = 0x803977fd0

but then it still dies

TTY> HV: Initializing secondary 2
TTY> Exception: SYNC
TTY> Exception taken from EL2h
TTY> Running in EL2
TTY> MPIDR: 0x80000002
TTY> Registers: (@0x803913dd0)
TTY>   x0-x3: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
TTY>   x4-x7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY>  x8-x11: 0000000030901085 0000000000000001 a005a500f005f00f f005a500f00ff00f
TTY> x12-x15: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY> x16-x19: 0000000000000000 0000000000000000 00000000fffffff5 00000008022402a0
TTY> x20-x23: 000000080223800a 0000000802240000 0000000000000001 0000000000000000
TTY> x24-x27: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY> x28-x30: 0000000000000000 00000008021a9134 0000000002000000
TTY> PC:       0x2000000 (rel: 0xfffffff7ffe80000)
TTY> SP:       0x803913ed0
TTY> SPSR:     0x10c9
TTY> FAR:      0x2000000
TTY> ESR:      0x86000006 (instruction abort (current))
TTY> L2C_ERR_STS: 0x11000ffc00000000
TTY> L2C_ERR_ADR: 0x0
TTY> L2C_ERR_INF: 0x0
TTY> E_LSU_ERR_STS: 0x0
TTY> E_FED_ERR_STS: 0x0
TTY> E_MMU_ERR_STS: 0x0
TTY> Unhandled exception, rebooting...

[svenpeter42]

I removed the wfi instruction from deep_wfi and replaced the beginning that usually
reads SYS_IMP_APL_CYC_OVRD to x0 and then updates it using x1 to instead just load
0xde0 | TPIDR_EL2 to x0, i.e.

mrs x0, TPIDR_EL2
add x0, x0, #0xde0

I wanted to confirm that the stack is indeed overwritten by another core.
It looks like the stack is corrupted by the the same core:

TTY> HV: Initializing secondary 2
TTY> Exception: SYNC
TTY> Exception taken from EL2h
TTY> Running in EL2
TTY> MPIDR: 0x80000002
TTY> Registers: (@0x803f37dc0)
TTY>   x0-x3: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
TTY>   x4-x7: 0000000000000010 0000000000000000 00000000ffffffff 0000000000000000
TTY>  x8-x11: 0000000030901085 0000000000000001 a005a500f005f00f f005a500f00ff00f
TTY> x12-x15: 0000000803f37c85 000000000000002c 000000000000001e 000000000000001e
TTY> x16-x19: 00000000fffffffe 0000000000000000 00000000fffffff5 00000008028652a8
TTY> x20-x23: 0000000000000002 000000080285c00a 0000000802857e58 0000000802865000
TTY> x24-x27: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
TTY> x28-x30: 0000000000000000 00000008027cd1e0 0000000000000de2
TTY> PC:       0xde2 (rel: 0xfffffff7fd85cde2)
TTY> SP:       0x803f37ec0
TTY> SPSR:     0x10c9
TTY> FAR:      0xde2
TTY> ESR:      0x8a000000 (pc misaligned)
TTY> L2C_ERR_STS: 0x11000ffc00000000
TTY> L2C_ERR_ADR: 0x0
TTY> L2C_ERR_INF: 0x0
TTY> E_LSU_ERR_STS: 0x0
TTY> E_FED_ERR_STS: 0x0
TTY> E_MMU_ERR_STS: 0x0