firmware: fw_base: add an acquire fence after the _boot_status wait loop

[zeldovich]

Summary

This bug was found in the process of formally verifying the correctness of OpenSBI on top of the Sail RISC-V semantics using Lean.

Non-boot harts leave _wait_for_boot_hart on a plain load of _boot_status and immediately read data the boot hart produced before publishing. Nothing on the waiting side orders that load before the later loads. Under RVWMO a waiter can observe the "done" status and still read pre-relocation values, including the unrelocated platform.hart_index2id pointer, and fault through it before mtvec is set. One fence r, rw after the loop closes it.

An execution that triggers it

Setup: fw_dynamic loaded at an address different from its link address, so the boot hart runs the self-relocation pass; eight harts; RVWMO hardware (or any model that lets a plain load be satisfied early, e.g. the Sail model used to find this).

1. Hart 1 reaches _wait_for_boot_hart while hart 0 is still relocating. Hart 1's loop body is ld t1, _boot_status; div; div; div; bne. Its load of _boot_status returns the pre-done value, it loops.
2. Hart 1's core speculatively issues the next iteration's loads early. RVWMO allows a later load to be performed before an earlier load to a different address when nothing orders them. In particular the loads that follow the loop exit, lwu platform.hart_count, ld platform.hart_index2id, may be satisfied now, from memory that hart 0 has not yet rewritten: platform.hart_index2id still holds its link-time value 0x449f8.
3. Hart 0 finishes: it fixes up platform.hart_index2id to 0x800449f8 via the R_RISCV_RELATIVE pass, executes fence rw, rw, stores _boot_status = BOOT_STATUS_BOOT_HART_DONE.
4. Hart 1's loop load now returns DONE, the bne falls through. The already-performed load of platform.hart_index2id is not re-executed: s9 = 0x449f8.
5. Hart 1 executes lwu a5, 0(s9): a load from 0x449f8, which is outside every PMA region on the virt platform. The access faults.
6. Hart 1 has not yet executed _start_warm's csrw mtvec; mtvec holds its reset value. The trap vectors to an undefined address and the hart is lost. The system boots with 7 harts, or hangs in smp_init waiting for it.

The release on the boot hart (fence rw, rw before the status store) is correct; the bug is the missing acquire on the waiting side. The fence added here sits after the loop and runs once per hart. A 2019 attempt placed a fence rw, rw inside the loop before the load (05602e2) and was reverted (69d794c) after a warm-reset regression on the HiFive Unleashed whose root cause was never found; that fence did not provide the needed ordering anyway.

Verification

Proved: with the fence, the non-boot harts' path from _start to _start_warm satisfies its contract against the Sail RISC-V model under an RVWMO-faithful memory model. Without the fence the stale read is an allowed execution and the contract is unprovable.