Kernel hang when trying to set TPIDRRO_EL0 register at EL1

[Abhishek-612]

Hi Team,

I am working on a fuzzing tool for the OPTEE kernel. I am trying to get obtain a coverage bitmap for all system calls made when executing a TA. However, I need to store the bitmap between system calls.

Following the implementation by Riscure, I want to store the base address of the bitmap in the TPIDRRO_EL0 register. Below is the reference code

static inline void __afl_set_ctx_ptr(struct afl_ctx* ptr) {
    asm volatile("msr tpidrro_el0, %0" :: "r" (ptr) : "memory");
}

However, when I build the kernel, the program fails to set any value (except 0x0) in the TPIDRRO_EL0 register, and kernel hangs.
Since the call is made from core/kernel/scall.c, the set instruction happens in EL1, which seems to be the minimum EL level at which this register can be written to. The same happens when trying to run any other TPIDR* register.

Can you please suggest how to resolve this?

Thanks

[jenswi-linaro]

What happens when it fails to write to TPIDRRO_EL0? Does the instruction cause an exception? Note that TPIDRRO_EL0 is used as a temporary register with CFG_CORE_UNMAP_CORE_AT_EL0 when transitioning to and from EL0.

[Abhishek-612]

With some debug messages, I figured that the program enters the function but does not proceed after the MSR instruction. The program is stuck indefinitely on that line.

Yes, for this implementation, I have set CFG_CORE_UNMAP_CORE_AT_EL0=n.

[jenswi-linaro]

So far, we don't know anything about the hardware you're running on. Can you try it in QEMU and connect with gdb?

[Abhishek-612]

Correction: MSR instruction seems to stored the value, but the program does not return to caller. (I tried putting debug messaged on the qemu console)

I am building in QEMU on Ubuntu, and I have tried this on two devices - an x86 Ubuntu machine, and a Mac M2 (inside an Ubuntu VM)

In core/include/kernel/afl.h

static inline void __afl_set_ctx_ptr(uint64_t ptr) {
    register uint64_t reg_value =  ptr;
    if (ptr != 0) {
        snprintf(debug_message, sizeof(debug_message), "__afl_set_ctx_ptr: TPIDRRO_EL0 before - %p\n",__afl_ctx_ptr());
        debug_printer(debug_message);

        __asm__ volatile("msr tpidrro_el0, %0" : : "r"(reg_value));

        snprintf(debug_message, sizeof(debug_message), "__afl_set_ctx_ptr: TPIDRRO_EL0 after- %p\n",__afl_ctx_ptr());
        debug_printer(debug_message);
    }

    return;
}

In core/arch/arm/include/kernel/arch_scall.h:

static inline void __afl_cov_trace_start(struct ts_session *sess) {
    assert(sess->afl_ctx != NULL);
    assert(__afl_ctx_ptr() == NULL);

    sess->afl_ctx->prev_loc = 0;

    DMSG("Integer as pointer: %p\n", (uintptr_t) addr_int);

    DMSG("TPIDRRO_EL0 curent value: %p\n", __afl_ctx_ptr());

    __afl_set_ctx_ptr(sess->afl_ctx);

	DMSG("Set CTX Complete");
	DMSG("Set CTX Pointer: %x", __afl_ctx_ptr());

    sess->afl_ctx->enabled = true;
}

Output
Normal World:

# tee_invoke_svc /tmp/filedg1cIz 
[   36.798576] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[   36.803716] rcu: 	0-...0: (0 ticks this GP) idle=028c/1/0x4000000000000000 softirq=482/482 fqs=2606
[   36.804962] 	(detected by 1, t=5252 jiffies, g=-227, q=4 ncpus=2)
[   36.805547] Task dump for CPU 0:
[   36.805810] task:tee_invoke_svc  state:R  running task     stack:0     pid:137   ppid:133    flags:0x00000002
[   36.807057] Call trace:
[   36.807417]  __switch_to+0xe4/0x160
[   36.808319]  0x0

Secure World:

D/TC:0 0 __afl_cov_trace_start:148 Integer as pointer: 0x451f2d00
D/TC:0 0 __afl_cov_trace_start:151 TPIDRRO_EL0 curent value: 0x0

QEMU Console:

(qemu) __afl_set_ctx_ptr: TPIDRRO_EL0 before - 0x0
__afl_set_ctx_ptr: TPIDRRO_EL0 after- 0x451f2d00

So it seems that the function did complete, but did not execute the remaining code in __afl_cov_trace_start function.

Trying GDB now