NuttX RTOS for PinePhone: Fixing the Interrupts

📝 1 Sep 2022

Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy

Creating our own Operating System (non-Linux) for Pine64 PinePhone can be super challenging…

We’ll answer these questions today as we port Apache NuttX RTOS to PinePhone.

Let’s dive into our Porting Journal for NuttX on PinePhone…

And relive the very first Interrupt issue that we hit…

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
arm64_gic_initialize: no distributor detected, giving up

Partial list of Shared Peripheral Interrupts for Allwinner A64’s GIC

Partial list of Shared Peripheral Interrupts for Allwinner A64’s GIC

1 Generic Interrupt Controller

What’s a GIC?

PinePhone’s Generic Interrupt Controller (GIC) works like a typical Interrupt Controller in a CPU. It manages Interrupts for the Arm64 CPU.

Except that GIC is a special chunk of silicon that lives inside the Allwinner A64 SoC. (Outside the Arm64 CPU)

Huh? Arm64 CPU doesn’t have its own Interrupt Controller?

Interrupting gets complicated… Remember PinePhone runs on 4 Arm64 CPUs?

The 4 CPUs must handle the Interrupts triggered by all kinds of Peripherals: UART, I2C, SPI, DMA, USB, microSD, eMMC, …

We do this the flexible, efficient way with a GIC, which supports…

Allwinner A64’s GIC supports 157 Interrupt Sources: 16 Software-Generated, 16 Private and 125 Shared.

The GIC in Allwinner A64 is a little problematic, let’s talk…

Allwinner A64 runs on Arm GIC Version 2

Allwinner A64 runs on Arm GIC Version 2

2 Allwinner A64 GIC

What’s this GIC error we saw earlier?

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
arm64_gic_initialize: no distributor detected, giving up

When we boot NuttX RTOS, it expects PinePhone to provide a modern Generic Interrupt Controller (GIC), Version 3.

But the Allwinner A64 User Manual (page 210, “GIC”) says that PinePhone runs on…

Our GIC Version 2 is from 2011, when Arm CPUs were still 32-bit… That’s 11 years ago!

So we need to fix NuttX and downgrade GIC Version 3 back to GIC Version 2, specially for PinePhone.

We’re sure that PinePhone runs on GIC Version 2?

Let’s verify! This code reads the GIC Version from PinePhone: arch/arm64/src/common/arm64_gicv3.c

// Init GIC v2 for PinePhone
int arm64_gic_initialize(void) {
  sinfo("TODO: Init GIC for PinePhone\n");
  sinfo("CONFIG_GICD_BASE=%p\n", CONFIG_GICD_BASE);
  sinfo("CONFIG_GICR_BASE=%p\n", CONFIG_GICR_BASE);

  // To verify the GIC Version, read the Peripheral ID2 Register (ICPIDR2) at Offset 0xFE8 of GIC Distributor.
  // Bits 4 to 7 of ICPIDR2 are...
  // - 0x1 for GIC Version 1
  // - 0x2 for GIC Version 2
  // GIC Distributor is at 0x01C80000 + 0x1000
  const uint8_t *ICPIDR2 = (const uint8_t *) (CONFIG_GICD_BASE + 0xFE8);
  uint8_t version = (*ICPIDR2 >> 4) & 0b1111;
  sinfo("GIC Version is %d\n", version);
  DEBUGASSERT(version == 2);

Here’s the output…

TODO: Init GIC for PinePhone
CONFIG_GICD_BASE=0x1c81000
CONFIG_GICR_BASE=0x1c82000
GIC Version is 2

(Source)

Yep PinePhone runs on GIC Version 2. Bummer.

What are GICD and GICR?

GICD (GIC Distributor) and GICR (GIC CPU Interface) are the addresses for accessing the GIC on PinePhone.

According to Allwinner A64 User Manual (page 74, “Memory Mapping”), the GIC is located at…

ModuleAddressRemarks
GIC_DIST0x01C8 0000 + 0x1000GIC Distributor (GICD)
GIC_CPUIF0x01C8 0000 + 0x2000GIC CPU Interface (GICR)

Which we define in NuttX as: arch/arm64/include/qemu/chip.h

// PinePhone Generic Interrupt Controller
// GIC_DIST:  0x01C80000 + 0x1000
// GIC_CPUIF: 0x01C80000 + 0x2000
#define CONFIG_GICD_BASE 0x01C81000  
#define CONFIG_GICR_BASE 0x01C82000  

Back to our headache of GIC Version 2…

3 GIC Version 2

Does NuttX support GIC Version 2 for PinePhone?

Yes NuttX supports Generic Interrupt Controller (GIC) Version 2 but there’s a catch… It’s for Arm32 CPUs, not Arm64 CPUs!

Remember: GIC Version 2 was created for Arm32.

So we port NuttX’s GIC Version 2 from Arm32 to Arm64?

Kinda. We did a horrible hack… Don’t try this at home! (Unless you have a ten-foot pole) arch/arm64/src/common/arm64_gicv3.c

// GIC v2 for PinePhone:
// Reuse the implementation of Arm32 GIC v2
#define PINEPHONE_GICv2
#define CONFIG_ARMV7A_HAVE_GICv2
#define CONFIG_ARCH_TRUSTZONE_NONSECURE

// Override...
// MPCORE_ICD_VBASE: GIC Distributor
// MPCORE_ICC_VBASE: GIC CPU Interface
#include "../arch/arm/src/armv7-a/mpcore.h"
#undef  MPCORE_ICD_VBASE
#undef  MPCORE_ICC_VBASE
#define MPCORE_ICD_VBASE CONFIG_GICD_BASE  // 0x01C81000  
#define MPCORE_ICC_VBASE CONFIG_GICR_BASE  // 0x01C82000  

// Inject Arm32 GIC v2 Implementation
#include "../arch/arm/src/armv7-a/arm_gicv2.c"

(We commented out the GIC Version 3 code as NOTUSED)

What! Did we just #include the GIC Version 2 Source Code from Arm32 into Arm64?

Yep it’s an awful trick but it seems to work!

We made minor tweaks to GIC Version 2 to compile with Arm64…

We rewrote this function for Arm64 because we’re passing 64-bit Registers (instead of 32-bit): arm64_gicv3.c

// Decode IRQ for PinePhone.
// Based on arm_decodeirq in arm_gicv2.c.
// Previously we passed 32-bit Registers as `uint32_t *`
uint64_t * arm64_decodeirq(uint64_t * regs) {
  /* Omitted: Get the interrupt ID */
  ...
  /* Dispatch the Arm64 interrupt */
  regs = arm64_doirq(irq, regs);

Everything else stays the same! Well except for…

Injecting Arm32 code into Arm64 sounds so reckless… Will it work?

Let’s test our reckless GIC Version 2 with QEMU Emulator…

Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy

Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy

4 Test PinePhone GIC with QEMU

Will our hacked GIC Version 2 run on PinePhone?

Before testing on PinePhone, let’s test our Generic Interrupt Controller (GIC) Version 2 on QEMU Emulator.

Follow these steps to build NuttX for QEMU with GIC Version 2

Enter this to start QEMU with NuttX and GIC Version 2…

# Run GIC Version 2 with QEMU
qemu-system-aarch64 \
  -smp 4 \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=2 \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

Note that “gic-version=2” instead of the usual GIC Version 3 for Arm64.

Also we simulated 4 Cores of Arm Cortex-A53 (similar to PinePhone): “-smp 4

We see this in QEMU…

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
arm64_gic_initialize: GIC Version is 2

up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0

work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found

NuttShell (NSH) NuttX-10.3.0-RC2
nsh>
nx_start: CPU0: Beginning Idle Loop

(See the Complete Log)

NuttX with GIC Version 2 boots OK on QEMU, and will probably run on PinePhone!

We tested Interrupts with GIC Version 2?

Yep the pic above shows “TX” whenever an Interrupt Handler is dispatched.

(We added Debug Logging to arm64_vectors.S and arm64_vector_table.S)

How did we get the GIC Base Addresses for QEMU?

CONFIG_GICD_BASE=0x8000000
CONFIG_GICR_BASE=0x8010000

We got the Base Addresses for GIC Distributor (CONFIG_GICD_BASE) and GIC CPU Interface (CONFIG_GICR_BASE) by dumping the Device Tree from QEMU…

# Dump Device Tree for GIC Version 2
qemu-system-aarch64 \
  -smp 4 \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=2,dumpdtb=gicv2.dtb \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

# Convert Device Tree to text format
dtc \
  -o gicv2.dts \
  -O dts \
  -I dtb \
  gicv2.dtb

The Base Addresses are revealed in the GIC Version 2 Device Tree: gicv2.dts

intc@8000000 {
reg = <
    0x00 0x8000000 0x00 0x10000  //  GIC Distributor:   0x8000000
    0x00 0x8010000 0x00 0x10000  //  GIC CPU Interface: 0x8010000
    0x00 0x8030000 0x00 0x10000  //  VGIC Virtual Interface Control: 0x8030000
    0x00 0x8040000 0x00 0x10000  //  VGIC Virtual CPU Interface:     0x8040000
>;
compatible = "arm,cortex-a15-gic";

(More about this)

Which we defined in NuttX at…

5 PinePhone Hangs At Startup

NuttX should boot OK on PinePhone right?

We followed these steps to boot NuttX on PinePhone (with GIC Version 2)…

But NuttX got stuck on PinePhone in a very curious way…

arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
uart_regi

NuttX got stuck while printing a line!

And it happened a short while after we started the System Timer: up_timer_initialize

(More about System Timer)

Something in the System Timer caused this?

Yep! If we disabled the System Timer, PinePhone will continue to boot.

Remember that the System Timer will trigger Interrupts periodically…

Perhaps we’re handling Interrupts incorrectly?

Let’s investigate…

6 Timer Interrupt Isn’t Handled

Why did PinePhone hang while handling System Timer Interrupts?

Was the Timer Interrupt Handler called?

We verified that Timer Interrupt Handler arm64_arch_timer_compare_isr was NEVER called.

(We checked by calling up_putc, which prints directly to the UART Port)

So something went wrong BEFORE calling the Interrupt Handler. Let’s backtrack…

Is the Interrupt Vector Table pointing correctly to the Timer Interrupt Handler?

NuttX defines an Interrupt Vector Table for dispatching Interrupt Handlers…

We dumped NuttX’s Interrupt Vector Table…

And verified that the Timer Interrupt Handler is set correctly in the table.

Maybe something went wrong when NuttX tried to call the Interrupt Handler?

NuttX should call Interrupt Dispatcher irq_dispatch to dispatch the Interrupt Handler…

But nope, irq_dispatch was never called.

Some error occurred and NuttX threw an Unexpected Interrupt?

Nope, the Unexpected Interrupt Handler irq_unexpected_isr was never called either.

OK I’m really stumped. Did something go bad deep inside Arm64 Interrupts?

Possibly! Let’s talk about the Arm64 Vector Table…

Vector Base Address Register, EL1

7 Arm64 Vector Table Is Wrong

When an Interrupt is triggered, what happens in the Arm64 CPU?

According to the Arm Cortex-A53 Technical Reference Manual (page 4-121), the CPU reads the Vector Base Address Register (EL1) to locate the Arm64 Vector Table. (Pic above)

(Why EL1? We’ll explain in a while)

The Arm64 Vector Table looks like this…

Arm64 Vector Table

(Source)

Which we define in NuttX as _vector_table: arch/arm64/src/common/arm64_vector_table.S

GTEXT(_vector_table)
SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)
  ...
  /* Current EL with SP0 / IRQ */
  .align 7
  arm64_enter_exception x0, x1
  b    arm64_irq_handler
  ...
  /* Current EL with SPx / IRQ */
  .align 7
  arm64_enter_exception x0, x1
  b    arm64_irq_handler

(arm64_enter_exception saves the Arm64 Registers)

(arm64_irq_handler is the NuttX IRQ Handler)

So Vector Base Address Register (EL1) should point to _vector_table?

Let’s find out! This is how we read Vector Base Address Register (EL1): arch/arm64/src/common/arm64_arch_timer.c

void up_timer_initialize(void) {
  ...
  // Read Vector Base Address Register EL1
  extern void *_vector_table[];
  sinfo("_vector_table=%p\n", _vector_table);
  sinfo("Before writing: vbar_el1=%p\n", read_sysreg(vbar_el1));

Here’s the output on PinePhone…

_vector_table=0x400a7000
Before writing: vbar_el1=0x40227000

Aha! _vector_table is at 0x400a 7000… But Vector Base Address Register (EL1) says 0x4022 7000!

Our Arm64 CPU is pointing to the wrong Arm64 Vector Table… Hence our Interrupt Handler is never called!

Let’s fix it: arch/arm64/src/common/arm64_arch_timer.c

  // Write Vector Base Address Register EL1
  write_sysreg((uint64_t)_vector_table, vbar_el1);
  ARM64_ISB();

  // Read Vector Base Address Register EL1
  sinfo("After writing: vbar_el1=%p\n", read_sysreg(vbar_el1));

This writes the correct value of _vector_table back into Vector Base Address Register EL1. Here’s the output on PinePhone…

_vector_table=0x400a7000
Before writing: vbar_el1=0x40227000
After writing:  vbar_el1=0x400a7000

Yep Vector Base Address Register (EL1) is now correct.

Our Interrupt Handlers are now working fine… And PinePhone boots successfully yay! 🎉

Starting kernel ...

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000

arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2

up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000

uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0

work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400a7000 _einit: 0x400a7000 _stext: 0x40080000 _etext: 0x400a8000
nsh: sysinit: fopen failed: 2

nshn:x _msktfaarttf:s :C PcUo0m:m aBnedg innonti nfgo uInddle  L oNouptt
 Shell (NSH) NuttX-10.3.0-RC2

(Yeah the output is slightly garbled, the UART Driver needs fixing)

Now that we have UART Interrupts, NuttX Shell works perfectly OK on PinePhone…

nsh> uname -a
NuttX 10.3.0-RC2 fc909c6-dirty Sep  1 2022 17:05:44 arm64 qemu-a53

nsh> help
help usage:  help [-v] [<cmd>]

  .         cd        dmesg     help      mount     rmdir     true      xd        
  [         cp        echo      hexdump   mv        set       truncate  
  ?         cmp       exec      kill      printf    sleep     uname     
  basename  dirname   exit      ls        ps        source    umount    
  break     dd        false     mkdir     pwd       test      unset     
  cat       df        free      mkrd      rm        time      usleep    

Builtin Apps:
  getprime  hello     nsh       ostest    sh        

nsh> hello
task_spawn: name=hello entry=0x4009b1a0 file_actions=0x400c9580 attr=0x400c9588 argv=0x400c96d0
spawn_execattrs: Setting policy=2 priority=100 for pid=3
Hello, World!!

nsh> ls /dev
/dev:
 console
 null
 ram0
 ram2
 ttyS0
 zero

Watch the Demo on YouTube

Another Demo Video

Let’s talk about EL1…

8 Exception Levels

What’s EL1?

EL1 is Exception Level 1. As defined in Arm Cortex-A53 Technical Reference Manual page 3-5 (“Exception Level”)…

The ARMv8 exception model defines exception levels EL0-EL3, where:

So EL1 is (kinda) privileged, suitable for running OS Kernel code. (Like NuttX)

NuttX runs mostly in EL1 and briefly in EL2 (at startup)…

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

(Remember that EL1 is less privileged than EL2, which supports Processor Virtualization. Host OS will run at EL2, Guest OS at EL1)

That’s why we talked about the EL1 Vector Base Address Register in the previous section.

So there’s a Vector Base Address Register for EL1, EL2 and EL3?

Indeed! Each Exception Level has its own Arm64 Vector Table.

(Except EL0)

Who loads the EL1 Vector Base Address Register?

The EL1 Vector Base Address Register is loaded during EL1 Initialisation at startup: arch/arm64/src/common/arm64_boot.c

void arm64_boot_el1_init(void) {
  /* Setup vector table */
  write_sysreg((uint64_t)_vector_table, vbar_el1);
  ARM64_ISB();

arm64_boot_el1_init is called by our Startup Code: arch/arm64/src/common/arm64_head.S

    PRINT(switch_el1, "- Boot from EL1\r\n")

    /* EL1 init */
    bl    arm64_boot_el1_init

    /* set SP_ELx and Enable SError interrupts */
    msr   SPSel, #1
    msr   DAIFClr, #(DAIFCLR_ABT_BIT)
    isb

jump_to_c_entry:
    PRINT(jump_to_c_entry, "- Boot to C runtime for OS Initialize\r\n")
    ret x25

The Boot Sequence for NuttX RTOS is explained here…

So how did our Vector Base Address Register get messed up? And why is it off by exactly 0x18 0000?

We might have missed something when we changed the Kernel Start Address from 0x4028 0000 to 0x4008 0000 in dramboot.ld.

Or maybe we made a mistake in our Memory Map

We’ll check some more.

9 What’s Next

There’s plenty to be done for NuttX on PinePhone, please lemme know if you would like to join me 🙏

Check out the other articles on NuttX RTOS for PinePhone

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/interrupt.md