📝 1 Sep 2022
UPDATE: PinePhone is now officially supported by Apache NuttX RTOS (See this)
Creating our own Operating System (non-Linux) for Pine64 PinePhone can be super challenging…
How does PinePhone handle Interrupts?
What’s a Generic Interrupt Controller? (GIC)
Why is PinePhone’s GIC particularly problematic?
What’s an Exception Level? (EL)
Why does EL matter for handling Arm64 Interrupts?
We’ll answer these questions today as we port Apache NuttX RTOS to PinePhone.
Let’s dive into our Porting Journal for NuttX on PinePhone…
And relive the very first Interrupt issue that we hit…
HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
arm64_gic_initialize: no distributor detected, giving up
Partial list of Shared Peripheral Interrupts for Allwinner A64’s GIC
What’s a GIC?
PinePhone’s Generic Interrupt Controller (GIC) works like a typical Interrupt Controller in a CPU. It manages Interrupts for the Arm64 CPU.
Except that GIC is a special chunk of silicon that lives inside the Allwinner A64 SoC. (Outside the Arm64 CPU)
Huh? Arm64 CPU doesn’t have its own Interrupt Controller?
Interrupting gets complicated… Remember PinePhone runs on 4 Arm64 CPUs?
The 4 CPUs must handle the Interrupts triggered by all kinds of Peripherals: UART, I2C, SPI, DMA, USB, microSD, eMMC, …
We do this the flexible, efficient way with a GIC, which supports…
Shared Peripheral Interrupts (SPI)
GIC can route Peripheral Interrupts to one or multiple CPUs
(Pic above)
Private Peripheral Interrupts (PPI)
GIC can route Peripheral Interrupts to a single CPU
Software-Generated Interrupts (SGI)
GIC lets CPUs to talk to each other by triggering Software Interrupts
(Anyone remember Silicon Graphics?)
Allwinner A64’s GIC supports 157 Interrupt Sources: 16 Software-Generated, 16 Private and 125 Shared.
The GIC in Allwinner A64 is a little problematic, let’s talk…
Allwinner A64 runs on Arm GIC Version 2
What’s this GIC error we saw earlier?
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
arm64_gic_initialize: no distributor detected, giving up
When we boot NuttX RTOS, it expects PinePhone to provide a modern Generic Interrupt Controller (GIC), Version 3.
But the Allwinner A64 User Manual (page 210, “GIC”) says that PinePhone runs on…
Arm GIC PL400, which is based on…
Our GIC Version 2 is from 2011, when Arm CPUs were still 32-bit… That’s 11 years ago!
So we need to fix NuttX and downgrade GIC Version 3 back to GIC Version 2, specially for PinePhone.
We’re sure that PinePhone runs on GIC Version 2?
Let’s verify! This code reads the GIC Version from PinePhone: arch/arm64/src/common/arm64_gicv3.c
// Init GIC v2 for PinePhone
int arm64_gic_initialize(void) {
sinfo("TODO: Init GIC for PinePhone\n");
sinfo("CONFIG_GICD_BASE=%p\n", CONFIG_GICD_BASE);
sinfo("CONFIG_GICR_BASE=%p\n", CONFIG_GICR_BASE);
// To verify the GIC Version, read the Peripheral ID2 Register (ICPIDR2) at Offset 0xFE8 of GIC Distributor.
// Bits 4 to 7 of ICPIDR2 are...
// - 0x1 for GIC Version 1
// - 0x2 for GIC Version 2
// GIC Distributor is at 0x01C80000 + 0x1000
const uint8_t *ICPIDR2 = (const uint8_t *) (CONFIG_GICD_BASE + 0xFE8);
uint8_t version = (*ICPIDR2 >> 4) & 0b1111;
sinfo("GIC Version is %d\n", version);
DEBUGASSERT(version == 2);
Here’s the output…
TODO: Init GIC for PinePhone
CONFIG_GICD_BASE=0x1c81000
CONFIG_GICR_BASE=0x1c82000
GIC Version is 2
Yep PinePhone runs on GIC Version 2. Bummer.
What are GICD and GICR?
GICD (GIC Distributor) and GICR (GIC CPU Interface) are the addresses for accessing the GIC on PinePhone.
According to Allwinner A64 User Manual (page 74, “Memory Mapping”), the GIC is located at…
| Module | Address | Remarks |
|---|---|---|
| GIC_DIST | 0x01C8 0000 + 0x1000 | GIC Distributor (GICD) |
| GIC_CPUIF | 0x01C8 0000 + 0x2000 | GIC CPU Interface (GICR) |
Which we define in NuttX as: arch/arm64/include/a64/chip.h
// PinePhone Generic Interrupt Controller
// GIC_DIST: 0x01C80000 + 0x1000
// GIC_CPUIF: 0x01C80000 + 0x2000
#define CONFIG_GICD_BASE 0x01C81000
#define CONFIG_GICR_BASE 0x01C82000
Back to our headache of GIC Version 2…
Does NuttX support GIC Version 2 for PinePhone?
Yes NuttX supports Generic Interrupt Controller (GIC) Version 2 but there’s a catch… It’s for Arm32 CPUs, not Arm64 CPUs!
Remember: GIC Version 2 was created for Arm32.
So we port NuttX’s GIC Version 2 from Arm32 to Arm64?
Kinda. We did a horrible hack… Don’t try this at home! (Unless you have a ten-foot pole) arch/arm64/src/common/arm64_gicv3.c
// GIC v2 for PinePhone:
// Reuse the implementation of Arm32 GIC v2
#define PINEPHONE_GICv2
#define CONFIG_ARMV7A_HAVE_GICv2
#define CONFIG_ARCH_TRUSTZONE_NONSECURE
// Override...
// MPCORE_ICD_VBASE: GIC Distributor
// MPCORE_ICC_VBASE: GIC CPU Interface
#include "../arch/arm/src/armv7-a/mpcore.h"
#undef MPCORE_ICD_VBASE
#undef MPCORE_ICC_VBASE
#define MPCORE_ICD_VBASE CONFIG_GICD_BASE // 0x01C81000
#define MPCORE_ICC_VBASE CONFIG_GICR_BASE // 0x01C82000
// Inject Arm32 GIC v2 Implementation
#include "../arch/arm/src/armv7-a/arm_gicv2.c"
(We commented out the GIC Version 3 code as NOTUSED)
What! Did we just #include the GIC Version 2 Source Code from Arm32 into Arm64?
Yep it’s an awful trick but it seems to work!
We made minor tweaks to GIC Version 2 to compile with Arm64…
We rewrote this function for Arm64 because we’re passing 64-bit Registers (instead of 32-bit): arm64_gicv3.c
// Decode IRQ for PinePhone.
// Based on arm_decodeirq in arm_gicv2.c.
// Previously we passed 32-bit Registers as `uint32_t *`
uint64_t * arm64_decodeirq(uint64_t * regs) {
/* Omitted: Get the interrupt ID */
...
/* Dispatch the Arm64 interrupt */
regs = arm64_doirq(irq, regs);
Everything else stays the same! Well except for…
Injecting Arm32 code into Arm64 sounds so reckless… Will it work?
Let’s test our reckless GIC Version 2 with QEMU Emulator…
UPDATE: NuttX Mainline now supports GIC Version 2 (See this)
Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy
Will our hacked GIC Version 2 run on PinePhone?
Before testing on PinePhone, let’s test our Generic Interrupt Controller (GIC) Version 2 on QEMU Emulator.
Follow these steps to build NuttX for QEMU with GIC Version 2…
Enter this to start QEMU with NuttX and GIC Version 2…
## Run GIC Version 2 with QEMU
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=2 \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
Note that “gic-version=2” instead of the usual GIC Version 3 for Arm64.
Also we simulated 4 Cores of Arm Cortex-A53 (similar to PinePhone): “-smp 4”
We see this in QEMU…
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-10.3.0-RC2
nsh>
nx_start: CPU0: Beginning Idle Loop
NuttX with GIC Version 2 boots OK on QEMU, and will probably run on PinePhone!
We tested Interrupts with GIC Version 2?
Yep the pic above shows “TX” whenever an Interrupt Handler is dispatched.
(We added Debug Logging to arm64_vectors.S and arm64_vector_table.S)
How did we get the GIC Base Addresses for QEMU?
CONFIG_GICD_BASE=0x8000000
CONFIG_GICR_BASE=0x8010000
We got the Base Addresses for GIC Distributor (CONFIG_GICD_BASE) and GIC CPU Interface (CONFIG_GICR_BASE) by dumping the Device Tree from QEMU…
## Dump Device Tree for GIC Version 2
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=2,dumpdtb=gicv2.dtb \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
## Convert Device Tree to text format
dtc \
-o gicv2.dts \
-O dts \
-I dtb \
gicv2.dtb
The Base Addresses are revealed in the GIC Version 2 Device Tree: gicv2.dts…
intc@8000000 {
reg = <
0x00 0x8000000 0x00 0x10000 // GIC Distributor: 0x8000000
0x00 0x8010000 0x00 0x10000 // GIC CPU Interface: 0x8010000
0x00 0x8030000 0x00 0x10000 // VGIC Virtual Interface Control: 0x8030000
0x00 0x8040000 0x00 0x10000 // VGIC Virtual CPU Interface: 0x8040000
>;
compatible = "arm,cortex-a15-gic";
Which we defined in NuttX at…
UPDATE: NuttX Mainline now provides a Board Config “qemu-armv8a:nsh_gicv2” for testing GIC Version 2 with QEMU (See this)
NuttX should boot OK on PinePhone right?
We followed these steps to boot NuttX on PinePhone (with GIC Version 2)…
But NuttX got stuck on PinePhone in a very curious way…
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
uart_regi
NuttX got stuck while printing a line!
And it happened a short while after we started the System Timer: up_timer_initialize
Something in the System Timer caused this?
Yep! If we disabled the System Timer, PinePhone will continue to boot.
Remember that the System Timer will trigger Interrupts periodically…
Perhaps we’re handling Interrupts incorrectly?
Let’s investigate…
UPDATE: This problem doesn’t happen with the latest code in NuttX Mainline (See this)
Why did PinePhone hang while handling System Timer Interrupts?
Was the Timer Interrupt Handler called?
We verified that Timer Interrupt Handler arm64_arch_timer_compare_isr was NEVER called.
(We checked by calling up_putc, which prints directly to the UART Port)
So something went wrong BEFORE calling the Interrupt Handler. Let’s backtrack…
Is the Interrupt Vector Table pointing correctly to the Timer Interrupt Handler?
NuttX defines an Interrupt Vector Table for dispatching Interrupt Handlers…
We dumped NuttX’s Interrupt Vector Table…
And verified that the Timer Interrupt Handler is set correctly in the table.
Maybe something went wrong when NuttX tried to call the Interrupt Handler?
NuttX should call Interrupt Dispatcher irq_dispatch to dispatch the Interrupt Handler…
But nope, irq_dispatch was never called.
Some error occurred and NuttX threw an Unexpected Interrupt?
Nope, the Unexpected Interrupt Handler irq_unexpected_isr was never called either.
OK I’m really stumped. Did something go bad deep inside Arm64 Interrupts?
Possibly! Let’s talk about the Arm64 Vector Table…
When an Interrupt is triggered, what happens in the Arm64 CPU?
According to the Arm Cortex-A53 Technical Reference Manual (page 4-121), the CPU reads the Vector Base Address Register (EL1) to locate the Arm64 Vector Table. (Pic above)
(Why EL1? We’ll explain in a while)
The Arm64 Vector Table looks like this…
Which we define in NuttX as _vector_table: arch/arm64/src/common/arm64_vector_table.S
GTEXT(_vector_table)
SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)
...
/* Current EL with SP0 / IRQ */
.align 7
arm64_enter_exception x0, x1
b arm64_irq_handler
...
/* Current EL with SPx / IRQ */
.align 7
arm64_enter_exception x0, x1
b arm64_irq_handler
(arm64_enter_exception saves the Arm64 Registers)
(arm64_irq_handler is the NuttX IRQ Handler)
So Vector Base Address Register (EL1) should point to _vector_table?
Let’s find out! This is how we read Vector Base Address Register (EL1): arch/arm64/src/common/arm64_arch_timer.c
void up_timer_initialize(void) {
...
// Read Vector Base Address Register EL1
extern void *_vector_table[];
sinfo("_vector_table=%p\n", _vector_table);
sinfo("Before writing: vbar_el1=%p\n", read_sysreg(vbar_el1));
Here’s the output on PinePhone…
_vector_table=0x400a7000
Before writing: vbar_el1=0x40227000
Aha! _vector_table is at 0x400a 7000… But Vector Base Address Register (EL1) says 0x4022 7000!
Our Arm64 CPU is pointing to the wrong Arm64 Vector Table… Hence our Interrupt Handler is never called!
Let’s fix it: arch/arm64/src/common/arm64_arch_timer.c
// Write Vector Base Address Register EL1
write_sysreg((uint64_t)_vector_table, vbar_el1);
ARM64_ISB();
// Read Vector Base Address Register EL1
sinfo("After writing: vbar_el1=%p\n", read_sysreg(vbar_el1));
This writes the correct value of _vector_table back into Vector Base Address Register EL1. Here’s the output on PinePhone…
_vector_table=0x400a7000
Before writing: vbar_el1=0x40227000
After writing: vbar_el1=0x400a7000
Yep Vector Base Address Register (EL1) is now correct.
Our Interrupt Handlers are now working fine… And PinePhone boots successfully yay! 🎉
Starting kernel ...
HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400a7000 _einit: 0x400a7000 _stext: 0x40080000 _etext: 0x400a8000
nsh: sysinit: fopen failed: 2
nshn:x _msktfaarttf:s :C PcUo0m:m aBnedg innonti nfgo uInddle L oNouptt
Shell (NSH) NuttX-10.3.0-RC2
(Yeah the output is slightly garbled, here’s the workaround)
Now that we have UART Interrupts, NuttX Shell works perfectly OK on PinePhone…
nsh> uname -a
NuttX 10.3.0-RC2 fc909c6-dirty Sep 1 2022 17:05:44 arm64 qemu-armv8a
nsh> help
help usage: help [-v] [<cmd>]
. cd dmesg help mount rmdir true xd
[ cp echo hexdump mv set truncate
? cmp exec kill printf sleep uname
basename dirname exit ls ps source umount
break dd false mkdir pwd test unset
cat df free mkrd rm time usleep
Builtin Apps:
getprime hello nsh ostest sh
nsh> hello
task_spawn: name=hello entry=0x4009b1a0 file_actions=0x400c9580 attr=0x400c9588 argv=0x400c96d0
spawn_execattrs: Setting policy=2 priority=100 for pid=3
Hello, World!!
nsh> ls /dev
/dev:
console
null
ram0
ram2
ttyS0
zero
Let’s talk about EL1…
UPDATE: This patching isn’t needed with the latest code in NuttX Mainline (See this)
What’s EL1?
EL1 is Exception Level 1. As defined in Arm Cortex-A53 Technical Reference Manual page 3-5 (“Exception Level”)…
The ARMv8 exception model defines exception levels EL0-EL3, where:
- EL0 has the lowest software execution privilege, and execution at EL0 is called unprivileged execution.
- Increased exception levels, from 1 to 3, indicate increased software execution privilege.
- EL2 provides support for processor virtualization.
- EL3 provides support for a secure state, see Security state on page 3-6.
So EL1 is (kinda) privileged, suitable for running OS Kernel code. (Like NuttX)
NuttX runs mostly in EL1 and briefly in EL2 (at startup)…
HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
(Remember that EL1 is less privileged than EL2, which supports Processor Virtualization. Host OS will run at EL2, Guest OS at EL1)
That’s why we talked about the EL1 Vector Base Address Register in the previous section.
So there’s a Vector Base Address Register for EL1, EL2 and EL3?
Indeed! Each Exception Level has its own Arm64 Vector Table.
(Except EL0)
Who loads the EL1 Vector Base Address Register?
The EL1 Vector Base Address Register is loaded during EL1 Initialisation at startup: arch/arm64/src/common/arm64_boot.c
void arm64_boot_el1_init(void) {
/* Setup vector table */
write_sysreg((uint64_t)_vector_table, vbar_el1);
ARM64_ISB();
arm64_boot_el1_init is called by our Startup Code: arch/arm64/src/common/arm64_head.S
PRINT(switch_el1, "- Boot from EL1\r\n")
/* EL1 init */
bl arm64_boot_el1_init
/* set SP_ELx and Enable SError interrupts */
msr SPSel, #1
msr DAIFClr, #(DAIFCLR_ABT_BIT)
isb
jump_to_c_entry:
PRINT(jump_to_c_entry, "- Boot to C runtime for OS Initialize\r\n")
ret x25
The Boot Sequence for NuttX RTOS is explained here…
There’s plenty to be done for NuttX on PinePhone, please lemme know if you would like to join me 🙏
Please check out the other articles on NuttX for PinePhone…
Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…