Star64 JH7110 + NuttX RTOS: RISC-V PLIC Interrupts and Serial I/O

📝 2 Aug 2023

We’re almost ready with our barebones port of Apache NuttX Real-Time Operating System (RTOS) to Pine64 Star64 64-bit RISC-V Single-Board Computer! (Pic below)

(Based on StarFive JH7110, the same SoC in VisionFive2)

In this article, we find out…

• Why there’s No Console Output from NuttX Apps

• How Serial I/O works in NuttX QEMU

• How UART I/O differs for Star64 vs QEMU

• What’s the RISC-V Platform-Level Interrupt Controller (pic above)

• Why we delegate RISC-V Machine-Mode Interrupts to Supervisor-Mode

• How NuttX Star64 handles UART Interrupts

• Which leads to a new problem: 16550 UART Controller fires too many Spurious Interrupts!

  (Watch the Demo Video on YouTube)

We’ll see later that NuttX Star64 actually works fine! It’s just very very slooow because of the Spurious Interrupts.

(UPDATE: We fixed the Spurious UART Interrupts!)

§1 No Console Output from NuttX Apps

At the end of our previous article, NuttX seems to boot fine on Star64 (pic below)…

Starting kernel ...
123067DFHBCI
nx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
board_late_initialize: 
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
nx_start_application: ret=3
up_exit: TCB=0x404088d0 exiting
nx_start: CPU0: Beginning Idle Loop

(See the Output Log)

But NuttX Shell doesn’t appear!

Maybe NuttX Shell wasn’t started correctly?

Let’s find out! When NuttX Apps (and NuttX Shell) print to the Serial Console (via printf), this function will be called in the NuttX Kernel: uart_write

Thus we add Debug Logs to uart_write. Something interesting happens…

uart_write (0xc000a610):
0000  0a 4e 75 74 74 53 68 65 6c 6c 20 28 4e 53 48 29  .NuttShell (NSH)
0010  20 4e 75 74 74 58 2d 31 32 2e 30 2e 33 0a         NuttX-12.0.3.  

uart_write (0xc0015338):
0000  6e 73 68 3e 20                                   nsh>            

uart_write (0xc0015310):
0000  1b 5b 4b                                         .[K             

This says that NuttX Shell is actually started, and trying to print something!

Just that NuttX Shell couldn’t produce any Console Output.

But we see other messages from NuttX Kernel!

That’s because NuttX Kernel doesn’t call uart_write to print messages.

Instead, NuttX Kernel calls up_putc. Which calls u16550_putc to write directly to the UART Output Register.

So uart_write is a lot more sophisticated than up_putc?

Yep NuttX Apps will (indirectly) call uart_write to do Serial I/O with Buffering and Interrupts.

Somehow uart_write is broken for all NuttX Apps on Star64.

Let’s find out why…

§2 Serial Output in NuttX QEMU

What happens in NuttX Serial Output?

To understand how NuttX Apps print to the Serial Console (via printf), we add Debug Logs to NuttX QEMU (pic below)…

ABC
nx_start: Entry
up_irq_enable: 
up_enable_irq: irq=17, RISCV_IRQ_SOFT=17

uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
up_enable_irq: irq=35, extirq=10, RISCV_IRQ_EXT=25

work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
up_exit: TCB=0x802088d0 exiting

(See the Complete Log)

(See the Build Outputs)

(up_enable_irq is defined here)

In the log above, NuttX QEMU enables UART Interrupts at NuttX IRQ 35.

(Equivalent to RISC-V IRQ 10, with IRQ Offset of 25)

Then NuttX Shell runs in QEMU…

$%&riscv_doirq: irq=8
$%&riscv_doirq: irq=8
$%&riscv_doirq: irq=8
...
$%&riscv_doirq: irq=8
$%&riscv_doirq: irq=8
$%&riscv_doirq: irq=8

(riscv_doirq is defined here)

NuttX IRQ 8 appears frequently in our log. That’s for RISCV_IRQ_ECALLU: ECALL from RISC-V User Mode to Supervisor Mode.

This happens when our NuttX App (in User Mode) makes a System Call to NuttX Kernel (in Supervisor Mode).

Like for printing to the Serial Console…

uart_write (0xc000a610):
0000  0a 4e 75 74 74 53 68 65 6c 6c 20 28 4e 53 48 29  .NuttShell (NSH)
0010  20 4e 75 74 74 58 2d 31 32 2e 30 2e 33 0a         NuttX-12.0.3.  

Then this Alphabet Soup appears…

FAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ADEF
FNFuFtFtFSFhFeFlFlF F(FNFSFHF)F FNFuFtFtFXF-F1F2F.F0F.F3F

This says that the NuttX Kernel calls uart_write (print to Serial Console), which calls…

[A] uart_putxmitchar (write to Serial Buffer), which calls…

[D] uart_xmitchars (print the Serial Buffer), which calls…

[E] uart_txready (check for UART ready) and…

[F] u16550_send (write to UART output)

And that’s what happens when a NuttX App prints to the Serial Console (via printf)…

1. NuttX App (in User Mode) makes a System Call to NuttX Kernel (in Supervisor Mode)

   (uart_write)

2. NuttX Kernel writes the output to the Serial Buffer

   (uart_putxmitchar)

3. NuttX Kernel reads the Serial Buffer, one character at a time…

   (uart_xmitchars)

4. If the UART Transmit Status is ready…

   (uart_txready)

5. Write the character to UART Output

   (u16550_send)

What if UART Transmit Status is NOT ready?

UART will trigger a Transmit Ready Interrupt when it’s ready to transmit more data.

When this happens, our UART Interrupt Handler will call uart_xmitchars to send the Serial Buffer.

(Which loops back to steps above)

Now we do Serial Input…

§3 Serial Input in NuttX QEMU

What happens when we type something in NuttX QEMU?

Typing something in the Serial Console will trigger a UART Interrupt…

$%^&
riscv_doirq: irq=35
#*
ADEFa
$%&
riscv_doirq: irq=8

(See the Complete Log)

That triggers a call to…

• [$] exception_common (RISC-V Exception Handler) which calls…

• [%^&] riscv_dispatch_irq (Dispatch QEMU Interrupt), which calls…

• riscv_doirq (Dispatch RISC-V Interrupt), which calls…

• irq_dispatch (Dispatch NuttX Interrupt), which calls…

• [#] u16550_interrupt (UART Interrupt Handler), which calls…

• uart_recvchars (write to Serial Receive Buffer)

Looks complicated, but that’s how Serial I/O works with Buffering and Interrupts in NuttX!

Why 2 Interrupts? IRQ 35 and IRQ 8?

• NuttX IRQ 35 (RISC-V IRQ 10) is the QEMU UART Interrupt that’s triggered when a character is received

  (That’s us typing something)

• NuttX IRQ 8 (RISCV_IRQ_ECALLU) happens when a NuttX App makes a System Call to NuttX Kernel

  (NuttX Shell calls NuttX Kernel to do something)

Now we compare the above QEMU Log with Star64…

§4 Star64 vs QEMU Serial I/O

Earlier we said that NuttX Star64 couldn’t print to Serial Console. Why?

Let’s observe the Star64 Debug Log (and compare with QEMU Log)…

up_enable_irq:
  irq=57
  extirq=32
  RISCV_IRQ_EXT=25

(See the Complete Log)

NuttX Star64 now enables UART Interrupts at NuttX IRQ 57. (RISC-V IRQ 32)

(More about this in the next section)

We see NuttX Shell making System Calls to NuttX Kernel (via NuttX IRQ 8)…

$%&riscv_doirq: irq=8
...
$%&riscv_doirq: irq=8

Then NuttX Shell tries to print to Serial Output…

uart_write (0xc0015338):
0000  6e 73 68 3e 20                                   nsh>            

AAAAAD

From the QEMU Log, we know that uart_write (print to Serial Console) calls…

• [A] uart_putxmitchar (write to Serial Buffer), which calls…

• [D] uart_xmitchars (print the Serial Buffer), but wait…

Something looks different from QEMU?

Yeah these are missing from the Star64 Log…

• [E] uart_txready (check for UART ready) and…

• [F] u16550_send (write to UART output)

Which means that UART is NOT ready to transmit!

(Hence we can’t write to UART Output)

What happens next?

We said earlier that UART will trigger a Transmit Ready Interrupt when it’s ready to transmit more data.

(Which triggers our UART Interrupt Handler that calls uart_xmitchars to send data)

But NuttX IRQ 57 is never triggered in the Star64 Log!

Thus there’s our problem: NuttX on Star64 won’t print to the Serial Output because UART Interrupts are never triggered.

(NuttX Star64 won’t respond to keypresses either)

There’s a problem with our Interrupt Controller?

We checked the Star64 Interrupt Settings and Memory Map…

• irq.h: Map RISC-V IRQ to NuttX IRQ

• qemu_rv_memorymap.h: PLIC Address

• board_memorymap.h: Memory Map

• knsh64/defconfig: Memory Configuration

  (See the JH7110 U74 Memory Map)

But everything looks OK!

Maybe we got the wrong UART IRQ Number? Let’s verify…

Global Interrupts for JH7110

§5 JH7110 UART Interrupt

Is the UART IRQ Number correct?

From the JH7110 UART Doc, the UART Interrupt is at RISC-V IRQ 32…

Which becomes NuttX IRQ 57. (Offset by 25)

(See RISCV_IRQ_SEXT)

That’s why we configure the NuttX UART IRQ like so: nsh/defconfig

CONFIG_16550_UART0_IRQ=57

Is it the same UART IRQ as Linux?

We dumped the Linux Device Tree for JH7110…

## Convert Device Tree to text format
dtc \
  -o jh7110-visionfive-v2.dts \
  -O dts \
  -I dtb \
  jh7110-visionfive-v2.dtb

(dtc decompiles a Device Tree)

Linux Port UART0 is indeed at RISC-V IRQ 32: jh7110-visionfive-v2.dts

serial@10000000 {
  compatible = "snps,dw-apb-uart";
  reg = <0x00 0x10000000 0x00 0x10000>;
  reg-io-width = <0x04>;
  reg-shift = <0x02>;
  clocks = <0x08 0x92 0x08 0x91>;
  clock-names = "baudclk\0apb_pclk";
  resets = <0x21 0x53 0x21 0x54>;
  interrupts = <0x20>;
  status = "okay";
  pinctrl-names = "default";
  pinctrl-0 = <0x24>;
};

What about the Global Interrupt Number?

According to JH7110 Interrupt Connections, u0_uart is at global_interrupts[27] (pic above).

Which is correct because the SiFive U74 Manual (Page 198) says that…

RISC-V IRQ = Global Interrupt Number + 5

Maybe IRQ 32 is too high? (QEMU UART IRQ is only 10)

The doc on JH7110 Interrupt Connections says that Global Interrupts are numbered 0 to 126. (127 total interrupts)

That’s a lot more than NuttX QEMU can handle. So we patched it…

• irq.h: Increase to 127 IRQs

• qemu_rv_irq.c: Initialise 127 IRQs

Though some parts are hardcoded to 64 IRQs. (Needs more fixing)

Let’s talk about the Interrupt Controller…

§6 Platform-Level Interrupt Controller

What’s this PLIC?

Inside JH7110, the Platform-Level Interrupt Controller (PLIC) handles Global Interrupts (External Interrupts) that are triggered by Peripherals. (Like the UART Controller)

• SiFive U74-MC Core Complex Manual

  “Platform-Level Interrupt Controller” (Page 192)

• RISC-V PLIC Specification

  (PLIC works like Arm’s Global Interrupt Controller)

The pic above shows how we may configure the PLIC to Route Interrupts to each of the 5 RISC-V Cores.

Wow there are 5 RISC-V Cores in JH7110?

According to the SiFive U74 Manual (Page 96), these are the RISC-V Cores in JH7110…

• Hart 0: S7 Monitor Core (RV64IMACB)

• Harts 1 to 4: U74 Application Cores (RV64GCB)

NuttX boots on the First Application Core, which is Hart 1.

(Though we pass the Hart ID to NuttX as Hart 0, since NuttX expects Hart ID to start at 0)

So we’ll route Interrupts to Hart 1?

Yep, later we might add Harts 2 to 4 when we boot NuttX on the other Application Cores.

(But probably not Hart 0, since it’s a special limited Monitor Core)

Let’s check our PLIC Code in NuttX…

§6.1 Memory Map

How do we program the PLIC?

We write to the PLIC Registers defined in the SiFive U74 Manual (Page 193)…

Above are the PLIC Registers for Interrupt Priorities (Page 198) and Interrupt Pending Bits (Page 198).

(Yep PLIC supports 136 Interrupts)

To enable (or disable) Interrupts, we write to the Interrupt Enable Registers (Page 199)…

This says that each Hart (RISC-V Core) can be programmed individually to receive Interrupts, in Machine or Supervisor Modes.

(We’ll only do Hart 1 in Supervisor Mode)

The Priority Threshold (Page 200) works like an Interrupt Mask, it suppresses Lower Priority Interrupts…

Things can get messy when Multiple Harts service Interrupts at the same time.

That’s why we service Interrupts in 3 steps…

1. Claim the Interrupt

2. Handle the Interrupt

3. Mark the Interrupt as Complete

(If we don’t mark the Interrupt as Complete, we won’t receive any subsequent Interrupts)

These are the PLIC Registers to Claim and Complete Interrupts (Page 201)…

Based on the above Memory Map, we set the PLIC Addresses in NuttX to use Hart 1 in Supervisor Mode: jh7110_plic.h

// PLIC Addresses for NuttX Star64
// (Hart 1 in Supervisor Mode)
// | 0x0C00_0004 | RW | Source 1 priority
// | 0x0C00_1000 | RO | Start of pending array
#define QEMU_RV_PLIC_PRIORITY (QEMU_RV_PLIC_BASE + 0x000000)
#define QEMU_RV_PLIC_PENDING1 (QEMU_RV_PLIC_BASE + 0x001000)

// NuttX Star64 runs in Supervisor Mode
#ifdef CONFIG_ARCH_USE_S_MODE

// | 0x0C00_2100 | RW | Start Hart 1 S-Mode Interrupt Enables
#define QEMU_RV_PLIC_ENABLE1 (QEMU_RV_PLIC_BASE + 0x002100)
#define QEMU_RV_PLIC_ENABLE2 (QEMU_RV_PLIC_BASE + 0x002104)

// | 0x0C20_2000 | RW | Hart 1 S-Mode Priority Threshold
// | 0x0C20_2004 | RW | Hart 1 S-Mode Claim / Complete 
#define QEMU_RV_PLIC_THRESHOLD (QEMU_RV_PLIC_BASE + 0x202000)
#define QEMU_RV_PLIC_CLAIM     (QEMU_RV_PLIC_BASE + 0x202004)

FYI these are the earlier PLIC Settings for NuttX QEMU (which runs in Machine Mode): qemu_rv_plic.h

// Previously for NuttX QEMU:
// #define QEMU_RV_PLIC_ENABLE1   (QEMU_RV_PLIC_BASE + 0x002080)
// #define QEMU_RV_PLIC_ENABLE2   (QEMU_RV_PLIC_BASE + 0x002084)
// #define QEMU_RV_PLIC_THRESHOLD (QEMU_RV_PLIC_BASE + 0x201000)
// #define QEMU_RV_PLIC_CLAIM     (QEMU_RV_PLIC_BASE + 0x201004)

Let’s figure out QEMU_RV_PLIC_BASE…

What’s the PLIC Base Address?

From JH7110 U74 Memory Map, the Base Addresses are…

Which are correct in NuttX: jh7110_memorymap.h

// Base Address of PLIC
#define QEMU_RV_PLIC_BASE  0x0c000000

§6.2 Initialise Interrupts

In NuttX, this is how we initialise the PLIC Interrupt Controller: jh7110_irq.c

// Initialise Interrupts for Star64
void up_irqinitialize(void) {

  // Disable Machine interrupts 
  up_irq_save();

  // Disable all global interrupts 
  // TODO: Extend to PLIC Interrupt ID 136
  putreg32(0x0, QEMU_RV_PLIC_ENABLE1);
  putreg32(0x0, QEMU_RV_PLIC_ENABLE2);

  // Set priority for all global interrupts to 1 (lowest) 
  // TODO: Extend to PLIC Interrupt ID 136
  for (int id = 1; id <= NR_IRQS; id++) {
    putreg32(
      1,  // Register Value
      (uintptr_t)(QEMU_RV_PLIC_PRIORITY + 4 * id)  // Register Address
    );
  }

  // Set irq threshold to 0 (permits all global interrupts) 
  putreg32(0, QEMU_RV_PLIC_THRESHOLD);

  // Attach the common interrupt handler 
  riscv_exception_attach();

  // And finally, enable interrupts 
  up_irq_enable();
}

(up_irq_save is defined here)

The code above calls up_irq_enable to enable RISC-V Interrupts: jh7110_irq.c

// Enable Interrupts
irqstate_t up_irq_enable(void) {

  // Enable external interrupts (sie) 
  SET_CSR(CSR_IE, IE_EIE);

  // Read and enable global interrupts (sie) in sstatus 
  irqstate_t oldstat = READ_AND_SET_CSR(CSR_STATUS, STATUS_IE);
  return oldstat;
}

(SET_CSR is defined here)

(READ_AND_SET_CSR is defined here)

§6.3 Enable Interrupts

To enable a specific External Interrupt (like for UART), we configure PLIC to forward the External Interrupt to Hart 1 in Supervisor Mode: jh7110_irq.c

// Enable the IRQ specified by 'irq'
void up_enable_irq(int irq) {

  // For Software Interrupt:
  // Read sstatus and set Software Interrupt Enable in sie 
  if (irq == RISCV_IRQ_SOFT) {
    SET_CSR(CSR_IE, IE_SIE);

  // For Timer Interrupt:
  // Read sstatus and set Timer Interrupt Enable in sie 
  } else if (irq == RISCV_IRQ_TIMER) {
    SET_CSR(CSR_IE, IE_TIE);

  // For External Interrupts:
  // Set Enable bit for the IRQ 
  // TODO: Extend to PLIC Interrupt ID 136
  } else if (irq > RISCV_IRQ_EXT) {
    int extirq = irq - RISCV_IRQ_EXT;
    if (0 <= extirq && extirq <= 63) {
      modifyreg32(
        QEMU_RV_PLIC_ENABLE1 + (4 * (extirq / 32)),  // Address
        0,  // Clear Bits
        1 << (extirq % 32)  // Set Bits
      );
    } else { PANIC(); }
  }
}

(SET_CSR is defined here)

§6.4 Claim and Complete Interrupts

Remember that we service External Interrupts in 3 steps…

1. Claim the Interrupt

2. Handle the Interrupt

3. Mark the Interrupt as Complete

This is how we do it: jh7110_irq_dispatch.c

// Dispatch the RISC-V Interrupt
void *riscv_dispatch_irq(uintptr_t vector, uintptr_t *regs) {

  // For External Interrupts:
  // Claim the Interrupt
  int irq = (vector >> RV_IRQ_MASK) | (vector & 0xf);
  if (RISCV_IRQ_EXT == irq) {
    // Add the value to NuttX IRQ which is offset to the mext 
    uintptr_t val = getreg32(QEMU_RV_PLIC_CLAIM);
    irq += val;
  }

  // For External Interrupts:
  // Call the Interrupt Handler
  if (RISCV_IRQ_EXT != irq) {
    regs = riscv_doirq(irq, regs);
  }

  // For External Interrupts:
  // Mark the Interrupt as Complete
  if (RISCV_IRQ_EXT <= irq) {
    putreg32(
      irq - RISCV_IRQ_EXT,  // Register Value
      QEMU_RV_PLIC_CLAIM    // Register Address
    );
  }
  return regs;
}

(riscv_doirq is defined here)

There’s also a Core-Local Interruptor (CLINT) (Page 185) that handles Software Interrupt and Timer Interrupt. But we won’t cover it today. (Pic below)

TODO: Do we need to handle CLINT?

Let’s check that the RISC-V Interrupts are delegated correctly…

§7 Delegate Machine-Mode Interrupts to Supervisor-Mode

Why do we delegate Interrupts?

According to the SiFive U74 Manual (Page 176)…

“By default, all Traps are handled in Machine Mode”

“Machine Mode Software can selectively delegate Interrupts and Exceptions to Supervisor Mode by setting the corresponding bits in mideleg and medeleg CSRs”

NuttX runs in Supervisor Mode, so we need to be sure that the Interrupts have been delegated correctly to Supervisor Mode…

Or our UART Interrupt Handler will never be called!

What’s this “Machine Mode Software”? Who controls the Delegation?

On Star64, OpenSBI (Supervisor Binary Interface) boots in Machine Mode and controls the Delegation of Interrupts.

From the OpenSBI Log, we see the value of mideleg (“Delegate Machine Interrupt”)…

Boot HART MIDELEG:
  0x0222
Boot HART MEDELEG:
  0xb109

What does mideleg say?

(Ring-ding-ding-ding-dingeringeding!)

mideleg is defined by the following bits: csr.h

// Bit Definition for mideleg
#define MIP_SSIP (0x1 << 1)  // Delegate Software Interrupt
#define MIP_STIP (0x1 << 5)  // Delegate Timer Interrupt
#define MIP_MTIP (0x1 << 7)  // Delegate Machine Timer Interrupt
#define MIP_SEIP (0x1 << 9)  // Delegate External Interrupts

So mideleg 0x0222 means…

• Delegate Software Interrupt to Supervisor Mode (SSIP)

• Delegate Timer Interrupt to Supervisor Mode (STIP)

• Delegate External Interrupts to Supervisor Mode (SEIP)

  (But not MTIP: Delegate Machine Timer Interrupt)

Thus we’re good! OpenSBI has correctly delegated External Interrupts from Machine Mode to Supervisor Mode. (For NuttX to handle)

We’re finally ready to test the Fixed PLIC Code on Star64!

§8 Spurious UART Interrupts

After fixing the PLIC Code for Star64…

Are UART Interrupts OK?

We fixed the PLIC Memory Map in NuttX…

• qemu_rv_plic.h: Fix PLIC Memory Map

  (Route Interrupts to Hart 1 in Supervisor Mode)

  (See the Build Outputs)

Now we see UART Interrupts fired at NuttX IRQ 57 (RISC-V IRQ 32) yay!

uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
up_enable_irq: irq=57, extirq=32, RISCV_IRQ_EXT=25
$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
...
#*$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
#*$%^&riscv_doirq: irq=57
#*$%^&nx_start: CPU0: Beginning Idle Loop

(See the Complete Log)

But we have the Opposite Problem: Too many UART Interrupts!

NuttX gets too busy handling millions of spurious UART Interrupts, and can’t do anything meaningful.

Are they valid UART Interrupts?

Well we see Valid UART Interrupts for…

• UART Transmit Ready (INTID_THRE)

• UART Input Received (INTID_RDA)

But most of the UART Interrupts are for…

• UART Interrupt Status = 0 (INTSTATUS)

Which means that we got interrupted…

FOR NO REASON AT ALL!!!

(UPDATE: We fixed the Spurious UART Interrupts!)

Why? Maybe we should throttle the UART Interrupts?

This definitely needs to be fixed, but for now we made a Quick Hack: Defer the Enabling of UART Interrupts till later.

We comment out the UART Interrupt in u16550_attach: uart_16550.c

// When we attach to UART Interrupt...
static int u16550_attach(struct uart_dev_s *dev) {
  ...
  // Attach to UART Interrupt
  ret = irq_attach(priv->irq, u16550_interrupt, dev);
  if (ret == OK) {
    // Changed this: Don't enable UART Interrupt yet
    // up_enable_irq(priv->irq);

And instead we enable the UART Interrupt in uart_write: serial.c

static ssize_t uart_write(FAR struct file *filep, FAR const char *buffer, size_t buflen) {
  // Added this: Enable UART Interrupt
  // on the 4th print
  static int count = 0;
  if (count++ == 3) {
    up_enable_irq(57); 
  }

Ater hacking, watch what happens when we enter ls at the NuttX Shell…

(Watch the Demo Video on YouTube)

uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
nx_start_application: ret=3
up_exit: TCB=0x404088d0 exiting
up_enable_irq: irq=57, extirq=32, RISCV_IRQ_EXT=25

NuttShell (NSH) NuttX-12.0.3
nsh> ......++.+.
l......s......
................................................

We see the exec_spawn warning…

(Which is OK to ignore)

p.o.s.i.x._.s.p.a.w.n..:. .p.i.d.=...0.x.c.0.2.0.2.9.7.8. .p.a.t.h.=..l.s. .f.i.l.e._.a.c.t.i.o.n.s.=...0.x.c.0.2.0.2.9.8.0. .a.t.t.r.=...0.x.c.0.2.0.2.9.8.8. .a.r.g.v.=...0.x.c.0.2.0.2.a.2.8.
.........................................................
e.x.e.c._.s.p.a.w.n.:. .E.R.R.O..R.:. .F.a.i.l.e.d. .t.o. .l.o.a.d. .p.r.o.g.r.a.m. .'..l.s.'.:. ..-.2.
.......
n.x.p.o.s.i.x._.s.p.a.w.n._.e.x.e.c.:. .E.R.R.O.R.:. .e.x.e.c. .f.a.i.l.e.d.:. ..2.
..............................................................................................................

Followed by the output of ls…

/:............................................................... 
dev........
/.............. 
proc........
/............... 
system.........
/.............................................................
nsh> 

(See the Complete Log)

Yep NuttX Shell works OK on Star64!

But it’s super slow. Each dot is One Million Calls to the UART Interrupt Handler, with UART Interrupt Status INTSTATUS = 0!

(UPDATE: We fixed the Spurious UART Interrupts!)

Why is UART Interrupt triggered repeatedly with INTSTATUS = 0?

Michael Engel says it’s a DesignWare UART issue…

“The JH7110 uses a DesignWare UART component which has some “interesting” extra features. The spurious interrupts are probably caused by a busy interrupt generated by the UART (which is caused by writing the LCR when the chip is busy). If this interrupt is not cleared, you’ll end up in an interrupt storm.“

“See e.g. the Linux DesignWare UART driver for a workaround.”

(Also on Hacker News)

Thanks to the suggestion by Michael Engel, we fixed the Spurious UART Interrupts yay!

We must wait till UART is not busy before setting the Line Control Register (LCR), here’s how…

• “Fix the Spurious UART Interrupts”

We seem to be rushing?

Well NuttX Star64 might get stale and out of sync with NuttX Mainline.

We better chop chop hurry up and merge with NuttX Mainline soon!

(So amazing that NuttX Apps and Context Switching are OK… Even though we haven’t implemented the RISC-V Timer!)

§9 What’s Next

NuttX on Star64 JH7110 RISC-V SBC is almost ready!

• We fixed the Console Output from NuttX Apps

• By tracing through Serial I/O in NuttX QEMU

• And comparing UART I/O for Star64 vs QEMU

• We fixed the NuttX code for Platform-Level Interrupt Controller (PLIC)

• And verified that OpenSBI delegate Machine-Mode Interrupts to Supervisor-Mode

• NuttX Star64 now handles UART Interrupts correctly

• But there’s a new problem: 16550 UART Controller fires too many Spurious Interrupts

• Which we have just fixed: Wait before setting the Line Control Register

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

• Sponsor me a coffee

• Discuss this article on Hacker News

• Discuss this article on Pine64 Forum

• My Current Project: “Apache NuttX RTOS for Ox64 BL808”

• My Other Project: “NuttX for Star64 JH7110”

• Older Project: “NuttX for PinePhone”

• Check out my articles

• RSS Feed

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/plic.md

§10 Appendix: Fix the Spurious UART Interrupts

Earlier we said that NuttX on JH7110 fires too many Spurious UART Interrupts…

• “Spurious UART Interrupts”

This section explains how we fixed the problem.

Based on the JH7110 UART Developing Guide, the StarFive JH7110 SoC uses a Synopsys DesignWare 8250 UART.

(Because that page mentions 8250_dw.c, which is the DesignWare 8250 Driver for Linux)

As documented in the Linux Driver for DesignWare 8250…

“The Synopsys DesignWare 8250 has an extra feature whereby it detects if the LCR is written whilst busy”

“If it is, then a busy detect interrupt is raised, the LCR needs to be rewritten and the uart status register read”

Which is also mentioned by Michael Engel.

This means that before we set the Line Control Register (LCR), we must wait until the UART is not busy.

Thus our fix for JH7110 is to wait for UART before setting LCR. This is how we wait for the UART until it’s not busy: uart_16550.c

#ifdef CONFIG_16550_WAIT_LCR
/***************************************************************************
 * Name: u16550_wait
 *
 * Description:
 *   Wait until UART is not busy. This is needed before writing to LCR.
 *   Otherwise we will get spurious interrupts on Synopsys DesignWare 8250.
 *
 * Input Parameters:
 *   priv: UART Struct
 *
 * Returned Value:
 *   Zero (OK) on success; ERROR if timeout.
 *
 ***************************************************************************/

static int u16550_wait(FAR struct u16550_s *priv)
{
  int i;

  for (i = 0; i < UART_TIMEOUT_MS; i++)
    {
      uint32_t status = u16550_serialin(priv, UART_USR_OFFSET);

      if ((status & UART_USR_BUSY) == 0)
        {
          return OK;
        }

      up_mdelay(1);
    }

  _err("UART timeout\n");
  return ERROR;
}
#endif /* CONFIG_16550_WAIT_LCR */

(UART_USR_OFFSET and UART_USR_BUSY have been added to uart_16550.h)

We wait up to 100 milliseconds: uart_16550.c

/* Timeout for UART Busy Wait, in milliseconds */
#define UART_TIMEOUT_MS 100

Here’s how we wait for UART before setting the Baud Rate in LCR: uart_16550.c

static int u16550_setup(FAR struct uart_dev_s *dev)
{
  ...
#ifdef CONFIG_16550_WAIT_LCR
  /* Wait till UART is not busy before setting LCR */

  if (u16550_wait(priv) < 0)
    {
      _err("UART wait failed\n");
      return ERROR;
    }
#endif /* CONFIG_16550_WAIT_LCR */

  /* Enter DLAB=1 */

  u16550_serialout(priv, UART_LCR_OFFSET, (lcr | UART_LCR_DLAB));

  /* Set the BAUD divisor */

  div = u16550_divisor(priv);
  u16550_serialout(priv, UART_DLM_OFFSET, div >> 8);
  u16550_serialout(priv, UART_DLL_OFFSET, div & 0xff);

#ifdef CONFIG_16550_WAIT_LCR
  /* Wait till UART is not busy before setting LCR */

  if (u16550_wait(priv) < 0)
    {
      _err("UART wait failed\n");
      return ERROR;
    }
#endif /* CONFIG_16550_WAIT_LCR */

  /* Clear DLAB */

  u16550_serialout(priv, UART_LCR_OFFSET, lcr);

We also wait for UART before setting the Break Control in LCR: uart_16550.c

static inline void u16550_enablebreaks(FAR struct u16550_s *priv,
                                       bool enable)
{
  uint32_t lcr = u16550_serialin(priv, UART_LCR_OFFSET);

  if (enable)
    {
      lcr |= UART_LCR_BRK;
    }
  else
    {
      lcr &= ~UART_LCR_BRK;
    }

#ifdef CONFIG_16550_WAIT_LCR
  /* Wait till UART is not busy before setting LCR */

  if (u16550_wait(priv) < 0)
    {
      _err("UART wait failed\n");
    }
#endif /* CONFIG_16550_WAIT_LCR */

  u16550_serialout(priv, UART_LCR_OFFSET, lcr);
}

By default, 16550_WAIT_LCR is Disabled. (Don’t wait for UART)

When 16550_WAIT_LCR is Disabled (default), JH7110 will fire Spurious UART Interrupts and fail to start NuttX Shell (because it’s too busy servicing interrupts)…

• NuttX Log for UART Wait LCR Disabled

Starting kernel ...
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
BCnx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0

When 16550_WAIT_LCR is Enabled, JH7110 will start NuttX Shell correctly…

• NuttX Log for UART Wait LCR Enabled

Starting kernel ...
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
123067BCnx_start: Entry
up_irq_enable: 
up_enable_irq: irq=17
up_enable_irq: RISCV_IRQ_SOFT=17
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
up_enable_irq: irq=57
up_enable_irq: extirq=32, RISCV_IRQ_EXT=25
work_start_lowpri: Starting low-priority kernel worker thread(s)
board_late_initialize: 
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
nx_start_application: ret=3
up_exit: TCB=0x404088d0 exiting
nx_start: CPU0: Beginning Idle Loop
***main

NuttShell (NSH) NuttX-12.0.3
nsh> uname -a
posix_spawn: pid=0xc0202978 path=uname file_actions=0xc0202980 attr=0xc0202988 argv=0xc0202a28
exec_spawn: ERROR: Failed to load program 'uname': -2
nxposix_spawn_exec: ERROR: exec failed: 2
NuttX 12.0.3 2ff7d88 Jul 28 2023 12:35:31 risc-v rv-virt
nsh> ls -l
posix_spawn: pid=0xc0202978 path=ls file_actions=0xc0202980 attr=0xc0202988 argv=0xc0202a28
exec_spawn: ERROR: Failed to load program 'ls': -2
nxposix_spawn_exec: ERROR: exec failed: 2
/:
 dr--r--r--       0 dev/
 dr--r--r--       0 proc/
 dr--r--r--       0 system/
nsh> 

(Regression Test with NuttX QEMU is OK)

Also mentioned in the Synopsys DesignWare DW_apb_uart Databook (Line Control Register, Page 100)…

“DLAB: Divisor Latch Access Bit. Writeable only when UART is not busy (USR[0] is zero)”

So rightfully we should wait for UART whenever we set LCR. Otherwise the LCR Settings might not take effect.

(We already do this in NuttX for PinePhone: a64_serial.c)

This fix has been merged into NuttX Mainline…

• serial/uart_16550: Wait before setting Line Control Register (Synopsys DesignWare 8250)