Star64 JH7110 + NuttX RTOS: RISC-V Privilege Levels and UART Registers

📝 19 Jul 2023

We’re in the super-early stage of porting Apache NuttX Real-Time Operating System (RTOS) to the Pine64 Star64 64-bit RISC-V Single-Board Computer.

(Based on StarFive JH7110, the same SoC in VisionFive2)

In this article we’ll talk about the interesting things that we learnt about RISC-V and Star64 JH7110…

• What are RISC-V Privilege Levels (pic above)

  (And why they make our OS a little more complicated)

• What is NuttX Kernel Mode

  (And how it differs from Flat Mode)

• All about JH7110’s UART Registers

  (And how they are different from other 16550 UARTs)

• Why (naively) porting NuttX from QEMU to Star64 might become really challenging!

  (Thankfully we have the LiteX Arty-A7 and PolarFire Icicle ports)

We begin with the simpler topic: UART…

Star64 JH7110 SBC with Woodpecker USB Serial Adapter

§1 Wait Forever in UART Transmit

Here’s a fun quiz…

This NuttX Kernel Code prints a character to the UART Port. Guess why it waits forever on Star64 JH7110…

// Print a character to UART Port
static void u16550_putc(
  FAR struct u16550_s *priv,  // UART Struct
  int ch                      // Character to be printed
) {
  // Wait for UART Port to be ready to transmit.
  // TODO: This will get stuck!
  while (
    (
      u16550_serialin(   // Read UART Register...
        priv,            //   From UART Base Address...
        UART_LSR_OFFSET  //   At offset of Line Status Register.
      ) & UART_LSR_THRE  // If THRE Flag (Transmit Holding Register Empty)...
    ) == 0               //   Says that Transmit Register is Not Empty...
  );                     //   Then loop until it's empty.

  // Write the character
  u16550_serialout(priv, UART_THR_OFFSET, (uart_datawidth_t)ch);
}

(Source)

Is the UART Base Address correct?

It’s correct, actually. Previously we validated the 16550 UART Base Address for JH7110…

• “UART Controller on Star64”

• “Boot NuttX on Star64”

And we successfully printed to UART…

// Print `A` to the UART Port at
// Base Address 0x1000 0000
*(volatile uint8_t *) 0x10000000 = 'A';

(Previously here)

But strangely it loops forever waiting for the UART Port to be ready!

What’s inside u16550_serialin?

Remember we call u16550_serialin like this…

u16550_serialin(   // Read UART Register...
  priv,            // From UART Base Address...
  UART_LSR_OFFSET  // At offset of Line Status Register
)

Inside u16550_serialin, we read a UART Register at the Offset…

*((FAR volatile uart_datawidth_t *)
  priv->uartbase +  // UART Base Address
  offset);          // Offset of UART Register

What’s the UART Register Offset?

UART_LSR_OFFSET (Offset of Line Status Register) is…

// UART Line Status Register
// is Register #5
#define UART_LSR_INCR 5

// Offset of Line Status Register
// is 16550_REGINCR * 5
#define UART_LSR_OFFSET \
  (CONFIG_16550_REGINCR * UART_LSR_INCR)

16550_REGINCR defaults to 1…

config 16550_REGINCR
  int "Address increment between 16550 registers"
  default 1
  ---help---
    The address increment between 16550 registers.
    Options are 1, 2, or 4.
    Default: 1

Which we copied from NuttX for QEMU Emulator.

Ah but is 16550_REGINCR correct for Star64?

Let’s find out…

§2 UART Registers are Spaced Differently

Earlier we talked about the Address Increment between 16550 UART Registers (16550_REGINCR), which defaults to 1…

config 16550_REGINCR
  int "Address increment between 16550 registers"
  default 1

(Source)

Which means that the 16550 UART Registers are spaced 1 byte apart…

But is it the same for Star64 JH7110?

JH7110 (oddly) doesn’t document the UART Registers, so we follow the trial of JH7110 Docs…

• JH7110 UART Datasheet

• JH7110 UART Developing Guide

• JH7110 UART Device Tree

• JH7110 UART Source Code

From the JH7110 UART Device Tree…

reg = <0x0 0x10000000 0x0 0xl0000>;
reg-io-width = <4>;
reg-shift = <2>;

We see that regshift is 2.

What’s regshift?

According to the JH7110 UART Source Code, this is how we write to a UART Register: 8250_dw.c

// Linux Kernel Driver: Write to 8250 UART Register
static void dw8250_serial_out(struct uart_port *p, int offset, int value) {
  ...
  // Write to UART Register
  writeb(
    value,                     // Register Value
    p->membase +               // UART Base Address plus...
      (offset << p->regshift)  // Offset shifted by `regshift`
  );

(8250 UART is compatible with 16550)

We see that the UART Register Offset is shifted by 2 (regshift).

Which means we multiply the UART Offset by 4!

Thus the UART Registers are spaced 4 bytes apart. And 16550_REGINCR should be 4, not 1!

How to fix 16550_REGINCR?

We fix the NuttX Configuration in “make menuconfig”…

• Device Drivers > Serial Driver Support > 16550 UART Chip support > Address Increment Between 16550 Registers

And change it from 1 to 4: nsh/defconfig

CONFIG_16550_REGINCR=4

Now UART Transmit works perfectly yay! (Pic below)

Starting kernel ...
123067DFHBC
qemu_rv_kernel_mappings: map I/O regions
qemu_rv_kernel_mappings: map kernel text
qemu_rv_kernel_mappings: map kernel data
qemu_rv_kernel_mappings: connect the L1 and L2 page tables
qemu_rv_kernel_mappings: map the page pool
qemu_rv_mm_init: mmu_enable: satp=1077956608
nx_start: Entry

(Source)

Lesson Learnt: 8250 UARTs (and 16550) might work a little differently across Hardware Platforms! (Due to Word Alignment maybe?)

We move on to the tougher topic: Machine Mode vs Supervisor Mode…

§3 Critical Section Doesn’t Return

We ran into another problem when printing to the UART Port…

NuttX on Star64 gets stuck when we enter a Critical Section: uart_16550.c

// Print a character to the UART Port
int up_putc(int ch) {
  ...
  // Enter the Critical Section
  // TODO: This doesn't return!
  flags = enter_critical_section();

  // Print the character
  u16550_putc(priv, ch);

  // Exit the Critical Section
  leave_critical_section(flags);

What’s this Critical Section?

To prevent garbled output, NuttX stops mutiple threads (or interrupts) from printing to the UART Port simultaneously.

It uses a Critical Section to lock the chunk of code above, so only a single thread can print to UART at any time.

But the locking isn’t working… It never returns!

How is it implemented?

When we browse the RISC-V Disassembly of NuttX, we see the implementation of the Critical Section: nuttx.S

int up_putc(int ch) {
  ...
up_irq_save():
nuttx/include/arch/irq.h:675
  __asm__ __volatile__
    40204598: 47a1      li    a5, 8
    4020459a: 3007b7f3  csrrc a5, mstatus, a5
up_putc():
nuttx/drivers/serial/uart_16550.c:1726
  flags = enter_critical_section();

Which has this curious RISC-V Instruction…

// (Atomically) Read and Clear Bits
// in `mstatus` Register
csrrc a5, mstatus, a5

According to the RISC-V Spec, csrrc (Atomic Read and Clear Bits in CSR) will…

• Read the mstatus Register

  (Which is a CSR: Control and Status Register)

• Clear the mstatus bits specified by Register a5 (with value 8)

  (Which is the MIE Bit: Machine Interrupt Enable)

• Return the initial value of mstatus in Register a5

  (Before clearing the bits)

Effectively we’re disabling interrupts, so we won’t possibly switch to another thread.

But we have a problem: NuttX can’t modify the mstatus Register, because of its Privilege Level…

§4 RISC-V Privilege Levels

What’s this Privilege Level?

RISC-V Machine Code runs at three Privilege Levels…

• M: Machine Mode (Most powerful)

• S: Supervisor Mode (Less powerful)

• U: User Mode (Least powerful)

NuttX on Star64 runs in Supervisor Mode. Which doesn’t allow write access to Machine-Mode CSR Registers. (Pic above)

Remember this?

// (Atomically) Read and Clear Bits
// in `mstatus` Register
csrrc a5, mstatus, a5

The “m” in mstatus signifies that it’s a Machine-Mode Register.

That’s why NuttX failed to modify the mstatus!

What’s the equivalent of mstatus for Supervisor Mode?

NuttX should use the sstatus Register instead.

(We should switch all Machine-Mode m Registers to Supervisor-Mode s Registers)

What runs in Machine Mode?

OpenSBI (Supervisor Binary Interface) is the first thing that boots on Star64.

It runs in Machine Mode and starts the U-Boot Bootloader.

(More about OpenSBI)

What about U-Boot Bootloader?

U-Boot Bootloader runs in Supervisor Mode. And starts NuttX, also in Supervisor Mode.

Thus OpenSBI is the only thing that runs in Machine Mode. And can access the Machine-Mode Registers. (Pic above)

(More about U-Boot)

QEMU doesn’t have this problem?

We (naively) copied the code above from NuttX for QEMU Emulator.

But QEMU doesn’t have this problem, because it runs NuttX in (super-powerful) Machine Mode!

Let’s make it work for Star64…

§5 RISC-V Machine Mode becomes Supervisor Mode

Earlier we saw the csrrc instruction…

From whence it came?

// (Atomically) Read and Clear Bits
// in `mstatus` Register
csrrc a5, mstatus, a5

We saw the above RISC-V Assembly emitted by up_putc and enter_critical_section, let’s track it down.

enter_critical_section calls up_irq_save, which is defined as…

// Disable interrupts
static inline irqstate_t up_irq_save(void) {
  ...
  // Read `mstatus` and clear 
  // Machine Interrupt Enable (MIE) in `mstatus`
  __asm__ __volatile__
  (
    "csrrc %0, " __XSTR(CSR_STATUS) ", %1\n"
    : "=r" (flags)
    : "r"(STATUS_IE)
    : "memory"
  );

Ah so CSR_STATUS maps to mstatus?

Yes indeed, CSR_STATUS becomes mstatus: mode.h

// If NuttX runs in Supervisor Mode...
#ifdef CONFIG_ARCH_USE_S_MODE
  // Use Global Status Register 
  // for Supervisor Mode
  #define CSR_STATUS sstatus

#else  // If NuttX runs in Machine Mode...
  // Use Global Status Register 
  // for Machine Mode 
  #define CSR_STATUS mstatus
#endif

…BUT only if NuttX Configuration ARCH_USE_S_MODE is disabled!

So if ARCH_USE_S_MODE is enabled, NuttX will use sstatus instead?

Yep! We need to enable ARCH_USE_S_MODE, so that NuttX will use sstatus (instead of mstatus)…

Which is perfectly hunky dory for RISC-V Supervisor Mode!

We dig around for the elusive (but essential) ARCH_USE_S_MODE…

§6 NuttX Flat Mode becomes Kernel Mode

How to enable ARCH_USE_S_MODE in NuttX?

In the previous section we discovered that we should enable ARCH_USE_S_MODE, so that NuttX will run in RISC-V Supervisor Mode…

// If NuttX runs in Supervisor Mode...
#ifdef CONFIG_ARCH_USE_S_MODE
  // Use Global Status Register 
  // for Supervisor Mode
  #define CSR_STATUS sstatus

#else  // If NuttX runs in Machine Mode...
  // Use Global Status Register 
  // for Machine Mode 
  #define CSR_STATUS mstatus
#endif

(Because Star64 boots NuttX in Supervisor Mode)

Searching NuttX for ARCH_USE_S_MODE gives us this Build Configuration for NuttX Kernel Mode: knsh64/defconfig

CONFIG_ARCH_USE_S_MODE=y

Perfect! Exactly what we need!

Thus we switch the NuttX Build Configuration from Flat Mode to Kernel Mode…

## Configure NuttX for Kernel Mode and build NuttX
tools/configure.sh rv-virt:knsh64
make

## Previously: Configure NuttX for Flat Mode
## tools/configure.sh rv-virt:nsh64

(Complete Steps for Kernel Mode)

What’s this Kernel Mode?

According to the NuttX Docs on Kernel Mode…

“All of the code that executes within the Kernel executes in Privileged, Kernel Mode”

“All User Applications are executed with their own private address environments in Unprivileged, User-Mode”

Hence Kernel Mode is a lot more secure than the normal NuttX Flat Mode, which runs the Kernel and User Applications in the same Unprotected, Privileged Mode.

(More about Kernel Mode)

Does it work?

When we grep for csr Instructions in the rebuilt NuttX Disassembly nuttx.S…

We see (nearly) all Machine-Mode m Registers replaced by Supervisor-Mode s Registers.

No more problems with Critical Section yay!

Let’s eliminate the remaining Machine-Mode Registers…

§7 Initialise RISC-V Supervisor Mode

We rebuilt NuttX from Flat Mode to Kernel Mode…

Why does it still need RISC-V Machine-Mode Registers?

NuttX accesses the RISC-V Machine-Mode Registers during NuttX Startup…

1. NuttX Boot Code calls jh7110_start

   (As explained here)

2. jh7110_start (previously) assumes it’s in Machine Mode

   (Because QEMU boots NuttX in Machine Mode)

3. jh7110_start (previously) initialises the Machine-Mode Registers

   (And some Supervisor-Mode Registers)

4. jh7110_start jumps to jh7110_start_s in Supervisor Mode

5. jh7110_start_s initialises the Supervisor-Mode Registers

   (And starts NuttX)

So we need to remove the Machine-Mode Registers from jh7110_start?

Yep, because NuttX boots in Supervisor Mode on Star64.

(And can’t access the Machine-Mode Registers)

This is how we patched jh7110_start to remove the Machine-Mode Registers: jh7110_start.c

// Called by NuttX Boot Code
// to init System Registers
void jh7110_start(int mhartid) {

  // For the First CPU Core...
  if (0 == mhartid) {

    // Clear the BSS
    qemu_rv_clear_bss();

    // Initialize the per CPU areas
    riscv_percpu_add_hart(mhartid);
  }

  // Disable MMU and enable PMP
  WRITE_CSR(satp, 0x0);
  // Removed: pmpaddr0 and pmpcfg0

  // Set exception and interrupt delegation for S-mode
  // Removed: medeleg and mideleg

  // Allow to write satp from S-mode
  // Set mstatus to S-mode and enable SUM
  // Removed: mstatus

  // Set the trap vector for S-mode
  WRITE_CSR(stvec, (uintptr_t)__trap_vec);

  // Set the trap vector for M-mode
  // Removed: mtvec

  // TODO: Call up_mtimer_initialize
  // https://github.com/apache/nuttx/blob/master/arch/risc-v/src/qemu-rv/qemu_rv_timerisr.c#L151-L210

  // Set mepc to the entry
  // Set a0 to mhartid explicitly and enter to S-mode
  // Removed: mepc

  // Added: Jump to S-Mode Init ourselves
  jh7110_start_s(mhartid);
}

(jh7110_start_s is defined here)

We’re not sure if this is entirely correct… But it’s a good start!

(Yeah we’re naively copying code again sigh)

Now NuttX boots further!

123067DFHBC
qemu_rv_kernel_mappings: map I/O regions
qemu_rv_kernel_mappings: map kernel text
qemu_rv_kernel_mappings: map kernel data
qemu_rv_kernel_mappings: connect the L1 and L2 page tables
qemu_rv_kernel_mappings: map the page pool
qemu_rv_mm_init: mmu_enable: satp=1077956608
Inx_start: Entry
elf_initialize: Registering ELF
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x404069e8

(Source)

But NuttX crashes due to a Semihosting Problem. (Pic above)

riscv_exception: EXCEPTION: Breakpoint. MCAUSE: 0000000000000003, EPC: 0000000040200434, MTVAL: 0000000000000000
riscv_exception: PANIC!!! Exception = 0000000000000003
_assert: Current Version: NuttX  12.0.3 2261b80-dirty Jul 15 2023 20:38:57 risc-v
_assert: Assertion failed panic: at file: common/riscv_exception.c:85 task: Idle Task 0x40200ce6
up_dump_register: EPC: 0000000040200434
up_dump_register: A0: 0000000000000001 A1: 0000000040406778 A2: 0000000000000000 A3: 0000000000000001

(Source)

We’ll find out why in the next article!

• “Star64 JH7110 + NuttX RTOS: RISC-V Semihosting and Initial RAM Disk”

TODO: Port up_mtimer_initialize to Star64

(See the Modified Files)

(See the Build Steps)

(See the Build Outputs)

Semihosting on RISC-V NuttX

§8 Other RISC-V Ports of NuttX

Porting NuttX from QEMU to Star64 looks challenging…

Are there other ports of NuttX for RISC-V?

We found the following NuttX Ports that run in RISC-V Supervisor Mode with OpenSBI.

(They might be good references for Star64 JH7110)

LiteX Arty-A7 boots from OpenSBI to NuttX (but doesn’t call back to OpenSBI)…

(VexRISCV SMP uses a RAM Disk for NuttX Apps)

PolarFire Icicle (based on PolarFire MPFS) runs a copy of OpenSBI inside NuttX (so it boots in Machine Mode before Supervisor Mode)…

(QEMU has an Emulator for PolarFire Icicle)

How to call OpenSBI in NuttX?

We run this ecall to jump from NuttX (in RISC-V Supervisor Mode) to OpenSBI (in RISC-V Machine Mode)…

• riscv_sbi.c: Calling OpenSBI in NuttX

  (How ecall works in OpenSBI)

  (More about OpenSBI)

§9 What’s Next

I hope we learnt a bit more about RISC-V and Star64 JH7110 SBC today…

• RISC-V Privilege Levels

  (Why they make our OS a little more complicated)

• NuttX Kernel Mode

  (How it differs from Flat Mode)

• JH7110’s UART Registers

  (How they are different from other 16550 UARTs)

• Porting NuttX from QEMU to Star64 might become really challenging!

  (Thankfully we have the LiteX Arty-A7 and PolarFire Icicle ports)

Please join me in the next article as we solve the RISC-V Semihosting Problem. (We’ll use an Initial RAM Disk with ROMFS)

• “Star64 JH7110 + NuttX RTOS: RISC-V Semihosting and Initial RAM Disk”

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

• Sponsor me a coffee

• Discuss this article on Hacker News

• Discuss this article on Pine64 Forum

• My Current Project: “Apache NuttX RTOS for Ox64 BL808”

• My Other Project: “NuttX for Star64 JH7110”

• Older Project: “NuttX for PinePhone”

• Check out my articles

• RSS Feed

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/privilege.md