📝 7 Jul 2024
Earlier this year we made a RISC-V Emulator for Ox64 BL808 SBC. Every day we run it for testing the daily build of Apache NuttX RTOS, thanks to our customised TinyEMU RISC-V Emulator (not the small flightless bird)…
Now that NuttX supports Sophgo SG2000 SoC: Let’s create a similar emulator for Pine64 Oz64 SBC (pic above) and Milk-V Duo S…
We take TinyEMU Emulator for Ox64 BL808 SBC
Update the RISC-V Memory Map to match Sophgo SG2000 SoC
Fix the auipc Overflow in TinyEMU Boot Code
We emulate the 16550 UART Controller
By intercepting Reads and Writes to the UART I/O Registers
But TinyEMU supports only 32 Interrupts, we bump up to 64
Eventually we’ll emulate SG2000 Peripherals like GPIO
Right now it’s good enough for Daily Automated Testing of NuttX for SG2000
SG2000 Reference Manual (Page 17)
We begin with the TinyEMU RISC-V Emulator for Ox64 BL808 SBC, and we tweak it for Sophgo SG2000 SoC.
This is how we update the RISC-V Memory Map for SG2000: riscv_machine.c
// Base Addresss of System RAM, Core Local Interrupt Controller (unused)
// And Platform-Level Interrupt Controller
#define RAM_BASE_ADDR 0x80200000ul
#define CLINT_BASE_ADDR 0x74000000ul
#define PLIC_BASE_ADDR 0x70000000ul
(Interrupt Controller is at 0x7000_0000)
Then we build and run TinyEMU Emulator for SG2000…
## Build SG2000 Emulator for macOS
## For Linux: See https://github.com/lupyuen/nuttx-sg2000/blob/main/.github/workflows/sg2000-test.yml#L29-L45
cd $HOME
git clone https://github.com/lupyuen2/sg2000-emulator
cd sg2000-emulator
make \
CFLAGS="-I$(brew --prefix)/opt/openssl/include -I$(brew --prefix)/opt/sdl2/include" \
LDFLAGS="-L$(brew --prefix)/opt/openssl/lib -L$(brew --prefix)/opt/sdl2/lib" \
CONFIG_MACOS=y
## Build NuttX for SG2000
cd $HOME/nuttx
tools/configure.sh milkv_duos:nsh
make
## Omitted: Create the NuttX Image
## See https://lupyuen.github.io/articles/sg2000#appendix-build-nuttx-for-sg2000
## Boot TinyEMU with NuttX for SG2000
## To Exit TinyEMU: Ctrl-A x
wget https://raw.githubusercontent.com/lupyuen/nuttx-sg2000/main/nuttx.cfg
$HOME/sg2000-emulator/temu nuttx.cfg
(nuttx.cfg points to the NuttX Image)
Something baffling appears…
auipc Overflow in Boot CodeWhen we boot NuttX on TinyEMU, it crashes at a curious location (pic above)…
$ sg2000-emulator/temu nuttx.cfg
TinyEMU Emulator for Sophgo SG2000 SoC
raise_exception2:
cause=1
tval= 0xffffffff_80200000
pc= 0xffffffff_80200000
tinyemu:
Illegal instruction, quitting
What just happened?
Our Emulator tried to execute the code at MTVAL 0xffffffff_80200000. And crashed because it’s not a valid address!
The Effy Address looks sus, it seems related to RAM Base Address: 0x80200000
Our Emulator is booting the wrong address?
Apparently. We check the Original TinyEMU Boot Code: riscv_machine.c
// Init the TinyEMU Boot Code
void copy_bios(...) {
...
// `q` points to the Boot Code
q = (uint32_t *)(ram_ptr + 0x1000);
// Load `RAM_BASE_ADDR` into Register T0:
// `auipc t0, RAM_BASE_ADDR`
// `RAM_BASE_ADDR` is 0x80200000
q[0] = 0x297 + RAM_BASE_ADDR - 0x1000;
// Later: Jump to Register T0
To load the RAM Base Address into Register T0: TinyEMU tries to assemble this RISC-V Instruction (into our Boot Code)…
auipc t0, 0x80200000
Maybe auipc has a problem?
We verify with the RISC-V Online Assembler.
When we assemble the auipc instruction above, the Online Assembler fails with an error (pic below)…
Error: lui expression not in range 0..1048575
Error: value of 0000080200000000 too large
for field of 4 bytes at 0000000000000000
Aha 0x8020_0000 is too big to assemble as an auipc Address!
But 0x8020_0000 is a perfectly valid address?
Remember that RISC-V is a RISC Platform after all. Some operations won’t fit into 4 bytes of Machine Code.
We upsize to 8 bytes of Machine Code…
auipc to li in Boot CodeHow else can we load 0x8020_0000 into Register T0?
Instead of aupic (which has size limits), we load our RAM Base Address with the li Instruction…
li t0, 0x80200000
When we feed the above into RISC-V Online Assembler, we see the resulting (8-byte) Machine Code…
4010029b addiw t0, zero, 1025
01529293 slli t0, t0, 0x15
That’s because li is a Pseudo-Instruction that expands into two RISC-V Instructions…
addiw: Add zero to 1025 and store into T0
slli: Shift-Left T0 by 0x15 bits
Producing: 1025 << 0x15 = 0x8020_0000
Thus we copy the above Machine Code into our TinyEMU Boot Code: riscv_machine.c
// Init the TinyEMU Boot Code
void copy_bios(...) {
...
// Load `RAM_BASE_ADDR` into Register T0:
// `li t0, 0x80200000`
// Which is assembled as...
q[pc++] = 0x4010029b; // addiw t0, zero, 1025
q[pc++] = 0x01529293; // slli t0, t0, 0x15
// TODO: Remove the hardcoding of 0x80200000
Our Emulator now boots NuttX correctly at 0x8020_0000!
So TinyEMU will jump directly to 0x8020_0000?
Not quite. TinyEMU boots in RISC-V Machine Mode, but NuttX expects to boot in RISC-V Supervisor Mode! (Pic above)
That’s why we customised the TinyEMU Boot Code, so that it jumps from Machine Mode to Supervisor Mode via the MRET Instruction. (Pic below)
(Which will start NuttX at 0x8020_0000)
Nothing appears when we boot NuttX?
$ sg2000-emulator/temu nuttx.cfg
TinyEMU Emulator for Sophgo SG2000 SoC
[...crickets...]
That’s because we haven’t emulated the 16550 UART Controller in TinyEMU!
To figure out what’s needed, we refer to the 16550 UART Driver in NuttX: uart_16550.c
// To send one byte to UART Output...
void u16550_send(struct uart_dev_s *dev, int ch) {
...
// We write the byte to the 16550 UART Register...
u16550_serialout(
priv, // UART Device
UART_THR_OFFSET, // UART Register: Transmit Holding Register (THR)
ch // Byte to be sent
);
}
// To check if the UART Transmit FIFO is ready...
bool u16550_txready(struct uart_dev_s *dev) {
...
// We read the 16550 UART Register...
return ((
u16550_serialin(
priv, // UART Device
UART_LSR_OFFSET // UART Register: Line Status Register (LSR)
) & UART_LSR_THRE // And check the THRE Bit (Transmit Holding Register Empty)
) != 0);
}
Which says that…
UART_THR: Transmit Holding Register
Will receive the byte that NuttX is transmitting
UART_LSR: Line Status Register
Will be read by NuttX to check if the Transmit FIFO is Ready
UART_LSR_THRE: Transmit Holding Register Empty
This is the bit in UART_LSR that will indicate whether Transmit FIFO is Ready
Let’s fix this in TinyEMU…
How will we emulate the 16550 UART Registers in TinyEMU?
When TinyEMU needs to Read or Write a Memory Address, it will call the functions below.
This is how we Intercept the Memory Writes to emulate the UART Output Register (Transmit Holding): riscv_cpu.c
// TinyEMU calls this function to execute Memory Writes
int target_write_slow(...) {
...
// If TinyEMU is writing to this address...
switch(paddr) {
// Address is UART Transmit Holding Register
case UART0_BASE_ADDR + UART_THR_OFFSET:
// Print the character that's written by NuttX
char buf[1] = { val };
print_console(NULL, buf, 1);
And this is how we Intercept the Memory Reads to emulate the UART Status Register (Line Status): riscv_cpu.c
// TinyEMU calls this function to execute Memory Reads
int target_read_slow(...) {
...
// If TinyEMU is reading from this address...
switch(paddr) {
// Address is UART Line Status Register
case UART0_BASE_ADDR + UART_LSR_OFFSET:
// Always tell NuttX that
// Transmit Holding Register is Empty
ret = UART_LSR_THRE;
// If UART Input is available:
// Tell NuttX that Receive Data is Available
if (read_input() != 0) {
ret |= UART_LSR_DR;
}
(More about UART Input in a while)
What happens when we run this?
Now we see the NuttX Shell yay!
$ sg2000-emulator/temu nuttx.cfg
TinyEMU Emulator for Sophgo SG2000 SoC
NuttShell (NSH) NuttX-12.5.1
nsh>
NuttX Shell won’t accept input, let’s fix it…
What about UART Input? How to emulate the UART Registers?
Again we refer to the NuttX Driver for 16550 UART, to discover the inner workings of UART Input: uart_16550.c
// NuttX Interrupt Handler for 16550 UART
int u16550_interrupt(int irq, FAR void *context, FAR void *arg) {
...
// Loop until no characters to be transferred
for (passes = 0; passes < 256; passes++) {
// Get the current UART Status
status = u16550_serialin(priv, UART_IIR_OFFSET);
// If no Pending Interrupts, exit
if ((status & UART_IIR_INTSTATUS) != 0) { break; }
// Handle the UART Interrupt
switch (status & UART_IIR_INTID_MASK) {
// If UART Input is available,
// receive the Input Data
case UART_IIR_INTID_RDA:
uart_recvchars(dev);
break;
...
}
// Receive one character from UART.
// Called by the above Interrupt Handler.
int u16550_receive(struct uart_dev_s *dev, unsigned int *status) {
...
// Return the Line Status and Receive Buffer
*status = u16550_serialin(priv, UART_LSR_OFFSET);
rbr = u16550_serialin(priv, UART_RBR_OFFSET);
return rbr;
}
Which says that (pic above)…
UART_IIR: Interrupt ID Register
Should return INTID_RDA (data available)
Followed by IIR_INTSTATUS (no more data)
UART_LSR: Line Status Register
Should return LSR_DR (data available)
Followed by 0 (no more data)
UART_RBR: Receiver Buffer Register
Should return the UART Input Data
Thus we emulate the above UART Registers in TinyEMU: riscv_cpu.c
// TinyEMU calls this function to execute Memory Reads
int target_read_slow(...) {
...
// If TinyEMU is reading from this address...
switch(paddr) {
// Address is UART Interrupt ID Register
case UART0_BASE_ADDR + UART_IIR_OFFSET:
// If the Input Buffer is Empty:
// Then Receive Data is NOT Available
if (read_input() == 0) {
ret = UART_IIR_INTSTATUS;
} else {
// Otherwise Receive Data is Available
ret = UART_IIR_INTID_RDA;
}
break;
// Address is UART Receive Buffer Register
case UART0_BASE_ADDR + UART_RBR_OFFSET:
// Return the Input Buffer
ret = read_input();
// Clear the Input Buffer and UART Interrupt
set_input(0);
virtio_ack_irq(NULL);
break;
What about UART_LSR? (Line Status Register)
Check out our earlier implementation of target_read_slow.
So UART Input is all hunky dory?
Not quite. When we press a key: TinyEMU crashes with a Segmentation Fault (pic above)…
$ sg2000-emulator/temu nuttx.cfg
NuttShell (NSH) NuttX-12.5.1
nsh>
[1] segmentation fault
Maybe we got the Wrong Interrupt Number?
We check our NuttX Config for UART Interrupt: nsh/defconfig
## NuttX IRQ for UART0 is 69
CONFIG_16550_UART0_IRQ=69
Which says…
RISC-V IRQ is 69 - 25 = 44
Which matches the SG2000 Reference Manual (Page 13)…
“3.1 Interrupt Subsystem”
“Int #44: UART0”
And matches the VirtIO IRQ in TinyEMU: riscv_machine.c
// UART0 IRQ becomes VirtIO IRQ for TinyEMU
#define VIRTIO_IRQ 44
Something seriously sinister is wrecking our rojak…
After lots of headscratching: We discover that TinyEMU supports only 32 RISC-V External Interrupts. Which is too few for our UART Controller! (IRQ 44)
Here’s the original definition in TinyEMU: riscv_machine.c
// TinyEMU defines the RISC-V Virtual Machine
typedef struct RISCVMachine {
...
// Platform-Level Interrupt Controller:
// Only 32 External RISC-V Interrupts!
uint32_t plic_pending_irq; // 32 Pending Interrupts
uint32_t plic_served_irq; // 32 Served Interrupts
IRQSignal plic_irq[32]; // 32 Interrupt Signals
...
} RISCVMachine;
Therefore we increase the number of TinyEMU Interrupts from 32 to 64…
Finally NuttX Shell runs OK yay!
$ sg2000-emulator/temu nuttx.cfg
TinyEMU Emulator for Sophgo SG2000 SoC
virtio_console_init
Patched DCACHE.IALL (Invalidate all Page Table Entries in the D-Cache) at 0x80200a28
Patched SYNC.S (Ensure that all Cache Operations are completed) at 0x80200a2c
Found ECALL (Start System Timer) at 0x8020b2c6
Patched RDTIME (Read System Time) at 0x8020b2cc
elf_len=0
virtio_console_resize_event
ABC
NuttShell (NSH) NuttX-12.5.1
nsh> uname -a
NuttX 12.5.1 50fadb93f2 Jun 18 2024 09:20:31 risc-v milkv_duos
nsh>
SG2000 Emulator seems slower than Ox64 BL808 Emulator?
Yeah probably because SG2000 runs on MTIMER Frequency of 25 MHz.
When we execute sleep 10, it completes in 25 seconds. We might need to adjust the TinyEMU System Timer.
(CPU-bound Operations like getprime won’t have this timing delay)
Where’s the rest of our SG2000 Emulator?
Yeah we need to emulate the SG2000 Peripherals: GPIO, I2C, SPI, …
Based on the SG2000 Reference Manual (Page 721): We’ll probably emulate SG2000 GPIO Controller like this: riscv_cpu.c
// TinyEMU calls this function to execute Memory Writes
int target_write_slow(...) {
...
// If TinyEMU is writing to this address...
switch(paddr) {
// Address is GPIOA Base Address
// with Offset 0 (GPIO_SWPORTA_DR)
case 0x03020000:
// Check if GPIOA1 is Off or On (Bit 1)
// `val` is the value written by NuttX to 0x03020000
const uint8_t gpio = 1;
const uint32_t mask = (1 << gpio);
const char b =
((val & mask) == 0)
? '0' : '1';
// Send an Emulator Notification to the Console:
// {"nuttxemu":{"gpioa1":1}}
char notify[] = "{\"nuttxemu\":{\"gpioa1\":0}}\r\n";
notify[strlen(notify) - 5] = b;
print_console(NULL, notify, strlen(notify));
What’s this nuttxemu?
nuttxemu will be printed on the TinyEMU Console to notify the caller that GPIOA1 is set to On or Off.
When we run TinyEMU in a Web Browser (via WebAssembly), our Web Browser can intercept this notification and visualise a Simulated LED on GPIOA1. (Pic above)
Why are we doing all this?
SG2000 Emulator will be helpful for testing NuttX Drivers and App, without a Real SBC
We’re running SG2000 Emulator for Daily Automated Testing at GitHub Actions (pic above): sg2000-test.yml
## Build the SG2000 Emulator
git clone https://github.com/lupyuen2/sg2000-emulator
cd sg2000-emulator
make
## Download the NuttX Daily Build for SG2000
date=2024-07-04
repo=https://github.com/lupyuen/nuttx-sg2000
release=$repo/releases/download/nuttx-sg2000-$date
wget $release/Image
wget $release/nuttx.hash
## Download the NuttX Test Script
wget $repo/raw/main/nuttx.cfg
wget $repo/raw/main/nuttx.exp
## Run the NuttX Test Script
chmod +x nuttx.exp
./nuttx.exp
Which calls this Expect Script to execute OSTest (and verify that everything is hunky dory): nuttx.exp
#!/usr/bin/expect
## Expect Script for Testing NuttX with SG2000 Emulator
## Wait at most 300 seconds
set timeout 300
## For every 1 character sent, wait 0.01 milliseconds
set send_slow {1 0.01}
## Start the SG2000 Emulator
spawn ./temu nuttx.cfg
## Wait for the prompt and enter `uname -a`
expect "nsh> "
send -s "uname -a\r"
## Wait for the prompt and enter `ostest`
expect "nsh> "
send -s "ostest\r"
## Check the response...
expect {
## If we see this message, exit normally
"ostest_main: Exiting with status 0" { exit 0 }
## If timeout, exit with an error
timeout { exit 1 }
}
What about QEMU Emulator?
QEMU Emulator is way too complex to customise for SG2000. Instead we’re running QEMU Emulator for Daily Testing of NuttX QEMU.
(Also based on GitHub Actions with Expect Scripting)
Isn’t it safer to run Daily Tests on a Real SG2000 SBC?
Oh yes we’re doing it too! (Pic below) Check out the article…
“Daily Automated Testing for Milk-V Duo S RISC-V SBC (IKEA TRETAKT / Apache NuttX RTOS)”
Creating the SG2000 Emulator… Doesn’t look so hard?
Yeah I’m begging all RISC-V SoC Makers: Please provide a Software Emulator for your RISC-V SoC! 🙏
Just follow the steps in this article to create your RISC-V Emulator. Some SoC Peripherals might be missing, but a Barebones Emulator is still super helpful for porting, booting and testing any Operating System. 🙏 🙏 🙏
Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…