Preparing a Pull Request for Apache NuttX RTOS

📝 28 Nov 2022

Typical Pull Request for Apache NuttX RTOS

(Watch the Video on YouTube)

This article explains how I prepared my Pull Requests for submission to Apache NuttX RTOS. So if we’re contributing code to NuttX, just follow these steps and things will (probably) go Hunky Dory!

(Like the fish)

Before we begin, please swim over to the official Development Workflow for NuttX…

Ready? Let’s dive in! (Like the fish)

Create Fork

§1 NuttX Repositories

We begin by creating our forks for the nuttx and apps repositories…

  1. Browse to NuttX Repository

    github.com/apache/nuttx

    Click “Fork” to create our fork. (Pic above)

    Click “Actions” and enable workflows…

    Enable Workflows

    (This will check that our code compiles OK at every commit)

  2. Do the same for the NuttX Apps Repository

    github.com/apache/nuttx-apps

    Click “Fork” to create our fork.

    Click “Actions” and enable workflows.

  3. As a principle, let’s keep our master branch always in sync with the NuttX Mainline master branch.

    (This seems cleaner for syncing upstream updates into our repo)

    Let’s create a branch to make our changes…

  4. In our NuttX Repository, click master.

    Enter the name of our new branch.

    Click “Create Branch

    Create Branch

    (I named my branch gic for Generic Interrupt Controller)

  5. Do the same for our NuttX Apps Repository

    (Because we should sync nuttx and apps too)

  6. Download the new branches of our nuttx and apps repositories…

    ## Download the "gic" branch of "lupyuen2/wip-nuttx"
    ## TODO: Change the branch name and repo URLs
    mkdir nuttx
    cd nuttx
    git clone \
      --branch gic \
      https://github.com/lupyuen2/wip-nuttx \
      nuttx
    git clone \
      --branch gic \
      https://github.com/lupyuen2/wip-nuttx-apps \
      apps
    

§2 Build and Test

We’re ready to code!

  1. Consider breaking our Pull Request into Smaller Pull Requests.

    This Pull Request implements One Single Feature (Generic Interrupt Controller)…

    “arch/arm64: Add support for Generic Interrupt Controller Version 2”

    That’s called by another Pull Request…

    “arch/arm64: Add support for PINE64 PinePhone”

    Adding a NuttX Arch (SoC) and Board might need 3 Pull Requests (or more)…

    “Add the NuttX Arch and Board”

  2. Modify the code in nuttx and apps to implement our awesome new feature.

    (Be careful with CMake and CMakeLists.txt)

    (Also watch out for NuttX Config defconfig)

  3. Build and test the modified code.

    (I configured F1 in VSCode to run this Build Script)

    (Testing the Docker Image for NuttX CI)

  4. Capture the Output Log and save it as a GitHub Gist

    “NuttX QEMU Log for Generic Interrupt Controller”

    We’ll add this to our Pull Request. Test Logs are super helpful for NuttX Maintainers!

    (Because we can’t tell which way the train went… Unless we have the Test Logs!)

  5. Commit the modified code to our repositories.

    Sometimes the GitHub Actions Workflow will fail with a strange error (like below). Just re-run the failed jobs. (Like this)

    Error response from daemon:
    login attempt to https://ghcr.io/v2/
    failed with status: 503 Service Unavailable"
    

§2.1 Regression Test

Will our modified code break other parts of NuttX?

That’s why it’s good to run a Regression Test (if feasible) to be sure that other parts of NuttX aren’t affected by our modified code.

For our Pull Request…

We tested with QEMU Emulator our new implementation of Generic Interrupt Controller v2…

tools/configure.sh qemu-armv8a:nsh_gicv2 ; make ; qemu-system-aarch64 ...

(See the NuttX QEMU Log)

And for Regression Testing we tested the existing implementation of Generic Interrupt Controller v3…

tools/configure.sh qemu-armv8a:nsh ; make ; qemu-system-aarch64 ...

(See the NuttX QEMU Log)

Remember to capture the Output Log, we’ll add it to our Pull Request.

(Yeah it will be hard to run a Regression Test if it requires hardware that we don’t have)

§2.2 Documentation

Please update the Documentation. The Documentation might be in a Text File

Or in the Official NuttX Docs

To update the Official NuttX Docs, follow the instructions here…

For macOS: We may need to use “brew install pipenv” instead of “pip install pipenv”. And we may need “pip install setuptools” to fix “No module named pkg_resources”

## Build NuttX Docs on macOS
cd nuttx/Documentation
brew install pipenv
pipenv install
pipenv shell
pip install setuptools
rm -r _build
make html

(See the Log)

Check Coding Style with nxstyle

§3 Check Coding Style

Our NuttX Code will follow this Coding Standard…

NuttX provides a tool nxstyle that will check the Coding Style of our source files…

## Compile nxstyle
## TODO: Change "$HOME/nuttx" to our NuttX Project Folder
gcc -o $HOME/nxstyle $HOME/nuttx/nuttx/tools/nxstyle.c

## Check coding style for our modified source files
## TODO: Change the file paths
$HOME/nxstyle $HOME/nuttx/nuttx/arch/arm64/Kconfig
$HOME/nxstyle $HOME/nuttx/nuttx/arch/arm64/src/common/Make.defs
$HOME/nxstyle $HOME/nuttx/nuttx/arch/arm64/src/common/arm64_gic.h
$HOME/nxstyle $HOME/nuttx/nuttx/arch/arm64/src/common/arm64_gicv2.c

(nxstyle.c is here)

(How I run nxstyle in my Build Script)

Will nxstyle check Kconfig and Makefiles?

Not yet, but maybe someday? That’s why we passed all the modified files to nxstyle for checking.

How do we fix our code?

The pic above shows the output from nxstyle. We’ll see messages like…

- C++ style comment
- Long line found
- Mixed case identifier found
- Operator/assignment must be followed/preceded with whitespace
- Upper case hex constant found
- #include outside of 'Included Files' section

We modify our code so that this…

// Initialize GIC. Called by CPU0 only.
int arm64_gic_initialize(void) {
  // Verify that GIC Version is 2
  int err = gic_validate_dist_version();
  if (err) { sinfo("no distributor detected, giving up ret=%d\n", err); return err; }

  // CPU0-specific initialization for GIC
  arm_gic0_initialize();

  // CPU-generic initialization for GIC
  arm_gic_initialize();
  return 0;
}

(Source)

Becomes this…

/****************************************************************************
 * Name: arm64_gic_initialize
 *
 * Description:
 *   Initialize GIC. Called by CPU0 only.
 *
 * Input Parameters
 *   None
 *
 * Returned Value:
 *   Zero (OK) on success; a negated errno value is returned on any failure.
 *
 ****************************************************************************/

int arm64_gic_initialize(void)
{
  int err;

  /* Verify that GIC Version is 2 */

  err = gic_validate_dist_version();
  if (err)
    {
      sinfo("no distributor detected, giving up ret=%d\n", err);
      return err;
    }

  /* CPU0-specific initialization for GIC */

  arm_gic0_initialize();

  /* CPU-generic initialization for GIC */

  arm_gic_initialize();

  return 0;
}

(Source)

If we see this…

/* */ not balanced

Check that our stars “*” are aligned (heh)…

/******************************
 * Name: missing_asterisk_below
 *
 *****************************/

After fixing, test our code one last time.

§4 Write the Pull Request

Our Pull Request will have…

  1. Title: NuttX Subsystem and One-Line Summary

  2. Summary: What’s the purpose of the Pull Request? What files are changed?

  3. Impact: Which parts of NuttX are affected by the Pull Request? Which parts won’t be affected?

  4. Testing: Provide evidence that our Pull Request does what it’s supposed to do.

    (And that it won’t do what it’s not supposed to do)

To write the above items, let’s walk through these Pull Requests…

§4.1 Title

Inside our Title is the NuttX Subsystem and One-Line Summary

“arch/arm64: Add support for Generic Interrupt Controller Version 2”

(Source)

Let’s write this into a Markdown File for easier copying…

We’ll add the following sections to the Markdown File…

§4.2 Summary

In the Summary we write the purpose of the Pull Request

“This PR adds support for GIC Version 2, which is needed by Pine64 PinePhone.”

(Source)

And we list the files that we changed

arch/arm64/Kconfig: Under “ARM64 Options”, we added an integer option ARM_GIC_VERSION (“GIC version”) that selects the GIC Version. Valid values are 2, 3 and 4, default is 3.“

(Source)

If it’s a long list, we might break into subsections like this…

For Code Provenance it’s good to state how we created the code

“This 64-bit implementation of GIC v2 is mostly identical to the existing GIC v2 for 32-bit Armv7-A (arm_gicv2.c, gic.h), with minor modifications to support 64-bit Registers (Interrupt Context).”

(Source)

(Adding GPL Code? Please check with the NuttX Maintainers)

§4.3 Impact

Under “Impact”, we write which parts of NuttX are affected by the Pull Request.

(And which parts won’t be affected)

“With this PR, NuttX now supports GIC v2 on Arm64.

There is no impact on the existing implementation of GIC v3, as tested below.“

(Source)

§4.4 Testing

Under “Testing”, we provide evidence that our Pull Request does what it’s supposed to do. We fill in the…

Like this…

“We tested with QEMU our implementation of GIC v2:

tools/configure.sh qemu-armv8a:nsh_gicv2 ; make ; qemu-system-aarch64 ...

(See the NuttX QEMU Log)

The log shows that GIC v2 has correctly handled interrupts“

(Source)

If we have done a Regression Test, provide the details too…

“For Regression Testing: We tested the existing implementation of GIC v3…”

tools/configure.sh qemu-armv8a:nsh ; make ; qemu-system-aarch64 ...

(See the NuttX QEMU Log)

(Source)

Test Logs are super helpful for NuttX Maintainers!

(Because we can’t tell which way the train went… By staring at the track!)

Now we tidy up our commits…

Squash Commits with GitHub Desktop

§5 Squash the Commits

(Note: Some kinds of Pull Requests won’t need to Squash Commits)

What’s this “squashing”?

“Squashing” means we’re combining Multiple Commits into One Single Commit.

Our Commit History can get awfully messy during development…

- Initial Commit
- Fixing Build
- Build OK!
- Oops fixing bug
- Tested OK yay!

(Source)

So we always Squash the Commits into One Single Commit (to help future maintainers)…

- arch/arm64: Add support for Generic Interrupt Controller Version 2

(Source)

How do we squash the commits?

We’ll use GitHub Desktop (because I’m terrible with the Git Command Line)…

  1. Install GitHub Desktop and launch it

  2. Click “File → Add Local Repository

    Select our downloaded nuttx folder.

    Click “Add Repository

  3. Click the “History” tab to reveal the Commit History

    (Pic above)

  4. Select the Commits to be Squashed.

    Right-click the Commits.

    Select “Squash Commits

    (Pic above)

  5. Copy the Title of our Pull Request and paste into the Title Box

    arch/arm64: Add support for Generic Interrupt Controller Version 2
    

    In the Description Box, erase the old Commit Messages.

    Copy the Summary of our Pull Request and paste into the Description Box

    This PR adds support for GIC Version 2.
    - `boards/arm64/qemu/qemu-armv8a/configs/nsh_gicv2/defconfig`: Added the Board Configuration
    

    Click “Squash Commits

    Squash Commits with GitHub Desktop

  6. Click “Begin Squash

    Squash Commits with GitHub Desktop

  7. Click “Force Push Origin

    Squash Commits with GitHub Desktop

  8. Click “I’m Sure

    Squash Commits with GitHub Desktop

    And we’re ready to merge upstream! (Like the salmon)

What if we prefer the Git Command Line?

Here are the steps to Squash Commits with the Git Command Line

Taking a long walk

§6 Meditate

Breathe. Take a break.

We’re about to make NuttX History… Our changes will be recorded for posterity!

Take a long walk and ponder…

(I walked 29 km for 5.5 hours while meditating on my Pull Request)

If we get an inspiration or epiphany, touch up the Pull Request.

(And resquash the commits)

What’s your epiphany?

Through my Pull Requests, I hope to turn NuttX into a valuable tool for teaching the internals of Smartphone Operating Systems.

That’s my motivation for porting NuttX to PINE64 PinePhone.

But for now… Let’s finish our Pull Request!

Submit the Pull Request

§7 Submit the Pull Request

Finally it’s time to submit our Pull Request!

  1. Create the Pull Request (pic above)

  2. Verify that it has only One Single Commit (pic above)

    (Squash the Commits)

  3. Copy these into the Pull Request…

    Title

    Summary

    Impact

    Testing

    (Like this)

  4. Submit the Pull Request

  5. Wait for the Automated Checks to be completed (might take an hour)…

    Automated Checks for Pull Request

  6. Fix any errors in the Automated Checks

    (Watch out for CMake and Ninja errors)

    (Fixing the NuttX Config defconfig errors)

  7. Wait for the NuttX Team to review and comment on our Pull Request.

    This might take a while (due to the time zones)… Grab a coffee and standby for fixes!

    (I bake sourdough while waiting)

    If all goes Hunky Dory, our Pull Request will be approved and merged! 🎉

Sometimes we need to Rebase To The Latest Master due to updates in the GitHub Actions Workflow (Continuous Integration Script). Here’s how…

  1. Browse to the master branch of our nuttx repository.

    Click “Sync Fork → Update Branch

    (Pic below)

  2. Launch GitHub Desktop

    Click “File → Add Local Repository

    Select our downloaded nuttx folder.

    Click “Add Repository

  3. Check that the Current Branch is our Working Branch for the Pull Request.

    (Like gic branch)

  4. Click “Fetch Origin

  5. Click “Branch → Rebase Current Branch

    Select the master branch

    Click “Rebase” and “Begin Rebase

  6. Click “Force Push Origin” and “I’m Sure

    (See the official docs)

Is it really OK to Rebase To The Latest Master?

Normally when the GitHub Actions build fails (not due to our code), I take a peek at other recent Pull Requests.

If they didn’t fail the build, then it’s probably OK to Rebase with Master to force the Rebuild.

Pull Updates from NuttX Mainline

§8 Update Our Repositories

(Watch the Video on YouTube)

After our Pull Request has been merged into NuttX Mainline, pull the updates into our repositories…

  1. Browse to the master branch of our nuttx repository.

    Click “Sync Fork → Update Branch

    (Pic above)

  2. Do the same for the master branch of our apps repository.

    Click “Sync Fork → Update Branch

  3. Test the code from the master branch of our nuttx and apps repositories.

When we’re ready to add our next awesome feature

  1. Pull the updates from NuttX Mainline into our nuttx repository…

    Select the master branch.

    Click “Sync Fork → Update Branch

    (Pic above)

  2. Do the same for our apps repository.

  3. In our nuttx repository, click master.

    Enter the name of our new branch.

    Click “Create Branch

    Create Branch

  4. Do the same for our apps repository.

  5. Modify the code in the new branch of our nuttx and apps repositories.

    Build, test and submit our new Pull Request.

And that’s the Complete Lifecycle of a Pull Request for Apache NuttX RTOS!

One last thing: Please help to validate that the NuttX Release works OK! We really appreciate your help with this… 🙏

§9 What’s Next

I hope this article will be helpful for folks contributing code to NuttX for the very first time.

In the next article we’ll explain this complex Pull Request that adds a new SoC and new Board to NuttX. Stay Tuned!

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/pr.md

§10 Notes

  1. Here’s an excellent guide for the Git Command Line

    “Flight rules for Git”

  2. Converting our code to the “NuttX C Coding Standard”

    Can we automate this with a VSCode Extension?

    Maybe like the Linux checkpatch Extension, but it will actually auto-reformat our lines?

  3. QEMU Emulator is incredibly helpful for Regression Testing…

    Can we extend it to emulate PinePhone? Maybe just the UART Hardware? Check out the articles…

    “(Possibly) Emulate PinePhone with Unicorn Emulator”

    “(Clickable) Call Graph for Apache NuttX Real-Time Operating System”

§11 Appendix: Build NuttX with CMake

Why did our NuttX Pull Request fail with this CMake Error?

Cmake in present: rv-virt/smp
riscv-none-elf-gcc CMakeFiles/nuttx.dir/empty.c.obj -o nuttx ...
riscv_createstack.c: undefined reference to board_autoled_on
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

(Source)

That’s because the NuttX Automated Check (Continuous Integration with GitHub Actions) runs CMake on some NuttX Configurations!

(Instead of the usual GNU Make)

What’s this CMake?

CMake is the Alternative Build System for NuttX. We build NuttX with CMake like this…

## For macOS: Install Build Tools
brew install pipenv cmake ninja

## For Ubuntu: Install Build Tools
sudo apt install pipenv cmake ninja-build

## Configure NuttX for QEMU RISC-V SMP `rv-virt:smp`
cd nuttx
pipenv install
pipenv shell
pip install kconfiglib
cmake \
  -B build \
  -DBOARD_CONFIG=rv-virt:smp \
  -GNinja

## Build NuttX for QEMU RISC-V SMP
## with CMake and Ninja
cmake --build build

## Note: No more `tools/configure.sh` and `make`!

But why did CMake fail?

Probably because the CMake Makefiles are out of sync with the GNU Make Makefiles.

Suppose we have a GNU Make Makefile: rv-virt/src/Makefile

## Always compile `qemu_rv_appinit.c`
CSRCS = qemu_rv_appinit.c

## If `CONFIG_ARCH_LEDS` is Enabled:
## Compile `qemu_rv_autoleds.c`
ifeq ($(CONFIG_ARCH_LEDS),y)
  CSRCS += qemu_rv_autoleds.c
endif

## If `CONFIG_USERLED` is Enabled:
## Compile `qemu_rv_userleds.c`
ifeq ($(CONFIG_USERLED),y)
  CSRCS += qemu_rv_userleds.c
endif

Note that it checks the Kconfig Options CONFIG_ARCH_LEDS and CONFIG_USERLED. And optionally compiles qemu_rv_autoleds.c and qemu_rv_userleds.c.

We need to Sync the Build Rules into the CMake Makefile like so: rv-virt/src/CMakeLists.txt

## Always compile `qemu_rv_appinit.c`
set(SRCS qemu_rv_appinit.c)

## If `CONFIG_ARCH_LEDS` is Enabled:
## Compile `qemu_rv_autoleds.c`
if(CONFIG_ARCH_LEDS)
  list(APPEND SRCS qemu_rv_autoleds.c)
endif()

## If `CONFIG_USERLED` is Enabled:
## Compile `qemu_rv_userleds.c`
if(CONFIG_USERLED)
  list(APPEND SRCS qemu_rv_userleds.c)
endif()

This will ensure that the Build Rules are consistent across the GNU Make Makefiles and CMake Makefiles. (Like this)

§12 Appendix: NuttX Configuration Files

Why did our NuttX Pull Request fail with this defconfig error?

Normalize rv-virt/leds64
Saving the new configuration file
HEAD detached at pull/12762/merge
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   boards/risc-v/qemu-rv/rv-virt/configs/leds64/defconfig
  no changes added to commit (use "git add" and/or "git commit -a")

(Source)

This means that the NuttX Configuration File for rv-virt:leds64 has a problem, like a Missing Newline.

To fix it, run tools/refresh.sh: nuttx/pull/12762

## Normalize the `defconfig` for `rv-virt:leds64`
$ cd nuttx
$ tools/refresh.sh --silent rv-virt:leds64
  Normalize rv-virt:leds64
  < CONFIG_USERLED_LOWER=y
  \ No newline at end of file
  ---
  > CONFIG_USERLED_LOWER=y
  Saving the new configuration file

## Remember to commit the updated `defconfig`!
$ git status
  modified: boards/risc-v/qemu-rv/rv-virt/configs/leds64/defconfig

refresh.sh will also fix Kconfig Options that are misplaced…

## `CONFIG_USERLED_LOWER` is misplaced in `rv-virt:leds64`
## Remember to commit the updated `defconfig`!
$ tools/refresh.sh --silent rv-virt:leds64
  Normalize rv-virt:leds64
  74d73
  < CONFIG_USERLED_LOWER=y
  75a75
  > CONFIG_USERLED_LOWER=y
  Saving the new configuration file

What else will cause defconfig errors?

When we modify the Kconfig Configuration Files, remember to update the defconfig Configuration Files! (Like this)

If we forget to update defconfig

Configuration/Tool: pinephone/sensor
Building NuttX...
Normalize pinephone/sensor
71d70
< CONFIG_UART1_SERIAL_CONSOLE=y
Saving the new configuration file
HEAD detached at pull/9243/merge
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
    modified:   boards/arm64/a64/pinephone/configs/sensor/defconfig

(Source)

How to create a new defconfig?

To Create or Update a defconfig, do this…

## TODO: Change this to your
## `boards/<archname>/<chipname>/<boardname>/config/<configname>`
cd nuttx
mkdir -p boards/risc-v/bl808/ox64/configs/nsh

## Create or Update the NuttX Config
make menuconfig \
  && make savedefconfig \
  && grep -v CONFIG_HOST defconfig \
  >boards/risc-v/bl808/ox64/configs/nsh/defconfig

Some Default Settings in .config are missing from defconfig. Can we copy them ourselves to defconfig?

Sorry it won’t work. Suppose we copy these Default UART3 Settings from .config to defconfig (to hard-code the UART3 Baud Rate)…

CONFIG_UART3_BAUD=115200
CONFIG_UART3_BITS=8
CONFIG_UART3_PARITY=0
CONFIG_UART3_2STOP=0

The Auto-Build will fail with the error below. Thus we can’t copy any Default Settings in .config to defconfig.

Configuration/Tool: pinephone/modem
Building NuttX...
Normalize pinephone/modem
69,72d68
< CONFIG_UART3_BAUD=115200
< CONFIG_UART3_BITS=8
< CONFIG_UART3_PARITY=0
< CONFIG_UART3_2STOP=0
Saving the new configuration file
HEAD detached at pull/9304/merge
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
    modified:   boards/arm64/a64/pinephone/configs/modem/defconfig

(Source)

What if the auto-build fails with “Untracked etctmp”?

HEAD detached at pull/11379/merge
Untracked files: (use "git add <file>..." to include in what will be committed)
boards/risc-v/k230/canmv230/src/etctmp.c
boards/risc-v/k230/canmv230/src/etctmp/

(Source)

Check that we’ve added “etctmp” to the Board-Specific Git Ignore: boards/risc-v/jh7110/star64/src/.gitignore

etctmp
etctmp.c

(NuttX CI validates the Config Files by calling cibuild.sh)

§13 Appendix: Validate NuttX Release

For each Official Release of NuttX, how do we check if it runs OK on all devices? Like PinePhone, ESP32, BL602, …

NuttX needs to be tested on every device, and we need your help! 🙏

Before every Official Release of NuttX, a Validation Request will be broadcast on the NuttX Developers Mailing List

Follow the instructions here to validate that the NuttX Release builds correctly and runs OK on your device…

Here’s the script I run to validate NuttX on PinePhone

And here’s the output of the validation script

Boot NuttX on our device, run “uname -a” and “free”…

NuttShell (NSH) NuttX-12.1.0
nsh> uname -a
NuttX 12.1.0 d40f4032fc Apr 12 2023 07:11:20 arm64 pinephone
nsh> free
      total     used   free      largest   nused nfree
Umem: 133414240 550768 132863472 132863376 56    2

The NuttX Hash (like “d40f4032fc” above) should match the Validation Request.

Copy the above into a Validation Response email…

And send back to the Mailing List. (Assuming all is hunky dory)

Since there are so many NuttX devices, we really appreciate your help with the NuttX Validation! 🙏

What are the updates to the NuttX Validation Instructions?

The NuttX Validation Instructions should be updated…

§14 Appendix: Building the Docker Image for NuttX CI

Why are we doing this? Suppose we need to tweak the NuttX CI Docker Image (for Continuous Integration), like to install a Rust Target for a RISC-V Build

## Install the Rust Target for QEMU RISC-V 64-bit
rustup target add riscv64gc-unknown-none-elf

(Source)

We need to test our Modified CI Docker Image. But if we’re on Arm64 macOS: Sorry we can’t download the CI Docker Image, we need to build ourselves! (Skip to the next section if we’re on x64 Linux)

Here’s how we created our CI Docker Image and Custom Dockerfile: tools/ci/docker/linux/Dockerfile

Install Rancher Desktop. In Rancher Desktop, click “Settings”…

Then we build the Docker Image

$ cd nuttx/tools/ci/docker/linux
$ docker build -t nuttx:v1 .

(See the Docker Log)

If we see…

76.31 error: Cannot download gmp-6.1.0.tar.bz2 from ftp://gcc.gnu.org/pub/gcc/infrastructure/

Then GNU Server might be busy. Try again later and it works!

But i386 packages are missing (because we’re hosting Docker on Arm64 macOS)…

 > [stage-12  5/50] RUN apt-get update -qq && DEBIAN_FRONTEND="noninteractive" TZ=Etc/UTC apt-get install -y -qq --no-install-recommends   -o APT::Immediate-Configure=0   avr-libc   ccache   clang   clang-tidy   g++-12-multilib   gcc-avr   gcc-12-multilib   genromfs   gettext   git   lib32z1-dev   libasound2-dev libasound2-dev:i386   libc6-dev-i386   libcurl4-openssl-dev   libmp3lame-dev:i386   libmad0-dev:i386   libncurses5-dev   libpulse-dev libpulse-dev:i386   libpython2.7   libtinfo5   libusb-1.0-0-dev libusb-1.0-0-dev:i386   libv4l-dev libv4l-dev:i386   libx11-dev libx11-dev:i386   libxext-dev libxext-dev:i386   linux-headers-generic   linux-libc-dev:i386   ninja-build   npm   qemu-system-arm   qemu-system-misc   python3   python3-pip   python-is-python3   u-boot-tools   unzip   wget   xxd   file   tclsh   && rm -rf /var/lib/apt/lists/*:
15.03 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy/universe/binary-i386/Packages  404  Not Found [IP: 185.125.190.39 80]
15.03 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-updates/multiverse/binary-i386/Packages  404  Not Found [IP: 185.125.190.39 80]
15.03 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-backports/universe/binary-i386/Packages  404  Not Found [IP: 185.125.190.39 80]
15.03 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-security/main/binary-i386/Packages  404  Not Found [IP: 185.125.190.39 80]
15.03 E: Some index files failed to download. They have been ignored, or old ones used instead.

We don’t need the i386 packages. So we comment out everything in Dockerfile except NuttX Build Tools, RISC-V Toolchain and Rust Toolchain:

Make sure RISC-V Toolchain is for Arm64 (not x64)…

# Download the latest RISCV GCC toolchain prebuilt by xPack
RUN mkdir riscv-none-elf-gcc && \
  curl -s -L "https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v13.2.0-2/xpack-riscv-none-elf-gcc-13.2.0-2-linux-arm64.tar.gz" \
  | tar -C riscv-none-elf-gcc --strip-components 1 -xz

If RISC-V Toolchain fails: Try again…

 => ERROR [nuttx-toolchain-riscv 1/1] RUN mkdir riscv-none-elf-gcc &&    195.2s
------                                                                          
 > [nuttx-toolchain-riscv 1/1] RUN mkdir riscv-none-elf-gcc &&   curl -s -L "https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v13.2.0-2/xpack-riscv-none-elf-gcc-13.2.0-2-linux-arm64.tar.gz"   | tar -C riscv-none-elf-gcc --strip-components 1 -xz:
195.2 
195.2 gzip: stdin: unexpected end of file
195.2 tar: Unexpected EOF in archive
195.2 tar: Unexpected EOF in archive
195.2 tar: Error is not recoverable: exiting now
------

zap seems to fail for Arm64. So we comment out zap in Dockerfile…

39.86 npm ERR! code 1
39.86 npm ERR! path /tools/zap/node_modules/canvas
39.86 npm ERR! command failed
39.86 npm ERR! command sh -c node-pre-gyp install --fallback-to-build --update-binary
39.86 npm ERR! Failed to execute '/usr/bin/node /usr/share/nodejs/node-gyp/bin/node-gyp.js configure --fallback-to-build --update-binary --module=/tools/zap/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/tools/zap/node_modules/canvas/build/Release --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
39.86 npm ERR! node-pre-gyp info it worked if it ends with ok
39.86 npm ERR! node-pre-gyp info using node-pre-gyp@1.0.10
39.86 npm ERR! node-pre-gyp info using node@12.22.9 | linux | arm64
39.86 npm ERR! node-pre-gyp http GET https://github.com/Automattic/node-canvas/releases/download/v2.11.2/canvas-v2.11.2-node-v72-linux-glibc-arm64.tar.gz
39.86 npm ERR! node-pre-gyp ERR! install response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.11.2/canvas-v2.11.2-node-v72-linux-glibc-arm64.tar.gz 
39.86 npm ERR! node-pre-gyp WARN Pre-built binaries not installable for canvas@2.11.2 and node@12.22.9 (node-v72 ABI, glibc) (falling back to source compile with node-gyp) 

And we’re done!

 => [stage-4 27/33] COPY --from=nuttx-tools /tools/bloaty/ bloaty/         1.8s
 => [stage-4 28/33] COPY --from=nuttx-tools /tools/kconfig-frontends/ kco  0.0s
 => [stage-4 29/33] COPY --from=nuttx-tools /tools/rust/ /tools/rust/      3.0s
 => [stage-4 30/33] COPY --from=nuttx-toolchain-riscv /tools/riscv-none-e  5.1s
 => [stage-4 31/33] RUN mkdir -p /tools/gn                                 0.2s
 => [stage-4 32/33] COPY --from=nuttx-tools /tools/gn/gn/out/gn /tools/gn  0.0s
 => [stage-4 33/33] RUN mkdir -p /tools/ccache/bin &&   ln -sf `which cca  0.3s
 => exporting to image                                                    15.0s
 => => exporting layers                                                   15.0s
 => => writing image sha256:27da351829f15ecf1dd333cc43b864045d462a595f512  0.0s
 => => naming to docker.io/library/nuttx:v1                                0.0s

After the above fixes, here’s our Custom Dockerfile for NuttX CI + RISC-V + Rust (that works on Arm64 macOS)…

We check the Docker Image and RISC-V Toolchain…

$ docker images
REPOSITORY  TAG   IMAGE ID       CREATED              SIZE
nuttx       v1    27da351829f1   About a minute ago   7.09GB

$ docker run -it nuttx:v1 /bin/bash 

root@?:/tools# uname -a
Linux 6d5db4f6b835 6.6.14-0-virt #1-Alpine SMP Fri, 26 Jan 2024 11:08:07 +0000 aarch64 aarch64 aarch64 GNU/Linux

root@?:/tools# riscv-none-elf-gcc -v       
gcc version 13.2.0 (xPack GNU RISC-V Embedded GCC aarch64) 

root@?:/tools# rustc --version
rustc 1.80.0 (051478957 2024-07-21)

root@?:/tools# rustup --version
rustup 1.27.1 (54dd3d00f 2024-04-24)

root@?:/tools# rustup toolchain list
stable-aarch64-unknown-linux-gnu (default)

root@?:/tools# rustup target list
aarch64-unknown-linux-gnu (installed)
riscv64gc-unknown-none-elf (installed)
thumbv6m-none-eabi (installed)
thumbv7m-none-eabi (installed)

root@?:/tools# exit

Looks hunky dory! We download the NuttX Source Files and start the build…

$ docker run -it nuttx:v1 /bin/bash 
# cd
# pwd
/root
# git clone https://github.com/apache/nuttx
# git clone https://github.com/apache/nuttx-apps apps
# cd nuttx/tools/ci
# ./cibuild.sh -c -A -N -R testlist/risc-v-02.dat

Why risc-v-02.dat? That’s because we’re running the “Linux (risc-v-02)” build. Which will build the Second Batch of RISC-V Targets, including rv-virt:* (QEMU RISC-V).

(Why cibuild.sh? See the next section)

We will see…

+ /root/nuttx/tools/testbuild.sh -A -N -R -j 2 -e '-Wno-cpp -Werror' testlist/risc-v-02.dat
====================================================================================
Configuration/Tool: hifive1-revb/nsh
2024-08-06 16:06:08
------------------------------------------------------------------------------------
  Cleaning...
  Configuring...
  Building NuttX...

Exactly like NuttX CI in GitHub Actions!

Finally we see our Rust Build completing successfully in our Docker Container yay!

Configuration/Tool: rv-virt/leds64_rust
  Cleaning...
  Configuring...
  Building NuttX...
riscv-none-elf-ld: warning: /root/nuttx/nuttx has a LOAD segment with RWX permissions
  Normalize rv-virt/leds64_rust

(See the Docker Log)

(See the Pull Request)

And that’s how we test a NuttX CI Docker Image on Arm64 macOS.

What happens if the CI Build fails?

Suppose we forget to add the Rust Target into the Docker Image. We’ll see an error like this…

Configuration/Tool: rv-virt/leds64_rust
  Cleaning...
  Configuring...
  Building NuttX...
error[E0463]: can't find crate for `core`
  |
  = note: the `riscv64gc-unknown-none-elf` target may not be installed
  = help: consider downloading the target with `rustup target add riscv64gc-unknown-none-elf`

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0463`.
make[2]: *** [/root/apps/Application.mk:293: leds_rust_main.rs.root.apps.examples.leds_rust.o] Error 1
make[2]: Target 'all' not remade because of errors.
make[1]: *** [Makefile:51: /root/apps/examples/leds_rust_all] Error 2
make[1]: Target 'all' not remade because of errors.
make: *** [tools/LibTargets.mk:232: /root/apps/libapps.a] Error 2
make: Target 'all' not remade because of errors.
/root/nuttx/tools/testbuild.sh: line 378: /root/nuttx/../nuttx/nuttx.manifest: No such file or directory
  Normalize rv-virt/leds64_rust

(See the Docker Log)

What about testing a Single Config?

Just do it the normal NuttX way…

$ docker run -it nuttx:v1 /bin/bash 
# cd
# git clone https://github.com/apache/nuttx
# git clone https://github.com/apache/nuttx-apps apps
# cd nuttx
# tools/configure.sh rv-virt:leds64_rust
# make

(Based on “Create a Docker Image for NuttX”)

§15 Appendix: Downloading the Docker Image for NuttX CI

If we’re on x64 Linux: We can Build the Docker Image for NuttX CI, without any changes to the Dockerfile…

$ sudo apt install podman-docker
$ cd nuttx/tools/ci/docker/linux
$ docker build -t nuttx:v1 .

(Takes a few hours to complete. Unless we trim down the Dockerfile, as explained in the previous section)

Or we can Download the Docker Image for NuttX CI and tweak it (for testing). This works only if we’re on x64 Linux…

$ sudo apt install podman-docker
$ docker pull \
    ghcr.io/apache/nuttx/apache-nuttx-ci-linux:latest
$ docker images
REPOSITORY                                  TAG         IMAGE ID      CREATED         SIZE
ghcr.io/apache/nuttx/apache-nuttx-ci-linux  latest      60a6cdc2c7cc  5 hours ago     15.9 GB

If we see “No matching manifest for linux/arm64/v8”: Nope sorry we’re not on x64 Linux, we can’t download and run the Docker Image!

After building or downloading the Docker Image: Run these steps (from previous section) to start the Docker Container, download the NuttX Source Files and start the build…

## If we built the Docker Image ourselves:
$ docker run -it nuttx:v1 /bin/bash 

## If we downloaded the Docker Image:
$ docker run -it ghcr.io/apache/nuttx/apache-nuttx-ci-linux:latest /bin/bash 

## Inside Docker Container:
## Download the NuttX Source Files and start the build
$ cd
$ pwd
/root
$ git clone https://github.com/apache/nuttx
$ git clone https://github.com/apache/nuttx-apps apps
$ cd nuttx/tools/ci
$ ./cibuild.sh -c -A -N -R testlist/risc-v-02.dat

Or for a Single Config…

$ cd
$ git clone https://github.com/apache/nuttx
$ git clone https://github.com/apache/nuttx-apps apps
$ cd nuttx
$ tools/configure.sh rv-virt:leds64_rust
$ make

Why cibuild.sh? We got it from the NuttX CI Log

(cibuild.sh is called by .github/workflows/build.yml)

(testbuild.sh is explained here)

## Run the CI Build for RISC-V Targets
./cibuild.sh -c -A -N -R \
  testlist/risc-v-02.dat

## Which calls Test Build for RISC-V Targets
/github/workspace/sources/nuttx/tools/testbuild.sh -A -N -R \
  -j 4 \
  -e '-Wno-cpp -Werror' \
  testlist/risc-v-02.dat

## Which runs inside the Docker Container for NuttX CI
/usr/bin/docker run \
  --name ghcrioapachenuttxapachenuttxcilinux_ba895b \
  --label bbeb86 \
  --workdir /github/workspace \
  --rm \
  -e "DOCKER_BUILDKIT" \
  -e "nuttx_sha" \
  -e "BLOBDIR" \
  -e "INPUT_RUN" \
  -e "HOME" \
  -e "GITHUB_JOB" \
  -e "GITHUB_REF" \
  -e "GITHUB_SHA" \
  -e "GITHUB_REPOSITORY" \
  -e "GITHUB_REPOSITORY_OWNER" \
  -e "GITHUB_REPOSITORY_OWNER_ID" \
  -e "GITHUB_RUN_ID" \
  -e "GITHUB_RUN_NUMBER" \
  -e "GITHUB_RETENTION_DAYS" \
  -e "GITHUB_RUN_ATTEMPT" \
  -e "GITHUB_REPOSITORY_ID" \
  -e "GITHUB_ACTOR_ID" \
  -e "GITHUB_ACTOR" \
  -e "GITHUB_TRIGGERING_ACTOR" \
  -e "GITHUB_WORKFLOW" \
  -e "GITHUB_HEAD_REF" \
  -e "GITHUB_BASE_REF" \
  -e "GITHUB_EVENT_NAME" \
  -e "GITHUB_SERVER_URL" \
  -e "GITHUB_API_URL" \
  -e "GITHUB_GRAPHQL_URL" \
  -e "GITHUB_REF_NAME" \
  -e "GITHUB_REF_PROTECTED" \
  -e "GITHUB_REF_TYPE" \
  -e "GITHUB_WORKFLOW_REF" \
  -e "GITHUB_WORKFLOW_SHA" \
  -e "GITHUB_WORKSPACE" \
  -e "GITHUB_ACTION" \
  -e "GITHUB_EVENT_PATH" \
  -e "GITHUB_ACTION_REPOSITORY" \
  -e "GITHUB_ACTION_REF" \
  -e "GITHUB_PATH" \
  -e "GITHUB_ENV" \
  -e "GITHUB_STEP_SUMMARY" \
  -e "GITHUB_STATE" \
  -e "GITHUB_OUTPUT" \
  -e "RUNNER_OS" \
  -e "RUNNER_ARCH" \
  -e "RUNNER_NAME" \
  -e "RUNNER_ENVIRONMENT" \
  -e "RUNNER_TOOL_CACHE" \
  -e "RUNNER_TEMP" \
  -e "RUNNER_WORKSPACE" \
  -e "ACTIONS_RUNTIME_URL" \
  -e "ACTIONS_RUNTIME_TOKEN" \
  -e "ACTIONS_CACHE_URL" \
  -e "ACTIONS_RESULTS_URL" \
  -e GITHUB_ACTIONS=true \
  -e CI=true \
  -v "/var/run/docker.sock":"/var/run/docker.sock" \
  -v "/home/runner/work/_temp/_github_home":"/github/home" \
  -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" \
  -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" \
  -v "/home/runner/work/nuttx/nuttx":"/github/workspace" \
  ghcr.io/apache/nuttx/apache-nuttx-ci-linux  \
  "/bin/bash" \
  "-ce" \
  "echo \"::add-matcher::sources/nuttx/.github/gcc.json\"