MicroPython and wasp-os hosted on Mynewt on PineTime Smart Watch. VSCode Debugger is shown on the monitor
đź“ť 2 Jun 2020
wasp-os, built with MicroPython, is highly popular with folks coding PineTime Smart Watch for the very first time...
Because there's no easier way to create a Watch App for PineTime!
Just write a simple Python script, upload it wirelessly to wasp-os and watch it run!
And if the script doesn't work right? Just connect over Bluetooth and troubleshoot live with the REPL (Read-Eval-Print Loop) command line!
Today wasp-os works great on Bare Metal (i.e. PineTime hardware). It doesn't require an embedded operating system, because the underlying MicroPython runtime serves as a bare-bones operating system.
How shall we make wasp-os better?
I'm now coding these enhancements in MicroPython...
Easier Installation
What if PineTime Owners can unbox their smart watches... And install wasp-os wirelessly through their phones? No more messy wiring to the SWD Port inside PineTime!
And we'll let PineTime Owners switch and test drive any operating system wirelessly: wasp-os, FreeRTOS, Mynewt, RIOT, Zephyr, ... Without opening their watches!
We're getting ready a Common Bootloader and Firmware Update mechanism that will be preloaded by Pine64 into PineTime watches... Would be terrific to have wasp-os and MicroPython on board!
More about this in a while.
PineTime Companion App for Android, iOS and PinePhone
A common Companion App for PineTime that updates the firmware, uploads Watch Apps and lets you interact via the REPL command line... That works on Android, iOS and PinePhone!
If you have a Raspberry Pi, the Companion App will run too! (Using Pi's onboard Bluetooth hardware)
We're creating the Companion App now... And we hope to have wasp-os adopting the Simple Management Protocol that's used by Mynewt, RIOT, FreeRTOS and Zephyr.
We have a solution for this... More about this later.
Simpler Firmware Debugging with Raspberry Pi and VSCode
For newbies who have mastered MicroPython programming on PineTime, perhaps they would like to dig deeper into the internals of wasp-os firmware, make some tweaks, experiment with the Bluetooth stack, ...
Let's make it easier for them to explore! Just connect a Raspberry Pi to PineTime's SWD Port (yep slightly messy) and step through the firmware code with the VSCode Debugger.
Or just study the helpful debugging messages that will be displayed in OpenOCD, thanks to the SWD Semihosting Console.
Check out this video of wasp-os and MicroPython debug messages displayed on the Semihosting Console with OpenOCD and VSCode...
Preemptive Multitasking
MicroPython comes with simple task management. Perhaps we should get ready to support Watch Apps that require full multitasking?
If we build a MicroPython Watch App that talks to other watches over Bluetooth Mesh... Surely we'll need some kind of background processing?
Best Buddies with other Operating Systems
PineTime feels like a rowdy playground sometimes... Kids (i.e. various PineTime Firmware) come and go as they please, messing up the sandbox (e.g. PineTime's SPI Flash) for the other players.
Can we turn PineTime into a safer, happier playground for all?
Adopting a common filesystem (like littlefs) with fixed parameters (SPI Flash location) may help.
PineTime Bootloader's Boot Graphic (stored in SPI Flash) unintentionally scratched (top left) by the firmware
To make wasp-os truly awesome with interoperable firmware updates (and a common companion app), we could add to MicroPython the following open source components...
MCU Manager Library for Simple Management Protocol
And a sophisticated Task Scheduler
Looks daunting!
Or we could host wasp-os and MicroPython on Mynewt, which already includes NimBLE, MCUBoot, MCU Manager and Preemptive Multitasking...
That's the approach we'll explore. Hosting MicroPython on another operating system isn't new... It has been done with Zephyr, which is similar to Mynewt.
What could go wrong?
There are plenty of risks in running wasp-os and MicroPython on Mynewt instead of Bare Metal...
MicroPython is Dynamic, Mynewt is Not
MicroPython allocates objects dynamically in Heap Memory. When Heap Memory runs low, the Garbage Collector sweeps the memory and reclaims space for new objects.
Traditional embedded operating systems are engineered to be highly predictable and less flexible. Objects live in Static Memory and Stack Memory, not in Heap Memory.
Embedded operating systems like Mynewt will get spooked by Dynamic Objects in MicroPython!
Heap Memory needs to be managed really carefully with Mynewt.
MicroPython is blissfully unaware of Multitasking
MicroPython runs mostly as a single task with a single stack. Easy peasy!
Mynewt runs with multiple tasks and multiple stacks. Tasks communicate by passing Events.
So it gets messy when we try to give MicroPython the illusion that it's running all alone by itself... MicroPython needs to share the CPU and RAM with Mynewt tasks now.
wasp-os Complexity
By inserting an embedded operating system (Mynewt) under MicroPython, it increases the complexity of the entire wasp-os firmware.
We'll have to manage this complexity very carefully: We shall make changes only to the MicroPython layer, not to the wasp-os layer.
And most ominous of all: I have failed before!
My first hacking experience with MicroPython was on STM32 Blue Pill... It failed miserably.
But now I'm older, wiser and I have much better hardware (PineTime vs Blue Pill)... I might succeed!
Here goes the experiment to assess whether it's really feasible to improve wasp-os by hosting it on top of Mynewt.
Our objective is simple: To boot up wasp-os on Mynewt and render a watch face on PineTime.
Read on for the details.
Objective Accomplished! wasp-os rendering a watch face with Mynewt on PineTime. At right is the ST-Link debugger
Embedded operating systems (like Mynewt) are designed to run on multuple hardware platforms (like PineTime). That's why they include a Hardware Abstraction Layer that wraps low-level hardware-specific code (like writing to SPI registers) into high-level hardware-independent functions (like sending a chunk of data to the SPI port).
The Hardware Abstraction Layer is used on PineTime like this...
In MicroPython, the Hardware Abstraction Layers for various hardware platforms are located in the Ports folder, coded in C...
github.com/micropython/micropython/portsPineTime uses the nRF Port, which runs on nRF51 and nRF52 hardware...
ports/nrfFor this experiment we cloned the nRF Port to create a new Mynewt Port that will run on Mynewt (instead of bare hardware)...
ports/mynewtLet's look inside the Mynewt Port.
Remember that a MicroPython Port works like a Hardware Abstraction Layer... It contains high-level functions for accessing the GPIO, SPI and I2C hardware interfaces. Let's learn how the Mynewt Port of MicroPython implements these high-level functions in C.
Here are the configuration files for the Mynewt Port of MicroPython...
mpconfigport.h: Configures MicroPython Runtime features, like REPL and code compilation
boards/pinetime/mpconfigboard.h: Configures PineTime's Hardware features, like SPI
boards/pinetime/pins.csv: Defines PineTime's Hardware Pins and their functions (GPIO, SPI, I2C)
PineTime's nRF52 Microcontroller has onboard functions like GPIO, I2C, SPI, UART, Timers, ... We call them Peripherals.
modules/machine folder contains the driver code in C for the nRF52 Peripherals
We have reprogrammed the GPIO, SPI and I2C Drivers to work with Mynewt's Hardware Abstraction Layer. More about this later.
Peripheral Drivers from MicroPython that we haven't ported to Mynewt: Analog-to-Digital Converter, Pulse Width Modulation, Real Time Clock, Temperature Sensor, Timer, UART
There are other MicroPython drivers for PineTime's nRF52 Microcontroller that we haven't ported to Mynewt yet. The drivers may be found in these folders...
modules: Bluetooth LE, LED, Music, Random Number Generator, BluePy, OS, Time
drivers: Bluetooth, Flash ROM, Soft PWM, Ticker, Watchdog
main.c contains the start_micropython() function that starts the MicroPython Runtime (including the REPL command line)
start_micropython() is called by Mynewt during startup
Heap Memory for MicroPython is also defined in main.c as a Static Array. More about this later.
Every PineTime Firmware requires a Reset Handler function to defined. It's the first function that's called when the firmware starts.
The firmware should also contain a Vector Table, which is a list of pointers to functions that will handle interrupts and exceptions on PineTime.
The Linker Script is used by the GCC Linker while building the firmware to determine the addresses in ROM and RAM for placing the firmware code and data.
The Reset Handler, Vector Table and Linker Script are mandatory for every MicroPython Port (including the nRF Port). But for the Mynewt Port, they are not used because we'll be using the Reset Handler, Vector Table and Linker Script from Mynewt instead.
device folder contains the MicroPython Reset Handler and Vector Table
They are not used in the Mynewt Port, we're using the Reset Handler and Vector Table from Mynewt: hw/bsp/nrf52/src/arch/cortex_m4/gcc_startup_nrf52.s
boards folder contains the MicroPython Linker Script
This Linker Script is not used in the Mynewt Port, we're using the Linker Script from Mynewt: hw/bsp/nrf52/nrf52xxaa.ld
Let's study the code for the GPIO, SPI and I2C Drivers in the Mynewt Port for MicroPython... And understand how they call Mynewt's Hardware Abstraction Layer.
The GPIO Driver is essential for our demo... Because PineTime's Backlight is controlled by GPIO and PineTime's screen will be totally dark without the Backlight!
Let's look at the current implementation of the GPIO Driver on MicroPython: ports/nrf/mphalport.h
#define mp_hal_pin_high(p) \
nrf_gpio_pin_set(p->pin)
#define mp_hal_pin_low(p) \
nrf_gpio_pin_clear(p->pin)
To set a pin high or low, the GPIO Driver (from the nRF Port) calls the functions nrf_gpio_pin_set() and nrf_gpio_pin_clear(), which are provided by the Nordic nRF5 SDK
Clearly this GPIO Driver (nRF Port) is directly manipulating the bare hardware of PineTime.
Compare that with the Mynewt Port: ports/mynewt/mphalport.h
#define mp_hal_pin_high(p) \
hal_gpio_write(p->pin, 1)
#define mp_hal_pin_low(p) \
hal_gpio_write(p->pin, 0)
This GPIO Driver calls the functions hal_gpio_write() and hal_gpio_write() provided by Mynewt's Hardware Abstraction Layer.
Thus we are no longer manipulating PineTime hardware directly... We are doing it through an embedded operating system (Mynewt).
And we get to enjoy all the benefits of the embedded operating system! Multitasking, firmware updates, Bluetooth support, ...
Here's how we configure and read a GPIO pin by calling Mynewt's Hardware Abstraction Layer...
#define mp_hal_pin_read(p) \
hal_gpio_read(p->pin)
#define mp_hal_pin_open_drain(p) \
hal_gpio_init_in(p->pin, HAL_GPIO_PULL_NONE)
With this simple reprogramming, we have done enough to switch on PineTime's Backlight for our experiment.
Some GPIO functions have yet to be implemented, like in modules/machine/pin.c...
STATIC mp_obj_t pin_irq(
size_t n_args,
const mp_obj_t *pos_args,
mp_map_t *kw_args) { ...
pin_irq() is called by MicroPython to configure the interrupt that will be triggered when an input GPIO pin switches its state from high to low (or low to high). This will be useful for handling the pressing of PineTime's watch button, since it's connected to a GPIO pin.
Eventually pin_irq() shall be reprogrammed to call the equivalent function in Mynewt: hal_gpio_irq_init()
Moving on to SPI...
Now that we have reprogrammed the GPIO Driver from Bare Metal PineTime to Mynewt, let's do the same for the SPI Driver.
PineTime's Sitronix ST77889 Display Controller is controlled via SPI, thus we shall port the SPI Driver to Mynewt for rendering graphics.
Porting the SPI Driver really is really easy... No initialisation needed!
Mynewt starts its SPI Driver automatically upon startup, according to the settings in apps/my_sensor_app/syscfg.yml
# System Configuration Setting Values:
# Below we override the driver and library settings. Settings defined in
# targets/nrf52_my_sensor/syscfg.yml will override the settings below.
syscfg.vals:
SPI_0_MASTER: 1 # Enable SPI port 0 for ST7789 display and SPI Flash
The pins for the SPI port are defined in hw/bsp/nrf52/syscfg.yml
syscfg.vals:
###########################################################################
# Default Pins for Peripherals
# Defined in http://files.pine64.org/doc/PineTime/PineTime%20Port%20Assignment%20rev1.0.pdf
# SPI port 0 connected to ST7789 display and XT25F32B flash
SPI_0_MASTER_PIN_SCK: 2 # P0.02/AIN0: SPI-SCK, LCD_SCK SPI clock for display and flash
SPI_0_MASTER_PIN_MOSI: 3 # P0.03/AIN1: SPI-MOSI, LCD_SDI SPI MOSI for display and flash
SPI_0_MASTER_PIN_MISO: 4 # P0.04/AIN2: SPI-MISO SPI MISO for flash only
What about the MicroPython driver code? Only one SPI Driver function needs to be reprogrammed in modules/machine/spi.c
/// Called by MicroPython to transmit and receive a chunk of data over the SPI port
void spi_transfer(
const machine_hard_spi_obj_t * self, // MicroPython instance of the SPI Driver (not used here)
size_t len, // Number of bytes to send / receive
const void * src, // Transmit Buffer
void * dest // Receive Buffer
) {
int rc = hal_spi_txrx(
SPI_PORT, // Mynewt SPI Port (hardcoded to 0)
(void *) src, // Transmit Buffer
dest, // Receive Buffer
len // Length
);
if (rc != 0) { mp_raise_ValueError("SPI TX failed"); }
}
Whenever MicroPython calls spi_transfer() in our Mynewt Port to transmit and receive a chunk of data over SPI, we forward the call to hal_spi_txrx(). That's the equivalent SPI function in the Mynewt Hardware Abstraction Layer.
The SPI Driver in the Mynewt Port for MicroPython is now ready!
We'll see the SPI Driver in action when wasp-os's ST7789 Driver calls our SPI Driver to render the watch face: wasp/drivers/st7789.py
Finally, I2C...
We won't be using the I2C Driver in the demo, because we won't be handling touch events from PineTime's Touch Controller. But for completeness, here's how we ported MicroPython's I2C Driver to Mynewt...
Mynewt starts the I2C port automatically upon startup because of the settings in apps/my_sensor_app/syscfg.yml
# System Configuration Setting Values:
# Below we override the driver and library settings. Settings defined in
# targets/nrf52_my_sensor/syscfg.yml will override the settings below.
syscfg.vals:
I2C_1: 1 # Enable I2C port 1 for CST816S touch controller, BMA421 accelerometer, HRS3300 heart rate sensor
The I2C pins are configured in hw/bsp/nrf52/syscfg.yml
syscfg.vals:
###########################################################################
# Default Pins for Peripherals
# Defined in http://files.pine64.org/doc/PineTime/PineTime%20Port%20Assignment%20rev1.0.pdf
# I2C port 1 connected to CST816S touch controller, BMA421 accelerometer, HRS3300 heart rate sensor
I2C_1_PIN_SCL: 7 # P0.07: BMA421-SCL, HRS3300-SCL, TP-SCLOUT
I2C_1_PIN_SDA: 6 # P0.06: BMA421-SDA, HRS3300-SDA, TP-SDAI/O
Note that the SPI and I2C ports are numbered globally in Mynewt... SPI goes to port 0 (SPI_0) and I2C goes to port 1 (I2C_1). Thus we should never attempt to configure SPI_1 or I2C_0.
Here's how we configure the I2C frequency for MicroPython: modules/machine/i2c.c
/// Called by MicroPython to initialise the I2C port
mp_obj_t machine_hard_i2c_make_new(
const mp_obj_type_t *type,
size_t n_args,
size_t n_kw,
const mp_obj_t *all_args
) {
struct hal_i2c_settings settings = {
.frequency = 400, // Set to the highest I2C frequency: 400 Kbps
};
int rc = hal_i2c_config( // Call Mynewt I2C HAL
I2C_PORT, // Mynewt I2C Port (hardcoded to 1)
&settings
);
if (rc != 0) { mp_raise_ValueError("I2C init failed"); }
const machine_hard_i2c_obj_t *self = &machine_hard_i2c_obj[I2C_PORT];
return MP_OBJ_FROM_PTR(self);
}
The code above calls hal_i2c_config() from Mynewt's I2C Hardware Abstraction Layer.
Here's how we transmit and receive I2C packets for MicroPython: modules/machine/i2c.c
/// Transmit or receive the buffer via I2C. Return the number of bytes transferred, or in case of error, a negative MP error code.
int machine_hard_i2c_transfer_single(
mp_obj_base_t *self_in, // MicroPython instance of the I2C Driver (not used here)
uint16_t addr, // I2C target address
size_t len, // Number of bytes to transfer / receive
uint8_t *buf, // Buffer to transfer and receive
unsigned int flags // Whether this is a read or write operation
) {
// Set the transmit/receive parameters
struct hal_i2c_master_data master_data = {
.address = addr,
.len = len,
.buffer = buf,
};
if (flags & MP_MACHINE_I2C_FLAG_READ) { // If reading from I2C...
int rc_read = hal_i2c_master_read( // Call Mynewt I2C HAL to receive data
I2C_PORT, // Mynewt I2C Port (hardcoded to 1)
&master_data, // Data will be received here
I2C_TIMEOUT, // Timeout in milliseconds (1,000)
1 // 1 means this is the last I2C operation. So we can terminate after this.
);
if (rc_read != 0) { return check_i2c_return_code(rc_read); }
} else { // If writing to I2C...
int rc_write = hal_i2c_master_write( // Call Mynewt I2C HAL to transmit data
I2C_PORT, // Mynewt I2C Port (hardcoded to 1)
&master_data, // Data to be transmitted
I2C_TIMEOUT, // Timeout in milliseconds (1,000)
1 // 1 means this is the last I2C operation. So we can terminate after this.
);
if (rc_write != 0) { return check_i2c_return_code(rc_write); }
}
return len;
}
The code above calls hal_i2c_master_read() and hal_i2c_master_write() from Mynewt's I2C Hardware Abstraction Layer.
How is this I2C Driver used in wasp-os?
Check out the driver for the CST816S Touch Controller: wasp/drivers/cst816s.py
How is RAM used by MicroPython?
When MicroPython runs on Bare Metal (without an operating system), it uses three chunks of RAM to store objects...
Static Memory: Contains C objects that are Global or Static
Stack Memory: Contains C objects allocated on the stack inside function calls
Heap Memory: Contains MicroPython objects allocated dynamically, similar to calling malloc() in C
Note that the Heap Memory and Stack Memory are not fixed. They are allowed to grow and shrink, as long as the combined size fits within RAM.
What happens when Stack Memory gets too close to Heap Memory?
MicroPython performs Garbage Collection. It scans the Heap Memory and reclaims unused space for new objects.
What happens when we run MicroPython on a multitasking operating system like Mynewt?
With multitasking, all RAM needs to be carefully budgeted and partitioned like this...
Mbuf Memory: Mbufs (Memory Buffers) are used for Bluetooth LE networking and by the Semihosting Console in Mynewt. Mbufs are fixed-size blocks of 110 bytes that may be chained together to form larger objects.
Why are Mbufs fixed size?
To prevent fragmentation of the memory space. Though coding with Mbufs is slightly more difficult.
Idle Task: With multitasking, there is no longer a common stack. Each task now requires its own Stack Memory. Even the Idle Task, which runs when no other tasks are running.
Main Task: This is the task that runs the MicroPython Runtime. The Stack Memory in the Main Task is used by MicroPython.
Bluetooth Task: This is the background task that handles Bluetooth LE packets. It requires its own Stack Memory too.
We no longer enjoy the grow-and-shink memory model that we have seen earlier... Every memory space must have a fixed size!
That's why we often discourage the use of Heap Memory on embedded operating systems... We can't really be sure when we'll run out of Heap Memory!
Let's find out how we allocated the Heap Memory in Mynewt.
For the Mynewt port of MicroPython, the Heap Memory is defined as a static array of 32 KB: ports/mynewt/main.c
/// Heap Memory for MicroPython
#define MICROPYTHON_HEAP_SIZE 32768
uint8_t micropython_heap[MICROPYTHON_HEAP_SIZE];
/// Called my Mynewt to start the MicroPython runtime
int start_micropython(void) {
...
// Set the top of the stack
void *stack_start = get_micropython_stack_start();
void *stack_end = get_micropython_stack_end();
mp_stack_set_top(stack_end);
// Stack limit not applicable to Mynewt, since stack and heap are in different regions
mp_stack_set_limit((char*) stack_end - (char*) stack_start);
// Initialise the Mynewt port
machine_init();
// Allocate the MicroPython heap memory
void *heap_start = µpython_heap[0];
void *heap_end = µpython_heap[MICROPYTHON_HEAP_SIZE];
gc_init(heap_start, heap_end);
Note that the Stack Limit checking has been disabled, since we don't support grow-and-shrink Heap Memory with Mynewt.
32 KB is not a lot of Heap Memory... But it's sufficient for wasp-os to render a watch face on PineTime (which has 64 KB RAM).
The Stack Memory and Mbuf Memory sizes are defined in Mynewt: apps/my_sensor_app/syscfg.yml
syscfg.vals:
OS_MAIN_STACK_SIZE: 2048 # Main stack (includes MicroPython) is 8 KB. Previously 4096 (16 KB)
# For Bluetooth LE: Lots of smaller mbufs are required for newtmgr using typical BLE ATT MTU values.
MSYS_1_BLOCK_COUNT: 22 # Defaults to 12. Previously 64
MSYS_1_BLOCK_SIZE: 110 # Defaults to 292
We reserved 8 KB of Stack Memory for Mynewt's Main Task, which also executes the MicroPython Runtime.
For the Mbuf Memory we allocated 22 Mbufs of 110 bytes each (total 2,420 bytes).
This Mbuf Memory setting was recommended because we'll be using MCU Manager (Simple Management Protocol) to upload firmware files during firmware update. See apache-mynewt-nimble/apps/bleprph/syscfg.yml
How much RAM and ROM are used by MicroPython on Mynewt?
According to the build output, the MicroPython Runtime takes up 33 KB of RAM. That includes 32 KB of Heap Memory that we have allocated above.
The compiled MicroPython Runtime code occupies 210 KB of Flash ROM.
FLASH RAM
214606 33557 libs_micropython.a
The MicroPython Runtime is compiled as a custom library in Mynewt, hence the name libs_micropython.a. See the build script
The combined MicroPython + Mynewt firmware (including the NimBLE Bluetooth Stack) occupies 340 KB of Flash ROM, 53 KB of RAM...
text data bss
347992 984 54728
Hence MicroPython takes up 62% of the firmware ROM, 61% of the firmware RAM.
Here are the ROM and RAM sizes of all code modules in the MicroPython + Mynewt firmware...
+ newt size -v nrf52_my_sensor
Size of Application Image: app
Mem FLASH: 0x8000-0x7bc00
Mem RAM: 0x20000000-0x20010000
FLASH RAM
800 324 *fill*
1018 98 apps_my_sensor_app.a
2268 116 boot_bootutil.a
18 0 boot_mynewt_flash_map_backend.a
438 26 boot_split.a
1180 0 crypto_mbedtls.a
2302 0 crypto_tinycrypt.a
401 0 encoding_base64.a
1622 0 encoding_cborattr.a
3002 0 encoding_tinycbor.a
452 444 hw_bsp_nrf52.a
52 0 hw_cmsis-core.a
1560 92 hw_drivers_flash_spiflash.a
706 1 hw_hal.a
6646 89 hw_mcu_nordic_nrf52xxx.a
2 0 hw_sensor_creator.a
1264 260 hw_sensor.a
8790 11296 kernel_os.a
3152 50 libc_baselibc.a
214606 33557 libs_micropython.a
16 0 libs_mynewt_rust.a
1410 0 libs_rust_app.a
1014 0 libs_rust_libcore.a
738 42 libs_semihosting_console.a
40 9 libs_sensor_coap.a
583 99 libs_sensor_network.a
677 212 libs_temp_stub.a
3428 72 mgmt_imgmgr.a
231 20 mgmt_mgmt.a
884 100 mgmt_newtmgr.a
1410 44 mgmt_newtmgr_nmgr_os.a
454 108 mgmt_newtmgr_transport_ble.a
405 388 net_oic.a
35496 2107 nimble_controller.a
4086 1203 nimble_drivers_nrf52.a
41721 2797 nimble_host.a
822 218 nimble_host_services_ans.a
241 112 nimble_host_services_dis.a
396 118 nimble_host_services_gap.a
204 62 nimble_host_services_gatt.a
1814 648 nimble_host_store_config.a
114 0 nimble_host_util.a
692 1096 nimble_transport_ram.a
1578 54 sys_config.a
634 128 sys_flash_map.a
2 0 sys_log_modlog.a
686 29 sys_mfg.a
839 51 sys_reboot.a
226 37 sys_sysdown.a
30 5 sys_sysinit.a
1746 0 time_datetime.a
120 0 util_mem.a
208 0 nrf52_my_sensor-sysinit-app.a
2016 0 libgcc.a
Loading compiler /Users/Luppy/PineTime/pinetime-rust-mynewt/repos/apache-mynewt-core/compiler/arm-none-eabi-m4, buildProfile debug
objsize
text data bss dec hex filename
347992 984 54728 403704 628f8 /Users/Luppy/PineTime/pinetime-rust-mynewt/bin/targets/nrf52_my_sensor/app/apps/my_sensor_app/my_sensor_app.elf
Mynewt includes the open-source NimBLE Bluetooth LE networking stack, so we won't be using the Nordic SoftDevice Bluetooth stack from the nRF Port of MicroPython.
Let's flip NimBLE on and watch what happens. We'll add these NimBLE libraries to our firmware: apps/my_sensor_app/pkg.yml
# Bluetooth LE
pkg.deps.BLUETOOTH_LE:
- "@apache-mynewt-core/boot/split"
- "@mcuboot/boot/bootutil"
- "@apache-mynewt-core/mgmt/imgmgr"
- "@apache-mynewt-core/mgmt/newtmgr"
- "@apache-mynewt-core/mgmt/newtmgr/transport/ble"
- "@apache-mynewt-nimble/nimble/host"
- "@apache-mynewt-nimble/nimble/host/services/ans"
- "@apache-mynewt-nimble/nimble/host/services/dis"
- "@apache-mynewt-nimble/nimble/host/services/gap"
- "@apache-mynewt-nimble/nimble/host/services/gatt"
- "@apache-mynewt-nimble/nimble/host/store/config"
- "@apache-mynewt-nimble/nimble/host/util"
- "@apache-mynewt-nimble/nimble/transport"
Here are the libraries that we have added to our Mynewt + MicroPython firmware to support Firmware Updates over Bluetooth LE...
mcuboot is the open-source MCUBoot Bootloader that we're using on PineTime. We include the mcuboot library so that PineTime Owners may query via Bluetooth LE the version number of the firmware that's installed.
imgmgr writes the received firmware image to SPI Flash during Firmware Update
newtmgr implements the MCU Manager
Simple Management Protocol for uploading firmware images over Bluetooth LE
nimble implements the Bluetooth LE Host and Transport functions (including GATT Services for Simple Management Protocol) by talking to the Bluetooth hardware on PineTime's nRF52 Microcontroller
How do we start the Bluetooth LE services on PineTime?
In Mynewt's startup function main(), we call start_ble() to start the Bluetooth LE services: rust/app/src/lib.rs
/// Rust Main Function main() will be called at Mynewt startup. /// It replaces the C version of the main() function. #[no_mangle] // Don't mangle the name "main" extern "C" fn main() -> ! { // Declare extern "C" because it will be called by Mynewt // Initialise the Mynewt drivers. Any startup functions defined in pkg.yml // of our custom drivers and libraries will be called by sysinit(). // Here are the startup functions consolidated by Mynewt: // bin/targets/nrf52_my_sensor/generated/src/nrf52_my_sensor-sysinit-app.c mynewt::sysinit(); // Start Bluetooth LE, including over-the-air firmware upgrade extern { fn start_ble() -> i32; } let rc = unsafe { start_ble() }; assert!(rc == 0, "BLE fail"); // Start MicroPython extern { fn start_micropython() -> i32; } let rc = unsafe { start_micropython() }; assert!(rc == 0, "MP fail"); ...
start_ble() is defined in my_sensor_app/src/ble_main.c, which calls supporting Bluetooth LE functions in ble_gatt_svr.c, ble_misc.c, ble_phy.c and ble_prph.h
Then main() calls start_micropython() to start the MicroPython Runtime, including wasp-os. We have seen start_micropython() in ports/mynewt/main.c
At the end of main() we see this: rust/app/src/lib.rs
extern "C" fn main() -> ! { ... // Mynewt Event Loop loop { // Loop forever... os::eventq_run( // Processing events... os::eventq_dflt_get() // From default event queue. .expect("GET fail") ).expect("RUN fail"); }
This is the Event Loop for Mynewt. It's critical because any Bluetooth LE packets received will be treated as Mynewt Events and will be handled in this loop.
So does Bluetooth LE work?
Sadly no! There's a conflict with the Task Scheduling in Mynewt and MicroPython. More about this in the next section.
Wait a minute... The main() function is in Rust not C!
Yep! We're using a variant of Mynewt that supports Rust!
So technically we're calling: C (Mynewt) → Rust → MicroPython!
But won't Rust bloat our Mynewt + MicroPython firmware unnecessarily?
Not at all! The Rust Application and Rust Core Library occupy only 2.4 KB of ROM, 0 KB of RAM! Amazing!
FLASH RAM
1410 0 libs_rust_app.a
1014 0 libs_rust_libcore.a
We have just enabled the NimBLE Bluetooth Stack on our Mynewt + MicroPython firmware... But we can't scan our PineTime device on the Nordic nRF Connect mobile app. Why?
Let's look again at the main() function: rust/app/src/lib.rs
/// Rust Main Function main() will be called at Mynewt startup. It replaces the C version of the main() function. #[no_mangle] // Don't mangle the name "main" extern "C" fn main() -> ! { // Declare extern "C" because it will be called by Mynewt // Initialise the Mynewt packages and internal temperature sensor driver. Any startup // functions defined in pkg.yml of our custom drivers and libraries will be called by // sysinit(). Here are the startup functions consolidated by Mynewt: // bin/targets/nrf52_my_sensor/generated/src/nrf52_my_sensor-sysinit-app.c mynewt::sysinit(); // Start Bluetooth LE, including over-the-air firmware upgrade extern { fn start_ble() -> i32; } let rc = unsafe { start_ble() }; assert!(rc == 0, "BLE fail"); // Start MicroPython extern { fn start_micropython() -> i32; } let rc = unsafe { start_micropython() }; assert!(rc == 0, "MP fail"); // OOPS: start_micropython() will never return, // because it's stuck in a MicroPython Event Loop. So the // Mynewt Event Loop below will never run... // Mynewt Event Loop loop { // Loop forever... os::eventq_run( // Processing events... os::eventq_dflt_get() // From default event queue. .expect("GET fail") ).expect("RUN fail"); } // Never comes here }
We see the watch face rendered by MicroPython and wasp-os, so start_micropython() must be working.
But there's a problem here... start_micropython() will be stuck in its own MicroPython Event Loop taking care of MicroPython tasks, so it never returns!
As a result, the Mynewt Event Loop will never be executed. And we know that the Mynewt Event Loop is critical for handling Bluetooth packets. Thus Bluetooth fails!
We have a conflict here between MicroPython and Mynewt and the way they handle tasks and events. We don't have a solution right now, but we'll probably need to modify the wasp-os / MicroPython Event Loop to handle Mynewt Events as well.
But for now, we have accomplished our objective of rendering a watch face on PineTime with wasp-os, MicroPython and Mynewt. Yay!
In the next section we'll learn to build the Mynewt + MicroPython firmware and flash it to PineTime.
MicroPython and wasp-os hosted on Mynewt on PineTime Smart Watch. VSCode Debugger is shown on the monitor
Follow these steps to build the Mynewt + MicroPython Firmware on Linux (including Raspberry Pi) and macOS...
Install GCC and Python build tools for Linux (or the macOS equivalent)...
sudo apt install gcc gcc-arm-none-eabi python3 make
Install rustup with support for nightly target thumbv7em-none-eabihf.
Follow the instructions at https://rustup.rs/
Press Enter to select 1) Proceed with installation (default)
Then execute...
# Latest nightly-2020-04-20 fails with asm error, so we use nightly-2020-02-16
source $HOME/.cargo/env
rustup default nightly-2020-02-16
rustup update
rustup target add thumbv7em-none-eabihf
Install the newt build tool for Mynewt. Refer to these scripts...
scripts/install-version.sh: To set the version numbers
scripts/install-pi.sh: To build and install newt, look under the section "Build newt"
We'll download the source files into these folders...
Download the source files to ~/pinetime...
mkdir ~/pinetime
cd ~/pinetime
git clone --recursive --branch master https://github.com/lupyuen/wasp-os
git clone --recursive --branch micropython https://github.com/lupyuen/pinetime-rust-mynewt
Update the MCUBoot version number. Edit ~/pinetime/pinetime-rust-mynewt/project.yml
Change...
repository.mcuboot:
type: github
vers: 1.5.0
to...
repository.mcuboot:
type: github
vers: 1.3.1
Download the Mynewt source code...
cd ~/pinetime/pinetime-rust-mynewt
newt install
We should see...
Downloading repository mynewt-core (commit: master) from https://github.com/apache/mynewt-core.git
Downloading repository mynewt-mcumgr (commit: master) from https://github.com/apache/mynewt-mcumgr.git
Downloading repository mynewt-nimble (commit: master) from https://github.com/apache/mynewt-nimble.git
Downloading repository mcuboot (commit: master) from https://github.com/JuulLabs-OSS/mcuboot.git
Making the following changes to the project:
install apache-mynewt-core (1.7.0)
install apache-mynewt-nimble (1.2.0)
install mcuboot (1.3.1)
apache-mynewt-core successfully installed version 1.7.0
apache-mynewt-nimble successfully installed version 1.2.0
Error: Error updating "mcuboot": error: The following untracked working tree files would be overwritten by checkout:
ext/mbedtls/include/mbedtls/check_config.h
ext/mbedtls/include/mbedtls/config.h
Please move or remove them before you switch branches.
Aborting
Ignore the mcuboot error above and proceed to the next step.
Build the Mynewt firmware (and generate the syscfg.h config file)...
cd ~/pinetime/pinetime-rust-mynewt
scripts/build-app.sh
We should see...
Linking my_sensor_app.elf
Error: ld: libc_baselibc.a(start.o): in function `_start':
start.c:39: undefined reference to `main'
collect2: error: ld returned 1 exit status
Ignore the Undefined reference to main error above and proceed to the next step.
cd ~/pinetime/wasp-os
export BUILD_VERBOSE=1
make -j 1 BOARD=pinetime micropython
We should see...
arm-none-eabi-size build-pinetime/micropython.a
text data bss dec hex filename
0 0 0 0 0 mpstate.o (ex build-pinetime/micropython.a)
32 0 0 32 20 nlr.o (ex build-pinetime/micropython.a)
0 0 0 0 0 nlrx86.o (ex build-pinetime/micropython.a)
...
1041 0 0 1041 411 pins_gen.o (ex build-pinetime/micropython.a)
make[1]: Leaving directory 'pinetime/wasp-os/micropython/ports/mynewt'
Build the Mynewt firmware again...
cd ~/pinetime/pinetime-rust-mynewt
scripts/build-app.sh
We should see...
objsize
text data bss dec hex filename
346216 984 54720 401920 62200 pinetime/pinetime-rust-mynewt/bin/targets/nrf52_my_sensor/app/apps/my_sensor_app/my_sensor_app.elf
+ set +x
+ newt create-image nrf52_my_sensor 1.0.0
App image successfully generated: pinetime/pinetime-rust-mynewt/bin/targets/nrf52_my_sensor/app/apps/my_sensor_app/my_sensor_app.img
If you see the error Undefined main, run scripts/build-app.sh again. It should fix the error.
Create the Firmware Image...
scripts/nrf52/image-app.sh
We should see...
App image successfully generated: pinetime/pinetime-rust-mynewt/bin/targets/nrf52_my_sensor/app/apps/my_sensor_app/my_sensor_app.img
Edit ~/pinetime/pinetime-rust-mynewt/scripts/config.sh
If we're using ST-Link v2 for flashing PineTime, set swd_device as follows...
# Select ST-Link v2 as SWD Programmer
swd_device=scripts/nrf52/swd-stlink.ocd
If we're using Raspberry Pi SPI for flashing PineTime, set swd_device as follows...
# Select Raspberry Pi as SWD Programmer
swd_device=scripts/nrf52-pi/swd-pi.ocd
Download the MCUBoot Bootloader for PineTime...
For other versions of the bootloader, see this article
Edit ~/pinetime/pinetime-rust-mynewt/scripts/nrf52/flash-boot.sh
Change openocd/bin/openocd to the path of our installed openocd (for ST-Link) or openocd-spi (for Raspberry Pi)...
# Flash the device
openocd/bin/openocd \
-f $swd_device \
-f scripts/nrf52/flash-boot.ocd
Edit ~/pinetime/pinetime-rust-mynewt/scripts/nrf52/flash-boot.ocd
Change bin/targets/nrf52_boot/app/boot/mynewt/mynewt.elf.bin to the path of the downloaded mynewt.elf.bin
# For MCUBoot (debugging not supported):
program bin/targets/nrf52_boot/app/boot/mynewt/mynewt.elf.bin verify 0x00000000
Flash the bootloader...
scripts/nrf52/flash-boot.sh
Edit ~/pinetime/pinetime-rust-mynewt/scripts/nrf52/flash-app.sh
Change openocd/bin/openocd to the path of our installed openocd (for ST-Link) or openocd-spi (for Raspberry Pi)...
# Flash the device
openocd/bin/openocd \
-f $swd_device \
-f scripts/nrf52/flash-app.ocd
The path of the built firmware file is defined in ~/pinetime/pinetime-rust-mynewt/scripts/nrf52/flash-app.ocd. We shouldn't need to change this.
program bin/targets/nrf52_my_sensor/app/apps/my_sensor_app/my_sensor_app.img verify 0x00008000
Flash the application...
scripts/nrf52/flash-app.sh
PineTime reboots, starts wasp-os and renders the watch face
Here's the debug log that appears in OpenOCD when we run flash-app.sh...
The firmware is flashed to PineTime via OpenOCD and ST-Link...
> Executing task in folder pinetime-rust-mynewt: bash -c -l ' scripts/nrf52/flash-app.sh && echo ✅ ◾ ️Done! ' <
+ source scripts/config.sh
++ swd_device=scripts/nrf52/swd-stlink.ocd
+ openocd/bin/openocd -f scripts/nrf52/swd-stlink.ocd -f scripts/nrf52/flash-app.ocd
GNU MCU Eclipse 64-bit Open On-Chip Debugger 0.10.0+dev-00462-gdd1d90111 (2019-01-15-13:49)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
debug_level: 0
adapter speed: 1000 kHz
force hard breakpoints
Stopping...
target halted due to debug-request, current mode: Thread
xPSR: 0x61000000 pc: 0x000001c6 msp: 0x2000ffd8
Flashing Application...
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x000000d8 msp: 0x20010000
Enabled ARM Semihosting to show debug output
semihosting is enabled
** Programming Started **
auto erase enabled
target halted due to breakpoint, current mode: Thread
xPSR: 0x61000000 pc: 0x2000001e msp: 0x20010000, semihosting
wrote 352256 bytes from file bin/targets/nrf52_my_sensor/app/apps/my_sensor_app/my_sensor_app.img in 8.465226s (40.637 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20010000, semihosting
verified 349048 bytes in 0.803118s (424.430 KiB/s)
** Verified OK **
Restarting...
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x000000d8 msp: 0x20010000, semihosting
Enabled ARM Semihosting to show debug output
semihosting is enabled
**** Done! Press Ctrl-C to exit...
PineTime reboots after flashing. The MCUBoot Booloader starts...
Starting Bootloader...
Displaying image...
Image displayed
Check button: 0
[INF] Primary image: magic=good, swap_type=0x4, copy_done=0x1, image_ok=0x1
[INF] Scratch: magic=bad, swap_type=0x1, copy_done=0x2, image_ok=0x2
[INF] Boot source: none
[INF] Swap type: none
Waiting 5 seconds for button: 0...
Waited for button: 0
Bootloader done
Mynewt starts running...
TMP create temp_stub_0
NET hwid 4a f8 cf 95 6a be c1 f6 89 ba 12 1a
NET standalone node
MicroPython initialises the stack and heap...
stack_start: 20000618
stack_end: 20002618
heap_start: 20004bfa
heap_end: 2000cbfa
TODO adc_init0
TODO timer_init0
TODO microbit_filesystem_init
TODO machine_rtc_make_new
TODO machine_rtc_start
TODO machine_adc_make_new
1 chips detected. Total flash size 4MiB.
stack_end: 20002618
gc_collect: sp=20001dc0, len=216
stack_end: 20002618
gc_collect: sp=20001f28, len=1bc
TODO machine_adc_value_read
TODO machine_adc_value_read
The ADC is probably used for reading the battery status.
wasp-os starts the REPL loop and renders the watch face...
Watch is running, use Ctrl-C to stop
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
stack_end: 20002618
gc_collect: sp=20001f38, len=1b8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
After a while PineTime reboots, probably because of the Mynewt watchdog.
And PineTime runs the same thing again and again...
Starting Bootloader...
Displaying image...
Image displayed
Check button: 0
[INF] Primary image: magic=good, swap_type=0x4, copy_done=0x1, image_ok=0x1
[INF] Scratch: magic=bad, swap_type=0x1, copy_done=0x2, image_ok=0x2
[INF] Boot source: none
[INF] Swap type: none
Waiting 5 seconds for button: 0...
Waited for button: 0
Bootloader done
TMP create temp_stub_0
NET hwid 4a f8 cf 95 6a be c1 f6 89 ba 12 1a
NET standalone node
stack_start: 20000618
stack_end: 20002618
heap_start: 20004bfa
heap_end: 2000cbfa
TODO adc_init0
TODO timer_init0
TODO microbit_filesystem_init
TODO machine_rtc_make_new
TODO machine_rtc_start
TODO machine_adc_make_new
1 chips detected. Total flash size 4MiB.
stack_end: 20002618
gc_collect: sp=20001dc0, len=216
stack_end: 20002618
gc_collect: sp=20001f28, len=1bc
TODO machine_adc_value_read
TODO machine_adc_value_read
Watch is running, use Ctrl-C to stop
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
stack_end: 20002618
gc_collect: sp=20001f38, len=1b8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
TODO machine_adc_value_read
stack_end: 20002618
gc_collect: sp=200022b8, len=d8
This article is getting waaaaay too long, and I haven't covered all the topics yet. Drop me a note if you would like me to cover the following...
VSCode Workspace for wasp-os and MicroPython: The Build Task for wasp-os and MicroPython is defined here: .vscode/tasks.json
Debugging wasp-os and MicroPython with VSCode and ST-Link
How debug messages from wasp-os and MicroPython are routed to OpenOCD's Semihosting Console (i.e. replacing printf by console_printf)
REPL Command Line: Will probably be implemented as a UART driver in MicroPython that talks to NimBLE
Supporting the other wasp-os drivers in wasp-os/wasp/drivers: SPI Flash, Battery, nRF Real Time Clock, Signal, Vibrator
Will we see wasp-os and MicroPython run on Mynewt someday?
The work continues! We have a showstopper: Unable to handle Bluetooth packets due to conflicts with the Mynewt and the MicroPython task scheduler.
I'll continue to experiment on wasp-os with guidance from Daniel and keep this article updated!
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here...
pinetime-rust-mynewt/rust/ app/src/micropython.md
wasp-os Application Writer’s Guide
"MCUBoot Bootloader for PineTime Smart Watch (nRF52)"
"Firmware Update over Bluetooth Low Energy on PineTime Smart Watch"
"Wireless Firmware Update In Action on PineTime Smart Watch (nRF52)"