Encode Sensor Data with CBOR on Apache NuttX OS

📝 10 Jan 2022

Suppose we’re creating an IoT Gadget with Apache NuttX OS that transmits Sensor Data from two sensors: Temperature Sensor and Light Sensor

{ 
  "t": 1234, 
  "l": 2345 
}

(Located in a Greenhouse perhaps)

And we’re transmitting over a low-power wireless network like LoRa, Zigbee or Bluetooth LE.

We could transmit 19 bytes of JSON. But there’s a more compact way to do it….

Concise Binary Object Representation (CBOR), which works like a binary, compressed form of JSON.

And we need only 11 bytes of CBOR!

Encoding Sensor Data with CBOR

Today we’ll learn to encode Sensor Data with the TinyCBOR Library that we have ported to Apache NuttX OS…

The library has been tested on PineDio Stack BL604, but it should work on any NuttX Platform (like ESP32)

(tinycbor-nuttx is a fork of TinyCBOR with minimal changes)

Must we scrimp and save every single byte?

Yes, every single byte matters for low-power wireless networks!

  1. Low-power wireless networks operate on Radio Frequency Bands that are shared with many other gadgets.

    They are prone to collisions and interference.

    The smaller the data packet, the higher the chance that it will be transmitted successfully!

  2. When we transmit LoRa packets to The Things Network (the free public global LoRa network), we’re limited by their Fair Use Policy.

    (Roughly 12 bytes per message, assuming 10 messages per hour)

    JSON is too big for this. But CBOR works well!

    (In the next article we’ll watch the TinyCBOR Library in action for encoding Sensor Data in The Things Network)

1 Encode Sensor Data with TinyCBOR

We begin by encoding one data field into CBOR…

{ 
  "t": 1234
}

We call this a CBOR Map that maps a Key (“t”) to a Value (1234)…

CBOR Map with 1 Key-Value Pair

Let’s look at the code from our NuttX App that encodes the above into CBOR…

1.1 Output Buffer and CBOR Encoder

First we create an Output Buffer that will hold the encoded CBOR data: tinycbor_test_main.c

#include "../libs/libtinycbor/src/cbor.h"  //  For TinyCBOR Library

/// Test CBOR Encoding for { "t": 1234 }
static void test_cbor(void) {

  //  Max output size is 50 bytes (which fits in a LoRa packet)
  uint8_t output[50];

(50 bytes is the max packet size for LoRaWAN AS923 Data Rate 2)

Output Buffer Size is important: Calls to the TinyCBOR library will fail if we run out of buffer space!

Next we define the CBOR Encoder (from TinyCBOR) that will encode our data…

  //  Our CBOR Encoder and Map Encoder
  CborEncoder encoder, mapEncoder;

As well as the Map Encoder that will encode our CBOR Map.

We initialise the CBOR Encoder like so…

  //  Init our CBOR Encoder
  cbor_encoder_init(
    &encoder,        //  CBOR Encoder
    output,          //  Output Buffer
    sizeof(output),  //  Output Buffer Size
    0                //  Options (always 0)
  );

1.2 Create Map Encoder

Now we create the Map Encoder that will encode our CBOR Map…

  //  Create a Map Encoder that maps keys to values
  CborError res = cbor_encoder_create_map(
    &encoder,     //  CBOR Encoder
    &mapEncoder,  //  Map Encoder
    1             //  Number of Key-Value Pairs
  );    
  assert(res == CborNoError);

The last parameter (1) is important: It must match the Number of Key-Value Pairs (like "t": 1234) that we shall encode.

1.3 Encode Key and Value

We encode the Key (“t”) into the CBOR Map…

  //  First Key-Value Pair: Map the Key
  res = cbor_encode_text_stringz(
    &mapEncoder,  //  Map Encoder
    "t"           //  Key
  );    
  assert(res == CborNoError);

Followed by the Value (1234)…

  //  First Key-Value Pair: Map the Value
  res = cbor_encode_int(
    &mapEncoder,  //  Map Encoder 
    1234          //  Value
  );
  assert(res == CborNoError);

cbor_encode_int encodes 64-bit Signed Integers.

(We’ll look at other data types in a while)

1.4 Close Map Encoder

We’re done with our CBOR Map, so we close the Map Encoder

  //  Close the Map Encoder
  res = cbor_encoder_close_container(
    &encoder,    //  CBOR Encoder
    &mapEncoder  //  Map Encoder
  );
  assert(res == CborNoError);

Our CBOR Encoding is complete!

1.5 Get Encoded Output

To work with the Encoded CBOR Output, we need to know how many bytes have been encoded…

  //  How many bytes were encoded
  size_t output_len = cbor_encoder_get_buffer_size(
    &encoder,  //  CBOR Encoder
    output     //  Output Buffer
  );
  printf("CBOR Output: %d bytes\n", output_len);

For the demo we dump the encoded CBOR data to the console…

  //  Dump the encoded CBOR output (6 bytes):
  //  0xa1 0x61 0x74 0x19 0x04 0xd2
  for (int i = 0; i < output_len; i++) {
    printf("  0x%02x\n", output[i]);
  }
}

And that’s how we call the TinyCBOR Library to work with CBOR data!

Let’s watch what happens when we run the firmware…

Calling the TinyCBOR Library

1.6 Download Source Code

To test CBOR Encoding on NuttX, download the modified source code for NuttX OS and NuttX Apps

mkdir nuttx
cd nuttx
git clone --recursive --branch cbor https://github.com/lupyuen/incubator-nuttx nuttx
git clone --recursive --branch cbor https://github.com/lupyuen/incubator-nuttx-apps apps

Or if we prefer to add TinyCBOR to our NuttX Project, follow these instructions…

(For PineDio Stack BL604: The TinyCBOR Library and Test App are already preinstalled)

1.7 Build The Firmware

Let’s build the NuttX Firmware with TinyCBOR inside…

  1. Install the build prerequisites…

    “Install Prerequisites”

  2. Assume that we have downloaded the NuttX Source Code

    “Download Source Code”

  3. Configure the build…

    cd nuttx
    
    # For BL602: Configure the build for BL602
    ./tools/configure.sh bl602evb:nsh
    
    # For PineDio Stack BL604: Configure the build for BL604
    ./tools/configure.sh bl602evb:pinedio
    
    # For ESP32: Configure the build for ESP32.
    # TODO: Change "esp32-devkitc" to our ESP32 board.
    ./tools/configure.sh esp32-devkitc:nsh
    
    # Edit the Build Config
    make menuconfig 
  4. In menuconfig, check the box for…

    “Library Routines”“TinyCBOR Library”

  5. Check the box for…

    “Application Configuration”“Examples”“TinyCBOR Test App”

  6. Save the configuration and exit menuconfig

    (See the .config for BL602 and BL604)

  7. Build, flash and run the NuttX Firmware…

    “Build, Flash and Run NuttX”

1.8 Magic Happens

In the NuttX Shell, enter…

tinycbor_test

We’ll see 6 bytes of Encoded CBOR Output for “test_cbor”…

test_cbor: Encoding { "t": 1234 }
CBOR Output: 6 bytes
  0xa1
  0x61
  0x74
  0x19
  0x04
  0xd2

We have just compressed 10 bytes of JSON

{ 
  "t": 1234
}

Into 6 bytes of CBOR.

We have scrimped and saved 4 bytes!

Encoded CBOR Output

2 Add Another Field

Now we add another field to our CBOR Encoding…

{ 
  "t": 1234, 
  "l": 2345 
}

And watch how our program changes to accommodate the second field.

CBOR Map with 2 Key-Value Pairs

2.1 Modify Map Encoder

We begin the same way as before: tinycbor_test_main.c

/// Test CBOR Encoding for { "t": 1234, "l": 2345 }
static void test_cbor2(void) {

  //  Max output size is 50 bytes (which fits in a LoRa packet)
  uint8_t output[50];

  //  Our CBOR Encoder and Map Encoder
  CborEncoder encoder, mapEncoder;

  //  Init our CBOR Encoder
  cbor_encoder_init( ... );

Now we create the Map Encoder with a tiny modification…

  //  Create a Map Encoder that maps keys to values
  CborError res = cbor_encoder_create_map(
    &encoder,     //  CBOR Encoder
    &mapEncoder,  //  Map Encoder
    2             //  Number of Key-Value Pairs
  );    
  assert(res == CborNoError);

We changed the Number of Key-Value Pairs to 2.

(Previously it was 1)

2.2 Encode First Key and Value

We encode the First Key and Value the same way as before…

  //  First Key-Value Pair: Map the Key
  res = cbor_encode_text_stringz(
    &mapEncoder,  //  Map Encoder
    "t"           //  Key
  );    
  assert(res == CborNoError);

  //  First Key-Value Pair: Map the Value
  res = cbor_encode_int(
    &mapEncoder,  //  Map Encoder 
    1234          //  Value
  );
  assert(res == CborNoError);

(Yep no changes above)

2.3 Encode Second Key and Value

This part is new: We encode the Second Key and Value (“l” and 2345)…

  //  Second Key-Value Pair: Map the Key
  res = cbor_encode_text_stringz(
    &mapEncoder,  //  Map Encoder
    "l"           //  Key
  );    
  assert(res == CborNoError);

  //  Second Key-Value Pair: Map the Value
  res = cbor_encode_int(
    &mapEncoder,  //  Map Encoder 
    2345          //  Value
  );
  assert(res == CborNoError);

And the rest of the code is the same…

  //  Close the Map Encoder
  res = cbor_encoder_close_container( ... );

  //  How many bytes were encoded
  size_t output_len = cbor_encoder_get_buffer_size( ... );

  //  Dump the encoded CBOR output (11 bytes):
  //  0xa2 0x61 0x74 0x19 0x04 0xd2 0x61 0x6c 0x19 0x09 0x29
  for (int i = 0; i < output_len; i++) {
    printf("  0x%02x\n", output[i]);
  }
}

Recap: To add a data field to our CBOR Encoding, we…

  1. Modify the call to cbor_encoder_create_map and update the Number of Key-Value Pairs (2)

  2. Add the new Key and Value (“l” and 2345) to the CBOR Map

Everything else stays the same.

Add a second field

2.4 Watch the Magic

In the NuttX Shell, enter…

tinycbor_test

We’ll see 11 bytes of Encoded CBOR Output for “test_cbor2”…

test_cbor2: Encoding { "t": 1234, "l": 2345 }
CBOR Output: 11 bytes
  0xa2
  0x61
  0x74
  0x19
  0x04
  0xd2
  0x61
  0x6c
  0x19
  0x09
  0x29

We have just compressed 19 bytes of JSON into 11 bytes of CBOR.

8 bytes saved!

Encoding Sensor Data with CBOR

3 CBOR Data Types

We’ve been encoding 64-bit Signed Integers. What other Data Types can we encode?

Below are the CBOR Data Types and their respective Encoder Functions from the TinyCBOR Library…

3.1 Numbers

3.2 Strings

3.3 Other Types

For the complete list of CBOR Encoder Functions, refer to the TinyCBOR docs…

CBOR Data Types are explained in the CBOR Specification…

To experiment with CBOR Encoding and Decoding, try the CBOR Playground

CBOR Playground

4 Floating-Point Numbers

The CBOR spec says that there are 3 ways to encode floats

How do we select the proper float encoding?

Suppose we’re encoding Temperature Data (like 12.34 ºC) that could range from 0.00 ºC to 99.99 ºC.

This means that we need 4 significant decimal digits.

Which is too many for a Half-Precision Float (16 bits), but OK for a Single-Precision Float (32 bits).

Thus we need 5 bytes to encode the Temperature Data. (Including the CBOR Initial Byte)

4.1 Encode Floats as Integers

Huh? If we encode an integer like 1234, we need only 3 bytes!

That’s why in this article we scale up 100 times for the Temperature Data and encode as an integer instead.

(So 1234 actually means 12.34 ºC)

2 bytes saved!

(Our scaling of Sensor Data is similar to Fixed-Point Representation)

4.2 Accuracy and Precision

Is it meaningful to record temperatures that are accurate to 0.01 ºC?

How much accuracy do we need for Sensor Data anyway?

The accuracy for our Sensor Data depends on…

  1. Our monitoring requirements, and

  2. Accuracy of our sensors

Learn more about Accuracy and Precision of Sensor Data…

5 Decode CBOR

For decoding CBOR packets, can we call the TinyCBOR Library?

Sure, we can call the Decoder Functions in the TinyCBOR Library…

If we’re transmitting CBOR packets to a server (or cloud), we can decode them with a CBOR Library for Node.js, Go, Rust,

We can decode CBOR Payloads in The Things Network with a CBOR Payload Formatter…

For Grafana we used a Go Library for CBOR

There’s even a CBOR Library for Roblox and Lua Scripting

TinyCBOR is available on various Embedded Operating Systems

Inside PineDio Stack BL604

6 What’s Next

In the next few articles we’ll build a complete IoT Sensor Device with NuttX…

  1. We’ll take the LoRaWAN Stack from the previous article…

    “LoRaWAN on Apache NuttX OS”

  2. Read BL602’s Internal Temperature Sensor to get real Sensor Data…

    “ADC and Internal Temperature Sensor Library”

  3. Compress the Sensor Data with CBOR

    (As explained in this article)

  4. Transmit the compressed Sensor Data to The Things Network over LoRaWAN

    (Pic below)

But first we’ll take a short detour to explore Rust on NuttX

Stay Tuned!

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/cbor2.md

NuttX transmits a CBOR Payload to The Things Network Over LoRaWAN

NuttX transmits a CBOR Payload to The Things Network Over LoRaWAN

7 Notes

  1. This article is the expanded version of this Twitter Thread

8 Appendix: Build, Flash and Run NuttX

(For BL602 and ESP32)

Below are the steps to build, flash and run NuttX on BL602 and ESP32.

The instructions below will work on Linux (Ubuntu), WSL (Ubuntu) and macOS.

(Instructions for other platforms)

(See this for Arch Linux)

8.1 Build NuttX

Follow these steps to build NuttX for BL602 or ESP32…

  1. Install the build prerequisites…

    “Install Prerequisites”

  2. Assume that we have downloaded the NuttX Source Code and configured the build…

    “Download Source Code”

    “Build the Firmware”

  3. To build NuttX, enter this command…

    make
  4. We should see…

    LD: nuttx
    CP: nuttx.hex
    CP: nuttx.bin

    (See the complete log for BL602)

  5. For WSL: Copy the NuttX Firmware to the c:\blflash directory in the Windows File System…

    #  /mnt/c/blflash refers to c:\blflash in Windows
    mkdir /mnt/c/blflash
    cp nuttx.bin /mnt/c/blflash

    For WSL we need to run blflash under plain old Windows CMD (not WSL) because it needs to access the COM port.

  6. In case of problems, refer to the NuttX Docs

    “BL602 NuttX”

    “ESP32 NuttX”

    “Installing NuttX”

Building NuttX

8.2 Flash NuttX

For ESP32: See instructions here (Also check out this article)

For BL602: Follow these steps to install blflash

  1. “Install rustup”

  2. “Download and build blflash”

We assume that our Firmware Binary File nuttx.bin has been copied to the blflash folder.

Set BL602 / BL604 to Flashing Mode and restart the board…

For PineDio Stack BL604:

  1. Set the GPIO 8 Jumper to High (Like this)

  2. Disconnect the USB cable and reconnect

    Or use the Improvised Reset Button (Here’s how)

For PineCone BL602:

  1. Set the PineCone Jumper (IO 8) to the H Position (Like this)

  2. Press the Reset Button

For BL10:

  1. Connect BL10 to the USB port

  2. Press and hold the D8 Button (GPIO 8)

  3. Press and release the EN Button (Reset)

  4. Release the D8 Button

For Pinenut and MagicHome BL602:

  1. Disconnect the board from the USB Port

  2. Connect GPIO 8 to 3.3V

  3. Reconnect the board to the USB port

Enter these commands to flash nuttx.bin to BL602 / BL604 over UART…

# For Linux: Change "/dev/ttyUSB0" to the BL602 / BL604 Serial Port
blflash flash nuttx.bin \
  --port /dev/ttyUSB0 

# For macOS: Change "/dev/tty.usbserial-1410" to the BL602 / BL604 Serial Port
blflash flash nuttx.bin \
  --port /dev/tty.usbserial-1410 \
  --initial-baud-rate 230400 \
  --baud-rate 230400

# For Windows: Change "COM5" to the BL602 / BL604 Serial Port
blflash flash c:\blflash\nuttx.bin --port COM5

(See the Output Log)

For WSL: Do this under plain old Windows CMD (not WSL) because blflash needs to access the COM port.

(Flashing WiFi apps to BL602 / BL604? Remember to use bl_rfbin)

(More details on flashing firmware)

Flashing NuttX

8.3 Run NuttX

For ESP32: Use Picocom to connect to ESP32 over UART…

picocom -b 115200 /dev/ttyUSB0

(More about this)

For BL602: Set BL602 / BL604 to Normal Mode (Non-Flashing) and restart the board…

For PineDio Stack BL604:

  1. Set the GPIO 8 Jumper to Low (Like this)

  2. Disconnect the USB cable and reconnect

    Or use the Improvised Reset Button (Here’s how)

For PineCone BL602:

  1. Set the PineCone Jumper (IO 8) to the L Position (Like this)

  2. Press the Reset Button

For BL10:

  1. Press and release the EN Button (Reset)

For Pinenut and MagicHome BL602:

  1. Disconnect the board from the USB Port

  2. Connect GPIO 8 to GND

  3. Reconnect the board to the USB port

After restarting, connect to BL602 / BL604’s UART Port at 2 Mbps like so…

For Linux:

screen /dev/ttyUSB0 2000000

For macOS: Use CoolTerm (See this)

For Windows: Use putty (See this)

Alternatively: Use the Web Serial Terminal (See this)

Press Enter to reveal the NuttX Shell

NuttShell (NSH) NuttX-10.2.0-RC0
nsh>

Congratulations NuttX is now running on BL602 / BL604!

(More details on connecting to BL602 / BL604)

Running NuttX

macOS Tip: Here’s the script I use to build, flash and run NuttX on macOS, all in a single step: run.sh

Script to build, flash and run NuttX on macOS

(Source)

9 Appendix: Porting TinyCBOR to NuttX

Below are the fixes we made while porting the TinyCBOR library to NuttX…