📝 10 Jan 2022
Suppose we’re creating an IoT Gadget with Apache NuttX OS that transmits Sensor Data from two sensors: Temperature Sensor and Light Sensor…
{
"t": 1234,
"l": 2345
}
(Located in a Greenhouse perhaps)
And we’re transmitting over a low-power wireless network like LoRa, Zigbee or Bluetooth LE.
We could transmit 19 bytes of JSON. But there’s a more compact way to do it….
Concise Binary Object Representation (CBOR), which works like a binary, compressed form of JSON.
And we need only 11 bytes of CBOR!
Today we’ll learn to encode Sensor Data with the TinyCBOR Library that we have ported to Apache NuttX OS…
The library has been tested on PineDio Stack BL604, but it should work on any NuttX Platform (like ESP32)
(tinycbor-nuttx is a fork of TinyCBOR with minimal changes)
Must we scrimp and save every single byte?
Yes, every single byte matters for low-power wireless networks!
Low-power wireless networks operate on Radio Frequency Bands that are shared with many other gadgets.
They are prone to collisions and interference.
The smaller the data packet, the higher the chance that it will be transmitted successfully!
When we transmit LoRa packets to The Things Network (the free public global LoRa network), we’re limited by their Fair Use Policy.
(Roughly 12 bytes per message, assuming 10 messages per hour)
JSON is too big for this. But CBOR works well!
(In the next article we’ll watch the TinyCBOR Library in action for encoding Sensor Data in The Things Network)
We begin by encoding one data field into CBOR…
{
"t": 1234
}
We call this a CBOR Map that maps a Key (“t”) to a Value (1234)…
Let’s look at the code from our NuttX App that encodes the above into CBOR…
First we create an Output Buffer that will hold the encoded CBOR data: tinycbor_test_main.c
#include "../libs/libtinycbor/src/cbor.h" // For TinyCBOR Library
/// Test CBOR Encoding for { "t": 1234 }
static void test_cbor(void) {
// Max output size is 50 bytes (which fits in a LoRa packet)
uint8_t output[50];
(50 bytes is the max packet size for LoRaWAN AS923 Data Rate 2)
Output Buffer Size is important: Calls to the TinyCBOR library will fail if we run out of buffer space!
Next we define the CBOR Encoder (from TinyCBOR) that will encode our data…
// Our CBOR Encoder and Map Encoder
CborEncoder encoder, mapEncoder;
As well as the Map Encoder that will encode our CBOR Map.
We initialise the CBOR Encoder like so…
// Init our CBOR Encoder
cbor_encoder_init(
&encoder, // CBOR Encoder
output, // Output Buffer
sizeof(output), // Output Buffer Size
0 // Options (always 0)
);
Now we create the Map Encoder that will encode our CBOR Map…
// Create a Map Encoder that maps keys to values
CborError res = cbor_encoder_create_map(
&encoder, // CBOR Encoder
&mapEncoder, // Map Encoder
1 // Number of Key-Value Pairs
);
assert(res == CborNoError);
The last parameter (1) is important: It must match the Number of Key-Value Pairs (like "t": 1234) that we shall encode.
We encode the Key (“t”) into the CBOR Map…
// First Key-Value Pair: Map the Key
res = cbor_encode_text_stringz(
&mapEncoder, // Map Encoder
"t" // Key
);
assert(res == CborNoError);
Followed by the Value (1234)…
// First Key-Value Pair: Map the Value
res = cbor_encode_int(
&mapEncoder, // Map Encoder
1234 // Value
);
assert(res == CborNoError);
cbor_encode_int encodes 64-bit Signed Integers.
(We’ll look at other data types in a while)
We’re done with our CBOR Map, so we close the Map Encoder…
// Close the Map Encoder
res = cbor_encoder_close_container(
&encoder, // CBOR Encoder
&mapEncoder // Map Encoder
);
assert(res == CborNoError);
Our CBOR Encoding is complete!
To work with the Encoded CBOR Output, we need to know how many bytes have been encoded…
// How many bytes were encoded
size_t output_len = cbor_encoder_get_buffer_size(
&encoder, // CBOR Encoder
output // Output Buffer
);
printf("CBOR Output: %d bytes\n", output_len);
For the demo we dump the encoded CBOR data to the console…
// Dump the encoded CBOR output (6 bytes):
// 0xa1 0x61 0x74 0x19 0x04 0xd2
for (int i = 0; i < output_len; i++) {
printf(" 0x%02x\n", output[i]);
}
}
And that’s how we call the TinyCBOR Library to work with CBOR data!
Let’s watch what happens when we run the firmware…
To test CBOR Encoding on NuttX, download the modified source code for NuttX OS and NuttX Apps…
mkdir nuttx
cd nuttx
git clone --recursive --branch cbor https://github.com/lupyuen/nuttx nuttx
git clone --recursive --branch cbor https://github.com/lupyuen/nuttx-apps apps
Or if we prefer to add TinyCBOR to our NuttX Project, follow these instructions…
(For PineDio Stack BL604: The TinyCBOR Library and Test App are already preinstalled)
Let’s build the NuttX Firmware with TinyCBOR inside…
Install the build prerequisites…
Assume that we have downloaded the NuttX Source Code…
Configure the build…
cd nuttx
## For BL602: Configure the build for BL602
./tools/configure.sh bl602evb:nsh
## For PineDio Stack BL604: Configure the build for BL604
./tools/configure.sh bl602evb:pinedio
## For ESP32: Configure the build for ESP32.
## TODO: Change "esp32-devkitc" to our ESP32 board.
./tools/configure.sh esp32-devkitc:nsh
## Edit the Build Config
make menuconfig
In menuconfig, check the box for…
“Library Routines” → “TinyCBOR Library”
Check the box for…
“Application Configuration” → “Examples” → “TinyCBOR Test App”
Save the configuration and exit menuconfig
Build, flash and run the NuttX Firmware…
In the NuttX Shell, enter…
tinycbor_test
We’ll see 6 bytes of Encoded CBOR Output for “test_cbor”…
test_cbor: Encoding { "t": 1234 }
CBOR Output: 6 bytes
0xa1
0x61
0x74
0x19
0x04
0xd2
We have just compressed 10 bytes of JSON…
{
"t": 1234
}
Into 6 bytes of CBOR.
We have scrimped and saved 4 bytes!
Now we add another field to our CBOR Encoding…
{
"t": 1234,
"l": 2345
}
And watch how our program changes to accommodate the second field.
We begin the same way as before: tinycbor_test_main.c
/// Test CBOR Encoding for { "t": 1234, "l": 2345 }
static void test_cbor2(void) {
// Max output size is 50 bytes (which fits in a LoRa packet)
uint8_t output[50];
// Our CBOR Encoder and Map Encoder
CborEncoder encoder, mapEncoder;
// Init our CBOR Encoder
cbor_encoder_init( ... );
Now we create the Map Encoder with a tiny modification…
// Create a Map Encoder that maps keys to values
CborError res = cbor_encoder_create_map(
&encoder, // CBOR Encoder
&mapEncoder, // Map Encoder
2 // Number of Key-Value Pairs
);
assert(res == CborNoError);
We changed the Number of Key-Value Pairs to 2.
(Previously it was 1)
We encode the First Key and Value the same way as before…
// First Key-Value Pair: Map the Key
res = cbor_encode_text_stringz(
&mapEncoder, // Map Encoder
"t" // Key
);
assert(res == CborNoError);
// First Key-Value Pair: Map the Value
res = cbor_encode_int(
&mapEncoder, // Map Encoder
1234 // Value
);
assert(res == CborNoError);
(Yep no changes above)
This part is new: We encode the Second Key and Value (“l” and 2345)…
// Second Key-Value Pair: Map the Key
res = cbor_encode_text_stringz(
&mapEncoder, // Map Encoder
"l" // Key
);
assert(res == CborNoError);
// Second Key-Value Pair: Map the Value
res = cbor_encode_int(
&mapEncoder, // Map Encoder
2345 // Value
);
assert(res == CborNoError);
And the rest of the code is the same…
// Close the Map Encoder
res = cbor_encoder_close_container( ... );
// How many bytes were encoded
size_t output_len = cbor_encoder_get_buffer_size( ... );
// Dump the encoded CBOR output (11 bytes):
// 0xa2 0x61 0x74 0x19 0x04 0xd2 0x61 0x6c 0x19 0x09 0x29
for (int i = 0; i < output_len; i++) {
printf(" 0x%02x\n", output[i]);
}
}
Recap: To add a data field to our CBOR Encoding, we…
Modify the call to cbor_encoder_create_map and update the Number of Key-Value Pairs (2)
Add the new Key and Value (“l” and 2345) to the CBOR Map
Everything else stays the same.
In the NuttX Shell, enter…
tinycbor_test
We’ll see 11 bytes of Encoded CBOR Output for “test_cbor2”…
test_cbor2: Encoding { "t": 1234, "l": 2345 }
CBOR Output: 11 bytes
0xa2
0x61
0x74
0x19
0x04
0xd2
0x61
0x6c
0x19
0x09
0x29
We have just compressed 19 bytes of JSON into 11 bytes of CBOR.
8 bytes saved!
We’ve been encoding 64-bit Signed Integers. What other Data Types can we encode?
Below are the CBOR Data Types and their respective Encoder Functions from the TinyCBOR Library…
Signed Integer (64 bits): cbor_encode_int
(We called this earlier. Works for positive and negative integers)
Unsigned Integer (64 bits): cbor_encode_uint
(Positive integers only)
Negative Integer (64 bits): cbor_encode_negative_int
(Negative integers only)
Floating-Point Number (16, 32 or 64 bits):
(See the next chapter)
Null-Terminated String: cbor_encode_text_stringz
(We called this earlier to encode our Keys)
Text String: cbor_encode_text_string
(For strings that are not null-terminated)
Byte String: cbor_encode_byte_string
(For strings containing binary data)
Boolean: cbor_encode_boolean
Null: cbor_encode_null
Undefined: cbor_encode_undefined
For the complete list of CBOR Encoder Functions, refer to the TinyCBOR docs…
CBOR Data Types are explained in the CBOR Specification…
To experiment with CBOR Encoding and Decoding, try the CBOR Playground…
The CBOR spec says that there are 3 ways to encode floats…
Half-Precision Float (16 bits): cbor_encode_half_float
(3.3 significant decimal digits. See this)
Single-Precision Float (32 bits): cbor_encode_float
(6 to 9 significant decimal digits. See this)
Double-Precision Float (64 bits): cbor_encode_double
(15 to 17 significant decimal digits. See this)
How do we select the proper float encoding?
Suppose we’re encoding Temperature Data (like 12.34 ºC) that could range from 0.00 ºC to 99.99 ºC.
This means that we need 4 significant decimal digits.
Which is too many for a Half-Precision Float (16 bits), but OK for a Single-Precision Float (32 bits).
Thus we need 5 bytes to encode the Temperature Data. (Including the CBOR Initial Byte)
Huh? If we encode an integer like 1234, we need only 3 bytes!
That’s why in this article we scale up 100 times for the Temperature Data and encode as an integer instead.
(So 1234 actually means 12.34 ºC)
2 bytes saved!
(Our scaling of Sensor Data is similar to Fixed-Point Representation)
Is it meaningful to record temperatures that are accurate to 0.01 ºC?
How much accuracy do we need for Sensor Data anyway?
The accuracy for our Sensor Data depends on…
Our monitoring requirements, and
Accuracy of our sensors
Learn more about Accuracy and Precision of Sensor Data…
For decoding CBOR packets, can we call the TinyCBOR Library?
Sure, we can call the Decoder Functions in the TinyCBOR Library…
If we’re transmitting CBOR packets to a server (or cloud), we can decode them with a CBOR Library for Node.js, Go, Rust, …
We can decode CBOR Payloads in The Things Network with a CBOR Payload Formatter…
For Grafana we used a Go Library for CBOR…
There’s even a CBOR Library for Roblox and Lua Scripting…
TinyCBOR is available on various Embedded Operating Systems…
In the next few articles we’ll build a complete IoT Sensor Device with NuttX…
We’ll take the LoRaWAN Stack from the previous article…
Read BL602’s Internal Temperature Sensor to get real Sensor Data…
Compress the Sensor Data with CBOR
(As explained in this article)
Transmit the compressed Sensor Data to The Things Network over LoRaWAN
(Pic below)
But first we’ll take a short detour to explore Rust on NuttX…
Stay Tuned!
Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…
lupyuen.github.io/src/cbor2.md
NuttX transmits a CBOR Payload to The Things Network Over LoRaWAN
(For BL602 and ESP32)
Below are the steps to build, flash and run NuttX on BL602 and ESP32.
The instructions below will work on Linux (Ubuntu), WSL (Ubuntu) and macOS.
(Instructions for other platforms)
Follow these steps to build NuttX for BL602 or ESP32…
Install the build prerequisites…
Assume that we have downloaded the NuttX Source Code and configured the build…
To build NuttX, enter this command…
make
We should see…
LD: nuttx
CP: nuttx.hex
CP: nuttx.bin
For WSL: Copy the NuttX Firmware to the c:\blflash directory in the Windows File System…
## /mnt/c/blflash refers to c:\blflash in Windows
mkdir /mnt/c/blflash
cp nuttx.bin /mnt/c/blflash
For WSL we need to run blflash under plain old Windows CMD (not WSL) because it needs to access the COM port.
In case of problems, refer to the NuttX Docs…
For ESP32: See instructions here (Also check out this article)
For BL602: Follow these steps to install blflash…
We assume that our Firmware Binary File nuttx.bin has been copied to the blflash folder.
Set BL602 / BL604 to Flashing Mode and restart the board…
For PineDio Stack BL604:
Set the GPIO 8 Jumper to High (Like this)
Disconnect the USB cable and reconnect
Or use the Improvised Reset Button (Here’s how)
For PineCone BL602:
Set the PineCone Jumper (IO 8) to the H Position (Like this)
Press the Reset Button
For BL10:
Connect BL10 to the USB port
Press and hold the D8 Button (GPIO 8)
Press and release the EN Button (Reset)
Release the D8 Button
For Ai-Thinker Ai-WB2, Pinenut and MagicHome BL602:
Disconnect the board from the USB Port
Connect GPIO 8 to 3.3V
Reconnect the board to the USB port
Enter these commands to flash nuttx.bin to BL602 / BL604 over UART…
## For Linux: Change "/dev/ttyUSB0" to the BL602 / BL604 Serial Port
blflash flash nuttx.bin \
--port /dev/ttyUSB0
## For macOS: Change "/dev/tty.usbserial-1410" to the BL602 / BL604 Serial Port
blflash flash nuttx.bin \
--port /dev/tty.usbserial-1410 \
--initial-baud-rate 230400 \
--baud-rate 230400
## For Windows: Change "COM5" to the BL602 / BL604 Serial Port
blflash flash c:\blflash\nuttx.bin --port COM5
For WSL: Do this under plain old Windows CMD (not WSL) because blflash needs to access the COM port.
(Flashing WiFi apps to BL602 / BL604? Remember to use bl_rfbin)
(More details on flashing firmware)
For ESP32: Use Picocom to connect to ESP32 over UART…
picocom -b 115200 /dev/ttyUSB0
For BL602: Set BL602 / BL604 to Normal Mode (Non-Flashing) and restart the board…
For PineDio Stack BL604:
Set the GPIO 8 Jumper to Low (Like this)
Disconnect the USB cable and reconnect
Or use the Improvised Reset Button (Here’s how)
For PineCone BL602:
Set the PineCone Jumper (IO 8) to the L Position (Like this)
Press the Reset Button
For BL10:
For Ai-Thinker Ai-WB2, Pinenut and MagicHome BL602:
Disconnect the board from the USB Port
Connect GPIO 8 to GND
Reconnect the board to the USB port
After restarting, connect to BL602 / BL604’s UART Port at 2 Mbps like so…
For Linux:
screen /dev/ttyUSB0 2000000
For macOS: Use CoolTerm (See this)
For Windows: Use putty (See this)
Alternatively: Use the Web Serial Terminal (See this)
Press Enter to reveal the NuttX Shell…
NuttShell (NSH) NuttX-10.2.0-RC0
nsh>
Congratulations NuttX is now running on BL602 / BL604!
(More details on connecting to BL602 / BL604)
macOS Tip: Here’s the script I use to build, flash and run NuttX on macOS, all in a single step: run.sh
Below are the fixes we made while porting the TinyCBOR library to NuttX…