Monitor IoT Devices in The Things Network with Prometheus and Grafana

đź“ť 21 Oct 2021

Suppose we have some IoT Devices that transmit Sensor Data (via LoRa and LoRaWAN) to The Things Network…

(That’s the free-to-use public global wireless network for IoT Devices)

IoT Devices transmitting Sensor Data to The Things Network

How shall we monitor the Sensor Data transmitted by the IoT Devices?

Today we shall monitor IoT Sensor Data by connecting open source Prometheus and Grafana to The Things Network…

Monitoring IoT Devices in The Things Network with Prometheus and Grafana

  1. The Things Network pushes our Sensor Data over MQTT in real time

  2. Our MQTT Gateway consumes the Sensor Data…

  3. And publishes the Sensor Data to our Prometheus Time Series Database…

  4. Which gets rendered as a Grafana Dashboard like this…

Monitoring Devices on The Things Network with Prometheus and Grafana

Why Prometheus and Grafana?

Prometheus works great for storing and querying IoT Sensor Data.

And Grafana works well with Prometheus for visualising IoT Sensor Data.

PineDio Stack BL604 RISC-V Board (foreground) talking to The Things Network via RAKWireless RAK7248 LoRaWAN Gateway (background)

In a while we shall demo this Prometheus + Grafana Setup with PineDio Stack BL604 RISC-V Board (pic above)

But it should work for any LoRaWAN Device connected to The Things Network… Assuming that we have configured a suitable Payload Formatter in The Things Network.

(Read on to learn how)

CBOR Payload Formatter for The Things Network

1 Payload Formatter

What’s a Payload Formatter in The Things Network?

A Payload Formatter is JavaScript Code that we configure in The Things Network to decode the Sensor Data in the LoRaWAN Message Payload.

For PineDio Stack our Sensor Data is encoded with CBOR (Concise Binary Object Representation), so we use this CBOR Payload Formatter…

Is it mandatory to use a Payload Formatter?

Yes, our MQTT Gateway will work only if we configure a suitable Payload Formatter that will decode our Sensor Data.

(More about Payload Formatters)

What if we can’t find a suitable Payload Formatter?

We can make one together! Post a comment here

1.1 Checkpoint Alpha

Let’s verify that our Payload Formatter works OK for decoding our Sensor Data…

  1. Start the LoRaWAN Firmware on our LoRaWAN Device (PineDio Stack).

    Transmit some Sensor Data every minute…

    “Run the LoRaWAN Firmware”

  2. Log on to The Things Network Console

  3. Click Applications → (Your Application) → Live Data

  4. Our Decoded Sensor Data should appear in the Live Data Table like so…

    Payload: { l: 4000, t: 4669 }

    Decoded Sensor Data in the Live Data Table

  5. Click on a message in the Live Data Table.

    We should see the decoded_payload field containing our Decoded Sensor Data…

    {
      ...
      "uplink_message": {
        ...
        "decoded_payload": {
          "l": 4000,
          "t": 4656
        }    

    These are the Light Sensor (“l”) and Temperature Sensor (“t”) values transmitted by our LoRaWAN Device (PineDio Stack).

    (Our Temperature Values are scaled up 100 times… 4656 means 46.56 ºC)

Also verify that the MQTT Server works OK at The Things Network…

  1. Start our LoRaWAN Firmware and transmit Sensor Data every minute

    (Like this)

  2. Copy the MQTT Public Address, Username and Password from The Things Network…

    “Configure The Things Network MQTT”

  3. Install the command-line tools for MQTT…

    “Download Eclipse Mosquitto”

  4. Enter this at the command line…

    # Change au1.cloud.thethings.network to our 
    # MQTT Public Address (without the port number)
    # Change YOUR_USERNAME to our MQTT Username
    # Change YOUR_PASSWORD to our MQTT Password
    
    # For Linux and macOS:
    mosquitto_sub \
      -h au1.cloud.thethings.network \
      -t "#" \
      -u "YOUR_USERNAME" \
      -P "YOUR_PASSWORD" \
      -d
    
    # For Windows:
    "c:\Program Files\Mosquitto\mosquitto_sub" ^
      -h au1.cloud.thethings.network ^
      -t "#" ^
      -u "YOUR_USERNAME" ^
      -P "YOUR_PASSWORD" ^
      -d
  5. We should see the Uplink Messages transmitted by our LoRaWAN Device…

    {
      ...
      "uplink_message": {
        ...
        "decoded_payload": {
          "l": 4000,
          "t": 4656
        }    

    Including decoded_payload and the Decoded Sensor Data.

    (See the complete message)

MQTT Gateway for Prometheus

2 MQTT Gateway for Prometheus

Now we connect our MQTT Gateway to The Things Network…

Our MQTT Gateway shall…

Follow these steps to configure our MQTT Gateway…

  1. Download the MQTT Gateway Configuration File…

    ttn-mqtt.yaml

  2. Edit the file. Fill in the MQTT Public Address, Username and Password for The Things Network (from here)…

    # Change au1.cloud.thethings.network to our MQTT Public Address
    server: tcp://au1.cloud.thethings.network:1883
    
    # Change luppy-application@ttn to our MQTT Username
    user: luppy-application@ttn
    
    # Change YOUR_API_KEY to our MQTT Password
    password: YOUR_API_KEY
  3. Note that we’re subscribing to all MQTT Topics…

    # Topic path to subscribe to. "#" means All Topics.
    topic_path: "#"
  4. Our MQTT Gateway shall extract the Device ID from the MQTT Topic Path…

    # Extract the device ID (eui-YOUR_DEVICE_EUI) 
    # from the topic path, which looks like...
    # v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up
    device_id_regex: "(.*/)?devices/(?P<deviceid>.*)/.*"

    (Which will be helpful for filtering our Sensor Data by Device ID in Grafana)

    MQTT Configuration File

2.1 Prometheus Metrics

What’s a Prometheus Metric?

A Metric is an item of Monitoring Data that’s collected and reported by Prometheus.

(Think of Metrics like CPU Usage and RAM Utilisation… Prometheus was originally created for monitoring servers)

In this article we shall use “Sensor Data” and “Metric” interchangeably, since Prometheus treats our Sensor Data as Metrics.

Let’s define the Sensor Data / Metrics that will be ingested by our MQTT Gateway…

  1. Edit ttn-mqtt.yaml

    Look for the metrics section…

    Prometheus Metrics for MQTT Gateway

  2. We define the Metric for Temperature like so…

    # Temperature Metric
    # Name of the metric in prometheus
    - prom_name: t
    
      # JSON Path of the metric in our MQTT JSON message
      mqtt_name: "uplink_message.decoded_payload.t"
    
      # Prometheus help text for this metric
      help: "Temperature"
    
      # Prometheus type for this metric.
      # Valid values are: "gauge" and "counter"
      type: gauge
    
      # Map of string to string for constant labels.
      # The labels will be attached to every Prometheus metric.
      const_labels:
        sensor_type: t    
  3. This tells MQTT Gateway: Our LoRaWAN Device (PineDio Stack) transmits a Temperature Value (named “t”) at this JSON Path…

    uplink_message.decoded_payload.t

    Which matches our JSON Message Format from Checkpoint Alpha…

    {
      ...
      "uplink_message": {
        ...
        "decoded_payload": {
          "l": 4000,
          "t": 4656
        }    

    (Our Temperature Values are scaled up 100 times… 4656 means 46.56 ºC)

    Prometheus Metrics for MQTT Gateway

  4. We define other Metrics the same way, like this Light Level Metric that’s transmitted by PineDio Stack…

    # Light Level Metric
    # Name of the metric in prometheus
    - prom_name: l
    
      # JSON Path of the metric in our MQTT JSON message
      mqtt_name: "uplink_message.decoded_payload.l"
    
      # Prometheus help text for this metric
      help: "Light Level"
    
      # Prometheus type for this metric.
      # Valid values are: "gauge" and "counter"
      type: gauge
    
      # Map of string to string for constant labels.
      # The labels will be attached to every Prometheus metric.
      const_labels:
        sensor_type: l

2.2 Start MQTT Gateway

We’re ready to start our MQTT Gateway!

Follow these steps to download and run MQTT2Prometheus…

  1. Install the latest version of Go…

    golang.org

  2. Enter this at the command line…

    # Download mqtt2prometheus
    go get github.com/hikhvar/mqtt2prometheus
    
    # For Linux and macOS:
    cd $GOPATH/src/github.com/hikhvar/mqtt2prometheus
    
    # For Windows:
    cd %GOPATH%\src\github.com\hikhvar\mqtt2prometheus
    
    # Build mqtt2prometheus
    go build ./cmd
    
    # Run mqtt2prometheus.
    # Change "ttn-mqtt.yaml" to the full path of our
    # MQTT Gateway Configuration File.
    go run ./cmd -log-level debug -config ttn-mqtt.yaml
  3. For Windows: Click “Private Network: Allow Access” when prompted

    (That’s because our MQTT Gateway starts a HTTP Server at port 9641)

  4. We should see our MQTT Gateway ingesting Sensor Data from The Things Network…

    mqttclient/mqttClient.go:20     
    Connected to MQTT Broker
    
    mqttclient/mqttClient.go:21     
    Will subscribe to topic "#"
    
    web/tls_config.go:191           
    "TLS is disabled.", "http2": false
    
    metrics/ingest.go:42    
    Got message     
    "topic": "v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up", "payload": 
    {
      ...
      "uplink_message": {
        "decoded_payload": {
          "l": 4000,
          "t": 5017
        }
  5. MQTT Gateway is now listening for HTTP Requests at port 9641!

2.3 Checkpoint Bravo

Let’s check the Sensor Data ingested by our MQTT Gateway…

  1. Start our LoRaWAN Firmware and transmit Sensor Data every minute

    (Like this)

  2. Enter this at the command-line…

    curl -v http://localhost:9641/metrics
  3. We should see our Sensor Data (Temperature and Light Level) ingested as Prometheus Metrics…

    # HELP l Light Level
    # TYPE l gauge
    l{sensor="eui-YOUR_DEVICE_EUI",
    sensor_type="l",
    topic="v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up"
    } 4000 1634364863274
    ...    
    
    # HELP t Temperature
    # TYPE t gauge
    t{sensor="eui-YOUR_DEVICE_EUI",
    sensor_type="t",
    topic="v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up"
    } 5056 1634364863274

    This says that the Light Level is 4000 and the Temperature is 50.56 ÂşC, recorded at the Timestamp of 1634364863274.

  4. Also watch for received_messages…

    # HELP received_messages received messages per topic and status
    # TYPE received_messages counter
    received_messages{status="success",
    topic="v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up"
    } 3

    This says that our MQTT Gateway has successfully processed 3 messages from The Things Network.

    Let’s move on to Prometheus…

    Prometheus Time Series Database

3 Prometheus Time Series Database

How do we push the Sensor Data / Metrics from MQTT Gateway to Prometheus?

Prometheus collects Metrics by scraping them over HTTP…

(Much like the curl command from Checkpoint Bravo)

And stores the Metrics in its Time Series Database.

(Which is super efficient for querying sensor values that vary over time)

Let’s configure and start Prometheus to scrape the Metrics from our MQTT Gateway…

  1. Download and unzip Prometheus…

    “Prometheus Download”

  2. In the unzipped folder, edit the Prometheus Configuration File…

    prometheus.yml

    Prometheus Configuration for MQTT Gateway

  3. Under the scrape_configs section, add the following…

    # Scrape configuration containing the endpoints to scrape
    scrape_configs:
    ...
    
      # Scrape The Things Network Metrics from MQTT2Prometheus
      - job_name: "ttn"
    
        # Metrics will be scraped from MQTT2Prometheus
        # at http://localhost:9641/metrics
        static_configs:
          - targets: ["localhost:9641"]
  4. Note that Prometheus will scrape the Metrics from MQTT Gateway every 15 seconds…

    # Global Configuration
    global:
    
      # Set the scrape interval to every 15 seconds
      scrape_interval: 15s
  5. Start the Prometheus Server…

    # Change this to the unzipped path of Prometheus
    cd prometheus
    
    # For Linux and macOS:
    ./prometheus
    
    # For Windows:
    prometheus.exe
  6. We should see…

    main.go:400
    "No time or size retention was set so using the default time retention" duration=15d
    
    main.go:438
    "Starting Prometheus" version="(version=2.30.3, branch=HEAD, revision=f29caccc42557f6a8ec30ea9b3c8c089391bd5df)"
    
    web.go:541
    "Start listening for connections" address=0.0.0.0:9090
    
    main.go:852
    "TSDB started"    
    
    main.go:794
    "Server is ready to receive web requests."
  7. Prometheus is now listening for HTTP Requests at port 9090!

3.1 Checkpoint Charlie

Let’s check the Metrics scraped by Prometheus from MQTT Gateway…

  1. Start our LoRaWAN Firmware and transmit Sensor Data every minute

    (Like this)

  2. Browse to our Prometheus Server…

    http://localhost:9090
  3. Enter the name of our Metric (like for Temperature)…

    t

    Like this…

    Checking the Metrics scraped by Prometheus from MQTT Gateway

  4. Click “Execute” and “Graph”

    Our Metric appears in the graph.

    (See pic above)

We’re ready for our final step… Connecting Prometheus to Grafana!

Grafana Dashboard for Prometheus

4 Grafana Dashboard

Finally we install and configure Grafana to pull the Metrics from Prometheus (over HTTP) for rendering in a Grafana Dashboard…

  1. Follow the steps below to download and install Grafana…

    “Install Grafana”

  2. Browse to our Grafana Server…

    http://localhost:3000

    Username: admin

    Password: admin

  3. In the left menu bar, click…

    Configuration → Data Sources

    Click “Add Data Source”

    Add Data Source

  4. Look for “Prometheus” and click “Select”

    Prometheus Data Source for Grafana

  5. Set the HTTP URL to…

    http://localhost:9090

    Grafana Data Source for Prometheus

  6. Click “Save & Test”

4.1 Checkpoint Delta

For our final checkpoint let’s render our Sensor Data in a Grafana Dashboard!

  1. Start our LoRaWAN Firmware and transmit Sensor Data every minute

    (Like this)

  2. In Grafana, click “Add Panel” (top right)

    Click “Add An Empty Panel”

    Add Panel

  3. Set the Data Source to “Prometheus”

    Under Metric Browser: Enter the name of our Metric (like for Temperature)…

    t

    Like this…

    Grafana Panel for Prometheus

  4. Click the “Save” button (top right)

  5. Our Sensor Data from The Things Network appears in the Grafana Dashboard!

    Monitoring Devices on The Things Network with Prometheus and Grafana

    (Remember: Our Temperature Values are scaled up 100 times)

5 Transform and Filter Sensor Data

Can we tweak the display of Sensor Data in Grafana?

Grafana lets us transform and filter the Sensor Data for our Dashboard.

First we show the Raw Sensor Data as a table…

  1. Click “Panel Title” and “Edit”

  2. Click “Table View” (at top)

    Table View for Grafana Panel

  3. Not quite what we expected… Everything gets lumped into a single column!

    Let’s split our Time Series Data into separate columns.

  4. Click “Transform” Tab (at bottom)

    Click “Add Transformation”

    Select “Labels To Fields”

  5. We should see this…

    Labels to Fields for Grafana Panel

    Much better! Our Device ID (“sensor”), Sensor Type (“t”) and Value are now in separate columns.

  6. If we’re rendering Multiple Devices or Sensor Types, we should set the Value Field Name

    (See this)

Next we filter the Sensor Data that will be rendered in our Dashboard…

  1. Click “Transform” Tab (at bottom)

    Click “Add Transformation”

    Select “Filter Data By Values”

  2. Click “Add Condition” and set the Condition…

    Grafana Panel with Filter

  3. The above filter matches the Device ID with the Regular Expression…

    eui-70b3.*

    Which means that only Device IDs starting with “eui-70b3” will be rendered.

  4. When we’re done, click the “Apply” button (top right)

5.1 Auto Dashboard Refresh

Our Grafana Dashboard doesn’t refresh automatically for real-time Sensor Data?

No worries! This neat trick will auto-refresh our Grafana Dashboard to render real-time Sensor Data…

  1. In our Grafana Dashboard, click the “Settings” button (top right)

    Dashboard Settings

  2. Under “Time Options”, uncheck “Hide Time Picker”

    Hide Time Picker

  3. Click “Save Dashboard”

  4. Click the “Refresh Interval” (top right)

    Select “5 Seconds”

    Refresh Interval

    Now our Grafana Dashboard auto-refreshes every 5 seconds!

6 MQTT with TLS Encryption

There’s a Security Risk in our configuration of MQTT Gateway…

Our MQTT Password is transmitted as clear text from our computer to The Things Network!

To secure our MQTT Password with TLS Encryption, follow the instructions here…

MQTT with TLS Encryption

6.1 Checkpoint Echo

What if we have problems enabling TLS Encryption for MQTT?

Run Wireshark on our computer and trace the TLS Certificates that are presented by The Things Network and by our computer.

The certificates should appear like this…

Tracing MQTT with TLS Encryption

7 Sensor Data Alerts

Can we create Alerts for monitoring our Sensor Data?

Like when the Temperature gets too hot?

Yes that’s possible with Prometheus Alert Manager!

“The Alertmanager handles alerts sent by client applications such as the Prometheus server.”

“It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie.”

“It also takes care of silencing and inhibition of alerts.”

More details here…

Drop me a note if you’re keen to learn about Prometheus Alerts!

Prometheus Architecture

(Prometheus Architecture)

8 What’s Next

I had fun integrating The Things Network with Prometheus and Grafana… It’s something I always wanted to do. I hope you enjoyed it too!

In the next article I’ll head back to PineDio Stack BL604 and run more IoT Experiments with LoRaWAN and The Things Network.

(Thankfully we now have a proper platform for Sensor Data visualisation and analysis: Prometheus + Grafana!)

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/prometheus.md

9 Notes

  1. If we’re rendering Multiple Sensor Types in a Grafana Panel (like Temperature and Light Level)…

    Set Labels To Fields → Value Field Name to “sensor_type”…

    Rendering multiple Sensor Types

    This fixes the graph to plot one line per Sensor Type.

  2. If we’re rendering Multiple Devices in a Grafana Panel…

    Set Labels To Fields → Value Field Name to “sensor”…

    Rendering multiple Sensor Devices

    This fixes the graph to plot one line per Device.

  3. This article is the expanded version of this Twitter Thread

10 Appendix: Configure The Things Network MQTT

Follow these steps to enable the MQTT Server in The Things Network…

  1. Log on to The Things Network Console

  2. Click Applications → (Your Application) → Integrations → MQTT

    Configure The Things Network MQTT Server

  3. Click “Generate New API Key” and copy the values for…

    (This is the only time we can see the password. Don’t forget to copy it!)

11 Appendix: Install Grafana

Follow these steps to install Grafana on Linux, macOS and Windows…

  1. Browse to grafana.com/oss/grafana

    Click “Get Grafana → Self-Managed → Download Grafana”

  2. For “Edition” select “OSS”

  3. Click Linux, macOS, Windows, Arm or Docker

    (Grafana for Linux works on WSL too)

  4. Follow the instructions to download and install Grafana

  5. For Linux and macOS: Start the Grafana Server

    # For Ubuntu and WSL
    sudo service grafana-server restart
    sudo service grafana-server status
    
    # For macOS
    brew services start grafana
  6. To test Grafana, browse to

    http://localhost:3000

    Username: admin

    Password: admin

Grafana rendering PineDio Stack’s Internal Temperature over a one-hour period, thanks to Prometheus and The Things Network

Grafana rendering PineDio Stack’s Internal Temperature over a one-hour period, thanks to Prometheus and The Things Network