IoT APIv1beta1

Scaleway IoT Hub is a PaaS enabling device-to-device and device-to-Scaleway communication, you can see it as a managed message broker.

Main features:

• Pub/Sub architecture
• MQTT protocol, with or without TLS, with or without WebSockets
• Mutual authentication available on TLS connections
• High Availability and Scalability
• Routes to Scaleway's ecosystem for data ingestion
• Usage metrics

We have planned a lot of other exciting features, do not hesitate to reach out to us and say what you would like to see next!

We are available on Scaleway Community Slack. You may request access to the #iot channel by sending a message to @Gregoire.

• A single region is supported, your hub will be located in Paris, France (fr-par).
• Message rate is limited

Let's learn how to spawn your first IoT platform as a service with Scaleway.

All you need is a Scaleway account.

You will need to have curl, mosquitto-clients (mqtt client), and jq (json parsing tool) installed to follow this guide.

Your device will connect to a hub, from which you can publish or subscribe to topics. A message published to a topic will be sent to all devices which subscribed to this topic.

From a device perspective, the hub acts as a message broker. Depending on the product plan you spawn, you may end up on a shared message broker, a dedicated message broker, or a dedicated message broker cluster. The product plan also defines the number of devices you can create for this hub.

Available protocols:

• MQTT: port 1883
• MQTT-over-TLS: port 8883
• MQTT-over-Websocket: port 80
• MQTT-over-Websocket-over-TLS: port 443

There are 3 available product plans, which map to feature sets:

• plan_shared: as a rule of thumb, the broker will not store anything in memory, as such unsupported features are: persistent sessions, retained messages, QoS 2 publish messages and QoS 1 & 2 subscriptions
• plan_dedicated: single broker with full MQTT protocol support, allowed memory: 140MiB
• plan_ha: highly-available broker with full MQTT protocol support, allowed memory: 140MiB

A device connects to a hub and identifies itself with a token called the client ID and a TLS certificate matching this client ID. The client ID is the ID of the device as returned by the API.

Devices have an allow_insecure option. In this case, TLS client certificate verification is not required, and the device can also connect in plaintext MQTT or MQTT-over-TLS with server-only authentication.

The message_filters field allows for publish and subscribe restrictions for the device. For each operation (publish or subscribe) a topic list can be provided along with one of these policies:

• accept will only accept the topics listed, others will be rejected
• reject will reject listed topics, others will be allowed

Now you know what's under the hood, let's start with our first hub.

You will need to find 2 things to get you started: a token and your organization ID. You can get/create them from your Scaleway console, under the Account > Credentials page.

Some documentation here: https://www.scaleway.com/docs/generate-an-api-token/

Let's save this for later:

IOT_API="https://api.scaleway.com/iot/v1beta1"
SCW_TOKEN="<your scaleway token secret here>"
SCW_ORG="<your organization ID here>"

This is as simple as an API call, provide your organization id, a name and a product plan. We will save the output to hub.json file to make it easier later:

curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "organization_id": "$SCW_ORG",
        "name": "my_first_hub",
        "product_plan": "plan_dedicated"
        }
EOF
)" $IOT_API/hubs | jq | tee hub.json

hub.json will contain something like:

{
  "id": "a15ddd5b-ff73-4fb3-b043-1c9176dae295",
  "name": "my_first_hub",
  "organization_id": "<your organization ID>",
  "status": "creating",
  "product_plan": "plan_dedicated",
  "endpoint": "iot.fr-par.scw.cloud",
  "created_at": "2018-12-13T15:15:27.057005Z",
  "enabled": true
}

We can poll the hub status until it is ready:

curl -sS -H "X-Auth-Token: $SCW_TOKEN" $IOT_API/hubs/$(jq -r '.id' hub.json) | jq -r '.status'

At some point, the status will switch to ready.

Now let's create 2 devices, you just have to provide your organization id, hub id and a name. We will save the response to a file so we can use fields later.

curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "hub_id": "$(jq -r '.id' hub.json)",
        "name": "my_first_device",
        "allow_insecure": true
        }
EOF
)" $IOT_API/devices | jq | tee dev1.json

dev1.json should contain something like:

{
  "device": {
    "id": "ca973002-b87d-457e-8b9b-f5e19796e2b9",
    "name": "my_first_device",
    "organization_id": "<your organization ID>",
    "status": "ready",
    "hub_id": "a15ddd5b-ff73-4fb3-b043-1c9176dae295",
    "created_at": "2018-12-14T13:59:43.277945Z",
    "enabled": true,
    "allow_insecure": true
  },
  "crt": "<certificate here>",
  "key": "<certificate key here>"
}

Let's now create a second device:

curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "hub_id": "$(jq -r '.id' hub.json)",
        "name": "my_second_device",
        "allow_insecure": true
        }
EOF
)" $IOT_API/devices | jq | tee dev2.json

Ok, we have all set up, let's simulate 2 devices and send data now!

Setup the subscriber:

# In one terminal
mosquitto_sub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev1.json) \
    -t mytopic/mysubtopic

Run the publisher:

# In another terminal
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev2.json) \
    -t mytopic/mysubtopic \
    -m 'Hello, world!'

The subscriber is receiving the Hello, world! message. Yay!

If you require security, you can also send your messages with TLS mutual authentication.

You will need the IoT Hub endpoint certificate authority to verify its identity. Let's download it:

curl -sS -O https://iot.s3.nl-ams.scw.cloud/certificates/fr-par/iot-hub-ca.pem
sha1sum iot-hub-ca.pem
# 13cf3e59ed52d4c4b6bc249e85539d5fd5d572fb  iot-hub-ca.pem

Now let's extract the certificates from the device JSON files, so that the mosquitto clients may use them.

jq -r '.crt' dev1.json > dev1.crt
jq -r '.key' dev1.json > dev1.key
jq -r '.crt' dev2.json > dev2.crt
jq -r '.key' dev2.json > dev2.key

We can now run the same test as previously, with added security:

# In one terminal
mosquitto_sub \
    -h $(jq -r '.endpoint' hub.json) -p 8883 \
    --cert dev1.crt --key dev1.key --cafile iot-hub-ca.pem \
    -i $(jq -r '.device.id' dev1.json) \
    -t mytopic/mysubtopic

# In another terminal
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) -p 8883 \
    --cert dev2.crt --key dev2.key --cafile iot-hub-ca.pem \
    -i $(jq -r '.device.id' dev2.json) \
    -t mytopic/mysubtopic \
    -m 'Hello, SECURE world!'

You can mix MQTT and MQTTs clients on the same hub.

Devices information, such as events or errors, are reported on your Hub, with default topic prefix $SCW/logs. Log topic is like $SCW/logs/<severity>/<object-type>/<object-id> where:

• object-type is the type of the object concerned by the log (currently only device, for devices logs),
• object-id is the id of the object
• severity is one of error, warn, info or debug

Log payload is a json string, like:

{
    "time":"2020-01-21T14:00:52+01:00",
    "severity": "error",
    "object-type": "device",
    "object-id": "my-device-id",
    "msg":"Unauthorized topic [foo/bar] (accept policy)",
    "packet":"CONNECT: dup: false qos: 0 retain: false rLength: 13"
}

JSON fields are:

• time: timestamp of the event
• severity: message severity, same value as in log topic
• type: object type
• object-id: id of the object
• msg: log message

optional fields:

• packet: digest of the MQTT packet that triggered this log

Let's see with an example: First, we connect to previously created dev1 device as logs consumer (we want only error logs)

# In one terminal
mosquitto_sub \
    -h $(jq -r '.endpoint' hub.json) -p 1883 \
    -i $(jq -r '.device.id' dev1.json) \
    -t '$SCW/logs/error/#'

Then we create a secured-only device and try to connect it in insecure mode:

curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "hub_id": "$(jq -r '.id' hub.json)",
        "name": "my_secured_device",
        "allow_insecure": false
        }
EOF
)" $IOT_API/devices | jq | tee secure-dev.json

# In another terminal
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) -p 1883 \
    -i $(jq -r '.device.id' secure-dev.json) \
    -t mytopic/mysubtopic \
    -m 'This should fail'

dev1 will receive following message, on topic $SCW/logs/device/<secure-dev-id>/error:

{
    "time":"2020-01-17T15:01:29Z",
    "severity": "error",
    "object-type": "device",
    "object-id": "dev3-id",
    "msg":"secure device refusing mutual authentication",
    "packet":"CONNECT: dup: false qos: 0 retain: false rLength: 48"
}

Some additional information:

• We could also have used TLS without providing a client certificate, we would have had an encrypted connection but no mutual authentication.
• The plaintext test works because we set the allow_insecure flag in the devices. Same would have applied for TLS without client certificate.

You can harness the real power of MQTT Pub/Sub with a few more lines of reading.

Routes are integrations with the Scaleway ecosystem: they can route MQTT messages to Scaleway services.

This route allows you to put the payload of MQTT messages into Scaleway's Object Storage.

When creating such a route for one of your hubs, you just need to specify a topic (wildcards are ok) and a bucket. The route will subscribe on this hub to this topic, and put everything it receives into the bucket.

In the bucket, objects will be created according to the strategy you selected:

• per_topic: one object for each topic (eg: publishing to 'hello/world' will create 's3://my-bucket/my-prefix/hello/world'), new messages will overwrite objects of the same topic
• per_message: one object for each message (eg: publishing to 'hello/world' will create 's3://my-bucket/my-prefix/hello/world/2006-01-02T15:04:05.999999999Z'), new messages will create objects with different timestamps

Note: The route needs to be given write access to the bucket. To do it from the command-line, see the documentation of CreateS3Route.

This section is an addition to the "Getting started with IoT Hub" above, make sure you follow it first!

As a prerequisite, you should have s3cmd configured for Scaleway, see guide.

BUCKET="my-bucket-$RANDOM" # Buckets are globally unique, suffix with a random number
REGION="fr-par"
PREFIX="iot/messages"
# Create the bucket
s3cmd mb --region "$REGION" "s3://$BUCKET"
# Grant write access to IoT Hub S3 Route Service to your bucket
s3cmd setacl --region "$REGION" "s3://$BUCKET" --acl-grant=write:555c69c3-87d0-4bf8-80f1-99a2f757d031:555c69c3-87d0-4bf8-80f1-99a2f757d031
# Create the IoT Hub S3 Route
curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "hub_id": "$(jq -r '.id' hub.json)",
        "topic": "hello/world",
        "bucket_region": "$REGION",
        "bucket_name": "$BUCKET",
        "object_prefix": "$PREFIX",
        "strategy": "per_topic"
    }
EOF
)" $IOT_API/routes/s3 | jq | tee route.json

route.json will contain something like:

{
  "id": "e0d1e048-6f67-4c2d-8911-a75f0ef4d787",
  "organization_id": "<your organization ID>",
  "hub_id": "a15ddd5b-ff73-4fb3-b043-1c9176dae295",
  "topic": "hello/world",
  "created_at": "2018-12-13T15:17:22.052005Z",
  "bucket_region": "fr-par",
  "bucket_name": "my-bucket",
  "object_prefix": "iot/messages",
  "strategy": "per_topic"
}

Now you may publish and see the result!

sleep 5 # wait a little for the route to start
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev2.json) \
    -t hello/world \
    -m 'Hello, world!'
s3cmd get --region "$REGION" "s3://$BUCKET/$PREFIX/hello/world"

Database route allows you to store messages in your database.

When creating such a route for one of you hubs, you need to specify a topic (wildcards are allowed), a database (with valid credentials) and a query to execute. That query may contains \$TOPIC and \$PAYLOAD variables.

The route will subscribe on this hub to this topic, and execute the query onto the given database for each received message:

• First, \$TOPIC and \$PAYLOAD are replaced with the topic and payload of the received MQTT message,
• Then the generated query is executed

NOTE: topic database field must be a of text type, and payload a bytea

For now, we support only PostgreSQL database system. You can use a Scaleway Database instance, or any other PostgreSQL instance publicly accessible.

This section is an addition to the "Getting started with IoT Hub" above, make sure you follow it first!

As a prerequisite, you must have a working PostgreSQL database, with valid credentials (username and password).

# Database settings
DBHOST=<your db host>
DBPORT=<your db port>
DBNAME=<your db name>
DBUSER=<your db user>
DBPASS=<your db password>

# Create the target database table
psql -h $DBHOST -d $DBPORT -U $DBUSER -W $DBNAME -c "$(cat <<-EOF
    CREATE TABLE messages (
        time timestamp,
        topic text,
        payload bytea
    )
EOF
)"

# Create the IoT Hub Database Route
# The query will insert message topic and payload with current timestamp
curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "name": "my first database route",
        "hub_id": "$(jq -r '.id' hub.json)",
        "topic": "hello/world",
        "query": "INSERT INTO messages VALUES (NOW(), \$TOPIC, \$PAYLOAD)",
        "database": {
            "host": "$DBHOST",
            "port": $DBPORT,
            "dbname": "$DBNAME",
            "username": "$DBUSER",
            "password": "$DBPASS"
        }
    }
EOF
)" $IOT_API/routes/database | jq | tee route.json

route.json will contain something like

{
  "id": "486944d3-848e-4886-8945-938fbb828aa6",
  "name": "my first database route",
  "organizationId": "<your organization id>",
  "hubId": "<your hub id>",
  "topic": "hello/world",
  "createdAt": "2019-10-11T13:58:42.710Z",
  "query": "INSERT INTO messages VALUES (NOW(), $TOPIC, $PAYLOAD)",
  "database": {
    "host": "127.0.0.1",
    "port": 5432,
    "dbname": "route_tests",
    "username": "jdoe",
    "password": ""
  }
}

Now you can publish a message and checks it is inserted into the message table.

sleep 5 # wait a little for the route to start
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev2.json) \
    -t hello/world \
    -m 'Hello, world!'
psql -h $DBHOST -p $DBPORT -U $DBUSER $DBNAME -c "SELECT * FROM messages"

But you are not limited to binary payloads! You can use all postgresql functions to manipulate $TOPIC and $PAYLOAD placeholders. For example, if you know published payloads are always float values, you can store it as a float into the database.

Let's say you have a temperature sensor publishing on 'temperature/living-room' topic. You could setup the following configuration to store it into your database:

# Create the target database table
psql -h $DBHOST -d $DBPORT -U $DBUSER -W $DBNAME -c "$(cat <<-EOF
    CREATE TABLE temperatures (
        time timestamp,
        room text,
        value double precision
    )
EOF
)"

# Create the IoT Hub Database Route
# The query
#   1. extract topic 2nd part (temperature/living-room -> living-room) stored in _room_ field,
#   2. converts the binary payload into a float stored in _value_ field
curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "name": "temperature",
        "hub_id": "$(jq -r '.id' hub.json)",
        "topic": "temperature/+",
        "query": "INSERT INTO messages VALUES (NOW(), split_part(\$TOPIC, '/', 2), cast(encode(\$PAYLOAD, 'escape') as double precision))",
        "database": {
            "host": "$DBHOST",
            "port": $DBPORT,
            "dbname": "$DBNAME",
            "username": "$DBUSER",
            "password": "$DBPASS"
        }
    }
EOF
)" $IOT_API/routes/database

Try by publishing a sample temperature notification

sleep 5 # wait a little for the route to start
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev2.json) \
    -t 'temperature/bedroom' \
    -m '19.5'

In the same manner, you can also manipulate JSON payload, call plsql functions, ... Following example is logging building access attemps in non-working hours. Those are notified by badge readers, that publish a json payload on 'access-control/office/'

# Create the target database table
psql -h $DBHOST -d $DBPORT -U $DBUSER -W $DBNAME -c "$(cat <<-EOF
    CREATE TABLE access_logs (
        time timestamp PRIMARY KEY,
        login varchar(128)
    );

    CREATE OR REPLACE FUNCTION nonworkinghours(at timestamp, login text) RETURNS void
    AS $$
    DECLARE
        hour INTEGER;
    BEGIN
        hour := EXTRACT(hour FROM at);
        IF hour <= 8 OR hour >= 22 THEN
            INSERT INTO access_logs VALUES (at, user);
        END IF;
    END;
    $$ LANGUAGE plpgsql;
EOF
)"

# Create the IoT Hub Database Route
# NOTE: the query decodes the payload as json, then extract both 'timestamp' and 'user' values, and
finally executes `nonworkinghours` function with timestamp and user values as parameters
curl -sS -H "X-Auth-Token: $SCW_TOKEN" -d "$(cat <<-EOF
    {
        "name": "non-working hours office access",
        "hub_id": "$(jq -r '.id' hub.json)",
        "topic": "access-control/office/#",
        "query": "SELECT nonworkinghours(encode(\$PAYLOAD, 'escape')::json->'timestamp', encode(\$PAYLOAD, 'escape')::json->'user')",
        "database": {
            "host": "$DBHOST",
            "port": $DBPORT,
            "dbname": "$DBNAME",
            "username": "$DBUSER",
            "password": "$DBPASS"
        }
    }
EOF
)" $IOT_API/routes/database

Try by publishing some access events

sleep 5 # wait a little for the route to start
mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev2.json) \
    -t access-control/office/reader1 \
    -m "$(cat <<-EOF
        {
            "timestamp": "2019-10-11T11:21:12",
            "user": "jdoe",
            "door": "front",
            "access": True
        }
EOF
)"

mosquitto_pub \
    -h $(jq -r '.endpoint' hub.json) \
    -i $(jq -r '.device.id' dev2.json) \
    -t access-control/office/reader2 \
    -m "$(cat <<-EOF
        {
            "timestamp": "2019-10-11T04:10:44",
            "user": "thief",
            "door": "rear",
            "access": False
        }
EOF
)"

psql -h $DBHOST -p $DBPORT -U $DBUSER $DBNAME -c "SELECT * FROM access_logs"

Rest route allows you call any http(s) endpoint on received MQTT message. You can choose the HTTP verb use to call your REST uri, as well as adding extra headers.

Kickstarts are just a set of scripts based on cloud-init to help you build an IoT application as quickly as possible, they will install and setup the required software for you on a Scaleway compute instance.

Prerequisites:

• you have spawned an IoT Hub;
• you have created a device on this Hub with allow_insecure flag set to true.

The kickstart software will connect to this hub as this device.

They are very simple to use: when spawning a Scaleway compute instance (tested on Ubuntu Bionic x64), you should tick the Cloud-init box in the advanced options and copy/paste a script into the box.

This kickstart will provide you with a Time Series Database (InfluxDB) and a visualization software (Grafana) connected to your IoT Hub with a MQTT-to-InfluxDB bridge (Telegraf).

Cloud Init script:

#!/bin/bash
MQTT_ENDPOINT=iot.fr-par.scw.cloud
MQTT_CLIENT_ID=<put your Device ID here>
GRAFANA_PW=<choose a grafana admin password here>
source <(curl -s https://iot.s3.nl-ams.scw.cloud/apps/dataviz/setup-script.sh)

After setup you can access Grafana at http://<instance ip>:3000/ ( admin / <GRAFANA_PW chosen above> )

Now, any device, publishing a numeric on any topic will be recorded in the InfluxDB. The values will be displayed in the Grafana interface.

This kickstart will provide you with a flow programming interface allowing to build applications by just placing function blocks and connecting them together.

Cloud Init script:

#!/bin/bash
MQTT_ENDPOINT=iot.fr-par.scw.cloud
MQTT_CLIENT_ID=<put your Device ID here>
NODERED_PW=<choose a nodered admin password here>
source <(curl -s https://iot.s3.nl-ams.scw.cloud/apps/nodered/setup-script.sh)

After setup you can access Node Red at http://<instance ip>:1880/ ( admin / <NODERED_PW above> )

Now, any device, publishing on any topic, will trigger an event into the NodeRed engine.

This kickstart will provide you with an ElasticSearch / Kibana stack to add logging features to your IoT Hub. The kickstart will subscribe on the MQTT topic (wildcard allowed) you provide and store received messages in ElasticSearch, then you can visualize them in Kibana. Message payload format :

• Numeric : will appear in value field
• JSON : will be decoded (1st level depth), each field will appear in Kibana
• anything else: will appear in payload field

Cloud Init script:

#!/bin/bash
MQTT_ENDPOINT=iot.fr-par.scw.cloud
MQTT_CLIENT_ID=<put your Device ID here>
MQTT_TOPIC='#'
KIBANA_PW=aVEryNIcePassWord%42
source <(curl -s https://iot.s3.nl-ams.scw.cloud/apps/logging/setup-script.sh)

After setup you can access Kibana at http://<instance ip>/ ( admin / <KIBANA_PW above> )

This kickstart will set up a Nuclio function as a service with an example function connected to your IoT Hub. You can choose the MQTT topic (wildcard allowed) which will trigger your example function. The example function simply sends back a message to the events topic upon being triggered.

Cloud Init script:

#!/bin/bash
MQTT_ENDPOINT=iot.fr-par.scw.cloud
MQTT_CLIENT_ID=<put your Device ID here>
MQTT_TOPIC='please/react/to/#'
NUCLIO_PW=aVEryNIcePassWord%42
source <(curl -s https://iot.s3.nl-ams.scw.cloud/apps/faas/setup-script.sh)

After setup you can access Nuclio at http://<instance ip>/ ( admin / <NUCLIO_PW above> )

Please refer to our online documentation to get more information.