An Introduction to Data Connectors

In this short introductory guide, a general overview of how Data Connectors work will be given with the addition of how to create a new Data Connector for real-time events forwarding to an endpoint of your choice. If you're interested in learning more, our Data Connectors Reference goes into more details about the technical aspect of how they work.

Overview

All sensors send their data to the DT Cloud, where the data is stored for 30 days. However, many use-cases often require specific functionality beyond what DT Studio currently supports. For this, Data Connectors provide a simple yet reliable way to get your data into external services.

  1. A sensor sends a data event to DT Cloud via a Cloud Connector.
  2. DT Cloud uses the Data Connector to forward the events to your cloud.
  3. Your cloud acknowledges that the event was received.

data-connector-overview.svg

Each event is sent as an HTTPS POST request with a JSON payload. Depending on the sensor type, the fields contained in the payload may differ. However, the general structure does remain the same, as shown in the following snippet representing a temperature sensor event. A list of all event types can be found in the Events Reference.

{
  "event": {
    "eventId": "bjeho5nlafj3bdrehgsg",
    "targetName": "projects/bhmh0143iktucae701vg/devices/bchonod7rihjtvdmd2vg",
    "eventType": "temperature",
    "data": {
      "temperature": {
        "value": 24.9,
        "updateTime": "2019-05-16T08:15:18.318751Z"
      }
    },
    "timestamp": "2019-05-16T08:15:18.318751Z"
  },
  "labels": {}
}

Benefits

Data Connectors have three properties that make them particularly well suited for integrating your sensor data from DT Cloud into an external service.

At-Least-Once Guarantee

To keep your code lean, we promise an at-least-once delivery guarantee.

data-connector-at-least-once.svg

The Data Connector is similar to a webhook but with an added delivery guarantee. Every event received by DT Cloud is put in a dedicated per-Data Connector queue. Messages are only removed from this queue once acknowledged, or if the message is older than 12 hours. This means that if your endpoint goes offline for a while, you will still receive the data when it comes back up again.

An important side effect of this delivery guarantee is that, under certain conditions, you may receive duplicates of the same event. This is rare but should be considered.

Low Latency

Data Connectors give you the lowest possible end-to-end latency from the Sensor to your cloud.

data-connector-low-latency.svg

DT cloud will create one HTTPS POST request for every event we receive from the sensor. Because we do not introduce artificial wait states to aggregate events, latency is kept as low as possible no matter how many sensors you have.

High Scalability

Data Connectors scale well. DT cloud will never wait for one HTTPS POST request to complete before sending the next. A new event will always instantly be sent to your cloud, running as many HTTPS POST requests in parallel as there are events at any given moment.

Creating a New Data Connector

New Cloud Connectors can be created by following only a few simple steps.

  1. Navigate to your Project in DT Studio and locate Data Connectors in the left menu. Press Add to create a new Data Connector.

    AddFirstDataConnector.png


  2. Give the Data Connector a display name. The Endpoint URL is where the Data Connector will forward the data. We will use dweet.io in this example, a free hosted service that fulfills the same role as a server. If you already have a hosted HTTPS POST endpoint set up, feel free to edit the Data Connector to point toward your server instead.

    DweetExample.png

    Please note that Disruptive Technologies is not affiliated with dweet.io in any way and that data sent in this way is publicly available.

  3. We will here skip Signature Secret, Payload Configuration, Label Data, and Custom HTTP Request Headers. These are, however, explained in more detail in our Data Connectors Reference. In particular, including a Signature Secret is highly recommended for integrations. For now, save your new Data Connector.

  4. You are now finished setting up a new Data Connector. To forward an event, either wait for a sensor in the project to send a new event by its heartbeat interval or force it by touch. The Data Connector page success counter will iterate as they occur.
    dcon_success.png
  5. By navigating to https://dweet.io/follow/SOME-UNIQUE-NAME, under the Raw tab, the event payload should be clearly visible as they are forwarded.