Data Backfill is a feature that is available optionally for our Temperature sensor and is a standard inclusion for our Temperature Probe sensors.
Where to use?
Data Backfill operates reliably in stable and permanent installations, with good and reliable connectivity as a requirement. However, in environments characterized by unstable and unreliable connectivity or use cases demanding mobility of either the sensor or Cloud Connector, Data Backfill isn't guaranteed to function.
Data Backfill primarily functions as a safety measure, designed to safeguard against data loss during service interruptions, power outages, and comparable situations that could result in data loss.
It is not intended for use as a data logging option where periodic disconnections between the sensor and Cloud Connector are deliberate and planned as a use case.
How does it work?
If the sensor goes offline, it will start storing temperature measurements locally until the connection to the cloud is restored. The sensor will then start backfilling data, starting with the most recent samples first. The sensor will overwrite the oldest data if the memory becomes full.
The number of data points that can be stored in an offline period depends on the sampling rate, heartbeat configuration, and temperature fluctuations.
The Temperature sensor has a storage capacity of up to 100,000 events, while Temperature Probe sensors can store up to 50,000 events.
It takes at least two heartbeats for backfilling to start after sensor reconnection to the Cloud Connector:
- One to trigger the retrieval request
- Another for the sensor to respond.
Whether the reported Temperature event was due to Backfill or not is currently only visible through the API. The event contains a field called isBackfilled and is documented in our Developer Docs. This is not yet available in Studio.
How long does it take?
Assuming flawless communication and already initialized backfilling, 20 backfilled heartbeats (HBIs) per 1 live HBI is the best-case scenario. However, message loss can occur, causing delays in backfilling, especially with longer heartbeats.
Example
Let's assume that the information below can be applied:
- Configured Heartbeat interval (HBI) - 30 minutes
- Heartbeats (HBI) before backfill starts - 2
- Backfill data per HBI (best case scenario) - 20
The table below shows an estimated backfill duration, but actual times may vary due to factors like connection stability.
| Offline time | HBIs skipped | Minimum HBIs needed to read offline data | Backfill duration time |
|---|---|---|---|
| 60 min | 2 | 3 | 90 min |
| 90 min | 3 | 3 | 90 min |
| 13h | 27 | 4 | 2h |
| 25h | 50 | 5 | 2.5h |
To further explain on an example from the table:
| Offline time in minutes | HBIs skipped | HBIs needed to read offline data | Backfill duration in minutes |
|---|---|---|---|
| 90 min | 3 | 3 | 90 min |
- the sensor was offline for 90 minutes, skipping 3 HBIs of data.
- assuming stable connectivity, backfilling should start after 2 HBIs, with each following HBI backfilling up to 20 HBIs of data
- to backfill the 3 missed HBIs, the sensor might need 2 HBIs to trigger backfill + 1 more to recover the data in ideal conditions.
The backfill duration remains unaffected by the sampling rate. Touching the sensor will not speed up the backfilling process.
Related articles
- Is data lost for offline sensors?
- How often is data sent from sensors?
- When is a sensor considered offline?