Polling the Events API

For the majority of server-to-server integration use-cases, our API offers an easy-to-setup, scalable HTTPS Data Connector with an at-least-once guarantee.

There might be cases where polling the API periodically might be preferred instead, such as:

  • Integrating with legacy batch-based APIs, where it is too expensive calling the API with one event at a time. 
  • Integrating with an application behind a strict firewall, where opening the port needed for a Data Connector might not be possible.

This article will cover some best practices when polling our events API and a code example.

Summary of the Events API

All events for the last 30 days for each sensor is available at: https://api.disruptive-technologies.com/v2/projects/PROJECT-ID/devices/DEVICE-ID/events

Key parameters for polling the Events API are:

  • start_time and end_time. This is the span from which to fetch events. If these parameters are not set, it will default to fetching the last 24h.
  • event_types, a list of the type of events to fetch.

Also, from the API documentation:

Indexing delay
There is a delay from when an event is received by our servers until they are indexed and available via this endpoint. This is typically 1-2 seconds, but can be up to 10 seconds.

For full documentation, see the Events API documentation.

Polling best practices

Some best practices for polling the Events API are:

  1. Always use the start_time and end_time to retrieve events.

    Ever new poll interval, the call to the Event API should ensure that:

    • start_time is set to the previous end_time minus the worst-case indexing delay, 10 seconds.
    • end_time is set to the current time.

    By doing this you avoid:

    • Losing events between calls - which can happen if you would always set start_time to be e.g. 5min into the past instead of using the old end_time, or because of indexing delays.
    • Fetching more data than you need - which is often the default if you don’t specify a timespan.
  2. Respect the Retry_After header in case of a 429 - Too Many Requests.

  3. Only fetch the type of events that you are interested in by using event_types.

Code example

Below is a python code example which will:

  1. Fetch all sensors in a given project once.
  2. Poll the Events API of all sensors at a fixed interval and print a summary to the console.
  3. Supports pagination of both devices, for large projects, and for events, for long intervals.
  4. Respects the Retry-After header in case of a 429 - Too Many Requests response status code.

Prerequisites

To try out this code example you need to have Python 3 installed.

Create a new folder for this example open a command-line console in this folder, run the following 3 lines: 

python3 -m venv venv
source venv/bin/activate
pip3 install requests
  1. setup a standard Python 3 virtual environment (or venv),
  2. activate this environment and finally
  3. install the required packages via the built-in Python package manager.

Keep this console window open as we will use it to run the example in a moment.

The source

Using your favorite editor, create a new file called poll-event-api.py with the following content:

import time
import os
import requests
import json
import pprint
import datetime

MAX_INDEXING_DELAY_SECONDS=10

def requests_get_with_retry(url, **kwargs):
  max_retries = 10

  for _ in range(max_retries):
    response = requests.get(url, **kwargs)

    # Retry on status 429, Too Many Requests, after Retry-After seconds up to max_retries times
    if response.status_code == 429 and 'Retry-After' in response.headers:
      time.sleep(response.headers['Retry-After'])
    # Return immediately on any other status code
    else:
      break

  return response

def get_project_devices(project_id, auth):
  sensor_device_filter="device_types=touch&device_types=temperature&device_types=proximity"
  project_devices_url="https://api.disruptive-technologies.com/v2/projects/{}/devices?{}".format(project_id,sensor_device_filter)

  # Get all devices in project - with support for paganation
  devices = []
  nextPageToken = ''
  while True:
    raw_response = requests_get_with_retry(project_devices_url + "&page_token={}".format(nextPageToken), auth=auth)
    if raw_response.status_code != 200:
      print("Failed to access API with HTTP status code {}".format(raw_response.status_code))
      quit()

    response = raw_response.json()
    devices = devices + response["devices"]

    nextPageToken = response["nextPageToken"]
    if nextPageToken == '':
      break
  return devices

def get_device_events(device_name, start_time, end_time, auth):
  sensor_event_filter="event_types=touch&event_types=proximity&event_types=temperature&event_types=networkStatus"
  device_events_url = "https://api.disruptive-technologies.com/v2/{}/events?{}&start_time={}&end_time={}".format(device_name, sensor_event_filter, start_time, end_time)

  # Get all events from device - with support for paganation
  events = []
  nextPageToken = ''
  while True:
    raw_response = requests_get_with_retry(device_events_url + "&page_token={}".format(nextPageToken), auth=auth)
    if raw_response.status_code != 200:
      print("Failed to access API with HTTP status code {}".format(raw_response.status_code))
      quit()
    
    response = raw_response.json()
    events = events + response["events"]
    
    nextPageToken = response["nextPageToken"]
    if nextPageToken == '':
      break
  return events

def main():
  # Use Basic Auth. enabled Service Account Key ID and secret, and project ID, from environment
  username=os.environ.get('DT_SERVICE_ACCOUNT_KEY_ID')
  password=os.environ.get('DT_SERVICE_ACCOUNT_KEY_SECRET')
  project_id=os.environ.get('DT_SENSOR_PROJECT_ID')
  poll_interval_minutes=os.environ.get('DT_POLL_INTERVAL_MINUTES')
  
  # Start with an empty last-seen
  devices_last_seen = {}

  # Fetch all sensors in project, with paganation, once
  devices = get_project_devices(project_id, auth=(username, password))

  # Start with end_time being poll_interval_minutes back in time from now (UTC)
  end = datetime.datetime.utcnow() - datetime.timedelta(minutes=int(poll_interval_minutes))  

  while True:
    # Update start_time and end_time timestamps
    start = end - datetime.timedelta(seconds=MAX_INDEXING_DELAY_SECONDS)
    end = datetime.datetime.utcnow()
    start_time = "{:%Y-%m-%dT%H:%M:%SZ}".format(start)
    end_time = "{:%Y-%m-%dT%H:%M:%SZ}".format(end)

    print("Fetching all events between {} and {}...".format(start_time, end_time))

    # Get all events from all sensors between start_time and end_time
    for device in devices:
      events = get_device_events(device["name"], start_time, end_time, auth=(username, password))

      print("\t got {} events from {}".format(len(events),device['name']))

    print("Done! Waiting for {} minutes...".format(poll_interval_minutes))

    # Wait for poll_interval_minutes
    time.sleep(60*int(poll_interval_minutes))

if __name__ == "__main__":
  main()

Running the code

Before running the application, you need to set a few environmental variables:

DT_SERVICE_ACCOUNT_KEY_ID - The Key ID of the Service Account used to access the project

DT_SERVICE_ACCOUNT_KEY_SECRET - The Secret of Key ID of the Service Account used to access the project

DT_SENSOR_PROJECT_ID - The project to poll

DT_POLL_INTERVAL_MINUTES - At what interval to poll, in minutes

When running the code below, it will assume that these environment variables are set.

Run the code on the command line as follows:

python ./poll-event-api.py

You should see output similar to:

Fetching all events between 2019-03-22T14:50:51Z and 2019-03-22T14:55:51Z...
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b5rj9el7rihk942p49p0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6m6s4d7rihhhm5omis0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6m6s4l7rihhhm5omku0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6roh657rihmn9oji66g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6roh657rihmn9oji83g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b9ngr9l7rihg93n9puqg
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bchonod7rihjtvdmd40g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bchoqbl7rihkg46592b0
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bchoqbt7rihkg46594m0
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktcl7rihjbm0406vg
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktcl7rihjbm040760
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktd57rihjbm040bn0
	 got 12 events from projects/bccgtp8en7inbeit5fbg/devices/bdokte57rihjbm040pmg
	 got 11 events from projects/bccgtp8en7inbeit5fbg/devices/bdokted7rihjbm040q80
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdokted7rihjbm040s80
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdokted7rihjbm040sag
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktel7rihjbm040uj0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktel7rihjbm04103g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubccgvh5krle0009efg4g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubccgvjdkrle0009efg50
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubccgvktkrle0009efg5g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emube3a3aavpkgg00au2oog
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emube8odsegse4g0082e4vg
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubfrtbunlfkr000eoequ0
Done! Waiting for 5 minutes...