Integrating with Azure Event Hubs

In this guide, we will show how to use a Data Connector to integrate with Azure by forwarding events directly to Azure Event Hubs.

Before you begin

  1. You need to have at least Project Administrator level access to a project in Studio.
  2. You need a Microsoft Azure account with access to create Azure resources.

Setup a new Event Hubs namespace and instance

  1. Go to the Azure Portal.
  2. Create a new Event Hubs Namespace, by navigating to Create a resource and search for Event Hubs. Select Event Hubs from the suggestions, and click Create.create-event-hubs-resource.png
  3. In the Create Namespace view, set Name to something unique, and fill out the rest of the required fields to match your usage and location.

    Tip: If you are planning to use the Azure Event Hubs Capture feature, you’ll need to select the Standard pricing tier.
  4. Expand the notification (“bell”) icon in the top right, and wait until the deployment status says Deployment succeeded.

    Click on Deployment succeeded and then click on the Go to resource button.
  5. From the Event Hubs Namespace Overview, click the + Event Hub button to create a new Event Hubs instance.create-event-hubs-instance.png

    Set Name to device-data and adjust the partition count to the number of concurrent readers you expect to have.

Setup a Shared Access Policy

To allow the Data Connector to publish data to the Event Hubs instance we’ll create a new Shared Access Policy to manage access.

  1. From the Namespace overview, scroll down to Entities and navigate to Event Hubs, and select device-data from the list.event-hubs-instance-list.png
  2. Navigate to Shared access policies and click Add.create-shared-access-policy.png
  3. Set Policy name to dt-dataconnector, check the Send checkbox and click Create.create-shared-access-policy-dialog.png
  4. Select the newly created policy, and copy the Primary key. We’ll need the policy Name and Primary key in our next step.

Create a Shared Access Signature authentication token

To allow the Data Connector HTTPS requests to be authorized by Event Hubs, well need to create a SAS token that we will add as a custom HTTP header to the Data Connector.

Here is a Python (3.x) example of how to generate an authentication token. You can find examples for other programming languages and platforms in the Azure Event Hubs documentation.

import argparse
import urllib.parse
import hmac
import hashlib
import base64

from datetime import datetime, timedelta


def get_auth_token(namespace: str, instance_name: str, policy_name: str,
                policy_key: str, expire_at: datetime) -> str:
    uri = urllib.parse.quote_plus('https://{}.servicebus.windows.net/{}' \
                                .format(namespace, instance_name))
    key_value = policy_key.encode('utf-8')
    expiry = str(int(expire_at.timestamp()))
    string_to_sign = (uri + '\n' + expiry).encode('utf-8')
    signed_hmac_sha256 = hmac.HMAC(key_value, string_to_sign, hashlib.sha256)
    signature = urllib.parse.quote(base64.b64encode(
        signed_hmac_sha256.digest()))
    return 'SharedAccessSignature sr={}&sig={}&se={}&skn={}'.format(
        uri, signature, expiry, policy_name)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--namespace',
                        required=True,
                        help='Event Hubs namespace name')
    parser.add_argument('--instance',
                        required=True,
                        help='Event Hubs instance name')
    parser.add_argument('--policy-name',
                        required=True,
                        help='Shared access policy name')
    parser.add_argument('--policy-key',
                        required=True,
                        help='Shared access policy key')
    parser.add_argument('--days-valid',
                        required=True,
                        type=int,
                        help='Number of days until the token should expire')
    args = parser.parse_args()

    expire_at = datetime.now() + timedelta(days=args.days_valid)
    token = get_auth_token(args.namespace, args.instance, args.policy_name,
                        args.policy_key, expire_at)
    print(token)

This example can also be used from the command line to generate an authentication token:

  1. Save the example as sas_token.py.

  2. Run the following command to generate an authentication token.

     python3 sas_token.py --namespace $EVENTHUBS_NAMESPACE_NAME \
                         --instance device-data \
                         --policy-name dt-dataconnector \
                         --policy-key $POLICY_PRIMARY_KEY \
                         --days-valid 60
    

    Replace the parameter $EVENTHUBS_NAMESPACE_NAME with the name of your Event Hubs namespace, and $POLICY_PRIMARY_KEY with Primary key value from the dt-dataconnector Shared Access Policy created in the previous section.

    The authentication token generated from this example command will stop working 60 days from when it was created. You can adjust this with the --days-valid parameter, and set it to be longer or shorter depending on your needs.

    Tip: By using our REST API, you can programmatically refresh the authentication token on the Data Connector by running e.g a time-triggered Azure Function.

Create a Data Connector

  1. Go to Studio.

  2. Navigate to the project you want to integrate with.

  3. Open the Data Connector view for the project via the menu on the left.

  4. Click Create new.

  5. Set the Name to “Azure Event Hubs” and set the Endpoint URL to https://<yournamespace>.servicebus.windows.net/device-data/messages (replace <yournamespace> with the name of your Azure Event Hubs namespace).

    create-data-connector.png

  6. Scroll down to the Custom HTTP request headers section and set the header Name to Authorization. For the header Value, add the authentication token generated from the python script in the previous section.

    add-authorization-header.png

  7. Click the Save New Data Connector button on the bottom to complete the process.

Test the integration

To confirm that integration is up and running, we will look at the Data Connector metrics in Studio.

Before proceeding, make sure your sensors have sent some data by pressing them a few times.

Navigate back to the Data Connector that you created previously in DT Studio.

At the top of the page, you will find how many times your new Data Connector has been run. <data-connector-metrics.png

The Success count shows the number of times Azure Event Hubs has been called and returned HTTP status 200 OK the last 24h.

The Error count shows how many times it has returned an error code or timed out.

Next steps

See: