Automatically discovering SNS message structure

Last month during Re:Invent preview of EventBridge Schema Registry had been announced. One of unique feature that the service brings is the automatic discovery of any custom event publish to EventBridge. As soon as the discovery will be enabled on Event Bus with the service will aggregate the event over period of time and register them as OpenAPI document into the registry. You might ask yourself what is exactly the use case in which you will need a feature like that? Typically the discovery of event will be particularly useful whenever you don’t own the source that publishes the events typically when it’s owned by other team in you organization or even a third party. Even for consuming the AWS EventBridge events prior to the release of the service typical onboarding process required fallowing the contract of the events defined through the AWS documentation and mapping that into language of choice. That could be really tedious.

As for today the service allows integration with EventBridge, though it is definitely not limited to it. In matter of fact it will be able to easy integrate with any JSON base event source. To demonstrate that capabilities with little amount of code we can demonstrate how to automate the discovery of schema from any SNS topic.

To achieve that a little bit of wiring needs to be done to pipe the events from SNS to EventBridge. Unfortunately EventBridge is not a natively supported target for SNS at the moment. Instead we can use AWS Lambda to consume the event from SNS and forward it to EventBridge.

This leads us to very straight forward integration.

AWS To wire everything together we can create a SAM template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  aws-schema-discovery-sns

  Sample SAM Template for aws-schema-discovery-sns

Globals:
  Function:
    Timeout: 60

Resources:
  SnsDiscoveryTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: 'sns-discovery'

  SnsSchemaDiscoveryFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: app/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Environment:
        Variables:
          event_source: sns.topic
          event_detail_type: !GetAtt SnsDiscoveryTopic.TopicName
      Events:
        SnsEvents:
          Type: SNS
          Properties:
            Topic: !Ref SnsDiscoveryTopic
      Policies:
        - Statement:
            - Sid: EventBridgePutEventsPolicy
              Effect: Allow
              Action:
                - events:PutEvents
              Resource: '*'
Outputs:
  SnsSchemaDiscoveryFunction:
    Description: 'SNS Schema Discovery function Lambda Function ARN'
    Value: !GetAtt SnsSchemaDiscoveryFunction.Arn
  SnsSchemaDiscoveryFunctionIamRole:
    Description: 'Implicit IAM Role created for SNS Schema Discovery function'
    Value: !GetAtt SnsSchemaDiscoveryFunctionRole.Arn

This will deploy through CloudFormation a Lambda function that will be subscribed to dedicated SNS topic. Publishing any message to the SNS will forward it to EventBridge and providing that the schema discovery had been enabled on the default Event Bus auto discovered schema will be registered.

Next thing is to develop the Lambda code. To keep things since let’s use JavaScript for that.

const AWS = require('aws-sdk');
exports.lambdaHandler = async (event, context) => {
    try {
        var eventbridge = new AWS.EventBridge();
        var entries = [];
        event.Records.forEach(record => entries.push({
            Source: process.env.event_source,
            DetailType: process.env.event_detail_type,
            Detail: record.Sns.Message,
            Time: new Date()
        }));
        return eventbridge.putEvents({
            Entries: entries
        }).promise();
    } catch (err) {
        console.log(err, err.stack);
        throw err;
    }
};

The code is very simple. It will deliver any event published to SNS back to EventBridge. It uses the configured source and detail-type attributes to identify this particular event.

The last two things is to enable the discovery on the EventBus. This can be done from Console, CloudFormation or using CLI command.

aws schemas create-discoverer --source-arn arn:aws:events:us-east-1:${ACCOUNT_ID}:event-bus/default

Now by publishing an event to SNS topic we will get automatic registered schema.

To demonstrate that everything is working let’s publish a sample JSON document into the SNS topic.

{
  "Records": [{
    "AwsRegion": "us-east-1",
    "AwsAccountId": "12345678910",
    "MessageId": "4e4fac8e-cf3a-4de3-b33e-e614fd25c66f",
    "Message": {
      "instance-id":"i-abcd1111",
      "state":"pending"
    }
  }]
}

The above event results in fallowing OpenAPI document created in the registry:

{
  "openapi": "3.0.0",
  "info": {
    "version": "1.0.0",
    "title": "SnsDiscovery"
  },
  "paths": {},
  "components": {
    "schemas": {
      "AWSEvent": {
        "type": "object",
        "required": ["detail-type", "resources", "detail", "id", "source", "time", "region", "version", "account"],
        "x-amazon-events-detail-type": "sns-discovery",
        "x-amazon-events-source": "sns.topic",
        "properties": {
          "detail": {
            "$ref": "#/components/schemas/SnsDiscovery"
          },
          "account": {
            "type": "string"
          },
          "detail-type": {
            "type": "string"
          },
          "id": {
            "type": "string"
          },
          "region": {
            "type": "string"
          },
          "resources": {
            "type": "array",
            "items": {
              "type": "object"
            }
          },
          "source": {
            "type": "string"
          },
          "time": {
            "type": "string",
            "format": "date-time"
          },
          "version": {
            "type": "string"
          }
        }
      },
      "SnsDiscovery": {
        "type": "object",
        "required": ["Records"],
        "properties": {
          "Records": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/SnsDiscoveryItem"
            }
          }
        }
      },
      "SnsDiscoveryItem": {
        "type": "object",
        "required": ["AwsRegion", "Message", "AwsAccountId", "MessageId"],
        "properties": {
          "Message": {
            "$ref": "#/components/schemas/Message"
          },
          "AwsAccountId": {
            "type": "string"
          },
          "AwsRegion": {
            "type": "string"
          },
          "MessageId": {
            "type": "string"
          }
        }
      },
      "Message": {
        "type": "object",
        "required": ["instance-id", "state"],
        "properties": {
          "instance-id": {
            "type": "string"
          },
          "state": {
            "type": "string"
          }
        }
      }
    }
  }
}

The source code of the SAM application is available at Github repo. Feel free to try it out.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s