AWS EventBridge Pattern DSL

EventBridge had been lately updated with set of new pattern matching capabilities that had been nicely captured in the blog post by James Beswick. The existing functionality of matching the event based on exact equality of certain fields has now been expanded to support operations like prefix matches, not matches or matching base on a presence of certain attributes. This altogether helps to pre filter the events based on specific criteria, that previously was not only achievable by adding the needed boilerplate logic to your events consumer and dropping all events that are not meeting certain criteria.

Even before the pattern matching syntax had expanded with additional keywords and operators it was not uncommon to mis configure your event pattern by specifying wrong attribute name or pattern syntax, no matter whether use with AWS CLI, CloudFormation or through AWS Console.

To avoid this scenario and first of all make the creation of the Rule pattern more bulletproof experience as well as allow it to be fully reproducible and testable I had create a very small utility Java library that introduce DSL for EventBridge pattern language.

You can add the library to your existing project by simply dropping it to your Maven (or Gradle) project.

<dependency>
  <groupId>io.jmnarloch</groupId>
  <artifactId>aws-eventbridge-pattern-builder</artifactId>
  <version>1.0.0</version>
</dependency>

The utility is really simple to use. It defines a `EventsPattern` with flow API for defining the constraints of the matched events.

As an example a pattern that matches an event published by aws.ec2 that would be expressed in JSON like:

{
  "source": [ "aws.ec2" ]
}

Can be turn into small Java snippet.

EventsPattern.builder()
        .equal("source", "aws.ec2")
        .build();

Once a pattern had been created it can be turn into JSON simply by calling `toJson` method.

The newly supported operators are also supported.

Prefix matching
For instance if we wish to match events from specifying AWS regions we can create a pattern.

EventsPattern.builder()
        .prefix("region", "eu-")
        .build();

Will corespond to JSON syntax

{
  "region": [{"prefix": "eu-"}]
}

Anything but
Can be used to match everything except certain value.

EventsPattern.builder()
        .path("detail")
                .anythingBut("state", "initializing")
                .parent()
        .build();

Will result in.

{
  "detail": {
    "state": [ {"anything-but":"initializing"} ]
  }
}

AWS SDK

The above examples can be used as drop replacement whenever using the vanilla AWS SDK.

EventsPattern pattern = EventsPattern.builder()
        .equal("source", "aws.ec2")
        .build();

AmazonCloudWatchEvents eventsClient = AmazonCloudWatchEventsClient.builder().build();
eventsClient.putRule(new PutRuleRequest()
        .withName("EC2-Rule")
        .withRoleArn("arn:aws:iam::123456789012:role/ec2-events-delivery-role")
        .withEventPattern(pattern.toJson()));

Testability

The biggest benefit for moving the rule to the code would be arguable the possibility to test the rule even before it’s deployed. For that purpose a dedicated test could be added that would be part of the integration test suite.

    @Test
    public void patternShouldMatchEvent() {

        String event = "{\"id\":\"7bf73129-1428-4cd3-a780-95db273d1602\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.ec2\",\"account\":\"123456789012\",\"time\":\"2015-11-11T21:29:54Z\",\"region\":\"us-east-1\",\"resources\":[  \"arn:aws:ec2:us-east-1:123456789012:instance/i-abcd1111\"  ],\"detail\":{  \"instance-id\":\"i-abcd1111\",  \"state\":\"pending\"  }}";

        EventsPattern pattern = EventsPattern.builder()
                .equal("source", "aws.ec2")
                .build();

        AmazonCloudWatchEvents eventsClient = AmazonCloudWatchEventsClient.builder().build();
        TestEventPatternResult matchResult = eventsClient.testEventPattern(new TestEventPatternRequest()
                .withEvent(event)
                .withEventPattern(pattern.toJson()));
        assertTrue(matchResult.getResult());
    }

Now each time the event structure itself changes there is regression test in place that can guarantee that the created pattern will still be matching the event. Useful during early development stages or for prototyping purposes.

AWS CDK

AWS CDK is a development framework that allows you to define your AWS infrastructure as a code. At the moment the CDK does not yet support the full syntax of the pattern matching, but the ultimate goal would be to introduce such capacity in the CDK directly.

The project is open source and available on Github repo.

For additional information on the event matching capabilities please refer to the EventBridge documentation.

Automatically discovering SNS message structure

Last month during Re:Invent preview of EventBridge Schema Registry had been announced. One of unique feature that the service brings is the automatic discovery of any custom event publish to EventBridge. As soon as the discovery will be enabled on Event Bus with the service will aggregate the event over period of time and register them as OpenAPI document into the registry. You might ask yourself what is exactly the use case in which you will need a feature like that? Typically the discovery of event will be particularly useful whenever you don’t own the source that publishes the events typically when it’s owned by other team in you organization or even a third party. Even for consuming the AWS EventBridge events prior to the release of the service typical onboarding process required fallowing the contract of the events defined through the AWS documentation and mapping that into language of choice. That could be really tedious.

As for today the service allows integration with EventBridge, though it is definitely not limited to it. In matter of fact it will be able to easy integrate with any JSON base event source. To demonstrate that capabilities with little amount of code we can demonstrate how to automate the discovery of schema from any SNS topic.

To achieve that a little bit of wiring needs to be done to pipe the events from SNS to EventBridge. Unfortunately EventBridge is not a natively supported target for SNS at the moment. Instead we can use AWS Lambda to consume the event from SNS and forward it to EventBridge.

This leads us to very straight forward integration.

AWS To wire everything together we can create a SAM template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  aws-schema-discovery-sns

  Sample SAM Template for aws-schema-discovery-sns

Globals:
  Function:
    Timeout: 60

Resources:
  SnsDiscoveryTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: 'sns-discovery'

  SnsSchemaDiscoveryFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: app/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Environment:
        Variables:
          event_source: sns.topic
          event_detail_type: !GetAtt SnsDiscoveryTopic.TopicName
      Events:
        SnsEvents:
          Type: SNS
          Properties:
            Topic: !Ref SnsDiscoveryTopic
      Policies:
        - Statement:
            - Sid: EventBridgePutEventsPolicy
              Effect: Allow
              Action:
                - events:PutEvents
              Resource: '*'
Outputs:
  SnsSchemaDiscoveryFunction:
    Description: 'SNS Schema Discovery function Lambda Function ARN'
    Value: !GetAtt SnsSchemaDiscoveryFunction.Arn
  SnsSchemaDiscoveryFunctionIamRole:
    Description: 'Implicit IAM Role created for SNS Schema Discovery function'
    Value: !GetAtt SnsSchemaDiscoveryFunctionRole.Arn

This will deploy through CloudFormation a Lambda function that will be subscribed to dedicated SNS topic. Publishing any message to the SNS will forward it to EventBridge and providing that the schema discovery had been enabled on the default Event Bus auto discovered schema will be registered.

Next thing is to develop the Lambda code. To keep things since let’s use JavaScript for that.

const AWS = require('aws-sdk');
exports.lambdaHandler = async (event, context) => {
    try {
        var eventbridge = new AWS.EventBridge();
        var entries = [];
        event.Records.forEach(record => entries.push({
            Source: process.env.event_source,
            DetailType: process.env.event_detail_type,
            Detail: record.Sns.Message,
            Time: new Date()
        }));
        return eventbridge.putEvents({
            Entries: entries
        }).promise();
    } catch (err) {
        console.log(err, err.stack);
        throw err;
    }
};

The code is very simple. It will deliver any event published to SNS back to EventBridge. It uses the configured source and detail-type attributes to identify this particular event.

The last two things is to enable the discovery on the EventBus. This can be done from Console, CloudFormation or using CLI command.

aws schemas create-discoverer --source-arn arn:aws:events:us-east-1:${ACCOUNT_ID}:event-bus/default

Now by publishing an event to SNS topic we will get automatic registered schema.

To demonstrate that everything is working let’s publish a sample JSON document into the SNS topic.

{
  "Records": [{
    "AwsRegion": "us-east-1",
    "AwsAccountId": "12345678910",
    "MessageId": "4e4fac8e-cf3a-4de3-b33e-e614fd25c66f",
    "Message": {
      "instance-id":"i-abcd1111",
      "state":"pending"
    }
  }]
}

The above event results in fallowing OpenAPI document created in the registry:

{
  "openapi": "3.0.0",
  "info": {
    "version": "1.0.0",
    "title": "SnsDiscovery"
  },
  "paths": {},
  "components": {
    "schemas": {
      "AWSEvent": {
        "type": "object",
        "required": ["detail-type", "resources", "detail", "id", "source", "time", "region", "version", "account"],
        "x-amazon-events-detail-type": "sns-discovery",
        "x-amazon-events-source": "sns.topic",
        "properties": {
          "detail": {
            "$ref": "#/components/schemas/SnsDiscovery"
          },
          "account": {
            "type": "string"
          },
          "detail-type": {
            "type": "string"
          },
          "id": {
            "type": "string"
          },
          "region": {
            "type": "string"
          },
          "resources": {
            "type": "array",
            "items": {
              "type": "object"
            }
          },
          "source": {
            "type": "string"
          },
          "time": {
            "type": "string",
            "format": "date-time"
          },
          "version": {
            "type": "string"
          }
        }
      },
      "SnsDiscovery": {
        "type": "object",
        "required": ["Records"],
        "properties": {
          "Records": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/SnsDiscoveryItem"
            }
          }
        }
      },
      "SnsDiscoveryItem": {
        "type": "object",
        "required": ["AwsRegion", "Message", "AwsAccountId", "MessageId"],
        "properties": {
          "Message": {
            "$ref": "#/components/schemas/Message"
          },
          "AwsAccountId": {
            "type": "string"
          },
          "AwsRegion": {
            "type": "string"
          },
          "MessageId": {
            "type": "string"
          }
        }
      },
      "Message": {
        "type": "object",
        "required": ["instance-id", "state"],
        "properties": {
          "instance-id": {
            "type": "string"
          },
          "state": {
            "type": "string"
          }
        }
      }
    }
  }
}

The source code of the SAM application is available at Github repo. Feel free to try it out.