Documentation
Plugins
Sources
AWS
Event-Based Sync

This feature is currently in a closed preview. Sign up for access using this form (opens in a new tab)

Event-Based Sync

AWS CloudTrail enables users to get an audit log of events occurring within their account. By subscribing to a stream of AWS CloudTrail events in a Kinesis Data stream CloudQuery can trigger selective syncs to update just the singular resource that had a configuration change.

Each table in the supported list is a top level table. When an event is received for a table, all child tables are re-synced too by default. To skip some child tables you can use skip_tables

Supported Services and Events

ServiceEventCloudQuery Tables
ec2.amazonaws.comAssociateRouteTableaws_ec2_route_tables
ec2.amazonaws.comAttachInternetGatewayaws_ec2_internet_gateways
ec2.amazonaws.comAuthorizeSecurityGroupEgressaws_ec2_security_groups
ec2.amazonaws.comAuthorizeSecurityGroupIngressaws_ec2_security_groups
ec2.amazonaws.comCreateImageaws_ec2_images
ec2.amazonaws.comCreateInternetGatewayaws_ec2_internet_gateways
ec2.amazonaws.comCreateNetworkInterfaceaws_ec2_network_interfaces
ec2.amazonaws.comCreateSecurityGroupaws_ec2_security_groups
ec2.amazonaws.comCreateSubnetaws_ec2_subnets
ec2.amazonaws.comCreateTagsaws_ec2_instances
ec2.amazonaws.comCreateVpcaws_ec2_vpcs
ec2.amazonaws.comDeleteTagsaws_ec2_instances
ec2.amazonaws.comDetachInternetGatewayaws_ec2_internet_gateways
ec2.amazonaws.comModifySubnetAttributeaws_ec2_subnets
ec2.amazonaws.comRevokeSecurityGroupEgressaws_ec2_security_groups
ec2.amazonaws.comRevokeSecurityGroupIngressaws_ec2_security_groups
ec2.amazonaws.comRunInstancesaws_ec2_instances
iam.amazonaws.comCreateGroupaws_iam_groups
iam.amazonaws.comCreateGroupaws_iam_groups
iam.amazonaws.comDeleteGroupaws_iam_groups
iam.amazonaws.comUpdateGroupaws_iam_groups
iam.amazonaws.comCreateRoleaws_iam_roles
iam.amazonaws.comDeleteRoleaws_iam_roles
iam.amazonaws.comTagRoleaws_iam_roles
iam.amazonaws.comUntagRoleaws_iam_roles
iam.amazonaws.comUpdateRoleaws_iam_roles
iam.amazonaws.comUpdateRoleDescriptionaws_iam_roles
iam.amazonaws.comCreateUseraws_iam_users
iam.amazonaws.comDeleteUseraws_iam_users
iam.amazonaws.comTagUseraws_iam_users
iam.amazonaws.comUntagUseraws_iam_users
iam.amazonaws.comUpdateUseraws_iam_users
rds.amazonaws.comCreateDBClusteraws_rds_clusters
rds.amazonaws.comCreateDBInstanceaws_rds_instances
rds.amazonaws.comModifyDBClusteraws_rds_clusters
rds.amazonaws.comModifyDBInstanceaws_rds_instances

Configuration

  1. Configure an AWS CloudTrail Trail to send management events to a Kinesis Data Stream via CloudWatch Logs. The most straight-forward way to do this is to use the CloudFormation template provided by CloudQuery.

The CloudFormation template will deploy the following architecture:

Event based syncing cloud infrastructure

aws cloudformation deploy --template-file ./streaming-deployment.yml --stack-name <STACK-NAME> --capabilities CAPABILITY_IAM --disable-rollback --region <DESIRED-REGION>
  1. Copy the ARN of the Kinesis stream. If you used the CloudFormation template you can run the following command:
aws cloudformation describe-stacks --stack-name <STACK-NAME> --query "Stacks[].Outputs" --region <DESIRED-REGION>
  1. Define a config.yml file like the one below
kind: source
spec:
  name: "aws-event-based"
  registry: "local"
  path: <PATH/TO/BINARY>
  tables:
    - aws_ec2_instances
    - aws_ec2_internet_gateways
    - aws_ec2_security_groups
    - aws_ec2_subnets
    - aws_ec2_vpcs
    - aws_ecs_cluster_tasks
    - aws_iam_groups
    - aws_iam_roles
    - aws_iam_users
    - aws_rds_instances
  destinations: ["postgresql"]
  spec:
    event_based_sync:
      # account:
      #  local_profile: "<ROLE-NAME>"
      kinesis_stream_arn: <OUTPUT-FROM-CLOUDFORMATION-STACK>
  1. Sync the data!
cloudquery sync config.yml

This will start a long lived process that will only stop when there is an error or you stop the process

Limitations

  • Kinesis Stream can only have a single shard. (This is a limitation that we expect to remove in the future)
  • Stale records will only be deleted if the plugin stops consuming the Kinesis Stream, which only can occur if there is an error
TablesPolicies