Announcing the CloudQuery BigQuery Destination Plugin

Name

Herman Schaaf

Twitter

hermanschaaf

Introduction

Hot on the heels of the Snowflake destination plugin, today we are introducing another CloudQuery destination plugin: BigQuery. BigQuery is a cloud-based data warehousing and analytics platform offered by Google Cloud that allows users to store and query large datasets quickly and efficiently. With the CloudQuery BigQuery destination plugin, you can now consolidate your organization's data from across cloud platforms into BigQuery.

Use Cases

Cloud Infrastructure data lake and data warehouse

CloudQuery brings all your cloud infrastructure configuration data to the same place, consolidating your security and infrastructure data and enabling new insights. If your organization already uses BigQuery for analysis, these in-house datasets can now also be combined with CloudQuery datasets to unlock insights that were previously impossible without several intermediate steps.

Historical data

Maintaining historical data is a common use case for data warehouses. By using BigQuery as a destination, you can now cost-effectively store all your cloud infrastructure data in a data warehouse for long term storage, analysis and investigation use-cases.

Scheduled Queries

By combining the CloudQuery data in BigQuery with scheduled queries (opens in a new tab), further business insights and security reports can be derived from the data as part of a completely serverless, cloud-based deployment.

Syncing data

Syncing data to BigQuery can be done as with any other destination plugin, so check out our Quickstart and BigQuery plugin documentation for instructions on how to get started.

Once data is synced you can start querying either with the native BigQuery interface, code or with any of the many BI tools that support BigQuery. Here is the "Hello World" of cloud infrastructure queries, a query that finds all AWS S3 buckets that are permitted to be public:

And yes, the BigQuery destination is not just for GCP data! As the example above demonstrates, data from all major cloud providers can now be synced to BigQuery, allowing a holistic view of your cloud assets.

Summary

This is the initial BigQuery destination plugin release. More improvements are coming, and I'd love to hear your use cases and ideas on how to leverage it. Feel free to open an issue on GitHub (opens in a new tab) or join our Discord (opens in a new tab) to share your ideas.