Shopify Source Plugin

Latest: v3.1.4

The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication

In order to fetch information from Shopify, cloudquery needs to be authenticated. Either an API key and password (in the case of basic custom/private apps) or an access token (for OAuth apps) is required for authentication.

Refer to the Shopify Help Center article on Custom apps (opens in a new tab) and create a custom app. Follow Get the API credentials for a custom app section to get the credentials for Admin API and put them in your plugin configuration as api_key and api_secret.

If you have a large or busy store, API key/secret type credentials might not be enough due to the heavy rate limiting. In this case, you can use OAuth in your custom app to get an access token which allow many more requests a second. To use that token in your plugin configuration instead, just set it in access_token and remove api_key and api_secret sections. For more information, refer to Shopify.dev (opens in a new tab) on the subject.

Incremental Syncing

The Shopify plugin supports incremental syncing. This means that only new data will be fetched from Shopify and loaded into your destination for supported tables (support depending on API endpoint). This is done by keeping track of the last item fetched and only fetching data that has been created since then. To enable this, backend option must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.

Example

This example syncs from Shopify to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference. Incremental syncing is enabled and will be saved to a .cq/state/ directory by default.

kind: source
# Common source-plugin configuration
spec:
  name: shopify
  path: cloudquery/shopify
  version: "v3.1.4"
  tables: ["*"]
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_shopify"
    connection: "@@plugins.postgresql.connection"
  # Shopify specific configuration
  spec:
    api_key: "<YOUR_API_KEY_HERE>"
    api_secret: "<YOUR_API_SECRET_HERE>"
    shop_url: "https://<YOUR_SHOP>.myshopify.com"
#    concurrency: 1000

Note that if backend_options is omitted, by default no backend will be used. This will result in all items being fetched on every sync.

For more information about managing state for incremental tables, see Managing Incremental Tables.

Configuration Reference

This is the (nested) spec used by the Shopify source plugin:

api_key (string) (required if access_token isn't used):

The API Key for your custom app in your store.
api_secret (string) (required if access_token isn't used):

The API Secret for your custom app in your store.
access_token (string) (required if api_key & api_secret aren't used):

An access token for your Shopify custom app. This is an alternative way of authenticating, use either this or the ones above.
shop_url (string) (required):

The URL of your Shopify store. Must start with https:// and end with .myshopify.com.
api_version (string) (optional) (default: 2023-01):

The Shopify Admin API version to use. See here (opens in a new tab) for more information.
timeout_secs (integer) (optional) (default: 10):

Timeout (in seconds) for requests against the Shopify Admin API.
max_retries (integer) (optional) (default: 30):

Number of retries if a request was rate limited.
page_size (integer) (optional) (default: 50):

Maximum number of items queried each request. Find an optimum value to balance amount of data fetched and requests timing out. Maximum value 250.
concurrency (integer) (optional) (default: 1000):

Maximum number of concurrent requests to the Shopify Admin API.

Query Examples

Get all your active products with a specific tag

SELECT * FROM shopify_products WHERE status='active' AND 'your-tag' = ANY(tags);

Tables Tables