ClickHouse destination plugin

Latest: v3.4.0

This destination plugin lets you sync data from a CloudQuery source to ClickHouse (opens in a new tab) database.

It supports append write mode only. Write mode selection is required through write_mode.

Supported database versions: >= 22.1.2

Configuration

Example

kind: destination
spec:
  name: "clickhouse"
  registry: "github"
  path: "cloudquery/clickhouse"
  version: "v3.4.0"
  write_mode: "append"
  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
    # Optional parameters
    # cluster: ""
    # ca_cert: ""
    # engine:
    #   name: MergeTree
    #   parameters: []
    #
    # batch_size: 10000
    # batch_size_bytes: 5242880 # 5 MiB
    # batch_timeout: 20s

This example configures a ClickHouse instance, located at localhost:9000. It expects CH_USER, CH_PASSWORD and CH_DATABASE environment variables to be set. The (top level) spec section is described in the Destination Spec Reference.

Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly.

ClickHouse spec

This is the (nested) spec used by the ClickHouse destination plugin.

connection_string (string) (required)

Connection string to connect to the database. See SDK documentation (opens in a new tab) for more details.
cluster (string) (optional) (default: not used)

Cluster name to be used for distributed DDL (opens in a new tab). If the value is empty, DDL operations will affect only the server the plugin is connected to.
ca_cert (string) (optional) (default: not used)

PEM-encoded certificate authorities. When set, a certificate pool will be created by appending the certificates to the system pool.

See file variable substitution for how to read this value from a file.
engine (optional, table engine settings. Default: MergeTree engine)

Engine to be used for tables. Only *MergeTree family (opens in a new tab) is supported at the moment.
batch_size (integer) (optional) (default: 10000)

This parameter controls the maximum amount of items may be grouped together to be written as a single write.
batch_size_bytes (integer) (optional) (default: 5242880 (= 5 MiB))

This parameter controls the maximum size of items that may be grouped together to be written as a single write.
batch_timeout (duration) (optional) (default: 20s)

This parameter controls the maximum interval between batch writes.

ClickHouse table engine

This option allows to specify a custom table engine to be used.

name (string) (required)

Name of the table engine. Only *MergeTree family (opens in a new tab) is supported at the moment.
parameters (array of parameters) (optional) (default: empty)

Engine parameters. Currently, no restrictions are imposed on the parameter types.

kind: destination
spec:
  name:       "clickhouse"
  registry:   "github"
  path:       "cloudquery/clickhouse"
  version:    "v3.4.0"
  write_mode: "append"

  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
    engine:
      name: ReplicatedMergeTree
      parameters:
      - "/clickhouse/tables/{shard}/{database}/{table}"
      - "{replica}"

Verbose logging for debug

The ClickHouse destination can be run in debug mode. To achieve this pass the debug=true option to connection_string. See SDK documentation (opens in a new tab) for more details.

Note: This will use SDK (opens in a new tab) built-in logging and might output data and sensitive information to logs. Make sure not to use it in production environment.

kind: destination
spec:
  name:       "clickhouse"
  registry:   "github"
  path:       "cloudquery/clickhouse"
  version:    "v3.4.0"
  write_mode: "append"

  spec:
    connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"

Types Overview