> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-trino-dialect.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Seamlessly connect your external data sources to ClickHouse Cloud.

# Integrating with ClickHouse Cloud

export const Image = ({img, alt, size = "lg", background}) => {
  const normalizedSize = ["sm", "md", "lg"].includes(size) ? size : "lg";
  const backgroundColor = background === "white" ? "white" : background === "black" ? "rgb(31 31 28)" : undefined;
  return <div className={`ch-image-${normalizedSize}`}>
      <Frame>
        <img src={img} alt={alt} style={{
    backgroundColor
  }} />
      </Frame>
    </div>;
};

<h2 id="overview">
  Overview
</h2>

[ClickPipes](/integrations/clickpipes/home) is a managed integration platform that makes ingesting data from a diverse set of sources as simple as clicking a few buttons. Designed for the most demanding workloads, ClickPipes's robust and scalable architecture ensures consistent performance and reliability. ClickPipes can be used for continuous data ingestion or one-time data loads.

ClickPipes can be deployed and managed using the ClickPipes UI, as well as programmatically using [OpenAPI](/integrations/clickpipes/programmatic-access/openapi) and [Terraform](/integrations/clickpipes/programmatic-access/terraform).

<h2 id="clickpipes-connectors">
  Connectors
</h2>

<h3 id="clickpipes-for-streaming">
  ClickPipes for Streaming
</h3>

| Connector                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Feature lifecycle                             | Data sources                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/SwBmQiR_ocl5w4KS/images/integrations/logos/kafka.svg?fit=max&auto=format&n=SwBmQiR_ocl5w4KS&q=85&s=0f8a4d7f58e46d6bde9775449d67082d" alt="Apache Kafka logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/kafka.svg" /> [Kafka](/integrations/clickpipes/kafka)                                                                                                                            | <Badge color="green">GA</Badge>               | Apache Kafka, [Confluent Cloud](/resources/support-center/knowledge-base/cloud-services/confluent-cloud-private-connectivity-for-clickpipes), Redpanda, [AWS MSK](/resources/support-center/knowledge-base/cloud-services/aws-privatelink-setup-for-msk-clickpipes), Google Cloud Managed Service for Apache Kafka, [Azure Event Hubs](/integrations/clickpipes/kafka/faq#azure-eventhubs), [WarpStream](/integrations/clickpipes/kafka/best-practices#warpstream-settings), and other Kafka API-compatible data sources. |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/qOv7KJkvDWsDYhyE/images/integrations/logos/amazon_kinesis_logo.svg?fit=max&auto=format&n=qOv7KJkvDWsDYhyE&q=85&s=cb473bee9b1a0d568a7061ae470c3209" alt="Amazon Kinesis logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/amazon_kinesis_logo.svg" /> [Amazon Kinesis](/integrations/clickpipes/kinesis) | <Badge color="green">GA</Badge>               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/SwBmQiR_ocl5w4KS/images/integrations/logos/google_pubsub.svg?fit=max&auto=format&n=SwBmQiR_ocl5w4KS&q=85&s=c8b19c8117df6d541c18c82dd9188303" alt="Google Cloud Pub/Sub logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/google_pubsub.svg" /> [GCP Pub/Sub](/integrations/clickpipes/pubsub)                                           | <Badge color="purple">Private Preview</Badge> |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |

<h3 id="clickpipes-for-object-storage">
  ClickPipes for Object Storage
</h3>

| Connector                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Feature lifecycle               | Data sources                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/qOv7KJkvDWsDYhyE/images/integrations/logos/amazon_s3_logo.svg?fit=max&auto=format&n=qOv7KJkvDWsDYhyE&q=85&s=4d99f78a5f77afa0e86bf1ba624a5ad6" alt="Amazon S3 logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/amazon_s3_logo.svg" /> [Amazon S3](/integrations/clickpipes/object-storage/s3/overview)                                  | <Badge color="green">GA</Badge> | [Amazon S3](/integrations/clickpipes/object-storage/s3/get-started), Cloudflare R2, DigitalOcean Spaces, OVH Object Storage, and other S3 API-compatible data sources. |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/SwBmQiR_ocl5w4KS/images/integrations/logos/gcs.svg?fit=max&auto=format&n=SwBmQiR_ocl5w4KS&q=85&s=0165b9e2fc94c2c480390e3289164985" alt="Google Cloud Storage logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/gcs.svg" /> [Google Cloud Storage](/integrations/clickpipes/object-storage/gcs/overview)                                                                                                   | <Badge color="green">GA</Badge> |                                                                                                                                                                        |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/qOv7KJkvDWsDYhyE/images/integrations/logos/azureblobstorage.svg?fit=max&auto=format&n=qOv7KJkvDWsDYhyE&q=85&s=0d66eaa78cb9fc6d7a1d430b60b7b9ff" alt="Azure Blob Storage logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/azureblobstorage.svg" /> [Azure Blob Storage](/integrations/clickpipes/object-storage/abs/overview) | <Badge color="green">GA</Badge> |                                                                                                                                                                        |

<h3 id="clickpipes-for-cdc">
  ClickPipes for CDC
</h3>

| Connector                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Feature lifecycle                             | Data sources                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/SwBmQiR_ocl5w4KS/images/integrations/logos/postgresql.svg?fit=max&auto=format&n=SwBmQiR_ocl5w4KS&q=85&s=576bc6bae6d9d3fa5b450e67f83f1414" alt="Postgres logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/postgresql.svg" /> [Postgres](/integrations/clickpipes/postgres) | <Badge color="green">GA</Badge>               | [Amazon RDS](/integrations/clickpipes/postgres/source/rds), [Amazon Aurora](/integrations/clickpipes/postgres/source/aurora), [Google Cloud SQL](/integrations/clickpipes/postgres/source/google-cloudsql), [AlloyDB](/integrations/clickpipes/postgres/source/alloydb), [Azure Flexible Server](/integrations/clickpipes/postgres/source/azure-flexible-server-postgres), [Supabase](/integrations/clickpipes/postgres/source/supabase), [Neon](/integrations/clickpipes/postgres/source/neon-postgres), [Crunchy Bridge](/integrations/clickpipes/postgres/source/crunchy-postgres), [TimescaleDB](/integrations/clickpipes/postgres/source/timescale), [PlanetScale](/integrations/clickpipes/postgres/source/planetscale), [self-managed](/integrations/clickpipes/postgres/source/generic), and other compatible data sources. |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/SwBmQiR_ocl5w4KS/images/integrations/logos/mysql.svg?fit=max&auto=format&n=SwBmQiR_ocl5w4KS&q=85&s=c4fd0664855241bca1ace561dcbc0598" alt="MySQL logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/mysql.svg" /> [MySQL](/integrations/clickpipes/mysql)                                                  | <Badge color="yellow">Public Beta</Badge>     | [Amazon RDS](/integrations/clickpipes/mysql/source/rds), [Amazon Aurora](/integrations/clickpipes/mysql/source/aurora), [Cloud SQL](/integrations/clickpipes/mysql/source/gcp), [Azure Flexible Server](/integrations/clickpipes/mysql/source/azure-flexible-server-mysql), [self-managed MySQL](/integrations/clickpipes/mysql/source/generic), [Amazon RDS for MariaDB](/integrations/clickpipes/mysql/source/rds_maria), [self-managed MariaDB](/integrations/clickpipes/mysql/source/generic_maria), and other compatible data sources.                                                                                                                                                                                                                                                                                         |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/SwBmQiR_ocl5w4KS/images/integrations/logos/mongodb.svg?fit=max&auto=format&n=SwBmQiR_ocl5w4KS&q=85&s=d755894e3378f41a35cb6031245195d8" alt="MongoDB logo" className="integration-table-logo" width="24" height="24" data-path="images/integrations/logos/mongodb.svg" /> [MongoDB](/integrations/clickpipes/mongodb)                              | <Badge color="yellow">Public Beta</Badge>     | [MongoDB Atlas](/integrations/clickpipes/mongodb/source/atlas), [Amazon DocumentDB](/integrations/clickpipes/mongodb/source/documentdb), [self-managed](/integrations/clickpipes/mongodb/source/generic), and other compatible data sources.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| <img src="https://mintcdn.com/private-7c7dfe99-trino-dialect/TaJVOJrb2nH0JFss/images/logo-bigquery.svg?fit=max&auto=format&n=TaJVOJrb2nH0JFss&q=85&s=a9e379c79a38f63fa00c895cc9482a6b" alt="BigQuery logo" className="integration-table-logo" width="24" height="24" data-path="images/logo-bigquery.svg" /> [BigQuery](/integrations/clickpipes/bigquery/overview)                                                                                                                          | <Badge color="purple">Private Preview</Badge> |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

<Tip>
  For unlisted services: due to differences in API implementations and replication support across service providers, not every compatible service is guaranteed to work out-of-the-box. If you're running into issues with a service that isn't listed, please [reach out to our team](https://clickhouse.com/company/contact?loc=clickpipes).
</Tip>

More connectors will get added to ClickPipes, you can find out more by [contacting us](https://clickhouse.com/company/contact?loc=clickpipes).

<h2 id="aws-iam-database-authentication">
  AWS IAM database authentication
</h2>

ClickPipes supports AWS IAM database authentication for Amazon RDS and Aurora. The configuration differs by database engine and AWS service, so choose the source setup guide that matches your database:

| Database engine | AWS service   | Setup guide                                                                                                |
| --------------- | ------------- | ---------------------------------------------------------------------------------------------------------- |
| PostgreSQL      | Amazon RDS    | [RDS for PostgreSQL](/integrations/clickpipes/postgres/source/rds#iam-authentication)                      |
| PostgreSQL      | Amazon Aurora | [Aurora PostgreSQL-Compatible Edition](/integrations/clickpipes/postgres/source/aurora#iam-authentication) |
| MySQL           | Amazon RDS    | [RDS for MySQL](/integrations/clickpipes/mysql/source/rds#iam-authentication)                              |
| MySQL           | Amazon Aurora | [Aurora MySQL-Compatible Edition](/integrations/clickpipes/mysql/source/aurora#iam-authentication)         |
| MariaDB         | Amazon RDS    | [RDS for MariaDB](/integrations/clickpipes/mysql/source/rds-maria#iam-authentication)                      |

<h2 id="list-of-static-ips">
  List of Static IPs
</h2>

The list of ClickPipes static NAT IPs has moved to the [ClickPipes networking documentation](/integrations/clickpipes/networking/static-ips).

<h2 id="adjusting-clickhouse-settings">
  Adjusting ClickHouse settings
</h2>

ClickHouse Cloud provides sensible defaults for most of the use cases. However, if you need to adjust some ClickHouse settings for the ClickPipes destination tables, a dedicated role for ClickPipes is the most flexible solution.
Steps:

1. create a custom role `CREATE ROLE my_clickpipes_role SETTINGS ...`. See [CREATE ROLE](/reference/statements/create/role) syntax for details.
2. add the custom role to ClickPipes user on step `Details and Settings` during the ClickPipes creation.

<Image img="https://mintcdn.com/private-7c7dfe99-trino-dialect/zkBy8QRjLpx6BosZ/images/integrations/data-ingestion/clickpipes/cp_custom_role.webp?fit=max&auto=format&n=zkBy8QRjLpx6BosZ&q=85&s=c26c64c4a45a73246aa8926e5ea646d7" alt="Assign a custom role" size="lg" border width="1744" height="470" data-path="images/integrations/data-ingestion/clickpipes/cp_custom_role.webp" />

<h2 id="clickpipes-advanced-settings">
  Adjusting ClickPipes advanced settings
</h2>

ClickPipes provides sensible defaults that cover the requirements of most use cases. If your use case requires additional fine-tuning, you can adjust the following settings:

<h3 id="clickpipes-advanced-settings-object-storage">
  Object storage ClickPipes
</h3>

| Setting                              | Default value | Description                                                                                                                                                 |
| ------------------------------------ | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Max insert bytes`                   | 10 GB         | Number of bytes to process in a single insert batch.                                                                                                        |
| `Max file count`                     | 100           | Maximum number of files to process in a single insert batch.                                                                                                |
| `Max threads`                        | auto(3)       | [Maximum number of concurrent threads](/reference/settings/session-settings/max-threads#max_threads) for file processing.                                   |
| `Max insert threads`                 | 1             | [Maximum number of concurrent insert threads](/reference/settings/session-settings/max-insert#max_insert_threads) for file processing.                      |
| `Min insert block size bytes`        | 1 GB          | [Minimum size of bytes in the block](/reference/settings/session-settings/min-insert#min_insert_block_size_bytes) which can be inserted into a table.       |
| `Max download threads`               | 4             | [Maximum number of concurrent download threads](/reference/settings/session-settings/max-download#max_download_threads).                                    |
| `Object storage polling interval`    | 30 s          | Configures the maximum wait period before inserting data into the ClickHouse cluster.                                                                       |
| `Parallel distributed insert select` | 2             | [Parallel distributed insert select setting](/reference/settings/session-settings/parallel#parallel_distributed_insert_select).                             |
| `Parallel view processing`           | false         | Whether to enable pushing to attached views [concurrently instead of sequentially](/reference/settings/session-settings/parallel#parallel_view_processing). |
| `Use cluster function`               | true          | Whether to process files in parallel across multiple nodes.                                                                                                 |

<Image img="https://mintcdn.com/private-7c7dfe99-trino-dialect/zkBy8QRjLpx6BosZ/images/integrations/data-ingestion/clickpipes/cp_advanced_settings.webp?fit=max&auto=format&n=zkBy8QRjLpx6BosZ&q=85&s=f4dcc8dc162755569f26a993ae22326a" alt="Advanced settings for ClickPipes" size="lg" border width="1724" height="620" data-path="images/integrations/data-ingestion/clickpipes/cp_advanced_settings.webp" />

<h3 id="clickpipes-advanced-settings-streaming">
  Streaming ClickPipes
</h3>

| Setting                          | Default value | Description                                                                           |
| -------------------------------- | ------------- | ------------------------------------------------------------------------------------- |
| `Streaming max insert wait time` | 5 s           | Configures the maximum wait period before inserting data into the ClickHouse cluster. |

<h2 id="error-reporting">
  Error reporting
</h2>

ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process.

<Note>
  This does not currently apply to CDC pipes (Postgres, MySQL, and MongoDB), for which operational and error logging are available directly in the ClickHouse Cloud console.
</Note>

<h3 id="record-errors">
  Record errors
</h3>

ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. This table has a [TTL](/reference/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.

<h3 id="system-errors">
  System errors
</h3>

Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/reference/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.

If ClickPipes can't connect to a data source after 15 min or to a destination after 1 hr, the ClickPipes instance stops and stores an appropriate message in the system error table (provided the ClickHouse instance is available).

<h2 id="monitoring">
  Monitoring
</h2>

In addition to in-console monitoring, ClickPipes exposes metrics to a [Prometheus-compatible endpoint](/products/cloud/features/monitoring/prometheus) for scraping. These metrics are published with other ClickHouse Cloud service metrics, and allow you to integrate ClickPipes monitoring with your existing observability stack (e.g., [Grafana](/products/cloud/features/monitoring/prometheus#integrating-with-grafana), [Datadog](/products/cloud/features/monitoring/prometheus#integrating-with-datadog)). See [Monitoring ClickPipes](/integrations/clickpipes/monitoring) for the full list of available metrics.

<h2 id="faq">
  FAQ
</h2>

* **What is ClickPipes?**

  ClickPipes is a ClickHouse Cloud feature that makes it easy for you to connect your ClickHouse services to external data sources, specifically Kafka. With ClickPipes for Kafka, you can easily continuously load data into ClickHouse, making it available for real-time analytics.

* **Does ClickPipes support data transformation?**

  Yes, ClickPipes supports basic data transformation by exposing the DDL creation. You can then apply more advanced transformations to the data as it is loaded into its destination table in a ClickHouse Cloud service leveraging ClickHouse's [materialized views feature](/concepts/features/materialized-views/cascading-materialized-views).

* **Does using ClickPipes incur an additional cost?**

  ClickPipes is billed on two dimensions: Ingested Data and Compute. The full details of the pricing are available on [this page](/products/cloud/reference/billing/clickpipes/index). Running ClickPipes might also generate an indirect compute and storage cost on the destination ClickHouse Cloud service similar to any ingest workload.

* **Is there a way to handle errors or failures when using ClickPipes for Kafka?**

  Yes, ClickPipes for Kafka will automatically retry in the event of failures when consuming data from Kafka for any operational issue including network issues, connectivity issues, etc. In the event of malformed data or invalid schema, ClickPipes will store the record in the record\_error table and continue processing.