> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-trino-dialect.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Discover ClickPipe source schema

> **This endpoint is in beta.** API contract is stable, and no breaking changes are expected in the future. <br /><br /> Infers the schema (field names and ClickHouse data types) of a ClickPipe source without creating a pipe. Supported for Kafka, Kinesis, Pub/Sub, and object storage sources. Object storage inference runs on the destination service, which must be running.

<span data-endpoint-badge="Beta"><Badge color="blue">Beta</Badge></span>

**This endpoint is in beta.** API contract is stable, and no breaking changes are expected in the future. <br /><br /> Infers the schema (field names and ClickHouse data types) of a ClickPipe source without creating a pipe. Supported for Kafka, Kinesis, Pub/Sub, and object storage sources. Object storage inference runs on the destination service, which must be running.


## OpenAPI

````yaml _specs/cloud-openapi.json POST /v1/organizations/{organizationId}/services/{serviceId}/clickpipes/schemaDiscovery
openapi: 3.1.2
info:
  title: OpenAPI spec for ClickHouse Cloud
  version: '1.0'
  contact:
    name: ClickHouse Support
    url: >-
      https://clickhouse.com/docs/en/cloud/manage/openapi?referrer=openapi-1078296
    email: support@clickhouse.com
servers:
  - url: https://api.clickhouse.cloud
security:
  - basicAuth: []
tags:
  - name: Organization
  - name: User management
  - name: Billing
  - name: Role Management
  - name: Service
  - name: Backup
  - name: API keys
  - name: Prometheus
  - name: ClickPipes
  - name: ClickStack
  - name: Postgres
  - name: UDF
paths:
  /v1/organizations/{organizationId}/services/{serviceId}/clickpipes/schemaDiscovery:
    post:
      tags:
        - ClickPipes
      summary: Discover ClickPipe source schema
      description: >-
        **This endpoint is in beta.** API contract is stable, and no breaking
        changes are expected in the future. <br /><br /> Infers the schema
        (field names and ClickHouse data types) of a ClickPipe source without
        creating a pipe. Supported for Kafka, Kinesis, Pub/Sub, and object
        storage sources. Object storage inference runs on the destination
        service, which must be running.
      operationId: clickPipeSchemaDiscovery
      parameters:
        - in: path
          name: organizationId
          description: ID of the organization that owns the service.
          required: true
          schema:
            type: string
            format: uuid
        - in: path
          name: serviceId
          description: ID of the service to run schema discovery against.
          required: true
          schema:
            type: string
            format: uuid
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ClickPipeSchemaDiscoveryRequest'
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: number
                    description: HTTP status code.
                    example: 200
                  requestId:
                    type: string
                    description: Unique id assigned to every request. UUIDv4
                    format: uuid
                  result:
                    $ref: '#/components/schemas/ClickPipeSchemaDiscoveryResponse'
        '400':
          description: >-
            The request cannot be processed due to a client error. Please verify
            your request parameters and try again.
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: number
                    description: HTTP status code.
                    example: 400
                  error:
                    type: string
                    description: Detailed error description.
                  requestId:
                    type: string
                    description: Unique id assigned to every request. UUIDv4
                    format: uuid
        '500':
          description: >-
            An internal server error has occurred. If this issue persists,
            please contact ClickHouse Cloud support for assistance.
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: integer
                    description: HTTP status code.
                    example: 500
                  error:
                    type: string
                    description: Detailed error description.
                  requestId:
                    type: string
                    description: Unique id assigned to every request. UUIDv4
                    format: uuid
components:
  schemas:
    ClickPipeSchemaDiscoveryRequest:
      properties:
        source:
          $ref: '#/components/schemas/ClickPipeSchemaDiscoverySource'
    ClickPipeSchemaDiscoveryResponse:
      properties:
        fields:
          type: array
          description: Inferred schema fields with their ClickHouse data types.
          items:
            $ref: '#/components/schemas/ClickPipeSchemaDiscoveryField'
        meta:
          oneOf:
            - $ref: '#/components/schemas/ClickPipeSchemaDiscoveryMeta'
            - type: 'null'
    ClickPipeSchemaDiscoverySource:
      properties:
        kafka:
          oneOf:
            - $ref: '#/components/schemas/ClickPipePostKafkaSource'
            - type: 'null'
        kinesis:
          oneOf:
            - $ref: '#/components/schemas/ClickPipePostKinesisSource'
            - type: 'null'
        pubsub:
          oneOf:
            - $ref: '#/components/schemas/ClickPipePostPubSubSource'
            - type: 'null'
        objectStorage:
          oneOf:
            - $ref: '#/components/schemas/ClickPipePostObjectStorageSource'
            - type: 'null'
    ClickPipeSchemaDiscoveryField:
      properties:
        name:
          description: Name of the inferred field.
          type: string
          example: user_id
        type:
          description: Inferred ClickHouse data type of the field.
          type: string
          example: Int64
        optional:
          description: Whether the field is optional (nullable) in the source.
          type:
            - boolean
            - 'null'
    ClickPipeSchemaDiscoveryMeta:
      type: object
      description: Source-specific schema discovery metadata.
      additionalProperties:
        type: string
    ClickPipePostKafkaSource:
      properties:
        type:
          description: Type of the Kafka source.
          type: string
          enum:
            - kafka
            - redpanda
            - msk
            - gcmk
            - confluent
            - warpstream
            - azureeventhub
            - dokafka
        format:
          description: Format of the Kafka source.
          type: string
          enum:
            - JSONEachRow
            - Avro
            - AvroConfluent
            - Protobuf
        brokers:
          description: Brokers of the Kafka source.
          type: string
        topics:
          description: Topics of the Kafka source.
          type: string
        consumerGroup:
          description: >-
            Consumer group of the Kafka source. If not provided
            "clickpipes-<<ID>>" will be used.
          type:
            - string
            - 'null'
          example: my-clickpipe-consumer-group
        authentication:
          description: >-
            Authentication method of the Kafka source. Supported authentication
            methods: kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS,
            msk: SCRAM-SHA-512, IAM_ROLE, IAM_USER, MUTUAL_TLS, gcmk: PLAIN,
            MUTUAL_TLS, confluent: PLAIN, MUTUAL_TLS, warpstream: PLAIN,
            azureeventhub: PLAIN, redpanda: SCRAM-SHA-256, SCRAM-SHA-512,
            MUTUAL_TLS, dokafka: SCRAM-SHA-256, MUTUAL_TLS
          type: string
          enum:
            - PLAIN
            - SCRAM-SHA-256
            - SCRAM-SHA-512
            - IAM_ROLE
            - IAM_USER
            - MUTUAL_TLS
        iamRole:
          description: >-
            IAM role for the Kafka source. Use with IAM role authentication.
            Read more in ClickPipes documentation:
            https://clickhouse.com/docs/en/integrations/clickpipes/kafka#iam
          type:
            - string
            - 'null'
          example: arn:aws:iam::123456789012:role/MyRole
        offset:
          oneOf:
            - $ref: '#/components/schemas/ClickPipeKafkaOffset'
            - type: 'null'
        schemaRegistry:
          oneOf:
            - $ref: '#/components/schemas/ClickPipeMutateKafkaSchemaRegistry'
            - type: 'null'
        caCertificate:
          description: PEM encoded CA certificates to validate the broker's certificate.
          type:
            - string
            - 'null'
        reversePrivateEndpointIds:
          type: array
          description: >-
            Reverse private endpoint UUIDs used for a secure private connection
            to the Kafka source.
          items:
            type: string
        exactlyOnce:
          description: >-
            Enable exactly-once delivery. Guarantees every Kafka record is
            inserted exactly once across restarts and rebalances. Can only be
            set at pipe creation.
          type:
            - boolean
            - 'null'
        credentials:
          description: >-
            Credentials for Kafka source. Choose one that is supported by the
            authentication method.
          oneOf:
            - $ref: '#/components/schemas/PLAIN'
            - $ref: '#/components/schemas/MskIamUser'
            - $ref: '#/components/schemas/AzureEventHub'
            - $ref: '#/components/schemas/MutualTLS'
    ClickPipePostKinesisSource:
      properties:
        format:
          description: Format of the Kinesis stream.
          type: string
          enum:
            - JSONEachRow
            - Avro
            - AvroConfluent
        streamName:
          description: Name of the Kinesis stream.
          type: string
          example: my-stream
        region:
          description: AWS region of the Kinesis stream.
          type: string
          example: us-east-1
        useEnhancedFanOut:
          description: Use enhanced fan-out for the Kinesis stream.
          type:
            - boolean
            - 'null'
        iteratorType:
          description: >-
            Type of iterator to use when reading from the Kinesis stream. If
            AT_TIMESTAMP is used, the timestamp field must be provided.
          type: string
          enum:
            - TRIM_HORIZON
            - LATEST
            - AT_TIMESTAMP
        timestamp:
          description: >-
            UNIX timestamp to start reading from the Kinesis stream. Required if
            iteratorType is AT_TIMESTAMP.
          type:
            - integer
            - 'null'
          example: 1615766400
        authentication:
          description: Authentication method to use with the Kinesis stream.
          type: string
          enum:
            - IAM_ROLE
            - IAM_USER
        iamRole:
          description: IAM role to use for authentication. Required if IAM_ROLE is used.
          type:
            - string
            - 'null'
          example: arn:aws:iam::123456789012:role/MyRole
        accessKey:
          oneOf:
            - $ref: '#/components/schemas/MskIamUser'
            - type: 'null'
    ClickPipePostPubSubSource:
      properties:
        format:
          description: >-
            Format of messages in the Pub/Sub topic. GCP Pub/Sub ClickPipes are
            in limited preview — contact support to enable this feature for your
            organization.
          type: string
          enum:
            - JSONEachRow
            - Avro
            - Protobuf
          example: JSONEachRow
        projectId:
          description: GCP project ID that owns the Pub/Sub topic.
          type: string
          example: my-gcp-project
        topic:
          description: Pub/Sub topic name (not the fully-qualified path).
          type: string
          example: my-topic
        authentication:
          description: >-
            Authentication method to use with GCP Pub/Sub. Currently only
            SERVICE_ACCOUNT is supported.
          type: string
          enum:
            - SERVICE_ACCOUNT
          example: SERVICE_ACCOUNT
        seekType:
          description: >-
            Starting position strategy for consuming the subscription. The
            seekTimestamp companion is required only when seekType is
            "timestamp"; setting it for a mismatched seek type is rejected.
          type: string
          enum:
            - latest
            - earliest
            - timestamp
          example: earliest
        seekTimestamp:
          description: >-
            RFC 3339 / ISO 8601 timestamp to seek to. Required when seekType is
            "timestamp"; must be omitted otherwise.
          type:
            - string
            - 'null'
          format: date-time
          example: '2026-04-10T12:00:00Z'
        filter:
          description: >-
            Optional Pub/Sub subscription filter expression (CEL). Maximum 256
            characters.
          type:
            - string
            - 'null'
          maxLength: 256
        enableOrdering:
          description: >-
            Whether to enable ordered delivery of messages (requires messages to
            be published with ordering keys).
          type:
            - boolean
            - 'null'
        ackDeadline:
          description: >-
            Acknowledgement deadline for messages, in seconds. Must be between
            10 and 600.
          type:
            - integer
            - 'null'
          minimum: 10
          maximum: 600
        serviceAccountKey:
          $ref: '#/components/schemas/ServiceAccount'
      required:
        - format
        - projectId
        - topic
        - authentication
        - seekType
        - serviceAccountKey
    ClickPipePostObjectStorageSource:
      properties:
        type:
          description: Type of the ObjectStorage source.
          type: string
          enum:
            - s3
            - gcs
            - dospaces
            - azureblobstorage
            - cloudflarer2
            - ovhobjectstorage
        format:
          description: Format of the files.
          type: string
          enum:
            - JSONEachRow
            - JSONAsObject
            - CSV
            - CSVWithNames
            - TabSeparated
            - TabSeparatedWithNames
            - Parquet
            - Avro
        url:
          description: >-
            Provide a path to the file(s) you want to ingest. You can specify
            multiple files using bash-like wildcards. For more information, see
            the documentation on using wildcards in path:
            https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations
          type: string
          example: >-
            https://datasets-documentation.s3.eu-west-3.amazonaws.com/http/**.ndjson.gz
        delimiter:
          description: Delimiter used in the files.
          type:
            - string
            - 'null'
          example: ','
        compression:
          description: Compression algorithm used for the files.
          type:
            - string
            - 'null'
          enum:
            - none
            - gzip
            - gz
            - brotli
            - br
            - xz
            - LZMA
            - zstd
            - auto
          example: auto
        isContinuous:
          description: >-
            If set to true, the pipe will continuously read new files from the
            source. If set to false, the pipe will read the files only once. New
            files have to be uploaded lexically order.
          type:
            - boolean
            - 'null'
        queueUrl:
          description: >-
            Queue URL for event-based continuous ingestion. For S3, provide an
            SQS queue URL. For GCS, provide a Pub/Sub subscription (e.g.
            projects/{project}/subscriptions/{name}). When provided, files are
            ingested based on event notifications rather than lexicographical
            order. Only applicable when isContinuous is true and authentication
            is not public.
          type:
            - string
            - 'null'
          example: https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue
        skipInitialLoad:
          description: >-
            If set to true, skips the initial load and only ingests files
            delivered by queue notifications. Only applicable when queueUrl is
            provided.
          type:
            - boolean
            - 'null'
        startAfter:
          description: >-
            Skip all files up to and including this object key during the
            initial load. Cannot be provided when skipInitialLoad is true.
          type:
            - string
            - 'null'
          example: events/2026-06-01/
        authentication:
          description: >-
            Authentication method. IAM_USER is for S3, GCS, and DigitalOcean
            Spaces. IAM_ROLE is for S3 only. SERVICE_ACCOUNT is for GCS only.
            CONNECTION_STRING is for Azure Blob Storage. PUBLIC uses no
            authentication.
          type:
            - string
            - 'null'
          enum:
            - IAM_ROLE
            - IAM_USER
            - CONNECTION_STRING
            - SERVICE_ACCOUNT
        iamRole:
          description: >-
            IAM role to be used with IAM role authentication. Read more in
            ClickPipes documentation:
            https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#authentication
          type:
            - string
            - 'null'
          example: arn:aws:iam::123456789012:role/MyRole
        connectionString:
          description: >-
            Connection string for Azure Blob Storage authentication. Required
            when authentication is CONNECTION_STRING.
          type:
            - string
            - 'null'
          example: >-
            DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net
        path:
          description: >-
            Path to the file(s) within the Azure container. Used for Azure Blob
            Storage sources. You can specify multiple files using bash-like
            wildcards. For more information, see the documentation on using
            wildcards in path:
            https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations
          type:
            - string
            - 'null'
          example: data/logs/*.json
        azureContainerName:
          description: >-
            Container name for Azure Blob Storage. Required when type is
            azureblobstorage.
          type:
            - string
            - 'null'
          example: mycontainer
        accessKey:
          oneOf:
            - $ref: '#/components/schemas/MskIamUser'
            - type: 'null'
        serviceAccountKey:
          description: >-
            Base64-encoded GCP service account JSON key. Required when
            authentication is SERVICE_ACCOUNT.
          type:
            - string
            - 'null'
    ClickPipeKafkaOffset:
      properties:
        strategy:
          description: Offset strategy.
          type: string
          enum:
            - from_beginning
            - from_latest
            - from_timestamp
        timestamp:
          description: >-
            A minute precision UTC timestamp to start from. Required for
            "from_timestamp" strategy.
          type:
            - string
            - 'null'
          example: 2021-01-01T00:00
    ClickPipeMutateKafkaSchemaRegistry:
      properties:
        url:
          description: Schema URL. HTTPS required.
          type: string
          example: https://psrc-aa00.us-east-2.aws.confluent.cloud/schemas/ids/100004
        authentication:
          description: Authentication type of the schema registry.
          type: string
          enum:
            - PLAIN
        caCertificate:
          description: >-
            PEM encoded CA certificates to validate the schema registry's
            certificate.
          type:
            - string
            - 'null'
        credentials:
          $ref: '#/components/schemas/ClickPipeKafkaSchemaRegistryCredentials'
    PLAIN:
      properties:
        username:
          description: Database username.
          type: string
          example: postgres_user
        password:
          description: Database password.
          type: string
          format: password
          example: your_secure_password
    MskIamUser:
      properties:
        accessKeyId:
          description: IAM access key ID.
          type: string
        secretKey:
          description: IAM secret key.
          type: string
    AzureEventHub:
      properties:
        connectionString:
          description: Connection string for Azure EventHub source.
          type: string
    MutualTLS:
      properties:
        certificate:
          description: PEM encoded client certificate for mTLS authentication.
          type: string
        privateKey:
          description: PEM encoded client private key for mTLS authentication.
          type: string
          format: password
    ServiceAccount:
      properties:
        serviceAccountFile:
          description: Google Cloud service account JSON key file content, base64 encoded.
          type: string
    ClickPipeKafkaSchemaRegistryCredentials:
      properties:
        username:
          description: Username for the schema registry.
          type: string
        password:
          description: Password for the schema registry.
          type: string
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
      description: >-
        Use key ID and key secret obtained in ClickHouse Cloud console:
        https://clickhouse.com/docs/cloud/manage/openapi

````