Confluent Cloud¶

This tool was designed to work with Confluent Cloud. It can manage service accounts, topics, and ACLs for Confluent Cloud clusters.

Getting Started¶

Ensure you have installed kafka-gitops or are using the kafka-gitops docker image as described in the installation instructions.

You must have the ccloud command line tools installed if you wish to auto-populate the principal fields on services.

Desired State File¶

Create a basic desired state file, state.yaml, such as:

settings:
  ccloud:
    enabled: true

topics:
  test-topic:
    partitions: 6
    replication: 3

services:
  test-service:
    type: application
    produces:
      - test-topic

To give an overview, throughout this guide, this will create:

• A topic named test-topic
• A service account named test-service
• A WRITE ACL for topic test-topic tied to the service account test-service

Configuration¶

To use kafka-gitops with Confluent Cloud, you'll need to set a few environment variables.

• KAFKA_BOOTSTRAP_SERVERS: your Confluent Cloud cluster URL
• KAFKA_SASL_JAAS_USERNAME: your Confluent Cloud API key
• KAFKA_SASL_JAAS_PASSWORD: your Confluent Cloud API secret
• KAFKA_SECURITY_PROTOCOL: SASL_SSL
• KAFKA_SASL_MECHANISM: PLAIN
• KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

The username/password shortcut requires KAFKA_SASL_MECHANISM, so keep that variable set when using the Confluent Cloud API key and secret pair.

Additionally, you'll need to login to the ccloud tool. You can automate this by setting the following environment variables:

• XX_CCLOUD_EMAIL: your Confluent Cloud administrator email
• XX_CCLOUD_PASSWORD: your Confluent Cloud administrator password

Then, you can run ccloud login and it will run without a prompt. This is great for CI builds.

You can optionally specify a path to a ccloud executable:

• CCLOUD_EXECUTABLE_PATH: /full/path/to/ccloud

Otherwise, ccloud must be on your path.

Validate¶

First, validate your state file is correct by running:

kafka-gitops -f state.yaml validate

An example success message would look like:

[VALID] Successfully validated the desired state file.

Accounts¶

Before generating an execution plan, you will need to create the service accounts. This can be done by running:

kafka-gitops -f state.yaml account

This currently only creates service accounts; it will not delete any.

Plan¶

We're now ready to generate a plan to execute against the cluster. By using the plan command, we are NOT changing the cluster.

Generate a plan by running:

kafka-gitops -f state.yaml plan -o plan.json

This will generate an execution plan to run against the cluster and save the output to plan.json.

The command will also pretty-print what changes it wants to make to the cluster.

Apply¶

To execute a plan against the cluster, we use the apply command.

By default, the apply command will generate a plan and then apply it. For most situations, you should output a plan file from the plan command and pass it to the apply command.

Example:

kafka-gitops -f state.yaml apply -p plan.json

You are now using kafka-gitops to manage your Confluent Cloud cluster. Once you are comfortable with the workflow, commit your state file to a repository and automate plan and apply steps in CI/CD.