Quick Start¶

Getting started with kafka-gitops is simple. For this tutorial, we will assume:

• You have installed kafka-gitops as described in Installation.
• You have a Kafka cluster running on localhost:9092.

Desired State File¶

First, create a desired state file named state.yaml. In this file, we'll define a topic. To showcase how to set topic configs, we'll make it a compacted topic.

topics:
  my-example-topic:
    partitions: 6
    replication: 1
    configs:
      cleanup.policy: compact

Configuration¶

Currently, configuring kafka-gitops is done via environment variables. To configure properties, prefix them with KAFKA_. For example:

• KAFKA_BOOTSTRAP_SERVERS: injects as bootstrap.servers
• KAFKA_CLIENT_ID: injects as client.id

For our quick start example, open a terminal where your state.yaml file is located and set the bootstrap servers:

export KAFKA_BOOTSTRAP_SERVERS=localhost:9092

Validate¶

We can validate the desired state file conforms to the specification. To do this, run:

kafka-gitops validate

By default, it will look for state.yaml in the current directory. To specify a different file name, run:

kafka-gitops -f my-state-file.yaml validate

An example success message would look like:

[VALID] Successfully validated the desired state file.

Plan¶

We're now ready to generate a plan to execute against the cluster. By using the plan command, we are NOT changing the cluster.

Generate a plan by running:

kafka-gitops plan

This will generate an execution plan that looks like:

Generating execution plan...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

The following actions will be performed:

Topics: 1 to create, 0 to update, 0 to delete.

+ [TOPIC] my-example-topic
    + cleanup.policy: compact


ACLs: 0 to create, 0 to update, 0 to delete.

Plan: 1 to create, 0 to update, 0 to delete.

In most cases, you will want to output the plan to a file which can then be passed to the apply command. Plan files are JSON files. This can be done by running:

kafka-gitops plan -o plan.json

If running against a Kafka cluster with no authorizer configured or if you simply want to only manage topics, you can ignore ACLs completely. This can be done by running:

kafka-gitops --skip-acls plan

Apply¶

To execute a plan against the cluster, we use the apply command.

Run:

kafka-gitops apply

By default, the apply command will generate a plan and then apply it. For most situations, you should output a plan file from the plan command and pass it to the apply command.

Example:

kafka-gitops apply -p plan.json

An output of a successful apply would look like:

Executing apply...

Applying: [CREATE]

+ [TOPIC] my-example-topic
    + cleanup.policy: compact

Successfully applied.

[SUCCESS] Apply complete! Resources: 1 created, 0 updated, 0 deleted.

If there is a partial failure, successes will not be rolled back. A failure error would look something like:

[ERROR] Error thrown when attempting to create a Kafka topic:
org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3

[ERROR] An error has occurred during the apply process.
[ERROR] The apply process has stopped in place. There is no rollback.
[ERROR] Fix the error, re-create a plan, and apply the new plan to continue.

You now have the basic local workflow in place. If your cluster uses ACLs, continue with Services to define applications and generated ACLs.