Services¶

If you have security on your cluster, you can use kafka-gitops to manage ACLs. Kafka GitOps automatically builds the necessary ACLs for most common use cases.

Application Example¶

A basic example shown below defines one topic, test-topic, and one service, my-application.

The service my-application both consumes from and produces to test-topic. This will generate the necessary ACLs for my-application to do this.

topics:
  test-topic:
    partitions: 6
    replication: 1

services:
  my-application:
    type: application
    principal: User:myapp
    consumes:
      - test-topic
    produces:
      - test-topic

Behind the scenes, this will generate three ACLs:

• READ for topic test-topic
• WRITE for topic test-topic
• READ for consumer group my-application

Group ID¶

The group.id used for consumer ACLs defaults to the service name. You can override this by specifying the group-id property, as shown below:

services:
  my-application:
    type: application
    principal: User:myapp
    group-id: my-application-service
    consumes:
      - test-topic
    produces:
      - test-topic

This allows your consumer group to access Kafka using my-application-service as the group.id.

Kafka Streams Example¶

A basic example shown below defines one topic, test-topic, and one Kafka Streams application, my-stream.

The service my-stream both consumes from and produces to test-topic. This will generate the necessary ACLs for my-stream to do this.

topics:
  test-topic:
    partitions: 6
    replication: 1

services:
  my-stream:
    type: kafka-streams
    principal: User:mystream
    consumes:
      - test-topic
    produces:
      - test-topic

Behind the scenes, this generates ACLs such as:

• READ for topic test-topic
• WRITE for topic test-topic
• READ for consumer group my-stream
• Various ACLs for Kafka streams internal topic management

Application ID¶

The application.id used for streams ACLs defaults to the service name. You can override this by specifying the application-id property, as shown below:

services:
  my-stream:
    type: kafka-streams
    principal: User:mystream
    application-id: my-stream-application
    consumes:
      - test-topic
    produces:
      - test-topic

This allows your Streams application to access Kafka using my-stream-application as the application.id.

Kafka Connect Example¶

A basic example which defines one Kafka Connect cluster that has one connector running.

topics:
  connect-configs-my-connect-cluster:
    partitions: 1
    replication: 1
    configs:
      cleanup.policy: compact

  connect-offsets-my-connect-cluster:
    partitions: 25
    replication: 1
    configs:
      cleanup.policy: compact

  connect-status-my-connect-cluster:
    partitions: 5
    replication: 1
    configs:
      cleanup.policy: compact

  rabbitmq-data:
    partitions: 6
    replication: 1

services:
  my-connect-cluster:
    type: kafka-connect
    principal: User:myconnectcluster
    connectors:
      rabbitmq-sink:
        consumes:
          - rabbitmq-data

Behind the scenes, this generates ACLs such as:

• READ for topic rabbitmq-data
• READ for the consumer group connect-rabbitmq-sink
• READ and WRITE for the internal kafka connect topics
• READ for the consumer group my-connect-cluster

Storage Topics¶

By default, kafka-gitops generates ACLs for the internal storage topics following this format:

• connect-configs-{service-name}
• connect-offsets-{service-name}
• connect-status-{service-name}

You can specify custom internal storage topics using the storage-topics property:

services:
  my-connect-cluster:
    type: kafka-connect
    principal: User:myconnectcluster
    storage-topics:
      config: custom-config-topic
      offset: custom-offset-topic
      status: custom-status-topic
    connectors:
      rabbitmq-sink:
        consumes:
          - rabbitmq-data

Group ID¶

The group.id used for the connect cluster ACLs defaults to the service name. You can override this by specifying the group-id property, as shown below:

services:
  my-connect-cluster:
    type: kafka-connect
    principal: User:myconnectcluster
    group-id: kafka-connect-cluster
    connectors:
      rabbitmq-sink:
        consumes:
          - rabbitmq-data

This allows your Connect cluster to access Kafka using kafka-connect-cluster as the group.id.