Managed Kafka on AWS

Kafka®-compatible brokers by Selfhost.dev, with the data kept in S3.

Brokers compatible with Apache Kafka® that run on AWS and keep topic data in S3 instead of on their own disks, built on AutoMQ. Pick one broker or three across availability zones, connect any Kafka client with SASL/SCRAM and watch throughput, partitions and S3 traffic in the console, with the price shown before you create.

Create a Kafka cluster See how it works

Kafka 3.9.1 protocol / Topic data in S3 / 1 broker or 3 across zones / Pay-as-you-go

The reliable home for your data.

Dedicated instances, run for you

Dedicated PostgreSQL, MySQL, Redis, ClickHouse, OpenSearch and Kafka on AWS, with BYOC into your own account. Multi-AZ on all six, PITR on Postgres and MySQL.

In our AWS account or your own with BYOC. You get AWS uptime and performance; we handle the ops.

How it works

Brokers without the storage problem.
Take one out and watch.

Producers write to the brokers, the brokers keep topic data in S3 and consumer groups read it back. Take a broker out of a three-broker cluster: it is fenced, a replacement joins and the data never moves, because it was never on that broker's disk.

topic: orders, 3 producers, 3 consumer groups

What you get

Built into the platform.
Streaming next to your databases.

Every cluster you create in the console comes with all of this, on the same account and the same bill as your PostgreSQL, ClickHouse and OpenSearch.

Kafka 3.9.1 protocol

Brokers speak the Kafka 3.9.1 protocol, so your existing producers, consumers and tools connect the way they connect to any cluster. The engine underneath is AutoMQ.

Topic data in S3

Brokers keep topic data in S3 rather than on their own disks. Durability comes from S3, so topics run with a replication factor of 1 by design.

One broker or three across zones

Start with a single broker or create a three-broker KRaft cluster with one broker per availability zone. When a broker stops responding it is fenced out and a replacement is provisioned automatically.

SASL/SCRAM, ACLs and TLS

Every client authenticates with SASL/SCRAM-SHA-256 as a user you create, with a role that maps to Kafka ACLs. On a single broker, TLS with a publicly trusted certificate is an option at create.

Schema Registry

A Confluent-compatible registry for Avro, Protobuf and JSON Schema on port 8081, chosen at create on a single broker. It signs in with the cluster's own Kafka user.

Metrics, logs and backups

Messages and bytes in and out, produce and fetch requests, partitions, S3 traffic and offline partitions, with alert rules on them. Tail the broker log and open a point-in-time cluster snapshot with consumer-group lag. Backups archive topics with their settings, ACLs and users.

Connect in a minute

Any Kafka client.
Three settings and a user.

Copy the bootstrap server and a user from the console's Connection tab. The same settings work for the Kafka command-line tools, librdkafka clients such as confluent-kafka and kafkajs.

SASL_PLAINTEXT or SASL_SSL

A single broker created with TLS switches to SASL_SSL once its certificate lands, a few minutes after the broker starts; the Connection tab always shows the protocol to use. Without TLS, SCRAM still protects the password.

# client.properties, as the Connection tab gives it
bootstrap.servers=YOUR_BROKER:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="YOUR_USER" password="YOUR_PASSWORD";

# create a topic: replication factor 1, the data lives in S3
kafka-topics.sh --bootstrap-server YOUR_BROKER:9092 \
  --command-config client.properties \
  --create --topic events --partitions 3 --replication-factor 1

# pip install confluent-kafka
import os
from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "YOUR_BROKER:9092",
    "security.protocol": "SASL_PLAINTEXT",  # SASL_SSL once TLS is applied
    "sasl.mechanism": "SCRAM-SHA-256",
    "sasl.username": "YOUR_USER",
    "sasl.password": os.environ["KAFKA_PASSWORD"],
})
producer.produce("events", b'{"order": 1042}')
producer.flush()

// npm install kafkajs
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  brokers: ['YOUR_BROKER:9092'],
  ssl: false, // true once TLS is applied
  sasl: { mechanism: 'scram-sha-256', username: 'YOUR_USER', password: process.env.KAFKA_PASSWORD },
});

const producer = kafka.producer();
await producer.connect();
await producer.send({ topic: 'events', messages: [{ value: '{"order":1042}' }] });
await producer.disconnect();

Before you start

Six things we would
want to know first.

S3 changes how a broker works, so some of what you know about Kafka clusters does not carry over. Here it is up front.

1 Topics run with a replication factor of 1 by design: durability comes from S3 and the broker refuses a higher factor.
2 Backups archive topics, their settings, ACLs and users, not messages. A restore brings the topics back empty.
3 The shape is fixed at create: one broker or three, on the machine size you pick. The broker volume can grow.
4 TLS and the Schema Registry are single-broker options chosen at create. A three-broker cluster connects with SASL_PLAINTEXT: SCRAM protects the password, but messages cross the network unencrypted.
5 The Schema Registry's certificate is issued by the broker machine for now, so serializers that verify certificates, the Confluent Java and Python ones among them, cannot use it yet.
6 Topic data in S3 is billed on usage, storage and requests, outside the estimate the console shows before you create.

Frequently asked questions

Is this Apache Kafka?

It is Kafka-compatible. The brokers run AutoMQ, which speaks the Kafka 3.9.1 protocol, so Kafka producers, consumers and tools connect as usual. The difference is where topic data lives: in S3 instead of on broker disks.

Why keep topic data in S3?

Brokers stop being the place your data lives. Durability comes from S3, so topics run with a replication factor of 1 and a failed broker is fenced and replaced rather than rebuilt from its peers.

How do I create a cluster?

In the console's create wizard: pick Kafka, choose one broker or three, the region and the machine size, decide on TLS and a Schema Registry if it is a single broker, then create. The cluster appears with its bootstrap server, users and metrics; the Connection tab hands you a ready client.properties.

What does it cost?

Pay-as-you-go against prepaid credits, like every engine on Selfhost.dev. The broker machines and their volumes are metered by the hour and the create wizard shows that estimate before anything is created. Topic data lives in S3, so its storage and requests are billed on usage, outside the estimate. There are no tiers or subscriptions.

One broker or three?

One broker suits development and streams that can wait out a restart. For production, three brokers run one per availability zone as a KRaft quorum; a failed broker is replaced automatically. The shape is chosen at create.

How do clients authenticate?

With SASL/SCRAM-SHA-256 as a user you create, whose role maps to Kafka ACLs. On a single broker you can ask for TLS at create: its publicly trusted certificate lands a few minutes after the broker starts and from then on clients connect with SASL_SSL. A three-broker cluster connects with SASL_PLAINTEXT for now.

Is there a Schema Registry?

Yes, on a single broker, chosen at create: a Confluent-compatible registry for Avro, Protobuf and JSON Schema on port 8081 that signs in with the cluster's own Kafka user. It runs its own Java process next to the broker, so 8 GB of RAM or more is recommended. Its certificate is issued by the broker machine for now, so serializers that verify certificates cannot use it yet. There is no Kafka UI.

Can I manage it from my AI editor?

Not yet. The Selfhost.dev MCP server manages PostgreSQL, MySQL, Redis, ClickHouse and OpenSearch today; Kafka is created and managed in the console.

Streaming without the disk problem.
Priced before you press create.

Create a Kafka cluster

$5 in free credit to start, no card needed.