Kafka Replication Factor: We Stopped Brokers at 1, 2 and 3

Set the Kafka replication factor too low and one lost disk takes your data with it. Set it too high and every write is stored, and shipped between zones, more often than it needs to be.

The Kafka replication factor is the number of copies Kafka keeps of each partition. It decides two different things: how many broker failures stop your writes, and how many lose your data. We measured both for replication factor 1, 2 and 3, with min.insync.replicas, on Apache Kafka 3.9.1.

The usual rule says replication factor 1 is never safe in production. On disk-based Kafka, our tests agree.

On Selfhost.dev managed Kafka, which keeps topic data in S3, we force-stopped a broker mid-run and lost none of 35,723 acknowledged messages. Every command and output below is from those runs.

Key Facts

• On Apache Kafka 3.9.1, replication factor 3 with min.insync.replicas=2 and acks=all kept writing with one of three brokers down, and rejected writes with two down.
• Replication factor 2 forces a choice during an outage: min.insync.replicas=2 stops writes, min.insync.replicas=1 writes to a single copy.
• Replication factor 1 on disk-based Kafka took the partition offline (Leader: none) as soon as its broker stopped.
• On Selfhost.dev managed Kafka (AutoMQ engine, topic data in S3), a replication factor 1 topic lost 0 of 35,723 acknowledged messages when a broker was force-stopped.
• Raising about 1 GB from replication factor 1 to 3 took 9 seconds unthrottled, 77 seconds at 10 MiB/s.

What is the Kafka replication factor?

The Kafka replication factor is the number of copies Kafka keeps of each partition, counting the leader. With replication factor 3, a partition lives on three brokers, and its data survives two broker failures if all three copies were in sync.

• Where you set it: per topic, with –replication-factor. Without it, the broker’s default.replication.factor applies, and that defaults to 1.
• Its limit: it can’t exceed the broker count, since each copy needs its own broker.
• What “in sync” means: replicas that are caught up with the leader form the in-sync replica set (ISR), the Isr column of kafka-topics.sh –describe.
• What it protects: RF N keeps a partition’s data through N−1 broker failures, if the copies were in sync.

This is Kafka’s topic setting. Cassandra and HDFS use the same term for their own data copies, with different rules.

Replication factor vs partitions vs brokers

Partitions decide how many consumers can read a topic in parallel. The replication factor decides how many brokers hold a copy of each partition. Brokers are the machines both live on.

The two topic settings multiply. Our 6-partition test topic at RF 3 is 18 partition copies on 3 brokers, so every broker holds every partition.

Ask for more copies than there are brokers, and the create fails ($BOOT lists our three brokers): 
$ kafka-topics.sh --bootstrap-server $BOOT --create --topic rf4-test --partitions 1 --replication-factor 4
Error while executing topic command : Unable to replicate the partition 4 time(s): The target replication factor of 4 cannot be reached because only 3 broker(s) are registered.
[2026-11-05 07:15:20,795] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 4 time(s): The target replication factor of 4 cannot be reached because only 3 broker(s) are registered.
 (org.apache.kafka.tools.TopicCommand)

Result: the create fails, and the command exits with status 1.

Kafka replication factor 1, 2 or 3: How to Choose

Use replication factor 3 with min.insync.replicas=2 when you have three or more brokers, and RF 1 only for data you can rebuild. Each setting has two failure counts, and they differ; the table comes from the runs below.

How to read it:

• The counts assume producers use acks=all.
• “Stop writes” means the producer gets an error.
• “Lose data” means acknowledged writes are gone because every disk that held them is gone for good. A broker that stops and comes back loses nothing, as the runs show.

Then pick:

• Three or more brokers: RF 3 with min.insync.replicas=2. It is the only row where a broker can be down, even for a planned restart, and every new write still gets two copies.
• Two brokers: RF 3 isn’t possible (the RF 4 error above), so choose between the RF 2 rows.

On brokers that keep topic data in object storage, this choice changes. See what we measured.

How we tested

• Disk-based Apache Kafka, for comparison: three brokers on t4g.medium, one each in ap-south-1a, 1b and 1c, with broker.rack set to the zone.
  • Apache Kafka 3.9.1 (apache.org tarball), OpenJDK 17.0.21 (Amazon Corretto), Amazon Linux 2023 arm64, AWS ap-south-1.
  • One KRaft controller on t4g.small in 1a, kept separate so that stopping two brokers leaves the quorum up. It also ran the clients.
  • 20 GB gp3 per node (3,000 IOPS, 125 MB/s), 1 GB heap, PLAINTEXT inside the VPC.
  • Brokers were stopped with kafka-server-stop.sh, a clean shutdown.
• Selfhost.dev Kafka: three brokers on t4g.medium in AWS ap-south-1, AutoMQ engine, Kafka 3.9.1 protocol, SASL_SSL with SCRAM-SHA-256.
  • The RF tests ran on November 4, 2026, with brokers in ap-south-1a, 1b and 1c.
  • The broker-failure test ran on November 5 on a second cluster with brokers in 1a, 1b and 1a, so two shared a zone.
• Clients: the Apache Kafka 3.9.1 command-line tools shown in each block, run from the apache/kafka:3.9.1 image against Selfhost.dev.
• Producer for the disk-based failure tests: retries=0, enable.idempotence=false, request.timeout.ms=10000, delivery.timeout.ms=15000, max.block.ms=15000, so errors show at once instead of being retried.
• In the outputs: private IPs show as broker-1 to broker-3, and the Selfhost.dev endpoint as <endpoint>. Commands drop the /opt/kafka/bin/ path and our scripts’ timeout and docker run wrappers. Long lines are cut with …; repeated client WARN lines and lines with internal instance IDs are removed.

Shorthand for the disk-based commands:

BOOT=broker-1:9092,broker-2:9092,broker-3:9092
PP="--producer-property retries=0 --producer-property enable.idempotence=false \
--producer-property request.timeout.ms=10000 --producer-property delivery.timeout.ms=15000 \
--producer-property max.block.ms=15000"

Replication factor vs min.insync.replicas: what prevents data loss

The replication factor sets how many copies of a partition exist. acks=all makes the producer wait until every replica currently in sync has the write. min.insync.replicas sets the smallest in-sync set the leader accepts: below it, the leader refuses acks=all writes.

With RF 3 and min.insync.replicas=2:

• All three in sync: an acks=all write waits for all three.
• One broker down: it waits for the two left, and is accepted.
• Two brokers down: one in-sync replica is below the minimum, so the leader rejects the write.

Neither setting is enough alone:

• acks=all with an ISR of one is one copy.
• min.insync.replicas is ignored for acks=1 writes, which wait only for the leader’s own write.

RF 3, min.insync.replicas=2, acks=all

All three replicas in sync, then broker 1 stopped:

$ kafka-topics.sh --bootstrap-server $BOOT --create --topic rf3-min2 --partitions 1 --replication-factor 3 --config min.insync.replicas=2
Created topic rf3-min2.
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf3-min2
Topic: rf3-min2  TopicId: _jAtq3tFQqC_PVVBZzhwMA  PartitionCount: 1  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Topic: rf3-min2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:   LastKnownElr:
$ echo a-1 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf3-min2 --producer-property acks=all $PP
$ kafka-server-stop.sh on broker 1   # 07:15:36Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf3-min2
Topic: rf3-min2  TopicId: _jAtq3tFQqC_PVVBZzhwMA  PartitionCount: 1  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Topic: rf3-min2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3,2  Elr:   LastKnownElr:
$ echo a-2 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf3-min2 --producer-property acks=all $PP

The producer printed no error for a-2, so the write was accepted, with two copies, on brokers 3 and 2. Then broker 2 stopped too:

$ kafka-server-stop.sh on broker 2   # 07:16:44Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf3-min2
Topic: rf3-min2  TopicId: _jAtq3tFQqC_PVVBZzhwMA  PartitionCount: 1  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Topic: rf3-min2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3  Elr:   LastKnownElr:
$ echo a-3 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf3-min2 --producer-property acks=all $PP
[2026-11-05 07:17:53,727] ERROR Error when sending message to topic rf3-min2 with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
$ echo a-4 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf3-min2 --producer-property acks=1 $PP

After both brokers came back (Isr: 3,1,2), a consumer read:

$ kafka-console-consumer.sh --bootstrap-server $BOOT --topic rf3-min2 --from-beginning --max-messages 10 --timeout-ms 15000
a-1
a-2
a-4
[2026-11-05 07:18:27,283] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 3 messages

Result: a-3 was never stored: with one in-sync replica left, the leader rejected the acks=all write. a-4, sent with acks=1, still went through on one copy. The TimeoutException at the end is how the console consumer exits once –timeout-ms passes with no new message.

RF 2, min.insync.replicas=2: one broker down stops writes

One broker down is enough to stop writes. This was a clean kafka-server-stop.sh, the same kind of stop a rolling restart makes:

$ kafka-topics.sh --bootstrap-server $BOOT --create --topic rf2-min2 --partitions 1 --replication-factor 2 --config min.insync.replicas=2
Created topic rf2-min2.
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min2
Topic: rf2-min2  TopicId: 26zhThAjTGSOsfpa3ihefw  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=2
  Topic: rf2-min2  Partition: 0  Leader: 3  Replicas: 3,1  Isr: 3,1  Elr:   LastKnownElr:
$ echo b-1 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf2-min2 --producer-property acks=all $PP
$ kafka-server-stop.sh on broker 1   # 07:18:39Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min2
Topic: rf2-min2  TopicId: 26zhThAjTGSOsfpa3ihefw  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=2
  Topic: rf2-min2  Partition: 0  Leader: 3  Replicas: 3,1  Isr: 3  Elr:   LastKnownElr:
$ echo b-2 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf2-min2 --producer-property acks=all $PP
[2026-11-05 07:19:47,936] ERROR Error when sending message to topic rf2-min2 with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

Result: with one of two brokers down, every acks=all write was rejected.

RF 2, min.insync.replicas=1: writes continue, and that is the risk {#rf2-min1} (H3)

The follower, broker 3, stopped first:

$ kafka-topics.sh --bootstrap-server $BOOT --create --topic rf2-min1-rerun --partitions 1 --replication-factor 2 --config min.insync.replicas=1
Created topic rf2-min1-rerun.
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min1-rerun
Topic: rf2-min1-rerun  TopicId: 11UA0OMZRLaagZBlJFrXwA  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=1
  Topic: rf2-min1-rerun  Partition: 0  Leader: 2  Replicas: 2,3  Isr: 2,3  Elr:   LastKnownElr:
$ echo c-0 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf2-min1-rerun --producer-property acks=all $PP
$ kafka-server-stop.sh on broker 3   # 07:32:24Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min1-rerun
Topic: rf2-min1-rerun  TopicId: 11UA0OMZRLaagZBlJFrXwA  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=1
  Topic: rf2-min1-rerun  Partition: 0  Leader: 2  Replicas: 2,3  Isr: 2  Elr:   LastKnownElr:
$ echo c-1 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf2-min1-rerun --producer-property acks=all $PP

c-1 was accepted while it existed only on broker 2. Then broker 2 stopped, and only the stale broker 3 came back:

$ kafka-server-stop.sh on broker 2   # 07:34:04Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min1-rerun
Topic: rf2-min1-rerun  TopicId: 11UA0OMZRLaagZBlJFrXwA  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=1
  Topic: rf2-min1-rerun  Partition: 0  Leader: none  Replicas: 2,3  Isr: 2  Elr:   LastKnownElr:
$ echo c-2 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf2-min1-rerun --producer-property acks=all $PP
[2026-11-05 07:36:00,439] ERROR Error when sending message to topic rf2-min1-rerun with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for rf2-min1-rerun-0:15003 ms has passed since batch creation
$ kafka-server-start.sh on broker 3  # 07:36:01Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min1-rerun
Topic: rf2-min1-rerun  TopicId: 11UA0OMZRLaagZBlJFrXwA  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=1
  Topic: rf2-min1-rerun  Partition: 0  Leader: none  Replicas: 2,3  Isr: 2  Elr:   LastKnownElr:
$ kafka-console-consumer.sh --bootstrap-server $BOOT --topic rf2-min1-rerun --from-beginning --max-messages 10 --timeout-ms 15000
[2026-11-05 07:36:42,228] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages

Kafka kept the partition offline (Leader: none) rather than elect broker 3, which doesn’t have c-1. That’s because unclean.leader.election.enable is false by default. c-1 came back only with broker 2:

$ kafka-server-start.sh on broker 2  # 07:36:42Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf2-min1-rerun
Topic: rf2-min1-rerun  TopicId: 11UA0OMZRLaagZBlJFrXwA  PartitionCount: 1  ReplicationFactor: 2  Configs: min.insync.replicas=1
  Topic: rf2-min1-rerun  Partition: 0  Leader: 2  Replicas: 2,3  Isr: 2,3  Elr:   LastKnownElr:
$ kafka-console-consumer.sh --bootstrap-server $BOOT --topic rf2-min1-rerun --from-beginning --max-messages 10 --timeout-ms 15000
c-0
c-1
[2026-11-05 07:37:11,080] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 2 messages

Result: c-1 survived only because broker 2 came back. Had broker 2’s disk been lost instead of stopped, c-1 would be gone. The partition would stay offline, or you would enable unclean election and broker 3 would lead without it.

What NotEnoughReplicasAfterAppend means

NotEnoughReplicasAfterAppendException means the leader stored the batch, but the ISR shrank below min.insync.replicas before the followers confirmed it.

• The rejections above were different: NotEnoughReplicasException means the leader refused the write before storing it.
• After append, the batch is in the leader’s log but not acknowledged: “Messages are written to the log, but to fewer in-sync replicas than required.” A retry can store it twice unless the producer is idempotent.
• We didn’t trigger it: it needs the ISR to shrink mid-write, which our clean stops didn’t cause.

If you script these tests:

• The console producer exits with status 0 even when a send fails, so the ERROR line is the only signal.
• Our stops were clean, so the ISR shrank in a controlled way. After a crash, shrinking can take up to replica.lag.time.max.ms (30 seconds by default).

What Kafka replication factor 1 means on disk-based Kafka

On disk-based Kafka, a replication factor 1 partition goes offline whenever its broker stops. If that broker’s disk is lost, the partition is lost for good. acks=all adds no protection, because the only in-sync replica is the leader itself.

A single-copy topic and an acks=all write:

$ kafka-topics.sh --bootstrap-server $BOOT --create --topic rf1-disk --partitions 1 --replication-factor 1
Created topic rf1-disk.
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf1-disk
Topic: rf1-disk  TopicId: ZAnAqk7LT3aVVEmvp85Ruw  PartitionCount: 1  ReplicationFactor: 1  Configs:
  Topic: rf1-disk  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
$ echo d-1 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf1-disk --producer-property acks=all $PP

acks=all was accepted, but with one replica, “every in-sync replica” is one broker. The empty Configs: means the topic uses the broker’s min.insync.replicas, which defaults to 1. So nothing refuses the write and nothing adds a copy.

Then the broker holding the partition stopped:

$ kafka-server-stop.sh on broker 1   # 07:26:15Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf1-disk
Topic: rf1-disk  TopicId: ZAnAqk7LT3aVVEmvp85Ruw  PartitionCount: 1  ReplicationFactor: 1  Configs:
  Topic: rf1-disk  Partition: 0  Leader: none  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
$ echo d-2 | kafka-console-producer.sh --bootstrap-server $BOOT --topic rf1-disk --producer-property acks=all $PP
[2026-11-05 07:27:46,400] ERROR Error when sending message to topic rf1-disk with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for rf1-disk-0:15001 ms has passed since batch creation
$ kafka-console-consumer.sh --bootstrap-server $BOOT --topic rf1-disk --from-beginning --max-messages 10 --timeout-ms 15000
[2026-11-05 07:28:04,036] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
$ kafka-topics.sh --bootstrap-server $BOOT --describe --unavailable-partitions
  Topic: rf1-disk  Partition: 0  Leader: none  Replicas: 1  Isr: 1  Elr:   LastKnownElr:

The partition is offline:

• Isr still names broker 1, so Leader: none is the field to watch.
• The producer gets no clear error. It waits for a leader until delivery.timeout.ms runs out (15 seconds with our settings, 120 seconds by default).
• The consumer gets nothing.

When broker 1 came back, so did the data, because its disk was intact:

$ kafka-server-start.sh on broker 1  # 07:28:07Z
$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic rf1-disk
Topic: rf1-disk  TopicId: ZAnAqk7LT3aVVEmvp85Ruw  PartitionCount: 1  ReplicationFactor: 1  Configs:
  Topic: rf1-disk  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
$ kafka-console-consumer.sh --bootstrap-server $BOOT --topic rf1-disk --from-beginning --max-messages 10 --timeout-ms 15000
d-1
[2026-11-05 07:28:43,328] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 1 messages

unclean.leader.election.enable can’t help at RF 1. It lets an out-of-sync replica take over when no in-sync one is left, and at RF 1 there is no other replica.

Result: the partition went offline when its only broker stopped, and acks=all added no protection. On disk-based Kafka, RF 1 is as risky as the usual advice says; keep it for data you can rebuild.

When is replication factor 1 safe? On brokers that keep data in S3

Replication factor 1 can be durable when the broker’s disk doesn’t hold the only copy. On architectures such as AutoMQ, which Selfhost.dev Kafka runs, topic data lives in redundant object storage, and an acks=all write is confirmed only once it is in S3. That is the setup we tested; other S3-backed designs may differ.

Durability isn’t availability. In our test the data stayed safe in S3, but producers still saw errors during the broker failover. A single broker can also leave the cluster unavailable until it restarts.

Selfhost.dev managed Kafka uses the AutoMQ engine, which keeps topic data in object storage (see AutoMQ’s S3 Storage page). We ran the same kind of tests there.

A higher replication factor is stored as 1

An RF 1 topic:

$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --create --topic rf1-test --partitions 3 --replication-factor 1
Created topic rf1-test.
$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --describe --topic rf1-test
Topic: rf1-test  TopicId: VbPvqlbjSLmS2UHdwqoHCg  PartitionCount: 3  ReplicationFactor: 1  Configs: min.insync.replicas=1
  Topic: rf1-test  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: rf1-test  Partition: 1  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: rf1-test  Partition: 2  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:

Asking for RF 3 doesn’t fail. The topic is created, the command exits with status 0, and –describe shows one replica per partition. A request for RF 2 was accepted the same way.

$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --create --topic rf3-test --partitions 3 --replication-factor 3
Created topic rf3-test.
$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --describe --topic rf3-test
Topic: rf3-test  TopicId: W0b1zQN-QLeh6PniWf8XFw  PartitionCount: 3  ReplicationFactor: 1  Configs: min.insync.replicas=1
  Topic: rf3-test  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: rf3-test  Partition: 1  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: rf3-test  Partition: 2  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --create --topic rf2-test --partitions 3 --replication-factor 2
Created topic rf2-test.

AutoMQ’s controller rewrites the factor when it creates a topic, and sets the topic’s min.insync.replicas to 1 (ReplicationControlManager.java, AutoMQ 1.7.4):

if (ElasticStreamSwitch.isEnabled()) {
    // MIN_IN_SYNC_REPLICAS should be forced to 1 since replication factor is 1 (see createTopic method).
    keyToOps.put(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, new AbstractMap.SimpleEntry<>(OpType.SET, String.valueOf(1)));

    // Then, we force replication factor to 1 here since "replicas" on broker layer can only be 1.
    if (replicationFactor != 1) {
        topic.setReplicationFactor((short) 1);
        log.info("force replication factor to 1 for create topic {}, the real replication factor is decided by elastic stream", topic.name());
    }
}

Result: scripts that create topics at RF 3 keep working, and –describe reports RF 1.

What happened when a broker failed

On the second cluster, we force-stopped a broker while a producer was writing to an RF 1 topic:

• The verifiable producer sent 36,000 numbered messages to a 6-partition RF 1 topic over 30 minutes, 20 per second, with –acks -1 (acks=all).
• It logs each acknowledgment and error with a timestamp and doesn’t retry, so every error counts.
• A loop ran –describe about every 15 seconds.

$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --create --topic failover-test --partitions 6 --replication-factor 1
Created topic failover-test.
$ kafka-topics.sh --bootstrap-server <endpoint>:9092 --command-config /tmp/c.properties --describe --topic failover-test
Topic: failover-test  TopicId: SczHtO_LRG6iD-GOGrBvCA  PartitionCount: 6  ReplicationFactor: 1  Configs: min.insync.replicas=1
  Topic: failover-test  Partition: 0  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 1  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 2  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 3  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 4  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 5  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
$ kafka-verifiable-producer.sh --bootstrap-server <endpoint>:9092 --producer.config /tmp/c.properties --topic failover-test --throughput 20 --max-messages 36000 --acks -1

At 08:22:18 UTC we force-stopped the EC2 instance of broker 2, the leader of partitions 2 and 4, with aws ec2 stop-instances –force. The timeline of this run:

### Sample at 08:22:27Z
  Topic: failover-test  Partition: 0  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 1  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 2  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 3  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 4  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 5  Leader: 0  Replicas: 0  Isr: 0  Elr:   LastKnownElr:
producer: acked=1581 errors=30
…
### Sample at 08:30:24Z
  Topic: failover-test  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 1  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 2  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 3  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 4  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 5  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
producer: acked=11083 errors=82

### Sample at 08:30:39Z
  Topic: failover-test  Partition: 0  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 1  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 2  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 3  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 4  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
  Topic: failover-test  Partition: 5  Leader: 2  Replicas: 2  Isr: 2  Elr:   LastKnownElr:
producer: acked=11881 errors=277

What the samples show:

• At the failure, writes kept going. Partitions 2 and 4 moved to a live broker (Replicas 0 instead of 2). AutoMQ’s comparison page says such a move needs only “a small amount of metadata” to be synchronized.
• The stop came later, about eight minutes after the failure, when all six partitions moved to broker 1, then to the replacement. Writes flowed again from 08:31:16.

After the run, a console consumer read the whole topic: 35,735 messages. A script compared them with the producer’s log:

$ python3 -I verify.py producer.jsonl consumed.txt
acked by broker:        35723
send errors:            277
  276 x class org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests
  1 x class org.apache.kafka.common.errors.NetworkException: Disconnected from node 1
  error window: 496.9s (first to last error)
consumed:               35735 (35735 distinct)
acked but not consumed: 0
duplicates consumed:    0

Result: on disk-based Kafka, replication factor 1 leaves a partition dependent on one broker’s disk. On Selfhost.dev Kafka, which uses the AutoMQ engine and S3-backed storage, all 35,723 acknowledged messages survived a force-stopped broker, and none was read twice.

• The 12 extra messages were sends that returned an error but had been stored anyway. An error doesn’t prove a message isn’t in the log.
• With default producer settings (retries on, delivery.timeout.ms of 120 seconds), most of the 277 failed sends would have been retried.
• This tests durability, not what a live consumer saw during the stop: we read the topic after the run.

Why it works

AutoMQ moves Kafka’s storage layer out of the broker, so the copy that matters lives in the object store:

• Its Stateless Broker page says it separates storage and compute “to offload Kafka’s storage layer to cloud storage via S3Stream, making Broker nodes stateless”.
• Its Deployment Recommendations say “object storage itself provides high durability and redundancy capabilities”, so “the following replica-related configurations are no longer needed or have no effect in AutoMQ”. The list includes default.replication.factor=1 and min.insync.replicas=1.
• Its Difference with Apache Kafka page says “Data is stored in S3 shared storage (high reliability with three replicas), eliminating the need for multiple replica replication”. It adds: “When adding nodes or replacing faulty nodes, data reassignment is not needed for quick switching.”

So whether RF 1 is safe depends on where the single Kafka-level copy lives: on a broker’s disk, it is one point of loss.

A three-broker Selfhost.dev Kafka cluster, one broker per availability zone with topic data in S3, comes to about $106 a month in us-east-1. The calculator shows the price for your region and broker size before you create anything.

See the 3-zone cluster priced, no signup

Is diskless Kafka in Apache Kafka yet?

Not in a release yet. KIP-1150: Diskless Topics is accepted as a direction, and its JIRA, KAFKA-19161, is open with no fix version.

AutoMQ is a separate project, not KIP-1150. Kafka-compatible brokers that keep topic data in S3 ship today, and other diskless Kafka options include WarpStream as well as AutoMQ.

How to change the replication factor of an existing topic

Use kafka-reassign-partitions.sh with a JSON plan that lists each partition’s new replicas: start it with –execute, then check it with –verify. For new topics, set the broker’s default.replication.factor instead.

• default.replication.factor applies to auto-created topics and to topics created without a factor (Kafka 3.9 broker configs). It doesn’t touch existing ones.
• On the S3-backed cluster, the controller fixes the factor at 1, so this section is about disk-based Kafka.

Example: raising a topic from replication factor 1 to 3

We raised a 6-partition topic of about 1 GB from RF 1 to RF 3, once without a limit and once with a throttle:

$ kafka-topics.sh --bootstrap-server $BOOT --create --topic reassign-test --partitions 6 --replication-factor 1
Created topic reassign-test.
$ kafka-producer-perf-test.sh --topic reassign-test --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=$BOOT acks=1 batch.size=262144 linger.ms=20 | tail -1
1000000 records sent, 122010.736945 records/sec (119.15 MB/sec), …

The plan keeps each partition’s current leader first, so leadership doesn’t move, and adds the other two brokers:

{"version":1,"partitions":[
  {"topic":"reassign-test","partition":0,"replicas":[2,1,3]},
  {"topic":"reassign-test","partition":1,"replicas":[3,1,2]},
  {"topic":"reassign-test","partition":2,"replicas":[1,2,3]},
  {"topic":"reassign-test","partition":3,"replicas":[3,1,2]},
  {"topic":"reassign-test","partition":4,"replicas":[3,1,2]},
  {"topic":"reassign-test","partition":5,"replicas":[2,1,3]}
]}

--execute starts the move and prints the current assignment. Save that JSON: it is the rollback plan, and running --execute with it restores the old replica lists. We didn’t run a rollback in this test.

$ kafka-reassign-partitions.sh --bootstrap-server $BOOT --reassignment-json-file /tmp/reassign-test.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"reassign-test","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"reassign-test","partition":1,"replicas":[3],"log_dirs":["any"]},…]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for reassign-test-0,reassign-test-1,reassign-test-2,reassign-test-3,reassign-test-4,reassign-test-5

Our script then polled --verify and --describe, printing how many partitions were done and each partition’s ISR. The final --verify also clears throttles:

t+3s  completed=1/6  isr=[2,1,3 3,1 1,3,2 3,1 3,1 2,1,3 ]
t+9s  completed=6/6  isr=[2,1,3 3,1,2 1,3,2 3,1,2 3,1,2 2,1,3 ]
$ kafka-reassign-partitions.sh --bootstrap-server $BOOT --reassignment-json-file /tmp/reassign-test.json --verify
Status of partition reassignment:
Reassignment of partition reassign-test-0 is completed.
Reassignment of partition reassign-test-1 is completed.
Reassignment of partition reassign-test-2 is completed.
Reassignment of partition reassign-test-3 is completed.
Reassignment of partition reassign-test-4 is completed.
Reassignment of partition reassign-test-5 is completed.

Clearing broker-level throttles on brokers 1,2,3
Clearing topic-level throttles on topic reassign-test

How long it took, and what the brokers saw

Without a throttle, all six partitions were done at the second check, 9 seconds after --execute returned. We repeated the move on a second 1 GB topic with --throttle 10485760 (10 MiB/s). --execute warned that --verify must keep running until the end, or the throttle stays on:

$ kafka-reassign-partitions.sh --bootstrap-server $BOOT --reassignment-json-file /tmp/reassign-throttled.json --execute --throttle 10485760
Current partition replica assignment

{"version":1,"partitions":[{"topic":"reassign-throttled","partition":0,"replicas":[2],"log_dirs":["any"]},…]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run --verify periodically, until the reassignment completes, to ensure the throttle is removed.
The inter-broker throttle limit was set to 10485760 B/s
Successfully started partition reassignments for reassign-throttled-0,reassign-throttled-1,reassign-throttled-2,…
t+2s  completed=0/6  isr=[2 3 1 1 2 3 ]
…
t+52s  completed=0/6  isr=[2 3 1 1 2 3 ]
t+58s  completed=0/6  isr=[2 3,1 1 1 2 3,1 ]
t+65s  completed=0/6  isr=[2 3,1 1 1,3 2 3,1,2 ]
t+71s  completed=2/6  isr=[2,1 3,1,2 1,3 1,3,2 2 3,1,2 ]
t+77s  completed=6/6  isr=[2,1,3 3,1,2 1,3,2 1,3,2 2,1,3 3,1,2 ]

We read each broker’s network receive counter (/proc/net/dev) and data directory size (du -sh) before and after each move:

Result: each broker received the partitions it didn’t have, about 2.2 GB in total, the two extra copies of a topic of about 1 GB. Afterwards, every broker held a full copy:

$ kafka-topics.sh --bootstrap-server $BOOT --describe --topic reassign-test
Topic: reassign-test  TopicId: Yrd-lnuOQQ65N2v1gU1IcQ  PartitionCount: 6  ReplicationFactor: 3  Configs:
  Topic: reassign-test  Partition: 0  Leader: 2  Replicas: 2,1,3  Isr: 2,1,3  Elr:   LastKnownElr:
  Topic: reassign-test  Partition: 1  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:   LastKnownElr:
  Topic: reassign-test  Partition: 2  Leader: 1  Replicas: 1,2,3  Isr: 1,3,2  Elr:   LastKnownElr:
  Topic: reassign-test  Partition: 3  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:   LastKnownElr:
  Topic: reassign-test  Partition: 4  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:   LastKnownElr:
  Topic: reassign-test  Partition: 5  Leader: 2  Replicas: 2,1,3  Isr: 2,1,3  Elr:   LastKnownElr:

The time is roughly the data each broker must receive divided by the throttle. 717 to 752 MB at 10.5 MB/s is about 70 seconds, and the move finished at 77.

To follow progress, watch the Isr column:

• In the throttled run, every partition kept one in-sync replica for the first 52 seconds. The new replicas joined one by one between 58 and 77 seconds.
• kafka-topics.sh --under-replicated-partitions won’t show this. During a reassignment it compares the ISR with the replica count minus the replicas being added (TopicCommand.java). So a partition going from one copy to three isn’t listed.
• Use --verify to know when the move is done.

What each extra copy costs

On disk-based Kafka, every extra copy is a full copy on another broker. Replication factor 3 stores each GB three times and sends two GB from the leader to its followers.

The reassignment above measured both: a full copy on every broker, and about 2.2 GB of traffic for two extra copies of about 1 GB.

• With one broker per zone and broker.rack set, as in our disk-based cluster, an RF 3 partition has a copy in every zone. So each write is sent to two followers in other zones.
• Raising RF later sends that traffic once for the data already there.

On brokers that keep topic data in object storage, the Kafka-level factor stays at 1, as shown above. There is no follower copy on another broker and no follower traffic. AutoMQ’s comparison with Apache Kafka says it “does not require cross-node multiple replica replication when writing messages”; the redundant copies live inside the object store.

On Selfhost.dev Kafka you pay for topic data by S3 usage, at $0.031 per GB-month, not for extra copies on broker disks.

See how Selfhost.dev managed Kafka runs without broker replica copies

The Short Version

• Three or more brokers: replication factor 3, min.insync.replicas=2, acks=all.
• Two brokers: pick the RF 2 row by what you’d rather lose in an outage: writes, or the second copy.
• RF 1: only for data you can rebuild on disk-based Kafka; the intended setting on AutoMQ-based brokers that keep topic data in S3, such as Selfhost.dev Kafka.

One Selfhost.dev Kafka broker costs about $35 a month in us-east-1, so the $5 welcome credit runs it for about four days. That’s long enough to point your own producer at it and rerun the RF 3 test above. One broker is for testing; run three across zones for production.

Start a Kafka broker on $5 of credit

Keep reading: 24 Kafka alternatives, sorted by whether your code still works · Managed Kafka with topic data in S3 · Best managed database services, priced across engines

Frequently Asked Questions