У меня была установка kafka с одним узлом. Он работал нормально. Затем я добавил еще одного брокера и создал кластер кафки с двумя узлами. Я не устанавливал отдельный zookeeper и не использовал тот же zookeeper, который поставляется с пакетом kafka. Я сделал следующие изменения, чтобы создать кластер.
Изменения в zookeeper.properties в обоих узлах:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=1000
server.1=10.20.40.120:2888:3888
server.2=10.20.40.119:2888:3888
initLimit=10
syncLimit=5
tickTime=2000
Скопировано server.properties в broker1.properties на первом узле и broker2.properties на втором узле:
Содержание broker1.properties:
broker.id=1
listeners=PLAINTEXT://10.20.40.120:9092
advertised.listeners=PLAINTEXT://10.20.40.120:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
log.retention.hours=168
zookeeper.connect=10.20.40.120:2181,10.20.40.119:2181
zookeeper.connection.timeout.ms=6000
replica.fetch.max.bytes=4000012
message.max.bytes=2690123replica.fetch.max.bytes=4000012
message.max.bytes=2690123
max.message.bytes=4000012
Содержимое broker2.properties во втором узле:
broker.id=2
listeners=PLAINTEXT://10.20.40.119:9092
advertised.listeners=PLAINTEXT://10.20.40.119:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
zookeeper.connect=10.20.40.120:2181,10.20.40.119:2181
zookeeper.connection.timeout.ms=6000
Создал файл myid в broker1 и поместил в него 1. 2 в broker2.
Запустил zookeeper в обоих, используя следующую команду:
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
Запустил kafka в broker1 и broker2 с помощью следующих команд соответственно:
nohup bin/kafka-server-start.sh config/broker1.properties &
nohup bin/kafka-server-start.sh config/broker2.properties &
Теперь, когда я пытаюсь описать __consumer_offsets, я вижу ниже:
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 5 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 6 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 7 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 8 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 9 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 10 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 11 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 12 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 13 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 14 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 15 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 16 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 17 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 18 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 19 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 20 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 21 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 22 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 23 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 24 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 25 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 26 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 27 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 28 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 29 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 30 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 31 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 32 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 33 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 34 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 35 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 36 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 37 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 38 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 39 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 40 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 41 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 42 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 43 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 44 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 45 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 46 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 47 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 48 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 49 Leader: 0 Replicas: 0 Isr: 0
Разве актор репликации не должен быть установлен как 2, а лидер - как 1 и 2 для разных разделов, а реплики и Isr - как 1,2?
Когда я пытаюсь запустить потребителя, я получаю COORDINATOR NOT AVAILABLE и error_code=15. Из-за этой ошибки я обнаружил, что могут быть проблемы с моими __consumer_offsets и, следовательно, с кластером.
Где недостающее звено и как исправить то же самое?
