Kafka集群-添加节点

发表于2019年7月5日作者评心静气

环境：

CentOS 7 + jdk-12.0.1 + kafka_2.12-2.2.0

添加环境：

Replications: 3 [编号：0，1，2]
Topic : SP2
Partitions : 3

添加节点：

复制其中一个节点的配置文件，修改以下三个参数：broker.id | listeners | log.dirs
启动服务器，就OK了

问题：

新添加的Kafka节点并不会自动地分配数据，所以无法分担集群的负载，除非我们新建一个topic，即：只对新的Topic有效

解决步骤：

kafka-reassign-partitions.sh工具来重新分布分区。该工具有三种使用模式：
　　1、generate模式，给定需要重新分配的Topic，自动生成reassign plan（并不执行）
　　2、execute模式，根据指定的reassign plan重新分配Partition
　　3、verify模式，验证重新分配Partition是否成功

操作：

定义一个文件，里面说明哪些topic需要重新分区，文件内容如下：

 { "topics": [{ "topic": "SP2" }] }

生成重新分配的计划【此步可以手写】

./bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topic-ressign.js --broker-list "0,1,2,3" --generate

# 生成内容
Current partition replica assignment  
 #当前集群分区信息， 原始的分区信息，保存之后，可以用来回滚 
{"version":1,"partitions":[{"topic":"SP2","partition":2,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"topic":"SP2","partition":1,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"SP2","partition":0,"replicas":[1,0,2],"log_dirs":["any","any","any"]}]}   

Proposed partition reassignment configuration    
# 重新分区后的配置，保存成文件[result.json]，上面的不需要    {"version":1,"partitions":[{"topic":"SP2","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"SP2","partition":2,"replicas":[0,3,1],"log_dirs":["any","any","any"]},{"topic":"SP2","partition":1,"replicas":[3,2,0],"log_dirs":["any","any","any"]}]}

执行分配计划

./bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file result.json --execute

# 执行结果：
Current partition replica assignment
{"version":1,"partitions":[{"topic":"SP2","partition":2,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"topic":"SP2","partition":1,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"SP2","partition":0,"replicas":[1,0,2],"log_dirs":["any","any","any"]}]}
       # 
       Save this to use as the --reassignment-json-file option during rollback
       Successfully started reassignment of partitions.

验证

./bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file result.json --verify

# 验证结果
Status of partition reassignment: 
       Reassignment of partition SP2-0 completed successfully
       Reassignment of partition SP2-2 completed successfully
       Reassignment of partition SP2-1 completed successfully

./bin/kafka-topics.sh --bootstrap-server localhost:9091 --topic SP2 --describe 
Topic:SP2 PartitionCount:3  ReplicationFactor:3 Configs:segment.bytes=1073741824   
Topic: SP2  Partition: 0  Leader: 1 Replicas: 3,0,1 Isr: 1,0,3   
Topic: SP2  Partition: 1  Leader: 0 Replicas: 0,1,2 Isr: 0,2,1   
Topic: SP2  Partition: 2  Leader: 2 Replicas: 1,2,3 Isr: 2,1,3

# 出现以上原因是：在做 SP2 主题的时候，只有3个分区，添加新了集群节点3，没有分区用，--alter 命令添加分区后，再检查

./bin/kafka-topics.sh --bootstrap-server localhost:9091 --topic SP2 --alter --partitions 4

./bin/kafka-topics.sh --bootstrap-server localhost:9091 --topic SP2 --describe

Topic:SP2 PartitionCount:4  ReplicationFactor:3 Configs:segment.bytes=1073741824   
Topic: SP2  Partition: 0  Leader: 3 Replicas: 3,0,1 Isr: 1,0,3   
Topic: SP2  Partition: 1  Leader: 0 Replicas: 0,1,2 Isr: 0,2,1   
Topic: SP2  Partition: 2  Leader: 1 Replicas: 1,2,3 Isr: 2,1,3   
Topic: SP2  Partition: 3  Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

环境：