Kafka动态增加Topic的副本(Replication)

Posted by Yancy on 2017-07-21

Kafka动态增加Topic的副本(Replication)

当前我的topic :countly_event 只有1个副本,如果集群其中一台机器出现问题,就会丢失数据。所以这里以后新建新的topic 最少2个副本,如果资源非常充足可以考虑副本3个。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[jollybi@kafka1 kafka_2.10-0.9.0.1]$ ./bin/kafka-topics.sh --zookeeper 172.31.2.6:2182,172.31.2.7:2182,172.31.2.8:2182 -describe -topic countly_event
Topic:countly_event PartitionCount:12 ReplicationFactor:1 Configs:
Topic: countly_event Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: countly_event Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: countly_event Partition: 2 Leader: 3 Replicas: 3 Isr: 3
Topic: countly_event Partition: 3 Leader: 4 Replicas: 4 Isr: 4
Topic: countly_event Partition: 4 Leader: 1 Replicas: 1 Isr: 1
Topic: countly_event Partition: 5 Leader: 2 Replicas: 2 Isr: 2
Topic: countly_event Partition: 6 Leader: 3 Replicas: 3 Isr: 3
Topic: countly_event Partition: 7 Leader: 4 Replicas: 4 Isr: 4
Topic: countly_event Partition: 8 Leader: 1 Replicas: 1 Isr: 1
Topic: countly_event Partition: 9 Leader: 2 Replicas: 2 Isr: 2
Topic: countly_event Partition: 10 Leader: 3 Replicas: 3 Isr: 3
Topic: countly_event Partition: 11 Leader: 4 Replicas: 4 Isr: 4

增加现有分区的复制因子很容易。只需在自定义重新分配json文件中指定额外副本,并将其与–execute选项一起使用即可增加指定分区的复制因子。

例如,以下示例将主题countly_event的分区0的复制因子从1增加到3.在增加复制因子之前,该分区的唯一副本存在于代理1上。作为增加复制因子的一部分,我们将添加更多副本经纪人2和3和4。

第一步是在json文件中手工制作自定义重新分配计划 - 这个自己定义每个分区设置副本,例如:0分区设置在3和2和4.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{"version":1,
"partitions":
[{"topic":"countly_event","partition":0,"replicas":[3,2,4]},
{"topic":"countly_event","partition":1,"replicas":[3,2,4]},
{"topic":"countly_event","partition":2,"replicas":[4,3,1]},
{"topic":"countly_event","partition":3,"replicas":[2,1,3]},
{"topic":"countly_event","partition":4,"replicas":[3,4,1]},
{"topic":"countly_event","partition":5,"replicas":[4,1,2]},
{"topic":"countly_event","partition":6,"replicas":[1,2,3]},
{"topic":"countly_event","partition":7,"replicas":[2,3,4]},
{"topic":"countly_event","partition":8,"replicas":[3,1,2]},
{"topic":"countly_event","partition":9,"replicas":[4,2,3]},
{"topic":"countly_event","partition":10,"replicas":[1,3,4]},
{"topic":"countly_event","partition":11,"replicas":[2,4,1]}]
}

然后,使用带有–execute选项的json文件启动重新分配过程

1
2
3
4
5
6
7
8
9
10
./bin/kafka-reassign-partitions.sh --zookeeper 10.155.90.153:2281,10.155.90.155:2281,10.155.90.138:2281 --reassignment-json-file increase-replication-factor.json --execute
[jollybi@kafka1 kafka_2.10-0.9.0.1]$ ./bin/kafka-reassign-partitions.sh --zookeeper 10.155.90.153:2281,10.155.90.155:2281,10.155.90.138:2281 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"countly_event","partition":6,"replicas":[4]},{"topic":"countly_event","partition":4,"replicas":[2]},{"topic":"countly_event","partition":5,"replicas":[3]},{"topic":"countly_event","partition":7,"replicas":[1]},{"topic":"countly_event","partition":3,"replicas":[1]},{"topic":"countly_event","partition":9,"replicas":[3]},{"topic":"countly_event","partition":11,"replicas":[1]},{"topic":"countly_event","partition":0,"replicas":[2]},{"topic":"countly_event","partition":10,"replicas":[4]},{"topic":"countly_event","partition":2,"replicas":[4]},{"topic":"countly_event","partition":1,"replicas":[3]},{"topic":"countly_event","partition":8,"replicas":[2]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"countly_event","partition":0,"replicas":[1,2,3,4]}]}

–verify选项可与该工具一起使用,以检查分区重新分配的状态。请注意,与–verify选项一起使用相同的increase-replication-factor.json(与–execute选项一起使用)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[jollybi@kafka1 kafka_2.10-0.9.0.1]$ ./bin/kafka-reassign-partitions.sh --zookeeper 10.155.90.153:2281,10.155.90.155:2281,10.155.90.138:2281 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [countly_event,0] is still in progress
Reassignment of partition [countly_event,1] is still in progress
Reassignment of partition [countly_event,2] is still in progress
Reassignment of partition [countly_event,3] is still in progress
Reassignment of partition [countly_event,4] is still in progress
Reassignment of partition [countly_event,5] is still in progress
Reassignment of partition [countly_event,6] is still in progress
Reassignment of partition [countly_event,7] is still in progress
Reassignment of partition [countly_event,8] is still in progress
Reassignment of partition [countly_event,9] is still in progress
Reassignment of partition [countly_event,10] is still in progress
Reassignment of partition [countly_event,11] is still in progress
您还可以使用kafka-topics工具验证复制因子的增加情况
[root@kafka1 kafka_2.10-0.9.0.1]# ./bin/kafka-topics.sh --zookeeper 10.155.90.153:2281,10.155.90.155:2281,10.155.90.138:2281 -describe -topic countly_event
Topic:countly_event PartitionCount:12 ReplicationFactor:3 Configs:
Topic: countly_event Partition: 0 Leader: 2 Replicas: 3,2,4 Isr: 2,3,4
Topic: countly_event Partition: 1 Leader: 3 Replicas: 3,2,4 Isr: 3,2,4
Topic: countly_event Partition: 2 Leader: 4 Replicas: 4,3,1 Isr: 4,1,3
Topic: countly_event Partition: 3 Leader: 1 Replicas: 2,1,3 Isr: 1,2,3
Topic: countly_event Partition: 4 Leader: 3 Replicas: 3,4,1 Isr: 4,1,3
Topic: countly_event Partition: 5 Leader: 4 Replicas: 4,1,2 Isr: 1,4,2
Topic: countly_event Partition: 6 Leader: 1 Replicas: 1,2,3 Isr: 2,1,3
Topic: countly_event Partition: 7 Leader: 2 Replicas: 2,3,4 Isr: 4,2,3
Topic: countly_event Partition: 8 Leader: 2 Replicas: 3,1,2 Isr: 2,1,3
Topic: countly_event Partition: 9 Leader: 3 Replicas: 4,2,3 Isr: 3,2,4
Topic: countly_event Partition: 10 Leader: 4 Replicas: 1,3,4 Isr: 4,1,3
Topic: countly_event Partition: 11 Leader: 1 Replicas: 2,4,1 Isr: 1,4,2

参考地址:http://kafka.apache.org/documentation/

Communicative learning:
🐧 Linux shell_ senior operation and maintenance faction: QQ group 459096184 circle (system operation and maintenance - application operation and maintenance - automation operation and maintenance - virtualization technology research, welcome to join)
🐧 BigData-Exchange School:QQ group 521621407 circles (big data Yun Wei) (Hadoop developer) (big data research enthusiasts) welcome to join

Bidata have internal WeChat exchange group, learn from each other, join QQ group has links.