__consumer_offsets

__consumer_offsets 是 kafka 自行创建的，和普通的 topic 相同。它存在的目的之一就是保存 consumer 提交的位移。

kafka 默认为该 topic 创建了50个分区，并且对每个 group.id 做哈希求模运算，从而将负载分散到不同的 __consumer_offsets 分区上。

如果我们没有在server.properties文件中指定topic分区的副本数的话，它的默认值就是1，若对应分区的broker宕机将影响所有的消费者。所以我们要增加这个topic的副本数，有两个方式，一个是添加配置重启kafka服务，还有一个是手动新增该topic的副本。

新增配置方式，需重启服务

在开发和测试环境，我们打算使用修改配置的方式来新增__consumer_offsets的副本数。

在kafka配置文件server.properties里新增如下配置：

offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3

若是容器环境，比如运行在kubernetes环境下，则在Kafka的statefulset添加如下环境变量

        - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
          value: "3"
        - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
          value: "3"
        - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
          value: "3"

添加完成之后，kafka pod重启，此时__consumer_offsets已经存在，需要删掉，然后自动重新生成的就是3副本了。

手动扩容副本，不需要重启服务

可以使用 kafka-reassign-partitions.sh 生成分区分配方案

1.创建 topic-list.json (可以多个topic）

{
    "topics":[
        {
            "topic":"__consumer_offsets"
        }
    ],
    "version":1
}

2.生成分区分配方案

（k8s环境kafka操作方式，到kafka对应节点上执行df -h|grep csi或者阿里云alicloud或者其他类型，可以找到对应挂载块文件的host路径，然后手动上传文件到该路径下，再通过kubectl exec进入pod内容器执行命令）

./kafka-reassign-partitions.sh --zookeeper kafkazk.common:2181 --topics-to-move-json-file /kafka/topic-list.json --broker-list "100,101,102" --generate > /kafka/topic.json

3.获取到的topic.json文件需要修改下

原始文件

Current partition replica assignment
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":19,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[102,100,101],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":19,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[100,101,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[102,101,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[102,100,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[100,102,101],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[101,102,100],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[101,100,102],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[102,100,101],"log_dirs":["any","any","any"]}]}

只保留 Proposed partition reassignment configuration 部分json文件，然后执行重新分区方案，若是觉得这个文件丑，可以网上找个工具美化下json，如下(注意中间省略了很多）：

{
    "version": 1,
    "partitions": [
      {
        "topic": "__consumer_offsets",
        "partition": 19,
        "replicas": [
          101,
          102,
          100
        ],
        "log_dirs": [
          "any",
          "any",
          "any"
        ]
      },
      {
        "topic": "__consumer_offsets",
        "partition": 30,
        "replicas": [
          100,
          101,
          102
        ],
        "log_dirs": [
          "any",
          "any",
          "any"
        ]
      },
    ......
      {
        "topic": "__consumer_offsets",
        "partition": 2,
        "replicas": [
          102,
          100,
          101
        ],
        "log_dirs": [
          "any",
          "any",
          "any"
        ]
      }
    ]
  }

# 执行重新分区方案
./kafka-reassign-partitions.sh --zookeeper kafkazk.common:2181 --reassignment-json-file /kafka/topic.json --execute

4.查看进度

./kafka-reassign-partitions.sh --zookeeper kafkazk.common:2181 --reassignment-json-file /kafka/topic.json --verify

Kafka Manager 上查看topic出现 Brokers Leader Skew %不为0，怎么办

有时候我们会在Kafka Manager 上查看topic，出现了Brokers Leader Skew %不为0，之前基本是通过重启对应的broker节点解决

之前的处理方法是业务低峰期（大概率得半夜爬起来了）通过重启对应的broker节点解决，

除了重启节点，还有哪个办法可以解决呢，我们也可以通过kafka-reassign-partitions.sh来重新分配下分区解决，这样就不需要通过重启节点来解决了。