kafka参数调优
Producer: 
acks: all
buffer.memory: 536870912
compression.type :snappy
retries: 100
max.in.flight.requests.per.connection = 1

batch.size: 10240字节 不是条数
max.request.size = 2097152
request.timeout.ms = 360000 大于 replica.lag.time.max.ms
metadata.fetch.timeout.ms= 360000
timeout.ms = 360000

linger.ms 5s (生产不用)
max.block.ms 1800000

Broker: CDH
message.max.bytes 2560KB 1条消息的大小
zookeeper.session.timeout.ms 180000
replica.fetch.max.bytes 5M 大于message.max.bytes
num.replica.fetchers 6
replica.lag.max.messages 6000
replica.lag.time.max.ms 15000

log.flush.interval.messages 10000
log.flush.interval.ms 5s

Consumer:
https://issues.apache.org/jira/browse/SPARK-22968
"max.partition.fetch.bytes" -> (5242880: java.lang.Integer) //default: 1048576
"request.timeout.ms" -> (90000: java.lang.Integer) //default: 60000
"session.timeout.ms" -> (60000: java.lang.Integer) //default: 30000
"heartbeat.interval.ms" -> (5000: java.lang.Integer)
"receive.buffer.bytes" -> (10485760: java.lang.Integer)
Author: Tunan
Link: http://yerias.github.io/2020/04/27/kafka/7/
Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless stating additionally.