kafka总是在启动一段时间后自动停止

Posted by Yancy on 2017-02-11

kafka总是在启动一段时间后自动停止

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
[2018-04-01 11:24:08,269] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions [namespace_jolly_brands_zy702_who_wms_order_user_info,2] (kafka.server.ReplicaFetcherManager)
[2018-04-01 11:24:08,269] INFO Truncating log namespace_jolly_brands_zy702_who_wms_order_user_info-2 to offset 69346. (kafka.log.Log)
[2018-04-01 11:24:08,292] INFO [Socket Server on Broker 2], Shutdown completed (kafka.network.SocketServer)
[2018-04-01 11:24:08,294] INFO [Kafka Request Handler on Broker 2], shutting down (kafka.server.KafkaRequestHandlerPool)
[2018-04-01 11:24:08,353] INFO [Kafka Request Handler on Broker 2], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2018-04-01 11:24:08,368] INFO [ThrottledRequestReaper-Produce], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-04-01 11:24:08,856] INFO [ThrottledRequestReaper-Produce], Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-04-01 11:24:08,857] INFO [ThrottledRequestReaper-Produce], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-04-01 11:24:08,857] INFO [ThrottledRequestReaper-Fetch], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-04-01 11:24:09,847] INFO [ThrottledRequestReaper-Fetch], Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-04-01 11:24:09,847] INFO [ThrottledRequestReaper-Fetch], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-04-01 11:24:09,848] INFO [KafkaApi-2] Shutdown complete. (kafka.server.KafkaApis)
[2018-04-01 11:24:09,852] INFO [Replica Manager on Broker 2]: Shutting down (kafka.server.ReplicaManager)
[2018-04-01 11:24:09,855] INFO [ReplicaFetcherManager on broker 2] shutting down (kafka.server.ReplicaFetcherManager)
[2018-04-01 11:24:09,856] INFO [ReplicaFetcherThread-0-4], Shutting down (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:09,877] INFO [ReplicaFetcherThread-0-4], Stopped (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:09,877] INFO [ReplicaFetcherThread-0-4], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:09,879] INFO [ReplicaFetcherThread-0-1], Shutting down (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:09,901] INFO [ReplicaFetcherThread-0-1], Stopped (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:09,901] INFO [ReplicaFetcherThread-0-1], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:09,903] INFO [ReplicaFetcherThread-0-3], Shutting down (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:10,194] INFO [ReplicaFetcherThread-0-3], Stopped (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:10,194] INFO [ReplicaFetcherThread-0-3], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2018-04-01 11:24:10,196] INFO [ReplicaFetcherManager on broker 2] shutdown completed (kafka.server.ReplicaFetcherManager)
[2018-04-01 11:24:10,196] INFO [ExpirationReaper-2], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:10,375] INFO [ExpirationReaper-2], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:10,375] INFO [ExpirationReaper-2], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:10,375] INFO [ExpirationReaper-2], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:10,515] INFO [ExpirationReaper-2], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:10,516] INFO [ExpirationReaper-2], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:10,523] INFO [Replica Manager on Broker 2]: Shut down completely (kafka.server.ReplicaManager)
[2018-04-01 11:24:10,529] INFO Shutting down. (kafka.log.LogManager)
[2018-04-01 11:24:27,288] INFO Shutdown complete. (kafka.log.LogManager)
[2018-04-01 11:24:27,289] INFO [GroupCoordinator 2]: Shutting down. (kafka.coordinator.GroupCoordinator)
[2018-04-01 11:24:27,290] INFO [ExpirationReaper-2], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:27,409] INFO [ExpirationReaper-2], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:27,409] INFO [ExpirationReaper-2], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:27,410] INFO [ExpirationReaper-2], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:27,563] INFO [ExpirationReaper-2], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:27,563] INFO [ExpirationReaper-2], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-04-01 11:24:27,565] INFO [GroupCoordinator 2]: Shutdown complete. (kafka.coordinator.GroupCoordinator)
[2018-04-01 11:24:27,640] INFO [Kafka Server 2], shut down completed (kafka.server.KafkaServer)

一开始我以为是 centos 的 OOM Killer 关了它,但是当我改了 oom_score 之后依旧不行。而且在/val/log/message 里面没有找的相关操作的 log。
我现在应该做些什么能修复他,或者进一步的确定错误原因?

经过查看 kafka 的启动脚本,上周尝试使用

1
bin/kafka-server-start.sh -daemon ./config/server.properties

进行启动,到现在为止 kafka 还在正常运行。
和不加 -daemon 区别在于:

1
2
3
4
5
6
7
8
9
10
bin/kafka-run-class.sh
# Launch mode
if [ "x$DAEMON_MODE" = "xtrue" ]; then
#加 daemon 会使用该命令
nohup $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" > "$CONSOLE_OUTPUT_FILE" 2>&1 < /dev/null &
else
#不加时使用的命令
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@"
fi

机器配置?集群情况?啥都没有…只有日志…
zookeeper和broker都需要一定资源,这俩放在一起也会降低稳定性…

1
su - jollybi -c "/data/tools/kafka_2.10-0.9.0.1/bin/kafka-server-start.sh -daemon /data/tools/kafka_2.10-0.9.0.1/config/server.properties 2>&1 > /dev/null"