Monitor Kafka with Prometheus +Grafana

Posted by Yancy on 2017-08-04

Monitoring Kafka with Prometheus

We’ve previously looked at how to monitor Cassandra with Prometheus. Let’s see the process for getting metrics from another popular Java application, Kafka.

we download Kafka, the JMX exporter and the config file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
tar -xzf kafka _ *。tgz
cd kafka_ *
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.6/jmx_prometheus_javaagent-0.6.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml
Download the good files and put them under the Kafka directory:
[hadoop@hadoop6 kafka_2.10-0.9.0.1]$ ll
drwxr-xr-x 3 hadoop hadoop 4096 9月 27 13:34 bin
drwxr-xr-x 2 hadoop hadoop 4096 9月 27 21:18 config
-rw-rw-r-- 1 hadoop hadoop 1225418 2月 5 2016 jmx_prometheus_javaagent-0.6.jar
-rw-rw-r-- 1 hadoop hadoop 2824 9月 26 17:48 kafka-0-8-2.yml
drwxr-xr-x 2 hadoop hadoop 4096 9月 27 13:31 libs
-rw-r--r-- 1 hadoop hadoop 11358 2月 12 2016 LICENSE
drwxrwxr-x 2 hadoop hadoop 266240 9月 27 21:19 logs
-rw-r--r-- 1 hadoop hadoop 162 2月 12 2016 NOTICE
drwxrwxr-x 5 hadoop hadoop 4096 9月 27 15:56 prometheus-1.2.1.linux-amd64
drwxr-xr-x 2 hadoop hadoop 4096 2月 12 2016 site-docs

We start a Zookeeper (a Kafka dependency) and Kafka with the JMX exporter running as a Java agent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
First step:
Configure the zookeeper zoo.zoo.cfg port: 2181
Start zookeeper:
./zookeeper-3.4.6/bin/zkServer.sh start
Second steps:
Configuration modification: kafka_2.10-0.9.0.1/config/zookeeper.properties
Port modification: 2182 do not conflict with the zookeeper service port
dataDir=/data2/zookeeper-3.4.6/zookeeper-data
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
./bin/zookeeper-server-start.sh config/zookeeper.properties &
KAFKA_OPTS="$KAFKA_OPTS -javaagent:$PWD/jmx_prometheus_javaagent-0.6.jar=7071:$PWD/kafka-0-8-2.yml" \
./bin/kafka-server-start.sh config/server.properties &

If you visit http://localhost:7071/metrics you’ll see the metrics.

Let’s setup a quick Prometheus server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
wget https://github.com/prometheus/prometheus/releases/download/v1.2.1/prometheus-1.2.1.linux-amd64.tar.gz
tar -xzf prometheus-*.tar.gz
cd prometheus-*
cat <<'EOF' > prometheus.yml
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: 'kafka'
static_configs:
- targets:
- localhost:7071
EOF
./prometheus

Following run access: http://localhost:9090/graph

1
2
3
4
5
6
7
8
9
[hadoop@hadoop8 prometheus-1.2.1.linux-amd64]$ ./prometheus
INFO[0000] Starting prometheus (version=1.2.1, branch=master, revision=dd66f2e94b2b662804b9aa1b6a50587b990ba8b7) source=main.go:75
INFO[0000] Build context (go=go1.7.1, user=root@fd9b0daff6bd, date=20161010-15:58:23) source=main.go:76
INFO[0000] Loading configuration file prometheus.yml source=main.go:247
INFO[0000] Loading series map and head chunks... source=storage.go:354
INFO[0000] 49 series loaded. source=storage.go:359
WARN[0000] No AlertManagers configured, not dispatching any alerts source=notifier.go:176
INFO[0000] Listening on :9090 source=web.go:240
INFO[0000] Starting target manager... source=targetmanager.go:76

The Prometheus platform monitors all data from the Kafka index:
This function is very powerful, late will explain what each representative means:

Install open source Grafana monitoring, combine prometheus.io to obtain Prometheus platform data.

Installing Grafana

1
2
3
4
5
6
7
# Download and unpack Grafana from binary tar (adjust version as appropriate).
curl -L -O https://grafanarel.s3.amazonaws.com/builds/grafana-2.5.0.linux-x64.tar.gz
tar zxf grafana-2.5.0.linux-x64.tar.gz
# Start Grafana.
cd grafana-2.5.0/
./bin/grafana-server web

Using

By default, Grafana will be listening on http://localhost:3000. The default login is “admin” / “admin”.

Creating a Prometheus data source

To create a Prometheus data source:

  1. Click on the Grafana logo to open the sidebar menu.
  2. Click on “Data Sources” in the sidebar.
  3. Click on “Add New”.
  4. Select “Prometheus” as the type.
  5. Set the appropriate Prometheus server URL (for example, http://localhost:9090/)
  6. Adjust other data source settings as desired (for example, turning the proxy access off).
  7. Click “Add” to save the new data source.
  8. The following shows an example data source configuration:

Reference address: GRAFANA SUPPORT FOR PROMETHEUS

Finally load the Kafka Overview dashboard from grafana.net into your Grafana to get the above console!

If you want to run Kafka inside docker, there’s another blog post covering that.

Reference address: monitoring-kafka-with-prometheus

Communicative learning:

🐧 Linux shell_ senior operation and maintenance faction: QQ group 459096184 circle (system operation and maintenance - application operation and maintenance - automation operation and maintenance - virtualization technology research, welcome to join)
🐧 BigData-Exchange School:QQ group 521621407 circles (big data Yun Wei) (Hadoop developer) (big data research enthusiasts) welcome to join

Bidata have internal WeChat exchange group, learn from each other, join QQ group has links.