Apache Kafka
SINGLE-NODE
CLUSTER CREATION
This
guide covers the installation of a Kafka single-node cluster only. Be aware that this setup is not compatible with Cloudera, Hortonworks or other vendors' clusters and it’s based on the official Apache Kafka documentation for standalone clusters.
See also: Kafka - How to create a multi-node Apache Kafka cluster
See also: Kafka - How to create a multi-node Apache Kafka cluster
1. Download
Kafka
a. Log in to your node via SSH, switch to root user and download
the Kafka tar file.
$ sudo -i
$ mkdir /tmp/kafka
$ cd /tmp/kafka
b. Extract the tar file, create the required
directories and copy the extracted files within, then apply specific
permissions on them:
$ tar -xzf
kafka_2.11-0.10.2.0.tgz
$ mkdir /usr/lib/kafka
$ chown -R root:root
/usr/lib/kafka
$ rsync -av
kafka_2.11-0.10.2.0/* /usr/lib/kafka/
Move to the /usr/lib/kafka directory and create a
symlink between the config subdirectory and /etc/kafka/conf
$ mkdir
/etc/kafka
$ cd
/etc/kafka
$ ln -s
/usr/lib/kafka/config /etc/kafka/conf
Create a directory for Kafka in /var/log and a
symlink between /usr/lib/kafka/logs and /var/log/kafka/logs:
$ mkdir
/var/log/kafka/
$
chmod -R 774 /var/log/kafka/
$ chown -R root:root
/var/log/kafka/
$ cd /var/log/kafka
$ ln
-s /usr/lib/kafka/logs /var/log/kafka/logs
2. Configure
Kafka
To properly configure a single-node Kafka cluster,
you need to know how many brokers need to be setup. Since each broker will
handle their own jobs, you should check how many brokers are required per your our your client's needs. In the below example, we will configure 3 broker instances.
Note that all the brokers will exist in the same node as we are configuring a
single-node Kafka cluster.
a. Look for the extra mounted disk (not covered in this guide) that will be used to
store Kafka data. In the below example, the disk is mounted in /DATA
$ df -h
Create the Kafka’s data and Zookeeper directories,
and apply permissions as below:
$ mkdir -p /DATA/kafka/data-1
$ mkdir -p /DATA/kafka/data-2
$ mkdir -p /DATA/kafka/data-3
$ chown -R root:root
/DATA/kafka
$ chmod -R 775 /DATA/kafka
$ mkdir -p
/DATA/kafka/zookeeper
$ chown -R root:root /DATA/kafka/zookeeper
$ chmod -R 775 /DATA/kafka/zookeeper
b. Create a backup of the original server.properties
file:
$ cp
/etc/kafka/conf/server.properties /etc/kafka/conf/server.properties.bkp
c. Create 3 server.properties files as below. These
files will be edited in the next step:
$ cp
/etc/kafka/conf/server.properties /etc/kafka/conf/server-1.properties
$ cp
/etc/kafka/conf/server.properties /etc/kafka/conf/server-2.properties
$ mv
/etc/kafka/conf/server.properties /etc/kafka/conf/server-3.properties
d. Edit the above files respectively and add/modify
each line as shown below. Make sure to replace the lines that already exist
instead of adding them twice:
$ vi /etc/kafka/conf/server-1.properties
broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/DATA/kafka/data-1
zookeeper.connect=<Fully_Qualified_Domain_Name>:2181
$ vi /etc/kafka/conf/server-2.properties
broker.id=2
listeners=PLAINTEXT://:9093
log.dirs=/DATA/kafka/data-2
zookeeper.connect=<Fully_Qualified_Domain_Name>:2181
$ vi /etc/kafka/conf/server-3.properties
broker.id=3
listeners=PLAINTEXT://:9094
log.dirs=/DATA/kafka/data-3
zookeeper.connect=<Fully_Qualified_Domain_Name>:2181
Make sure to replace <Fully_Qualified_Domain_Name>
by the actual FQDN of the Kafka server. It will be the same one on the 3 files
as we’re setting up a single-node cluster.
e. Create a backup of the original zookeeper.properties
file
$ cp /etc/kafka/conf/zookeeper.properties
/etc/kafka/conf/zookeeper.properties.bkp
f. Edit the file and add/modify each line as shown below. Make sure to replace the lines that already exist instead of adding them twice:
$ vi /etc/kafka/conf/zookeeper.properties
dataDir=/DATA/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=<Fully_Qualified_Domain_Name>:2888:3888
initLimit=5
syncLimit=2
g. Modify the permissions of the start and stop
scripts so that no one other than root can execute
them.
$ cd /usr/lib/kafka/bin
$ chmod 774 zookeeper-server-start.sh
zookeeper-server-stop.sh
$ chmod 774 kafka-server-start.sh
kafka-server-stop.sh
3. Start and
Test Kafka
To start Kafka, we first
need to start the embedded Zookeeper daemon, which will act as a coordinator
for the different Kafka brokers and their corresponding jobs.
Once the Zookeeper daemon is
started, we can proceed to start the Kafka servers (a.k.a. brokers) as per our
previous configuration.
·
To start
all the above-mentioned daemons in the background, we will add the
“-daemon” argument right before specifying the configuration files (where the
daemons get the required settings to start).
·
To start
the daemons in the foreground, no arguments must be passed to the
script, other than including the configuration files (as mentioned above).
Since we’re starting the
daemons for the first time, we will execute them in the foreground
(using several different terminals) to easily visualize any error that may
occur.
a. Open a terminal and connect to the Kafka node, then start the Zookeeper daemon in the foreground
as root. Make sure to pay attention to the command’s output in case of any error:
Terminal
1:
$ sudo /usr/lib/kafka/bin/zookeeper-server-start.sh
/etc/kafka/conf/zookeeper.properties
b. On a different terminal (connected to the
Kafka node as well), start the first Kafka broker (configured as per the
server-1.properties file) in the foreground as root:
Terminal
2:
$ sudo /usr/lib/kafka/bin/kafka-server-start.sh
/usr/lib/kafka/config/server-1.properties
Open two more terminals and repeat the same process for the rest of the
Kafka brokers.
Terminal
3:
$ sudo /usr/lib/kafka/bin/kafka-server-start.sh
/usr/lib/kafka/config/server-2.properties
Terminal 4:
$ sudo /usr/lib/kafka/bin/kafka-server-start.sh
/usr/lib/kafka/config/server-3.properties
c. We now have Zookeeper running, as well as 3
instances of Kafka broker.
The next step will be to create a test topic with a replication factor of
3 (consistent with the number of brokers previously created). Make sure to
replace <FQDN> by the actual FQDN of the Kafka node:
Terminal
5:
$ sudo
/usr/lib/kafka/bin/kafka-topics.sh --create --zookeeper <FQDN>:2181
--replication-factor 3 --partitions 1 --topic testtopic
You can validate the topic was created by running
the below script:
$
/usr/lib/kafka/bin/kafka-topics.sh --describe --zookeeper <FQDN>:2181
--topic testtopic
d. Once the topic is created and validated, we can test
the following scenarios:
·
Sending
messages from a Kafka producer
·
Visualizing
those messages in a Kafka consumer
·
Testing
failover between the 3 brokers (i.e. force-stopping brokers to validate that
the remaining ones keep on sending/receiving messages)
In the same terminal, execute a Kafka consumer
pointing to our testtopic, which will listen for incoming messages from
a Kafka producer:
Terminal
5:
$ /usr/lib/kafka/bin/kafka-console-consumer.sh
--bootstrap-server <FQDN>:9092, <FQDN>:9093, <FQDN>:9094
--topic testtopic --from-beginning
Open another terminal and execute a Kafka producer
pointing to the same topic:
Terminal
6:
$
/usr/lib/kafka/bin/kafka-console-producer.sh --broker-list <FQDN>:9092, <FQDN>:9093,
<FQDN>:9094 --topic testtopic
Once the producer is started, type any message you
want and press <ENTER> to submit it. You can send as much messages as you
want.
Go back to the Terminal 5 and make sure that
all messages are being received in the Kafka consumer console.
See thee below examples for clarification (disregard
the number of brokers showed after “--broker-list”):
Producer terminal (sending messages)
Consumer terminal (receiving messages)
e. Open yet another terminal to force-stop brokers and
test a failover scenario:
Terminal
7:
Look for the process associated with the server-1.properties
file and stop it.
$ ps -ef | grep
server-1.properties
(…)
$ kill -9 12831
Once stopped, you should still be able to send and
receive messages with the Kafka producer and consumer (if there’s at least one
broker up). Repeat the process described in step d to validate that you
can still send and receive messages.
f. Once you’ve validated that the failover scenario
works, stop all the running kafka processes, including Zookeeper. The below
script will stop all the processes associated with Kafka brokers:
$ sudo
/usr/lib/kafka/bin/kafka-server-stop.sh
Wait a few seconds and then stop the Zookeeper process:
$ sudo
/usr/lib/kafka/bin/zookeeper-server-stop.sh
Validate that no Kafka/Zookeeper daemons are still running:
$ ps -ef | grep kafka
g. In the next step, we will start Zookeeper and Kafka
as daemons so that they can run in the background and don’t depend on any open user session.
4. Start/Stop
Zookeeper and Kafka daemons
In this section, we will
finally start Zookeeper and Kafka as background daemons so that the cluster is
operational. We will do so using the root user, but you can always create and use a specific account for that purpose (if that's the case, make sure to grant sudo permissions accordingly in /etc/sudoers).
a. Start Zookeeper and Kafka brokers. Make sure to
start them in order as shown below:
$ sudo
/usr/lib/kafka/bin/zookeeper-server-start.sh -daemon
/usr/lib/kafka/config/zookeeper.properties
Wait a few seconds and then start the Kafka brokers:
$ sudo
/usr/lib/kafka/bin/kafka-server-start.sh -daemon /usr/lib/kafka/config/server-1.properties
$ sudo
/usr/lib/kafka/bin/kafka-server-start.sh -daemon /usr/lib/kafka/config/server-2.properties
$ sudo /usr/lib/kafka/bin/kafka-server-start.sh
-daemon /usr/lib/kafka/config/server-3.properties
Once all is started, you can repeat the steps described in 3.d to make sure that everything works as expected.
b. In case you need to, here’s how you can stop Kafka
brokers and then Zookeeper. Make sure to stop them in order as shown below:
$ sudo su - kafka
$ sudo
/usr/lib/kafka/bin/kafka-server-stop.sh
Wait a few seconds and then stop Zookeeper:
$ sudo
/usr/lib/kafka/bin/zookeeper-server-stop.sh
