|
Setting Up ZooKeeper, Kafka, and Schema Registry
|
NOTE |
Windows is currently not a supported platform for running Confluent Kafka, ensure that your OS is on the list of supported operating systems: https://docs.confluent.io/4.0.0/installation/installing_cp.html#system-requirements
|
Before proceeding, you must install and setup the following prerequisites:
q Java JDK 64-bit, version 1.7 or later
q System Environment variable JAVA_HOME set to the Java JDK 64-bit
Steps:
1. Download one of the Confluent Kafka archives from http://confluent.io/download.
2. Extract the contents of the archive to a new location.
3. Below are the top-level folders of the archive:
confluent-3.1.1/bin/ # Driver scripts for starting/stopping services
confluent-3.1.1/etc/ # Configuration files
confluent-3.1.1/share/java/ # Jars
4. Start the ZooKeeper, Kafka and Schema Registry processes in the correct order. Make sure the previous process has been started before continuing to the next one.
· Start ZooKeeper
$ ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
· Start the Kafka broker
$ ./bin/kafka-server-start ./etc/kafka/server.properties
· Start Schema Registry
$ ./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
When these three processes have been started, you can now connect Panopticon Streams to your local Kafka cluster to execute and deploy your applications.
For more details, refer to the Confluent Kafka Installation-and Quick Start guides:
· https://docs.confluent.io/3.1.1/installation.html
· https://docs.confluent.io/3.1.1/quickstart.html
|
NOTE |
When connecting to a Kafka broker on a separate machine, exposing different IP addresses internally and externally, you need to configure KAFKA_ADVERTISED_LISTENERS. This is typically the case when running Kafka in a Docker container. The symptoms of the missing configuration are: · Panopticon Streams can connect to ZooKeeper and the Kafka Broker · No data is written to topics In [Kafka]/etc/kafka/server.properties, uncomment advertised.listeners and replace “your.host.name” with the externally exposed host name or IP address. # Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value # returned from java.net.InetAddress.getCanonicalHostName(). advertised.listeners=PLAINTEXT://your.host.name:9092 When using the Confluent Docker image, you can pass the KAFKA_ADVERTISED_LISTENERS as a parameter: docker run -d --restart=always \ --net=confluent \ --name=kafka \ -p 9092:9092 \ -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://your.host.name:9092 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ confluentinc/cp-kafka:5.1.0
|