Junyangz's docs
  • Introduction
  • Ops
    • Linux-tips
    • MySQL-5.7.20
    • Upgrading MySQL
    • Upgrade OpenSSH to 7.7p1 in CentOS 6
    • Linux PERSISTENT NAMING
    • Use Kafka with Flume - CRS2
    • Setup Chroot SFTP in CentOS
    • Setup software RAID-5 in CentOS
    • SSH-port-forwarding
    • Elasticsearch In Production
    • ELK-simple-tutorial
    • Ansible Playbooks for Apache Kafka in production
    • GitHub Actions depoly Hexo
    • Test HTTP3/QUIC docker
    • Docker tutorial
    • SFTP-auth-pubkey
    • Linux Process Substitution
  • Note
    • Interview
      • interview-prepare
      • 2020-campus-recruiting
    • Android Tips
    • MacOS tips
    • Secret knowledge
    • GPG-Note
    • ud185
    • ud185-2
    • Introducing Tensorflow Federated
    • Tensorflow Federated
    • Expert Python Programing
    • What happens when zh_CN
    • TILGC
    • VScode keyboard shortcuts
    • Abseil Python
    • Latex Note
    • Git Cheatsheet
    • Study Smarter Not Harder
    • Machine Learning Interviews
    • 深度学习中的优化
    • Beej's Guide to Network Programming Note
      • ch4
      • ch5
      • ch6
      • ch7
  • [Share]
    • What to do after what to do
    • Truman is everywhere
    • Way2outer
    • 未来十五年
  • Quote
Powered by GitBook
On this page
  • 前言
  • Apache Kafka & Systemd
  • Requirements
  • Source tree
  • 理解和使用
  • Reference

Was this helpful?

  1. Ops

Ansible Playbooks for Apache Kafka in production

PreviousELK-simple-tutorialNextGitHub Actions depoly Hexo

Last updated 2 years ago

Was this helpful?

前言

参考了playbook批量安装部署Apache Kafka 2.2.0. 已在上开源

注:集群的zookeeper集群已通过Cloudera Manager安装了,所以不包括Zookeeper的安装部分

Apache Kafka & Systemd

Requirements

  • Ansible setup on your terminal

  • rhel7/CentOS7

  • Zookeeper cluster

  • Ansible playbook()

  • offline dist()

Source tree

Source tree

注:仓库里不含Kafka发行包kafka_2.12-2.2.0.tgz,需要自行下载并放置在上图的位置中

  • hosts.yml 里列出所有的主机角色,包括Zookeeper和Kafka broker

  • all.yml 里列出对位于broker分组中的主机执行task任务

理解和使用

运行

#ansible-playbook -C all.yml #运行之前可以先检查下,检查没问题后再运行
ansible-playbook -i hosts.yml all.yml

roles/kafka-broker/defaults/main.yml

配置安装过程中的所需变量即Zookeeper端口,Kafka broker配置,Systemd等相关内容

roles/kafka-broker/handlers/main.yml

配置完Systemd服务后执行daemon-reload并重启Kafka broker

roles/kafka-broker/tasks/main.yml

任务执行流,创建Kafka用户及用户组,创建Kafka数据目录,分发Kafka发行包,更改权限,拷贝broker配置文件并修改,创建systemd文件并启动Kafka

roles/kafka-broker/templates/

该文件夹下放置了broker的配置文件和Systemd文件,执行的时候将defaults设定的变量写入到文件中
默认配置文件参考了confluent Kafka的配置,请自行修改成自己所需的配置。

部署后可以使用 journalctl -xefu kafka检查Kafka的运行日志,部署目录/opt/apps/kafka/logs下也有相应的日志文件,此外还可以在Syslog中自行配置Identifier=kafka的日志处理。例如传输到各种监控程序中。

Reference

Ansible Playbooks for Confluent Platform
Running Kafka in Production
cp-ansible
Ansible playbooks for Kafka and Zookeeper
HowTo: Organize Ansible Playbook to install, uninstall, start and stop Kafka and Kafka Connect
confluentinc/cp-ansible
Github
repo
Apacher Kafka