RocketMQ 双主双从高可用部署 | Eddie'Blog
RocketMQ 双主双从高可用部署

RocketMQ 双主双从高可用部署

eddie 399 2021-07-14

目录

准备工作

序号IP地址角色备注
1192.168.8.108主节点rocketmq-nameserver1,rocketmq-master1
2192.168.8.240主节点rocketmq-nameserver2,rocketmq-master2
3192.168.8.246从节点rocketmq-nameserver3,rocketmq-master1-slave
4192.168.8.247从节点rocketmq-nameserver4,rocketmq-master2-slave

108 和 246 一主一从

240 和 247 一主一从

Linux终端

Hosts

vim /etc/hosts

192.168.8.108 rocketmq-nameserver1
192.168.8.108 rocketmq-master1

192.168.8.240 rocketmq-nameserver2
192.168.8.240 rocketmq-master2

192.168.8.246 rocketmq-nameserver3
192.168.8.246 rocketmq-master1-slave

192.168.8.247 rocketmq-nameserver4
192.168.8.247 rocketmq-master2-slave

四台服务器都需要配置以上的hosts

上传解压

==四个节点都需要同样操作==

# 创建文件夹
[root@240 rocketmq]# mkdir -p /opt/software/rocketmq

[root@240 rocketmq]# ll
total 221540
-rw-r--r-- 1 root root 195143852 May  6 10:27 jdk-8u241-linux-i586.tar.gz
-rw-r--r-- 1 root root  11639497 Oct 31  2018 rocketmq-all-4.3.0-bin-release.zip

# 解压
[root@240 rocketmq]# mkdir -p /usr/local/apache-rocketmq && unzip rocketmq-all-4.3.0-bin-release.zip && mv rocketmq-all-4.3.0-bin-release/* /usr/local/apache-rocketmq
[root@108 rocketmq]# ll /usr/local/apache-rocketmq/
total 40
drwxr-xr-x 2 root root    83 Jul 26  2018 benchmark
drwxr-xr-x 2 root root  4096 Jul 26  2018 bin
drwxr-xr-x 5 root root   187 Jul 26  2018 conf
drwxr-xr-x 2 root root  4096 Jul 26  2018 lib
-rw-r--r-- 1 root root 17336 Jul 26  2018 LICENSE
-rw------- 1 root root    84 Jul 26  2018 nohup.out
-rw-r--r-- 1 root root  1337 Jul 26  2018 NOTICE
-rw-r--r-- 1 root root  2426 Jul 26  2018 README.md

# 创建软连接
[root@108 rocketmq]# cd /usr/local/
[root@108 local]# ln -s apache-rocketmq rocketmq

创建存储目录

==四个节点都需要同样操作==

mkdir -p /usr/local/rocketmq/store/commitlog
mkdir -p /usr/local/rocketmq/store/consumequeue
mkdir -p /usr/local/rocketmq/store/index
mkdir -p /usr/local/rocketmq/logs
文件说明
/usr/local/rocketmq/store存储路径
/usr/local/rocketmq/store/commitlogcommitlog 存储路径
/usr/local/rocketmq/store/consumequeue消费队列存储路径存储路径
/usr/local/rocketmq/store/index消息索引存储路径
/usr/local/rocketmq/logs存放 rocketmq 日志

RocketMQ 配置文件

==四个节点都需要同样操作==

[root@108 local]# cd /usr/local/rocketmq/conf/
[root@108 conf]# ll
total 32
drwxr-xr-x 2 root root   118 Jul 26  2018 2m-2s-async   # 异步复制
drwxr-xr-x 2 root root   118 Jul 26  2018 2m-2s-sync    # 同步双写 (为了可靠性,生产环境推荐)
drwxr-xr-x 2 root root    60 Jul 26  2018 2m-noslave    # 双主模式
-rw-r--r-- 1 root root   949 Jul 26  2018 broker.conf
-rw-r--r-- 1 root root 14978 Jul 26  2018 logback_broker.xml
-rw-r--r-- 1 root root  3690 Jul 26  2018 logback_filtersrv.xml
-rw-r--r-- 1 root root  3692 Jul 26  2018 logback_namesrv.xml
-rw-r--r-- 1 root root  3761 Jul 26  2018 logback_tools.xml

修改同步双写配置

在 192.168.8.108 上操作

备份配置文件

cp /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties.bak
cp /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties.bak

cp /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties.bak 
cp /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties.bak

vim /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties

点击查看
#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字,注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master,>0 表示 Slave
brokerId=0
#nameServer 地址,分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时,自动创建服务器不存在的 topic,默认创建的队列数
defaultTopicQueueNums=4

#是否允许 Broker 自动创建 Topic,建议线下开启,线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组,建议线下开启,线上关闭
autoCreateSubscriptionGroup=true

#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点,默认凌晨 4 点
deleteWhen=04
#文件保留时间,默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/usr/local/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/usr/local/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/usr/local/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/usr/local/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false

#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

brokerIP1=192.168.8.108
brokerIP2=192.168.8.108

vim /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties

点击查看
#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字,注意此处不同的配置文件填写的不一样,与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master,>0 表示 Slave
brokerId=1
#nameServer 地址,分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时,自动创建服务器不存在的 topic,默认创建的队列数
defaultTopicQueueNums=4

#是否允许 Broker 自动创建 Topic,建议线下开启,线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组,建议线下开启,线上关闭
autoCreateSubscriptionGroup=true

#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点,默认凌晨 4 点
deleteWhen=04
#文件保留时间,默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/usr/local/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/usr/local/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/usr/local/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/usr/local/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

brokerIP1=192.168.8.246

vim /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties

点击查看
#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字,注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master,>0 表示 Slave
brokerId=0
#nameServer 地址,分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时,自动创建服务器不存在的 topic,默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic,建议线下开启,线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组,建议线下开启,线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点,默认凌晨 4 点
deleteWhen=04
#文件保留时间,默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/usr/local/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/usr/local/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/usr/local/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/usr/local/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

brokerIP1=192.168.8.240
brokerIP2=192.168.8.240

vim /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties

点击查看
#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字,注意此处不同的配置文件填写的不一样,与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master,>0 表示 Slave
brokerId=1
#nameServer 地址,分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时,自动创建服务器不存在的 topic,默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic,建议线下开启,线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组,建议线下开启,线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点,默认凌晨 4 点
deleteWhen=04
#文件保留时间,默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/usr/local/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/usr/local/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/usr/local/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/usr/local/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

brokerIP1=192.168.8.247

同步配置文件到240、246、247

通过 192.168.8.108 发送 *.properties 到240、246、247的/usr/local/rocketmq/conf/2m-2s-sync/下进行覆盖

cd /usr/local/rocketmq/conf/2m-2s-sync/
scp broker-a.properties broker-a-s.properties broker-b.properties broker-b-s.properties 192.168.8.240:/usr/local/rocketmq/conf/2m-2s-sync/
scp broker-a.properties broker-a-s.properties broker-b.properties broker-b-s.properties 192.168.8.246:/usr/local/rocketmq/conf/2m-2s-sync/
scp broker-a.properties broker-a-s.properties broker-b.properties broker-b-s.properties 192.168.8.247:/usr/local/rocketmq/conf/2m-2s-sync/

替换日志配置文件路径

108、240、246、247 都需要替换

cd /usr/local/rocketmq/conf && sed -i 's#${user.home}#/usr/local/rocketmq#g' *.xml

修改JVM参数

Broker

108、240、246、247 都需要修改

  • cp /usr/local/rocketmq/bin/runbroker.sh /usr/local/rocketmq/bin/runbroker.sh.bak
  • vim /usr/local/rocketmq/bin/runbroker.sh
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"
JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=1g"
Nserver
  • cp /usr/local/rocketmq/bin/runserver.sh /usr/local/rocketmq/bin/runserver.sh.bak
  • vim /usr/local/rocketmq/bin/runserver.sh
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

108、240、246、247 都需要启动

cd /usr/local/rocketmq/bin && nohup sh mqnamesrv &

查看日志:tail -f -n 200 /usr/local/rocketmq/logs/rocketmqlogs/namesrv.log

启动 Broker

192.168.8.108
cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 &
192.168.8.240
cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 &
192.168.8.246
cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 &
192.168.8.247
cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 &

jps 或 netstat -ntlp | grep 9876 查看是否启动

tail -f -n 500 /usr/local/rocketmq/logs/rocketmqlogs/broker.log

Web控制台

docker run -e "JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.8.108:9876;192.168.8.240:9876;192.168.8.246:9876;192.168.8.247:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false" -p 8080:8080 --name=eddie-rocketmq-cluster -t styletang/rocketmq-console-ng

访问:ip:8080

图片.png