RocketMQ 双主双从高可用部署

[TOC] # 目录 ## 准备工作序号 | IP地址 | 角色 | 备注 ---|---|---|--- 1 | 192.168.8.108 | 主节点 | rocketmq-nameserver1，rocketmq-master1 2 | 192.168.8.240 | 主节点 | rocketmq-nameserver2，rocketmq-master2 3 | 192.168.8.246 | 从节点 | rocketmq-nameserver3，rocketmq-master1-slave 4 | 192.168.8.247 | 从节点 | rocketmq-nameserver4，rocketmq-master2-slave > 108 和 246 一主一从 > 240 和 247 一主一从 ## Linux终端 ### Hosts vim /etc/hosts ```shell 192.168.8.108 rocketmq-nameserver1 192.168.8.108 rocketmq-master1 192.168.8.240 rocketmq-nameserver2 192.168.8.240 rocketmq-master2 192.168.8.246 rocketmq-nameserver3 192.168.8.246 rocketmq-master1-slave 192.168.8.247 rocketmq-nameserver4 192.168.8.247 rocketmq-master2-slave ``` > 四台服务器都需要配置以上的hosts ### 上传解压 ==++四个节点都需要同样操作++== ```shell # 创建文件夹 [root@240 rocketmq]# mkdir -p /opt/software/rocketmq [root@240 rocketmq]# ll total 221540 -rw-r--r-- 1 root root 195143852 May 6 10:27 jdk-8u241-linux-i586.tar.gz -rw-r--r-- 1 root root 11639497 Oct 31 2018 rocketmq-all-4.3.0-bin-release.zip # 解压 [root@240 rocketmq]# mkdir -p /usr/local/apache-rocketmq && unzip rocketmq-all-4.3.0-bin-release.zip && mv rocketmq-all-4.3.0-bin-release/* /usr/local/apache-rocketmq [root@108 rocketmq]# ll /usr/local/apache-rocketmq/ total 40 drwxr-xr-x 2 root root 83 Jul 26 2018 benchmark drwxr-xr-x 2 root root 4096 Jul 26 2018 bin drwxr-xr-x 5 root root 187 Jul 26 2018 conf drwxr-xr-x 2 root root 4096 Jul 26 2018 lib -rw-r--r-- 1 root root 17336 Jul 26 2018 LICENSE -rw------- 1 root root 84 Jul 26 2018 nohup.out -rw-r--r-- 1 root root 1337 Jul 26 2018 NOTICE -rw-r--r-- 1 root root 2426 Jul 26 2018 README.md # 创建软连接 [root@108 rocketmq]# cd /usr/local/ [root@108 local]# ln -s apache-rocketmq rocketmq ``` ### 创建存储目录 ==++四个节点都需要同样操作++== ```shell mkdir -p /usr/local/rocketmq/store/commitlog mkdir -p /usr/local/rocketmq/store/consumequeue mkdir -p /usr/local/rocketmq/store/index mkdir -p /usr/local/rocketmq/logs ``` 文件 | 说明 ---|--- /usr/local/rocketmq/store | 存储路径 /usr/local/rocketmq/store/commitlog | commitlog 存储路径 /usr/local/rocketmq/store/consumequeue | 消费队列存储路径存储路径 /usr/local/rocketmq/store/index | 消息索引存储路径 /usr/local/rocketmq/logs | 存放 rocketmq 日志 ### RocketMQ 配置文件 ==++四个节点都需要同样操作++== ```shell [root@108 local]# cd /usr/local/rocketmq/conf/ [root@108 conf]# ll total 32 drwxr-xr-x 2 root root 118 Jul 26 2018 2m-2s-async # 异步复制 drwxr-xr-x 2 root root 118 Jul 26 2018 2m-2s-sync # 同步双写（为了可靠性，生产环境推荐） drwxr-xr-x 2 root root 60 Jul 26 2018 2m-noslave # 双主模式 -rw-r--r-- 1 root root 949 Jul 26 2018 broker.conf -rw-r--r-- 1 root root 14978 Jul 26 2018 logback_broker.xml -rw-r--r-- 1 root root 3690 Jul 26 2018 logback_filtersrv.xml -rw-r--r-- 1 root root 3692 Jul 26 2018 logback_namesrv.xml -rw-r--r-- 1 root root 3761 Jul 26 2018 logback_tools.xml ``` #### 修改同步双写配置在 192.168.8.108 上操作备份配置文件 ```shell cp /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties.bak cp /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties.bak cp /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties.bak cp /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties.bak ``` vim /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties <details> <summary>点击查看</summary> ```properties #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-a #0 表示 Master，>0 表示 Slave brokerId=0 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/usr/local/rocketmq/store #commitLog 存储路径 storePathCommitLog=/usr/local/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/usr/local/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/usr/local/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/usr/local/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SYNC_MASTER #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 brokerIP1=192.168.8.108 brokerIP2=192.168.8.108 ``` </details> vim /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties <details> <summary>点击查看</summary> ```properties #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对 brokerName=broker-a #0 表示 Master，>0 表示 Slave brokerId=1 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/usr/local/rocketmq/store #commitLog 存储路径 storePathCommitLog=/usr/local/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/usr/local/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/usr/local/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/usr/local/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SLAVE #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 brokerIP1=192.168.8.246 ``` </details> vim /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties <details> <summary>点击查看</summary> ```properties #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-b #0 表示 Master，>0 表示 Slave brokerId=0 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/usr/local/rocketmq/store #commitLog 存储路径 storePathCommitLog=/usr/local/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/usr/local/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/usr/local/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/usr/local/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=ASYNC_MASTER #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 brokerIP1=192.168.8.240 brokerIP2=192.168.8.240 ``` </details> vim /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties <details> <summary>点击查看</summary> ```properties #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对 brokerName=broker-b #0 表示 Master，>0 表示 Slave brokerId=1 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/usr/local/rocketmq/store #commitLog 存储路径 storePathCommitLog=/usr/local/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/usr/local/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/usr/local/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/usr/local/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SLAVE #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 brokerIP1=192.168.8.247 ``` </details> #### 同步配置文件到240、246、247 通过 192.168.8.108 发送 *.properties 到240、246、247的/usr/local/rocketmq/conf/2m-2s-sync/下进行覆盖 ```shell cd /usr/local/rocketmq/conf/2m-2s-sync/ scp broker-a.properties broker-a-s.properties broker-b.properties broker-b-s.properties 192.168.8.240:/usr/local/rocketmq/conf/2m-2s-sync/ scp broker-a.properties broker-a-s.properties broker-b.properties broker-b-s.properties 192.168.8.246:/usr/local/rocketmq/conf/2m-2s-sync/ scp broker-a.properties broker-a-s.properties broker-b.properties broker-b-s.properties 192.168.8.247:/usr/local/rocketmq/conf/2m-2s-sync/ ``` #### 替换日志配置文件路径 108、240、246、247 都需要替换 ```shell cd /usr/local/rocketmq/conf && sed -i 's#${user.home}#/usr/local/rocketmq#g' *.xml ``` #### 修改JVM参数 ##### Broker 108、240、246、247 都需要修改 - cp /usr/local/rocketmq/bin/runbroker.sh /usr/local/rocketmq/bin/runbroker.sh.bak - vim /usr/local/rocketmq/bin/runbroker.sh ```shell JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m" JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=1g" ``` ##### Nserver - cp /usr/local/rocketmq/bin/runserver.sh /usr/local/rocketmq/bin/runserver.sh.bak - vim /usr/local/rocketmq/bin/runserver.sh ```shell JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m" ``` #### 启动 NameServer 108、240、246、247 都需要启动 ```shell cd /usr/local/rocketmq/bin && nohup sh mqnamesrv & ``` > 查看日志：tail -f -n 200 /usr/local/rocketmq/logs/rocketmqlogs/namesrv.log #### 启动 Broker ##### 192.168.8.108 ```shell cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & ``` ##### 192.168.8.240 ```shell cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & ``` ##### 192.168.8.246 ```shell cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & ``` ##### 192.168.8.247 ```shell cd /usr/local/rocketmq/bin && nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & ``` > jps 或 netstat -ntlp | grep 9876 查看是否启动 > tail -f -n 500 /usr/local/rocketmq/logs/rocketmqlogs/broker.log ### Web控制台 ```shell docker run -e "JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.8.108:9876;192.168.8.240:9876;192.168.8.246:9876;192.168.8.247:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false" -p 8080:8080 --name=eddie-rocketmq-cluster -t styletang/rocketmq-console-ng ``` 访问：ip:8080 ![图片.png](https://qiniuyun.eddilee.cn/%E5%9B%BE%E7%89%87_1626275317155.png?imageView2/0/q/75%7Cwatermark/2/text/YmxvZy5lZGRpbGVlLmNu/font/5a6L5L2T/fontsize/360/fill/IzAwMDAwMA==/dissolve/100/gravity/Center/dx/10/dy/10)