【王老师说运维】：Centos7 k8s 开发单服务器部署rocketmq

1736 05-20

这几天对开发服务器中的 k8s和相关模块进行升级，做个记录

环境，与系统
系统：centos 7.6

内核： 5.1.9-1.el7.elrepo.x86_64

Kubernetes(K8s) 1.14.3

服务器IP：192.168.0.254

服务器IP 说明
如果更改了服务器IP 为其他IP，那么请全局全部替换成你改的IP
fox.风

如何安装 k8s
请看最近写的一篇文章，

https://blog.csdn.net/fenglailea/article/details/88745642
常用 k8s 命令
https://foxwho.blog.csdn.net/article/details/92137467

服务器资源限制
对CPU,内存都做了限制，请自行针对修改

没有使用持久化存储，这里直接使用宿主机存储，如有需要，请自行增加

目录
请把项目放置到 /www/k8s/foxdev/ 目录下，所有都是基于此目录
或者请看下面一节的拉取代码

拉取代码

mkdir -p /www/k8s/
cd /www/k8s/

git clone https://github.com/foxiswho/k8s-nacos-sentinel-rocketmq-zipkin-elasticsearch-redis-mysql.git foxdev

cd foxdev

以下是独立部署相关模块

k8s rocketmq 独立部署

rocketmq 采用 server:1主,Broker:1主1从模式

带有 rocketmq-console web 简单管理界面

查看 service

kubectl get svc

输出

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                           AGE
rmqbroker         NodePort    10.1.183.152   <none>        10909:10909/TCP,10911:10911/TCP,10912:10912/TCP   3h59m
rmqbroker-slave   NodePort    10.1.128.180   <none>        10919:10919/TCP,10921:10921/TCP,10922:10922/TCP   3h7m
rmqui             NodePort    10.1.95.26     <none>        8080:8180/TCP                                     3h7m
rocketmq          NodePort    10.1.128.90    <none>        9876:9876/TCP                                     3h7m

在开发服务器上部署成功，有问题的，可以联系我

资源编排文件
后缀为-rc.yml 是 StatefulSet/Pod 文件
后缀为-service.yml 是 service 文件

开头rocketmq 是 rocketmq server
开头broker 是 rocketmq broker
开头broker-s 是 rocketmq broker slave ,其他的是 master

开头ui 是 rocketmq console 简单页面管理

rocketmq server 资源编排
rocketmq-rc.yml

# 版本
apiVersion: apps/v1
# 资源对象
kind: StatefulSet
metadata:
  #名称
  name: rocketmq
  #标签
  labels:
  #标签名和标签值
    app: rocketmq
    #命名空间
#  namespace: default
spec:
  #服务名称
  serviceName: rocketmq
  replicas: 1
  selector:
  #匹配标签
    matchLabels:
    #标签名和值
      app: rocketmq
  template:
    metadata:
      labels:
        app: rocketmq
    spec:
      containers:
      #容器名称
        - name: rocketmq
          # 镜像
          image: foxiswho/rocketmq:server
          #Always：不管镜像是否存在都会进行一次拉取
          imagePullPolicy: Always
          #资源限制
          resources:
            limits:
              cpu: 250m
              memory: 1000Mi
            requests:
              cpu: 100m
              memory: 768Mi
           #环境变量
          env:
           #设置时区
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT
              value: -Duser.home=/opt
            - name: JAVA_OPT_EXT
              value: -server -Xms768m -Xmx768m -Xmn256m
          ports:
            # 容器端口
            - containerPort: 9876
          # 加载外部券路径等
          volumeMounts:
            - name: vol-logs
              mountPath: /opt/logs
            - name: vol-init-d
              mountPath: /opt/store
      volumes:
        - name: vol-logs
          hostPath:
            path: /www/k8s/foxdev/rocketmq/logs
            type: Directory
        - name: vol-init-d
          hostPath:
            path: /www/k8s/foxdev/rocketmq/store
            type: Directory

rocketmq-service.yml

apiVersion: v1
kind: Service
metadata:
  #  namespace: default
  name: rocketmq
  labels:
    app: rocketmq
spec:
 #nodePort 是提供给集群外部客户访问service的入口
  type: NodePort
  ports:
    - port: 9876
      targetPort: 9876
      # 外部访问端口
      nodePort: 30876
  selector:
    app: rocketmq

rocketmq broker master 资源编排

broker-rc.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rmqbroker
  labels:
    app: rmqbroker
#  namespace: default
spec:
  serviceName: rmqbroker
  replicas: 1
  selector:
    matchLabels:
      app: rmqbroker
  template:
    metadata:
      labels:
        app: rmqbroker
    spec:
      containers:
        - name: rmqbroker
          image: foxiswho/rocketmq:broker
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 250m
              memory: 1000Mi
            requests:
              cpu: 100m
              memory: 512Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: NAMESRV_ADDR
              value: rocketmq:9876
            - name: JAVA_OPT_EXT
              value:  -Duser.home=/opt -server -Xms512m -Xmx512m -Xmn128m
          command: ["mqbroker","-c","/etc/rocketmq/broker.conf","-n","rocketmq:9876","autoCreateTopicEnable=true"]
          ports:
            - containerPort: 10909
            - containerPort: 10911
            - containerPort: 10912
          volumeMounts:
            - name: vol-logs
              mountPath: /opt/logs
            - name: vol-store
              mountPath: /opt/store
            - name: vol-broker
              mountPath: /etc/rocketmq/broker.conf
      volumes:
        - name: vol-logs
          hostPath:
            path: /www/k8s/foxdev/rocketmq/logs-b
            type: Directory
        - name: vol-store
          hostPath:
              path: /www/k8s/foxdev/rocketmq/store-b
              type: Directory
        - name: vol-broker
          hostPath:
            path: /www/k8s/foxdev/rocketmq/conf/broker.conf
            type: File

broker-service.yml

apiVersion: v1
kind: Service
metadata:
  #  namespace: default
  name: rmqbroker
  labels:
    app: rmqbroker
spec:
  #type: NodePort
  ports:
    - port: 10909
      targetPort: 10909
      #nodePort: 10909
      name: slave-master
    - port: 10911
      targetPort: 10911
      #nodePort: 10911
      name: listen
    - port: 10912
      targetPort: 10912
      #nodePort: 10912
      name: ha-service
  selector:
    app: rmqbroker

rocketmq broker slave 资源编排

broker-s-rc.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rmqbroker-slave
  labels:
    app: rmqbroker-slave
#  namespace: default
spec:
  serviceName: rmqbroker-slave
  replicas: 1
  selector:
    matchLabels:
      app: rmqbroker-slave
  template:
    metadata:
      labels:
        app: rmqbroker-slave
    spec:
      containers:
        - name: rmqbroker-slave
          image: foxiswho/rocketmq:broker
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 250m
              memory: 1000Mi
            requests:
              cpu: 100m
              memory: 512Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: NAMESRV_ADDR
              value: rocketmq:9876
            - name: JAVA_OPT_EXT
              value:  -Duser.home=/opt -server -Xms512m -Xmx512m -Xmn128m
          command: ["mqbroker","-c","/etc/rocketmq/broker-s.conf","-n","rocketmq:9876","autoCreateTopicEnable=true"]
          ports:
            - containerPort: 10919
            - containerPort: 10921
            - containerPort: 10922
          volumeMounts:
            - name: vol-logs
              mountPath: /opt/logs
            - name: vol-store
              mountPath: /opt/store
            - name: vol-broker-s
              mountPath: /etc/rocketmq/broker-s.conf
      volumes:
        - name: vol-logs
          hostPath:
            path: /www/k8s/foxdev/rocketmq/logs-b-s
            type: Directory
        - name: vol-store
          hostPath:
              path: /www/k8s/foxdev/rocketmq/store-b-s
              type: Directory
        - name: vol-broker-s
          hostPath:
              path: /www/k8s/foxdev/rocketmq/conf/broker-s.conf
              type: File

broker-s-service.yml

apiVersion: v1
kind: Service
metadata:
  #  namespace: default
  name: rmqbroker-slave
  labels:
    app: rmqbroker-slave
spec:
  #type: NodePort
  ports:
    - port: 10919
      targetPort: 10919
      #nodePort: 10919
      name: slave-master
    - port: 10921
      targetPort: 10921
      #nodePort: 10921
      name: broker-listen
    - port: 10922
      targetPort: 10922
      #nodePort: 10922
      name: ha-service
  selector:
    app: rmqbroker-slave

rocketmq ui console 资源编排

ui-rc.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rmqui
  labels:
    app: rmqui
#  namespace: default
spec:
  serviceName: rmqui
  replicas: 1
  selector:
    matchLabels:
      app: rmqui
  template:
    metadata:
      labels:
        app: rmqui
    spec:
      containers:
        - name: rmqui
          image: styletang/rocketmq-console-ng
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 450m
              memory: 1768Mi
            requests:
              cpu: 100m
              memory: 768Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPTS
              value: -Drocketmq.namesrv.addr=rocketmq:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false -Duser.home=/root  -Xms768m -Xmx768m
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: vol-logs
              mountPath: /root/logs
      volumes:
        - name: vol-logs
          hostPath:
            path: /www/k8s/foxdev/rocketmq/logs-ui
            type: Directory

ui-service.yml

apiVersion: v1
kind: Service
metadata:
  #  namespace: default
  name: rmqui
  labels:
    app: rmqui
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30879
  selector:
    app: rmqui

broker 配置文件

两个文件只有以下配置不一样，其他都是一样的

#0 表示Master，>0 表示Slave
brokerId=1
#Broker 的角色
#- ASYNC_MASTER 异步复制Master
#- SYNC_MASTER 同步双写Master
#- SLAVE
brokerRole=SLAVE
#Broker 对外服务的监听端口
listenPort=10921

broker master 配置文件

broker.conf

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.


#所属集群名字
brokerClusterName=DefaultCluster

#broker名字，注意此处不同的配置文件填写的不一样，如果在broker-a.properties使用:broker-a,
#在broker-b.properties使用:broker-b
brokerName=broker-a

#0 表示Master，>0 表示Slave
brokerId=0

#nameServer地址，分号分割
#namesrvAddr=rocketmq-0:9876;rocketmq-1:9876
namesrvAddr=rocketmq:9876

#启动IP,如果 docker 报 com.alibaba.rocketmq.remoting.exception.RemotingConnectException: connect to <192.168.0.120:10909> failed
# 解决方式1 加上一句producer.setVipChannelEnabled(false);，解决方式2 brokerIP1 设置宿主机IP，不要使用docker 内部IP
brokerIP1=192.168.0.254

#在发送消息时，自动创建服务器不存在的topic，默认创建的队列数
defaultTopicQueueNums=4

#是否允许 Broker 自动创建Topic，建议线下开启，线上关闭 ！！！这里仔细看是false，false，false
#原因下篇博客见~ 哈哈哈哈
autoCreateTopicEnable=true

#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true

#Broker 对外服务的监听端口
listenPort=10911
#haService中使用  默认值为：listenPort + 1
#haListenPort=10912
#主要用于slave同步master  listenPort - 2
#fastListenPort=10909

#删除文件时间点，默认凌晨4点
deleteWhen=04

#文件保留时间，默认48小时
fileReservedTime=120

#commitLog每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824

#ConsumeQueue每个文件默认存30W条，根据业务情况调整
mapedFileSizeConsumeQueue=300000

#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/opt/store
#commitLog 存储路径
#storePathCommitLog=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/commitlog
#消费队列存储
#storePathConsumeQueue=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/consumequeue
#消息索引存储路径
#storePathIndex=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/index
#checkpoint 文件存储路径
#storeCheckpoint=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/checkpoint
#abort 文件存储路径
#abortFile=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制Master
#- SYNC_MASTER 同步双写Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

broker slave 配置文件