Operation Management

发表于 2021-05-25 21:22:46 更新于 2024-05-15 10:18:04 阅读次数：

本文字数： 18k 阅读时长 ≈ 16 分钟

1 Deploy cluster

1.1 Single Master mode

This mode is risky, upon broker restart or broken down, the whole service is unavailable. It's not recommended in production environment, it can be used for local test.

1）Start NameServer

### Start Name Server
$ nohup sh mqnamesrv &
 
### check whether Name Server is successfully started
$ tail -f ~/logs/rocketmqlogs/namesrv.log
The Name Server boot success...

2）Start Broker

### start Broker
$ nohup sh bin/mqbroker -n localhost:9876 &

check whether Name Server is successfully started, eg: Broker’s IP is 192.168.1.2, Broker’s name is broker-a
$ tail -f ~/logs/rocketmqlogs/Broker.log
The broker[broker-a, 192.169.1.2:10911] boot success…

1.2 Multi Master mode

Cluster contains Master node only, no Slave node, eg: 2 Master nodes, 3 Master nodes, advantages and disadvantages of this mode are shown below:

advantages：simple configuration, single Master node broke down or restart do not impact application. Under RAID10 disk config, even if machine broken down and cannot recover, message do not get lost because of RAID10's high reliable(async flush to disk lost little message, sync to disk do not lost message), this mode get highest performance.
disadvantages：during the machine's down time, messages have not be consumed on this machine can not be subscribed before recovery. That will impacts message's instantaneity.

1）Start NameServer

NameServer should be started before broker. If under production environment, we recommend start 3 NameServer nodes for high available. Startup command is equal, as shown below:

### start Name Server
$ nohup sh mqnamesrv &
 
### check whether Name Server is successfully started
$ tail -f ~/logs/rocketmqlogs/namesrv.log
The Name Server boot success...

2）start Broker cluster

### start the first Master on machine A, eg:NameServer's IP is ：192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-noslave/broker-a.properties &
 
### start the second Master on machine B, eg:NameServer's IP is ：192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-noslave/broker-b.properties &

…

The above commands only used for single NameServer. In multi NameServer cluster, multi addresses concat by semicolon followed by -n in broker start command.

1.3 Multi Master Multi Slave mode - async replication

Each Master node is equipped with one Slave node, this mode has many Master-Slave group, using async replication for HA, slaver has a lag(ms level) behind master, advantages and disadvantages of this mode are shown below:

advantages: message lost a little, even if disk is broken; message instantaneity do not loss; Consumer can still consume from slave when master is down, this process is transparency to user, no human intervention is required; Performance is almost equal to Multi Master mode.
disadvantages: message lost a little data, when Master is down and disk broken.

1）Start NameServer

### start Name Server
$ nohup sh mqnamesrv &
 
### check whether Name Server is successfully started
$ tail -f ~/logs/rocketmqlogs/namesrv.log
The Name Server boot success...

2）Start Broker cluster

### start first Master on machine A, eg: NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-async/broker-a.properties &
 
### start second Master on machine B, eg: NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-async/broker-b.properties &
 
### start first Slave on machine C, eg: NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-async/broker-a-s.properties &
 
### start second Slave on machine D, eg: NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-async/broker-b-s.properties &

1.4 Multi Master Multi Slave mode - synchronous double write

Each Master node is equipped with one Slave node, this mode has many Master-Slave group, using synchronous double write for HA, application's write operation is successful means both master and slave write successful, advantages and disadvantages of this mode are shown below:

advantages:both data and service have no single point failure, message has no lantancy even if Master is down, service available and data available is very high;
disadvantages:this mode's performance is 10% lower than async replication mode, sending latency is a little high, in the current version, it do not have auto Master-Slave switch when Master is down.

1）Start NameServer

### start Name Server
$ nohup sh mqnamesrv &
 
### check whether Name Server is successfully started
$ tail -f ~/logs/rocketmqlogs/namesrv.log
The Name Server boot success...

2）Start Broker cluster

### start first Master on machine A, eg:NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-sync/broker-a.properties &
 
### start second Master on machine B, eg:NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-sync/broker-b.properties &
 
### start first Slave on machine C, eg: NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-sync/broker-a-s.properties &
 
### start second Slave on machine D, eg: NameServer's IP is 192.168.1.1
$ nohup sh mqbroker -n 192.168.1.1:9876 -c $ROCKETMQ_HOME/conf/2m-2s-sync/broker-b-s.properties &

The above Broker matches Slave by specifying the same BrokerName, Master's BrokerId must be 0, Slave's BrokerId must larger than 0. Besides, a Master can have multi Slaves that each has a distinct BrokerId. $ROCKETMQ_HOME indicates RocketMQ's install directory, user needs to set this environment parameter.

2 mqadmin management tool

Attentions：
execute command：./mqadmin {command} {args}
almost all commands need -n indicates NameSerer address, format is ip:port
almost all commands can get help info by -h
if command contains both Broker address(-b) and cluster name(-c), it's prior to use broker address. If command do not contains broker address, it will executed on all hosts in this cluster. Support only one broker host. -b format is ip:port, default port is 10911
there are many commands under tools, but not all command can be used, only commands that initialized in MQAdminStartup can be used, you can modify this class, add or self-define command.
because of version update, little command do not update timely, please refer to source code directly when occur error.

2.1 Topic

name	meaning	command items	explaination
updateTopic	create or update Topic's config	-b	Broker address, means which Broker that topic is located, only support single Broker, address format is ip:port
		-c	cluster name, whic cluster that topic belongs to(query cluster info by clusterList)
		-h-	print help info
		-n	NameServer Service address, format is ip:port
		-p	assign read write authority to new topic(W=2\|R=4\|WR=6)
		-r	the count of queue that can be read(default is 8)
		-w	the count of queue that can be wrote(default is 8)
		-t	topic name(can only use characters ^[a-zA-Z0-9_-]+$ )
deleteTopic	delete Topic	-c	cluster name, which cluster that topic will be deleted belongs to(query cluster info by clusterList)
		-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic name(can only use characters ^[a-zA-Z0-9_-]+$ )
topicList	query Topic list info	-h	print help info
		-c	return topic list only if do not contains -c, if containis -c, it will return cluster name, topic name, consumer group name
		-n	NameServer Service address, format is ip:port
topicRoute	query Topic's route info	-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
topicStatus	query Topic's offset	-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
topicClusterList	query cluster list where Topic belongs to	-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
updateTopicPerm	update Topic's produce and consume authority	-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
		-b	Broker address which topic belongs to, support single broker only, format is ip:port
		-p	assign read and write authority to the new topic(W=2\|R=4\|WR=6)
		-c	cluster name, which topic belongs to(query cluster info by clusterList), if do not have -b, execute comman an all brokers.
updateOrderConf	create delete get specified namespace's kv config from NameServer, have not enabled at present	-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic, key
		-v	orderConf, value
		-m	method, including get, put, delete
allocateMQ	calculate consumer list rebalance result by average rebalance algorithm	-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
		-i	ipList, seperate by comma, calculate which topic queue that ips will load.
statsAll	print Topic's subscribe info, TPS, size of message blocked, count of read and write at last 24h, eg.	-h	print help info
		-n	NameServer Service address, format is ip:port
		-a	only print active topic or not
		-t	assign topic

2.2 Cluster

名称	meaning	command items	explaination
clusterList	query cluster info, including cluster, BrokerName, BrokerId, TPS, eg.	-m	print more infos(eg: #InTotalYest, #OutTotalYest, #InTotalToday ,#OutTotalToday)
		-h	print help info
		-n	NameServer Service address, format is ip:port
		-i	print interval, unit second
clusterRT	send message to detect each cluster's Broker RT. Message will be sent to ${BrokerName} Topic。	-a	amount, count of detection, RT = sum time / amount
		-s	size of message, unit B
		-c	which cluster will be detected
		-p	whether print format log, splitted by \|, default is not print
		-h	print help info
		-m	which machine room it belongs to, just for print
		-i	send interval, unit second
		-n	NameServer Service address, format is ip:port

2.3 Broker

名称	meaning	command items	explaination
updateBrokerConfig	update Broker's config file, it will modify Broker.conf	-b	Broker address, format is ip:port
		-c	cluster name
		-k	key
		-v	value
		-h	print help info
		-n	NameServer Service address, format is ip:port
brokerStatus	get Broker's statistics info, running status(including whatever you want).	-b	Broker address, fomat isip:port
		-h	print help info
		-n	NameServer Service address, format is ip:port
brokerConsumeStats	Broker's consumer info, including Consume Offset, Broker Offset, Diff, Timestamp that ordered by essage Queue	-b	Broker address, fomat isip:port
		-t	request timeout time
		-l	diff threshold, it will print when exceed this threshold.
		-o	whether is sequencial topic, generally false
		-h	print help info
		-n	NameServer Service address, format is ip:port
getBrokerConfig	get Broker's config	-b	Broker address, fomat isip:port
getBrokerConfig	get Broker's config	-n	NameServer Service address, format is ip:port
wipeWritePerm	revoke broker's write authority from NameServer.	-b	Broker address, fomat isip:port
		-n	NameServer Service address, format is ip:port
		-h	print help info
cleanExpiredCQ	clean Broker's expired Consume Queue that maybe generated by decrease queue count.	-n	NameServer Service address, format is ip:port
		-h	print help info
		-b	Broker address, fomat isip:port
		-c	cluster name
cleanUnusedTopic	clean Broker's unused Topic that deleted mannually to release memory that Topic's Consume Queue occupied.	-n	NameServer Service address, format is ip:port
		-h	print help info
		-b	Broker address, fomat isip:port
		-c	cluster name
sendMsgStatus	send message to Broker, return send status and RT	-n	NameServer Service address, format is ip:port
		-h	print help info
		-b	BrokerName, is different from broker address
		-s	message size, unit B
		-c	send count

2.4 Message

名称	meaning	command items	explaination
queryMsgById	query message by offsetMsgId. If use opensource console, it should use offsetMsgId. Please refer to QueryMsgByIdSubCommand for detail.	-i	msgId
		-h	print help info
		-n	NameServer Service address, format is ip:port
queryMsgByKey	query message by Message's Key	-k	msgKey
		-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
queryMsgByOffset	query message by Offset	-b	Broker name(it's not broker address, can query Broker name by clusterList).
		-i	query queue id
		-o	offset value
		-t	topic name
		-h	print help info
		-n	NameServer Service address, format is ip:port
queryMsgByUniqueKey	query by msgId, msgId is different from offsetMsgId, please refer to Frequently asked questions about operations for details. Use -g and -d to let specified consumer return consume result.	-h	print help info
		-n	NameServer Service address, format is ip:port
		-i	uniqe msg id
		-g	consumerGroup
		-d	clientId
		-t	topic name
checkMsgSendRT	detect RT of sending a message to a topic, similiar to clusterRT	-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic name
		-a	detection count
		-s	size of the message
sendMessage	send a message, also can send to a specified Message Queue.	-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic name
		-p	body, message entity
		-k	keys
		-c	tags
		-b	BrokerName
		-i	queueId
consumeMessage	consume message. Differert consume logic depends on offset, start & end timestamp, message queue, please refer to ConsumeMessageCommand for details.	-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic name
		-b	BrokerName
		-o	offset that consumer start consume
		-i	queueId
		-g	consumer gropu
		-s	timestamp at start, refer to -h to get format开
		-d	timestamp at the end
		-c	size of message that consumed
printMsg	consume and print messages from broker, support a time range	-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic name
		-c	charset, eg: UTF-8
		-s	subExpress, filter expression
		-b	timestap at start, refer to -h to get format
		-e	timestamp at the end
		-d	whether print message entity or not
printMsgByQueue	similar to printMsg, but it need specified Message Queue	-h	print help info
		-n	NameServer Service address, format is ip:port
		-t	topic name
		-i	queueId
		-a	BrokerName
		-c	charset, eg: UTF-8
		-s	subExpress, filter expression
		-b	timestamp at start, refer to -h to get format
		-e	timestamp at the end
		-p	whether print message or not
		-d	whether print message entity or not
		-f	whether count and print tag or not
resetOffsetByTime	reset offset by timestamp, Broker and consumer will all be reseted	-h	print help info
		-n	NameServer Service address, format is ip:port
		-g	consumer group
		-t	topic name
		-s	reset offset corresponding to this timestamp
		-f	whether enforce to reset or not, if set false, only can reset offset, if set true, it omit the relationship between timestamp and consumer offset.
		-c	whether reset c++ sdk's offset or not

2.5 Consumer, Consumer Group

name	meaning	command items	explaination
consumerProgress	query subscribe status, can get blocking counts of a concrete client ip.	-g	consumer group name
		-s	whether print client IP or not
		-h	print help info
		-n	NameServer Service address, format is ip:port
consumerStatus	query consumer status, including message blocking, and consumer's jstack result(please refer to ConsumerStatusSubCommand)	-h	print help info
		-n	NameServer Service address, format is ip:port
		-g	consumer group
		-i	clientId
		-s	whether execute jstack or not
updateSubGroup	create or update subscribe info	-n	NameServer Service address, format is ip:port
		-h	print help info
		-b	Broker address
		-c	cluster name
		-g	consumer group name
		-s	consumer group is allowed to consume or not
		-m	start consume from minimal offset or not
		-d	broadcast mode or not
		-q	capacity of retry queue
		-r	max retry count
		-i	It works when slaveReadEnable enabled, and that not consumed from slave. Suggesting that consume from slave node by specify slave id.
		-w	If broker consume from slave, whic slave node depends on this config that configed by BrokerId, eg: 1.
		-a	whether notify other consumers to rebalance or not when the count of consumer changes
deleteSubGroup	delete subscribe info from Broker	-n	NameServer Service address, format is ip:port
		-h	print help info
		-b	Broker address
		-c	cluster name
		-g	consumer group name
cloneGroupOffset	use source group's offset at target group	-n	NameServer Service address, format is ip:port
		-h	print help info
		-s	source consumer group
		-d	target consumer group
		-t	topic name
		-o	not used at present

2.6 Connection

name	meaning	command items	explaination
consumerConnec tion	query Consumer's connection	-g	consumer group name
		-n	NameServer Service address, format is ip:port
		-h	print help info
producerConnec tion	query Producer's connection	-g	producer group name
		-t	topic name
		-n	NameServer Service address, format is ip:port
		-h	print help info

2.7 NameServer

name	meaning	command items	explaination
updateKvConfig	update NameServer's kv config, not used at present	-s	namespace
		-k	key
		-v	value
		-n	NameServer Service address, format is ip:port
		-h	print help info
deleteKvConfig	delete NameServer's kv config	-s	namespace
		-k	key
		-n	NameServer Service address, format is ip:port
		-h	print help info
getNamesrvConfig	get NameServer's config	-n	NameServer Service address, format is ip:port
getNamesrvConfig	get NameServer's config	-h	print help info
updateNamesrvConfig	modify NameServer's config	-n	NameServer Service address, format is ip:port
		-h	print help info
		-k	key
		-v	value

2.8 Other

name	meaning	command items	explaination
startMonitoring	Start the monitoring process, monitor message deletion and the number of retried messages in the queue	-n	NameServer Service address, format is ip:port
startMonitoring		-h	print help info

3 Frequently asked questions about operations

3.1 RocketMQ's mqadmin command error

question description：execute mqadmin occur below exception after deploy RocketMQ cluster.
org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to  failed

Solution: execute command export NAMESRV_ADDR=ip:9876 (ip is NameServer's ip address), then execute mqadmin commands.

3.2 RocketMQ consumer cannot consume, because of different version of producer and consumer.

question description: one producer produce message, consumer A can consume, consume B cannot consume, RocketMQ console print:
Not found the consumer group consume stats, because return offset table is empty, maybe the consumer not consume any message。

Solution: make sure that producer and consumer has the same version of rocketmq-client.

3.3 Consumer cannot consume oldest message, when a new consumer group is added.

question description: when a new consumer group start, it consumes from current offset, do not fetch oldest message.

Solution: rocketmq's default policy is consume from latest, that is skip oldest message. If you want consume oldest message, you need to set org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#setConsumeFromWhere. The following is three common configurations:

default configuration, a new consumer group consume from latest position at first startup, then consume from last time's offset at next startup, that is skip oldest message;

consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);

a new consumer group consume from oldest postion at first startup, then consume from last time's offset at next startup, that is consume the unexpired message;

consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);

a new consumer group consume from specified timestamp at first startup, then consume from last time's offset at next startup, cooperate with consumer.setConsumeTimestamp(), default is half an hour before;

consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_TIMESTAMP);

3.4 How to enable consume from Slave

In some cases, consumer need reset offset to a day or two before, if Master Broker has limited memory, it's CommitLog will have a high IO load, then it will impact other message's read and write that on this broker. When slaveReadEnable=true is set, and consumer's offset exceeds accessMessageInMemoryMaxRatio=40%, Master Broker will recommend consumer consume from Slave Broker to lower Master Broker IO.

3.5 Performance tuning

A spin lock is recommended for asynchronous disk flush, a reentrant lock is recommended for synchronous disk flush, configuration item is useReentrantLockWhenPutMessage, default is false; Enable TransientStorePoolEnable is recommended when use asynchronous disk flush; Recommend to close transferMsgByHeap to improve fetch efficiency; Set a little larger sendMessageThreadPoolNums, when use synchronous disk flush.

3.6 The meaning and difference between msgId and offsetMsgId in RocketMQ

You will usually see the following log print message after sending message by using RocketMQ sdk.

SendResult [sendStatus=SEND_OK, msgId=0A42333A0DC818B4AAC246C290FD0000, offsetMsgId=0A42333A00002A9F000000000134F1F5, messageQueue=MessageQueue [topic=topicTest1, BrokerName=mac.local, queueId=3], queueOffset=4]

msgId, is generated by producer sdk. In particular, call method MessageClientIDSetter.createUniqIDBuffer() to generate unique Id;
offsetMsgId, offsetMsgId is generated by Broker server(format is "Ip Address + port + CommitLog offset"). offsetMsgId is messageId that is RocketMQ console's input.