云野 New-运维运维 2023-04-14

Redis集群

Redis简介：Redis 是一个开源（BSD许可）的，内存中的数据结构存储系统，它可以用作数据库、缓存和消息中间件。它支持多种类型的数据结构，如字符串（strings），散列（hashes），列表（lists），集合（sets），有序集合（sorted sets）与范围查询， bitmaps， hyperloglogs 和地理空间（geospatial）索引半径查询。 Redis 内置了复制（replication），LUA脚本（Lua scripting）， LRU驱动事件（LRU eviction），事务（transactions）和不同级别的磁盘持久化（persistence），并通过 Redis哨兵（Sentinel）和自动分区（Cluster）提供高可用性（high availability）。

以上来自Redis中文官网：http://redis.cn/ , 下载地址：https://redis.io/download/

Redis集群介绍

Redis 集群是一个提供在多个Redis间节点间共享数据的程序集。

Redis集群并不支持处理多个keys的命令,因为这需要在不同的节点间移动数据,从而达不到像Redis那样的性能,在高负载的情况下可能会导致不可预料的错误.

Redis 集群通过分区来提供一定程度的可用性,在实际环境中当某个节点宕机或者不可达的情况下继续处理命令. Redis 集群的优势:

自动分割数据到不同的节点上。
整个集群的部分节点失败或者不可达的情况下能够继续处理命令。

Redis 集群的数据分片

Redis 集群没有使用一致性hash, 而是引入了 哈希槽的概念.

Redis 集群有16384个哈希槽,每个key通过CRC16校验后对16384取模来决定放置哪个槽.集群的每个节点负责一部分hash槽,举个例子,比如当前集群有3个节点,那么:

节点 A 包含 0 到 5500号哈希槽.
节点 B 包含5501 到 11000 号哈希槽.
节点 C 包含11001 到 16384号哈希槽.

这种结构很容易添加或者删除节点. 比如如果我想新添加个节点D, 我需要从节点 A, B, C中得部分槽到D上. 如果我想移除节点A,需要将A中的槽移到B和C节点上,然后将没有任何槽的A节点从集群中移除即可. 由于从一个节点将哈希槽移动到另一个节点并不会停止服务,所以无论添加删除或者改变某个节点的哈希槽的数量都不会造成集群不可用的状态.

集群搭建

1. redis里面内嵌了集群，在redis-5.0.14/utils/create-cluster/下执行./create-cluster start自动创建实例集群的脚本，一共会创建六个示例。

[root@Redis2 ~]# cd redis/redis-5.0.14/utils/create-cluster/
[root@Redis2 create-cluster]# ls
create-cluster README
[root@Redis2 create-cluster]# ./create-cluster start
Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006

2. 在一个节点上同时启动多个实例监听不同的端口且创建的这些实例都是可以直接使用的。

[root@Redis2 create-cluster]# redis-cli -p 30001 info
......
# Replication
role:master      ##可以看到创建的实例30001为master节点
connected_slaves:1
slave0:ip=127.0.0.1,port=30005,state=online,offset=168,lag=0    ##创建的实例30005为slave节点
master_replid:9f62447ce0b635eb2d1b3bac124160b7a496471e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:168
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:168

3. 当一个master挂掉以后会有专门的slave自动来接管，内部保证了高可用而且整体会进行负载均衡。

[root@Redis2 create-cluster]# ./create-cluster create  ##一共有六个实例，所以自动创建三组一主一从的集群

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:30005 to 127.0.0.1:30001
Adding replica 127.0.0.1:30006 to 127.0.0.1:30002
Adding replica 127.0.0.1:30004 to 127.0.0.1:30003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots:[0-5460] (5461 slots) master
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
S: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
replicates 60c1137dcca11bf6ad1358708e7771dc931f9f52
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
Can I set the above configuration? (type 'yes' to accept): yes  ##输入yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots: (0 slots) slave
replicates 60c1137dcca11bf6ad1358708e7771dc931f9f52
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

4. 查看集群的info信息。

[root@Redis2 create-cluster]# redis-cli --cluster info 127.0.0.1:30001 
127.0.0.1:30001 (60c1137d...) -> 0 keys | 5461 slots | 1 slaves.  
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
可以看到30001，30002，30003为master

5 . 查看六个实例所对应的master和slave主从关系。

[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30001
127.0.0.1:30001 (60c1137d...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001  ##通过ID可以看到master30001对应slave30005
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots: (0 slots) slave
replicates 60c1137dcca11bf6ad1358708e7771dc931f9f52
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003   ##master30003对应slave30004
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002   ##master30002对应slave30006
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

6. 集群的无中心化，集群内都可交互式访问且读写，不需要担心key存在在那个节点，访问集群内任意一个实例节点都可以看到key。

[root@Redis2 create-cluster]# redis-cli -c -p 30001  ##交互式打开集群
127.0.0.1:30001> set name redis
-> Redirected to slot [5798] located at 127.0.0.1:30002   ##可以看到做了一个重定向，通过哈希槽定位到了30002上
OK
127.0.0.1:30002> get name
"redis"
127.0.0.1:30002>
[root@Redis2 create-cluster]# redis-cli -c -p 30004  ##集群内都是可以访问的
127.0.0.1:30004> get name
-> Redirected to slot [5798] located at 127.0.0.1:30002  
"redis"
127.0.0.1:30002> 
[root@Redis2 create-cluster]# redis-cli -c -p 30005  ##从30005可以看到30001建立的key
127.0.0.1:30005> get name
-> Redirected to slot [5798] located at 127.0.0.1:30002
"redis"
127.0.0.1:30002> 
[root@Redis2 create-cluster]# redis-cli -c -p 30002   ##从30002也可以看到30001建立的key
127.0.0.1:30002> get name
"redis"
127.0.0.1:30002>

7. 当一个master节点停掉以后，他所对应的slave节点会自动接替他成为master节点。

7. 1 停掉30001的master节点。

[root@Redis2 create-cluster]# redis-cli -p 30001 shutdown

7.2 上面提到过，master30001所对用的slave节点是30005，所以查看一下slave30005节点的信息。

[root@Redis2 create-cluster]# redis-cli -p 30005 info
......
# Replication
role:master      ##可以看到30005自动接替30001成为了master节点
connected_slaves:0
master_replid:11948071f8e9b17bb86c62dd1f79dfb9ad043473
master_replid2:9f62447ce0b635eb2d1b3bac124160b7a496471e
master_repl_offset:4676
second_repl_offset:4677
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:4676

7.3 30005不仅接替30001成为了master，也继承了原属于30001的哈希槽。

[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005   
Could not connect to Redis at 127.0.0.1:30001: Connection refused
127.0.0.1:30005 (118d25e3...) -> 0 keys | 5461 slots | 0 slaves.
127.0.0.1:30002 (c41e2069...) -> 1 keys | 5462 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005    ##可以看到30001已经挂掉，且30005继承了原属于他的5461个哈希槽
slots:[0-5460] (5461 slots) master
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

7.4 把30001重新启动起起来，里面的数据是依然不会改变的，但是唯一会变得是30005替代30001成为master，30001成为30005的slave，身份转换了。

[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005
127.0.0.1:30005 (118d25e3...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 1 keys | 5462 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[0-5460] (5461 slots) master     ##30005成为了master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave     ##30001成为了30005的slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

8. 除了原有的三组主从，还可以加节点进去。

8.1 更改创建集群的脚本。

[root@Redis2 create-cluster]# pwd
/root/redis/redis-5.0.14/utils/create-cluster
[root@Redis2 create-cluster]# vim create-cluster 
[root@Redis2 create-cluster]# head -8 create-cluster 
#!/bin/bash

# Settings
PORT=30000
TIMEOUT=2000
NODES=8    ##把原来的6个改成8个，其余的地方不动
REPLICAS=1

8.2 重新启动脚本，创建新加的节点。

[root@Redis2 create-cluster]# ./create-cluster start
Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006
Starting 30007
Starting 30008

8.3 把创建的节点加到原来的集群中。

[root@Redis2 create-cluster]# redis-cli --cluster add-node 127.0.0.1:30007 127.0.0.1:30001   ##把创建的30007加到原有集群中
>>> Adding node 127.0.0.1:30007 to cluster 127.0.0.1:30001
>>> Performing Cluster Check (using node 127.0.0.1:30001)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:30007 to make it join the cluster.
[OK] New node added correctly.

8.4 查看30007在集群中的信息。

[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005
127.0.0.1:30005 (118d25e3...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 1 keys | 5462 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30007 (79ab6170...) -> 0 keys | 0 slots | 0 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
M: 79ab6170496ff41b969ecda181dcacbd5cc3f018 127.0.0.1:30007     ##可以看到30007已经加入到了集群中且自动成为master
slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

8.5 把30008作为30007的slave节点也加入到集群中。（注意–cluster-master-id后面跟的是30007的ID）。

[root@Redis2 create-cluster]# redis-cli --cluster add-node 127.0.0.1:30008 127.0.0.1:30001 --cluster-slave --cluster-master-id 79ab6170496ff41b969ecda181dcacbd5cc3f018
>>> Adding node 127.0.0.1:30008 to cluster 127.0.0.1:30001
>>> Performing Cluster Check (using node 127.0.0.1:30001)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: 79ab6170496ff41b969ecda181dcacbd5cc3f018 127.0.0.1:30007
slots: (0 slots) master
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:30008 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 127.0.0.1:30007.
[OK] New node added correctly.

8.6 再查看集群的信息，30008成为了30007的slave节点。

[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005
127.0.0.1:30005 (118d25e3...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 1 keys | 5462 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30007 (79ab6170...) -> 0 keys | 0 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
S: 556c23e28a2ffec4a5ff966a9d9816b9357a2f55 127.0.0.1:30008
slots: (0 slots) slave    ##已经添加30008成为了30007的slave节点
replicates 79ab6170496ff41b969ecda181dcacbd5cc3f018
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
M: 79ab6170496ff41b969ecda181dcacbd5cc3f018 127.0.0.1:30007
slots: (0 slots) master    ##虽然30007已经添加到了集群中，但是可以看到他的哈希槽是空的
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

8.7 因为之前最开始集群创建的时候已经把16384个哈希槽平均分配给了原有的三组节点，现在新添加了一组，所以现在要再从之前的三组节点给新添加的30007这组节点重新分配哈希槽，30007这组节点才能使用。

[root@Redis2 create-cluster]# redis-cli --cluster reshard 127.0.0.1:30001
>>> Performing Cluster Check (using node 127.0.0.1:30001)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[999-5460] (4462 slots) master
1 additional replica(s)
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[6462-10922] (4461 slots) master
1 additional replica(s)
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: 79ab6170496ff41b969ecda181dcacbd5cc3f018 127.0.0.1:30007
slots:[0-998],[5461-6461],[10923-11921] (2999 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[11922-16383] (4462 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 556c23e28a2ffec4a5ff966a9d9816b9357a2f55 127.0.0.1:30008
slots: (0 slots) slave
replicates 79ab6170496ff41b969ecda181dcacbd5cc3f018
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 3000  ##要从之前三组节点迁移到新添加这组节点的哈希槽的数量
What is the receiving node ID? 79ab6170496ff41b969ecda181dcacbd5cc3f018   ##要添加的master30007的ID
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: all   ##从所有的节点迁移，集群会自动平均从每组节点分配出1000个哈希槽迁移到新添加的节点中

8.8 查看迁移后的master30007的哈希槽数量。

[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005
127.0.0.1:30005 (118d25e3...) -> 0 keys | 4462 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 0 keys | 4461 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 4462 slots | 1 slaves.
127.0.0.1:30007 (79ab6170...) -> 1 keys | 2999 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[999-5460] (4462 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
S: 556c23e28a2ffec4a5ff966a9d9816b9357a2f55 127.0.0.1:30008
slots: (0 slots) slave
replicates 79ab6170496ff41b969ecda181dcacbd5cc3f018
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[6462-10922] (4461 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[11922-16383] (4462 slots) master
1 additional replica(s)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
M: 79ab6170496ff41b969ecda181dcacbd5cc3f018 127.0.0.1:30007     
slots:[0-998],[5461-6461],[10923-11921] (2999 slots) master     ##可以看到已经从原来的三组节点迁移了3000个哈希槽到30007
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

9. 集群内的master是平分了16384个哈希槽的，如果这16384个哈希槽不完整那么这个集群是不可用的。

9.1 把30007和30008的这组集群节点停掉。

[root@Redis2 create-cluster]# redis-cli -p 30007 shutdown
[root@Redis2 create-cluster]# redis-cli -p 30008 shutdown

9.2 可以看到停掉一组集群后，虽然集群内其他的节点可以连接，但整个集群是不可用的，因为他整个集群的哈希槽不完整了。

[root@Redis2 create-cluster]# redis-cli -p 30001
127.0.0.1:30001> get name
(error) CLUSTERDOWN The cluster is down
127.0.0.1:30001>
[root@Redis2 create-cluster]# redis-cli -p 30002
127.0.0.1:30002> get name
(error) CLUSTERDOWN The cluster is down
127.0.0.1:30002>
[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005
Could not connect to Redis at 127.0.0.1:30008: Connection refused
Could not connect to Redis at 127.0.0.1:30007: Connection refused
127.0.0.1:30005 (118d25e3...) -> 0 keys | 4462 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 0 keys | 4461 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 4462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[999-5460] (4462 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[6462-10922] (4461 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[11922-16383] (4462 slots) master
1 additional replica(s)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.    ###集群内的哈希槽不完整

9.3 要恢复集群，只需要重新执行脚本就好。

[root@Redis2 create-cluster]# ./create-cluster start    ##启动重新脚本
Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006
Starting 30007
Starting 30008
[root@Redis2 create-cluster]# redis-cli --cluster check 127.0.0.1:30005
127.0.0.1:30005 (118d25e3...) -> 0 keys | 4462 slots | 1 slaves.
127.0.0.1:30008 (556c23e2...) -> 1 keys | 2999 slots | 1 slaves.
127.0.0.1:30002 (c41e2069...) -> 0 keys | 4461 slots | 1 slaves.
127.0.0.1:30003 (20c1aca9...) -> 0 keys | 4462 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30005)
M: 118d25e366cbfc6659a28a735dec37e592306841 127.0.0.1:30005
slots:[999-5460] (4462 slots) master
1 additional replica(s)
S: 383bd4458b7ce29554973e5834d083b627724668 127.0.0.1:30006
slots: (0 slots) slave
replicates c41e2069a2f223133a4a9beff5b81ea40b1119bc
S: 82cf0c79a2dc271529131879dd1a5c837673993c 127.0.0.1:30004
slots: (0 slots) slave
replicates 20c1aca9d0882c4f17cba966fe38b1470e93c269
M: 556c23e28a2ffec4a5ff966a9d9816b9357a2f55 127.0.0.1:30008
slots:[0-998],[5461-6461],[10923-11921] (2999 slots) master
1 additional replica(s)
M: c41e2069a2f223133a4a9beff5b81ea40b1119bc 127.0.0.1:30002
slots:[6462-10922] (4461 slots) master
1 additional replica(s)
M: 20c1aca9d0882c4f17cba966fe38b1470e93c269 127.0.0.1:30003
slots:[11922-16383] (4462 slots) master
1 additional replica(s)
S: 60c1137dcca11bf6ad1358708e7771dc931f9f52 127.0.0.1:30001
slots: (0 slots) slave
replicates 118d25e366cbfc6659a28a735dec37e592306841
S: 79ab6170496ff41b969ecda181dcacbd5cc3f018 127.0.0.1:30007
slots: (0 slots) slave
replicates 556c23e28a2ffec4a5ff966a9d9816b9357a2f55
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

所以，如果要删除集群内主从节点的时候，一定要先把要删除节点的哈希槽迁移走，不然直接删除会导致集群不完整，从而集群不可用。

10. 删除建立的集群。

[root@Redis2 create-cluster]# ls 
30001.log 30004.log 30007.log appendonly-30002.aof appendonly-30005.aof appendonly-30008.aof nodes-30002.conf nodes-30005.conf nodes-30008.conf
30002.log 30005.log 30008.log appendonly-30003.aof appendonly-30006.aof create-cluster nodes-30003.conf nodes-30006.conf README
30003.log 30006.log appendonly-30001.aof appendonly-30004.aof appendonly-30007.aof nodes-30001.conf nodes-30004.conf nodes-30007.conf
[root@Redis2 create-cluster]# ./create-cluster stop    ##先停掉集群
Stopping 30001
Stopping 30002
Stopping 30003
Stopping 30004
Stopping 30005
Stopping 30006
Stopping 30007
Stopping 30008
[root@Redis2 create-cluster]# ./create-cluster clean   ##执行脚本删除集群
[root@Redis2 create-cluster]# ls
create-cluster README

Redis