环境
docker desktop for windows 4.23.0
redis 7.2
一、前提准备 1. 主从集群 首先需要有一个redis主从集群,才能接着做redis哨兵。具体可以参考下面这篇文章Docker-Compose部署Redis(v7.2)主从模式 (之后简称”主从模式博文
“)
2. 文件夹结构 和主从模式不同的是,redis sentinel
(哨兵)会更改你的conf
文件,无论是redis server
节点还是sentinel
节点本身,都可能被修改,所以这里需要注意文件权限问题。不然会一直警告Sentinel was not able to save the new configuration on disk
。
有兴趣可以参考以下几个帖子,或者接着本文做就行了。
总的来说,需要对主从模式博文
里提到的文件夹结构做一定改善和添加,具体如下:
language-txt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 cluster/ ├── docker-compose.yaml ├── master │ └── conf │ └── redis.conf ├── replica1 │ └── conf │ └── redis.conf ├── replica2 │ └── conf │ └── redis.conf ├── sentinel1 │ └── conf │ └── sentinel.conf ├── sentinel2 │ └── conf │ └── sentinel.conf └── sentinel3 └── conf └── sentinel.conf
其中redis.conf
和docker-compose.yaml
和主从模式博文
内容暂时保持一致,其余的都是新增的,暂时保持空白即可。
二、配置文件 1. redis server配置文件 保持不变
2. redis sentinel配置文件 对于上述三个sentinel.conf
内容都填入以下
language-sql 1 2 3 4 5 sentinel monitor mymaster 172.30.1.2 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 60000 sentinel auth-pass mymaster 1009 dir "/data"
意思分别是
监控的主节点:通过 sentinel monitor
指定要监控的主节点。这包括一个用户定义的名称(如 mymaster
)、主节点的地址、端口号和一个”仲裁”阈值,后者表示要进行故障转移所需的最小 Sentinel
投票数量。
故障检测:设置 Sentinel
判断主节点是否下线所需的时间
故障转移设置:配置故障转移的行为,如故障转移的超时时间
认证密码(如果主节点设置了密码):如果主节点设置了密码,Sentinel
需要这个密码来连接主节点和副本节点
设置 Sentinel
的工作目录
3. docker compose文件 language-yaml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 version: '3.8' networks: redis-network: driver: bridge ipam: driver: default config: - subnet: 172.30.1.0/24 services: redis-master: container_name: redis-master image: redis:7.2 volumes: - ./master/conf:/usr/local/etc/redis # - ./master/data:/data ports: - "7001:6379" command: ["redis-server", "/usr/local/etc/redis/redis.conf"] networks: redis-network: ipv4_address: 172.30.1.2 redis-replica1: container_name: redis-replica1 image: redis:7.2 volumes: - ./replica1/conf:/usr/local/etc/redis # - ./replica1/data:/data ports: - "7002:6379" command: ["redis-server", "/usr/local/etc/redis/redis.conf"] depends_on: - redis-master networks: redis-network: ipv4_address: 172.30.1.3 redis-replica2: container_name: redis-replica2 image: redis:7.2 volumes: - ./replica2/conf:/usr/local/etc/redis # - ./replica2/data:/data ports: - "7003:6379" command: ["redis-server", "/usr/local/etc/redis/redis.conf"] depends_on: - redis-master networks: redis-network: ipv4_address: 172.30.1.4 redis-sentinel1: container_name: redis-sentinel1 image: redis:7.2 volumes: - ./sentinel1/conf:/usr/local/etc/redis ports: - "27001:26379" command: ["redis-sentinel", "/usr/local/etc/redis/sentinel.conf"] depends_on: - redis-master networks: redis-network: ipv4_address: 172.30.1.11 redis-sentinel2: container_name: redis-sentinel2 image: redis:7.2 volumes: - ./sentinel2/conf:/usr/local/etc/redis ports: - "27002:26379" command: [ "redis-sentinel", "/usr/local/etc/redis/sentinel.conf" ] depends_on: - redis-master networks: redis-network: ipv4_address: 172.30.1.12 redis-sentinel3: container_name: redis-sentinel3 image: redis:7.2 volumes: - ./sentinel3/conf:/usr/local/etc/redis ports: - "27003:26379" command: [ "redis-sentinel", "/usr/local/etc/redis/sentinel.conf" ] depends_on: - redis-master networks: redis-network: ipv4_address: 172.30.1.13
需要注意以下几点
和主从模式博文
不同,这里所有的配置文件挂载都采用文件夹挂载
而非单文件挂载
这里自定义了bridge
子网并限定了范围,如果该范围已经被使用,请更换。
这里没有对data
进行-v
挂载,如果要挂载,请注意宿主机对应文件夹权限问题。
主节点地址为172.30.1.2
,如果更改请注意sentinel.conf
中也需要更改。
三、运行
在运行之前,记得备份一下所有的conf文件,因为sentinel会修改挂载到容器的conf。
language-bash 1 docker-compose -p redis-cluster up -d
查看其中一个sentinel节点的日志,可以看到监听端口是26379,同时监测主节点mymaster 172.30.1.2 6379
,以及添加了172.30.1.4 6379
和172.30.1.3 6379
两个从节点,并且感应到了位于172.30.1.13 26379
和172.30.1.12 26379
两个同为sentinel
节点的服务。
language-txt 1 2 3 4 5 6 7 8 9 10 11 12 2024-01-05 18:06:40 1:X 05 Jan 2024 10:06:40.758 * Running mode=sentinel, port=26379. 2024-01-05 18:06:40 1:X 05 Jan 2024 10:06:40.789 * Sentinel new configuration saved on disk 2024-01-05 18:06:40 1:X 05 Jan 2024 10:06:40.790 * Sentinel ID is 499007c98c0a165b13e026a4443ceb890695c191 2024-01-05 18:06:40 1:X 05 Jan 2024 10:06:40.790 # +monitor master mymaster 172.30.1.2 6379 quorum 2 2024-01-05 18:06:40 1:X 05 Jan 2024 10:06:40.791 * +slave slave 172.30.1.4:6379 172.30.1.4 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:06:40 1:X 05 Jan 2024 10:06:40.815 * Sentinel new configuration saved on disk 2024-01-05 18:06:42 1:X 05 Jan 2024 10:06:42.055 * +sentinel sentinel bcfaed15fb01e7ad03b013fe5e964479c1a1f138 172.30.1.13 26379 @ mymaster 172.30.1.2 6379 2024-01-05 18:06:42 1:X 05 Jan 2024 10:06:42.093 * Sentinel new configuration saved on disk 2024-01-05 18:06:42 1:X 05 Jan 2024 10:06:42.356 * +sentinel sentinel 92d9a1419be1256d1715df2aa17cea4bbacfdf60 172.30.1.12 26379 @ mymaster 172.30.1.2 6379 2024-01-05 18:06:42 1:X 05 Jan 2024 10:06:42.376 * Sentinel new configuration saved on disk 2024-01-05 18:06:50 1:X 05 Jan 2024 10:06:50.823 * +slave slave 172.30.1.3:6379 172.30.1.3 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:06:50 1:X 05 Jan 2024 10:06:50.837 * Sentinel new configuration saved on disk
四、测试 直接让redis-master
容器停止运行,查看sentinel
日志,可以看到sentinel
监测到master
节点挂掉后,选举了172.30.1.3
为新的主节点,并将其余两个作为slave
节点。
language-txt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.896 # +sdown master mymaster 172.30.1.2 6379 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.968 # +odown master mymaster 172.30.1.2 6379 #quorum 2/2 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.968 # +new-epoch 1 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.968 # +try-failover master mymaster 172.30.1.2 6379 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.987 * Sentinel new configuration saved on disk 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.987 # +vote-for-leader 499007c98c0a165b13e026a4443ceb890695c191 1 2024-01-05 18:10:08 1:X 05 Jan 2024 10:10:08.990 * 92d9a1419be1256d1715df2aa17cea4bbacfdf60 voted for 92d9a1419be1256d1715df2aa17cea4bbacfdf60 1 2024-01-05 18:10:09 1:X 05 Jan 2024 10:10:09.021 * bcfaed15fb01e7ad03b013fe5e964479c1a1f138 voted for 499007c98c0a165b13e026a4443ceb890695c191 1 2024-01-05 18:10:09 1:X 05 Jan 2024 10:10:09.054 # +elected-leader master mymaster 172.30.1.2 6379 2024-01-05 18:10:09 1:X 05 Jan 2024 10:10:09.054 # +failover-state-select-slave master mymaster 172.30.1.2 6379 2024-01-05 18:10:09 1:X 05 Jan 2024 10:10:09.125 # +selected-slave slave 172.30.1.3:6379 172.30.1.3 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:09 1:X 05 Jan 2024 10:10:09.125 * +failover-state-send-slaveof-noone slave 172.30.1.3:6379 172.30.1.3 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:09 1:X 05 Jan 2024 10:10:09.209 * +failover-state-wait-promotion slave 172.30.1.3:6379 172.30.1.3 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.033 * Sentinel new configuration saved on disk 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.033 # +promoted-slave slave 172.30.1.3:6379 172.30.1.3 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.033 # +failover-state-reconf-slaves master mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.094 * +slave-reconf-sent slave 172.30.1.4:6379 172.30.1.4 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.262 * +slave-reconf-inprog slave 172.30.1.4:6379 172.30.1.4 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.262 * +slave-reconf-done slave 172.30.1.4:6379 172.30.1.4 6379 @ mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.338 # +failover-end master mymaster 172.30.1.2 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.338 # +switch-master mymaster 172.30.1.2 6379 172.30.1.3 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.338 * +slave slave 172.30.1.4:6379 172.30.1.4 6379 @ mymaster 172.30.1.3 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.338 * +slave slave 172.30.1.2:6379 172.30.1.2 6379 @ mymaster 172.30.1.3 6379 2024-01-05 18:10:10 1:X 05 Jan 2024 10:10:10.373 * Sentinel new configuration saved on disk
接着让我们看看172.30.1.3
的日志,也就是redis-replica1
的日志,可以看到与主节点连接失败后,它开启了主节点模式MASTER MODE enabled
。
language-txt 1 2 3 4 5 6 7 8 9 10 11 2024-01-05 18:10:03 1:S 05 Jan 2024 10:10:03.812 * Reconnecting to MASTER 172.30.1.2:6379 2024-01-05 18:10:03 1:S 05 Jan 2024 10:10:03.813 * MASTER <-> REPLICA sync started 2024-01-05 18:10:03 1:S 05 Jan 2024 10:10:03.813 # Error condition on socket for SYNC: Connection refused 2024-01-05 18:10:04 1:S 05 Jan 2024 10:10:04.582 * Connecting to MASTER 172.30.1.2:6379 2024-01-05 18:10:04 1:S 05 Jan 2024 10:10:04.582 * MASTER <-> REPLICA sync started 2024-01-05 18:10:09 1:M 05 Jan 2024 10:10:09.209 * Discarding previously cached master state. 2024-01-05 18:10:09 1:M 05 Jan 2024 10:10:09.209 * Setting secondary replication ID to 5032654a1279c56d758c93a4eb1c4b89c99975a9, valid up to offset: 40756. New replication ID is d3464601d550e1159d91234567a366fa1f1a0b5e 2024-01-05 18:10:09 1:M 05 Jan 2024 10:10:09.209 * MASTER MODE enabled (user request from 'id=8 addr=172.30.1.11:55710 laddr=172.30.1.3:6379 fd=13 name=sentinel-499007c9-cmd age=199 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=188 qbuf-free=20286 argv-mem=4 multi-mem=169 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23717 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver=') 2024-01-05 18:10:09 1:M 05 Jan 2024 10:10:09.229 * CONFIG REWRITE executed with success. 2024-01-05 18:10:10 1:M 05 Jan 2024 10:10:10.120 * Replica 172.30.1.4:6379 asks for synchronization 2024-01-05 18:10:10 1:M 05 Jan 2024 10:10:10.120 * Partial resynchronization request from 172.30.1.4:6379 accepted. Sending 567 bytes of backlog starting from offset 40756.
并且还有redis-replica2
的日志,里面会显示将数据同步请求地址变成了172.30.1.3
而不是先前的172.30.1.2
。
接着连接redis-replica1
容器看看,发现这个节点以前作为从节点时是只读
节点,现在可以写入数据了。
language-txt 1 2 3 4 5 6 7 root@1eefea35001f:/data# redis-cli 127.0.0.1:6379> auth 1009 OK 127.0.0.1:6379> set num 8766 OK 127.0.0.1:6379> get num "8766"
并且会发现另外两个节点变成只读
了,同时,即使先前的主节点又恢复正常了,它不会去夺回master
地位。
测试成功。