Docker运行RabbitMQ并使用SpringAMQP操作

Docker运行RabbitMQ并使用SpringAMQP操作

一、RabbitMQ运行

拉取docker镜像

language-bash
1
docker pull rabbitmq:3-management

基础运行命令

language-bash
1
2
3
4
5
6
7
8
docker run \
-e RABBITMQ_DEFAULT_USER=rabbitmq \
-e RABBITMQ_DEFAULT_PASS=rabbitmq \
--name rabbitmq \
-p 15672:15672 \
-p 5672:5672 \
-d \
rabbitmq:3-management

15672是网页后台管理系统,5672是给服务用的。

官方入门教程可以看这里RabbitMQ Tutorials — RabbitMQ

二、整合SpringAMQP

1. 引入依赖

language-xml
1
2
3
4
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
language-yml
1
2
3
4
5
6
spring:
rabbitmq:
host: localhost # rabbitMQ的ip地址
port: 5672 # rabbitMQ服务端口
username: rabbitmq
password: rabbitmq

三、测试

这边采用常用的消费者-生产者模型,使用默认的Direct类型exchange。不懂的可以先继续学习rabbitmq再来实践。

1. 消费者

在消费者服务随便新建一个listener

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Slf4j
@Component
public class SpringRabbitListener {


@RabbitListener(bindings = @QueueBinding(
value = @Queue(name = "simple.queue"),
exchange = @Exchange(name = "simple.exchange"),
key = "simple"
))
public void listenSimpleQueue(String msg) {

log.info("消费者接收到simple.queue的消息:【" + msg + "】");
}
}

2. 生产者

在生产者服务的Test模块新建一个测试

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class SpringAmqpTest {

@Autowired
private RabbitTemplate rabbitTemplate;

@Test
public void testSendMessage2SimpleQueue() throws InterruptedException {

String message = "hello, spring amqp!";
rabbitTemplate.convertAndSend("simple.exchange", "simple", message);
}
}

3. 运行

先启动消费者。
登录http://localhost:15672/,可以看到simple.exchangesimple.queue已被创建。

然后启动测试testSendMessage2SimpleQueue,出现类似以下日志,消息发送成功。

language-txt
1
2
12:17:43:771  INFO 21064 --- [           main] o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: [localhost:5672]
12:17:43:808 INFO 21064 --- [ main] o.s.a.r.c.CachingConnectionFactory : Created new connection: rabbitConnectionFactory#50f40653:0/SimpleConnection@536d97f8 [delegate=amqp://rabbitmq@127.0.0.1:5672/, localPort= 59641]

消费者出现类似以下日志,收到消息。

language-txt
1
2
3
12:17:31:074  INFO 8924 --- [           main] o.s.a.r.c.CachingConnectionFactory       : Created new connection: rabbitConnectionFactory#2a27cb34:0/SimpleConnection@671facee [delegate=amqp://rabbitmq@127.0.0.1:5672/, localPort= 59634]
12:17:31:141 INFO 8924 --- [ main] cn.itcast.mq.ConsumerApplication : Started ConsumerApplication in 1.011 seconds (JVM running for 1.462)
12:17:43:848 INFO 8924 --- [ntContainer#0-1] c.i.mq.listener.SpringRabbitListener : 消费者接收到simple.queue的消息:【hello, spring amqp!】
多级缓存架构(五)缓存同步

多级缓存架构(五)缓存同步

通过本文章,可以完成多级缓存架构中的缓存同步。

一、Canal服务

1. mysql添加canal用户

连接在上一次multiCache项目中运行的mysql容器,创建canal用户。

language-sql
1
2
3
4
CREATE USER canal IDENTIFIED BY 'canal';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

2. mysql配置文件

docker/mysql/conf/my.cnf添加如下配置

language-txt
1
2
3
4
server-id=1000
log-bin=/var/lib/mysql/mysql-bin
binlog-do-db=heima
binlog_format=row

3. canal配置文件

添加canal服务块到docker-compose.yml,如下

language-yml
1
2
3
4
5
6
7
8
9
10
11
12
13
canal:
container_name: canal
image: canal/canal-server:v1.1.7
volumes:
- ./canal/logs:/home/admin/canal-server/logs
- ./canal/conf:/home/admin/canal-server/conf
ports:
- "11111:11111"
depends_on:
- mysql
networks:
multi-cache:
ipv4_address: 172.30.3.7
language-bash
1
docker pull canal/canal-server:v1.1.7

任意启动一个canal-server容器,将里面的/home/admin/canal-server/conf文件夹复制到宿主机,对应docker/canal/conf文件夹。
删除此临时容器。

修改docker/canal/conf/canal.properties如下条目

language-dart
1
2
canal.destinations=example
canal.instance.tsdb.enable=true

修改docker/canal/conf/example/instance.properties如下条目

language-dart
1
2
3
4
5
6
7
canal.instance.master.address=172.30.3.2:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.connectionCharset = UTF-8
canal.instance.tsdb.enable=true
canal.instance.gtidon=false
canal.instance.filter.regex=heima\\..*

二、引入依赖

pom.xml

language-xml
1
2
3
4
5
<dependency>
<groupId>top.javatool</groupId>
<artifactId>canal-spring-boot-starter</artifactId>
<version>1.2.1-RELEASE</version>
</dependency>

application.yml

language-yml
1
2
3
canal:
destination: example
server: 172.30.3.7:11111

三、监听Canal消息

这是canal-spring-boot-starter官方仓库,含使用文档

新建canal.ItemHandler类,内容如下

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
package com.heima.item.canal;

import com.github.benmanes.caffeine.cache.Cache;
import com.heima.item.config.RedisHandler;
import com.heima.item.pojo.Item;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import top.javatool.canal.client.annotation.CanalTable;
import top.javatool.canal.client.handler.EntryHandler;

@CanalTable(value = "tb_item")
@Component
public class ItemHandler implements EntryHandler<Item> {


@Autowired
private RedisHandler redisHandler;
@Autowired
private Cache<Long, Item> itemCache;

@Override
public void insert(Item item) {

itemCache.put(item.getId(), item);
redisHandler.saveItem(item);
}

@Override
public void update(Item before, Item after) {

itemCache.put(after.getId(), after);
redisHandler.saveItem(after);
}

@Override
public void delete(Item item) {

itemCache.invalidate(item.getId());
redisHandler.deleteItemById(item.getId());
}
}

修改pojo.Item类,如下

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
package com.heima.item.pojo;

import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
import org.springframework.data.annotation.Id;
import org.springframework.data.annotation.Transient;

import java.util.Date;

@Data
@TableName("tb_item")
public class Item {

@TableId(type = IdType.AUTO)
@Id
private Long id;//商品id
private String name;//商品名称
private String title;//商品标题
private Long price;//价格(分)
private String image;//商品图片
private String category;//分类名称
private String brand;//品牌名称
private String spec;//规格
private Integer status;//商品状态 1-正常,2-下架
private Date createTime;//创建时间
private Date updateTime;//更新时间
@TableField(exist = false)
@Transient
private Integer stock;
@TableField(exist = false)
@Transient
private Integer sold;
}

四、运行

到此为止,docker-compose.yml内容应该如下

language-yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
version: '3.8'

networks:
multi-cache:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.30.3.0/24

services:
mysql:
container_name: mysql
image: mysql:8
volumes:
- ./mysql/conf/my.cnf:/etc/mysql/conf.d/my.cnf
- ./mysql/data:/var/lib/mysql
- ./mysql/logs:/logs
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=1009
networks:
multi-cache:
ipv4_address: 172.30.3.2

nginx:
container_name: nginx
image: nginx:stable
volumes:
- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf
- ./nginx/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./nginx/dist:/usr/share/nginx/dist
ports:
- "8080:8080"
networks:
multi-cache:
ipv4_address: 172.30.3.3

canal:
container_name: canal
image: canal/canal-server:v1.1.7
volumes:
- ./canal/logs:/home/admin/canal-server/logs
- ./canal/conf:/home/admin/canal-server/conf
ports:
- "11111:11111"
depends_on:
- mysql
networks:
multi-cache:
ipv4_address: 172.30.3.7

openresty1:
container_name: openresty1
image: openresty/openresty:1.21.4.3-3-jammy-amd64
volumes:
- ./openresty1/conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf
- ./openresty1/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./openresty1/lua:/usr/local/openresty/nginx/lua
- ./openresty1/lualib/common.lua:/usr/local/openresty/lualib/common.lua
networks:
multi-cache:
ipv4_address: 172.30.3.11

redis:
container_name: redis
image: redis:7.2
volumes:
- ./redis/redis.conf:/usr/local/etc/redis/redis.conf
ports:
- "6379:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
multi-cache:
ipv4_address: 172.30.3.21

删除原来的multiCache,重新启动各项服务。

language-bash
1
docker-compose -p multi-cache up -d

启动springboot程序。

五、测试

springboot不断输入类似如下日志,属于正常监听canal消息中。

language-txt
1
2
3
09:27:17:175  INFO 1 --- [l-client-thread] t.j.c.client.client.AbstractCanalClient  : 获取消息 Message[id=-1,entries=[],raw=false,rawEntries=[]]
09:27:18:177 INFO 1 --- [l-client-thread] t.j.c.client.client.AbstractCanalClient : 获取消息 Message[id=-1,entries=[],raw=false,rawEntries=[]]
09:27:19:178 INFO 1 --- [l-client-thread] t.j.c.client.client.AbstractCanalClient : 获取消息 Message[id=-1,entries=[],raw=false,rawEntries=[]]

访问http://localhost:8081/item/10001,此时信息为tomcat查询数据库所得数据,而后存入Caffeine缓存。
访问http://localhost:8080/item.html?id=10001,此时信息为Redis缓存数据。

然后,
访问http://localhost:8081/来到商品管理页面。

修改id=10001的数据的商品分类

确认后
springboot日志出现类似如下日志

language-txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
09:31:29:234  INFO 1 --- [l-client-thread] t.j.c.client.client.AbstractCanalClient  : 获取消息 Message[id=1,entries=[header {
version: 1
logfileName: "binlog.000007"
logfileOffset: 236
serverId: 1
serverenCode: "UTF-8"
executeTime: 1705051889000
sourceType: MYSQL
schemaName: ""
tableName: ""
eventLength: 93
}
entryType: TRANSACTIONBEGIN
storeValue: " \r"
, header {
version: 1
logfileName: "binlog.000007"
logfileOffset: 411
serverId: 1
serverenCode: "UTF-8"
executeTime: 1705051889000
sourceType: MYSQL
schemaName: "heima"
tableName: "tb_item"
eventLength: 626
eventType: UPDATE
props {
key: "rowsCount"
value: "1"
}
}
entryType: ROWDATA
storeValue: "\bV\020\002P\000b\332\n\n&\b\000\020\373\377\377\377\377\377\377\377\377\001\032\002id \001(\0000\000B\00510001R\006bigint\nd\b\001\020\f\032\005title \000(\0000\000BCRIMOWA 21\345\257\270\346\211\230\350\277\220\347\256\261\346\213\211\346\235\206\347\256\261 SALSA AIR\347\263\273\345\210\227\346\236\234\347\273\277\350\211\262 820.70.36.4R\fvarchar(264)\n)\b\002\020\f\032\004name \000(\0000\000B\tSALSA AIRR\fvarchar(128)\n)\b\003\020\373\377\377\377\377\377\377\377\377\001\032\005price \000(\0000\000B\00516900R\006bigint\n\226\001\b\004\020\f\032\005image \000(\0000\000Buhttps://m.360buyimg.com/mobilecms/s720x720_jfs/t6934/364/1195375010/84676/e9f2c55f/597ece38N0ddcbc77.jpg!q70.jpg.webpR\fvarchar(200)\n0\b\005\020\f\032\bcategory \000(\0000\000B\f\346\213\211\346\235\206\347\256\261777R\fvarchar(200)\n\'\b\006\020\f\032\005brand \000(\0000\000B\006RIMOWAR\fvarchar(100)\nG\b\a\020\f\032\004spec \000(\0000\000B\'{\"\351\242\234\350\211\262\": \"\347\272\242\350\211\262\", \"\345\260\272\347\240\201\": \"26\345\257\270\"}R\fvarchar(200)\n\032\b\b\020\004\032\006status \000(\0000\000B\0011R\003int\n6\b\t\020]\032\vcreate_time \000(\0000\000B\0232019-05-01 00:00:00R\bdatetime\n6\b\n\020]\032\vupdate_time \000(\0000\000B\0232019-05-01 00:00:00R\bdatetime\022&\b\000\020\373\377\377\377\377\377\377\377\377\001\032\002id \001(\0000\000B\00510001R\006bigint\022d\b\001\020\f\032\005title \000(\0000\000BCRIMOWA 21\345\257\270\346\211\230\350\277\220\347\256\261\346\213\211\346\235\206\347\256\261 SALSA AIR\347\263\273\345\210\227\346\236\234\347\273\277\350\211\262 820.70.36.4R\fvarchar(264)\022)\b\002\020\f\032\004name \000(\0000\000B\tSALSA AIRR\fvarchar(128)\022)\b\003\020\373\377\377\377\377\377\377\377\377\001\032\005price \000(\0000\000B\00516900R\006bigint\022\226\001\b\004\020\f\032\005image \000(\0000\000Buhttps://m.360buyimg.com/mobilecms/s720x720_jfs/t6934/364/1195375010/84676/e9f2c55f/597ece38N0ddcbc77.jpg!q70.jpg.webpR\fvarchar(200)\0220\b\005\020\f\032\bcategory \000(\0010\000B\f\346\213\211\346\235\206\347\256\261888R\fvarchar(200)\022\'\b\006\020\f\032\005brand \000(\0000\000B\006RIMOWAR\fvarchar(100)\022G\b\a\020\f\032\004spec \000(\0000\000B\'{\"\351\242\234\350\211\262\": \"\347\272\242\350\211\262\", \"\345\260\272\347\240\201\": \"26\345\257\270\"}R\fvarchar(200)\022\032\b\b\020\004\032\006status \000(\0000\000B\0011R\003int\0226\b\t\020]\032\vcreate_time \000(\0000\000B\0232019-05-01 00:00:00R\bdatetime\0226\b\n\020]\032\vupdate_time \000(\0000\000B\0232019-05-01 00:00:00R\bdatetime"
],raw=false,rawEntries=[]]
09:31:30:572 INFO 1 --- [l-client-thread] t.j.c.client.client.AbstractCanalClient : 获取消息 Message[id=2,entries=[header {
version: 1
logfileName: "binlog.000007"
logfileOffset: 1037
serverId: 1
serverenCode: "UTF-8"
executeTime: 1705051889000
sourceType: MYSQL
schemaName: ""
tableName: ""
eventLength: 31
}
entryType: TRANSACTIONEND
storeValue: "\022\00287"
],raw=false,rawEntries=[]]

这里可以先用redis连接工具查询数据,发现rediis已被更新。

再次访问http://localhost:8081/item/10001直接向springbootcontroller发送请求,发现caffeine数据更新,并且springboot日志没有出现查询记录,说明走的是caffeine

多级缓存架构(三)OpenResty Lua缓存

多级缓存架构(三)OpenResty Lua缓存

通过本文章,可以完成多级缓存架构中的Lua缓存。

一、nginx服务

docker/docker-compose.yml中添加nginx服务块。

language-yaml
1
2
3
4
5
6
7
8
9
10
11
12
nginx:
container_name: nginx
image: nginx:stable
volumes:
- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf
- ./nginx/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./nginx/dist:/usr/share/nginx/dist
ports:
- "8080:8080"
networks:
multi-cache:
ipv4_address: 172.30.3.3

删除原来docker里的multiCache项目并停止springboot应用。

nginx部分配置如下,监听端口为8080,并且将请求反向代理至172.30.3.11,下一小节,将openresty固定在172.30.3.11

language-txt
1
2
3
4
5
6
7
8
9
10
11
12
13
upstream nginx-cluster {
server 172.30.3.11;
}

server {
listen 8080;
listen [::]:8080;
server_name localhost;

location /api {
proxy_pass http://nginx-cluster;
}
}

重新启动multiCache看看nginx前端网页效果。

language-css
1
docker-compose -p multi-cache up -d

访问http://localhost:8080/item.html?id=10001查询id=10001商品页

这里是假数据,前端页面会向/api/item/10001发送数据请求。

二、OpenResty服务

1. 服务块定义

docker/docker-compose.yml中添加openresty1服务块。

language-yaml
1
2
3
4
5
6
7
8
9
10
11
openresty1:
container_name: openresty1
image: openresty/openresty:1.21.4.3-3-jammy-amd64
volumes:
- ./openresty1/conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf
- ./openresty1/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./openresty1/lua:/usr/local/openresty/nginx/lua
- ./openresty1/lualib/common.lua:/usr/local/openresty/lualib/common.lua
networks:
multi-cache:
ipv4_address: 172.30.3.11

2. 配置修改

前端向后端发送/api/item/10001请求关于id=10001商品信息。

根据nginx的配置内容,这个请求首先被nginx拦截,反向代理到172.30.3.11 (即openresty1)。

language-txt
1
2
3
4
5
6
7
8
9
upstream nginx-cluster {
server 172.30.3.11;
}

server {
location /api {
proxy_pass http://nginx-cluster;
}
}

openresty1收到的也是/api/item/10001,同时,openresty/api/item/(\d+)请求代理到指定lua程序,在lua程序中完成数据缓存。

因此,openrestyconf/conf.d/default.conf如下

language-txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
upstream tomcat-cluster {
hash $request_uri;
server 172.30.3.4:8081;
# server 172.30.3.5:8081;
}

server {
listen 80;
listen [::]:80;
server_name localhost;

# intercept /item and join lua
location ~ /api/item/(\d+) {
default_type application/json;
content_by_lua_file lua/item.lua;
}

# intercept lua and redirect to back-end
location /path/ {
rewrite ^/path/(.*)$ /$1 break;
proxy_pass http://tomcat-cluster;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/dist;
}
}

conf/nginx.confhttp块最后添加3行,引入依赖。

language-txt
1
2
3
4
5
6
#lua 模块
lua_package_path "/usr/local/openresty/lualib/?.lua;;";
#c模块
lua_package_cpath "/usr/local/openresty/lualib/?.so;;";
#本地缓存
lua_shared_dict item_cache 150m;

3. Lua程序编写

common.lua被挂载到lualib,表示可以被其他lua当做库使用,内容如下

language-lua
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
-- 创建一个本地缓存对象item_cache
local item_cache = ngx.shared.item_cache;

-- 函数,向openresty本身发送类似/path/item/10001请求,根据conf配置,将被删除/path前缀并代理至tomcat程序
local function read_get(path, params)
local rsp = ngx.location.capture('/path'..path,{

method = ngx.HTTP_GET,
args = params,
})
if not rsp then
ngx.log(ngx.ERR, "http not found, path: ", path, ", args: ", params);
ngx.exit(404)
end
return rsp.body
end

-- 函数,如果本地有缓存,使用缓存,如果没有代理到tomcat然后将数据存入缓存
local function read_data(key, expire, path, params)
-- query local cache
local rsp = item_cache:get(key)
-- query tomcat
if not rsp then
ngx.log(ngx.ERR, "redis cache miss, try tomcat, key: ", key)
rsp = read_get(path, params)
end
-- write into local cache
item_cache:set(key, rsp, expire)
return rsp
end

local _M = {

read_data = read_data
}

return _M

item.lua是处理来自形如/api/item/10001请求的程序,内容如下

language-lua
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
-- include
local commonUtils = require('common')
local cjson = require("cjson")

-- get url params 10001
local id = ngx.var[1]
-- redirect item, 缓存过期时间1800s, 适合长时间不改变的数据
local itemJson = commonUtils.read_data("item:id:"..id, 1800,"/item/"..id,nil)
-- redirect item/stock, 缓存过期时间4s, 适合经常改变的数据
local stockJson = commonUtils.read_data("item:stock:id:"..id, 4 ,"/item/stock/"..id, nil)
-- json2table
local item = cjson.decode(itemJson)
local stock = cjson.decode(stockJson)
-- combine item and stock
item.stock = stock.stock
item.sold = stock.sold
-- return result
ngx.say(cjson.encode(item))

4. 总结

  1. 这里luaitem(tb_item表)和stock(tb_stock表)两个信息都有缓存,并使用cjson库将两者合并后返回到前端。
  2. 关于expire时效性的问题,如果后台改变了数据,但是openresty关于此数据的缓存未过期,前端得到的是旧数据
  3. 大致来说openresty = nginx + lua,不仅具有nginx反向代理的能力,还能介入lua程序进行扩展。

三、运行

到此为止,docker-compose.yml应该如下

language-yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
version: '3.8'

networks:
multi-cache:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.30.3.0/24

services:
mysql:
container_name: mysql
image: mysql:8
volumes:
- ./mysql/conf/my.cnf:/etc/mysql/conf.d/my.cnf
- ./mysql/data:/var/lib/mysql
- ./mysql/logs:/logs
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=1009
networks:
multi-cache:
ipv4_address: 172.30.3.2

nginx:
container_name: nginx
image: nginx:stable
volumes:
- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf
- ./nginx/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./nginx/dist:/usr/share/nginx/dist
ports:
- "8080:8080"
networks:
multi-cache:
ipv4_address: 172.30.3.3

openresty1:
container_name: openresty1
image: openresty/openresty:1.21.4.3-3-jammy-amd64
volumes:
- ./openresty1/conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf
- ./openresty1/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./openresty1/lua:/usr/local/openresty/nginx/lua
- ./openresty1/lualib/common.lua:/usr/local/openresty/lualib/common.lua
networks:
multi-cache:
ipv4_address: 172.30.3.11

启动各项服务

language-bash
1
docker-compose -p multi-cache up -d

启动springboot程序。

四、测试

清空openresty容器日志。
访问http://localhost:8080/item.html?id=10001
查看openresty容器日志,可以看到两次commonUtils.read_data都没有缓存,于是代理到tomcat,可以看到springboot日志出现查询相关记录。

language-txt
1
2
3
2024-01-12 11:45:53 2024/01/12 03:45:53 [error] 7#7: *1 [lua] common.lua:99: read_data(): redis cache miss, try tomcat, key: item:id:10001, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 11:45:53 2024/01/12 03:45:53 [error] 7#7: *1 [lua] common.lua:99: read_data(): redis cache miss, try tomcat, key: item:stock:id:10001 while sending to client, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 11:45:53 172.30.3.3 - - [12/Jan/2024:03:45:53 +0000] "GET /api/item/10001 HTTP/1.0" 200 486 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

再次访问此网址,强制刷新+禁用浏览器缓存+更换浏览器
间隔超过4s但小于1800s时,日志如下,只出现一次miss。

language-txt
1
2
2024-01-12 11:48:04 2024/01/12 03:48:04 [error] 7#7: *4 [lua] common.lua:99: read_data(): redis cache miss, try tomcat, key: item:stock:id:10001, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 11:48:04 172.30.3.3 - - [12/Jan/2024:03:48:04 +0000] "GET /api/item/10001 HTTP/1.0" 200 486 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

再次访问此网址,强制刷新+禁用浏览器缓存+更换浏览器
间隔小于4s,日志如下,未出现miss。

language-txt
1
2024-01-12 11:49:16 172.30.3.3 - - [12/Jan/2024:03:49:16 +0000] "GET /api/item/10001 HTTP/1.0" 200 486 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

五、高可用集群

1. openresty

对于openresty高可用,可以部署多个openresty docker实例,并在nginxdocker/nginx/conf/conf.d/default.confupstream nginx-cluster将多个openresty地址添加进去即可。比如

language-txt
1
2
3
4
5
6
7
upstream nginx-cluster {
hash $request_uri;
# hash $request_uri consistent;
server 172.30.3.11;
server 172.30.3.12;
server 172.30.3.13;
}

多个openresty 无论是conf还是lua都保持一致即可。
并且使用hash $request_uri负载均衡作为反向代理策略,防止同一请求被多个实例缓存数据。

2. tomcat

对于springboot程序高可用,也是类似。可以部署多个springboot docker实例,并在openresty docker/openresty1/conf/conf.d/default.confupstream nginx-cluster将多个springboot地址添加进去即可。比如

language-txt
1
2
3
4
5
upstream tomcat-cluster {
hash $request_uri;
server 172.30.3.4:8081;
server 172.30.3.5:8081;
}
多级缓存架构(二)Caffeine进程缓存

多级缓存架构(二)Caffeine进程缓存

通过本文章,可以完成多级缓存架构中的进程缓存。

一、引入依赖

item-service中引入caffeine依赖

language-xml
1
2
3
4
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
</dependency>

二、实现进程缓存

这是Caffeine官方文档地址

1. 配置Config类

创建config.CaffeineConfig

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@Configuration
public class CaffeineConfig {

@Bean
public Cache<Long, Item> itemCache(){

return Caffeine.newBuilder()
.initialCapacity(100)
.maximumSize(10_000)
.build();
}

@Bean
public Cache<Long, ItemStock> stockCache(){

return Caffeine.newBuilder()
.initialCapacity(100)
.maximumSize(10_000)
.build();
}
}

2. 修改controller

ItemController中注入两个Cache对象,并修改业务逻辑

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
@RestController
@RequestMapping("item")
public class ItemController {


@Autowired
private IItemService itemService;
@Autowired
private IItemStockService stockService;
@Autowired
private Cache<Long, Item> itemCache;
@Autowired
private Cache<Long, ItemStock> stockCache;

@GetMapping("/{id}")
public Item findById(@PathVariable("id") Long id){

return itemCache.get(id, key->
itemService.query()
.ne("status", 3).eq("id", id)
.one()
);
// return itemService.query()
// .ne("status", 3).eq("id", id)
// .one();
}

@GetMapping("/stock/{id}")
public ItemStock findStockById(@PathVariable("id") Long id){

return stockCache.get(id, key->
stockService.getById(id)
);
// return stockService.getById(id);
}
}

三、运行

Idea结合Docker将springboot放入docker容器中运行,并指定使用multi-cache_multi-cache网络,以及固定172.30.3.4地址。
详细参考如下文章

启动好后,可以看到springboot容器和mysql容器处于同一网络下。(Docker Desktop for Windows插件PortNavigator)

四、测试

访问http://localhost:8081/item/10001可以看到springboot日志输出如下

language-txt
1
2
3
02:45:58:841 DEBUG 1 --- [nio-8081-exec-1] c.h.item.mapper.ItemMapper.selectOne     : ==>  Preparing: SELECT id,name,title,price,image,category,brand,spec,status,create_time,update_time FROM tb_item WHERE (status <> ? AND id = ?)
02:45:58:889 DEBUG 1 --- [nio-8081-exec-1] c.h.item.mapper.ItemMapper.selectOne : ==> Parameters: 3(Integer), 10001(Long)
02:45:58:951 DEBUG 1 --- [nio-8081-exec-1] c.h.item.mapper.ItemMapper.selectOne : <== Total: 1

当我们二次访问此网址,强制刷新+禁用浏览器缓存+更换浏览器,springboot日志都没有新的查询记录,说明使用了Caffeine缓存。

基于Docker Compose单机实现多级缓存架构2024

基于Docker Compose单机实现多级缓存架构2024

一、环境参考

Name Version
Docker Desktop for Windows 4.23.0
Openjdk 8
MySQL 8.2.0
Redis 7.2
Canal 1.1.7
OpenResty 1.21.4.3-3-jammy-amd64
Lua -
Caffeine -

二、专栏简介

多级缓存实现过程比较长,将拆分为多个文章分步讲述。如果一切顺利,大致会得到如下一个多级缓存架构:

本专栏主要对Lua缓存Redis缓存Caffeine缓存进行实践,以及缓存同步实践。依次为以下几篇:

  1. 多级缓存架构(一)项目初始化
  2. 多级缓存架构(二)Caffeine进程缓存
  3. 多级缓存架构(三)OpenResty Lua缓存
  4. 多级缓存架构(四)Redis缓存
  5. 多级缓存架构(五)缓存同步

三、扩展

对于高可用,集群等扩展,例如下图的构造,本专栏只包含部分展开但并不提供实践指导

多级缓存架构(四)Redis缓存

多级缓存架构(四)Redis缓存

通过本文章,可以完成多级缓存架构中的Redis缓存。

一、Redis服务

docker/docker-compose.yml中,添加redis服务块

language-yml
1
2
3
4
5
6
7
8
9
10
11
redis:
container_name: redis
image: redis:7.2
volumes:
- ./redis/redis.conf:/usr/local/etc/redis/redis.conf
ports:
- "6379:6379"
command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
networks:
multi-cache:
ipv4_address: 172.30.3.21

二、Redis缓存预热

spirngboot项目启动时,将固定的热点数据提前加载到redis中。

1. 引入依赖

pom.xml添加如下依赖

language-xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
<version>6.1.4.RELEASE</version> <!-- 或更高版本 -->
</dependency>
<!-- https://mvnrepository.com/artifact/com.alibaba.fastjson2/fastjson2 -->
<dependency>
<groupId>com.alibaba.fastjson2</groupId>
<artifactId>fastjson2</artifactId>
<version>2.0.41</version>
</dependency>

application.yml添加如下配置

language-yml
1
2
3
spring:
redis:
host: 172.30.3.21

2. handler类实现

新建config.RedisHandler类,内容如下,主要是重写afterPropertiesSet,完成缓存预热逻辑,saveItemdeleteItemById函数给之后的章节使用。

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
@Component
public class RedisHandler implements InitializingBean {

@Autowired
private StringRedisTemplate redisTemplate;
@Autowired
private IItemService itemService;
@Autowired
private IItemStockService stockService;
@Override
public void afterPropertiesSet() throws Exception {

List<Item> itemList = itemService.list();
for (Item item : itemList) {

String json = JSON.toJSONString(item);
redisTemplate.opsForValue().set("item:id:"+item.getId(), json);
}
List<ItemStock> stockList = stockService.list();
for (ItemStock stock : stockList) {

String json = JSON.toJSONString(stock);
redisTemplate.opsForValue().set("item:stock:id:"+stock.getId(), json);
}
}

public void saveItem(Item item){

String json = JSON.toJSONString(item);
redisTemplate.opsForValue().set("item:id:"+item.getId(), json);
}

public void deleteItemById(Long id){

redisTemplate.delete("item:id:"+id);
}
}

三、整合Redis缓存

改进openrestydocker/openresty1/lualib/common.lua,如下

language-lua
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
local redis = require('resty.redis')
local red = redis:new()
red:set_timeouts(1000, 1000, 1000)
-- 创建一个本地缓存对象item_cache
local item_cache = ngx.shared.item_cache;

-- 关闭redis连接的工具方法,其实是放入连接池
local function close_redis(red)
local pool_max_idle_time = 10000 -- 连接的空闲时间,单位是毫秒
local pool_size = 100 --连接池大小
local ok, err = red:set_keepalive(pool_max_idle_time, pool_size)
if not ok then
ngx.log(ngx.ERR, "放入redis连接池失败: ", err)
end
end

-- 查询redis的方法 ip和port是redis地址,key是查询的key
local function read_redis(ip, port, key)
-- 获取一个连接
local ok, err = red:connect(ip, port)
if not ok then
ngx.log(ngx.ERR, "连接redis失败 : ", err)
return nil
end
-- 查询redis
local resp, err = red:get(key)
-- 查询失败处理
if not resp then
ngx.log(ngx.ERR, "查询Redis失败: ", err, ", key = " , key)
end
--得到的数据为空处理
if resp == ngx.null then
resp = nil
ngx.log(ngx.ERR, "查询Redis数据为空, key = ", key)
end
close_redis(red)
return resp
end

-- 函数,向openresty本身发送类似/path/item/10001请求,根据conf配置,将被删除/path前缀并代理至tomcat程序
local function read_get(path, params)
local rsp = ngx.location.capture('/path'..path,{

method = ngx.HTTP_GET,
args = params,
})
if not rsp then
ngx.log(ngx.ERR, "http not found, path: ", path, ", args: ", params);
ngx.exit(404)
end
return rsp.body
end

-- 函数,如果本地有缓存,使用缓存,如果没有代理到tomcat然后将数据存入缓存
local function read_data(key, expire, path, params)
-- query local cache
local rsp = item_cache:get(key)
-- query redis
if not rsp then
ngx.log(ngx.ERR, "local cache miss, try redis, key: ", key)
rsp = read_redis("172.30.3.21", 6379, key)
if not rsp then
ngx.log(ngx.ERR, "redis cache miss, try tomcat, key: ", key)
rsp = read_get(path, params)
end
end
-- write into local cache
item_cache:set(key, rsp, expire)
return rsp
end

local _M = {

read_get = read_get,
read_redis = read_redis,
read_data = read_data
}

return _M

item.lua不需要用改动。

四、运行

到此为止,docker-compose.yml内容应该如下

language-yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
version: '3.8'

networks:
multi-cache:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.30.3.0/24

services:
mysql:
container_name: mysql
image: mysql:8
volumes:
- ./mysql/conf/my.cnf:/etc/mysql/conf.d/my.cnf
- ./mysql/data:/var/lib/mysql
- ./mysql/logs:/logs
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=1009
networks:
multi-cache:
ipv4_address: 172.30.3.2

nginx:
container_name: nginx
image: nginx:stable
volumes:
- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf
- ./nginx/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./nginx/dist:/usr/share/nginx/dist
ports:
- "8080:8080"
networks:
multi-cache:
ipv4_address: 172.30.3.3

openresty1:
container_name: openresty1
image: openresty/openresty:1.21.4.3-3-jammy-amd64
volumes:
- ./openresty1/conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf
- ./openresty1/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./openresty1/lua:/usr/local/openresty/nginx/lua
- ./openresty1/lualib/common.lua:/usr/local/openresty/lualib/common.lua
networks:
multi-cache:
ipv4_address: 172.30.3.11

redis:
container_name: redis
image: redis:7.2
volumes:
- ./redis/redis.conf:/usr/local/etc/redis/redis.conf
ports:
- "6379:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
multi-cache:
ipv4_address: 172.30.3.21

删除原来的multiCache,重新启动各项服务。

language-bash
1
docker-compose -p multi-cache up -d

启动springboot程序。

五、测试

1. redis缓存预热

springboot程序启动后,出现查询日志,查看redis数据库发现自动存入了数据。

2. redis缓存命中

清空openresty容器日志,访问http://localhost:8080/item.html?id=10001,查看日志,发现两次commonUtils.read_data都只触发到查询redis,没到查询tomcat

language-txt
1
2
3
2024-01-12 16:06:18 2024/01/12 08:06:18 [error] 7#7: *1 [lua] common.lua:59: read_data(): local cache miss, try redis, key: item:id:10001, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 16:06:18 2024/01/12 08:06:18 [error] 7#7: *1 [lua] common.lua:59: read_data(): local cache miss, try redis, key: item:stock:id:10001, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 16:06:18 172.30.3.3 - - [12/Jan/2024:08:06:18 +0000] "GET /api/item/10001 HTTP/1.0" 200 466 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

查看springboot程序日志,也没有查询记录,说明redis缓存命中成功。

六、高可用集群

对于redis高可用集群,可以参考以下专栏文章。
https://blog.csdn.net/m0_51390969/category_12546314.html?spm=1001.2014.3001.5482

多级缓存架构(一)项目初始化

多级缓存架构(一)项目初始化

一、项目克隆

克隆此项目到本地
https://github.com/Xiamu-ssr/MultiCache
来到start目录下,分别有以下文件夹

  • docker:docker相关文件
  • item-service:springboot项目

二、数据库准备

docker/docker-compose.yml中已经定义好如下mysql

language-yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
mysql:
container_name: mysql
image: mysql:8
volumes:
- ./mysql/conf/my.cnf:/etc/mysql/conf.d/my.cnf
- ./mysql/data:/var/lib/mysql
- ./mysql/logs:/logs
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=1009
networks:
multi-cache:
ipv4_address: 172.30.3.2

my.cnf如下

language-bash
1
2
3
4
5
[mysqld]
bind-address=0.0.0.0
skip-name-resolve
character_set_server=utf8
datadir=/var/lib/mysql

运行以下命令启动docker-compose

language-bash
1
docker-compose -p multi-cache up -d

之后使用数据库连接工具连接mysql容器,创建heima数据库,并对其执行docker/mysql/item.sql脚本。

三、项目工程准备

idea打开item-service文件夹,等待idea加载本springboot项目。

如果在docker-compose中服务ip改动,请注意一些可能关联的地方也需要做同样改动,比如item-serviceapplication.yml

language-yaml
1
2
3
4
5
6
7
8
spring:
application:
name: itemservice
datasource:
url: jdbc:mysql://172.30.3.2:3306/heima?useSSL=false&allowPublicKeyRetrieval=true
username: root
password: 1009
driver-class-name: com.mysql.cj.jdbc.Driver

观察controller

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
package com.heima.item.web;

import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.heima.item.pojo.Item;
import com.heima.item.pojo.ItemStock;
import com.heima.item.pojo.PageDTO;
import com.heima.item.service.IItemService;
import com.heima.item.service.IItemStockService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.List;
import java.util.stream.Collectors;

@RestController
@RequestMapping("item")
public class ItemController {


@Autowired
private IItemService itemService;
@Autowired
private IItemStockService stockService;

@GetMapping("list")
public PageDTO queryItemPage(
@RequestParam(value = "page", defaultValue = "1") Integer page,
@RequestParam(value = "size", defaultValue = "5") Integer size){

// 分页查询商品
Page<Item> result = itemService.query()
.ne("status", 3)
.page(new Page<>(page, size));

// 查询库存
List<Item> list = result.getRecords().stream().peek(item -> {

ItemStock stock = stockService.getById(item.getId());
item.setStock(stock.getStock());
item.setSold(stock.getSold());
}).collect(Collectors.toList());

// 封装返回
return new PageDTO(result.getTotal(), list);
}

@PostMapping
public void saveItem(@RequestBody Item item){

itemService.saveItem(item);
}

@PutMapping
public void updateItem(@RequestBody Item item) {

itemService.updateById(item);
}

@PutMapping("stock")
public void updateStock(@RequestBody ItemStock itemStock){

stockService.updateById(itemStock);
}

@DeleteMapping("/{id}")
public void deleteById(@PathVariable("id") Long id){

itemService.update().set("status", 3).eq("id", id).update();
}

@GetMapping("/{id}")
public Item findById(@PathVariable("id") Long id){

return itemService.query()
.ne("status", 3).eq("id", id)
.one();
}

@GetMapping("/stock/{id}")
public ItemStock findStockById(@PathVariable("id") Long id){

return stockService.getById(id);
}
}

SpringBoot基于Redis(7.2)分片集群实现读写分离

SpringBoot基于Redis(7.2)分片集群实现读写分离

一、前置提要

SpringBoot访问Redis分片集群和Redis哨兵模式,使用上没有什么区别。唯一的区别在于application.yml配置上不一样。

二、集群搭建

首先,无论如何,得先有一个Redis分片集群,具体可以参考下面这篇文章

搭建完成后大致得到如下图描述的一个集群。

三、SpringBoot访问分片集群

其次,具体如何结合IdeaDocker让本地开发的SpringBoot项目访问Redis分片集群,可以参考下面这篇文章

要注意的是,yaml文件要从

language-yaml
1
2
3
4
5
6
7
8
9
10
spring:
redis:
sentinel:
master: mymaster
nodes:
- 172.30.1.11:26379
- 172.30.1.12:26379
- 172.30.1.13:26379
password: 1009
password: 1009

变成

language-yaml
1
2
3
4
5
6
7
8
9
10
spring:
redis:
cluster:
nodes:
- 172.30.2.11:6379
- 172.30.2.12:6379
- 172.30.2.13:6379
- 172.30.2.21:6379
- 172.30.2.22:6379
- 172.30.2.23:6379

其余基本一致。

Docker-Compose部署Redis(v7.2)分片集群(含主从)

Docker-Compose部署Redis(v7.2)分片集群(含主从)

环境

  • docker desktop for windows 4.23.0
  • redis 7.2

目标

搭建如下图分片+主从集群。

一、前提准备

1. 文件夹结构

因为Redis 7.2 docker镜像里面没有配置文件,所以需要去redis官网下载一个复制里面的redis.conf
博主这里用的是7.2.3版本的redis.conf,这个文件就在解压后第一层文件夹里。

然后构建如下文件夹结构。

language-txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
sharding/
├── docker-compose.yaml
├── master1
│ └── conf
│ └── redis.conf
├── master2
│ └── conf
│ └── redis.conf
├── master3
│ └── conf
│ └── redis.conf
├── replica1
│ └── conf
│ └── redis.conf
├── replica2
│ └── conf
│ └── redis.conf
└── replica3
└── conf
└── redis.conf

二、配置文件

1. redis.conf

对每个redis.conf都做以下修改。分片集群的redis主从的redis.conf目前都是一样的。

language-bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
port 6379
# 开启集群功能
cluster-enabled yes
# 集群的配置文件名称,不需要我们创建,由redis自己维护
cluster-config-file /data/nodes.conf
# 节点心跳失败的超时时间
cluster-node-timeout 5000
# 持久化文件存放目录
dir /data
# 绑定地址
bind 0.0.0.0
# 让redis后台运行
daemonize no
# 保护模式
protected-mode no
# 数据库数量
databases 1
# 日志
logfile /data/run.log

2. docker-compose文件

language-yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
version: '3.8'

networks:
redis-sharding:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.30.2.0/24

services:
master1:
container_name: master1
image: redis:7.2
volumes:
- ./master1/conf:/usr/local/etc/redis
ports:
- "7001:6379"
command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
networks:
redis-sharding:
ipv4_address: 172.30.2.11

master2:
container_name: master2
image: redis:7.2
volumes:
- ./master2/conf:/usr/local/etc/redis
ports:
- "7002:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
redis-sharding:
ipv4_address: 172.30.2.12

master3:
container_name: master3
image: redis:7.2
volumes:
- ./master3/conf:/usr/local/etc/redis
ports:
- "7003:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
redis-sharding:
ipv4_address: 172.30.2.13

replica1:
container_name: replica1
image: redis:7.2
volumes:
- ./replica1/conf:/usr/local/etc/redis
ports:
- "8001:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
redis-sharding:
ipv4_address: 172.30.2.21

replica2:
container_name: replica2
image: redis:7.2
volumes:
- ./replica2/conf:/usr/local/etc/redis
ports:
- "8002:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
redis-sharding:
ipv4_address: 172.30.2.22

replica3:
container_name: replica3
image: redis:7.2
volumes:
- ./replica3/conf:/usr/local/etc/redis
ports:
- "8003:6379"
command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
networks:
redis-sharding:
ipv4_address: 172.30.2.23


需要注意以下几点

  • 这里自定义了bridge子网并限定了范围,如果该范围已经被使用,请更换。
  • 这里没有对data进行-v挂载,如果要挂载,请注意宿主机对应文件夹权限问题。

随后运行

language-bash
1
docker-compose -p redis-sharding up -d

三、构建集群

接下来所有命令都在master1容器的命令行执行

1. 自动分配主从关系

这个命令会创建了一个集群,包括三个主节点和三个从节点,每个主节点分配一个从节点作为副本,前3个ip为主节点,后3个为从节点,主节点的从节点随机分配。

language-bash
1
redis-cli --cluster create 172.30.2.11:6379 172.30.2.12:6379 172.30.2.13:6379 172.30.2.21:6379 172.30.2.22:6379 172.30.2.23:6379 --cluster-replicas 1

如果希望手动指定主从关系,看下面,否则你可以跳过这一章节了。

2.1 构建3 master集群

language-bash
1
redis-cli --cluster create 172.30.2.11:6379 172.30.2.12:6379 172.30.2.13:6379 --cluster-replicas 0

2.2 手动配置从节点

查看3个主节点的ID

language-bash
1
redis-cli -h 172.30.2.11 -p 6379 cluster nodes

下面3个命令会将3个从节点加入集群中,其中172.30.2.11可以是三个主节点的任意一个。

language-bash
1
2
3
redis-cli -h 172.30.2.21 -p 6379 cluster meet 172.30.2.11 6379
redis-cli -h 172.30.2.22 -p 6379 cluster meet 172.30.2.11 6379
redis-cli -h 172.30.2.23 -p 6379 cluster meet 172.30.2.11 6379

然后为每个从节点指定主节点。

language-bash
1
2
3
redis-cli -h 172.30.2.21 -p 6379 cluster replicate <master-ID>
redis-cli -h 172.30.2.22 -p 6379 cluster replicate <master-ID>
redis-cli -h 172.30.2.23 -p 6379 cluster replicate <master-ID>

四、测试

1. 集群结构

可以通过以下命令查看集群中每个节点的id、角色、ip、port、插槽范围等信息

language-bash
1
redis-cli -h 172.30.2.11 -p 6379 cluster nodes

2. 分片测试

往集群存入4个键值

language-bash
1
2
3
4
redis-cli -c -h 172.30.2.11 -p 6379 set key1 value1
redis-cli -c -h 172.30.2.11 -p 6379 set key2 value2
redis-cli -c -h 172.30.2.11 -p 6379 set key3 value3
redis-cli -c -h 172.30.2.11 -p 6379 set key4 value4

查看每个主节点现有的键值,会发现每个节点只有一部分键值。

language-bash
1
2
3
redis-cli -h 172.30.2.11 -p 6379 --scan
redis-cli -h 172.30.2.12 -p 6379 --scan
redis-cli -h 172.30.2.13 -p 6379 --scan
SpringBoot基于哨兵模式的Redis(7.2)集群实现读写分离

SpringBoot基于哨兵模式的Redis(7.2)集群实现读写分离

环境

  • docker desktop for windows 4.23.0
  • redis 7.2
  • Idea

一、前提条件

先根据以下文章搭建一个Redis集群

部署完后,redis集群看起来大致如下图

二、SpringBoot访问Redis集群

1. 引入依赖

需要注意的是lettuce-core版本问题,不能太旧,否则不兼容新版的Redis

language-xml
1
2
3
4
5
6
7
8
9
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
<version>6.1.4.RELEASE</version> <!-- 或更高版本 -->
</dependency>

2. yaml配置

application.yml加入以下配置。第一个password是用于sentinel节点验证,第二个password用于数据节点验证。

language-yaml
1
2
3
4
5
6
7
8
9
10
spring:
redis:
sentinel:
master: mymaster
nodes:
- 172.30.1.11:26379
- 172.30.1.12:26379
- 172.30.1.13:26379
password: 1009
password: 1009

这里关于sentinelip问题后面会讲解。

3. 设置读写分离

在任意配置类中写一个Bean,本文简单起见,直接写在SpringBoot启动类了。

language-java
1
2
3
4
5
@Bean
public LettuceClientConfigurationBuilderCustomizer clientConfigurationBuilderCustomizer(){

return clientConfigurationBuilder -> clientConfigurationBuilder.readFrom(ReadFrom.REPLICA_PREFERRED);
}

这里的ReadFrom是配置Redis的读取策略,是一个枚举,包括下面选择:

  • MASTER:从主节点读取
  • MASTER_PREFERRED:优先从master节点读取,master不可用才读取replica
  • REPLICA:从slave (replica)节点读取
  • REPLICA_PREFERRED:优先从slave (replica)节点读取,所有的slave都不可用才读取master

至于哪些节点支持读,哪些支持写,因为redis 7 默认给从节点设置为只读,所以可以认为只有主节点有读写权限,其余只有读权限。如果情况不一致,就手动给每一个redis-server的配置文件都加上这一行。

language-txt
1
replica-read-only yes

4. 简单的controller

写一个简单的controller,等会用于测试。

language-java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@RestController
public class HelloController {


@Autowired
private StringRedisTemplate redisTemplate;

@GetMapping("/get/{key}")
public String hi(@PathVariable String key) {

return redisTemplate.opsForValue().get(key);
}

@GetMapping("/set/{key}/{value}")
public String hi(@PathVariable String key, @PathVariable String value) {

redisTemplate.opsForValue().set(key, value);
return "success";
}
}

三、运行

首先,因为所有redis节点都在一个docker bridge网络中,所以基于Idea编写的项目在宿主机(Windows)中运行spirngboot程序,不好去和redis集群做完整的交互。

虽然说无论是sentinel还是redis-server都暴露了端口到宿主机,我们可以通过映射的端口分别访问它们,但是我们的程序只访问sentinelsentinel管理redis-serversentinel会返回redis-serverip来让我们的程序来访问redis-server,这里的ipdocker bridge网络里的ip,所以即使我们的程序拿到ip也访问不了redis-server

这个时候就需要将我们的项目放到一个docker容器中运行,然后把这个容器放到和redis同一网络下,就像下图。

具体如何快捷让Idea结合Docker去运行SpringBoot程序,可以参考下面这篇文章。

记得要暴露你的程序端口到宿主机,这样才方便测试。

四、测试

1. 写

浏览器访问localhost:8080/set/num/7799

查看SpringBoot容器日志,可以看到向主节点172.30.1.2:6379发送写请求。

language-txt
1
2
3
4
5
6
7
8
9
10
11
01-06 07:23:59:848 DEBUG 1 --- [nio-8080-exec-6] io.lettuce.core.RedisChannelHandler      : dispatching command AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:23:59:848 DEBUG 1 --- [nio-8080-exec-6] i.l.c.m.MasterReplicaConnectionProvider : getConnectionAsync(WRITE)
01-06 07:23:59:848 DEBUG 1 --- [nio-8080-exec-6] io.lettuce.core.RedisChannelHandler : dispatching command AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:23:59:848 DEBUG 1 --- [nio-8080-exec-6] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379, epid=0xf] write() writeAndFlush command AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:23:59:848 DEBUG 1 --- [nio-8080-exec-6] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379, epid=0xf] write() done
01-06 07:23:59:848 DEBUG 1 --- [oEventLoop-4-10] io.lettuce.core.protocol.CommandHandler : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379, epid=0xf, chid=0x16] write(ctx, AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], promise)
01-06 07:23:59:849 DEBUG 1 --- [oEventLoop-4-10] io.lettuce.core.protocol.CommandEncoder : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379] writing command AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:23:59:851 DEBUG 1 --- [oEventLoop-4-10] io.lettuce.core.protocol.CommandHandler : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379, epid=0xf, chid=0x16] Received: 5 bytes, 1 commands in the stack
01-06 07:23:59:851 DEBUG 1 --- [oEventLoop-4-10] io.lettuce.core.protocol.CommandHandler : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379, epid=0xf, chid=0x16] Stack contains: 1 commands
01-06 07:23:59:851 DEBUG 1 --- [oEventLoop-4-10] i.l.core.protocol.RedisStateMachine : Decode done, empty stack: true
01-06 07:23:59:852 DEBUG 1 --- [oEventLoop-4-10] io.lettuce.core.protocol.CommandHandler : [channel=0x9b4ebc85, /172.30.1.5:46700 -> /172.30.1.2:6379, epid=0xf, chid=0x16] Completing command AsyncCommand [type=SET, output=StatusOutput [output=OK, error='null'], commandType=io.lettuce.core.protocol.Command]

2. 读

浏览器访问localhost:8080/get/num

查看SpringBoot容器日志,会向两个从节点之一发送读请求。

language-txt
1
2
3
4
5
6
7
8
9
10
11
01-06 07:25:45:342 DEBUG 1 --- [io-8080-exec-10] io.lettuce.core.RedisChannelHandler      : dispatching command AsyncCommand [type=GET, output=ValueOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:25:45:342 DEBUG 1 --- [io-8080-exec-10] i.l.c.m.MasterReplicaConnectionProvider : getConnectionAsync(READ)
01-06 07:25:45:342 DEBUG 1 --- [io-8080-exec-10] io.lettuce.core.RedisChannelHandler : dispatching command AsyncCommand [type=GET, output=ValueOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:25:45:342 DEBUG 1 --- [io-8080-exec-10] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379, epid=0x1c] write() writeAndFlush command AsyncCommand [type=GET, output=ValueOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:25:45:342 DEBUG 1 --- [io-8080-exec-10] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379, epid=0x1c] write() done
01-06 07:25:45:342 DEBUG 1 --- [oEventLoop-4-11] io.lettuce.core.protocol.CommandHandler : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379, epid=0x1c, chid=0x23] write(ctx, AsyncCommand [type=GET, output=ValueOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], promise)
01-06 07:25:45:343 DEBUG 1 --- [oEventLoop-4-11] io.lettuce.core.protocol.CommandEncoder : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379] writing command AsyncCommand [type=GET, output=ValueOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
01-06 07:25:45:346 DEBUG 1 --- [oEventLoop-4-11] io.lettuce.core.protocol.CommandHandler : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379, epid=0x1c, chid=0x23] Received: 10 bytes, 1 commands in the stack
01-06 07:25:45:346 DEBUG 1 --- [oEventLoop-4-11] io.lettuce.core.protocol.CommandHandler : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379, epid=0x1c, chid=0x23] Stack contains: 1 commands
01-06 07:25:45:346 DEBUG 1 --- [oEventLoop-4-11] i.l.core.protocol.RedisStateMachine : Decode done, empty stack: true
01-06 07:25:45:346 DEBUG 1 --- [oEventLoop-4-11] io.lettuce.core.protocol.CommandHandler : [channel=0x96ae68cf, /172.30.1.5:38102 -> /172.30.1.4:6379, epid=0x1c, chid=0x23] Completing command AsyncCommand [type=GET, output=ValueOutput [output=[B@7427ef47, error='null'], commandType=io.lettuce.core.protocol.Command]

3. 额外测试

以及还有一些额外的测试,可以自行去尝试,检验,这里列举一些,但具体不再赘述。

  1. 关闭两个从节点容器,等待sentinel完成维护和通知后,测试读数据和写数据会请求谁?
  2. 再次开启两个从节点,等待sentinel完成操作后,再关闭主节点,等待sentinel完成操作后,测试读数据和写数据会请求谁?
  3. 再次开启主节点,等待sentinel完成操作后,测试读数据和写数据会请求谁?