keepalived 服务配置

keepalived 服务配置

keepalived 起初是专为LVS设计的,专门用来监控LVS集群系统中各个服务节点的状态,后来又加入VRRP的功能,因此除了配合LVS服务外,也可以作为其他服务(nginx,haproxy)的高可用软件。
VRRP(virtual router redundancy protocol)虚拟路由器冗余协议:处理的目的就是为了解决静态路由传销的单点故障问题,
keepalived两大功能:healthcheck & failover 。

主机规划

主机名 IP VIP 功能
node8 192.168.66.18 192.168.66.250(DR) lvs+keepalived(主节点)
node9 192.168.66.19 192.168.66.250(DR) lvs+keepalived(备节点)
node1 192.168.66.11 nginx
node2 192.168.66.12 nginx

安装 lvs 软件

两台机器上都要操作

1
yum install ipvsadm -y

开启内核转发

1
2
3
cat /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward

keepalived 安装

两台机器上都要操作

1
yum install curl gcc openssl-devel libnl3-devel net-snmp-devel -y

二进制安装

1
2
3
4
5
6
7
8
9
cd /usr/local/src
wget https://www.keepalived.org/software/keepalived-2.1.5.tar.gz
tar -zxf keepalived-2.1.5.tar.gz
cd keepalived-2.1.5.tar.gz
./configure --prefix=/usr/local/src/keepalived-2.1.5_bin
make && make install
cd /usr/local/src/keepalived-2.1.5_bin
cp sbin/keepalived /usr/sbin/
cp bin/genhash /usr/bin/

实战

添加 keepalived 启动文件

两台机器上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vim /etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp -P Only run with VRRP subsystem.
# --check -C Only run with Health-checker subsystem.
# --dont-release-vrrp -V Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs -I Dont remove IPVS topology on daemon stop.
# --dump-conf -d Dump the configuration data.
# --log-detail -D Detailed log messages.
# --log-facility -S 0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D"

添加 keepalived 主配置文件

node8上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
mkdir -p /etc/keepalived/
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
biglittleant@admin.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id node8 #修改为本机名
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}

vrrp_instance VI_1 {
state MASTER # 主节点
interface eth1
virtual_router_id 51
priority 100 # 要大于备节点的值
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.66.250
}
}

include keepalived.d/*.conf

备节点配置文件

node9上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
mkdir -p /etc/keepalived/
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
biglittleant@admin.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id node9 #修改为本机名
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}

vrrp_instance VI_1 {
state BACKUP # 备节点
interface eth1
virtual_router_id 51
priority 99 # 要小于主节点的值
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.66.250
}
}

include keepalived.d/*.conf

添加 virtual_server 配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
mkdir -p /etc/keepalived/keepalived.d
vim /etc/keepalived/keepalived.d/vs-192.168.66.250_80.conf
virtual_server 192.168.66.250 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
persistence_timeout 50
protocol TCP

real_server 192.168.66.11 80 {
weight 1
}

real_server 192.168.66.12 80 {
weight 2
}
}

添加 keepalived 启动文件

两台机器上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 添加keepalived 启动文件
vim /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# 启动keepalvied & 加入开机自启动
systemctl daemon-reload
systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
systemctl start keepalived

验证服务启动

node8上操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Sep 15 17:10:34 node8 Keepalived_vrrp[30389]: Using LinkWatch kernel netlink reflector...
Sep 15 17:10:34 node8 Keepalived_vrrp[30389]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Got SIGHUP, reloading checker configuration
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Initializing ipvs
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Opening file 'keepalived.d/vs-192.168.66.250_80.conf'.
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: service [192.168.66.13]:80 no longer exist
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Gained quorum 1+0=1 <= 2 for VS [192.168.66.250]:80
Sep 15 17:10:34 node8 systemd[1]: Reloaded LVS and VRRP High Availability Monitor.
Sep 15 17:10:35 node8 Keepalived_vrrp[30389]: VRRP_Instance(VI_1) Transition to MASTER STATE

[root@node8 ~]# ip addr list eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:89:7e:4b brd ff:ff:ff:ff:ff:ff
inet 192.168.66.18/24 brd 192.168.66.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet 192.168.66.250/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe89:7e4b/64 scope link
valid_lft forever preferred_lft forever

[root@node8 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.66.250:80 wrr persistent 50
-> 192.168.66.11:80 Route 1 0 0
-> 192.168.66.12:80 Route 2 0 0

手动关闭主节点,验证虚拟IP自动漂移

node8上操作

1
2
3
4
5
6
7
[root@node8 ~]# systemctl stop keepalived
# 发现主节点lvs配置没了
[root@node8 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@node8 ~]#

node9上操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 通过日志,查看Keepalived主备切换过程
# 期间备节点发送ARP广播,让所有客户端更新本地的ARP表,以便客户端访问新接管VIP服务的节点。

Sep 15 17:33:21 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 15 17:33:21 node9 Keepalived_vrrp[30272]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Sep 15 17:33:21 node9 systemd[1]: Reloaded LVS and VRRP High Availability Monitor.
Sep 15 17:34:48 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250

为什么连续发送5个arp,因为vrrp_garp_master_repeat默认为5;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@node9 ~]# ip addr list eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:e5:bd:89 brd ff:ff:ff:ff:ff:ff
inet 192.168.66.19/24 brd 192.168.66.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet 192.168.66.250/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fee5:bd89/64 scope link
valid_lft forever preferred_lft forever

[root@node9 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.66.250:80 wrr persistent 50
-> 192.168.66.11:80 Route 1 0 0
-> 192.168.66.12:80 Route 2 0 0

重启启动主节点,验证是否切回

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 由于主节点优先级高于备节点,因此主节点抢占为MASTER,同时备节点成为BACKUP,并且移除VIP。

Sep 15 17:38:41 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 15 17:38:41 node8 Keepalived_vrrp[1214]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Sep 15 17:38:41 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) forcing a new MASTER election
Sep 15 17:38:42 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250


[root@node8 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.66.250:80 wrr persistent 50
-> 192.168.66.11:80 Route 1 0 0
-> 192.168.66.12:80 Route 2 0 0
[root@node8 ~]# ip addr list eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:89:7e:4b brd ff:ff:ff:ff:ff:ff
inet 192.168.66.18/24 brd 192.168.66.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet 192.168.66.250/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe89:7e4b/64 scope link
valid_lft forever preferred_lft forever

报错汇总

Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory

yum 安装会报错,查看ipset的包,发现库文件是libipset.so.11, 怀疑是yum版本太旧导致。使用src安装最新版本,未发现此报错。

参考文档

git仓库
官方文档