云野 Lvs负载均衡
简述
可伸缩网络服务涉及到几种不同的结构，它们都需要一个前端的负载调度器（或者多个进行主从备份）。
先分析实现虚拟网络服务的主要技术，指出 <strong>IP</strong> 负载均衡技术是在负载调度器的实现技术中效率最高的。在已有的IP负载均衡技术中，主要有通过网络地址转换NAT（Network Address Translation）将一组服务器构成一个高性能的、高可用的虚拟服务器，称之为VS/NAT技术（Virtual Server via Network Address Translation）。在分析VS/NAT的缺点和网络服务的非对称性的基础上，提出了通过IP隧道实现虚拟服务器的方法VS/TUN （Virtual Server via IP Tunneling），和通过直接路由实现虚拟服务器的方法VS/DR（Virtual Server via Direct Routing），它们可以极大地提高系统的伸缩性。VS/NAT、VS/TUN和VS/DR技术是LVS集群中实现的三种IP负载均衡技术，后面将详细描述它们的工作原理和各自的优缺点。
实现虚拟服务的相关方法
网络服务中，一端是客户程序，另一端是服务程序，在中间可能有代理程序。由此看来，可以在不同的层次上实现多台服务器的负载均衡。用集群解决网络服务性能问题的现有方法主要分为以下四类。
基于RR-DNS的解决方法
在VS/DR中，根据缺省的TCP/IP协议栈处理，请求报文的目标地址为VIP，响应报文的源地址肯定也为VIP，所以响应报文不需要作任何修改，可以直接返回给客户，客户认为得到正常的服务，而不会知道是哪一台服务器处理的。
——-以上来自百度百科
LVS负载均衡实现
LVS调度服务器的设置（lvs1的172.25.32.6为调度服务器）。
1. 下载ipvsadm，这个是属于linux内核的模块，基于用户层面来写策略的，类似于防火墙。
[root@lvs1 ~]# yum install ipvsadm.x86_64 -y Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package ipvsadm.x86_64 0:1.27-7.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: ipvsadm x86_64 1.27-7.el7 dvd 45 k Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Total download size: 45 k Installed size: 75 k Downloading packages: ipvsadm-1.27-7.el7.x86_64.rpm | 45 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : ipvsadm-1.27-7.el7.x86_64 1/1 Verifying : ipvsadm-1.27-7.el7.x86_64 1/1 Installed: ipvsadm.x86_64 0:1.27-7.el7 Complete!
2. 添加一个没有被占用的vip地址。
[root@lvs1 ~]# ip addr add 172.25.32.100/24 dev eth0 [root@lvs1 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:8f:ed:32 brd ff:ff:ff:ff:ff:ff inet 172.25.32.6/24 brd 172.25.32.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.25.32.100/24 scope global secondary eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe8f:ed32/64 scope link valid_lft forever preferred_lft forever
3. 给调度器定义策略目标。
[root@lvs1 ~]# ipvsadm -A -t 172.25.32.100:80 -s rr -A添加一个服务 -t代表添加的服务为tcp服务 172.25.32.100:80为虚拟地址 -s调度算法 rr轮叫调度（这种是最简单的调度算法，就是将请求A一个，B一个，A一个，B一个 ...... 循环的发。就算A主机挂掉了，调度器还是会将请求发送到A。十分均衡。）
4. 给tcp虚拟服务添加真实服务器ip（172.25.32.7,172.25.32.8,作为真实均摊访问vip地址流量的后端服务器）
[root@lvs1 ~]# ipvsadm -a -t 172.25.32.100:80 -r 172.25.32.8:80 -g ##g直连模式
[root@lvs1 ~]# ipvsadm -a -t 172.25.32.100:80 -r 172.25.32.7:80 -g
5. 查看书写的策略
[root@lvs1 ~]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.25.32.100:80 rr -> 172.25.32.7:80 Route 1 0 0 -> 172.25.32.8:80 Route 1 0 0
lvs后端服务器的设置（lvs2的172.25.32.7和lvs2的172.25.32.8,作为真实均摊访问vip地址流量的后端服务器）
1. 两台后端服务器下载http服务且设置开机自启。
[root@lvs2 ~]# yum install httpd -y Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.4.6-88.el7 will be installed --> Processing Dependency: httpd-tools = 2.4.6-88.el7 for package: httpd-2.4.6-88.el7.x86_64 --> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-88.el7.x86_64 --> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-88.el7.x86_64 --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-88.el7.x86_64 --> Running transaction check ---> Package apr.x86_64 0:1.4.8-3.el7_4.1 will be installed ---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed ---> Package httpd-tools.x86_64 0:2.4.6-88.el7 will be installed ---> Package mailcap.noarch 0:2.1.41-2.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: httpd x86_64 2.4.6-88.el7 dvd 1.2 M Installing for dependencies: apr x86_64 1.4.8-3.el7_4.1 dvd 103 k apr-util x86_64 1.5.2-6.el7 dvd 92 k httpd-tools x86_64 2.4.6-88.el7 dvd 90 k mailcap ...... [root@lvs2 ~]# systemctl enable --now httpd.service Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service. [root@lvs3 ~]# yum install httpd -y Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.4.6-88.el7 will be installed --> Processing Dependency: httpd-tools = 2.4.6-88.el7 for package: httpd-2.4.6-88.el7.x86_64 --> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-88.el7.x86_64 --> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-88.el7.x86_64 --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-88.el7.x86_64 --> Running transaction check ---> Package apr.x86_64 0:1.4.8-3.el7_4.1 will be installed ---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed ---> Package httpd-tools.x86_64 0:2.4.6-88.el7 will be installed ---> Package mailcap.noarch 0:2.1.41-2.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: httpd x86_64 2.4.6-88.el7 dvd 1.2 M Installing for dependencies: apr x86_64 1.4.8-3.el7_4.1 dvd 103 k apr-util x86_64 1.5.2-6.el7 dvd 92 k httpd-tools x86_64 2.4.6-88.el7 dvd 90 k mailcap ....... [root@lvs3 ~]# systemctl enable --now httpd.service Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
2. 在两台后端服务器中自定义不同的http访问首页。
[root@lvs2 ~]# echo lv2 > /var/www/html/index.html [root@lvs2 ~]# cat /var/www/html/index.html lv2 [root@lvs3 ~]# echo lv3 > /var/www/html/index.html [root@lvs3 ~]# cat /var/www/html/index.html lv3
4. 给后端服务器添加vip（172.25.32.100）地址，（注意：后端服务器添加vip地址的时候ip后的掩玛不要设置成和调度服务器vip的掩玛一样，不然会造成冲突）
[root@lvs2 ~]# ip addr add 172.25.32.100/32 dev eth0 [root@lvs3 ~]# ip addr add 172.25.32.100/32 dev eth0
在外部客户端测试是否均衡访问流量了。
[root@foundation32 ~]# for i in {1..10}; do curl 172.25.32.100;done
lv3
lv2
lv3
lv2
lv3
lv2
lv3
lv2
lv3
lv2
访问到了不同的后端服务器首页，显然达到了负载均衡的目的。
在调度服务器中也可以查看负载的效果
[root@lvs1 ~]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.25.32.100:80 rr -> 172.25.32.7:80 Route 1 0 5 -> 172.25.32.8:80 Route 1 0 5
但是这样设置的话，相同的vip地址处在同一个vlan里面，外部服务器访问的时候可能会直接访问到后端服务器，而不经过lvs调度造成冲突。所以要设置成外部服务器访问vian中的vip（172.25.32.100）这个地址的时候只能是调度器地址来响应，然后调度器转发给后端服务器，后端服务器不能直接对外部服务器做响应。这时候有人就说了，那直接把后端服务器的vip地址删了不就好了，注意lvs服务二层调度转发的时候检测不到调度服务器策略中书写的后端服务器有这个vip地址的话，后端服务器就直接被丢弃了，参与不到负载均衡的这个集群中，所以在DR这个模式中所有节点都要有vip这个地址。
要达成这种效果，有两种方法：
1. 修改linux内核（本人才疏学浅，这种方法就俩字“不会”）
2. 通过arptables这个工具来实现
在后端服务器屏蔽来自外部服务器对vip地址的访问
1. 下载arptables工具（arptables只对arp协议作管控，其他的不做管控），设置开机自启。（两个后端服务器作同样惭怍）
[root@lvs2 html]# yum install -y arptables.x86_64 Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package arptables.x86_64 0:0.0.4-8.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: arptables x86_64 0.0.4-8.el7 dvd 47 k Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Total download size: 47 k Installed size: 87 k Downloading packages: arptables-0.0.4-8.el7.x86_64.rpm | 47 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : arptables-0.0.4-8.el7.x86_64 1/1 Verifying : arptables-0.0.4-8.el7.x86_64 1/1 Installed: arptables.x86_64 0:0.0.4-8.el7 Complete! [root@lvs2 html]# systemctl enable --now arptables.service Created symlink from /etc/systemd/system/multi-user.target.wants/arptables.service to /usr/lib/systemd/system/arptables.service.
2. 在后端服务器书写策略arptables策略
[root@lvs2 html]# arptables -A INPUT -d 172.25.32.100 ##拒绝外部进来对172.25.32.100这个vip地址的访问，只接受vlan里lvs调度器转发的响应 [root@lvs2 html]# arptables -A OUTPUT -s 172.25.32.100 -j mangle --mangle-ip-s 172.25.32.7 ##出去对外部反馈信息的时候对172.25.32.100这个vip地址作伪装，用后端服务器自己的接口地址来对外部进行广播（这里后端服务器自己的地址是172.25.32.7） [root@lvs3 ~]# arptables -A INPUT -d 172.25.32.100 [root@lvs3 ~]# arptables -A OUTPUT -s 172.25.32.100 -j mangle --mangle-ip-s 172.25.32.8
3. arptables默认保存在内存里面，重启后就没了，想要永久生效要通过arptables-save命令写到/etc/sysconfig/arptables中
[root@lvs2 html]# arptables-save > /etc/sysconfig/arptables [root@lvs2 html]# cat /etc/sysconfig/arptables *filter :INPUT ACCEPT :OUTPUT ACCEPT :FORWARD ACCEPT -A INPUT -d 172.25.32.100 -A OUTPUT -j mangle -s 172.25.32.100 --mangle-ip-s 172.25.32.7
[root@lvs3 ~]# arptables-save > /etc/sysconfig/arptables [root@lvs3 ~]# cat /etc/sysconfig/arptables *filter :INPUT ACCEPT :OUTPUT ACCEPT :FORWARD ACCEPT -A OUTPUT -j mangle -s 172.25.32.100 --mangle-ip-s 172.25.32.8