Linux Magazine | November 2003 | CLUSTERS | Linux Virtual Server Clusters

IP load balancing techniques are quite scalable, and IPVS supports three different load balancing techniques: Virtual Server via NAT (VS/NAT), Virtual Server via Tunneling (VS/TUN), and Virtual Server via Direct Routing (VS/DR).

Virtual Server via NAT (VS/NAT)

Due to security considerations and the shortage of IP addresses in IPv4, more and more networks use private IP addresses that aren't allocated on the Internet. Network address translation is needed when hosts in internal networks want to access the Internet, or need to be accessed from the Internet. NAT can also be used to build a virtual server: parallel services at different IP addresses can appear as a virtual service on a single IP address. The architecture of Virtual Server via NAT is illustrated in Figure Four. The load balancer and real servers are interconnected by a switch or a hub.

Figure Four: Architecture of Virtual Server via NAT

The workflow of VS/NAT is as follows:

1. When a user accesses a virtual service provided by the server cluster, a request packet destined for the virtual IP address (the IP address to accept requests for virtual service) arrives at the load balancer.

2. The load balancer examines the packet's destination address and port number. If they match a virtual service in the virtual server rule table, a real server is selected from the cluster by a scheduling algorithm and the connection is added to hash table that records connections. Then, the destination address and the port of the packet are rewritten to those of the selected server, and the packet is forwarded to the server. When an incoming packet belongs to an established connection, the connection can be found in the hash table and the packet is rewritten and forwarded to the right server.

3. The request is processed by one of the physical servers.

4. When response packets come back, the load balancer rewrites the source address and port of the packets to those of the virtual service. When a connection terminates or timeouts, the connection record is removed from the hash table.

5. A reply is sent back to the user.

Virtual Server via IP Tunneling (VS/TUN)

IP tunneling (also called IP encapsulation) is a technique to encapsulate IP datagrams within IP datagrams, which allows datagrams destined for one IP address to be wrapped and redirected to another IP address.

This technique can also be used to build a virtual server: the load balancer tunnels the request packets to the different servers, the servers process the requests, and return the results to the clients directly. Thus, the service appears as a virtual service on a single IP address. The architecture of Virtual Server via IP Tunneling is illustrated in Figure Five.

Figure Five: Architecture of Virtual Server via IP Tunneling

In the figure, the real servers can have any real IP addresses in any network, and the servers can be geographically distributed. However, each server must support the IP tunneling protocol, and each must have one of their tunnel devices configured with virtual IP.

The flow of VS/TUN is the same as that of VS/NAT. In VS/TUN, the load balancer encapsulates the packet within an IP datagram and forwards it to a dynamically selected server. When the server receives the encapsulated packet, it decapsulates the packet, finds the inner packet destined for the virtual IP address on its tunnel device, processes the request, and returns the result to the user directly.

Virtual Server via Direct Routing (VS/DR)

The VS/DR balancing approach is similar to the one implemented in IBM's NetDispatcher. The architecture of VS/DR is illustrated in Figure Six.

Figure Six: Architecture of Virtual Server via Direct Routing

The load balancer and the real servers must have one of their interfaces physically linked by an uninterrupted segment of LAN such as an Ethernet switch. The virtual IP address is shared by real servers and the load balancer. Each real server has a non-ARPing, loopback alias interface configured with the virtual IP address, and the load balancer has an interface configured with the virtual IP address to accept incoming packets.

The workflow of VS/DR is similar to that of VS/NAT or VS/TUN. In VS/DR, the load balancer directly routes a packet to the selected server (the load balancer simply changes the MAC address of the data frame to that of the server and retransmits it on the LAN). When the server receives the forwarded packet, the server determines that the packet is for the address on its loopback alias interface, processes the request, and finally returns the result directly to the user.

The logical interfaces of the real servers that are configured with virtual IP address should not respond to ARP requests; otherwise, there would be an ARP conflict between the interface that accepts incoming traffic for virtual IP and the interfaces of real servers in the same network.

Comparing Technique

Which technique is best? For most Internet services (such as web hosting) where request packets are often short and response packets usually carry large amount of data, a VS/TUN load balancer can easily schedule over 100 real servers and won't become the bottleneck of the system. The load balancer just directs requests to the servers and the servers reply to the clients directly. While VS/TUN requires servers to support the IP tunneling protocol, the IP tunneling protocol is becoming a standard of all operating systems, and VS/TUN should be usable with servers running operating systems other than Linux.

Networking Options --> IP: Virtual Server Configuration --> <M> virtual server support (EXPERIMENTAL) [ ] IP virtual server debugging (12) IPVS connection table size (the Nth power of 2) --- IPVS scheduler <M> round-robin scheduling <M> weighted round-robin scheduling <M> least-connection scheduling scheduling <M> weighted least-connection scheduling <M> locality-based least-connection scheduling <M> locality-based least-connection with replication scheduling <M> destination hashing scheduling <M> source hashing scheduling <M> shortest expected delay scheduling <M> never queue scheduling --- IPVS application helper <M> FTP protocol helper

Networking Options --> <*> Packet socket [*] Packet socket: mmapped IO <*> Netlink device emulation [*] Network packet filtering (replaces ipchains) [ ] Network packet filtering debugging [*] Socket Filtering <*> Unix domain sockets [*] TCP/IP networking [ ] IP: multicasting [ ] IP: advanced router [ ] IP: kernel level autoconfiguration <M> IP: tunneling < > IP: GRE tunnels over IP [ ] IP: multicast routing [ ] IP: ARP daemon support (EXPERIMENTAL) [ ] IP: TCP Explicit Congestion Notification support [ ] IP: TCP syncookie support (disabled per default) Networking Options --> IP: Netfilter Configurations <M> Connection tracking (required for masq/NAT) <M> Full NAT <M> MASQUERADE target support <M> REDIRECT target support

# echo "1" > /proc/sys/net/ipv4/ip_forward # iptables t NAT A POSTROUTING s \ 172.18.1.0/24 j MASQUERADE # ipvsadm A t 10.23.8.80:80 p 600 # ipvsadm a t 10.23.8.80:80 R \ 172.18.1.11 m w 100 # ipvsadm a t 10.23.8.80:80 R \ 172.18.1.12 m w 100 # ipvsadm a t 10.23.8.80:80 R \ 172.18.1.13 m w 100

/etc/ha.d/ha.cf: logfacility local0 keepalive 2 deadtime 10 warntime 10 initdead 10 nice_failback on udpport 694 bcast eth1 node ld1 node ld2 /etc/ha.d/haresources: ld1 IPaddr::10.23.8.80/24/eth1 IPaddr::172.18.1.254/24/ \ eth1 ldirectord::ldirectord.cf /etc/ha.d/ldirectord.cf: checktimeout=10 checkinterval=2 autoreload=no logfile="local0" quiescent=yes virtual=10.23.8.80:80 fallback=127.0.0.1:80 real=172.18.1.11:80 masq real=172.18.1.12:80 masq real=172.18.1.13:80 masq service=http request="index.html" receive="Test Page" scheduler=wlc persistent=600 protocol=tcp checktype=negotiate

primary = 10.23.8.1 service = lvs rsh_command = rsh backup_active = 1 backup = 10.23.8.2 heartbeat = 1 heartbeat_port = 539 keepalive = 4 deadtime = 12 network = nat nat_router = 172.18.1.254 eth1:0 nat_nmask = 255.255.255.0 reservation_conflict_action = preempt debug_level = NONE virtual web { active = 1 address = 10.23.8.80 eth0:1 vip_nmask = 255.255.255.255 port = 80 persistent = 600 send = "GET / HTTP/1.0\r\n\r\n" expect = "HTTP" load_monitor = none scheduler = wlc protocol = tcp timeout = 6 reentry = 15 quiesce_server = 0 server webserver1 { address = 172.18.1.11 active = 1 weight = 100 } server webserver2 { address = 172.18.1.12 active = 1 weight = 100 } server webserver3 { address = 172.18.1.13 active = 1 weight = 100 } }

vrrp_sync_group VG1 { group { VI_1 VI_2 } } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.23.8.80 } } vrrp_instance VI_2 { state MASTER interface eth1 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.18.1.254 } } virtual_server 10.23.8.80 80 { delay_loop 6 lb_algo wlc lb_kind NAT persistence_timeout 600 protocol TCP real_server 172.18.1.11 80 { weight 100 TCP_CHECK { connect_timeout 3 } } real_server 172.18.1.12 80 { weight 100 TCP_CHECK { connect_timeout 3 } } real_server 172.18.1.13 80 { weight 100 TCP_CHECK { connect_timeout 3 } } }