::: ZOSEL ::: :: How to make High Available Load Balancer(L4/L7) with haproxy and Pacemake

How to make High Available Load Balancer(L4/L7) with haproxy and Pacemaker - 4/4

2013.02.26 16:29 from Tech

5. Test to failover resources

Test를 위해 migration-threshold = 3으로 설정

Case 1. Active Node 장애

Active Node에서 물리적/논리적 Server/Network 장애가 발생하면 Standby Node에서의 Heartbeat Check가 실패하므로 Resource를 Failover
장애 상황을 가정하여 Active Node의 Heartbeat Daemon을 정지
# /etc/init.d/heartbeat stop

Stopping High-Availability services: [ OK ]

Resource(VIP, haproxy)가 Standby Node에서 Failover 됨.

Active Node가 "OFFLINE" 상태.

============
Last updated: Thu Feb 21 13:17:53 2013
Stack: Heartbeat
Current DC: STD (7b640fc0-0089-4db9-8c83-e34476c5e065)	- partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
1 Resources configured.
============

Node ACT (07e78f2b-c4cc-42e6-b058-d7e0bd3a91e5):OFFLINE
Node STD (7b640fc0-0089-4db9-8c83-e34476c5e065): online
        haproxy (lsb:haproxy) Started
        VIP_192.168.0.10        (ocf::heartbeat:IPaddr) Started

Inactive resources:


Migration summary:
* Node STD:

Active Node 복구 후

Active Node가 "online" 상태로 복귀.

============
Last updated: Thu Feb 21 13:21:11 2013
Stack: Heartbeat
Current DC: STD (7b640fc0-0089-4db9-8c83-e34476c5e065)	- partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
1 Resources configured.
============

Node ACT (07e78f2b-c4cc-42e6-b058-d7e0bd3a91e5): online
Node STD (7b640fc0-0089-4db9-8c83-e34476c5e065): online
        haproxy (lsb:haproxy) Started
        VIP_192.168.0.10        (ocf::heartbeat:IPaddr) Started

Inactive resources:


Migration summary:
* Node STD:
* Node ACT:

Resource Failback

Manual Failback 구성이므로 수동으로 해야 함.

# crm
crm(live)# resource 
crm(live)resource# restart LB 
INFO: ordering LB to stop
INFO: ordering LB to start
crm(live)resource#

복구 완료(장애전과 다른점)

Current DC (Designated Coodinator: Cluster Master) 변화: ACT -> STD

Active Node가 장애 였으므로 Standby Node가 Cluster Master가 됨.

============
Last updated: Thu Feb 21 13:23:30 2013
Stack: Heartbeat
Current DC: STD (7b640fc0-0089-4db9-8c83-e34476c5e065)	- partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
1 Resources configured.
============

Node ACT (07e78f2b-c4cc-42e6-b058-d7e0bd3a91e5): online
        VIP_192.168.0.10        (ocf::heartbeat:IPaddr) Started
        haproxy (lsb:haproxy) Started
Node STD (7b640fc0-0089-4db9-8c83-e34476c5e065): online

Inactive resources:


Migration summary:
* Node STD: 
* Node ACT:

Case 2. haproxy Daemon Shutdown

haproxy Daemon이 죽었을 경우 Cluster는 (migration-threshold - 1)회까지는 해당 Daemon을 재시작하고 fail-count를 횟수만큼 증가 시킴.

fail-count == migration-threshold 이면 Node의 장애 여부에 상관없이 다른 Node (여기서는 Standby Node)로 Resource를 migration 한다.

fail-count = 2

============
Last updated: Thu Feb 21 13:32:26 2013
Stack: Heartbeat
Current DC: STD (7b640fc0-0089-4db9-8c83-e34476c5e065)	- partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
1 Resources configured.
============

Node ACT (07e78f2b-c4cc-42e6-b058-d7e0bd3a91e5): online
        VIP_192.168.0.10        (ocf::heartbeat:IPaddr) Started
        haproxy (lsb:haproxy) Started
Node STD (7b640fc0-0089-4db9-8c83-e34476c5e065): online

Inactive resources:


Migration summary:
* Node STD: 
* Node ACT: 
   haproxy: migration-threshold=3 fail-count=2

fail-count = 3

============
Last updated: Thu Feb 21 13:33:13 2013
Stack: Heartbeat
Current DC: STD (7b640fc0-0089-4db9-8c83-e34476c5e065)	- partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
1 Resources configured.
============

Node ACT (07e78f2b-c4cc-42e6-b058-d7e0bd3a91e5): online
Node STD (7b640fc0-0089-4db9-8c83-e34476c5e065): online
        haproxy (lsb:haproxy) Started
        VIP_192.168.0.10        (ocf::heartbeat:IPaddr) Started

Inactive resources:


Migration summary:
* Node STD: 
* Node ACT: 
   haproxy: migration-threshold=3 fail-count=3

Failed actions:
    haproxy_monitor_5000 (node=ACT, call=13, rc=7, status=complete): not running

Resource Failback

fail-count를 초기화하기 위해 cleanup 명령을 사용

# crm crm(live)# resource

crm(live)resource# cleanup LB

Cleaning up VIP_192.168.0.10 on ACT Cleaning up VIP_192.168.0.10 on STD Cleaning up haproxy on ACT Cleaning up haproxy on STD Waiting for 5 replies from the CRMd..... crm(live)resource# restart LB INFO: ordering LB to stop INFO: ordering LB to start crm(live)resource#