michi/k3s-ansible
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove Keepalived VIP setup - using MikroTik hardware VIP instead

Browse Source 
- Deleted vip-setup.yml playbook (Keepalived no longer needed)
- Updated MIKROTIK-VIP-SETUP-CUSTOM.md with corrected MikroTik syntax:
  * Fixed path notation: use spaces not slashes (/ip firewall nat not /ip/firewall/nat/)
  * Fixed action parameter: use dst-nat not dstnat
  * Added web interface alternative for NAT rule configuration
  * Added important syntax notes section
- Removed Keepalived documentation from README.md
- Kept MIKROTIK-VIP-SETUP.md as general reference guide
- Updated DNS and external access section to reference MikroTik VIP only

This simplifies the project by removing software-based VIP complexity since
the hardware-based MikroTik VIP provides better performance with no node overhead.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
Michael Skrynski
2026-01-08 17:02:19 +01:00
parent ed0e0d1af1
commit b78249e8a1
3 changed files with 60 additions and 408 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
MIKROTIK-VIP-SETUP-CUSTOM.md
View File

@@ -31,7 +31,7 @@ Since your K3s nodes are on the `br-lab` bridge, add the VIP there:
### Verify VIP is Added ### Verify VIP is Added
```mikrotik ```mikrotik
/ip/address/print detail where comment~"K3s" /ip address print detail where comment~"K3s"
``` ```
You should see: You should see:
@@ -39,6 +39,12 @@ You should see:
0 address=192.168.30.100/24 interface=br-lab disabled=no comment="K3s-Cluster-VIP" 0 address=192.168.30.100/24 interface=br-lab disabled=no comment="K3s-Cluster-VIP"
``` ```
**Important Syntax Notes:**
- Use `/ip address` (space) instead of `/ip/address/` (slashes)
- Use `action=dst-nat` (with hyphen) instead of `action=dstnat`
- The command path uses spaces: `/ip firewall nat` instead of `/ip/firewall/nat/`
## Step 2: Create NAT Rules for Traffic Routing ## Step 2: Create NAT Rules for Traffic Routing
Your VIP will route traffic to the master node by default. Create NAT rules to handle port 80 (HTTP) and 443 (HTTPS). Your VIP will route traffic to the master node by default. Create NAT rules to handle port 80 (HTTP) and 443 (HTTPS).
@@ -46,28 +52,64 @@ Your VIP will route traffic to the master node by default. Create NAT rules to h
### HTTP (Port 80) ### HTTP (Port 80)
```mikrotik ```mikrotik
/ip/firewall/nat/add chain=dstnat dst-address=192.168.30.100 dst-port=80 protocol=tcp action=dstnat to-addresses=192.168.30.101 to-ports=80 comment="K3s-VIP-HTTP" /ip firewall nat add chain=dstnat dst-address=192.168.30.100 dst-port=80 protocol=tcp action=dst-nat to-addresses=192.168.30.101 to-ports=80 comment="K3s-VIP-HTTP"
``` ```
### HTTPS (Port 443) ### HTTPS (Port 443)
```mikrotik ```mikrotik
/ip/firewall/nat/add chain=dstnat dst-address=192.168.30.100 dst-port=443 protocol=tcp action=dstnat to-addresses=192.168.30.101 to-ports=443 comment="K3s-VIP-HTTPS" /ip firewall nat add chain=dstnat dst-address=192.168.30.100 dst-port=443 protocol=tcp action=dst-nat to-addresses=192.168.30.101 to-ports=443 comment="K3s-VIP-HTTPS"
``` ```
### Verify NAT Rules ### Verify NAT Rules
```mikrotik ```mikrotik
/ip/firewall/nat/print detail where comment~"K3s-VIP" /ip firewall nat print detail where comment~"K3s-VIP"
``` ```
### Alternative: Configure NAT Rules via Web Interface
If you experience CLI syntax issues, use the MikroTik WebFig interface instead:
1. **Access MikroTik WebFig**
- Open `http://<router-ip>` in your browser
- Login with your admin credentials
2. **Navigate to NAT Rules**
- Go to: **IP** → **Firewall** → **NAT**
3. **Add HTTP NAT Rule**
- Click **+ New**
- Set:
- **Chain**: `dstnat`
- **Dst. Address**: `192.168.30.100`
- **Dst. Port**: `80`
- **Protocol**: `tcp`
- **Action**: `dst-nat`
- **To Addresses**: `192.168.30.101`
- **To Ports**: `80`
- **Comment**: `K3s-VIP-HTTP`
- Click **OK**
4. **Add HTTPS NAT Rule**
- Click **+ New**
- Set:
- **Chain**: `dstnat`
- **Dst. Address**: `192.168.30.100`
- **Dst. Port**: `443`
- **Protocol**: `tcp`
- **Action**: `dst-nat`
- **To Addresses**: `192.168.30.101`
- **To Ports**: `443`
- **Comment**: `K3s-VIP-HTTPS`
- Click **OK**
## Step 3: Add Static Routes (Optional but Recommended) ## Step 3: Add Static Routes (Optional but Recommended)
Ensure the K3s cluster nodes can reach each other through br-lab: Ensure the K3s cluster nodes can reach each other through br-lab:
```mikrotik ```mikrotik
/ip/route/add dst-address=192.168.30.0/24 gateway=192.168.30.1 \ /ip route add dst-address=192.168.30.0/24 gateway=192.168.30.1 comment="K3s-Cluster-Network"
comment="K3s-Cluster-Network"
``` ```
## Step 4: Configure Firewall Rules ## Step 4: Configure Firewall Rules
@@ -77,25 +119,13 @@ Make sure your firewall allows traffic on ports 80 and 443 to the VIP:
### Allow Ingress to VIP on Port 80 ### Allow Ingress to VIP on Port 80
```mikrotik ```mikrotik
/ip/firewall/filter/add \ /ip firewall filter add chain=forward dst-address=192.168.30.100 dst-port=80 protocol=tcp action=accept comment="Allow-HTTP-to-VIP"
chain=forward \
dst-address=192.168.30.100 \
dst-port=80 \
protocol=tcp \
action=accept \
comment="Allow-HTTP-to-VIP"
``` ```
### Allow Ingress to VIP on Port 443 ### Allow Ingress to VIP on Port 443
```mikrotik ```mikrotik
/ip/firewall/filter/add \ /ip firewall filter add chain=forward dst-address=192.168.30.100 dst-port=443 protocol=tcp action=accept comment="Allow-HTTPS-to-VIP"
chain=forward \
dst-address=192.168.30.100 \
dst-port=443 \
protocol=tcp \
action=accept \
comment="Allow-HTTPS-to-VIP"
``` ```
## Step 5: Test the VIP ## Step 5: Test the VIP
@@ -103,7 +133,7 @@ Make sure your firewall allows traffic on ports 80 and 443 to the VIP:
### Test from MikroTik Router ### Test from MikroTik Router
```mikrotik ```mikrotik
/tool/ping 192.168.30.100 count=5 /tool ping 192.168.30.100 count=5
``` ```
Expected output: All 5 pings should succeed Expected output: All 5 pings should succeed
@@ -296,20 +326,20 @@ If you want to run all commands in one go, here's the complete sequence:
```mikrotik ```mikrotik
# Add VIP address # Add VIP address
/ip/address/add address=192.168.30.100/24 interface=br-lab comment="K3s-Cluster-VIP" /ip address add address=192.168.30.100/24 interface=br-lab comment="K3s-Cluster-VIP"
# Add HTTP NAT rule # Add HTTP NAT rule
/ip/firewall/nat/add chain=dstnat dst-address=192.168.30.100 dst-port=80 protocol=tcp action=dstnat to-addresses=192.168.30.101 to-ports=80 comment="K3s-VIP-HTTP" /ip firewall nat add chain=dstnat dst-address=192.168.30.100 dst-port=80 protocol=tcp action=dst-nat to-addresses=192.168.30.101 to-ports=80 comment="K3s-VIP-HTTP"
# Add HTTPS NAT rule # Add HTTPS NAT rule
/ip/firewall/nat/add chain=dstnat dst-address=192.168.30.100 dst-port=443 protocol=tcp action=dstnat to-addresses=192.168.30.101 to-ports=443 comment="K3s-VIP-HTTPS" /ip firewall nat add chain=dstnat dst-address=192.168.30.100 dst-port=443 protocol=tcp action=dst-nat to-addresses=192.168.30.101 to-ports=443 comment="K3s-VIP-HTTPS"
# Add static route # Add static route
/ip/route/add dst-address=192.168.30.0/24 gateway=192.168.30.1 comment="K3s-Cluster-Network" /ip route add dst-address=192.168.30.0/24 gateway=192.168.30.1 comment="K3s-Cluster-Network"
# Verify # Verify
/ip/address/print detail /ip address print detail
/ip/firewall/nat/print detail where comment~"K3s" /ip firewall nat print detail where comment~"K3s"
``` ```
## Remove VIP (If Needed) ## Remove VIP (If Needed)

225
README.md
View File

@@ -887,9 +887,9 @@ test.zlor.fi A 192.168.30.100
Set up a virtual IP that automatically handles failover. You have two sub-options: Set up a virtual IP that automatically handles failover. You have two sub-options:
##### Option C1: MikroTik VIP (Recommended if you have MikroTik router) ##### Option C: MikroTik VIP (Recommended)
Configure VIP directly on your MikroTik router. See [MIKROTIK-VIP-SETUP.md](MIKROTIK-VIP-SETUP.md) for detailed instructions. Configure VIP directly on your MikroTik router. See [MIKROTIK-VIP-SETUP.md](MIKROTIK-VIP-SETUP.md) for customized setup instructions for your network topology.
Pros: Pros:
@@ -897,23 +897,7 @@ Pros:
- No additional software on cluster nodes - No additional software on cluster nodes
- Hardware-based failover (more reliable) - Hardware-based failover (more reliable)
- Better performance - Better performance
- Reduced CPU overhead on nodes
##### Option C2: Keepalived (Software-based VIP)
Configure floating IP using Keepalived on cluster nodes. See "Virtual IP Setup (Keepalived)" below for detailed instructions.
Pros:
- No router configuration needed
- Portable across different networks
- Works in cloud environments
Cons:
- Additional daemon on all nodes
- More configuration needed
Recommendation: If you have MikroTik, use Option C1 (MikroTik VIP). Otherwise, use Option C2 (Keepalived).
### Step 2: Configure Cluster Nodes for External DNS ### Step 2: Configure Cluster Nodes for External DNS
@@ -1137,209 +1121,6 @@ spec:
kubectl apply -f manifests/nginx-test-deployment.yaml kubectl apply -f manifests/nginx-test-deployment.yaml
``` ```
## Virtual IP Setup - Keepalived (Option C2)
If your DNS server only allows a single A record but you want high availability across all nodes, and you're not using MikroTik VIP, use a Virtual IP (VIP) with Keepalived.
### How It Works
- A virtual IP (192.168.30.100) floats between cluster nodes using VRRP protocol
- The master node holds the VIP by default
- If the master fails, a worker node automatically takes over
- All traffic reaches the cluster through a single IP address
- Clients experience automatic failover with minimal downtime
### Prerequisites
- All nodes must be on the same network segment
- Network must support ARP protocol (standard on most networks)
- No other services should use 192.168.30.100
### Installation
#### Step 1: Update Your VIP Address
Edit `vip-setup.yml` and change the VIP to an unused IP on your network:
```yaml
vars:
vip_address: "192.168.30.100" # Change this to your desired VIP
vip_interface: "eth0" # Change if your interface is different
```
#### Step 2: Run the VIP Setup Playbook
```bash
ansible-playbook vip-setup.yml
```
This will:
- Install Keepalived on all nodes
- Configure VRRP with master on cm4-01 and backup on workers
- Set up health checks for automatic failover
- Enable the virtual IP
#### Step 3: Verify VIP is Active
Check that the VIP is assigned to the master node:
```bash
# From your local machine
ping 192.168.30.100
# From any cluster node
ssh pi@192.168.30.101
ip addr show
# Look for your VIP address in the output
```
#### Step 4: Update DNS Records
Now you can use just one A record pointing to the VIP:
```dns
test.zlor.fi A 192.168.30.100
```
#### Step 5: Update Ingress (Optional)
If you want to reference the VIP in your ingress, update the manifest:
```yaml
spec:
rules:
- host: test.zlor.fi
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-test
port:
number: 80
```
The ingress is already correct - it will reach the cluster through any node IP.
### Monitoring the VIP
Check VIP status and failover behavior:
```bash
# View Keepalived status
ssh pi@192.168.30.101
systemctl status keepalived
# Watch VIP transitions (open in separate terminal)
watch 'ip addr show | grep 192.168.30.100'
# View Keepalived logs
sudo journalctl -u keepalived -f
# Check health check script
sudo cat /usr/local/bin/check_apiserver.sh
```
### Testing Failover
To test automatic failover:
1. Note which node has the VIP:
```bash
for ip in 192.168.30.{101..104}; do
echo "=== $ip ==="
ssh pi@$ip "ip addr show | grep 192.168.30.100" 2>/dev/null || echo "Not on this node"
done
```
1. SSH into the node holding the VIP and stop keepalived:
```bash
ssh pi@192.168.30.101 # or whichever node has the VIP
sudo systemctl stop keepalived
```
1. Watch the VIP migrate to another node:
```bash
# From another terminal, watch the migration
ping 192.168.30.100 -c 5
# Connection may drop briefly, then resume on new node
```
1. Restart keepalived on the original node:
```bash
sudo systemctl start keepalived
```
### Troubleshooting VIP
#### VIP is not appearing on any node
Check if Keepalived is running:
```bash
ssh pi@192.168.30.101
sudo systemctl status keepalived
sudo journalctl -u keepalived -n 20
```
Verify the interface name:
```bash
ip route | grep default # Should show your interface name
```
Update `vip_interface` in `vip-setup.yml` if needed and re-run.
#### VIP keeps switching between nodes
This indicates the health check is failing. Verify:
```bash
# Check if API server is responding
curl -k https://127.0.0.1:6443/healthz
# Check the health check script
cat /usr/local/bin/check_apiserver.sh
sudo bash /usr/local/bin/check_apiserver.sh
```
#### DNS resolves but connections time out
Verify all nodes have the VIP configured:
```bash
for ip in 192.168.30.{101..104}; do
echo "=== $ip ==="
ssh pi@$ip "ip addr show | grep 192.168.30.100"
done
```
Test direct connectivity to the VIP from each node:
```bash
ssh pi@192.168.30.101
curl -H "Host: test.zlor.fi" http://192.168.30.100
```
### Disabling VIP
If you no longer need the VIP:
```bash
# Stop Keepalived on all nodes
ansible all -m systemd -a "name=keepalived state=stopped enabled=no" --become
# Remove configuration
ansible all -m file -a "path=/etc/keepalived/keepalived.conf state=absent" --become
```
## Uninstall ## Uninstall
To completely remove k3s from all nodes: To completely remove k3s from all nodes:

159
vip-setup.yml
View File

@@ -1,159 +0,0 @@
---
- name: Configure Virtual IP (VIP) with Keepalived for k3s cluster
hosts: all
become: yes
vars:
vip_address: "192.168.30.100" # Change this to your desired VIP
vip_netmask: "255.255.255.0"
vip_interface: "eth0" # Change if your interface is different
cluster_nodes:
- 192.168.30.101
- 192.168.30.102
- 192.168.30.103
- 192.168.30.104
pre_tasks:
- name: Determine interface name
shell: |
ip route | grep default | awk '{print $5}' | head -1
register: default_interface
changed_when: false
- name: Set interface fact
set_fact:
vip_interface: "{{ default_interface.stdout }}"
tasks:
- name: Install Keepalived
apt:
name: keepalived
state: present
update_cache: yes
- name: Configure Keepalived on master node
template:
content: |
global_defs {
router_id K3S_MASTER
script_user root
enable_script_security
}
vrrp_script check_apiserver {
script "/usr/local/bin/check_apiserver.sh"
interval 3
weight -2
fall 5
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface {{ vip_interface }}
virtual_router_id 51
priority 100
advert_int 1
nopreempt
virtual_ipaddress {
{{ vip_address }}/24
}
track_script {
check_apiserver
}
}
dest: /etc/keepalived/keepalived.conf
owner: root
group: root
mode: '0644'
notify: Restart Keepalived
when: inventory_hostname in groups['master']
- name: Configure Keepalived on worker nodes
template:
content: |
global_defs {
router_id K3S_WORKER
script_user root
enable_script_security
}
vrrp_script check_apiserver {
script "/usr/local/bin/check_apiserver.sh"
interval 3
weight -2
fall 5
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface {{ vip_interface }}
virtual_router_id 51
priority 50
advert_int 1
virtual_ipaddress {
{{ vip_address }}/24
}
track_script {
check_apiserver
}
}
dest: /etc/keepalived/keepalived.conf
owner: root
group: root
mode: '0644'
notify: Restart Keepalived
when: inventory_hostname not in groups['master']
- name: Create API server health check script
copy:
content: |
#!/bin/bash
# Check if k3s API server is responding
curl -sf https://127.0.0.1:6443/healthz > /dev/null 2>&1
exit $?
dest: /usr/local/bin/check_apiserver.sh
owner: root
group: root
mode: '0755'
- name: Enable and start Keepalived
systemd:
name: keepalived
enabled: yes
state: started
daemon_reload: yes
- name: Verify VIP is assigned
shell: |
ip addr show {{ vip_interface }} | grep {{ vip_address }}
register: vip_check
retries: 3
delay: 2
until: vip_check is succeeded
changed_when: false
- name: Display VIP configuration
debug:
msg: |
Virtual IP configured successfully!
VIP Address: {{ vip_address }}
Interface: {{ vip_interface }}
Use this IP for your DNS records:
test.zlor.fi A {{ vip_address }}
The VIP will automatically failover to a worker node
if the master node becomes unavailable.
handlers:
- name: Restart Keepalived
systemd:
name: keepalived
state: restarted
daemon_reload: yes