docs: NetBird management DNS flap leaves daemon stuck (CT124 case)
Why: CT124 observed "Connecting" from iPhone for hours while CT124 self-reported Connected. Root cause: DNS timeout ~7h earlier broke management gRPC, daemon didn't auto-recover after DNS returned. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -243,6 +243,94 @@ Docker 容器(本文部署的方式):
|
||||
docker restart netbird
|
||||
```
|
||||
|
||||
## Management 斷線後 daemon 不自癒(DNS flap)
|
||||
|
||||
### 症狀
|
||||
|
||||
本部署 CT124 曾發生:從 iPhone 看 CT124 一直顯示 `Connecting`,但 CT124 自己跑 `netbird status` 顯示 `Peers count: 5/5 Connected`。細看:
|
||||
|
||||
```
|
||||
Management: Disconnected, reason: rpc error: code = DeadlineExceeded
|
||||
desc = context deadline exceeded while waiting for connections to become ready
|
||||
Signal: Connected
|
||||
```
|
||||
|
||||
且其他 peer 雖然顯示 `Connected`,但 `Last WireGuard handshake: -`(從未握手)、`Connection type: Relayed`(沒做 ICE)— 表示只是殘留的舊狀態,沒有實際流量。
|
||||
|
||||
### 根因
|
||||
|
||||
容器 log 往上翻會看到類似這段:
|
||||
|
||||
```
|
||||
grpc: addrConn.createTransport failed to connect to {Addr: "netbird.timmy.us.kg:443", ...}.
|
||||
Err: transport: Error while dialing: nbnet.NewDialer().DialContext:
|
||||
dial tcp: lookup netbird.timmy.us.kg: i/o timeout
|
||||
```
|
||||
|
||||
NetBird daemon 啟動後跟 management server 是 gRPC 長連線,若期間 DNS 解析短暫失敗(OpenWrt dnsmasq 抖動、上游 DNS timeout 等),**daemon 的重連邏輯不一定能自癒** — 在 0.68.3 觀察到 DNS 已恢復、其他連線都好了,但 management gRPC 一直卡在失敗狀態。
|
||||
|
||||
後果:
|
||||
- 不會收 network map 更新 → 新加入的 peer 握不到手
|
||||
- 既有 peer 顯示為 Connected 但實際是前一次 session 的陰影,handshake 永不更新
|
||||
- Dashboard 上該 peer 看起來像「Connecting」
|
||||
|
||||
### 辨識方式(黃金一行)
|
||||
|
||||
```bash
|
||||
docker exec netbird netbird status | grep -E "^Management:|^Signal:"
|
||||
```
|
||||
|
||||
正常:
|
||||
|
||||
```
|
||||
Management: Connected
|
||||
Signal: Connected
|
||||
```
|
||||
|
||||
壞掉:
|
||||
|
||||
```
|
||||
Management: Disconnected, reason: ...
|
||||
Signal: Connected
|
||||
```
|
||||
|
||||
### 修復
|
||||
|
||||
直接重啟 daemon / 容器:
|
||||
|
||||
```bash
|
||||
# Docker 部署
|
||||
docker restart netbird
|
||||
|
||||
# Linux systemd
|
||||
systemctl restart netbird
|
||||
|
||||
# macOS
|
||||
sudo launchctl kickstart -k system/netbird
|
||||
```
|
||||
|
||||
Signal 斷線通常會自動重連,但 Management 不一定 — 看到這個症狀**直接重啟是最快的**,不要試著 `netbird down && netbird up`,那不會重建 gRPC channel。
|
||||
|
||||
### 可選:自動 health check
|
||||
|
||||
若部署規模變大、手動檢查不實際,可以加一個 cron / systemd timer 每 5 分鐘掃一次:
|
||||
|
||||
```bash
|
||||
cat > /usr/local/sbin/netbird-mgmt-watchdog.sh << 'EOF'
|
||||
#!/bin/sh
|
||||
if docker exec netbird netbird status 2>/dev/null | grep -q "^Management: Disconnected"; then
|
||||
logger -t netbird-watchdog "Management disconnected, restarting container"
|
||||
docker restart netbird
|
||||
fi
|
||||
EOF
|
||||
chmod +x /usr/local/sbin/netbird-mgmt-watchdog.sh
|
||||
|
||||
# 加進 cron(每 5 分鐘)
|
||||
echo "*/5 * * * * root /usr/local/sbin/netbird-mgmt-watchdog.sh" > /etc/cron.d/netbird-mgmt-watchdog
|
||||
```
|
||||
|
||||
目前本部署未啟用自動 watchdog,peer 數量還在可人工巡檢的範圍。
|
||||
|
||||
## 意外發現:NetBird 內建 UPNP 自動打洞
|
||||
|
||||
觀察 CT101 的啟動日誌:
|
||||
|
||||
Reference in New Issue
Block a user