docs: NetBird management DNS flap leaves daemon stuck (CT124 case)
Why: CT124 observed "Connecting" from iPhone for hours while CT124 self-reported Connected. Root cause: DNS timeout ~7h earlier broke management gRPC, daemon didn't auto-recover after DNS returned. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -91,6 +91,7 @@ reopen fd 8: permission denied
|
||||
| 加 subnet route 後部分裝置拿不到路由(`Networks: -`、`Status: Connecting`) | 既有 NetBird daemon 不會自動重新套用 route,卡在舊 state | 各裝置重連 daemon:macOS `launchctl kickstart`、iOS App Disconnect→Connect、Docker `compose restart`;見 [network-routes.md](./network-routes.md) |
|
||||
| iPhone RTSP app 連不到 LAN 攝影機(Safari 打 HTTP 可以) | 某些 iOS app 不走系統 VPN 路由 | 換支援 VPN 的 app(VLC 等),或改用 NetBird FQDN |
|
||||
| Dashboard 顯示 "SSH Access is disabled" 但 `ssh` 指令照通 | Dashboard 的 "SSH Access" 是 NetBird 內建 SSH 通道,不等於目標機器的 OpenSSH | 兩種 SSH 是不同東西,不用理會那個提示;見 [ssh-access.md](./ssh-access.md) |
|
||||
| 某 peer 一直顯示 Connecting,但其本機 status 看起來 Connected | DNS 短暫 timeout → daemon 的 management gRPC 連線卡死,不自癒(Signal 還 Connected,但 Management Disconnected) | 直接重啟 daemon / container;可選加 watchdog;見 [peer-deployment-ops.md](./peer-deployment-ops.md) |
|
||||
|
||||
## 檔案位置
|
||||
|
||||
|
||||
@@ -243,6 +243,94 @@ Docker 容器(本文部署的方式):
|
||||
docker restart netbird
|
||||
```
|
||||
|
||||
## Management 斷線後 daemon 不自癒(DNS flap)
|
||||
|
||||
### 症狀
|
||||
|
||||
本部署 CT124 曾發生:從 iPhone 看 CT124 一直顯示 `Connecting`,但 CT124 自己跑 `netbird status` 顯示 `Peers count: 5/5 Connected`。細看:
|
||||
|
||||
```
|
||||
Management: Disconnected, reason: rpc error: code = DeadlineExceeded
|
||||
desc = context deadline exceeded while waiting for connections to become ready
|
||||
Signal: Connected
|
||||
```
|
||||
|
||||
且其他 peer 雖然顯示 `Connected`,但 `Last WireGuard handshake: -`(從未握手)、`Connection type: Relayed`(沒做 ICE)— 表示只是殘留的舊狀態,沒有實際流量。
|
||||
|
||||
### 根因
|
||||
|
||||
容器 log 往上翻會看到類似這段:
|
||||
|
||||
```
|
||||
grpc: addrConn.createTransport failed to connect to {Addr: "netbird.timmy.us.kg:443", ...}.
|
||||
Err: transport: Error while dialing: nbnet.NewDialer().DialContext:
|
||||
dial tcp: lookup netbird.timmy.us.kg: i/o timeout
|
||||
```
|
||||
|
||||
NetBird daemon 啟動後跟 management server 是 gRPC 長連線,若期間 DNS 解析短暫失敗(OpenWrt dnsmasq 抖動、上游 DNS timeout 等),**daemon 的重連邏輯不一定能自癒** — 在 0.68.3 觀察到 DNS 已恢復、其他連線都好了,但 management gRPC 一直卡在失敗狀態。
|
||||
|
||||
後果:
|
||||
- 不會收 network map 更新 → 新加入的 peer 握不到手
|
||||
- 既有 peer 顯示為 Connected 但實際是前一次 session 的陰影,handshake 永不更新
|
||||
- Dashboard 上該 peer 看起來像「Connecting」
|
||||
|
||||
### 辨識方式(黃金一行)
|
||||
|
||||
```bash
|
||||
docker exec netbird netbird status | grep -E "^Management:|^Signal:"
|
||||
```
|
||||
|
||||
正常:
|
||||
|
||||
```
|
||||
Management: Connected
|
||||
Signal: Connected
|
||||
```
|
||||
|
||||
壞掉:
|
||||
|
||||
```
|
||||
Management: Disconnected, reason: ...
|
||||
Signal: Connected
|
||||
```
|
||||
|
||||
### 修復
|
||||
|
||||
直接重啟 daemon / 容器:
|
||||
|
||||
```bash
|
||||
# Docker 部署
|
||||
docker restart netbird
|
||||
|
||||
# Linux systemd
|
||||
systemctl restart netbird
|
||||
|
||||
# macOS
|
||||
sudo launchctl kickstart -k system/netbird
|
||||
```
|
||||
|
||||
Signal 斷線通常會自動重連,但 Management 不一定 — 看到這個症狀**直接重啟是最快的**,不要試著 `netbird down && netbird up`,那不會重建 gRPC channel。
|
||||
|
||||
### 可選:自動 health check
|
||||
|
||||
若部署規模變大、手動檢查不實際,可以加一個 cron / systemd timer 每 5 分鐘掃一次:
|
||||
|
||||
```bash
|
||||
cat > /usr/local/sbin/netbird-mgmt-watchdog.sh << 'EOF'
|
||||
#!/bin/sh
|
||||
if docker exec netbird netbird status 2>/dev/null | grep -q "^Management: Disconnected"; then
|
||||
logger -t netbird-watchdog "Management disconnected, restarting container"
|
||||
docker restart netbird
|
||||
fi
|
||||
EOF
|
||||
chmod +x /usr/local/sbin/netbird-mgmt-watchdog.sh
|
||||
|
||||
# 加進 cron(每 5 分鐘)
|
||||
echo "*/5 * * * * root /usr/local/sbin/netbird-mgmt-watchdog.sh" > /etc/cron.d/netbird-mgmt-watchdog
|
||||
```
|
||||
|
||||
目前本部署未啟用自動 watchdog,peer 數量還在可人工巡檢的範圍。
|
||||
|
||||
## 意外發現:NetBird 內建 UPNP 自動打洞
|
||||
|
||||
觀察 CT101 的啟動日誌:
|
||||
|
||||
Reference in New Issue
Block a user