From cb1b5f4bf2682aeac731ec0e9531ac3cc73887f1 Mon Sep 17 00:00:00 2001 From: Timmy Date: Sat, 18 Apr 2026 23:26:02 +0800 Subject: [PATCH] docs: NetBird management DNS flap leaves daemon stuck (CT124 case) Why: CT124 observed "Connecting" from iPhone for hours while CT124 self-reported Connected. Root cause: DNS timeout ~7h earlier broke management gRPC, daemon didn't auto-recover after DNS returned. Co-Authored-By: Claude Opus 4.7 (1M context) --- SUMMARY.md | 1 + peer-deployment-ops.md | 88 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+) diff --git a/SUMMARY.md b/SUMMARY.md index 3fefd11..20a8f7e 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -91,6 +91,7 @@ reopen fd 8: permission denied | 加 subnet route 後部分裝置拿不到路由(`Networks: -`、`Status: Connecting`) | 既有 NetBird daemon 不會自動重新套用 route,卡在舊 state | 各裝置重連 daemon:macOS `launchctl kickstart`、iOS App Disconnect→Connect、Docker `compose restart`;見 [network-routes.md](./network-routes.md) | | iPhone RTSP app 連不到 LAN 攝影機(Safari 打 HTTP 可以) | 某些 iOS app 不走系統 VPN 路由 | 換支援 VPN 的 app(VLC 等),或改用 NetBird FQDN | | Dashboard 顯示 "SSH Access is disabled" 但 `ssh` 指令照通 | Dashboard 的 "SSH Access" 是 NetBird 內建 SSH 通道,不等於目標機器的 OpenSSH | 兩種 SSH 是不同東西,不用理會那個提示;見 [ssh-access.md](./ssh-access.md) | +| 某 peer 一直顯示 Connecting,但其本機 status 看起來 Connected | DNS 短暫 timeout → daemon 的 management gRPC 連線卡死,不自癒(Signal 還 Connected,但 Management Disconnected) | 直接重啟 daemon / container;可選加 watchdog;見 [peer-deployment-ops.md](./peer-deployment-ops.md) | ## 檔案位置 diff --git a/peer-deployment-ops.md b/peer-deployment-ops.md index 3c0c1e1..d1bae72 100644 --- a/peer-deployment-ops.md +++ b/peer-deployment-ops.md @@ -243,6 +243,94 @@ Docker 容器(本文部署的方式): docker restart netbird ``` +## Management 斷線後 daemon 不自癒(DNS flap) + +### 症狀 + +本部署 CT124 曾發生:從 iPhone 看 CT124 一直顯示 `Connecting`,但 CT124 自己跑 `netbird status` 顯示 `Peers count: 5/5 Connected`。細看: + +``` +Management: Disconnected, reason: rpc error: code = DeadlineExceeded + desc = context deadline exceeded while waiting for connections to become ready +Signal: Connected +``` + +且其他 peer 雖然顯示 `Connected`,但 `Last WireGuard handshake: -`(從未握手)、`Connection type: Relayed`(沒做 ICE)— 表示只是殘留的舊狀態,沒有實際流量。 + +### 根因 + +容器 log 往上翻會看到類似這段: + +``` +grpc: addrConn.createTransport failed to connect to {Addr: "netbird.timmy.us.kg:443", ...}. +Err: transport: Error while dialing: nbnet.NewDialer().DialContext: + dial tcp: lookup netbird.timmy.us.kg: i/o timeout +``` + +NetBird daemon 啟動後跟 management server 是 gRPC 長連線,若期間 DNS 解析短暫失敗(OpenWrt dnsmasq 抖動、上游 DNS timeout 等),**daemon 的重連邏輯不一定能自癒** — 在 0.68.3 觀察到 DNS 已恢復、其他連線都好了,但 management gRPC 一直卡在失敗狀態。 + +後果: +- 不會收 network map 更新 → 新加入的 peer 握不到手 +- 既有 peer 顯示為 Connected 但實際是前一次 session 的陰影,handshake 永不更新 +- Dashboard 上該 peer 看起來像「Connecting」 + +### 辨識方式(黃金一行) + +```bash +docker exec netbird netbird status | grep -E "^Management:|^Signal:" +``` + +正常: + +``` +Management: Connected +Signal: Connected +``` + +壞掉: + +``` +Management: Disconnected, reason: ... +Signal: Connected +``` + +### 修復 + +直接重啟 daemon / 容器: + +```bash +# Docker 部署 +docker restart netbird + +# Linux systemd +systemctl restart netbird + +# macOS +sudo launchctl kickstart -k system/netbird +``` + +Signal 斷線通常會自動重連,但 Management 不一定 — 看到這個症狀**直接重啟是最快的**,不要試著 `netbird down && netbird up`,那不會重建 gRPC channel。 + +### 可選:自動 health check + +若部署規模變大、手動檢查不實際,可以加一個 cron / systemd timer 每 5 分鐘掃一次: + +```bash +cat > /usr/local/sbin/netbird-mgmt-watchdog.sh << 'EOF' +#!/bin/sh +if docker exec netbird netbird status 2>/dev/null | grep -q "^Management: Disconnected"; then + logger -t netbird-watchdog "Management disconnected, restarting container" + docker restart netbird +fi +EOF +chmod +x /usr/local/sbin/netbird-mgmt-watchdog.sh + +# 加進 cron(每 5 分鐘) +echo "*/5 * * * * root /usr/local/sbin/netbird-mgmt-watchdog.sh" > /etc/cron.d/netbird-mgmt-watchdog +``` + +目前本部署未啟用自動 watchdog,peer 數量還在可人工巡檢的範圍。 + ## 意外發現:NetBird 內建 UPNP 自動打洞 觀察 CT101 的啟動日誌: