From 655f07a19811a2e55e21efdbc691003a1364dda0 Mon Sep 17 00:00:00 2001 From: timmy Date: Sat, 18 Apr 2026 09:23:23 +0800 Subject: [PATCH] docs: add NPM gRPC fix for external HTTPS exposure Document the fix for 'server closed the stream without sending trailers' gRPC error when exposing NetBird via Nginx Proxy Manager: - NPM proxy_host advanced_config with grpc_pass for Signal/Management - Caddy h2c listener enabled via 'protocols h1 h2 h2c' --- README.md | 1 + SUMMARY.md | 4 +- npm-grpc-fix.md | 195 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 npm-grpc-fix.md diff --git a/README.md b/README.md index 41dff81..0ba035e 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ |------|------| | [QUICKSTART.md](./QUICKSTART.md) | 快速部署步驟（適合複製貼上執行） | | [netbird-selfhosted-setup.md](./netbird-selfhosted-setup.md) | 完整安裝指南（含原理說明） | +| [npm-grpc-fix.md](./npm-grpc-fix.md) | 透過 Nginx Proxy Manager 對外公開時的 gRPC 修復 | | [SUMMARY.md](./SUMMARY.md) | 部署摘要與疑難排解記錄 | ## 🚀 快速開始 diff --git a/SUMMARY.md b/SUMMARY.md index a012614..287ca24 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -80,6 +80,7 @@ reopen fd 8: permission denied | Docker Compose `crun` runtime 未知 | `/etc/docker/daemon.json` 設定未生效 | `systemctl restart docker` | | apt 的 crun 1.14.1 報 `unknown version specified` | crun 的 OCI spec 1.0.0 與 Docker 29.4 不相容 | 從 GitHub 下載 crun 1.20 | | 502 Bad Gateway（所有 API 端點） | netbird-server 下載 GeoLite2 中，尚未開始監聽 port 80 | 等待 2-5 分鐘 | +| 手機 App：`server closed the stream without sending trailers` | 透過 NPM 對外公開時，NPM 用 `proxy_pass` 而非 `grpc_pass` 轉 gRPC；加上 Caddy `:80` 預設不支援 h2c | NPM 加 `advanced_config`（gRPC 路由）+ Caddy 全域啟用 `protocols h1 h2 h2c`；見 [npm-grpc-fix.md](./npm-grpc-fix.md) | ## 檔案位置 @@ -89,7 +90,8 @@ reopen fd 8: permission denied ├── README.md ← 專案總覽 ├── QUICKSTART.md ← 快速部署步驟 ├── SUMMARY.md ← 本檔（部署摘要） -└── netbird-selfhosted-setup.md ← 完整安裝指南 +├── netbird-selfhosted-setup.md ← 完整安裝指南 +└── npm-grpc-fix.md ← 經 NPM 對外公開時的 gRPC 修復 ``` ### 遠端伺服器（192.168.42.127:/opt/netbird/） diff --git a/npm-grpc-fix.md b/npm-grpc-fix.md new file mode 100644 index 0000000..491cbc4 --- /dev/null +++ b/npm-grpc-fix.md @@ -0,0 +1,195 @@ +# 透過公網域名暴露 NetBird (含 gRPC 修復) + +> 情境：原本 NetBird 部署於內網 `http://192.168.42.127`，需透過 Nginx Proxy Manager (`192.168.42.124`) 以公開域名 `https://netbird.timmy.us.kg` 對外提供服務。 + +## 問題現象 + +手機端 NetBird App 連線 `https://netbird.timmy.us.kg` 失敗： + +``` +login failed: failed getting Management Service public key: +rpc error: code = Internal desc = server closed the stream without sending trailers +``` + +### 觀察 + +- Dashboard 本身 `https://netbird.timmy.us.kg/` 正常（HTTP/2 200） +- OAuth2 discovery `/oauth2/.well-known/openid-configuration` 正常 +- 只有 gRPC（Management / Signal）失敗 + +「server closed the stream without sending trailers」是反向代理不正確處理 gRPC 的典型特徵 — 多數情況是代理用了 HTTP/1.1 的 `proxy_pass` 轉送 gRPC。 + +## 流量路徑 + +``` +Internet → 125.229.110.50 (HiNet WAN) + → 192.168.42.10 (OpenWrt, L4 DNAT :443) + → 192.168.42.124 (Nginx Proxy Manager / openresty) + → 192.168.42.127 (Caddy → netbird-server) +``` + +OpenWrt 只做 L4 DNAT，與 HTTP/2 無關。問題在最末兩段： +1. **NPM** 用 `proxy_pass` + `proxy_http_version 1.1`，gRPC 走不過 +2. **Caddy** 的 `:80` listener 預設只支援 HTTP/1.1，即使 NPM 改成 `grpc_pass`（HTTP/2 明文）也接不到 + +## 修復方案 + +### 1. NPM: 為 NetBird proxy host 加上 gRPC 路由 + +NPM 的 proxy host 設定存於 `/opt/nginx-proxy-manager/data/database.sqlite` 的 `proxy_host` 表。找到 NetBird 那筆（本例為 id=55），將 gRPC/WebSocket 路由寫入 `advanced_config`。 + +**設定內容** (`/tmp/netbird_advanced.conf`)： + +```nginx +# NetBird: long-lived gRPC/WebSocket connections need high timeouts +client_header_timeout 1d; +client_body_timeout 1d; + +# Native gRPC (Signal + Management) +location ~ ^/(signalexchange\.SignalExchange|management\.ManagementService)/ { + grpc_pass grpc://192.168.42.127:80; + grpc_read_timeout 1d; + grpc_send_timeout 1d; + grpc_socket_keepalive on; +} + +# WebSocket upgrades (relay, ws-proxy) +location ~ ^/(relay|ws-proxy/) { + proxy_pass http://192.168.42.127:80; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 1d; +} + +# Plain HTTP routes (API + embedded IdP) +location ~ ^/(api|oauth2)/ { + proxy_pass http://192.168.42.127:80; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; +} +``` + +#### 1a. 寫入資料庫 + +```bash +# 備份 +cp /opt/nginx-proxy-manager/data/database.sqlite \ + /opt/nginx-proxy-manager/data/database.sqlite.bak-$(date +%F) + +# 用 python 寫入（避免 shell quoting 踩到 "upgrade" 裡的雙引號） +python3 <<'PY' +import sqlite3 +with open('/tmp/netbird_advanced.conf') as f: + adv = f.read() +conn = sqlite3.connect('/opt/nginx-proxy-manager/data/database.sqlite') +conn.execute( + 'UPDATE proxy_host SET advanced_config = ?, modified_on = datetime("now") WHERE id = 55', + (adv,)) +conn.commit() +conn.close() +PY +``` + +#### 1b. 重新渲染 nginx config + +⚠️ **注意**：NPM 只有在 UI/API 儲存 proxy host 時才會重新渲染 `/data/nginx/proxy_host/.conf`，重啟容器**不會**觸發。 + +兩種處理方式： + +- **方式 A（推薦）**：登入 NPM UI，進入該 proxy host，按一下 Save。NPM 會從 DB 重新渲染。 +- **方式 B（SSH 直接處理）**：手動把 advanced_config 注入 `.conf`，reload nginx。DB 已更新，日後 NPM 正常渲染會一致。 + +本次採 B： + +```bash +cp /opt/nginx-proxy-manager/data/nginx/proxy_host/55.conf \ + /opt/nginx-proxy-manager/data/nginx/proxy_host/55.conf.bak + +python3 <<'PY' +path = '/opt/nginx-proxy-manager/data/nginx/proxy_host/55.conf' +with open(path) as f: conf = f.read() +with open('/tmp/netbird_advanced.conf') as f: adv = f.read() +marker = ' location / {' +adv_block = ('\n # --- NetBird advanced_config (injected) ---\n' + + '\n'.join(' '+l for l in adv.splitlines()) + + '\n # --- end advanced_config ---\n\n') +open(path,'w').write(conf.replace(marker, adv_block + marker, 1)) +PY + +docker exec nginxproxymanager nginx -t +docker exec nginxproxymanager nginx -s reload +``` + +### 2. Caddy: 啟用 h2c 讓 :80 接受 HTTP/2 明文 + +NPM 的 `grpc_pass grpc://192.168.42.127:80` 會以 HTTP/2 明文連到 Caddy，但 Caddy `:80` listener 預設只支援 HTTP/1.1。在全域設定加上 `protocols h1 h2 h2c`。 + +編輯 `/opt/netbird/Caddyfile`： + +```caddy +{ + servers :80 { + protocols h1 h2 h2c + } +} + +:80 { + @grpc header Content-Type application/grpc* + reverse_proxy @grpc h2c://netbird-server:80 + + @backend path /relay* /ws-proxy/* /api/* /oauth2/* + reverse_proxy @backend netbird-server:80 + + reverse_proxy /* netbird-dashboard:80 +} +``` + +重啟： + +```bash +cd /opt/netbird +docker exec netbird-caddy caddy validate --config /etc/caddy/Caddyfile +docker compose restart caddy +``` + +## 驗證 + +```bash +# gRPC 端點應回 HTTP/2 200 +curl -sS -o /dev/null -w 'HTTP/%{http_version} %{http_code}\n' \ + --http2-prior-knowledge \ + -X POST -H 'content-type: application/grpc' --data-binary '' \ + https://netbird.timmy.us.kg/management.ManagementService/GetPublicKey + +# OAuth2 也應維持正常 +curl -sS -o /dev/null -w 'HTTP/%{http_version} %{http_code}\n' \ + https://netbird.timmy.us.kg/oauth2/.well-known/openid-configuration +``` + +兩者皆需回 `HTTP/2 200`。 + +## 為什麼不在 Caddy 那層完全繞過 NPM？ + +NPM 是這個家用環境所有公開域名共用的入口（處理 Let's Encrypt、存取控管、HTTP/2），只為 NetBird 改 DNAT 會讓證書續簽流程複雜化。保留現有流量路徑，只在 NPM 跟 Caddy 兩邊分別解決各自的協定協商問題，是最小侵入的做法。 + +## 故障排除 + +| 症狀 | 可能原因 | 檢查 | +|------|---------|------| +| 手機 App 仍報同樣 gRPC 錯 | NPM 55.conf 沒 reload 或 Caddy 沒重啟 | `docker exec nginxproxymanager cat /data/nginx/proxy_host/55.conf \| grep grpc_pass` | +| Caddy 報 "unknown directive protocols" | Caddy 版本太舊 | 升級到 `caddy:2.6+` | +| `grpc_pass` 回 502 | Caddy h2c 沒啟用或 netbird-server 沒起來 | `docker ps` + `docker logs netbird-caddy` | +| 改完後 NPM UI 編輯這個 proxy host 看不到 advanced config 內容 | NPM UI 有快取 | 重新整理瀏覽器 | + +## 備份位置 + +- `/opt/nginx-proxy-manager/data/database.sqlite.bak-2026-04-18` +- `/opt/nginx-proxy-manager/data/nginx/proxy_host/55.conf.bak` +- `/opt/netbird/Caddyfile.bak`