proxy-station/.claude/skills/edge-node-multi-protocol/references/pitfalls.md

# Pitfalls (read before you run)

Every item here cost real debugging time on a live deployment. They are ordered by how likely they are to bite you.

## aaPanel

- **Never `systemctl restart btpanel`.** `btpanel.service` is `enabled` but its `ExecStart` is `/etc/init.d/bt start` (SysV); the systemd unit shows `active: inactive (dead)` while the panel process (`webserver`) runs fine. `systemctl restart btpanel` stops the panel and it does **not** come back → your reverse-proxied panel returns **502**. To restart use `bt 1` or `/etc/init.d/bt start`. Same for nginx (`/etc/init.d/nginx`, not `systemctl`).
- **Reverse-proxy needs `CDN_PROXY=True`** in `/www/server/panel/config/cdn.conf`, or the panel rejects proxied requests. Restart the panel after setting it.
- **Anti-scan fake 404/403 — the UA has to look like a REAL browser, not just contain the word "Mozilla".** aaPanel returns a *fake* nginx 404 to requests it judges to be scanners. On aaPanel 7.x this check got noticeably stricter: a minimal UA like `Mozilla/5.0 (Macintosh)` (which used to be enough) now gets judged as a scanner and served the fake 404 — even hitting `/login` or `/` directly on the box over the unix socket gives the same fake 404 with that UA. Use a full, realistic UA string, e.g. `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/605.1.15`, and it works immediately. **If the entrance/domain/path all test as 404 no matter what you try, check the UA string completeness FIRST** — before you start doubting `CDN_PROXY`, domain binding, or the cert. This one costs the most debugging time of anything in this file. A real browser was never affected either way.
- Also **403** on direct `IP:port` access once a panel domain is bound — that one's real and expected, not a bug.
- **aaPanel 7.x moved several backend modules into a new `class_v2` directory** (e.g. `theme_config.py`). Scripts that do `sys.path.insert(0, "/www/server/panel/class")` and then `import panelSite` (or anything that transitively imports `common.py` → `theme_config`) will fail with `ModuleNotFoundError: theme_config` unless `/www/server/panel/class_v2` is *also* on `sys.path`. `09-aapanel-sites.py` already does this; if you hand-write a new script that touches panel internals, add the same `sys.path.insert(0, "/www/server/panel/class_v2")` line.
- **Build sites through the panel, never hand-write vhosts.** A hand-written `/www/server/panel/vhost/nginx/*.conf` works in nginx but is invisible/unmanageable in the panel UI. Use the panel classes (`AddSite`/`SetSSL`/`CreateProxy`/`HttpToHttps`) — that's what `09-aapanel-sites.py` does. The panel HTTP API can't be driven by curl (CSRF + anti-scan); drive the classes with the panel's bundled `pyenv` python.
- **nginx 1.24 uses `listen 443 ssl http2;`** — it does **not** support the newer `http2 on;` directive; that syntax fails `nginx -t`.
- **The aaPanel install-wait loop must not use `pgrep -f install_7.0` to detect "still installing".** A command like `until run 'test -f BT-Panel && ! pgrep -f install_7.0'; do sleep 15; done` has a self-match bug: the check command's own argv literally contains the string `"install_7.0"`, so `pgrep -f` matches the check itself every time, and the condition can never go true — the loop spins forever even though the install finished in ~1 minute. `04-install-aapanel.sh` instead waits for the literal string `Congratulations` in the install log plus the `BT-Panel` binary existing. If you ever add a similar poll-by-pgrep check, make sure the pattern you grep for doesn't appear in the checking command's own command line (the classic fix elsewhere is `pgrep -f '[i]nstall_7.0'` — the bracket trick — but matching a log string is simpler and was chosen here).

## 3x-ui (newer versions, Xray 26.x) — multi-protocol notes

- **Keep `inbounds.settings.clients` populated.** The normalized `clients` and `client_inbounds` tables are projections. Upgrades/migrations rebuild them from settings JSON; a table-only deployment can lose all credentials. The migrated `08-xui-bootstrap.py` includes this fix; repair older nodes with `08b-embed-settings-clients.py` on the VPS after a database backup.

- **One client owns all six inbounds — don't create one client per protocol.** The whole point of the single-client design is that the panel's Clients page shows ONE row ("node-main") whose Info dialog lists all six share links + QR codes and a single subscription covering all six. That works because the `clients` table is protocol-generic (next bullet): one row carrying both a `uuid` and a `password`, linked via `client_inbounds` to all six inbounds. `08-xui-bootstrap.py` does exactly this. (An earlier iteration created three separate clients — node-main / trojan-main / ss-main — which "worked" for traffic but split the links across three panel rows and made the single-subscription view impossible; consolidating them later meant deleting two clients and re-pointing credentials. Build it as one client from the start.)
- **The `clients` table is protocol-generic, not protocol-specific.** It has both a `uuid` column and a `password` column (plus VMess/VLESS-only fields like `security`/`flow`). VMess/VLESS inbounds read `uuid`; Trojan/Shadowsocks inbounds read `password`. There's no "protocol" column on `clients` itself — which field 3x-ui's backend actually uses depends on the *inbound's* `protocol` value at the point it builds the live xray config, not on anything stored on the client row. This isn't documented anywhere obvious; it was confirmed by reading the live generated `/usr/local/x-ui/bin/config.json` after inserting rows and restarting `x-ui`, and seeing the right credential land in the right field for each protocol. **Consequence:** with one client, Trojan and Shadowsocks share ONE password (that client's `password` column); VMess uses its `uuid`. That's intended for a single-user node — it's what `NODE_PASSWORD` in `config.env` is.
- **`user_id` on an inbound must be set (1), or the inbound is INVISIBLE in the panel.** The panel's Inbounds and Clients lists filter by the logged-in admin's `user_id`. A DB-inserted inbound with a NULL/empty `user_id` exists in the DB and proxies traffic perfectly (xray loads it, smoke test passes), but the panel UI silently filters it out — you'll see "only the vmess inbounds show up" even though all six are there. `08-xui-bootstrap.py` sets `user_id=1` on every inserted inbound. If you hand-insert an inbound later, set it too (or `UPDATE inbounds SET user_id=1 WHERE user_id IS NULL OR user_id=''`).
- **Every inbound needs an `externalProxy` in streamSettings, or the generated share links are useless.** These inbounds listen on `127.0.0.1` with `security:"none"` (nginx terminates TLS at the edge on `<cover>:443`). 3x-ui doesn't know the public address, so by default it generates links pointing at `127.0.0.1:<internal port>` with no TLS — which no client can use. Add `"externalProxy":[{"forceTls":"tls","dest":"<cover host>","port":443,"sni":"<cover host>","remark":"<name>"}]` to each inbound's streamSettings and every generated link/QR/subscription instead points at `<cover>:443` over TLS. `08-xui-bootstrap.py` does this. (This is a *link-generation* hint only — it's a 3x-ui-specific field that's stripped when building the live xray config, so it never affects the data path.)
- **Shadowsocks inbound `settings` needs a TOP-LEVEL `password` field, or the panel's "copy links" endpoint returns an empty 200 and the UI shows "No response data".** 3x-ui v3.5.0's share-link generator does `settings["password"].(string)` — an *unchecked* Go type assertion. A normal (non-2022) SS inbound has no top-level password (the real per-client password lives on the client), so that assertion hits `nil` and **panics**. gin's recovery swallows the panic (its error writer is `io.Discard`, so nothing reaches `journalctl`), the `/panel/api/clients/subLinks/<subId>` endpoint returns HTTP 200 with an EMPTY body, and the frontend — seeing no parseable JSON — reports "No response data" and drops *all* the links (not just SS). The subscription server returns 500 for the same reason. Fix: write a top-level `"password"` into the SS inbound's `settings` (auth still goes through the client table; for chacha20 ciphers this field doesn't even appear in the emitted `ss://` link — it exists only to keep the assertion from panicking). `08-xui-bootstrap.py` sets it. Debugging note: this presents as "200 empty body", not an error status — so watch for empty-but-200 API responses, and remember the sub server's Go logs are discarded.
- **`client_traffics.email` is GLOBALLY UNIQUE.** One client shared across two inbounds (e.g. a protocol's direct + warp inbound) gets **one** `client_traffics` row, not one per inbound — inserting two rows with the same email throws `UNIQUE constraint failed`.
- **`client_inbounds` has a `flow_override` column** (client_id, inbound_id, flow_override, created_at) — include it in the INSERT even though it's usually empty; some 3x-ui builds may reject a 3-column insert if the schema doesn't line up with what an older reference implementation expected.
- **Extend `xrayTemplateConfig`'s WARP routing rule, never overwrite the whole thing.** The WARP-egress inbounds' tags (and their `inbound-127.0.0.1:<port>` fallback form) need to be appended to the *existing* rule's `inboundTag` array. Re-generating and writing the whole `xrayTemplateConfig` from scratch (instead of read-modify-write) silently deletes whatever protocols were added in a previous run.
- **Auth is by UUID/password**, so a mismatch between the traffic-stat email label and the running config email is cosmetic, not a fault.
- **Protocol mismatch when testing.** A VLESS test client against a `vmess+ws` inbound is correctly rejected (`websocket close 1000`) — that's not a server bug, it's the wrong protocol. Match the test client's protocol (vmess/trojan/shadowsocks) exactly to the inbound you're testing, with the right `security`/`method`/`password` for that protocol.
- **Panel API `/login` may 403** on newer builds (security middleware). Do DB-direct edits + `systemctl restart x-ui` instead of the HTTP API — confirmed true both for the original VMess bootstrap and again when adding Trojan/Shadowsocks later.
- **Shadowsocks 2022-blake3-\* ciphers need an exact-byte-length base64 password/PSK** — get the length wrong and xray fails to start with no obvious error pointing at the cipher. `chacha20-poly1305` (or `aes-256-gcm`) takes a plain password string, is much harder to misconfigure, and is still a modern AEAD cipher — prefer it unless you have a specific reason to use the 2022 series.

## Cloudflare WARP

- **`gpg` must be installed first.** On a minimal Debian, `gpg --dearmor` for the repo keyring fails with `gpg: command not found`, and the later `apt update` then errors `Failed to parse keyring … sqv returned an error code`. `apt-get install -y gnupg` before adding the repo.
- Use the **`bookworm`** repo line even on Debian 13 (trixie) — the bookworm package works.
- **WARP can lie about being connected.** The control plane shows `Connected` while the SOCKS proxy stops egressing. The watchdog (`warp-healthcheck.*`) checks `warp=on` *through the proxy* and restarts `warp-svc` when it isn't.

## nginx / cover

- WS upgrade must return **101** with `--http1.1` + the `Upgrade`/`Connection` headers. If you test with HTTP/2 you'll get 400 (HTTP/2 doesn't carry the Upgrade frame).
- Default server for unmatched `server_name` should `return 444` so scanners hitting the bare IP / wrong host get dropped.

## Monitoring

- **Komari** auto-discovery registers the node with an auto-generated name (`Auto-<hostname>`). Rename/group it in your panel afterward (panel-specific; often a small DB update or a UI action). The agent falls back to a v1 WebSocket if the panel has no v2 API — that's normal.
- **Nezha** agent install one-liners change over time. If the scripted install fails, drop the `nezha-agent` binary + a `config.yml` (server, `tls`, `client_secret`, `uuid`) and a small systemd unit — the agent connects out to `SERVER:PORT` (gRPC), so only outbound is needed.

## SSH / access

- Some providers ship a `.pem`/`.ppk` bundle; use the `.pem`. `chmod 600` it or ssh refuses it.
- Fresh boxes may run fail2ban; repeated failed auth from a dynamic IP can lock you out. Get your key in early (step 0) so you're not relying on password auth.