proxy-station/.claude/skills/edge-node-multi-protocol/references/pitfalls.md

Pitfalls (read before you run)

Every item here cost real debugging time on a live deployment. They are ordered by how likely they are to bite you.

aaPanel

• Never systemctl restart btpanel. btpanel.service is enabled but its ExecStart is /etc/init.d/bt start (SysV); the systemd unit shows active: inactive (dead) while the panel process (webserver) runs fine. systemctl restart btpanel stops the panel and it does not come back → your reverse-proxied panel returns 502. To restart use bt 1 or /etc/init.d/bt start. Same for nginx (/etc/init.d/nginx, not systemctl).
• Reverse-proxy needs CDN_PROXY=True in /www/server/panel/config/cdn.conf, or the panel rejects proxied requests. Restart the panel after setting it.
• Anti-scan fake 404/403 — the UA has to look like a REAL browser, not just contain the word "Mozilla". aaPanel returns a fake nginx 404 to requests it judges to be scanners. On aaPanel 7.x this check got noticeably stricter: a minimal UA like Mozilla/5.0 (Macintosh) (which used to be enough) now gets judged as a scanner and served the fake 404 — even hitting /login or / directly on the box over the unix socket gives the same fake 404 with that UA. Use a full, realistic UA string, e.g. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/605.1.15, and it works immediately. If the entrance/domain/path all test as 404 no matter what you try, check the UA string completeness FIRST — before you start doubting CDN_PROXY, domain binding, or the cert. This one costs the most debugging time of anything in this file. A real browser was never affected either way.
• Also 403 on direct IP:port access once a panel domain is bound — that one's real and expected, not a bug.
• aaPanel 7.x moved several backend modules into a new class_v2 directory (e.g. theme_config.py). Scripts that do sys.path.insert(0, "/www/server/panel/class") and then import panelSite (or anything that transitively imports common.py → theme_config) will fail with ModuleNotFoundError: theme_config unless /www/server/panel/class_v2 is also on sys.path. 09-aapanel-sites.py already does this; if you hand-write a new script that touches panel internals, add the same sys.path.insert(0, "/www/server/panel/class_v2") line.
• Build sites through the panel, never hand-write vhosts. A hand-written /www/server/panel/vhost/nginx/*.conf works in nginx but is invisible/unmanageable in the panel UI. Use the panel classes (AddSite/SetSSL/CreateProxy/HttpToHttps) — that's what 09-aapanel-sites.py does. The panel HTTP API can't be driven by curl (CSRF + anti-scan); drive the classes with the panel's bundled pyenv python.
• nginx 1.24 uses listen 443 ssl http2; — it does not support the newer http2 on; directive; that syntax fails nginx -t.
• The aaPanel install-wait loop must not use pgrep -f install_7.0 to detect "still installing". A command like until run 'test -f BT-Panel && ! pgrep -f install_7.0'; do sleep 15; done has a self-match bug: the check command's own argv literally contains the string "install_7.0", so pgrep -f matches the check itself every time, and the condition can never go true — the loop spins forever even though the install finished in ~1 minute. 04-install-aapanel.sh instead waits for the literal string Congratulations in the install log plus the BT-Panel binary existing. If you ever add a similar poll-by-pgrep check, make sure the pattern you grep for doesn't appear in the checking command's own command line (the classic fix elsewhere is pgrep -f '[i]nstall_7.0' — the bracket trick — but matching a log string is simpler and was chosen here).

3x-ui (newer versions, Xray 26.x) — multi-protocol notes

• Keep inbounds.settings.clients populated. The normalized clients and client_inbounds tables are projections. Upgrades/migrations rebuild them from settings JSON; a table-only deployment can lose all credentials. The migrated 08-xui-bootstrap.py includes this fix; repair older nodes with 08b-embed-settings-clients.py on the VPS after a database backup.

• One client owns all six inbounds — don't create one client per protocol. The whole point of the single-client design is that the panel's Clients page shows ONE row ("node-main") whose Info dialog lists all six share links + QR codes and a single subscription covering all six. That works because the clients table is protocol-generic (next bullet): one row carrying both a uuid and a password, linked via client_inbounds to all six inbounds. 08-xui-bootstrap.py does exactly this. (An earlier iteration created three separate clients — node-main / trojan-main / ss-main — which "worked" for traffic but split the links across three panel rows and made the single-subscription view impossible; consolidating them later meant deleting two clients and re-pointing credentials. Build it as one client from the start.)

• The clients table is protocol-generic, not protocol-specific. It has both a uuid column and a password column (plus VMess/VLESS-only fields like security/flow). VMess/VLESS inbounds read uuid; Trojan/Shadowsocks inbounds read password. There's no "protocol" column on clients itself — which field 3x-ui's backend actually uses depends on the inbound's protocol value at the point it builds the live xray config, not on anything stored on the client row. This isn't documented anywhere obvious; it was confirmed by reading the live generated /usr/local/x-ui/bin/config.json after inserting rows and restarting x-ui, and seeing the right credential land in the right field for each protocol. Consequence: with one client, Trojan and Shadowsocks share ONE password (that client's password column); VMess uses its uuid. That's intended for a single-user node — it's what NODE_PASSWORD in config.env is.

• user_id on an inbound must be set (1), or the inbound is INVISIBLE in the panel. The panel's Inbounds and Clients lists filter by the logged-in admin's user_id. A DB-inserted inbound with a NULL/empty user_id exists in the DB and proxies traffic perfectly (xray loads it, smoke test passes), but the panel UI silently filters it out — you'll see "only the vmess inbounds show up" even though all six are there. 08-xui-bootstrap.py sets user_id=1 on every inserted inbound. If you hand-insert an inbound later, set it too (or UPDATE inbounds SET user_id=1 WHERE user_id IS NULL OR user_id='').

• Every inbound needs an externalProxy in streamSettings, or the generated share links are useless. These inbounds listen on 127.0.0.1 with security:"none" (nginx terminates TLS at the edge on <cover>:443). 3x-ui doesn't know the public address, so by default it generates links pointing at 127.0.0.1:<internal port> with no TLS — which no client can use. Add "externalProxy":[{"forceTls":"tls","dest":"<cover host>","port":443,"sni":"<cover host>","remark":"<name>"}] to each inbound's streamSettings and every generated link/QR/subscription instead points at <cover>:443 over TLS. 08-xui-bootstrap.py does this. (This is a link-generation hint only — it's a 3x-ui-specific field that's stripped when building the live xray config, so it never affects the data path.)

• Shadowsocks inbound settings needs a TOP-LEVEL password field, or the panel's "copy links" endpoint returns an empty 200 and the UI shows "No response data". 3x-ui v3.5.0's share-link generator does settings["password"].(string) — an unchecked Go type assertion. A normal (non-2022) SS inbound has no top-level password (the real per-client password lives on the client), so that assertion hits nil and panics. gin's recovery swallows the panic (its error writer is io.Discard, so nothing reaches journalctl), the /panel/api/clients/subLinks/<subId> endpoint returns HTTP 200 with an EMPTY body, and the frontend — seeing no parseable JSON — reports "No response data" and drops all the links (not just SS). The subscription server returns 500 for the same reason. Fix: write a top-level "password" into the SS inbound's settings (auth still goes through the client table; for chacha20 ciphers this field doesn't even appear in the emitted ss:// link — it exists only to keep the assertion from panicking). 08-xui-bootstrap.py sets it. Debugging note: this presents as "200 empty body", not an error status — so watch for empty-but-200 API responses, and remember the sub server's Go logs are discarded.

• client_traffics.email is GLOBALLY UNIQUE. One client shared across two inbounds (e.g. a protocol's direct + warp inbound) gets one client_traffics row, not one per inbound — inserting two rows with the same email throws UNIQUE constraint failed.

• client_inbounds has a flow_override column (client_id, inbound_id, flow_override, created_at) — include it in the INSERT even though it's usually empty; some 3x-ui builds may reject a 3-column insert if the schema doesn't line up with what an older reference implementation expected.

• Extend xrayTemplateConfig's WARP routing rule, never overwrite the whole thing. The WARP-egress inbounds' tags (and their inbound-127.0.0.1:<port> fallback form) need to be appended to the existing rule's inboundTag array. Re-generating and writing the whole xrayTemplateConfig from scratch (instead of read-modify-write) silently deletes whatever protocols were added in a previous run.

• Auth is by UUID/password, so a mismatch between the traffic-stat email label and the running config email is cosmetic, not a fault.

• Protocol mismatch when testing. A VLESS test client against a vmess+ws inbound is correctly rejected (websocket close 1000) — that's not a server bug, it's the wrong protocol. Match the test client's protocol (vmess/trojan/shadowsocks) exactly to the inbound you're testing, with the right security/method/password for that protocol.

• Panel API /login may 403 on newer builds (security middleware). Do DB-direct edits + systemctl restart x-ui instead of the HTTP API — confirmed true both for the original VMess bootstrap and again when adding Trojan/Shadowsocks later.

• Shadowsocks 2022-blake3-* ciphers need an exact-byte-length base64 password/PSK — get the length wrong and xray fails to start with no obvious error pointing at the cipher. chacha20-poly1305 (or aes-256-gcm) takes a plain password string, is much harder to misconfigure, and is still a modern AEAD cipher — prefer it unless you have a specific reason to use the 2022 series.

Cloudflare WARP

• gpg must be installed first. On a minimal Debian, gpg --dearmor for the repo keyring fails with gpg: command not found, and the later apt update then errors Failed to parse keyring … sqv returned an error code. apt-get install -y gnupg before adding the repo.
• Use the bookworm repo line even on Debian 13 (trixie) — the bookworm package works.
• WARP can lie about being connected. The control plane shows Connected while the SOCKS proxy stops egressing. The watchdog (warp-healthcheck.*) checks warp=on through the proxy and restarts warp-svc when it isn't.

nginx / cover

• WS upgrade must return 101 with --http1.1 + the Upgrade/Connection headers. If you test with HTTP/2 you'll get 400 (HTTP/2 doesn't carry the Upgrade frame).
• Default server for unmatched server_name should return 444 so scanners hitting the bare IP / wrong host get dropped.

Monitoring

• Komari auto-discovery registers the node with an auto-generated name (Auto-<hostname>). Rename/group it in your panel afterward (panel-specific; often a small DB update or a UI action). The agent falls back to a v1 WebSocket if the panel has no v2 API — that's normal.
• Nezha agent install one-liners change over time. If the scripted install fails, drop the nezha-agent binary + a config.yml (server, tls, client_secret, uuid) and a small systemd unit — the agent connects out to SERVER:PORT (gRPC), so only outbound is needed.

SSH / access

• Some providers ship a .pem/.ppk bundle; use the .pem. chmod 600 it or ssh refuses it.
• Fresh boxes may run fail2ban; repeated failed auth from a dynamic IP can lock you out. Get your key in early (step 0) so you're not relying on password auth.