🔌 Why your Chatwoot API keeps saying 'You need to sign in'

· 3 min read

I was building a billing automation for a managed Chatwoot platform for one of my clients. The plan was simple: daily cron, read the inbox list from Chatwoot, turn it into Moneybird invoices. The API docs were clear. The access token was fresh. And every single call came back with:

{"errors":["You need to sign in or sign up before continuing."]}

HTTP 401. Every endpoint. Every variation of the auth header I tried. User access token, platform token, query parameter, Bearer prefix, no prefix. Same answer.

This is the kind of bug where you spend twenty minutes second-guessing your token and then realize the token was never the problem.

The actual problem

Chatwoot authenticates API calls with a header called api_access_token. Note the underscores.

I am going to spoil this for you right now:

Nginx, by default, silently drops any HTTP header that contains an underscore.

Not rejects. Not warns. Drops. Your request sails through the ingress controller, arrives at the Chatwoot pod, and the application looks at the headers, doesn’t see its auth header, and says “hi, who are you?”

There’s an eight-year-old forum thread where the Cloudron folks ran into the same issue. The fix is a single nginx config line:

underscores_in_headers on;

Why would nginx do this

The short answer: CGI. The long answer: the RFC 3875 CGI specification maps HTTP headers to environment variables by uppercasing them and replacing dashes with underscores. So Content-Type becomes HTTP_CONTENT_TYPE. If a header already contains an underscore, you now have an ambiguity — is HTTP_FOO_BAR the original header Foo-Bar or the original header Foo_Bar? A malicious client could use this to forge headers a CGI backend thinks it’s getting from a trusted source.

Nginx’s authors decided the safest default was to never pass underscored headers at all. That was reasonable in 2004. In 2026, with everyone standardized on JSON APIs and nobody running CGI, it mostly just silently breaks applications built by teams who didn’t know about this quirk.

The fix, in GitOps

I run ingress-nginx as a HelmRelease via Flux. The controller supports this as a config option, exposed through the chart values:

# infrastructure/ingress-nginx/helmrelease.yaml
spec:
  values:
    controller:
      config:
        enable-underscores-in-headers: "true"

Commit, push, Flux reconciles, done. No pod restart needed — the ingress-nginx controller watches its own configmap and reloads on change.

The lesson

When an HTTP API rejects you and you’re sure the credential is right, check whether anything in the path is rewriting or dropping your headers. In order of likelihood:

  1. A reverse proxy (nginx, HAProxy, a CDN) stripping or mangling headers
  2. A middleware layer normalizing header names
  3. Client libraries that lowercase or re-case headers in surprising ways

The test that nailed it for me: curl the Chatwoot pod directly, bypassing the ingress. Worked instantly. That’s when I knew the token was fine and something between me and the pod was the problem.

Also: any time an application authenticates via a header name that doesn’t match the Authorization / X-* conventions, be a little suspicious. Underscores, dots, non-ASCII — if it’s weird, someone in the chain is going to sanitize it.