ubuntu-post-install/services/mattermost.md

## Android push notifications inconsistent after a migration (e.g. from PikaPods)

Symptom: "Enable Push Notifications" is on in System Console, but only some
Android users actually get background push notifications — one user gets
them reliably, others on the same server don't. Since this is per-device
rather than server-wide, work through these in order; the first that
reproduces the symptom is almost always the actual cause.

### 1. Rule out server → push-proxy connectivity first (quick, and if this is
broken it explains ALL users failing, not just some)

The mobile app talks to Google's FCM directly for the device token, but it's
*your* Mattermost server that calls out to the push relay (default
`https://push.mattermost.com`, System Console → Environment → Push
Notification Server) every time it needs to fan out a push. Confirm the new
VPS can actually reach it — a fresh box's outbound rules, or a NAT/firewall
inherited from the migration, can block this silently:

```bash
docker exec mattermost curl -Is https://push.mattermost.com | head -1
```

Then check the server's own logs for push attempts/failures:

```bash
docker compose logs mattermost | grep -i push
```

If specific users' pushes error out while others succeed, that already rules
out a global connectivity/config problem and points at something per-account
(section 2) or per-device (section 3).

### 2. Stale device registration carried over by the migration

A SQL dump import (`migrate-from-pikapods.sh` or any other DB restore) brings
the `Sessions` table with it — including each user's `DeviceId`, the
FCM token that was registered against the *old* server. That registration
only gets refreshed on a real login, not by the app quietly staying open:
a session that survived the move keeps working perfectly for live chat
(the websocket connection has nothing to do with push registration) while
its background push silently stops working, because the token behind it may
now be stale.

This matches "one user is always fine, everyone else isn't" almost exactly —
the working user is typically the one who happened to log out/in (or
reinstalled the app) since the migration, refreshing their `DeviceId`, while
everyone else's session rode through the import unchanged.

**Fix:** have affected users fully log out of the Mattermost Android app
(not just background it — Menu → Log Out) and log back in. This forces a
fresh device-token registration against the current server.

### 3. Android OEM battery optimization (the most common purely-device-side cause)

Xiaomi/MIUI, Huawei, Samsung, OnePlus, and Oppo/Vivo all ship aggressive
battery managers that kill background apps and their FCM listeners by
default — independent of anything about the server. This is the single most
common reason some Android phones on the exact same server get pushes and
others don't, migration or no migration. Have affected users check, per
device:

- **Settings → Apps → Mattermost → Battery** → set to "Unrestricted" / "No
  restrictions" / disable "Battery Saver" for the app (menu wording varies
  by OEM/Android version).
- **Notification permission itself** is still granted — Android 13+ requires
  an explicit runtime permission that can get silently revoked (e.g. after
  an OS update), separate from the app's own in-app notification settings.
- Some OEMs (Xiaomi especially) also gate this behind a separate
  "Autostart" toggle for the app.

### 4. Push notification content setting, if section 1 and 2 don't explain it

System Console → Environment → Push Notification Server → **Push
Notification Contents**. If set to anything other than "Send full message
contents", the client has to phone the server's own `SiteURL` back for the
real content after getting the push shell — so if the *new* domain isn't
reliably reachable from a given user's network (split-horizon DNS, a mobile
carrier blocking something, a half-finished Caddy/DNS cutover for the new
VPS), that user can receive the push notification itself but never see
real content, or see it inconsistently. Temporarily switching to "Send full
message contents" removes this variable while narrowing down the cause.

### 5. A changed FQDN specifically — DNS propagation and cert readiness

If the migration also moved the server to a new domain (not just a new box
under the same domain), that alone can produce exactly this
some-users-fine/some-users-not pattern for a few days after cutover, on top
of section 2 and 4 above:

- **DNS propagation lags per device.** Different users' resolvers (ISP DNS,
  carrier DNS on cellular, cached records with old TTLs) pick up the new
  FQDN's IP at different times. A user on a fast public resolver sees it
  immediately; someone on carrier DNS with a stale cache might not resolve
  it correctly for hours. Combined with section 4 (anything but "full
  message contents" requires a content-fetch call back to `SiteURL`), a
  device with a stale answer for the new FQDN fails that step while others
  succeed.
- **TLS certificate not fully issued/propagated yet** for the new FQDN
  (Caddy/Let's Encrypt) causes the same content-fetch failure via cert
  validation instead of DNS.
- **Confirm `MM_SERVICESETTINGS_SITEURL` in `.env` is actually the new FQDN**
  — a value left over from before the domain change points every client's
  content-fetch at the wrong place, consistently, not just intermittently.
- Users who never explicitly added the new server URL in the mobile app (a
  kept redirect from the old domain let them keep working without
  noticing) are still running on their old, stale device registration —
  this is section 2's mechanism, just caused directly by the FQDN change
  rather than by the DB import alone.