ubuntu-post-install/services/mattermost.md

Android push notifications inconsistent after a migration (e.g. from PikaPods)

Symptom: "Enable Push Notifications" is on in System Console, but only some Android users actually get background push notifications — one user gets them reliably, others on the same server don't. Since this is per-device rather than server-wide, work through these in order; the first that reproduces the symptom is almost always the actual cause.

1. Rule out server → push-proxy connectivity first (quick, and if this is

broken it explains ALL users failing, not just some)

The mobile app talks to Google's FCM directly for the device token, but it's your Mattermost server that calls out to the push relay (default https://push.mattermost.com, System Console → Environment → Push Notification Server) every time it needs to fan out a push. Confirm the new VPS can actually reach it — a fresh box's outbound rules, or a NAT/firewall inherited from the migration, can block this silently:

docker exec mattermost curl -Is https://push.mattermost.com | head -1

Then check the server's own logs for push attempts/failures:

docker compose logs mattermost | grep -i push

If specific users' pushes error out while others succeed, that already rules out a global connectivity/config problem and points at something per-account (section 2) or per-device (section 3).

2. Stale device registration carried over by the migration

A SQL dump import (migrate-from-pikapods.sh or any other DB restore) brings the Sessions table with it — including each user's DeviceId, the FCM token that was registered against the old server. That registration only gets refreshed on a real login, not by the app quietly staying open: a session that survived the move keeps working perfectly for live chat (the websocket connection has nothing to do with push registration) while its background push silently stops working, because the token behind it may now be stale.

This matches "one user is always fine, everyone else isn't" almost exactly — the working user is typically the one who happened to log out/in (or reinstalled the app) since the migration, refreshing their DeviceId, while everyone else's session rode through the import unchanged.

Fix: have affected users fully log out of the Mattermost Android app (not just background it — Menu → Log Out) and log back in. This forces a fresh device-token registration against the current server.

3. Android OEM battery optimization (the most common purely-device-side cause)

Xiaomi/MIUI, Huawei, Samsung, OnePlus, and Oppo/Vivo all ship aggressive battery managers that kill background apps and their FCM listeners by default — independent of anything about the server. This is the single most common reason some Android phones on the exact same server get pushes and others don't, migration or no migration. Have affected users check, per device:

• Settings → Apps → Mattermost → Battery → set to "Unrestricted" / "No restrictions" / disable "Battery Saver" for the app (menu wording varies by OEM/Android version).
• Notification permission itself is still granted — Android 13+ requires an explicit runtime permission that can get silently revoked (e.g. after an OS update), separate from the app's own in-app notification settings.
• Some OEMs (Xiaomi especially) also gate this behind a separate "Autostart" toggle for the app.

4. Push notification content setting, if section 1 and 2 don't explain it

System Console → Environment → Push Notification Server → Push Notification Contents. If set to anything other than "Send full message contents", the client has to phone the server's own SiteURL back for the real content after getting the push shell — so if the new domain isn't reliably reachable from a given user's network (split-horizon DNS, a mobile carrier blocking something, a half-finished Caddy/DNS cutover for the new VPS), that user can receive the push notification itself but never see real content, or see it inconsistently. Temporarily switching to "Send full message contents" removes this variable while narrowing down the cause.

5. A changed FQDN specifically — DNS propagation and cert readiness

If the migration also moved the server to a new domain (not just a new box under the same domain), that alone can produce exactly this some-users-fine/some-users-not pattern for a few days after cutover, on top of section 2 and 4 above:

• DNS propagation lags per device. Different users' resolvers (ISP DNS, carrier DNS on cellular, cached records with old TTLs) pick up the new FQDN's IP at different times. A user on a fast public resolver sees it immediately; someone on carrier DNS with a stale cache might not resolve it correctly for hours. Combined with section 4 (anything but "full message contents" requires a content-fetch call back to SiteURL), a device with a stale answer for the new FQDN fails that step while others succeed.
• TLS certificate not fully issued/propagated yet for the new FQDN (Caddy/Let's Encrypt) causes the same content-fetch failure via cert validation instead of DNS.
• Confirm MM_SERVICESETTINGS_SITEURL in .env is actually the new FQDN — a value left over from before the domain change points every client's content-fetch at the wrong place, consistently, not just intermittently.
• Users who never explicitly added the new server URL in the mobile app (a kept redirect from the old domain let them keep working without noticing) are still running on their old, stale device registration — this is section 2's mechanism, just caused directly by the FQDN change rather than by the DB import alone.