Production never sleeps.

But your on-call doesn't have to.

RemOps

RemOps AI for prod
Datadogmetrics · traces
Grafanadashboards · alerts
PagerDutySEV routing
Kubernetesin-cluster agent
GitHubdeploys · diffs
Postgrespool · locks
app.remopsai.com/dashboard
Live
acme / Dashboard
1 openIntake ready

Production Signals

Acme Payments · 3 incidents · live

LiveAuto-refresh 10s
Alerts3
Open1
Investigating1
Awaiting0
Resolved2
AllOpenAwaitingResolved
AlertSeverityStateUpdated
payments.api.p99 > 400msPagerDuty · checkout-api critical investigating just now
KubePodRestartingAlertmanager · payments warning resolved 2h ago
RDS CPU > 80%Datadog · payments-db warning resolved Yesterday
↗
Webhook receivedPagerDuty · SEV-1 · INC-2847
app.remopsai.com/incidents/inc-2847
investigatingINC-2847started 12s ago

payments.api.p99 > 400ms

PagerDuty · checkout-api · prod-us-east-1

Plan5 steps
Investigation plan
✓
Correlate with recent deploysGitHub · last 2h · 1 candidate
Validate latency spike in metricsDatadog · payments.api.p99 · running…
3
Query error logsDatadog Logs · connection errors
4
Check Postgres pool saturationRDS · connections · wait events
5
Identify root cause with evidenceCross-correlate hypotheses
Datadog · Metrics live
payments.api.p99last 30m · prod
14:02 deploy
Spike +340% · 1.84s p99 · started 62s after deploy
Datadog · Logs query
service:payments-api status:error
"connection pool exhausted"
ERROR pool exhausted — waiting for connection
ERROR pool exhausted — waiting for connection
WARN active=10 max=10 · queue depth 48
ERROR could not obtain connection from pool
47 matches · rewritten query ×3
GitHub · Deploy diff
PR #4821 · merged 14:01 UTC · payments-db-pool
chore: tighten db pool for cost
payments/values.yaml
- default_pool_size: 50
+ default_pool_size: 10
Latent config change · passed code review
app.remopsai.com/incidents/inc-2847
Root cause · 94% confidence

Connection pool exhaustion after deploy

PR #4821 cut default_pool_size from 50 → 10. Under checkout load, Postgres wait events spiked and p99 collapsed.

✓
Timingp99 spike 62s after deploy
✓
Logs47× “pool exhausted”
✓
Diffpool_size 50 → 10
✓
RDSwait/IO · connection saturation
payments/values.yaml · rev 13
- default_pool_size: 50
+ default_pool_size: 10
SK
app.remopsai.com/incidents/inc-2847
awaiting approvalINC-2847expires in 14:32

Recommended fix

ROLLBACK · configmap

Restore payments-db-pool to rev 12

Set default_pool_size: 50 and restart payments-api

Risk · low · blast radius 1 service
✓ Dry-run passed against cluster
✓ Policy checks green
✓ No write credentials on SaaS

Decision

Incident received14:02:18
Investigation ready14:02:41
Awaiting approvalnow
SaaS RemOps Cloud read-only investigate
  • metrics · logs · code
  • no write keys
Your VPC RemOps Agent executing approved fix
  • kubectl · in-cluster
  • policy gated
agent@payments-prod · session 8f2a

$ kubectl -n payments patch cm payments-db-pool --patch '…pool_size: 50'

$ kubectl -n payments rollout restart deploy/payments-api

waiting for rollout…

✓ rollout complete · p99 1.84s → 210ms

✓ INC-2847 resolved

Postmortem · draft

INC-2847 — payments p99 regression

Summary Pool size reduced in PR #4821 caused connection exhaustion under load.

Impact Checkout latency 1.84s for 3m 12s · ~4.2k affected requests.

Fix Rollback approved · MTTR 54s from alert.

Runbook · guardrail

DB pool sizing policy

Block deploys that reduce default_pool_size below 40 without load test evidence.

Policy PR opened · #4830
Ask RemOps · shared session
alex Why did pool size change?
RemOps PR #4821 (“tighten db pool for cost”) merged at 14:01 by @dev. Diff cut max connections 50→10. Linked to INC-2847.
sara Can we prevent this class?
RemOps Yes — drafted runbook + policy PR #4830 for your review.

Keep production moving.

remopsai.com

Try RemOps