Hardening 55 Microservices for Millions of Citizens - Without a Single Maintenance Window

The operational gaps translated directly into citizen-facing service failures.

An attempt to reduce compute costs by lowering CPU limits on the birth service caused a cold-start cascade during the morning login rush. Birth certificate services went offline. The services could not complete their startup sequence under the reduced CPU limit, and the load balancer drained the target group before any instance became healthy.

Separately, an authorised team member manually invoked a database operations Lambda to test its behaviour in isolation. The function, designed to apply configuration changes across RDS instances, had no dry-run mode and no scope restrictions. A single invocation triggered an immediate, simultaneous restart across all 10 production databases, producing a 22-minute platform-wide outage. A legitimate, well-intentioned action against a tool with no guardrails had the blast radius of a full-scale incident.

Every deployment carried similar risk. Without reliable health checks, pinned container platforms, or Terraform state discipline, each release was as likely to introduce a new failure mode as it was to ship the intended change.

Edstem was engaged as the platform engineering partner to close these gaps. The team, comprising a Senior Solution Architect, a DevOps Engineer, and a Senior Backend Engineer, executed a structured AWS platform engineering programme, validating every change against March busy-hour baselines before promoting to production.

Kubernetes platform hardening:

• All four clusters upgraded from EKS 1.31 to 1.35 using a blue-green node group swap strategy, with old nodes continuing to serve traffic throughout
• A single managed node group swap per version: no maintenance windows, no downtime
• Prod nodes migrated to r7g.4xlarge Graviton (AL2023), resolving disk eviction issues caused by the previous undersized root disk configuration

The Result: a platform capable of absorbing infrastructure upgrades without scheduling maintenance windows or exposing citizens to service interruptions.

Database connection stability:

• Five RDS Proxy instances deployed across the core, property tax, file management, civil registration, finance and HRMS service groups, collapsing 1,800+ app-side JDBC connections into stable, backend-pooled connections
• HRMS peak connections reduced from 1,782 to 283, a reduction of 84%; CPU on the HRMS instance dropped from 16% to 7%
• Migration executed with documented guardrails to avoid known proxy configuration traps that can destabilise connection pools during cutover

Stable connection pools eliminated the risk of database-layer outages cascading into citizen-facing service failures.

Autoscaling and reliability floors:

• Autoscaling configured on all 55 services with CPU-based scaling and per-service minimum floors set to cold-burst capacity
• JVM services floored at 500m CPU to prevent cold-start probe failures
• A SAFE_MIN_FLOORS safety mechanism added to the nightly scale-down job, preventing a failure mode that had previously taken services offline during the peak login period

These changes ensured the platform could absorb the morning citizen login rush without pod failures or cascading service degradation.

ALB and health check standardisation:

• All load balancer target groups migrated to a dedicated liveness endpoint, eliminating the downstream-aggregation timeout problem
• WAF IP allowlists corrected to the NAT gateway address, resolving a misconfiguration that had been silently blocking internal traffic
• Legacy node security group patched onto all new node ENIs after a node-group swap had left new nodes missing the load balancer ingress path

These fixes removed a class of silent misconfigurations that had been intermittently dropping citizen traffic without triggering any visible alerts.

CI/CD discipline:

• Deployment workflows standardised via shared reusable pipelines with enforced architecture and build flags, eliminating silent startup failures caused by architecture mismatches
• Infrastructure state locking and a CI gate implemented to address configuration drift
• Commit convention enforced across all service repos, requiring simultaneous updates to version, changelog, and deployment configuration files

Every deployment across all 55 services became a traceable, repeatable operation, eliminating the category of human error that had previously caused production incidents.

The platform now runs 55 services at ~1,000 rpm each, absorbing Kubernetes upgrades without planned maintenance, and recovering from pod churn without database connection exhaustion, all on a 12-node Graviton cluster with headroom to scale to 50 nodes on demand.

If you're accountable for platform reliability on AWS, this engagement's experience shows that the most damaging failure modes (connection exhaustion, silent misconfigurations, deployment drift) are also the most preventable. A structured engineering programme, validated against real peak-load data, can improve platform stability without disrupting live services.