How We Cut a Client’s AWS Bill 56% — By Fixing the Application Before Touching the Infrastructure
W
e cut a client’s AWS bill from ~$1,127/month to ~$498/month — a 56% reduction, with zero unplanned downtime.
The platform: a router-based parental controls and DNS filtering system, handling real-time allow/deny policy decisions for live users. Every request that hits it is a small, latency-sensitive decision — get infrastructure changes wrong here, and real users feel it in milliseconds.
That constraint shaped everything about how we approached this. Most cost-optimization engagements go straight to the obvious move: look at CPU utilization, pick a smaller instance, resize. We didn’t. Here’s why, and what we did instead.
Phase 1 — Reduce the Load Before You Reduce the Capacity
The instinct to downsize first is understandable — smaller instances are the visible, easy win. But resizing before fixing what’s actually consuming resources is how these projects fail: you hit throttling under real load, performance degrades, and you’re forced to roll the change back — often after the client has already been told it’s done.
We started by measuring instead of assuming.
A monitoring loop firing far more than it needed to
The system’s device health-check process was running a task fan-out pattern — spinning up a separate task per device, every cycle — that added up to 13,684 tasks every 5 minutes. We restructured this to use bulk inserts and batched probes instead of per-device fan-out, bringing that down to roughly 490 tasks per cycle — a ~27x reduction in task volume, with the same monitoring coverage.
Load that wasn’t even real traffic
Digging into the device registry, we found ~6,800 staging/inactive devices being polled on the exact same schedule as active, paying customers. These devices were sitting in a “staged” state — not yet live — but nothing in the polling logic distinguished them from production devices. Filtering them out of the polling cycle removed a huge, invisible chunk of unnecessary load in one change.
A monitoring database with no retention policy at all
InfluxDB, used for infrastructure metrics, had been accumulating three years of data with no cleanup — 16GB of it. We set a 35-day retention policy. The database dropped to 614MB, and the running InfluxDB process handed back 4.4GB of RAM it had been holding for data nobody was querying anymore.
Worker concurrency tuned for load that no longer existed
Celery’s autoscale ceiling was set to spin up as many as 36 worker processes under load; we brought that down to 14, matching real concurrency needs post-cleanup. Application server (uwsgi) workers went from 8 to 4. Debug-level logging — which had likely been left on from an earlier troubleshooting session — was turned back down to info.
A forgotten 6.7GB file
An old Redis RDB dump, left over from an incident roughly 11 months earlier, was still sitting on disk — pure waste that had gone unnoticed because it wasn’t in the way of anything.
None of these were infrastructure changes. They were application and configuration fixes. And the effect was substantial before we’d resized a single instance: server CPU dropped from ~35% to ~5%, and database CPU load dropped from a sustained 55–90% down to under 10%.
Phase 2 — Only Now, Resize
With real load actually reduced, rightsizing stopped being a guess and became a straightforward exercise in matching capacity to confirmed usage.
- The application server went from 8 vCPU / 32GB RAM down to 2 vCPU / 8GB — a size that would have been genuinely risky to jump to before Phase 1, and was comfortably safe afterward.
- The database instance was rightsized, and its storage type switched from provisioned IOPS (io1) to gp3. That single change alone eliminated $105/month in IOPS charges — the volume had been provisioned for performance it was using at under 1% capacity — and gp3 delivered roughly 3x the IOPS headroom at no additional cost.
- AWS Global Accelerator was removed entirely, after confirming — not assuming — that it wasn’t providing meaningful value for this traffic pattern. It was replaced with a direct static IP.
- The production database dropped from 116GB to 3.6GB.
The Step Worth Pausing On
That last number deserves its own explanation, because the path to it is the part of this project worth highlighting most.
One internal table — a device ping-check log — had grown to over 85GB and roughly 304 million rows, with no retention policy, adding rows continuously. Our original plan, and the technically “correct” approach for a table that size on a live production database, was a staged migration: build a new table containing only the recent data, verify it, do an atomic rename to swap it in, and drop the old table afterward. Safe, low-lock-time, but still a real piece of work — realistically, multiple night maintenance windows to execute carefully.
Before writing a line of that migration, we asked a simpler question: does anything actually read this historical data?
The answer was no — nothing in the application queried ping-check history beyond a narrow, recent operational window. Once that was confirmed, the entire multi-night migration plan became unnecessary. The table was truncated directly, after archiving the historical data to S3 first as a safeguard. What would have been a multi-session infrastructure project became a five-second operation.
The Pattern
Every step in this project followed the same discipline: verify before you act, snapshot before you change, monitor after every step — and before reaching for the complex, well-engineered solution, check whether it’s actually needed at all.
The database migration plan we didn’t end up needing is, in some ways, the most representative part of this whole engagement. Not because avoiding it was clever, but because it’s a reminder that a lot of “necessary” infrastructure complexity is really just complexity nobody has questioned yet.