Case Study: How We Cut MySQL CPU Usage from 87% to 40% with ProxySQL and Read Replicas

Every two to three days, our production MySQL database would spike to 85%+ CPU usage, degrading every service that depended on it. This is the story of how we diagnosed the environment, evaluated four competing solutions, and shipped a master-replica architecture with ProxySQL that cut peak CPU usage by more than half—without downtime and with a rollback plan ready at every step.

---

The Situation

The system comprises 15–20 microservices managed on Kubernetes. All of these services shared a single MySQL 5.7 instance storing more than 800 GB of data and serving 100+ requests per second.

And every 2 to 3 days, that single instance would hit high CPU usage—dragging down every service that depended on it.

A single database behind an entire microservices fleet is a classic scaling bottleneck: there was no way to offload read traffic, no redundancy if the instance failed, and every slow query competed with production writes for the same CPU.

The Results (Up Front)

Because this is a case study, let's lead with the outcome:

CPU usage across the migration window (June 6–7). The pre-migration spike reached ~87%; after cutover, peak usage stayed around 40%.

Since the migration, we have not faced a single high CPU incident on the master database.

Here's how we got there.

---

Step 1: Understand the Environment Before Designing Anything

The first step in resolving any production problem is an in-depth understanding of its environment. Before drafting a solution, we worked through a questionnaire of critical questions:

1. Which cloud provider or on-premise solution do we use—AWS, GCP, or something else?
2. What is the overall structure of our infrastructure?
3. Do we follow a one-service-one-database approach, or do multiple services share databases?
4. Are we running a single MySQL instance or multiple?
5. How frequently do we encounter infrastructure problems?
6. Is the database self-hosted or a managed cloud service?
7. What version of MySQL are we on?
8. How many services and workers connect to the MySQL instance?
9. How large is the dataset, and what is the VM configuration?
10. What is the query rate per second/minute?
11. When are peak hours, and what is throughput during them?
12. How critical is this database to business uptime?
13. Are all queries slow, or is a particular query underperforming?
14. Do we cache data or API responses anywhere to reduce database load?
15. What is the backup strategy?
16. How do we monitor database load and infrastructure?
17. What happens today when the database slows down (not downtime—degraded performance)?
18. Does compliance factor into disaster recovery objectives, or is it purely about uptime?

With these answers in hand, we could set concrete expectations for the ideal solution and compare candidates against them—rather than reaching for whatever tool was trendiest.

Step 2: Compare the Candidate Solutions

We shortlisted four solutions: InnoDB Cluster, Percona XtraDB Cluster, Google Cloud SQL, and a master-replica architecture fronted by ProxySQL. We compared them across replication support, monitoring, hosting, disaster recovery, ease of setup, community support, and pricing:

From this matrix, a few inferences drove the decision:

• Write latency was a hard constraint. Percona XtraDB's synchronous commit across all nodes could slow down writes, and we didn't have enough data to say conclusively that the system could absorb that. Percona was out.
• Cost mattered. Cloud SQL could be roughly twice as expensive as the self-hosted alternatives. Cloud SQL was out.
• Simplicity wins. A single master with multiple replicas has no master election, no quorum, and none of the other distributed-database complexities. It was the simplest architecture that solved the actual problem: read traffic overwhelming a single instance.

We chose the master-replica architecture with ProxySQL. It gave us monitoring support, straightforward setup, cost-effectiveness, and—critically—a way to distribute read load without touching write performance. High availability came along for free: with replicas in place, the system no longer had a single point of database failure.

Step 3: The Solution Architecture

The target architecture: application servers connect through a load balancer to two ProxySQL instances (active-passive) inside the VPN. ProxySQL routes writes to the primary database and distributes reads across the replicas.

The key components:

• Primary (master) database — continues to serve all writes.
• Two read replicas — asynchronously replicate from the primary and absorb read traffic.
• Two ProxySQL instances — sit between the applications and the databases, routing queries by type. Two instances in active-passive mode mean the proxy layer itself is not a single point of failure.
• Load balancer — fronts the ProxySQL pair, so applications keep a single connection endpoint and never need to know which proxy—or which database—serves them.

Step 4: The Migration Plan

We executed the migration in five deliberate steps:

1. Set up replicas for the master database. This immediately gave us high availability and data redundancy. If the master fails, a replica can be promoted with minimal impact on availability—the database remains accessible even during unexpected events.

2. Configure ProxySQL as the database proxy. ProxySQL distributes traffic and load-balances between the master and replica instances, transparently to the applications.

3. Run two ProxySQL instances in active-passive mode. A proxy that routes all database traffic is itself a potential single point of failure. With an active-passive pair, the backup instance seamlessly takes over proxying if the primary proxy fails—no single point of failure anywhere in the data path.

4. Implement monitoring with Grafana and Prometheus. We tracked CPU usage, query performance, and replication lag in real time. This let us identify bottlenecks, make data-driven decisions during the rollout, and catch issues before they impacted the system.

5. Add ProxySQL query rules to redirect slow reads to replicas—one by one. Rather than flipping all read traffic at once, we shifted the heaviest offending reads incrementally. Each rule offloaded work from the master, and monitoring confirmed the impact before we moved the next one.

Step 5: The Rollback Strategy

A rollback strategy is a crucial component of any migration—it's the safety net that minimizes potential downtime or data loss if anything goes wrong mid-flight. Ours was built on five pillars:

1. Checkpoints — Define specific points in the migration where the system's state is saved, so there is always a known stable state to restore to.
2. Backups — Take validated backups of critical data and configuration before any change, and keep them easily accessible.
3. Documentation — Write down the rollback steps in detail, accessible to every team member who might need to execute them.
4. Testing — Rehearse the rollback in a controlled environment to verify it works and to surface hidden dependencies.
5. Communication — Make sure every stakeholder—implementers, management, support—knows the plan and their role if a rollback is triggered.

We never needed it. But knowing it existed let us migrate a production database serving 100+ RPS with confidence rather than crossed fingers.

---

Key Takeaways

• Diagnose the environment before picking a tool. The 18-question audit shaped every decision that followed—including ruling out solutions that looked great on paper.
• The simplest architecture that solves the problem usually wins. We deliberately avoided multi-master clusters, quorum logic, and managed-service lock-in because read splitting was all the problem actually required.
• Kill single points of failure at every layer. Replicas removed the database SPOF; the active-passive ProxySQL pair removed the proxy SPOF the replicas introduced.
• Migrate incrementally, measure continuously. Redirecting slow reads one query rule at a time, with Grafana/Prometheus watching, meant every change was verifiable and reversible.
• Plan the rollback you hope to never use. It's what turns a risky migration into a controlled one.