How I Cut Campus Server Latency by 73%
The Problem
Our school had a website. When too many students tried to access it at the same time, like during course registration, the server would slow down. Most requests finished quickly, but some took a very long time.
This is called tail latency. When 99% of requests finish in 10ms but 1% take 500ms, the average looks fine. But those 1% of users had a terrible experience.
My Approach
I built a simulated server environment on a Raspberry Pi to test different load balancing strategies. The idea was to see which strategy reduced tail latency the most.
I tested three strategies:
Round Robin: Each request goes to the next server in order. Simple, but it does not consider how busy each server is.
Least Connections: Each request goes to the server with the fewest active connections. This sounds smart, but it requires checking every server before each decision.
Weighted Round Robin: Servers get requests based on their capacity. Faster servers get more requests.
The R-MLO Mechanism
I designed a Resource-Monitor Load Offloading (R-MLO) mechanism. It uses lightweight containers as offloading nodes and watches the hardware in real time. When traffic stays high for too long, it hands the extra requests to idle backup nodes.
Implementation
I built it with Docker containers and nginx for load balancing. I configured the upstream servers and wrote a shell script to handle the routing decisions.
What I Found
I used Locust, a load testing tool, to pretend a crowd of users was hitting the server at the same time. That let me measure how each strategy really behaved under load.
The results surprised me. Least Connections reduced average latency the most, but it did not help tail latency as much as I expected. The reason was that checking every server took time itself: the overhead of making a smarter decision sometimes outweighed the benefit.
Weighted Round Robin did better for tail latency because it spread the load more evenly without the overhead of checking connection counts.
The biggest insight was that the best strategy depends on the workload. If requests come in bursts, Least Connections works better. If the load is steady, Weighted Round Robin is simpler and often good enough.
The Campus Website
I also built a small campus website to test it for real. Students could check grades, look at class schedules, and download course materials.
It worked: the optimized strategy cut P99 latency by 73%. Even during peak hours, the website stayed responsive.
What I Learned
This project changed how I think about performance. Before, I thought faster hardware was always the answer. Now I know that the way you distribute work matters just as much as how fast each piece runs.
I also learned that there is no single best solution. Every strategy has tradeoffs. You just have to pick the one that fits your situation.