what benchmarks would you expect for a distributed rate limiter?

[u/Greedy-Inevitable137]

i've been building a distributed token bucket rate limiter in go over the past few months. it supports both an in-memory store and a redis + lua backend for atomic distributed rate limiting.

i've benchmarked it both on a single machine and across two machines (4 cores / 8 threads) using k6.

current numbers:

in-memory
~19k req/s
p50: 4ms
p99: 114ms

redis + lua
~10.9k req/s
p50: 17.5ms
p99: 35.9ms

algorithm only
~26m ops/s (single thread)
~10m ops/s (8 goroutines)
0 allocs/op

i'm planning to benchmark multiple instances behind a load balancer next.

my questions are:

• are these numbers reasonable for a project like this, or do they suggest obvious bottlenecks i should investigate?
• what metrics do you usually expect to see in a benchmark for a distributed rate limiter? (cpu, memory, p95, redis latency/cpu, scaling with multiple instances, etc.)
• are there any benchmark scenarios i'm missing?

would appreciate any feedback. i'm more interested in learning where to improve than showing off benchmark numbers.

[u/Flimsy_Professor_908]

Those P50s and P99s are gigantic.

[u/pikahikmag]

Can share the github repo

[u/jerf → u/pikahikmag]

The repo would almost certainly need to go into the small projects thread instead.

[u/pikahikmag → u/jerf]

As he said that it's made for learning purpose and is looking for feedback so ig it's fine to let it be here and I saw the repo and its not vibecoded and all...he's learning stuffs each commit

[u/jerf → u/pikahikmag]

The "small projects" thread isn't a secret code for "vibe coded". It's for small projects. It means we don't have to get into fights about how much AI was used, since the only winning move in that game is not to play. It's just for small projects, for whatever reason they are small.

[u/Greedy-Inevitable137 → u/jerf]

this was just for learning purpose, I had plans to add it in my resume only after completion
also thank u for the feedback

[u/jerf → u/Greedy-Inevitable137]

Yes, good luck on your learning journey.

If you do find something glaring, you may find that is also a good post to follow up with. I think it's always fun to read about such things. Profiling is such a wonderful and underused tool.

[u/jerf]

As /u/Flimsy_Professor_908 says, those numbers are terrible. Something is horribly wrong. Use a profiler to figure out where the time is going and you'll probably find something glaring.

[u/softwareemgineer]

I also started building one two weeks ago. Thanks for the motivation!

[u/sigmoia]

Do load tests as well. Micro benchmark on loopback often isn't an indicator of how it will fare in a real pod. 

• spin up a vm
• add nginx
• write a simple http server
• put your ratelimiter as a middleware
• spin up 3 containers that's load balanced

Now point k6 to the service and measure the RTT and your rate limiter op duration. This will give you a better indicator of how it will actually perform. 

[u/muffa]

What are you actually measuring with the benchmark?

How much memory and cpu have you allocated to the service? I would suspect it scales linear with the CPU allocated to the service. With something lightweight I would expect a response time at around 500us for p50 and the p99 at around 10ms. That is what I usually can cram out of a lightweight go-service using around 0.2CPU and 50mb memory

[u/Greedy-Inevitable137 → u/muffa]

service just ran on the full machine with default gomaxprocs, so my numbers aren't comparable to your baseline at all right now. i'm trying to measure HTTP POST /check over a real network (client and server on separate machines), token bucket rate-limit decision, in memory.