Rate limiting on mirrors.dotsrc.org/kiwix


EDIT: Just to clarify, this rate limiting was only set on /kiwix, and it was only active for 2-3 days. Turned out this was not very effective after all. Anyone seeing outages or slowing down is not due to the rate-limiting, but instead the immense traffic we are seeing. We want to increase capacity with another mirror host, but harddrives are expensive right now :-(. The text below is therefore out of date, so please do not send us emails about rate-limiting.

We recently started seeing a very large amount of traffic to the kiwix mirrors. This seems to be duplicate requests from single ip’s many times a second, or at fast the connection allows. Blocking these does not work as they move onto other ip’s.

This has largely overpowered our mirror server and internet connection. This traffic might be legitimate, but user agents are not very descriptive. To make sure our other services remain usable, I have decided to put a rather large rate limit on /kiwix.

If this in any way hinders a legitimate usage of /kiwix, then please please reach out and help me understand what is going on. I will keep this rate limiting for a couple of days, and then check whether the traffic persists.

See also