My web server is tipidee (skarnet.org/software/tipidee/) and it runs under a super-server: one process per connection. I can block IP ranges, or individual IPs, very efficiently, with an interface that's well-suited to automation, but I had never tested its full power yet. So I wondered: is it really impossible to block all the LLM crawlers' IP?

I wrote a program to analyze tipidee's logs over a window of time, to know exactly what I wanted to protect against. And sure enough, I noticed a pattern. Access to the static pages didn't matter at all, they were few and far between. What was really hammering my system was requests to cgit - the web interface to the git repositories of my projects. And there was also a clear pattern to how most requests were made to cgit as well. They were from brand new IPs hitting a seemingly random point under cgit, with an explicit branch, or an explicit commit. 3/8

So I wrote another program, meant to be used as a cgit wrapper, that would blacklist yet-unknown clients (and send them a 403 error) if their request matched a given regexp. But first-time clients appearing to have a normal access pattern would be whitelisted.

Blacklisting an IP is adding one symlink in a directory used as rulesdir for s6-tcpserver-access (skarnet.org/software/s6-networ…). I set the thing up on a large ext4 filesystem, and let it run. I fully expected to run out of inodes in a week or so, with basically no changes to the server's overload. I just wanted to try and see what happened, and then think about my next move. But surprisingly, the result was way above and beyond my expectations. 4/8

After a week, the number of blocked IPs had stabilized around 2.99M for IPv4, and 430k for IPv6. One more week later, today, these numbers are still going up, but very slowly. Creeping towards 3M for IPv4 (might reach it in December) and 450k for IPv6. And the load average of skarnet.org has dropped back to under 0.10.

There was one reported false positive, which was easy to unblock. And probably some false negatives; if there are too many of them I can delete the whitelist at any time and let it rebuild. I don't think it will be necessary, given that the crawler-induced load is pretty negligible now. 5/8

Is it reproducible? I don't know. Most sites are probably hammered a lot more, and from a lot more IPs, than mine. But I thought it would be an interesting data point that blocking 3.5M IPs cut the LLM crawler spam to basically zero.

And I have to admit, it feels really nice to kinda be able to out-bruteforce these assholes, with very few resources, and - as always - extremely little code. What I build is nothing if not efficient: there isn't a single system call that's unaccounted for in the whole stack. 6/8

Now an interesting question is: I have 3 million symlinks in one directory. How is that going?

It's actually going very well, and it's a testament to the robustness of ext4. The ip4 directory weighs 113 MB, for 3M symlinks each 16 bytes long. Accessing any given entry is still instant. Listing the whole directory (and a fortiori statting every file in it) is obviously slow, but that never happens unless I want to do something stupid like du the whole thing.

If I wanted, I could run s6-accessrules-cdb-from-fs (skarnet.org/software/s6/s6-acc…) on the whole thing, and turn it into a single cdb file weighing a fraction of the directory's size, and even more efficient. But that would make the database static, which would defeat the purpose. The point is that it's updated on every access from a new IP.

It has been a theory of mine for a long time that the filesystem makes for an excellent key-value store and people shouldn't hesitate to use it as such. This is the first time I can put that theory to the test, and really stress a filesystem, and it's passing with flying colors.

(I used ext4 because it's what I'm used to, and I'm used to it just working; I have no other ideology. Feel free to debate the virtues of filesystem foo vs. filesystem bar, but please don't include me in these debates - they have no value or appeal to me.) 7/8

@GhostOnTheHalfShell

I meant to add pedophiles and pedophile protectors, too. Time for a few edits.

@GhostOnTheHalfShell

Fixed it! 😉👍