Yes Man would never refuse to help you commit tax fraud.

Crawlers don’t have to follow conventions or specifications. If one has a setTimeout implementation that doesn’t wait the specified amount of time and simply executes the callback immediately, it defeats the system. Proof-of-work is meant to ensure that it’s impossible to get around the time factor because of computational inefficiency.

Anubis is an emergency solution against the flood of scrapers deployed by massive AI companies. Everybody wishes it wasn’t necessary.

Correct. Anubis’ goal is to decrease the web traffic that hits the server, not to prevent scraping altogether. I should also clarify that this works because it costs the scrapers time with each request, not because it bogs down the CPU.

It assumes that we sites are under constant ddos

It is literally happening. https://www.youtube.com/watch?v=cQk2mPcAAWo https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/

It assumes that anubis is effective against ddos attacks

It’s being used by some little-known entities like the LKML, FreeBSD, SourceHut, UNESCO, and the fucking UN, so I’m assuming it probably works well enough. https://policytoolbox.iiep.unesco.org/ https://xeiaso.net/notes/2025/anubis-works/

anti-AI wave

Oh, you’re one of those people. Enough said. (edit) By the way, Anubis’ author seems to be a big fan of machine learning and AI.

(edit 2 just because I’m extra cross that you don’t seem to understand this part)

Do you know what a web crawler does when a process finishes grabbing the response from the web server? Do you think it takes a little break to conserve energy and let all the other remaining processes do their thing? No, it spawns another bloody process to scrape the next hyperlink.

• A web server that can’t discriminate between a request made by a human and one made by a machine has to handle all requests. It may not be an issue for large companies like Amazon or Microsoft, but small websites will suffer timeouts and outages.
• Without a locally hosted solution like Anubis, small websites would have to move behind a large centralized service like Cloudflare.
• Otherwise they might not be able to continue operating and only large corporate-backed services like Twitter and Reddit would survive.

The alternative is having to choose between Reddit and Cloudflare. Does that look “free” and “open” to you?

Anubis is a simple anti-scraper defense that weighs a web client’s soul by giving it a tiny proof-of-work workload (some calculation that doesn’t have an efficient solution, like cryptography) before letting it pass through to the actual website. The workload is insignificant for human users, but very taxing for high-volume scrapers. The calculations are done on the client’s side using Javascript code.

(edit) For clarification: this works because the computation workload takes a relatively long time, not because it bogs down the CPU. Halting each request at the gate for only a few seconds adds up very quickly.

Recently, the FSF published an article that likened Anubis to malware because it’s basically arbitrary code that the user has no choice but to execute:

[…] The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.

Here’s the article, and here’s aussie linux man talking about it.

Hail Anubis-chan.

Searching for “twitter” redirects to the live version of this article: https://archive.ph/sJzRS

Surely this won’t upset people.

No.

The local machine boots using PXE. Clonezilla itself is transferred from a TFTP server as a squashfs and loaded into memory. When that OS boots, it mounts a network share using CIFS that contains the image to be installed. All of the local SATA disks are named sda, sdb, etc. A script determines which SATA disk is the correct one (must be non-rotational, must be a specific size and type), deletes every SCSI device (which includes ATA devices too), then mounts only the chosen disk to make sure it’s named sda.

Clonezilla will not allow an image cloned from a device named sda to be written to a device with a different name – this is why I had to make sure that sda is always the correct SSD.

There was no need to physically disconnect anything. We didn’t actually use any SCSI devices, but Linux (and in turn, the Debian-based Clonezilla) uses the SCSI kernel driver for all ATA devices, so SATA SSDs also appeared as SCSI hosts and could be handled as such. If I had to manually unplug and reconnect hundreds of physical cables, I’d send my resignation directly to my boss’ printer.

I presume you have had to run on RAM, considering you removed all drives

Yes. Mass deployment using Clonezilla in an extremely heterogenous environment. I had to make sure the OS got installed on the correct SSD, and that it was always named sda, otherwise Clonezilla would shit itself. The solution is a hack held together by spit and my own stubbornness, but it works.

Only if you’re working with SCSI hardware. On Linux, SATA (and probably PATA) devices use the same kernel driver as SCSI, and appear on the system as SCSI hosts. You can find them in /sys/class/scsi_disk or by running lsblk -o NAME,HCTL.

Broke: /dev/sd*
Woke: /dev/disk/by-id/*
Bespoke: finding the correct device’s SCSI host, detaching everything, then reattaching only the one host to make sure it’s always /dev/sda. (edit) In software. SATA devices also show up as SCSI hosts because they use the same kernel driver.

I’ve had to use all three methods. Fucking around in /sys feels like I’m wielding a power stolen from the gods.

Gparted. I’m pretty sure it’s bundled with Mint. Identify which partition it is, then simply delete it. Then you can create a new partition on the newly unallocated area or extend an existing partition onto it.

You might have to edit the bootloader’s boot entries and remove the Windows boot manager.

“Fuck around and find out”?

I can reliably trigger rate limiting by rapidly switching between files. Sometimes it’s after looking at 10 files, sometimes up to 30. At any rate, it’s not something a regular user would do (and at that point, why not just clone the repo?), and the rate limiting clears in less than a minute. It’s really no worse than having to deal with Anubis-chan.

People just love to freak out, I guess.

Calm the fuck down. This wasn’t a “Russian lathe accident” situation. We were trained professionals, and never left the machines unattended in an unsafe state. There were no injuries and only that one close call (which IIRC was traced back to a faulty e-stop button).

We never fell victim to complacency and I am quite proud of that.

Sometimes for maintenance, sometimes because manual intervention was necessary. The machines where we did this were built in the 90s and have been in near constant operation. Moving parts are worn out and the tolerances are gone. Replacement parts are difficult to find and expensive to manufacture, so if something more complex than a ball bearing or axle got out of alignment, we had to pound it back into place (sometimes literally).

I personally never bypassed the interlock, I wasn’t paid enough to take on that responsibility. I would just file a downtime notice and call the on-site mechanic when needed. I didn’t give a shit about reduced output.

Tagging @Remorhaz@lemmy.world