Dropsitenews published a list of websites Facebook uses to train its AI on. Multiple Lemmy instances are on the list as noticed by user BlueAEther
Hexbear is on there too. Also Facebook is very interested in people uploading their massive dongs to lemmynsfw.
Full article here.
Link to the full leaked list download: Meta leaked list pdf
My instance gets pillaged once a day for 20 minutes by what I think is a scraper for an LLM.
The scraper grabs every post and profile page and the load on the server triggers alerts but the site stays usable.
I haven’t been able to put a stop to it as the requests come from 1500+ IP addresses, with different user agents.
Anubis?
I have no idea. I spot check 20 or so IP addresses and they are all from different AS networks. Truly diverse botnet. Feel powerless.
they were suggesting a solution, this proof-of-work web firewall: https://github.com/TecharoHQ/anubis
Ah thank you, will check it out
I think Anubis would be able to prevent that. Sopuli uses it
Thanks I’ll have a look
Yeah, they’re scraping alright and it’s all purposefully done in such a way that you can’t stop it, you can’t control it.
AI companies are criminal as far as I am concerned
Run your access logs through something that will report the ASN for the client IPs. Goaccess would be my recommendation. It will require access to a GeoIP database which you can get from Maxmind by signing up for a free API key, or download them directly from P3TERX/GeoLite.mmdb on Github. We have identified a number of bot networks this way. Happy to help further if you’d like a hand 👍