Pro@programming.dev to Technology@programming.devEnglish · edit-24 months agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square114linkfedilinkarrow-up1707file-textcross-posted to: [email protected]
arrow-up1707imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@programming.devEnglish · edit-24 months agomessage-square114linkfedilinkfile-textcross-posted to: [email protected]
minus-squareS7rauss@discuss.tchncs.delinkfedilinkEnglisharrow-up4·4 months agoI was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.
I was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.