Pro@programming.dev to Technology@programming.devEnglish · edit-24 months agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square114linkfedilinkarrow-up1707file-textcross-posted to: [email protected]
arrow-up1707imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@programming.devEnglish · edit-24 months agomessage-square114linkfedilinkfile-textcross-posted to: [email protected]
minus-squareRedjard@lemmy.dbzer0.comlinkfedilinkEnglisharrow-up9·4 months agoCrawl-delay is just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers …
minus-squarewho@feddit.orglinkfedilinkEnglisharrow-up2·edit-24 months ago Crawl-delay It’s a nonstandard extension without consistent semantics or wide support, but I suppose it’s good to know about anyway. Thanks for mentioning it.
Crawl-delayis just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers …It’s a nonstandard extension without consistent semantics or wide support, but I suppose it’s good to know about anyway. Thanks for mentioning it.