The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers
«
»

Do not block User-agent ia_archiver because you think it is Alexa, as it makes it harder for the Internet Archive WayBack Machine to stay up-to-date

Posted by jpluimers on 2020/12/18

I observed some sites block the User-agent: ia_archiver in their robots.txt, thinking the would just block Alexa Internet: [Archive.is1/Archive.is2] Crawlers – Alexa Support (which oddly refuses the details to be archived in the Internet Archive).

It will indeed block, but also makes it harder for the Internet Archive WayBack Machine to stay up to date

This sounds complicated, and it is. The Internet Archive originally wrote ia_archiver, but it was ran by Alexa Internet, and still feeds lots of the Internet Archive. By now, the Internet Archive also uses their own both that uses User-agent: archive.org_bot and is called Heritrix.

What makes it complicated, is that most (maybe by now that is down “a lot of”) content is still donated to the internet archive by Alexa Internet.

So if you block ia_archiver, then the Internet Archive still misses data.

Further reading:

and:

Example: [Archive.is] https://assarbad.net/robots.txt

–jeroen

This entry was posted on 2020/12/18 at 06:00 and is filed under Power User. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

«
»