The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,466 other followers

Overview of Client Libraries · Internet Archive

Posted by jpluimers on 2021/09/14

Besides manual upload at [Archive.is] Upload to Internet Archive, there are also automated ways of uploading content.

One day I need this to archive pages or sites into the WayBack machine: [WayBack] Overview of Client Libraries · Internet Archive (most of which is Python based):

Overview of Client Libraries


The Internet Archive and its community have developed several tools to give developers more control the Archive’s content and services:

`internetarchive` Command Line Tool (Python, CLI)


The internetarchive tool by Jake enables programatic access to archive.org item metadata and bulk upload of content to the Internet Archive.

Download instructions are available at [WayBack] https://github.com/jjjake/internetarchive/

Read documentation at [WayBackhttps://internetarchive.readthedocs.io/en/latest/

`openlibrary-client` Client Library (Python, CLI)


The openlibrary-client is the equivalent of the internetarchive tool for OpenLibrary. It provides developers with programatic access to Book edition and author metadata, as well as the ability to create new works.

Download instructions and documentation are available at [WayBackhttps://github.com/internetarchive/openlibrary-client

`warc` Client Library (Python)


WARC (Web ARChive) is a file format for storing web crawls (learn more at: [WayBackhttp://bibnum.bnf.fr/WARC)

This warc library makes it very easy to work with WARC files.

Download instructions and documentation are available at [WayBackhttps://github.com/internetarchive/warc

Related:

Via: [WayBack] Uploading to the Internet Archive – Archiveteam

Uploading to archive.org

[WayBack] Upload any content you manage to preserve! Registering takes a minute.

Tools

The are three main methods to upload items to Internet Archive programmatically:

Don’t use FTP upload, try to keep your items below 400 GiB size, add plenty of metadata.

Wayback machine save page now

Many scripts have been written to use the live proxy:

Torrent upload

Torrent upload, useful if you need resume (for huge files or because your bandwidth is insufficient for upload in one go):

  • Just create the item, make a torrent with your files in it, name it like the item, and upload it to the item.
  • archive.org will connect to you and other peers via a Transmission daemon and keep downloading all the contents till done;
  • For a command line tool you can use e.g. mktorrent or buildtorrent, example: mktorrent -a udp://tracker.publicbt.com:80/announce -a udp://tracker.openbittorrent.com:80 -a udp://tracker.ccc.de:80 -a udp://tracker.istole.it:80 -a http://tracker.publicbt.com:80/announce -a http://tracker.openbittorrent.com/announce "DIRECTORYTOUPLOAD" ;
  • You can then seed the torrent with one of the many graphical clients (e.g. Transmission) or on the command line (Transmission and rtorrent are the most popular; btdownloadcurses reportedly doesn’t work with udp trackers.)
  • archive.org will stop the download if the torrent stalls for some time and add a file to your item called “resume.tar.gz”, which contains whatever data was downloaded. To resume, delete the empty file called IDENTIFIER_torrent.txt; then, resume the download by re-deriving the item (you can do that from the Item Manager.) Make sure that there are online peers with the data before re-deriving and don’t delete the torrent file from the item.

Formats

Formats: anything, but:

  • Sites should be uploaded in [WayBackWARC format;
  • Audio, video, [WayBackbooks and other prints are supported from [WayBack] a number of formats;
  • For .tar and .zip files archive.org offers an online browser to search and download the specific files one needs, so you probably want to use either unless you have good reasons (e.g. if 7z or bzip2 reduce the size tenfold).

This [WayBackunofficial documentation page explains various of the special files found in every item.

Upload speed

Quite often, it’s hard to use your full bandwidth to/from the Internet Archive, which can be frustrating. The bottleneck may be temporary (check the current [WayBacknetwork speed and [WayBacks3 errors) but also persistent, especially if your network is far (e.g. transatlantic connections).

If your connection is slow or unreliable and you’re trying to upload a lot of data, it’s strongly recommended to use the bittorrent method (see above).

Some users with Gigabit upstream links or more, on common GNU/Linux operating systems (such as [WayBackAlpine), have had some success in increasing their upload speed by using more memory on [WayBackTCP congestion control and telling the kernel to live with higher latency and lower responsiveness, as in this example:

# sysctl net.core.rmem_default=8388608 net.core.rmem_max=8388608 net.ipv4.tcp_rmem="32768 131072 8388608" net.core.wmem_default=8388608 net.core.wmem_max=8388608 net.ipv4.tcp_wmem="32768 131072 8388608" net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr
# sysctl kernel.sched_min_granularity_ns=1000000000 kernel.sched_latency_ns=1000000000 kernel.sched_migration_cost_ns=2147483647 kernel.sched_rr_timeslice_ms=100 kernel.sched_wakeup_granularity_ns=1000000000

–jeroen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: