Overview of Client Libraries · Internet Archive
Posted by jpluimers on 2021/09/14
Besides manual upload at [Archive.is] Upload to Internet Archive, there are also automated ways of uploading content.
One day I need this to archive pages or sites into the WayBack machine: [WayBack] Overview of Client Libraries · Internet Archive (most of which is Python based):
Overview of Client Libraries
The Internet Archive and its community have developed several tools to give developers more control the Archive’s content and services:
`internetarchive` Command Line Tool (Python, CLI)
The
internetarchivetool by Jake enables programatic access to archive.org item metadata and bulk upload of content to the Internet Archive.Download instructions are available at [WayBack] https://github.com/jjjake/internetarchive/
Read documentation at [WayBack] https://internetarchive.readthedocs.io/en/latest/
`openlibrary-client` Client Library (Python, CLI)
The
openlibrary-clientis the equivalent of theinternetarchivetool for OpenLibrary. It provides developers with programatic access to Book edition and author metadata, as well as the ability to create new works.Download instructions and documentation are available at [WayBack] https://github.com/internetarchive/openlibrary-client
`warc` Client Library (Python)
WARC (Web ARChive) is a file format for storing web crawls (learn more at: [WayBack] http://bibnum.bnf.fr/WARC)
This warc library makes it very easy to work with WARC files.
Download instructions and documentation are available at [WayBack] https://github.com/internetarchive/warc
Related:
- [WayBack] internetarchive · PyPI
- [WayBack] ArchiveBot – Archiveteam
- [WayBack] Wget with WARC output – Archiveteam
- [WayBack] Have a WARC that you would like to upload to the Internet Archive so that it can eventually be included in their Wayback Machine? Here’s how to upload it from the command line. · GitHub
- [WayBack] archive.org/help/abouts3.txt : Internet Archive’s S3 like server API
- [WayBack] Internet Archive S3-Like API Keys
Via: [WayBack] Uploading to the Internet Archive – Archiveteam
Uploading to archive.org
[WayBack] Upload any content you manage to preserve! Registering takes a minute.
Tools
The are three main methods to upload items to Internet Archive programmatically:
- [WayBack] internetarchive Python library is the main tool now, see the extensive [WayBack] https://archive.org/services/docs/api/
- [WayBack] Handy script for mass upload (ias3upload.pl) with automatic error checking and retry
- [WayBack] S3 interface (for direct usage with curl, or indirect with the tool of your choice)
Don’t use FTP upload, try to keep your items below 400 GiB size, add plenty of metadata.
Wayback machine save page now
- For quick one-shot webpage archiving, use the [WayBack] Wayback Machine‘s “Save Page Now” tool.
- See [WayBack] October 2019 update for details including access requests.
- To input a list of URLs, [WayBack] https://archive.org/services/wayback-gsheets/ (avoid trying to send many thousands URLs; there’s [WayBack] Archivebot for that)
- There’s also an email address where to send lists of URLs in the body, useful to submit automatic email digests (could not independently verify its functioning as of September 2019)
Many scripts have been written to use the live proxy:
- JavaScript Bookmarklet and Chrome extension made by @bitsgalore that provide a fast way to submit pages on the Internet Archive. You can get them here: [WayBack] http://www.bitsgalore.org/2014/08/02/How-to-save-a-web-page-to-the-Internet-Archive/
- [WayBack] UserScript: “AutoSave to Internet Archive – Wayback Machine” by user “Flare0n”. Mirrors: [WayBack] Mirror 1[IA•Wcite•.today•MemWeb] [WayBack] Mirror 2[IA•Wcite•.today•MemWeb]. (No longer developed since 2014, but still functional.)
Torrent upload
Torrent upload, useful if you need resume (for huge files or because your bandwidth is insufficient for upload in one go):
- Just create the item, make a torrent with your files in it, name it like the item, and upload it to the item.
- archive.org will connect to you and other peers via a Transmission daemon and keep downloading all the contents till done;
- For a command line tool you can use e.g. mktorrent or buildtorrent, example:
mktorrent -a udp://tracker.publicbt.com:80/announce -a udp://tracker.openbittorrent.com:80 -a udp://tracker.ccc.de:80 -a udp://tracker.istole.it:80 -a http://tracker.publicbt.com:80/announce -a http://tracker.openbittorrent.com/announce "DIRECTORYTOUPLOAD";- You can then seed the torrent with one of the many graphical clients (e.g. Transmission) or on the command line (Transmission and rtorrent are the most popular; btdownloadcurses reportedly doesn’t work with udp trackers.)
- archive.org will stop the download if the torrent stalls for some time and add a file to your item called “resume.tar.gz”, which contains whatever data was downloaded. To resume, delete the empty file called
IDENTIFIER_torrent.txt; then, resume the download by re-deriving the item (you can do that from the Item Manager.) Make sure that there are online peers with the data before re-deriving and don’t delete the torrent file from the item.Formats
Formats: anything, but:
- Sites should be uploaded in [WayBack] WARC format;
- Audio, video, [WayBack] books and other prints are supported from [WayBack] a number of formats;
- For .tar and .zip files archive.org offers an online browser to search and download the specific files one needs, so you probably want to use either unless you have good reasons (e.g. if 7z or bzip2 reduce the size tenfold).
This [WayBack] unofficial documentation page explains various of the special files found in every item.
Upload speed
Quite often, it’s hard to use your full bandwidth to/from the Internet Archive, which can be frustrating. The bottleneck may be temporary (check the current [WayBack] network speed and [WayBack] s3 errors) but also persistent, especially if your network is far (e.g. transatlantic connections).
If your connection is slow or unreliable and you’re trying to upload a lot of data, it’s strongly recommended to use the bittorrent method (see above).
Some users with Gigabit upstream links or more, on common GNU/Linux operating systems (such as [WayBack] Alpine), have had some success in increasing their upload speed by using more memory on [WayBack] TCP congestion control and telling the kernel to live with higher latency and lower responsiveness, as in this example:
# sysctl net.core.rmem_default=8388608 net.core.rmem_max=8388608 net.ipv4.tcp_rmem="32768 131072 8388608" net.core.wmem_default=8388608 net.core.wmem_max=8388608 net.ipv4.tcp_wmem="32768 131072 8388608" net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr # sysctl kernel.sched_min_granularity_ns=1000000000 kernel.sched_latency_ns=1000000000 kernel.sched_migration_cost_ns=2147483647 kernel.sched_rr_timeslice_ms=100 kernel.sched_wakeup_granularity_ns=1000000000
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply