I want to check out how to do POST requests using bookmarklets in order to save URLs to the WayBack machine
Posted by jpluimers on 2023/04/27
I want to check out how to do POST requests using bookmarklets in order to save URLs to the Wayback machine.
The reason is that every few months or so, saving a page the normal way through a something like https://web.archive.org/save/URL fails for one reason or the other, but going to https://web.archive.org/save, then entering URL, and pressing “SAVE PAGE” button works fine:
- [Archive] Jeroen Wiert Pluimers on Twitter: “Does anyone why the @waybackmachine has a lot of
job failedand404errors over the last few days? … and … just returned a 404; most of my archivals the last few days failed or had to be retried at least half a dozen times to succeed. …” / Twitter - [Archive] Jeroen Wiert Pluimers on Twitter: “Workaround – still slow, but at least succeeds sometimes: … then enter the URL to be archived.” / Twitter
- [Archive] Jeroen Wiert Pluimers on Twitter: “Workaround now consistently fails for https://t.co/ruKnfWlfJ0 Is here a @waybackmachine status page somewhere?” / Twitter
The the failing way above is using a GET request, the succeeding workaround will open https://web.archive.org/save/URL using the below POST request (where I omitted some HTTP cookies and HTTP header fields for brevity).
- POST request using
PowerShell:
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession $session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" Invoke-WebRequest -UseBasicParsing -Uri "https://web.archive.org/save/URL" ` -Method "POST" ` -WebSession $session ` -Headers @{ "method"="POST" "origin"="https://web.archive.org" "referer"="https://web.archive.org/save" } ` -ContentType "application/x-www-form-urlencoded" ` -Body "url=URL&capture_outlinks=on&capture_all=on&capture_screenshot=on" - POST request using cURL on
bash:
curl 'https://web.archive.org/save/URL' \ -H 'origin: https://web.archive.org' \ -H 'content-type: application/x-www-form-urlencoded' \ -H 'referer: https://web.archive.org/save' \ --data-raw 'url=URL&capture_outlinks=on&capture_all=on&capture_screenshot=on' \ --compressed - POST request using the fetch API in JavaScript:
fetch("https://web.archive.org/save/URL", { "headers": { "content-type": "application/x-www-form-urlencoded", }, "referrer": "https://web.archive.org/save", "body": "url=URL&capture_outlinks=on&capture_all=on&capture_screenshot=on", "method": "POST", "mode": "cors" });
BTW: Yes, I know that URL is not a valid URL, so it will return a page with “http://url/ URL syntax is not valid.“.
All links below via [Wayback/Archive] bookmarklet post request – Google Search:
- [Wayback/Archive] GlenCoakley/createFormSubmittingBookmarklets: Scans through the current web page finding all forms. It will open a new window that listing all of them along with the source for a bookmarklet that will submit that form with its current data.
- [Wayback/Archive] bookmarklet maker by tac (takes a piece of JavaScript, removes spaces, generates bookmarklet anchor which you can drag to your bookmarks)
- Source saved at [Archive] https://gist.github.com/jpluimers/893699a4dd287c1c5b76f50cd2b7f7e5
- Though I really dislike minified code, for a bookmarklet it is often the easiest way to keep it short, so:
- [Wayback/Archive] How to save a bookmark in Firefox with POST data? – Super User
- [Wayback/Archive] google chrome – Storing a HTTP POST request in a bookmark or something similar as you would with a GET request – Super User
- [Wayback/Archive] Javascript bookmarklet with POST command – Stack Overflow
- [Wayback/Archive] How to Add a Cross-Browser Bookmark Button to Your Website | The Web Flash
- [Wayback/Archive] Javascript to add a bookmark – Cross Browser
I tried to put createFormSubmittingBookmarklets/createFormSubmitBookmarklets.js in a bookmarklet using both userjs.up.seesaa.net/js/bookmarklet.html and skalman.github.io/UglifyJS-online. That failed: somehow this code does not want to run as bookmarklet.
Running it from the console is fine though, and gave me this basic bookmarklet template:
javascript:function sf(ur,ty,fd){function me(tg,pr){var el=document.createElement(tg);for(const[nm,vl]of Object.entries(pr)){el.setAttribute(nm,vl);}return el}const fm=me("form",{action:ur,method:ty,style:"display:hidden;"});for(const[nm,vl]of Object.entries(fd)){fm.appendChild(me("input",{name:nm, value:vl}))}document.body.appendChild(fm);fm.submit()}sf("https://web.archive.org/save","post",{"url":"URL","capture_outlinks":"on","capture_all":"on","capture_screenshot":"on","wm-save-mywebarchive":"on","email_result":"on","":"SAVE PAGE"});
There bold URL there is the URL to be saved. I need to test this, then rework it to become parameterised.
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply