The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers
«
»

linux – Get final URL after curl is redirected – Stack Overflow (plus some Twitter scraping tricks)

Posted by jpluimers on 2025/08/06

Sometimes I need [Wayback/Archive] Redirect Checker | Check your Statuscode 301 vs 302 on the command-line, so cURL to the rescue: [Wayback/Archive] linux – Get final URL after curl is redirected – Stack Overflow. The relevant portions of answers and comments further below.

TL;DR:

Since I prefer verbose command-line arguments (you can find them at the [Wayback/Archive] curl – How To Use on-line man page) especially in scripts this HTTP GET request is what works with Twitter:

% curl --location --silent --output /dev/null --write-out "%{url_effective}\n" https://twitter.com/anyuser/status/20
https://x.com/anyuser/status/20

This failed (twitter dislikes HTTP HEAD requests):

% curl --head --location --silent --output /dev/null --write-out "%{url_effective}\n" https://twitter.com/anyuser/status/20
https://twitter.com/anyuser/status/20

Notes

Given so many of my scripts now run on zsh, I added the new-line because of command line – Why does a cURL request return a percent sign (%) with every request in ZSH? – Stack Overflow. You can strip that bit.

Note that these do not perform client side redirects, so they do not return the ultimate originating URL https://x.com/jack/status/20 (which was the first ever Tweet on what was back then called twttr) as Twitter on the client-side overwrites window.location.href with the final URL. Similar behaviour for getting the Twitter user handle of a Twitter user ID, more on Twitter tricks below.

Tweet by TweetID trick via [Wayback/Archive] Accessing a tweet using only its ID (and without the Twitter API) – Bram.us.

Further reading (thanks [Wayback/Archive] vise, [Wayback/Archive] Daniel Stenberg, [Wayback/Archive] Ivan, [Wayback/Archive] AndrewF, [Wayback/Archive] Roger Campanera, and [Wayback/Archive] Dave Baird):

A

curl‘s -w option and the sub variable url_effective is what you are looking for.

Something like

curl -Ls -o /dev/null -w %{url_effective} https://example.com

More info

-L         Follow redirects
-s         Silent mode. Don't output anything
-o FILE    Write output to <file> instead of stdout
-w FORMAT  What to output after completion

More

You might want to add -I (that is an uppercase i) as well, which will make the command not download any “body”, but it then also uses the HEAD method, which is not what the question included and risk changing what the server does. Sometimes servers don’t respond well to HEAD even when they respond fine to GET.

C

Some websites also need a spoofed user agent with curl -A ... to redirect to the expected location.

C

@minseong -o /dev/null suppresses the HTTP response from being outputted anywhere. -s is for curl’s own output (like progress or errors) and specifically does not cover the HTTP response.

C

Condiser adding a newline char to have a nice output: curl -Ls -o /dev/null -w "%{url_effective}\n" https://example.com

A

curl can only follow http redirects. To also follow meta refresh directives and javascript redirects, you need a full-blown browser like headless chrome

…

Note that the first link in the answer has moved and rewritten, but the redirect is broken. It is now at [Wayback/Archive] Write out – everything curl.

Some more interesting Twitter link conversions and scraping tricks

I had some Twitter user IDs that I wanted the handles from and vice versa.

Examples are for Twitter user ID 12 for handle jack (Jack Dorsey, original founder of Twitter before it got taken over by Space Karen)

Note that for all the examples you can use http URLs instead of https URLs costing more redirects.

I might write some Bookmarklets for these one day.

Here we go:

Getting the Twitter user handle from a Twitter user ID

From Twitter user ID URL to Twitter user profile URL:

  1. Start: https://x.com/i/user/12 or https://twitter.com/i/user/12
  2. Finish: https://x.com/jack and in the past https://twitter.com/jack

From Twitter follow user by ID URL to Twitter follow user by handle URL:

  1. Start: https://x.com/intent/user?user_id=12 or https://twitter.com/intent/user?user_id=12
  2. Finish: https://x.com/intent/user?screen_name=jack and in the past https://twitter.com/intent/user?screen_name=jack

Getting the Twitter user ID from a Twitter user handle

  1. Start:  https://x.com/jack or https://twitter.com/jack
  2. Client side redirect to https://x.com/jack
    1. Append /header_photo: https://x.com/jack/header_photo
  3. Run this JavaScript: document.querySelector('img[src*="https://pbs.twimg.com"]').src
  4. The resulting image URL contains Twitter user ID 12 right after https://pbs.twimg.com/profile_banners/: https://pbs.twimg.com/profile_banners/12/1742427520/1500×500

To get the ID itself in step 4, the JavaScript gets a tad more complex: document.querySelector('img[src*="https://pbs.twimg.com"]').src.split('https://pbs.twimg.com/profile_banners/')[1].split('/')[0]

Note: you can shortcut from step 2, but be ware that the JavaScript then returns a linkl like https://pbs.twimg.com/profile_images/874985777524092928/77_ho04k_normal.jpg forcing the JavaScript to be a tad longer:

  1. Start:  https://x.com/jack or https://twitter.com/jack
  2. Client side redirect to https://x.com/jack
  3. Run this JavaScript: document.querySelector('img[src*="https://pbs.twimg.com/profile_banners/"]').src
  4. The resulting image URL contains Twitter user ID 12 right after https://pbs.twimg.com/profile_banners/: https://pbs.twimg.com/profile_banners/12/1742427520/600×200

To get the ID itself in step 3, the JavaScript gets a tad more complex: document.querySelector('img[src*="https://pbs.twimg.com/profile_banners/"]').src.split('https://pbs.twimg.com/profile_banners/')[1].split('/')[0]

Note the above querySelector paths are way shorter than what browser debugger tools return. For creating these shorter ones, read these three links and use some imagination:

The final two three above have examples explaining the third link in more detail.

Note I need to research how to get the results of https://syndication.twitter.com/srv/timeline-profile/screen-name/jack (JSON __NEXT_DATA__ script embedded in HTML) into a parseable form.

Get the Twitter user ID from a Tweet

  1. Start https://x.com/jack/status/20 or https://twitter.com/jack/status/20 (as explained above the first ever Tweet on Twitter)
  2. Tweet ID: 20
  3. CDN URL: https://cdn.syndication.twimg.com/tweet-result?id=20&token=!
  4. Load and parse the content using JSONPath query $.user.id_str
  5. For the Twitter user handle use $.user.name

Note: I didn’t have time yet to figure out how to do this in the client-side in a JavaScript Bookmarklet when starting from the Tweet page.

Hopefully by then I also have time to write Bookmarklets for Twitter image size suffixes reloaded: pbs.twimg.com URL information and the above snippets.

Displaying the Tweet if you only have the Tweet ID

Again using the example Tweet ID 20 (replace your own where needed) and these still work and client-side redirect to the x.com or mobile.x.com domain depending on your platform:

References

The above materials are adapted from this content:

Queries:

Further research

I need to check out these links, as some of their material is likely outdated, and others likely have gems in them:

--jeroen

This entry was posted on 2025/08/06 at 12:00 and is filed under *nix, *nix-tools, bash, Batch-Files, Bookmarklet, Communications Development, Conference Topics, Conferences, CSS, cURL, Development, Event, HTTP, Internet protocol suite, JavaScript/ECMAScript, Power User, Scripting, SocialMedia, Software Development, TCP, Twitter, Web Browsers, Web Development. Tagged: . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

«
»