The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 4,229 other subscribers

Archive for the ‘wget’ Category

« Previous Entries

Downloading files with wget on ESXi · random writes

Posted by jpluimers on 2021/05/31

Reminder to self to check if wget on ESXi now finally supports https downloading: [WayBack] Downloading files with wget on ESXi · random writes.

In the mean time, ESXi 6.7 Update 2 and up seems to support this; so the below workaround might only be needed for ESXi 6.7 update 1 and below.

[WayBack] VMware ESXi: help downloading large ISO – Server Fault

I will likely not do this, as by now all my ESXi boxes should have been recent enough.

I will keep the article because of the BusyBox commands section below.

If so, I might finally try and write a Python wrapper for this, as I know that Python 3 on ESXi supports https, but the ESXi BusyBox does not have a built-in cURL.

Some links and notes I might need by then:

BusyBox commands

Another cool thing in the above blog post is that it shows how to dump the BusyBox built in commands.

I ran it for ESXi 6.7 with a slight trick to get the full path (using back-ticks and escaped back-ticks) and content.

Since ESXi is BusyBox based, the commands that are in /bin are not actually binaries, but each command is a symlink to the BusyBox binary. BusyBox then knows the original name of the command, so it can deduct what part to execute. This makes for a very space efficient storage scheme.

The various bits of the tricks to get the location of the BusyBox binary, so the --list parameter can be passed to it:

  • The which wget gives the full path of wget.
  • The ls -l `which wget` shows the full path of wget and the symlink target (but there is no way for ls to only show the symlink target).
  • The readlink -f `which wget` shows the full path of where /bin/wget points to: the BusyBox binary.

The main trick consists of backtick evalution, and knowing that ls cannot get you just the symlink target, but readlink can:

Now the back-tick escapes, because you cannot nest back-ticks:

  • The `readlink -f \`which wget\`` executes the BusyBox binary without arguments.
  • The `readlink -f \`which wget\`` --list executes the BusyBox binary with the --list parameter.

Note I do not like the cat --help (see [WayBack] How do I check busybox version (from busybox)? – Unix & Linux Stack Exchange) way of getting the BusyBox version, as that gets echoed to stderr.

This is the output:

Read the rest of this entry »

Posted in *nix, *nix-tools, cURL, ESXi6, ESXi6.5, ESXi6.7, Power User, Virtualization, VMware ESXi, wget | Leave a Comment »

OpenCandy – Wikipedia

Posted by jpluimers on 2020/11/06

Hmm, one of my machines contained OpenCandy – Wikipedia as found by Malwarebytes (software) – Wikipedia:

Tracking back the installation, revealed it came with ImgBurn 2.5.8.0, which is now on my black-list.

In my case this was how to remove it:

rd /s /q %AppData%\OpenCandy

This is not universal; you might need to take additional measures like in [WayBack] How to Remove PUP.Optional.OpenCandy (Removal Guide).

I use this batch-file to get the most recent Malwarebytes and Chameleon:

get-malware-bytes.bat 

:: redirects to something like wget https://data-cdn.mbamupdates.com/web/mb3-setup-consumer/mb3-setup-consumer-3.6.1.2711-1.0.508-1.0.8211.exe
wget --content-disposition https://downloads.malwarebytes.com/file/mb3/
:: redirects to something like wget https://data-cdn.mbamupdates.com/web/mbam-chameleon-3.1.33.0.zip
wget --content-disposition https://downloads.malwarebytes.com/file/chameleon/

If you do not have wget on your system, then try this PowerShell alternative (which does not show progress) via [WayBack] Windows batch file file download from a URL – Stack Overflow

:: in case you do not have wget:
powershell -Command "(New-Object Net.WebClient).DownloadFile('https://downloads.malwarebytes.com/file/mb3/', 'mb3.exe')"
powershell -Command "(New-Object Net.WebClient).DownloadFile('https://downloads.malwarebytes.com/file/chameleon/', 'chameleon.exe')"
:: note these do not show progress!
:: https://stackoverflow.com/questions/4619088/windows-batch-file-file-download-from-a-url

Related: [WayBack] Jeroen Pluimers on Twitter: “What if the most recent @Malwarebytes on a Windows 8.1 x64 VM (all patches installed) on ESXi backed by NVME hangs for hours on one file with hardly any CPU usage? Screenshots of mbam.exe, mbamservice.exe and mbamtray.exe thread usage below.

–jeroen

Posted in *nix, *nix-tools, LifeHacker, Power User, Security, wget, Windows | Leave a Comment »

curl/wget: use content-disposition for the file names

Posted by jpluimers on 2020/04/27

For me, on Windows, curl works better than wget, but on Linux/Mac OS X, curl tends to work better. Some people find wget easier for downloading multiple URLs at the same time. So here the parameter switches for both so they download to the file specified by the Content-Disposition http header:

  • curl --remote-name --remote-header-name
  • wget --content-disposition

My experience is that wget is better at this, especially when redirects are involved (by adding a [WayBack] --location parameter to  thecurl command line).

So for instance the first fails, but the second succeeds determining the download to be VSCodeUserSetup-x64-1.27.2.exe (so curl keeps the name stable):

curl.exe --location --remote-name --remote-time --remote-header-name https://vscode-update.azurewebsites.net/latest/win32-x64-user/stable

wget.exe --content-disposition https://vscode-update.azurewebsites.net/latest/win32-x64-user/stable

This takes into account the name after all followed redirects.

Via:

–jeroen

Posted in *nix, cURL, Power User, wget | Leave a Comment »

Redirect Detective – A Free Tool To Trace Where Redirects End Up

Posted by jpluimers on 2020/02/27

Cool tool, but be aware it does aJavaScript:xmlhttpPost from , so your data can be logged [WayBack] Redirect Detective – A Free Tool To Trace Where Redirects End Up.

For http://www.xs4all.nl/~jp, it shows these redirects (where in bold I’ve listed the changes):

Oh and it runs this call: JavaScript:xmlhttpPost("/linkdetect.px")?word=www.xs4all.nl%2F%7Ejp from [WayBack] redirectdetective.com/ajax.js.

Note that this shows more redirects than the plain http ones, so wget from [WayBack] wget – How do I display all URLs in a redirect chain? – Unix & Linux Stack Exchange shows this:

$ wget http://www.xs4all.nl/~jp 2>&1 | grep Location:
Location: https://www.xs4all.nl/~jp [following]
Location: https://jp.home.xs4all.nl/ [following]

–jeroen

Posted in *nix, *nix-tools, Communications Development, Development, HTTP, Internet protocol suite, Power User, TCP, wget | Leave a Comment »

wget and curl: downloads that sometimes fail

Posted by jpluimers on 2018/10/19

For my archive somewhere between cURL 7.21.0 and 7.34.0 it does not like to be started from an RDP based tsclient share:

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.21.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  60805      0 --:--:--  0:00:01 --:--:-- 70012

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.34.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: web.archive.org

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: www.xs4all.nl

C:\Users\jeroen\Downloads>copy \\tsclient\bin\curl.7.61.0.exe
        1 file(s) copied.

C:\Users\jeroen\Downloads>curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13    0    13    0     0     10      0 --:--:--  0:00:01 --:--:--    10

It fails the same way after net use B: \\tsclient\bin, so that does not matter.

The best link I could find until I got to the real problem was [WayBack] curl: (6) Could not resolve host: application – Stack Overflow which shows a different problem: properly quoting.

In addition to remote-name, you can also grab the file name from the headers using --remote-header-name, and --remote-time use the remote file time. The --location follows 302-redirects. You can see that in the example below which I build based on

[WayBack] unix – Curl to grab remote filename after following location – Stack Overflow: The remote side sends the filename using the Content-Disposition header.curl 7.21.2 or newer does this automatically if you specify –remote-header-name / -J.curl -O -J -L $url

C:\Users\jeroen\Downloads>b:\curl.7.21.0.exe --location --remote-name --remote-time --remote-header-name "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  45748      0 --:--:--  0:00:01 --:--:-- 50772
curl: Saved to filename 'pkgWuppdiWP_DX102T_1-1-2.zip'

wget failed big time:

C:\Users\jeroen\Downloads>B:\wget.exe --no-check-certificate -v -v -v --content-disposition --restrict-file-names=windows "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
wget: Cannot read b:/.wgetrc (No such file or directory).
--2018-07-12 09:55:23--  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Resolving web.archive.org... 207.241.225.186
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Retrying.

...

--2018-07-12 09:55:23--  (try:20)  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Giving up.

This is not caused by the filename (Windows does not like the ? question mark in output file names, so  – like & ampersand in file URLs – you have to quote the full URL, but also provide the --restrict-file-names=windows parameter; see [WayBack] wget – I can’t download files with “?” – Super User).

–jeroen

Posted in *nix, *nix-tools, cURL, Power User, wget | Leave a Comment »

« Previous Entries
 
%d bloggers like this: