Archive for the ‘wget’ Category
Posted by jpluimers on 2016/09/29
I needed to script a few things on my Fritz!Box. Here are the cURL links that I used to research some Fritz!Box scripting.
My first try was wget, but that didn’t do everything I need, so cURL came to the rescue.
In the end, I didn’t need cookies (a post request with an MD5 based handshake sufficed to get a session SID which is not stored in a Cookie), but that surely will come in useful another time.
Curl man page entries:
The script is and docs are here: jpluimers/bash-fritzclient.
–jeroen
Posted in *nix, bash, cURL, Development, Fritz!, Fritz!Box, Network-and-equipment, Power User, Scripting, Software Development, wget | Leave a Comment »
Posted by jpluimers on 2016/04/05
Even though cURL seems to handle https better than wget, it is not a wget replacement.
From the cURL FAQ :
1.3 What is cURL not?
Curl is not a wget clone. That is a common misconception. Never, during curl’s development, have we intended curl to replace wget or compete on its market. Curl is targeted at single-shot file transfers.
Curl is not a web site mirroring program. If you want to use curl to mirror something: fine, go ahead and write a script that wraps around curl to make it reality (like curlmirror.pl does).
Curl is not an FTP site mirroring program. Sure, get and send FTP with curl but if you want systematic and sequential behavior you should write a script (or write a new program that interfaces libcurl) and do it.
–jeroen
via:
Posted in *nix, cURL, Power User, wget | Leave a Comment »
Posted by jpluimers on 2016/01/13
Many sites giving your public IP address return a web page with a bloat of html. From the command-line, you are usually only interested in the IP-address itself. Few services return exactly that.
Below are command-line examples to provide the public IP address mostly from a *nix perspective. Usually you can get similar commands to work with Windows binaries for wget and Windows binaries for curl.
In the end, I’ve opted for commands in this format, as I think akamai will last longer than the other sites (but does not include an end-of-line in the http result hence the echo on Mac/*nix):
I’ve not tried aria2 yet, but might provide commands for that in the future.
These are the Linux permutations for akamai:
curl whatismyip.akamai.com && echo
curl ipv4.whatismyip.akamai.com && echo
curl ipv6.whatismyip.akamai.com && echo
curl ipv4.whatismyip.akamai.com && echo && curl ipv6.whatismyip.akamai.com && echo
The last two are convenient when you have both IPv4 and IPv6 configured on “the outside”.
You can replace curl with wget -q -O – (which outputs to stdout) for each command. You can even ommit the http:// (as that is the default protocol for both curl and wget).
Read the rest of this entry »
Posted in *nix, *nix-tools, Apple, bash, bash, Batch-Files, cURL, Development, Linux, Mac, Mac OS X / OS X / MacOS, Mac OS X 10.4 Tiger, Mac OS X 10.5 Leopard, Mac OS X 10.6 Snow Leopard, Mac OS X 10.7 Lion, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, MacMini, OS X 10.10 Yosemite, OS X 10.8 Mountain Lion, OS X 10.9 Mavericks, Power User, Scripting, Software Development, SuSE Linux, wget | Leave a Comment »
Posted by jpluimers on 2014/10/03
Had to download a bunch of stuff over the command-line from an IIS server that was using authentication. Not basic authentication, but NTLM authentication.
wget kept failing, even wget 1.10 that usually does NTLM quite OK (but up to 1.10.2 has a security vulnerability so you should not use wget 1.10 any more).
So I installed a Windows x86 cURL binary, and downloaded+copied the root certificates, then did some reading on the command-line switches.
Without any, cURL does http basic authentication. But a Windows server usually expects NTLM authentication (hardly documented, but it uses the Negotiate protocol).
When not using NTLM, both would show (wget -d, or curl -v) this in the output, indicating you should use NTLM authentication: Read the rest of this entry »
Posted in *nix, *nix-tools, cURL, Linux, Power User, SuSE Linux, wget, Windows, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2 | Leave a Comment »
Posted by jpluimers on 2014/06/24
Last week, I thanked Jaykul for helping me out on PowerShell.
But he taught me another thing that was new for me: on his site, he has hidden his email address behind a 302-redirect.
I didn’t even realize that was possible until I saw his site start my mail program without initially showing a mailto in the url. A quick check showed me he was using a 302-redirect: Read the rest of this entry »
Posted in *nix, Apache2, bash, Development, Linux, Power User, Scripting, Software Development, SuSE Linux, wget | Leave a Comment »
Posted by jpluimers on 2014/04/29
I needed AntiVir on an openSUSE workstation.
Too bad the default installation package from YaST installed an old license: AntiVir would not work, and I was getting emails like these at regular intervals:
<br />Date: Thu, 25 Apr 2014 08:57:11 +0200<br />From: Cron Daemon <root@....><br />To: root@....<br />Subject: Cron <root@...> /usr/lib/AntiVir/guard/avupdate-guard --product=Scanner > /dev/null<br /><br />Error: No valid license was found<br />
After searching the web for a while, I found a lot of posts with wrong information, basically coming down to these 2:
At the bottom of the post, you will find a small shell script that I use to keep the hbedv.key up-to-date.
First some more about HBEDV, then how I found about the new download location, and a command to show you the current license information. Read the rest of this entry »
Posted in *nix, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux, wget | 3 Comments »
Posted by jpluimers on 2014/03/23
Interesting: ESXi 5.1 and rsync – damiendebin.net. It works in ESXi 5.5 too, and the French link below has a version that runs on ESXi 4.
Now you can do these forms of backup:
Posted in *nix, ESXi4, ESXi5, ESXi5.1, ESXi5.5, Power User, VMware, VMware ESXi, wget | 6 Comments »
Posted by jpluimers on 2014/03/22
If your downloaded WinSCP install ever throws this error, and you cannot find the cause on the WinSCP forums, then this is probably the reason:
The caption of the error message should be enough: Windows thinks it is a 16-bit MS-DOS program, not a Windows installer.
This means that the install download went wrong. In my case, I downloaded it through this wget command:
wget -m -np http://winscp.net/download/winscp552setup.exe
At first I thought that WinSCP has a referrer check, so this is the command that would work (similar to wget direct download with referer: SpeedFan):
wget -m -np --referer=http://winscp.net/eng/download.php http://winscp.net/download/winscp552setup.exe
Read the rest of this entry »
Posted in *nix, Power User, wget, Windows, Windows XP | Leave a Comment »
Posted by jpluimers on 2014/01/27
Lately I moved more and more away from wget, mainly because out of the box, wget (and also aria2, which I like for the bittorrent support) handle https downloads so badly: you need to manually setup your CA store on each and every installed system.
Not so with cURL, especially not on Windows any more, as “recently” (that is: since the last time I examined it, which is over a year ago now), there is a new kid in town: cURL for Windows: a Windows Installer for the Web Transfer Tool.
You don’t even need to download the installer. Grabbing the stuff from the bin directory in the zip download is enough: it contains a prepackaged CA certificate set that works splendid.
So now downloading https://dl.google.com/update2/installers/ChromeStandaloneSetup.exe to the current directory is as simple as Read the rest of this entry »
Posted in *nix, Batch-Files, Development, Power User, Scripting, Software Development, wget, Windows | Tagged: cURL, wget | Leave a Comment »
Posted by jpluimers on 2013/12/23
wget is immensely useful tool to download files using ftp, http and https, especially as it allows recursive downloads and mirroring with some very nice options.
Mac OS X doesn’t come with wget, and curl – the alternative for wget – cannot do recursion, so you need wrapper scripts for that.
Basically there are two ways to get wget installed on Mac OS X:
- Compile it from the source, then install it like Install wget in Mac OS X Without Homebrew or MacPorts.
- Download a prebuilt version like wget – Prebuilt binary for Mac OSX Lion, Snow Leopard and Mountain Lion | Tech Tach.
For both ways you need to remember that they won’t automatically update. So: keep an eye on wget security vulnerabilities, and update as soon as new ones have been found.
The first way (build from source) needs you to download and install Xcode first. Since I’m a Mac OS X developer, I already have that.
Luckily Install wget in Mac OS X Without Homebrew or MacPorts had instructions for the most current version when writing this blog entry. The binary from Tech Tach was outdated.
That, and the my feel for greater influence on the built proces makes me like the first way more.
Below are the commands I used (thanks OSXDaily!).
Check http://ftp.gnu.org/gnu/wget/ to make sure you downloaded the most current wget sourcecode. Read the rest of this entry »
Posted in *nix, Apple, Mac, Mac OS X / OS X / MacOS, Mac OS X 10.4 Tiger, Mac OS X 10.5 Leopard, Mac OS X 10.6 Snow Leopard, Mac OS X 10.7 Lion, OpenSSL, OS X 10.8 Mountain Lion, Power User, Security, wget | Leave a Comment »
The Wiert Corner - irregular stream of stuff 