The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 4,184 other subscribers

Archive for October 19th, 2018

wget and curl: downloads that sometimes fail

Posted by jpluimers on 2018/10/19

For my archive somewhere between cURL 7.21.0 and 7.34.0 it does not like to be started from an RDP based tsclient share:

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.21.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  60805      0 --:--:--  0:00:01 --:--:-- 70012

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.34.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: web.archive.org

C:\Users\jeroen\Downloads>\\tsclient\bin\curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: www.xs4all.nl

C:\Users\jeroen\Downloads>copy \\tsclient\bin\curl.7.61.0.exe
        1 file(s) copied.

C:\Users\jeroen\Downloads>curl.7.61.0.exe --remote-name https://www.xs4all.nl/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13    0    13    0     0     10      0 --:--:--  0:00:01 --:--:--    10

It fails the same way after net use B: \\tsclient\bin, so that does not matter.

The best link I could find until I got to the real problem was [WayBack] curl: (6) Could not resolve host: application – Stack Overflow which shows a different problem: properly quoting.

In addition to remote-name, you can also grab the file name from the headers using --remote-header-name, and --remote-time use the remote file time. The --location follows 302-redirects. You can see that in the example below which I build based on

[WayBack] unix – Curl to grab remote filename after following location – Stack Overflow: The remote side sends the filename using the Content-Disposition header.curl 7.21.2 or newer does this automatically if you specify –remote-header-name / -J.curl -O -J -L $url

C:\Users\jeroen\Downloads>b:\curl.7.21.0.exe --location --remote-name --remote-time --remote-header-name "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 86465    0 86465    0     0  45748      0 --:--:--  0:00:01 --:--:-- 50772
curl: Saved to filename 'pkgWuppdiWP_DX102T_1-1-2.zip'

wget failed big time:

C:\Users\jeroen\Downloads>B:\wget.exe --no-check-certificate -v -v -v --content-disposition --restrict-file-names=windows "https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965"
wget: Cannot read b:/.wgetrc (No such file or directory).
--2018-07-12 09:55:23--  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Resolving web.archive.org... 207.241.225.186
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Retrying.

...

--2018-07-12 09:55:23--  (try:20)  https://web.archive.org/web/20180712073755if_/https://www.danielwolf.eu/?wpdmdl=1965
Connecting to web.archive.org|207.241.225.186|:443... failed: Invalid argument.
Giving up.

This is not caused by the filename (Windows does not like the ? question mark in output file names, so  – like & ampersand in file URLs – you have to quote the full URL, but also provide the --restrict-file-names=windows parameter; see [WayBack] wget – I can’t download files with “?” – Super User).

–jeroen

Posted in *nix, *nix-tools, cURL, Power User, wget | Leave a Comment »

Euler’s identity – cool explanatory videos by Burkard Polster

Posted by jpluimers on 2018/10/19

[WayBack] e^iπ = -1 – Daniela Osterhagen – Google+ reminded me of [WayBackEuler’s identity (often called the most beautiful formula in the world) and these two great videos by Burkard Polster I bumped into a few years back below.

She points to a German link about “the most beautiful formula in the world”, to which I fully agree:

Gerade einmal sechs Symbole braucht es dafür

[WayBackFreistetters Formelwelt: Die schönste Formel der Welt – Spektrum der Wissenschaft

Read the rest of this entry »

Posted in Power User, science | Leave a Comment »

when btrfs-size shows a snapshot as 16777216.00TB or btrfs qgroup as 16.00EiB

Posted by jpluimers on 2018/10/19

A long time ago I wrote about the btrfs-size tool: [WayBackA bash script to btrfs snapshot details like disk sizes (requires btrfs quota to be enabled).

One day, it showed a ridiculously large size for /tmp:

# ./btrfs-size.sh 
=============================================================================================================================================================================================================================================================
Snapshot / Subvolume                                               ID   Total    Exclusive Data
=============================================================================================================================================================================================================================================================
257 gen 855182 top level 5 path .snapshots                         257  4.30MB   4.30MB   
258 gen 856438 top level 257 path .snapshots/1/snapshot            258  1.84GB   193.01MB 
...
262 gen 856438 top level 5 path srv                                262  1.83GB   1.83GB   
263 gen 856438 top level 5 path tmp                                263  16777216.00TB16777216.00TB
264 gen 856438 top level 5 path usr/local                          264  260.00KB 260.00KB 
...
990 gen 849192 top level 257 path .snapshots/583/snapshot          990  1.83GB   8.23MB   
991 gen 849224 top level 257 path .snapshots/584/snapshot          991  2.09GB   62.66MB  
=============================================================================================================================================================================================================================================================
                                                                Exclusive Total: 3.26GB    

This tracks back to the output of this command, which I’ve shortened a bit:

# btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5          16.00KiB     16.00KiB
0/257         4.30MiB      4.30MiB
...
0/262         1.83GiB      1.83GiB
0/263        16.00EiB     16.00EiB
0/264       260.00KiB    260.00KiB
...
255/274         0.00B        0.00B
255/797      16.00KiB     16.00KiB

This is a known issue as quotas in btrfs – though workable – aren’t fully stable yet: [WayBack] Linux BTRFS Storage: Re: During a btrfs balance nearly all quotas of the subvolumes became exceeded

It also provides this simple solution:

Read the rest of this entry »

Posted in *nix, *nix-tools, btrfs, File-Systems, Power User | Leave a Comment »

git 2.19.1 on Windows: NullReferenceException during https pull/push/fetch

Posted by jpluimers on 2018/10/19

When using git 2.19.1 on Windows, you will likely get a NullReferenceException during pull/push/fetch operations on https connections.

Easiest workaround is to downgrade to 2.19.0: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub

I am seeing the same thing using 2.19.1 against our internal Bitbucket server. Reverting back to 2.19.0 the problem no longer occurs.

Harder workaround is to install the Git Credential Manager 1.18.1: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub.

[WayBack] Release Git Credential Manager for Windows v1.18.1 · Microsoft/Git-Credential-Manager-for-Windows · GitHub

Bug Fixes:

  • Fixes Null Reference exceptions when parameters or contentType are not populated

To install the Git Credential Manager, download and double-click the GCMW-1.18.1.exe installer. It is that easy, it will even install Git for Windows 2.17.0 and the Microsoft .Net Framework for you if necessary.

Hopefully that one will integrated in a minor update to git for Windows soon: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub

moderately soon, hopefully, there will be a minor version release of Git for Windows that will include the updated GCM v1.18.1. [just need more volunteer time for the PR and tests ;-)
That should eliminate the `Fatal: NullReferenceException” but appears not to fix the #1874https://github.com/git-for-windows/git/issues/1874 delays. (still to hear from the OP if the issue is definitely the AV, or something elese)

If you really dare, you can try the nightly snapshot which already includes GCMW-1.18.1: [WayBack] git pull fatal: NullReferenceException encountered. On an local instance of Bitbucket · Issue #1868 · git-for-windows/git · GitHub

the next official Git for Windows version will have a new Git Credential Manager version. Also, the current snapshot at https://wingit.blob.core.windows.net/files/index.html should have it already.

–jeroen

Posted in Development, DVCS - Distributed Version Control, git, Software Development, Source Code Management | Leave a Comment »

 
%d bloggers like this: