The Wiert Corner – irregular stream of stuff

I have been using rsync for years now to sync directories around on VMware ESXi systems with great success.

Usually when upgrading to new physical machines, I synced the things to migrate to a big disk, then physically mounted that disk in a new machine.

Not this time: because of an unscheduled event I had to bring up the target machine in a different place before I could assemble the data on the big disk.

But since there was a network tunnel between the networks of both ESXi machines, I wanted to transfer over ssh: a thing I had done numerous times between various Linux and unix machines.

The cool thing is that rsync supports remote connections over ssh out of the box. Between ESXi machines, I bumped into this error though:

sh: rsync: not found
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.2]
Command exited with non-zero status 12

The last three lines are collateral of the first line, which I queried [Wayback/Archive] “sh: rsync: not found” – Google Search. Further below I will go deeper into the rabbit-hole I took, but the actual solution was to nudge the local rsync the location of the remote rsync. This is specified in the [Wayback/Archive] rsync: a fast, versatile, remote (and local) file-copying tool | rsync Commands | Man Pages | ManKier under the [Wayback/Archive] --rsync-path option

–rsync-path=PROGRAM

Use this to specify what program is to be run on the remote machine to start-up rsync.  Often used when rsync is not in the default remote-shell’s path (e.g. –rsync-path=/usr/local/bin/rsync).  Note that PROGRAM is run with the help of a shell, so it can be any program, script, or command sequence you’d care to run, so long as it does not corrupt the standard-in & standard-out that rsync is using to communicate.

One tricky example is to set a different default directory on the remote machine for use with the –relative option.  For instance:

rsync -avR --rsync-path="cd /a/b && rsync" host:c/d /e/

So my solution is to append either of these to the rsync parameters:

1. --rsync-path="/opt/bin/rsync"
2. --rsync-path="source /etc/profile; rsync"
3. --rsh="ssh --login"

In retrospect, I prefer the second solution (the third solution works when the local system has OpenSSH, but not with BusyBox like on ESXi).

I will explain all three solutions below:

Queries

Below is basically the route of queries I needed leading me to the solution above.

1. [Wayback/Archive] “sh: rsync: not found” – Google Search
2. [Wayback/Archive] rsync ssh show environment remote machine – Google Search
3. [Wayback/Archive] ssh list remote environment variables – Google Search
4. [Wayback/Archive] “ESXi” “/etc/.ssh/environment” – Google Search
5. [Wayback/Archive] esxi ssh set environment variable – Google Search
6. [Wayback/Archive] rsync specify remote location – Google Search

Query results

In retrospect, I should have looked at the second result of the first query instead of the Stack Overflow question (which ended up as first). The second result for me gave me the first solution above. It is from [Wayback/Archive] sh: rsync: not found – Brakertech:

This message means rsync is not in the $PATH of the remote user on the remote system.

…

You may specify a path to the remote system’s rsync using --rsync-path.

Example:
$ rsync -av –progress --rsync-path=/usr/local/bin/rsync Solaris_8_2004-02/ root@10.209.173.50:/export/install/OS

he linked to the now defunct page from 2008 [Wayback/Archive] sh: rsync: not found

You may specify a path to the remote system’s rsync using –rsync-path.

Example:
$ rsync -av --progress --rsync-path=/usr/local/bin/rsync Solaris_8_2004-02/ root@10.209.173.50:/export/install/OS

Instead, I went the Stack Overflow route which usually is a good source of information, at [Wayback/Archive] linux – rsync command is not working when executing from remote ssh – Stack Overflow (thanks [Wayback/Archive] DevOps_Guy_1903 and [Wayback/Archive] user3601578):

A

Check the remote server’s PATH environment variable: Jenkins executes commands in a shell environment, and the error indicates that the shell cannot find the rsync executable. Ensure that the PATH environment variable is properly set on the remote server.

C

Thanks, I moved rsync from executable from /usr/local/bin to /usr/bin/ and now it works

So I checked the remote environment for the value of PATH:

# ssh root@192.168.71.91 "env"
Password: 
USER=root
MAIL=/var/mail/root
SSH_CLIENT=192.168.71.97 50445 22
SHLVL=1
HOME=/
LOGNAME=root
PATH=/bin:/sbin
LANG=en_US.UTF-8
SHELL=/bin/sh
LC_ALL=en_US.UTF-8
PWD=/
SSH_CONNECTION=192.168.71.97 50445 192.168.71.91 22

So somehow this part of the remote /etc/profile script is not executed for non-interactive SSH sessions:

PATH=/bin:/sbin:/opt/bin/
export PATH

The second query led to the an interesting diversion that I might one day use for a different problem: [Wayback/Archive] Can rsync use a remote environment variable? – Super User (thanks [Wayback/Archive] nsheff, [Wayback/Archive] akira and [Wayback/Archive] Gordon Davisson)

Q

Say I want to use rsync to copy a remote folder to my local desktop:

rsync -av remote:$REMOTEDATA $LOCALDATA

But I want to access the $REMOTEDATA environment variable on the remote server, not the local desktop.

A

you have to get the environment from the remote machine into your local machine:

$> export $(ssh remote "env | grep REMOTEDATA) 
$> rsync -av remote:$REMOTEDATA $LOCALDATA

or something like that. the command (the fact that it is rsync does not really matter) is executed locally and your shell is replacing the $variables locally.

C

Environment variables aren’t set on computers, they’re per-process things. I suspect the environment variable you’re trying to get is set up by the shell initialization script(s) (/etc/profile or ~/.profile or something), so in order for the variable to even exist, you need to create a shell process on the remote computer, let the profile script create the variable, and then have the shell report the variable’s value. (Which is what this answer does.)

I started the third query to ensure that my knowledge of showing environment variables through env or export were still current:

• [Wayback/Archive] env: set the environment for command invocation | POSIX Commands | Man Pages | ManKier
• [Wayback/Archive] export: set the export attribute for variables | POSIX Commands | Man Pages | ManKier

That query got me to [Wayback/Archive] Why does an SSH remote command get fewer environment variables then when run manually? – Stack Overflow (thanks [Wayback/Archive] Tom Feiner, [Wayback/Archive] Vinko Vrsalovic, [Wayback/Archive] codebeard, [Wayback/Archive] R.Liu, [Wayback/Archive] dpedro and [Wayback/Archive] Ian Vaughan):

Q

I have a command that runs fine if I ssh to a machine and run it, but fails when I try to run it using a remote ssh command like :

ssh user@IP <command>

Comparing the output of “env” using both methods resutls in different environments. When I manually login to the machine and run env, I get much more environment variables then when I run :

ssh user@IP "env"

Any idea why ?

A

There are different types of shells. The SSH command execution shell is a non-interactive shell, whereas your normal shell is either a login shell or an interactive shell. Description follows, from man bash:

[Wayback/Archive] bash Invocation: GNU Bourne-Again SHell | bash Commands | Man Pages | ManKier

C

That’s quite a wall of text, I think it would be worth highlighting that ssh user@host "bash --login -c 'command arg1 ...'" will make the remote shell set up the login environment. The section you quoted does mention --login but it would be easy to overlook this.

C

In some UNIX system(like AIX), add environment variables into /etc/profile won’t help, but /etc/environment works.

A

Shell environment does not load when running remote ssh command. You can edit ssh environment file:

vi ~/.ssh/environment

Its format is:

VAR1=VALUE1
VAR2=VALUE2

Also, check sshd configuration for PermitUserEnvironment=yes option.

A

How about sourcing the profile before running the command?

ssh user@host "source /etc/profile; /path/script.sh"

You might find it best to change that to ~/.bash_profile, ~/.bashrc, or whatever.

(As here (linuxquestions.org))

[Wayback/Archive] Simple rsh issue

Quote:

Originally Posted by pwc101  How about sourcing the profile before running the command? Code: ssh user@host "source /etc/profile; /path/script.sh" You might find it best to change that to ~/.bash_profile, ~/.bashrc, or whatever.

Code:

ssh 172.24.40.100 "source /root/.bash_profile; /path/script.sh"

Worked! You ppl are great! Cheers.

I explained the first answer earlier in Filezilla: figuring out the cause of “Connection timed out after 20 seconds of inactivity”, but the problem here is that ESXi is not running bash, but dash (I wrote about this for instance in ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes) which has very little information on which scripts run under what circumstances.

The second answer relies on PermitUserEnvironment=yes which is definitely not recommended (it is why I sidestepped into the fourth query, just in case there was any support for it from ESXi: it wasn’t).

The last answer however is spot on: it is a great alternative when you do not know the remote location of rsync as this script shows the remote PATH is now correct:

# ssh root@192.168.71.91 "source /etc/profile; env"
Password:
USER=root
MAIL=/var/mail/root
SSH_CLIENT=192.168.71.97 53418 22
SHLVL=1
HOME=/
TERMINFO=/usr/share/terminfo
PS1=[root@\h:\w]
LOGNAME=root
VI_USERNAME=root
PATH=/bin:/sbin:/opt/bin/
LANG=en_US.UTF-8
SHELL=/bin/sh
LC_ALL=en_US.UTF-8
PWD=/
SSH_CONNECTION=192.168.71.97 53418 192.168.71.91 22
TMOUT=0

The comments to the first answer however lead me to figuring out if I could if I could pass parameters to ssh from rsync, and indeed that is possible via [Wayback/Archive] rsync --rsh: a fast, versatile, remote (and local) file-copying tool | rsync Commands | Man Pages | ManKier:

–rsh=COMMAND, -e

This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync.  Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network.

If this option is used with [user@]host::module/path, then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host.  See the Using Rsync-Daemon Features Via a Remote-Shell Connection section above.

Beginning with rsync 3.2.0, the RSYNC_PORT environment variable will be set when a daemon connection is being made via a remote-shell connection.  It is set to 0 if the default daemon port is being assumed, or it is set to the value of the rsync port that was specified via either the –port option or a non-empty port value in an rsync:// URL. This allows the script to discern if a non-default port is being requested, allowing for things such as an SSL or stunnel helper script to connect to a default or alternate port.

Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument.  You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes).  Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing).  Some examples:

-e 'ssh -p 2234'
-e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'

(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

See also the –blocking-io option which is affected by this option.

This way I could pass the --login parameter to ssh in order to force a login session, however the ESXi BusyBox ssh implementation does not support that parameter:

unknown option -- -
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface]
[-b bind_address] [-c cipher_spec] [-D [bind_address:]port]
[-E log_file] [-e escape_char] [-F configfile] [-I pkcs11]
[-i identity_file] [-J [user@]host[:port]] [-L address]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-Q query_option] [-R address] [-S ctl_path] [-W host:port]
[-w local_tun[:remote_tun]] destination [command]

I sidestepped the fifth query wondering if there might be a different shortcut to set server side environment variables from a local ssh. There wasn’t.

Similar with the sixth query: I wondered if there might be another approach specifying the remote rsync location other than I already found. There wasn’t. Either rsync is on the PATH, or you can use --rsync-path unless you have a very rare situation where it is in the path and still need to use --rsync-path (like in [Wayback/Archive] rsync requires rsync-path when remote rsync is in path – Server Fault, thanks [Wayback/Archive] Paulo Matos and [Wayback/Archive] SmallClanger) which – thanks to the ubiquity of package managers – hardly occurs any more.

Almost final remarks

This error is not limited to ESXi, for instance it also occurs on OpenELEC (and likekly LibreELEC): [Wayback/Archive] sh: rsync: not found · Issue #2539 · OpenELEC/OpenELEC.tv.

Of course rsync over a network is way slower than syncing locally (and a VPN usually even slower), it was great though being able to transfer some 100’s of gigabytes of data this way.

Since this way of running rsync remotely depends on ssh (the alternative is having an rsync daemon, but, though faster, is insecure as it has no means of encryption, see below) it is important to secure this as much as possible hence the section on using keyfiles with ssh.

rsync in daemon mode

Warning: don’t do this unless you can guarantee the network connection between local and remote machine to be absolutely secure.

In case you insist, you have to manually run rsync on the remote server in daemon mode (see rsync --daemon --help on how to do this) or go to hoops in order to run it as a service (I didn’t even investigate that, but you want to then start looking at /etc/init.d/SSH).

• Queries:
  • [Wayback/Archive] esxi rsync daemon mode – Google Search
  • [Wayback/Archive] rsync daemon mode security – Google Search
• Results:
  • [Wayback/Archive] ESXi 5.1 and rsync – damiendebin.net.
  • [Wayback/Archive] ESXi service description for rsync in daemon mode. has the firewall settingsfile to put into /etc/vmware/firewall/rsyncd.xml:
    

    <ConfigRoot>
        <service>
            <id>rsync</id>
            <rule id="0000">
                <direction>outbound</direction>
                <protocol>tcp</protocol>
                <porttype>dst</porttype>
                <port>873</port>
            </rule>
            <rule id="0001">
                <direction>outbound</direction>
                <protocol>udp</protocol>
                <porttype>dst</porttype>
                <port>873</port>
            </rule>
            <enabled>true</enabled>
            <required>false</required>
        </service>
    </ConfigRoot>

  • [Wayback/Archive] ssh – How to perform a secure rsync between servers across an unsecured network – Server Fault (thanks [Wayback/Archive] khosrow and [Wayback/Archive] Dave Cheney): this is why rsync supports ssh.
  • [Wayback/Archive] Running rsync as a daemon

ESXi and keyfiles for ssh

[Wayback/Archive] ssh – Specify identity file (id_rsa) with rsync – Unix & Linux Stack Exchange (thanks [Wayback/Archive] Jangari, [Wayback/Archive] Dan Garthwaite and [Wayback/Archive] Cbaker510).

If you insist on not using keyfiles on ESXi then the only options are to either enter them manually or store the password in a file.

For Linux local machines, you can use sshpass.

Any solution based on this is a terrible idea from a security perspective. I might cover more about this in a lar post.

Potential alternatives

I want to check these out at a later stage:

• [Wayback/Archive] Solved: rdiff-backup in esxi 4 – VMware Technology Network VMTN
• [Wayback/Archive] How to Secure Rsync | UpGuard
• [Wayback/Archive] bcpierce00/unison: Unison file synchronizer

Related

Blog post: There has been an ESXi static rsync 3.1.3 binary for a while now (and why I want a much newer version)

–jeroen