The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 4,227 other subscribers

The tale of [SSH into ESXi 6.7 box resulting in “debug1: expecting SSH2_MSG_KEXDH_REPLY”, delay and after entering password “Permission denied, please try again.”]

Posted by jpluimers on 2021/04/02

A similar ESXi 6.5 box worked well to ssh into, but on ESXi 6.7 it failed:

SSH into ESXi 6.7 box resulting in “debug1: expecting SSH2_MSG_KEXDH_REPLY“, delay and after entering password “Permission denied, please try again.

I had a hard time figuring out why: Login with the same user+password on the web user interface, DCUI and console shell work fine (see [WayBack] Enable SSH on VMware ESXi 6.x – VirtuBytes).

Searches that led me to EBCAK:

It almost felt like the /etc/passwd file thought the user had an empty password, but in fact it did not.

Adding an AllowUsers clause to ESXi in /etc/ssh/ssd_config, then performing /etc/init.d/SSH restart failed as well, and should not be needed anyway (default is all users having a valid shell can login, including root as on ESXi,  by default has PermitRootLogin yes) (via [WayBack] server – Permission denied please try again ssh error – Ask Ubuntu).

Setting LogLevel debug from LogLevel info in /etc/ssh/ssd_config did not change anything (not even after restarting sshd, or rebooting): it did not even add any more logging in /var/log/syslog.log or any of the log files under /var/log or /scratch/log.

Ruling out lock-down mode:

# vim-cmd vimsvc/auth/lockdown_is_possible
# vim-cmd vimsvc/auth/lockdown_is_enabled

See [WayBack] New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 3 and [WayBack] HOW TO: Enable or Disable Lockdown Mode on VMware vSphere ESXi host |

Q: What is Lockdown Mode?
A: Lockdown Mode prevents users from logging directly to the host. The host will only be accessible through local console or vCenter Server. None of remote management options e.g. vCLI, PowerCLI script, SSH will work. When it is enabled, only vpxuser () has authentication permissions and can connect to the host remotely.

No password login also means no passwordless login

The above rules out easy uploading my public keys for doing passwordless login in [WayBack] ssh root@host – Permission denied, please try again. – Tarran Jones.

Delay annoyance

There is also an annoyance: it takes about 10 seconds before you can enter the password (adding -v -v -v reveals the wait is on debug1: expecting SSH2_MSG_KEXDH_REPLY).

Disabling/enabling SSH from the DCUI: not fully disabled

After disabling SSH from the DCUI, I could still connect over SSH.

So then I disabled the TSM-SSH service from the web interface (despite DCUI telling SSH was disabled, TSM-SSH was still active, strange!) as it hosts the SSH service. I could still perform my ssh command!

Then it occurred to me: the IP address in the web browser was one off from the IP address in my ssh command.

By sheer coincidence, the IPMI IP address was one lower than the LAN1 IP address. I had been ssh-ing into the IPMI interface all the time, never realising IPMI had support for the first place!

Restring the TSM-SSH service now suddenly did get me LogLevel debug output in /var/log/auth.log (backed by /scratch/log/auth.log and duplicated in /vmfs/volumes/<<ssd-volume>>/.locker/log/auth.log).

Learned three things

So learned three things the hard way:

  1. Be more careful with IP-addresses
  2. IPMI does ssh (but it is very undocumented)
  3. DCUI enable/disable of SSH is not complete; TSM-SSH is

Some references:


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: