The tale of [SSH into ESXi 6.7 box resulting in “debug1: expecting SSH2_MSG_KEXDH_REPLY”, delay and after entering password “Permission denied, please try again.”]
Posted by jpluimers on 2021/04/02
A similar ESXi 6.5 box worked well to ssh into, but on ESXi 6.7 it failed:
SSH into ESXi 6.7 box resulting in “
debug1: expecting SSH2_MSG_KEXDH_REPLY“, delay and after entering password “Permission denied, please try again.“
I had a hard time figuring out why: Login with the same user+password on the web user interface, DCUI and console shell work fine (see [WayBack] Enable SSH on VMware ESXi 6.x – VirtuBytes).
Searches that led me to EBCAK:
- Source: ssh failing to “esxi 6.7” – Google Search
- Source: “ssh” “esxi 6.7” “Permission denied, please try again.” – Google Search
- Source: “ssh” “esxi 6” “Permission denied, please try again.” – Google Search
- Source: “ssh” “Permission denied, please try again.” – Google Search
It almost felt like the /etc/passwd file thought the user had an empty password, but in fact it did not.
Adding an AllowUsers clause to ESXi in /etc/ssh/ssd_config, then performing /etc/init.d/SSH restart failed as well, and should not be needed anyway (default is all users having a valid shell can login, including root as on ESXi, by default has PermitRootLogin yes) (via [WayBack] server – Permission denied please try again ssh error – Ask Ubuntu).
Setting LogLevel debug from LogLevel info in /etc/ssh/ssd_config did not change anything (not even after restarting sshd, or rebooting): it did not even add any more logging in /var/log/syslog.log or any of the log files under /var/log or /scratch/log.
Ruling out lock-down mode:
# vim-cmd vimsvc/auth/lockdown_is_possible
false
# vim-cmd vimsvc/auth/lockdown_is_enabled
falseSee [WayBack] New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 3 and [WayBack] HOW TO: Enable or Disable Lockdown Mode on VMware vSphere ESXi host | vStrong.info
Q: What is Lockdown Mode?
A: Lockdown Mode prevents users from logging directly to the host. The host will only be accessible through local console or vCenter Server. None of remote management options e.g. vCLI, PowerCLI script, SSH will work. When it is enabled, only vpxuser () has authentication permissions and can connect to the host remotely.
No password login also means no passwordless login
The above rules out easy uploading my public keys for doing passwordless login in [WayBack] ssh root@host – Permission denied, please try again. – Tarran Jones.
Delay annoyance
There is also an annoyance: it takes about 10 seconds before you can enter the password (adding -v -v -v reveals the wait is on debug1: expecting SSH2_MSG_KEXDH_REPLY).
Disabling/enabling SSH from the DCUI: not fully disabled
After disabling SSH from the DCUI, I could still connect over SSH.
So then I disabled the TSM-SSH service from the web interface (despite DCUI telling SSH was disabled, TSM-SSH was still active, strange!) as it hosts the SSH service. I could still perform my ssh command!
Then it occurred to me: the IP address in the web browser was one off from the IP address in my ssh command.
By sheer coincidence, the IPMI IP address was one lower than the LAN1 IP address. I had been ssh-ing into the IPMI interface all the time, never realising IPMI had support for the first place!
Restring the TSM-SSH service now suddenly did get me LogLevel debug output in /var/log/auth.log (backed by /scratch/log/auth.log and duplicated in /vmfs/volumes/<<ssd-volume>>/.locker/log/auth.log).
Learned three things
So learned three things the hard way:
- Be more careful with IP-addresses
- IPMI does ssh (but it is very undocumented)
- DCUI enable/disable of SSH is not complete; TSM-SSH is
Some references:
- [WayBack] How to Enable SSH in the VMware ESXi Embedded Host Client:
- the TSM-SSH service (SSH)
- TSM (ESXi Shell)
- [WayBack] View, Start and Stop Remote Tech Support Mode (TSM-SSH) on All ESXi Servers – PowerCLI Get-VMHostService – VMadmin.co.uk
- [WayBack] Accessing Supermicro IPMI via SSH – Server Fault
- [WayBack] Supermicro IPMI Power On Server via SSH – ByteSizedAlex
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply