Remote access for ESXi local user account 'root' has been locked for 120 seconds after xxx failed login attempts.At the same time he could no longer log in to the vSphere client although he was using the correct password. I immediately thought: Wow, this is new in 6.0, never heard of this before! What the heck was going on here?
Well, I just RTFM (read the fine manual) and quickly found out what was going on (yes, VMware's public product documentation is pretty good, highly recommended!).
In the section ESXi Passwords, ESXi Pass Phrases, and Account Lockout of the ESXi and vCenter Server 6.0 Documentation you can find the following information:
ESXi Account Lockout Behavior
Starting with vSphere 6.0, account locking is supported for access through SSH and through the vSphere Web Services SDK. The Direct Console Interface (DCUI) and the ESXi Shell do not support account lockout. By default, a maximum of ten failed attempts is allowed before the account is locked. The account is unlocked after two minutes by default.
You can configure the login behavior with the following advanced options:
- Security.AccountLockFailures. Maximum number of failed login attempts before a user's account is locked. Zero disables account locking.
- Security.AccountUnlockTime. Number of seconds that a user is locked out.
And indeed this is new in version 6.0!
In this specific case it turned out that the host was directly connected to the Internet and had shell access via ssh permanently enabled. Of course this is an invitation to hackers all over the world, and they soon and repeatedly tried to break into the ESXi host using brute force. The new account lockout feature helps to mitigate this kind of attack, but it will also prevents legitimate access to the system like in this case. It is important to understand that not only ssh access, but also access to the host via the vSphere client or API calls is prevented when the root account is locked out!
This case reminds us of some basic security rules and best practices for ESXi that are mandatory for production use, but should also be honored for home and lab use:
1. Connect the host management interface to a secured internal network, but not to the public Internet!
2. Enable ssh access only temporarily for troubleshooting, but do not keep it permanently enabled!
And I like to add the following recommendations, especially for cases where you have no other choice than connecting the host directly to the Internet:
3. Create a timeout for ESXi shell availability and for idle ESXi shell sessions. This way the ssh access will automatically be disabled again after a configurable timeout, and forgotten idle sessions will automatically be terminated.
4. If you frequently enable ssh access to your hosts for whatever reason then consider using key based ssh authentication and disable password log in. For further information see KB1002866 and the instructions on Uploading an SSH Key Using a vifs Command.
5. Use the ESXi builtin firewall to limit access for at least ssh (port 22) and the vSphere client (port 443) to known trusted IP addresses. You can read my earlier blog post about Protecting your ESXi hosts against Heartbleed attacks for a practical scenario and examples.
I hope you will find this information useful - stay safe!
This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz.
Follow him on Twitter to keep up to date with what he posts.
Follow him on Twitter to keep up to date with what he posts.
Useful addition to your article:
ReplyDeletepam_tally2 --user root --reset
That will reset the lockout, assuming you have ssh key access to login.
Nice. Thanks for sharing, Nathan!
DeleteI wonder if WinSCP or another SSH-based file transfer tool could use an ssh key? For those of us trying to automate vmdk backups on the free editions in our labs.
ReplyDeleteHi Justin,
Deleteyes, that's possible. See KB1002866 for configuring key based ssh access on ESXi.
Andreas
This was seriously helpful! Thanks Man!
ReplyDeleteThanks a lot, i disabled the ssh now and also going to create a superuser other then root if such happening again.
ReplyDeleteRegards
Ranvir Sharma
http://www.jsmtp.com
It happened in our environment running VMware 6.0 HPE customized image for HP DL 380 server. We use N-Central monitoring services and also to backup VMware infrastructure we use shadow protect. We disabled monitoring. Next step was to disable all backup jobs. We left the system for like 10 to 15 minutes and then tried again. It worked this time with root credentials. I saw in events that there were 7792 failed login attempts from user "root" and it may be because of several login failures from our shadow protect backup solution. We have now created a separate user for backup jobs. I read somewhere that multiple logons with same user can also end up account being locked.
ReplyDeletethank you.. was going nuts here
ReplyDeleteThanks for the info. It isn't just rogue SSH attempts which can cause this entry, we recently experienced the same thing using Dell OpenManage where the root password had changed and it triggered this kind of behavior.
ReplyDeleteVcenter 6.5 - root account lockout
ReplyDeleteRan into this issue where the root account was locked for 900 seconds after 1111 failed login attempts. It turned out that a security scan attempted to login to the server 5 times and failed. This locked the root account. The Vcenter server continually logins into the ESXi every 5 minutes but the account was locked. This is where the 1111 logins came from.
Each attempt by the Vcenter to login to the ESXi delayed the lockout for another 900 seconds resulting in a never ending loop.
Workaround -
From Vcenter --> Host --> Configure --> System --> Advanced System Settings -->
Change - Security.AccountLockFailures to a value of 0 to disable account lockout.
Wait 900 seconds and login to the ESXi are root.
After you get back into the ESXi, it's a good idea to review the security/user policies to determine how to prevent this. One obvious way is to disable the account lockout.