I have a couple of FreeBSD VMs on Vultr. For quite some time sshd regularly dies on one of them and I am not sure why.
A couple of theories I’ve had is that maybe
a) my VM was compromised and there is a persistent rootkit installed that kills sshd, or
b) file corruption after previous unclean shutdown has left some file needed by sshd corrupted and it leads to this behaviour, or
c) maybe it’s running out of memory sometimes
Each time I want to ssh into the machine I usually have to first connect with the VNC from the vultr dashboard to start sshd up again.
It’s running the latest FreeBSD, as every now and then I log in and do an upgrade on it some time after a new version has been released.
A persistent rootkit may have been installed if it was compromised between when some vulnerability became known and when I later upgraded next time.
If a file was corrupted in an unclean shutdown in the past maybe it’s a file that has not been changed between FreeBSD versions so even though upgrades replace some files maybe it’s the same corrupted file all along.
Ideally I’d just reinstall the machine, but that’s always more of a hassle than it should be so I continue running the VM in this broken state where sshd keeps dying every now and then.
Well, you can at least run "freebsd-update IDS" on this system to verify the base system, "pkg check -s -a" would check the integrity of files installed via pkg also.
It will only take a few min..
Neat! Haven’t tried either of those before.
While I’m at it I also took a quick look now at output of `top` and it’s sitting at 27 MB free RAM lol. So from that, out of memory is very likely the reason I keep having sshd die on me.
Try adding this to your /etc/rc.conf:
sshd_oomprotect=YES
Then run
service sshd restart
Not familiar with BSD but does it not syslog oom kills like linux?
After rebooting the VM now, sshd died even when there was hundreds of megabytes free RAM available. So it seems I spoke too soon when I said it seemed to be for that reason.
Previously I haven't seen much detailed reason for why it dies in system messages. But this time it said something very specific:
> sshd[2036]: fatal: pack_hostkeys: serialize hostkey private: string is too large
Which kind of sounds like one of the sshd hostkey files might be corrupt? And maybe it only triggers after a while becuase it happens when scanners try to connect to it and during ssh negotiation sshd ends up selecting a different hostkey type than the one it uses when I connect to the machine myself?
I'm going to regenerate all of the three hostkey files on the server, and after that also disable the two that I can do without anyway.
I used buyvm.net for BSD a while back. Didn't have any issues, and the few times I contacted their support they were quick to respond and helpful.
rootbsd.net before that, but they don't seem to exist anymore.
They were very helpful when one of the AMD firmware patches in 7.3 caused my VPS to not boot. They even offered to apply a manual fix until patch -015 was available which fixed it.
The admins have access to your data and unless things have changed, that isn't monitored. They also bend to NSLs like any other provider. What they charge vs what things cost hurts.
Any cloud compute vendor will have access to your data unless you encrypt it.
> unless you encrypt it
And to clarify, this means encrypt it before it gets to the VPS. Just having full-disk encryption is not enough because cloud providers can dump RAM. There are tools that easily extract encryption keys from RAM.
So, really, you need to trust the cloud provider unless everything is encrypted on computers you own.