Haven't really got a chance to catch up with my emails until this morning and saw this SolusVM drama during the weekend:

As many of you are aware, we were attacked with a zero day SolusVM exploit early this morning. Several of our VPS nodes were partially or fully wiped within two hours of SolusVM publishing the exploit. We are currently working to restore full functionality to our SolusVM CP, our Client Area, and all impacted nodes. If we have a backup of your VPS, we will be restoring it as soon as possible. For those without backups, you will be able to reinstall the OS once we enable SolusVM access.

As part of the exploit, our SolusVM database was briefly leaked. If your VPS is online, you should change your root password over SSH. Any passwords that were created or changed in SolusVM itself may be compromised. No Client Area data was compromised as part of this attack.

Please monitor our Twitter accounts (@RamNode and @NodeStatus) for updates. You can also stay up to date in our IRC channel (#ramnode on irc.esper.net).

Thanks for your continued support and prayers as we deal with this extremely difficult trial.

Thanks,

Nick

From what I can tell from this post on LEB, basically Robert saw this exploit being posted and he tried it against RamNode (or potentially several providers) and got "lucky" with RamNode.

First, I'd say I hate what he has done. In fact, my SolusVM master for my test server for the VPS business is hosted with RamNode, and it sucks to see all these happening.

However, on a second thought, should providers be more active to things like this? In fact, SolusVM has this automatic script to update the installation, set up a cron with it and the system will always be up-to-date.

I have seen at least 2 more providers on Twitter who were shutting down their SolusVM...presumably because they saw what happened to RamNode rather than they saw the exploit was posted and people could use it. I just hope that providers would be able to, somehow, keep their installations up to date and if anything happens, shut down the panel until a fix was posted rather than keep it on hoping nothing bad will happen.

Just my 2 cents.

---

Just received an update from Nick:

We have opened the SolusVM CP again with the appropriate security patch. To briefly explain what happened: SolusVM released a zero day exploit early on Sunday morning, EDT. Within a few hours, another host decided to run the exploit against our installation. Shortly thereafter, someone (perhaps multiple persons) logged into our SolusVM administration panel, stole the database, and deleted several nodes worth of VPSs. Most of our nodes were unharmed, but the damage was obviously significant enough to keep us busy trying to restore as much as possible over the past 24 hours. No intruders directly accessed the VPS nodes, and I have completely reinstalled the control panel itself to prevent malicious activity.

The following information was contained in the leaked database: first names, last names, email addresses, and SolusVM account information. No telephone numbers, street addresses, or billing information was compromised. Anyone who has not changed his or her VPS root password using SSH should change it promptly. You should also change your SolusVM password. We believe the attackers were simply out to leak our database and destroy as much as possible, not steal client information; however, anything that was changed in or generated by SolusVM (initial passwords, for instance) is potentially compromised. Again, this incident did not impact billing information (we do not store credit cards to begin with) and it did not impact the Client Area's integrity.

We still have a few restores available for ATLCVZ5 and several KVM nodes, but we opened the control panel since some of you need to reinstall your OS. Unfortunately, backups were not available in every situation. Please submit a ticket if you need a restore on a KVM node in particular and we will do what we can. If you're on ATLCVZ5, we might also have a backup for you. Almost all other OpenVZ nodes have been restored as much as possible. If your OpenVZ VPS is marked Offline, you'll probably have to reinstall the OS. You may of course submit a ticket with any restore or other requests, but please understand that we will not be able to respond with our usual quickness for a few days.

Lastly, please ignore any overdue invoice messages for now. Our Client Area backups were apparently glitching, so we lost a few days worth of invoices and tickets in the restoration process. As such, we'll have to manually enter payments over the next few days. We have disabled automatic suspension/termination until we are caught up. If you ordered a new VPS between June 12 and now, we will have to manually recreate your account (or the order itself if you already had a VPS with us). The VPS itself should still be in our SolusVM system regardless.

Thanks for all of the support we have received over the strenuous past 24 hours. We will continue working to ensure that everything is back to normal promptly so that you experience the same great service you've come to know and love at RamNode. If you'd like to keep up to date on both recent and future events, please join our IRC channel (#ramnode on irc.esper.net) and/or follow us on Twitter (@RamNode and @NodeStatus).

Nick
nick[at]ramnode.com

(This post was last modified: 06-17-2013, 11:18 PM by zhuanyi.)

I have a few of my more critical websites hosted on RamNode so as soon as I noticed my VPS was down I looked around to see what was up. I haven't had my RamNode VPS go down unannounced at all before this happened.

Nick did an amazing job of getting everything back up and running within just that day. The node that I'm on was up within a few hours of him announcing the problem on Twitter, and most of the others followed shortly.

Maybe software like SolusVM should have a built in feature that emails/texts/calls the admins every time there are any big zero day exploits, even if it's an add on that costs a little extra each month.

Maybe software like SolusVM should have a built in feature that emails/texts/calls the admins every time there are any big zero day exploits, even if it's an add on that costs a little extra each month.

Or they can just use the upcp script, scheduled via a cron, to update once or twice a day by itself. Personally I think that might make more sense.

Looks like everyone is in a panic mode as of this morning, after CVPS was hacked....I got at least 3 or 4 emails this morning telling me SolusVM from various providers are shutting down.

Looks like everyone is in a panic mode as of this morning, after CVPS was hacked....I got at least 3 or 4 emails this morning telling me SolusVM from various providers are shutting down.

It's crazy. On top of all that, there are likely going to be 10 new panels out in the coming months. I'm very curious to see which ones take off, and which ones are dead in 1 or 2 years. I welcome competition in the VPS control panel market as it pushes innovation and gets things done.

Looks like everyone is in a panic mode as of this morning, after CVPS was hacked....I got at least 3 or 4 emails this morning telling me SolusVM from various providers are shutting down.

It's crazy. On top of all that, there are likely going to be 10 new panels out in the coming months. I'm very curious to see which ones take off, and which ones are dead in 1 or 2 years. I welcome competition in the VPS control panel market as it pushes innovation and gets things done.

Agreed, I think this is a good thing because it will potentially raise the barrier of entry. Say for example, OnApp is good, but it is not cheap, and if everyone has to use that in the end, I would say we can almost completely eliminated the summer host problem.

My biggest concern with this industry right now is anyone with 20 bucks in his/her pocket could hit the road and start running a VPS business right away, without even properly securing even the most basic stuff. I bet some of the VPS providers still use root password on port 22 for SSH into the nodes.