I just had a very disturbing conversation with a Rackspace Cloud CSR. It went something like this:
CSR: Can I have your account user name and password?
Me: You want my password?
CSR: Yes sir.
Me: You know that's, like, security 101 that you should never reveal a password over the phone?
CSR: Yes sir, but in this case we need it to verify your account.
Me: OK, let me go change it to something I'm willing to tell you over the phone.
[Typety type type]
Me: OK, my password is now somereallylongbogusthing.
CSR (without any delay): Thank you. How can I help you?
Me: Wait, you must either be the world's fastest typist, or you can see my password on your screen.
CSR: That's right, sir, I can see your password.
Me: (The sound of my jaw hitting the floor.)
I am just stunned. I have used Rackspace for mission-critical servers in the past. They have always seemed reasonably competent, if not always 100% reliable. But this calls into question Rackspace's entire security policy. The first rule of computer security is that you do not store passwords in the clear. Never. Ever. No ifs ands or buts. You Just Don't Do That. And security is particularly critical in cloud computing, where your data ends up on hardware that can be reused by other people. If Rackspace is storing passwords in the clear, what else might they be screwing up? This really calls into question whether Rackspace can be trusted with mission-critical data.
Good grief, Rackspace, I really wanted to like you. But what were you thinking?
[UPDATE:] It really is as serious as I thought. WIth the account password you can change contact information and reset the root password on all your servers. So unless and until this is fixed you should not use RSC for anything mission critical. I hope they do fix this because other than that I really like RSC. Their UI is very well designed, and setting up a server was amazingly fast and painless.
This comment has been removed by the author.
ReplyDeleteI am sure every CSR can be trusted with the most valuable access info!
ReplyDeleteAt least it is more robust than their DDoS defenses (none to speak of!)
I've been a rackspace customer for a while, and I've only ever used that password to verify my identity over the phone. A different password is used to log into the web interface to access data and tickets.
ReplyDeleteEven so, I did find that whole password verification thing unsettling. Fortunately, as pointed out, it's easy to update, and you can choose your own verification question.
unencrypted password ok what the fuck is the point. domenick, umm..
ReplyDeleteIf you send a message to http://twitter.com/EdLeafe with whoever it is you spoke to, he can sort it out I'm sure. $10 says its a clueless CSR somewhere along the line.
ReplyDeleteMediaTemple does this exact same thing...
ReplyDeleteI find it funny that we have to use the phrase 'security 101' -- I mean .... 10+ years ago when I was into security we'd laugh at this stuff -- but it is STILL happening -- wtf!??! these people need to be shot -- who gave them their credence -- oh, crap, we did by paying them to use their servers ;) you vote with your dollars everyone!
ReplyDeleteAs a former hosting support tech for several firms, large and small, I can testify that this is actually common practice for most dedicated server providers, especially those with some sort of managed service. One of my employers actually used SSH keys and an AD domain instead of requiring root/admin passwords to be kept up to date, but this arguably creates even more security concerns than it resolves. Credentials are needed by tech support in order to resolve most issues, so rather than communicating the password plaintext each time assistance is needed, a filed password for an admin-level account is common practice.
ReplyDeleteI'd be interested in some discussion on alternatives to this model, apart from colocating your own hardware in a locked cabinet/cage, of course :)
This was surprising. In PHP it's simple enough to do:
ReplyDelete$sPrivateKey = 'a2238Fyj';
$sPasswordToEncrypt = 'monkey1';
$sShadow = md5($sPrivateKey . $sPasswordToEncrypt);
...and then store $sShadow in the database. When someone logs in, I take whatever they typed, run it through the md5() with the private key I always know, and then compare the two shadows to see if they are the same. If so, the user can login and I can use a session var to track them through page to page as an authenticated user.
I, too, can attest to the fact that this is common practice in the hosting world. We don't use it for verification purposes, though, and the customer is under no obligation to keep the record up-to-date. It's there for them to put in if they want us to be able to log in to their servers and fix shit.
ReplyDelete@volo - not good enough these days... MD5 is not up to storing useable length passwords securely these days.
ReplyDeletehttp://codahale.com/how-to-safely-store-a-password/
PUHLEEEZ d00dz. RESET to random is only possible option when talking about passwords. Either it matches the hash or it doesn't, but there should be no way to see it for any reason. I hope that this was a mistake. Has rackspace responded yet?
ReplyDeleteWhat the hell is wrong with you? Of course Rackspace keeps your password on file but your an idoit if you think its cleartext. I'm sure its stored crypted in a DB. You call in and they ask for you customer id, then query the account information and ask you to verify it.
ReplyDeleteHOW ELSE DO YOU EXPECT SURPPORT? Would you rather email them the password or try to spell it out over the phone for 30mins during an emergency?
You may be competent but most other customers are your mom or dad fyi.
> You may be competent but most other customers are your mom or dad fyi.
ReplyDeleteFor a cloud server? I highly doubt that.
> I'm sure its stored crypted in a DB.
It doesn't matter if it's encrypted if every CSR has access to the decryption keys. And passwords should not be stored even encrypted. They should be hashed. With BCrypt.
> HOW ELSE DO YOU EXPECT SURPPORT?
I expect my CSR verification credentials to be separate from my account password.
> If you send a message to http://twitter.com/EdLeafe with whoever it is you spoke to, he can sort it out I'm sure. $10 says its a clueless CSR somewhere along the line.
ReplyDeleteI actually spoke to two CSRs and they both confirmed that they could see my password in clear text on their screens. Also, ironically, both CSRs seemed sympathetic and to understand why this was a bad thing, but of course they were powerless to do anything about it. So this is clearly not a CSR screwup, it's company policy. If it were just a CSR screwup I would not have blogged about it.
As a support engineer at a managed hosting company, I can say with confidence that we never store customer passwords in plain text. Period.
ReplyDeleteThere are multiple passwords used:
ReplyDelete1. account verification password
- this is used to verify you're an active client of the company. this password is not associated with server logon credentials.
2. account access password
- if this is different than your account verification password, this verifies that support techs cannot login and impersonate a client.
3. server password(s)
- these are passwords that shouldn't be known to hosting service providers unless they need to troubleshoot. You usually communicate the password to the tech via a secured ticket system. you should change it after the issue is resolved.
Most hosting service providers save the password in a way that allows CSRs/Support techs to access it instantly. The password may or may not be encrypted. They do not store a 1-way hash of this password as it would be useless in a support scenario.
There is nothing alarming about them storing a password in "plain text" as you say. It was probably encrypted in the database and decrypted before it is presented to the technician.
FYI - I work at one of the many hosting service providers in North America.
> There is nothing alarming about them storing a password in "plain text" as you say. It was probably encrypted in the database and decrypted before it is presented to the technician.
ReplyDeleteFrom a security point of view there is essentially no difference between storing passwords in plaintext and storing them encrypted along with the encryption key.
RADIUS requires plain text passwords, so if Rackspace have that anywhere in their architecture they have no choice but to have plain-text passwords.
ReplyDeleteThis drives me crazy. One of the biggest trading sites in Australia, Commsec, has the most amazingly lax attitude to security. Every time I call up to sort something out they ask me my password over the phone... seriously, a TRADING site, dealing with people's life savings, not understanding security 101. It blows my mind. So every time I call them I have to change my password as soon as the phone call is over, not that it matters apparently, since they all have access to our plaintext passwords. Makes me angry just thinking about it.
ReplyDeleteReally?
ReplyDeleteI mean, really?
You honestly think that your passwords shouldn't be viewable in a human readable format to the CSR's you're calling for SUPPORT?
First off, Cloud Servers root passwords aren't available to support reps. Ditto for Database passwords.
1. Account password - this is to login to your control panel. OF COURSE THEY CAN SEE THIS.
2. Cloud Server root password. They CANNOT see this.
3. Database passwords. They CANNOT see this.
Secondly, what did you expect on a shared system? They OWN the hardware! Did you seriously believe you have 100% obfuscation capability on their own hardware. If so, check yourself into an asylum for delusions of competency.
OF COURSE CSR's can see your account password. This is the very basic fundamental reality of a managed service!!!!
Get a clue dude.
One you have a clue, go get yourself a colo box and stop complaining.
Your password is stored encrypted, they can view it in the CSR system. Duh! Who cares, if they wanted to get into your account they would just change your login credentials.
ReplyDeleteI can also confirm MediaTemple does the same thing as well. I was just as shocked when I first encountered it.
ReplyDeleteOh jeez, I moved from MediaTemple to Rackspace because of MediaTemples' stupidity. le sigh...
ReplyDeleteNot that big of a deal. Homeboy did call an initiate the phone call, so what were they suppose to do? Take your word that you were whom you said you were? Would that be more secure? I think perhaps this was an over reaction...
ReplyDeleteIt happens to the best of us at one point or another. We will let our emotions get the best of us, thus the emotions take over and your brain ceases to control your actions.
In response to GFuzz's comment
ReplyDeleteIf CSR's can't be trusted then we are all screwed...
"You honestly think that your passwords shouldn't be viewable in a human readable format to the CSR's you're calling for SUPPORT?"
ReplyDeleteAbsolutely. No support staff should ever, EVER see a password. There are plenty of other secure ways to verify the individual, such as last 4 of credit card (much, much more secure than passwords), address/etc, even a callback to the number on file, all very good ways of verification.
using password verification for this stuff is insane, and allowing CSR's to read passwords is even worse.
Of course, this is probably pretty low on the totem pole for rackspace screwups and vulnerabilities over the years. For a 'managed service provider', they're not very managed at all. I'd bet I could go into certain servers on there and find RHEL1, or pre-RHEL OS'es floating around there.
To clear up some advice in the comments:
ReplyDeleteMD5(secret.pass) is better than nothing, but if you use MD5, it's better to use MD5 HMAC or nonce.MD5(nonce.pass). When hashing all passwords with same salt, add salt after the password. Hashes are chainable and hash state of salt could be precomputed.
MD5 collisions don't help in cracking of hashed passwords, so known MD5 weaknesses are irrelevant (you want preimage, not collision). For passwords MD5 is as good/poor as it always was. SHA1 isn't improving it much, because it's still too fast (fast = quicker brute-force). For password hashes you want slowest algorithm you can get. For this *bcrypt* is best (slower by configurable orders of magnitude).
tl;dr: bcrypt
Of course even poorest hashing is much better than showing plaintext passwords to everybody in the call center!
Pingdom.com does the same.. =/
ReplyDeletehttp://news.ycombinator.com/item?id=1118012
You have observed that a CSR can read your password. But in order to establish that is a security problem, you also need to prove the CSR's job requirements do not allow equivalent access some other way.
ReplyDeleteThis is a more subtle argument than it may appear at first. Consider that if a CSR is allowed to update your user profile, including your email address, and the CSR also is able to reset your account password if you forget it, then obviously a malicious CSR will be able to obtain your password.
The malicious CSR can temporarily set your email address to an address owned by the CSR, reset your password, then read the password in the password-reset email.
Now you may argue this method of obtaining the password is more detectable than simply reading it off the screen, because it involves changing the password.
But that's a whole separate argument with its own set of nuances. Ultimately you have to trust the CSR with a level of privilege equivalent to having your password.
> in order to establish that is a security problem, you also need to prove the CSR's job requirements do not allow equivalent access some other way.
ReplyDeleteThe ability to read a password is a much more serious security risk than merely being able to access the system. A CSR who wants to use their privileged access to hack my system has to weigh the risk of getting caught (because their actions might -- and hopefully are -- logged somewhere). But if they can see my password then they can write that password down and access my account from any internet cafe in the world. They can even mail that password to an accomplice in Romania, whereupon the risk of getting caught is virtually zero. In fact, the risk of getting caught if they write my password on a post-it and sell it to an overseas hacker are so small that it is virtually *inevitable* that this will happen sooner or later if people start to store sensitive data on RSC.
> A CSR who wants to use their privileged access to hack my system has to weigh the risk of getting caught (because their actions might -- and hopefully are -- logged somewhere).
ReplyDeleteOK, so in this threat analysis, we are accepting that the possibility of getting caught is a useful deterrent.
> But if they can see my password then they can write that password down and access my account from any internet cafe in the world.
Right, but you can change the passsword after any customer service call. Assuming you make a habit of doing that, the malicious CSR is going to have to access your account information outside the context of a customer service call in order for the attack to work.
Here is where the analysis gets difficult, because it isn't clear what policies and procedures exist to ensure that CSRs can't pull up customer account information without accountability.
On the one hand, if a CSR can pull up all the account information for any account at any time without anyone knowing, then I think everyone would agree there is a serious problem. But I think the password is probably not the largest concern. Of more immediate value to an attacker is probably the billing information, which may contain a credit card number.
On the other hand, if a log entry is generated every time any CSR pulls up any customer account, and the CSR is aware these logs are being kept, then we have a proper deterrent.
> OK, so in this threat analysis, we are accepting that the possibility of getting caught is a useful deterrent.
ReplyDeleteOf course.
> you can change the passsword after any customer service call.
You're joking, right? You really expect me to generate a new password every single time I call customer service? Do you really not understand why that is a Really Bad Idea (tm)?
I'm going to guess that CSR also has access to change your password. If not, his supervisor does. They own the hardware. They can reboot the machine and with a recovery disk change everything.
ReplyDeleteI'm not saying it's a good policy, but at some point you decided to trust Rackspace, and they decided to trust their employees.