Which attacks are possible concerning my security layer concept?

Question

Despite all the advices to use SSL/https/etc. I decided to implement my own security layer on top of http for my application... The concept works as follows:

User registers -> a new RSA Keypair is generated
the Private Key gets encrypted with AES using the users login Password
(which the server doesnt know - it has only the sha256 for authentication...)

Server stores the hash of the users password
 and the Encrypted Private Key and Public Key

User logs in -> authenticates with nickname+password hash
(normal nick/password -> IP-bound sessionid authentication)
Server replies: sessionid, the Encrypted RSA Private Key
    and an Encrypted randomly generated Session Communication Password

Client decrypts the RSA Private Key with the users Password
Client decrypts the Session Communication Password with the RSA Private Key

---> From this point on the whole traffic gets AES-encrypted
     using that Session Password

I found no hole in that chain - neither the private key nor the login password get ever sent to the server as plaintext (I make no use of cookies, to exclude the possibility of the HTTP Cookie header to contain sensitive information)... but I am biased, so I ask - does my security implementation provide enough... security?

Answer 1

Why does everyone have to come up with their secure transport layer? What makes you think you've got something better than SSL or TLS? I simply do not understand the motivation to re-invent the wheel, which is a particularly dangerous thing to do when it comes to cryptography. HTTPS is a complex beast and it actually does a lot of work.

Remember, HTTPS also involves authentication (eg: being able to know you are actually talking to who you think you are talking to), which is why there exists a PKI and browsers are shipped with Root CA's. This is simply extremely difficult (if not impossible) to re-invent and prone to security holes. To answer you question, how are you defending against MITM attacks?

TLDR: Don't do it. SSL/TLS work just fine.

/endrant.

Answer 2

I'm not a crypto or security expert by any means, but I do see one serious flaw:

There is no way the client can know that it is running the right crypto code. With SSL/TLS there is an agreed upon standard that both your browser vendor and the server software vendor have implemented. You do not need to tell the browser how SSL works, it comes built in, and you can trust that it works correctly and safely. But, in your case, the browser only learns about the correct protocol by receiving plain-text JavaScript from your server.

This means that you can never trust that the client is actually running the correct crypto code. Any man-in-the-middle could deliver JavaScript that behaves identically to the script you normally serve, except that it sends all the decrypted messages to the attacker's servers. And there's no way for the client to protect against this.

That's the biggest flaw, and I suspect it's a fatal flaw for your solution. I don't see a way around this. As long as your system relies on delivering your crypto code to the client, you'll always be susceptible to man-in-the-middle attacks. Unless, of course, you delivered that code over SSL :)

Answer 3

It looks like you've made more complexity than is needed, as far as "home-grown" is concerned. Specifically, I see no need to involve assymetric keys. If the server already knows the user's hashed password, then just have the client generate a session id rolled into a message digest (symmetrically) encrypted via the client's hashed password.

The best an attacker might do is sniff that initial traffic, and attempt a reply attack...but the attacker would not understand the server's response.

Keep in mind, if you don't use TLS/SSL, then you won't get hardware-accelerated encryption (it will be slower, probably noticeably so).

You should also consider using HMAC, with the twist of simply using the user's password as the crypto key.