Prevent people from reverse-engineering my network protocol

Question

I know I cant prevent people from reverse-engineering my protocol but I'd like to take a security-through-obscurity approach to make it as hard as possible.

I have a server/client system that communicates through the network with http style packets.

Example:

Header
Attribute: Value
Attribute2: Other Value

Payload

I would like to make it as hard as possible for anything other than my client to access the network. Pushing problems with them decompiling my assemblies aside - what would be some good things I could do to this network spec that would make it VERY DIFFICULT to understand and make another implementation without the source?

I was thinking some kind of strange hashing approach or some kind of encryption algorithm that would be difficult.

EDIT I'm not trying to protect my assemblies or source-code. I'm trying to prevent someone from, for example, watching my protocol with WireShark or similar and then making their own implementation based on that information.

Answer 1

All right, three cases:

• Users can't access server code and can't access client code: Easiest way is to use a pregenerated shared secret stored in the binary, and aes encrypt/decrypt.

• Users can access client or server code but not both: Use a public/private key method. You can encrypt using the public key but the private one is needed to decrypt.

• Users can access both client and server code: You're screwed.

If you want to improve security, this static key should only be used during session initiation, to generate a new shared secret, which is then used for communication.

Edit: actually, a more easy and safe solution is to use ssl and certificates (it's a mantra that you shouldn't implement your own encryption) Each certificate comes with a secret private key. As long as users don't have access to that you're safe if you verify that the peer has the exact correct certificate.

Answer 2

For having reversed a few network protocols (from MMOs), I can tell you that you will never protect your protocol for very long, I'm sorry.

The best you can do is:

• Obfuscate it using a custom algorithm (because it takes longer to reverse than a known one). Using a known encryption scheme offers no protection whatsoever.
• Add noise. Try to be very, very confusing. Add random values that make no sense whatsoever. Try to use a dynamic layout for packets. Move fields. Send useless packets. Just like if it were garbage.
• Version your protocol, so that two consecutive version are incompatible. That can be hard to do, but it obliges the reverser to re-do the work for every subsequent version.

But these are just ways to slow down attackers. It's certainly not going to stop them.