Read a special stream without knowing the number of bytes

Question

Before you go and say use Jon Skeet's answer, I can't. I am using this stream which is great in the beginning, but now, I have run into a small problem. I am now at a point where i do not know the amount of data I expect, so I thought well I'd use the method Jon Skeet provided but unfortunately it waits on packet = mPackets.Take() when all the packets have been read.

I have no idea how to fix this, the stream does what it's supposed to do (in the beginning) but unfortunately this is an occasion where i do not want it to do what it was made for...

Should I try to rewrite the waitingstream or is there a different readall method which would work better in my case?

I have tried returning 0 when there were no more packets available(when I wanted it to read all and not wait) but then it throws an IO end of stream exception.

I am really at a loss here, I understand I've dug a deep hole for my self...

So far i have not been able to come up with a better solution than subtracting the TLS header and GCM IV and MAC from the total length (like mentioned below) This works fine for the 1 suite i'm using now but does not boast well for future cipher suites.

---

Some context as to why I'm doing this:
I'm using https://github.com/bcgit/bc-csharp to create a TLS based encryption, however Because of limitations i cannot assign it a TCP stream, all i get is a string from the TCP layer, I cannot provide it with a propper TCP stream.

I am using the mockPskTlsServer and client and I've created it like so:

WaitingStream mStdin = new WaitingStream();
WaitingStream mStdout = new WaitingStream();
CryptoPskTlsClient mServer = new CryptoPskTlsClient(null);
SecureRandom secureRandom = new SecureRandom();
TlsClientProtocol TlsProtocol = new TlsServerProtocol(Stdin, Stdout, secureRandom);
TlsProtocol.Connect(mServer);

This works fine, I've also added an event to the waiting stream which fires when there's output for the stream. This so I could get the multiple handshake messages when they were created.

This all works fine, could have been made better (less.... hacked?) but it does what it's supposed to do.

I do know of ways to get it to work but this is a HACK and i'm looking for a elegant way to do it. Here is a code example for one of those HACKS:

public override string DecryptData(string ciphertext)
{
    byte[] ciphertextBuff = ASCIIEncoding.Default.GetBytes(ciphertext);
    mStdin.Write(ciphertextBuff, 0, ciphertextBuff.Length);

    Stream tlsStream = mServerProtocol.Stream;
    byte[] plaintextBuffer = new byte[ciphertextBuff.Length - 29];//filthy HACK 29 bytes bij AES-GCM want TLS packet = 5, GCM IV = 8, GCM-MAC = 16 totaal = 29.
    Streams.ReadFully(tlsStream, plaintextBuffer);

    string plaintext = ASCIIEncoding.Default.GetString(plaintextBuffer);
    return plaintext;
}

or by prepending the plaintext lenght in x reserved bytes to the TLS packet and retrieving these bytes before decrypting.

but as you can clearly see from the code example the buffer in the read fully method must be the length of bytes I want to extract, it can be less than what the stream has but it cannot be more because then it will wait indefinitely.

When i'm talking about methods like ReadAll and ReadFully I mean these methods

Answer 1

all i get is a string from the TCP layer

This is surely the fatal flaw in your attempts to make this work. Implicit in what you are trying to do is that the data sent across the TCP connection is TLS encrypted. Which produces highly random byte values, any value between 0 and 255 is possible. The parcel of data you'd receive from a socket's Read() call is byte[], not a string.

Converting byte[] to a string requires using the .NET Encoding class. There are many varieties of it, the stock ones are ASCIIEncoding, UnicodeEncoding, UTF8Encoding. And a bunch of custom ones designed to work with a specific code page, you get them by using the Encoding(int codePage) constructor.

We don't know what Encoding is used by the "TCP layer" you need to work with, UTF8 would be likely, but it is a given that it will always produce corrupted data when it is asked to convert a byte[] with encrypted content. Some byte values don't have a corresponding Unicode codepoint. The encoding object punts that kind of problem by generating a ? or a � as a substitute character. Whatever the value of the Encoding.EncoderFallback happens to be. Some odds that you can see them in the debugger, albeit that the frequency is unpredictable.

In other words, the encrypted data will inevitably get corrupted by these substitutions. There is no workaround for this, nothing you can do to recover the original byte[] since content was lost by the substitutions. This must be tackled by the lower layers, they either need to encode the byte[] content to a string that can always convert (base64 is the standard solution) or it needs to stop converting the data to a string. You really do need the raw byte[] to make this work.