Using control character (\x1f) with string and/or StringBuilder

Question

I just want to use the ASCII Unit Separator character (decimal 31 and hex 1F) instead of a tab for a delimited file. I assume the problem is encoding but I sure can't find how to change it. In the following, I get the desired output on the console in the first line of output in my StreamWriter file but the second line is missing the '\x1f'.

static StreamWriter sw = null;

static void Main(string[] args)
{
    try
    {
        sw = new StreamWriter(OutFilename, false, Encoding.UTF8);
    }
    catch (Exception ex)
    {
        Console.WriteLine("File open error: " + ex.Message);
        return;
    }
    // This works
    Output("From▼To");  // Has a '\x1f' in it
    // This does not work
    StringBuilder sb = new StringBuilder();
    sb.Append("From");
    sb.Append('\x1f');
    sb.Append("To");
    Output(sb.ToString());
    //
    sw.Close();
}

static void Output(string s)
{
    Console.WriteLine(s);
    sw.WriteLine(s);
}

The output file has:

From▼To
FromTo

I want to build a string using StringBuilder except with the '\x1f' in the output.

Answer 1

Seems like there is a lot of confusion here. Let me see if I can clear things up somewhat.

First of all, let's agree on the following points that are easily verifiable:

'\x1f' == '\u001F'
'\x1f' == (char)31
'\x1f' != '▼' // <-- here appears to be your mistaken assumption.
'▼' == (char)9660
'▼' == '\u25BC'

So this...

// This works
Output("From▼To");  // Has a '\x1f' in it

... ironically is the exact line that does not work. There is no '\x1f' in this string. The triangle character is not '\x1f'. Not sure where you got that impression.

Which leads us to the last point: '\x1f' is not a visible character. So when you try to display it in the console, you will not see it, and that is 100% normal.

However, be assured that when you have a string with '\x1f' and write that out to a file, the character is still there. But you will never be able to "see" it, unless you read the bytes directly.

So whether or not you can use '\x1f' as a delimiter depends on whether you need the delimiter to be visible. If yes, then you need to pick another character. But if you only need it as a delimiter for when you programmatically parse the file, then using '\x1f' is appropriate.