C# - Text Encoding question

[evilertoaster]

Is there a reason you can't just write a byte array to the file rather than a string?

How do you get a byte array from a StringBuilder object?

[Richard Robertson]

You don't. I meant skipping string builder altogether.

You would manually encode any runtime defined string using your encoding of choice and for any literals, just use bytes. Write these to a file the good old fashioned way.

Edited November 15, 2010 by Richard Robertson

[evilertoaster]

hum, so I'd basically have to manually manage my byte arrays (resizing ect) when I append to them (or just allocate a safe upper-bound static one)...

I don't know if it would do very well performance wise...In the actual code I'm enumerating though the results of a very large result set of a select * in SQL, then for each column of each row I do some string manipulation (which the String object's built in methods are of great help) and format an output line to the text file I'm writing...

Sounds like alot of refactoring just to fix a supposedly basic encoding problem... I thought there'd by a better way to handle this with .NET...

[evilertoaster]

Well, I can't argue with the results, a simple wrapping class allowed me near drop-in replacement for StringBuilder methods I was using:

class ByteBuilder
    {
        public List<byte> bytes;
        public ByteBuilder()
        {
            bytes = new List<byte>();
        }
        public void Append(string inStr)
        {
            bytes.AddRange(Encoding.Default.GetBytes(inStr));
        }
        public void AppendByte(byte inByte)
        {
            bytes.Add(inByte);
        }       
    }

Then it's just:

ByteBuilder bb = new ByteBuilder();
bb.Append("stuff");
bb.AppendByte(0x97);
File.WriteAllBytes("something.txt", bb.bytes.ToArray());

Thanks for the help Richard + others.

[Richard Robertson]

I'm facepalming right now. I meant no builders altogether. Just convert each element to bytes and write it immediately.

However, if your solution works, then that's fine.

[jvanegmond]

Sounds like alot of refactoring just to fix a supposedly basic encoding problem... I thought there'd by a better way to handle this with .NET...

There is... You just need to supply it in a different encoding. Convert class is a utility which doesn't use ANSI so it only works for a very limited number of cases, it uses UTF16 to work in a lot of cases.

char longdash = Convert.ToChar(8212);

Edit: I see you removed your post now. I just want to make this very clear.

Edited November 15, 2010 by Manadar

[evilertoaster]

I'm facepalming right now. I meant no builders altogether. Just convert each element to bytes and write it immediately.

However, if your solution works, then that's fine.

The actual code involves a very heavy amount of looping, so you'd end up with thousands of FileWrite's doing that. You'd probably then use a Stream object for the writes instead of the static method, but at that point you're basically doing what I have there (assuming Stream objects are handled similar to an array of bytes)... and this way I don't have to refactor much.

I tested a sample case after the fact, and they're near identical in terms of performance so it's a moot point for me.

[evilertoaster]

There is... You just need to supply it in a different encoding. Convert class is a utility which doesn't use ANSI so it only works for a very limited number of cases, it uses UTF16 to work in a lot of cases.

char longdash = Convert.ToChar(8212);

Edit: I see you removed your post now. I just want to make this very clear.

Hum, I tried:

char longdash = Convert.ToChar(8212);
StringBuilder sb = new StringBuilder();
sb.Append("stuff" + longdash);
File.WriteAllText("something.txt", sb.ToString());

But it doesn't output correctly either... did you have a working example?

[Richard Robertson]

I intended for you to use a stream. Or even Win32's WriteFile. This File.WriteAll* stuff is silly in my opinion.

[jvanegmond]

Hum, I tried:

char longdash = Convert.ToChar(8212);
StringBuilder sb = new StringBuilder();
sb.Append("stuff" + longdash);
File.WriteAllText("something.txt", sb.ToString());

But it doesn't output correctly either... did you have a working example?

The example you posted works correctly.

[evilertoaster]

The example you posted works correctly.

It comes across as a dash in notepad or what-have-you, but that was never the issue. The raw output (for me at least) is:

"73 74 75 66 66 E2 80 94" which as noted isn't quite what I'm after.

Or does it come out differently for you?

[jvanegmond]

I missed that post. My bad.

Maybe from here I could try to spoon feed the solution, assuming I have an hour or two to spend to solve your problem - but I've already told you everything I know. Play with encodings, figure out what's going on - etc.

You can do it. I'm sure of it.

Edited November 18, 2010 by Manadar

[canny]

I have an old MySQL database with encoding set to UTF-8.