StreamWriter and UTF-8 Byte Order Marks

I‘m having an issue with StreamWriter and Byte Order Marks. The documentation seems to state that the Encoding.UTF8 encoding has byte order marks enabled but when files are being written some have the marks while other don‘t.

I‘m creating the stream writer in the following way:

this.Writer = new StreamWriter(this.Stream, System.Text.Encoding.UTF8);

Any ideas on what could be happening would be appreciated.

回答1

As someone pointed that out already, calling without the encoding argument does the trick. However, if you want to be explicit, try this:

using (var sw = new StreamWriter(this.Stream, new UTF8Encoding(false)))

The key is to construct a new UTF8Encoding(false), instead of using Encoding.UTF8Encoding. That‘s to control if BOM should be added or not.

This is the same as calling StreamWriter without the encoding argument, internally it‘s just doing the same thing.

回答2

The issue is due to the fact that you are using the static UTF8 property on the Encoding class.

When the GetPreamble method is called on the instance of the Encoding class returned by the UTF8 property, it returns the byte order mark (the byte array of three characters) and is written to the stream before any other content is written to the stream (assuming a new stream).

You can avoid this by creating the instance of the UTF8Encoding class yourself, like so:

// As before.
this.Writer = new StreamWriter(this.Stream, 
    // Create yourself, passing false will prevent the BOM from being written.
    new System.Text.UTF8Encoding());

As per the documentation for the default parameterless constructor (emphasis mine):

This constructor creates an instance that does not provide a Unicode byte order mark and does not throw an exception when an invalid encoding is detected.

This means that the call to GetPreamble will return an empty array, and therefore no BOM will be written to the underlying stream.

回答3

The only time I‘ve seen that constructor not add the UTF-8 BOM is if the stream is not at position 0 when you call it. For example, in the code below, the BOM isn‘t written:

using (var s = File.Create("test2.txt"))
{
    s.WriteByte(32);
    using (var sw = new StreamWriter(s, Encoding.UTF8))
    {
        sw.WriteLine("hello, world");
    }
}

As others have said, if you‘re using the StreamWriter(stream) constructor, without specifying the encoding, then you won‘t see the BOM.

UTF8Encoding(Boolean)

This constructor creates an instance that does not throw an exception when an invalid encoding is detected.

The encoderShouldEmitUTF8Identifier parameter controls the operation of the GetPreamble method. If true, the method returns a byte array containing the Unicode byte order mark (BOM) in UTF-8 format. If false, it returns a zero-length byte array. However, setting encoderShouldEmitUTF8Identifier to true does not cause the GetBytes method to prefix the BOM at the beginning of the byte array, nor does it cause the GetByteCount method to include the number of bytes in the BOM in the byte count.

UTF8Encoding(Boolean, Boolean)

The encoderShouldEmitUTF8Identifier parameter controls the operation of the GetPreamble method. If true, the method returns a byte array containing the Unicode byte order mark (BOM) in UTF-8 format. If false, it returns a zero-length byte array. However, setting encoderShouldEmitUTF8Identifier to true does not cause the GetBytes method to prefix the BOM at the beginning of the byte array, nor does it cause the GetByteCount method to include the number of bytes in the BOM in the byte count.

If throwOnInvalidBytes is true, a method that detects an invalid byte sequence throws an System.ArgumentException exception. Otherwise, the method does not throw an exception, and the invalid sequence is ignored.