HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
Ok, I feel like an idiot, but I’ve been experimenting with this for an hour now and I cannot for the life of me figure how to get the encoding correct to run output from an ASP.NET page into a file. Well, no I can get it to work with explicitly setting the encoding to the Windows 1252, but this is not really what I want…
Here’s the setup. I’m using the ASP.NET runtime inside of a desktop app. The ProcessRequest method needs to pass a TextWriter to the ASP.NET runtime into which it will then render the output. This is the same TextWriter that Page.Render writes into for example.
All works well, except I can’t get the output written to file to look correct. What I want is to get the output written in UTF-8. So I thought I can use:
TextWriter Output;
try
{
// *** Note you have to write the right 'codepage'. If you use the default UTF-8
// *** everything will be double encoded.
Output = new StreamWriter(this.OutputFile,false, Encoding.UTF8);
}
catch (Exception ex)
{
this.Error = true;
this.ErrorMessage = ex.Message;
return false;
}
// *** Reset the Response settings
this.ResponseHeaders = null;
this.Cookies = null;
this.ResponseStatusCode = 200;
wwWorkerRequest Request = new wwWorkerRequest(Page, QueryString, Output);
if (this.Context != null)
Request.Context = this.Context;
Request.PostData = this.PostData;
Request.PostContentType = this.PostContentType;
Request.RequestHeaders = this.RequestHeaders;
Request.PhysicalPath= this.PhysicalDirectory;
try
{
HttpRuntime.ProcessRequest(Request);
}
catch(Exception ex)
{
Output.Close();
this.ResponseStatusCode = 500;
this.ErrorMessage = ex.Message;
this.Error = true;
return false;
}
Output.Close();
this.ResponseHeaders = Request.ResponseHeaders;
this.ResponseStatusCode = Request.ResponseStatusCode;
// *** Capture the Cookies that were set by the server
this.Cookies = Request.Cookies;
if (Request.Context != null)
this.Context = Request.Context;
return true;
The ASP.NET application is setup to encode to UTF-8 in Web.config:
<globalization requestEncoding="utf-8" responseEncoding="utf-8" />
So, what happens? Output gets generated but the output actually gets double encoded. I have a string like this embedded in the HTML of the ASPX rendered:
¢ª
After running the code above I get (raw output):
¢ª
Which is some funky double encoded wanna-be UTF-8 output of the above characters.
Next, I thought Ok, so we’re double encoding – let’s try Encoding.Ascii on the stream, but that gives me invalid characters (??????), so that’s no good either. Using Encoding.Default produces different results yet:
¢ª
which is just plain garbage.
I don’t really understand what I should be passing in for a TextWriter here when I need to dump to file. Why is this double encoding occurring when I use Encoding.UTF8 on the stream? It seems what I need is raw binary stream into which the encoding TextWriter is writing. But then I’m stilling missing the byte order mark too…
What am I missing here? How do I set up my stream and TextWriter to get ASP.NET to write my output to file as properly encoded UTF-8 including the UTF-8 PreAmble and properly encoded upper characters?
Other Posts you might also like
- Adding minimal OWIN Identity Authentication to an Existing ASP.NET MVC Application
- Resolving Paths To Server Relative Paths in .NET Code
- Map Physical Paths with an HttpContext.MapPath() Extension Method in ASP.NET
- Back to Basics: Rewriting a URL in ASP.NET Core
- Getting the Client IP Address in ASP.NET Core
The Voices of Reason
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
I have no control over the code inside of the ASP.NET application since this is the hosting wrapper around the ASP.NET runtime.
I played around with using Unicode for the encoding in both the file stream and web.config, but this produced even more funky results with the extended characters.
Further oddities are if I create the stream and write out a PreAmble ASP.NET kills any of the stream output and re-writes the stream, even if I explicitly flushed the output to disk.
WTF are they doing with the stream? This should be trivial.
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
http://www.informit.com/guides/content.asp?g=dotnet&seqNum=163&rl=1
when i used system.text.encoding.default then it wrote out characters with an ASCII code of above 127 how I expected.... (in fact this is what you wrote at the beginning of your article - the default must be the codepage your system is set to i.e. 1252).
Is this the case that you have written this file out in UTF-8 but that whatever is reading it thinks that it is ASCII and therefore shows the weird character - doesn't UTF encode a special character as 2 bytes which would look like 2 chars in an ASCII file.
My thinking that the "preamble" tells whatever is reading it that what follows is UTF-8 and therefore the 2char thing is a special char. I am writing out html pages - maybe there is a setting i can put in the header to say it is utf-8 and it would then be fine. There definitely is one for xml.
To be honest I was very frustrated by this weird char thing and have stopped now that i can at least get what i want using the "default" encoding value. Is this to do with codepages being used which has now been superceded by Unicode, meaning we have weird problems when you try and swap between the two.
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
Its just that you are reading it with notepad so it looks horrible.
I was writing out web pages using the streamwriter (that I had read in using HttpWebRequest)
Surprise surprise I got these garbage characters.
If I added the following meta tag to the html
<meta http-equiv="content-type" content="text-html; charset=utf-8">
then when i opened them in a browser they looked fine. In notepad the pesky chars were still there.
The problem seems to me that Microsoft has not been very consistent - since I was writing out ASP.NET pages that I had written. The default should be for ASP.NET pages to be in UTF-8 with this meta tag, then the default for streamwriter being UTF-8 would make sense.
Anyhow you probably are not worried about this anymore but i bet many people are stumbling over the exact same problem (my google searches tend to confirm this).
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
{
try
{
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(strURL);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
Stream receiveStream = myHttpWebResponse.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(receiveStream, encode);
System.IO.FileStream fileoutput;
Char[] read = new Char[256];
Byte[] readb = new Byte[256];
int count = readStream.Read(read, 0, 256);
fileoutput = System.IO.File.Create(strFilename);
while (count > 0)
{
int l = 0;
for (int i = 0; i < count; i++)
{
if (read[i] <= (int)Byte.MaxValue && read[i] >= (int)Byte.MinValue)
{
readb[l] = Convert.ToByte(read[i]);
l++;
}
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
{
string struni;
Char[] readuni;
Byte[] readbuni;
fileoutput.Write(readb, 0, l);
struni = "&#" + Convert.ToInt16(read[i]).ToString() + ";";
readuni = new Char[struni.Length];
readbuni = new Byte[struni.Length];
readuni = struni.ToCharArray();
for (int p = 0; p < struni.Length; p++)
readbuni[p] = Convert.ToByte(readuni[p]);
fileoutput.Write(readbuni, 0, struni.Length);
l = 0;
}
}
fileoutput.Write(readb, 0, l);
count = readStream.Read(read, 0, 256);
}
fileoutput.Close();
myHttpWebResponse.Close();
readStream.Close();
return (true);
}
catch(Exception ex)
{
return (false);
}
}/*SaveHtmlToFile*/
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
Response.ContentType = "text/xml";
// since the XML is based on a XmlStringWriter its based on a string == unicode == UTF-16
Response.ContentEncoding = Encoding.Unicode;
// for some reason, the following is also needed - you would think it was set by default.. but it ain't
Response.BinaryWrite(Encoding.Unicode.GetPreamble());
// --- elsewhere I have.. ----
StringBuilder output = new StringBuilder();
XmlTextWriter writer = new XmlTextWriter(new StringWriter(output));
writer.WriteStartDocument();
writer.WriteStartElement("ajax-response");
.. etc..
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
How did you solve this issue with encoding?
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
Here's my code snippet on the page called by my AJAX code:
protected void Page_Load(object sender, EventArgs e) { Response.ContentType = "text/xml"; Response.ContentEncoding = Encoding.Unicode; Response.BinaryWrite(Encoding.Unicode.GetPreamble()); // <-- this is the key!! Response.Write(GetXmlResults(items)); } private static String GetXmlResults(Object[] items) { // Create xml setting object XmlWriterSettings xmlSettings = new XmlWriterSettings(); xmlSettings.OmitXmlDeclaration = false; xmlSettings.Indent = true; // Instantiate an XmlTextWriter object. StringBuilder sb = new StringBuilder(); using (XmlWriter xw = XmlWriter.Create(sb, xmlSettings)) { // Write the XML declaration node. xw.WriteStartDocument(true); foreach (Object item in items) { WriteXmlItems(item.Name, item.Value); } // Close the file and perform cleanup. xw.WriteEndDocument(); xw.Flush(); xw.Close(); } return sb.ToString(); } private static void WriteXmlItems(String name, String value) { xw.WriteStartElement("item"); xw.WriteElementString("name", name); xw.WriteElementString("value", value); xw.WriteEndElement(); // item }
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
I'm not sure if you still have this problem. I ran into it today as well. Slightly different solution to others mentioned in response.
I assume that your wwWorkerRequest class inherits from System.Web.Hosting.SimpleWorkerRequest?
If so, you'll need to override the SendResponseFromMemory() method and write to your StreamWriter in UTF8.
e.g.
public override void SendResponseFromMemory(byte[] data, int length)
{
_output.Write(System.Text.Encoding.UTF8.GetString(data));
}
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
This solution stopped HttpRuntime.ProcessRequest() mangling Japanese characters in my page.
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
When the record was printed, the word special characters appeared as garbage. The 2 lines mentioned above:
Response.ContentEncoding = Encoding.Unicode;
Response.BinaryWrite(Encoding.Unicode.GetPreamble());
fixed this issue. Thank you very much!
Lee
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
Response.ContentEncoding = Encoding.Unicode; Response.BinaryWrite(Encoding.Unicode.GetPreamble());
Saved my day!
Thanks!
# re: HtmlTextWriters, Encoding, ASP.NET and writing Encoded output to file
This is how I fixed it:
Response.ContentEncoding = Encoding.Unicode;
Response.BinaryWrite(Encoding.Unicode.GetPreamble());
And after that, just regular Response.Write for the content.
I guess it will work exactly the same using Encoding.UTF8.