Server Errors when updating live Web Applications online
In the last few weeks I've been noticing some problems with site updates I've made to my live server. I have an application running locally and it's running fine without problems. The app is running as a Web Application Project (WAP) and so when I update the application most of the time I only update the BIN folder and possibly one or two of the ASPX markup pages.
So I upload all files out of the BIN directory to the server and I quite frequently end up with a yellow screen of death like this:
which is not an unfamiliar dialog. If you've ever uploaded files to a live site you've probably been greeted by this dialog (or the safe redirected version thereof). When files are 'in transit' .NET can and often does detect the new file on the server even though it's incomplete and tries to load it, which fails and boom you get the above error. That's well understood (although not really an optimal way to handle this - .NET should be able to detect that this file is still being written to instead of blinding unloading and reloading assemblies - but that's another story for another day).
But what's happening to me now is that these this page keeps coming up even after the file upload to the server has been completed. If I refresh the above page I will see the app cycling through the BIN directory assemblies with similar errors. Somehow the files are locked or at least ASP.NET thinks they are.
It takes re-copying files (first impulse right? Must have screwed up the upload somehow!), a restart or editing of Web.Config to get the app back up and running. However on several occasions I've even restarted and not been able to get the assemblies to work. This even though the app runs locally. In that case I ended up copying the assemblies a couple more times (without changes mind you) and eventually it took.
It certainly looks like ASP.NET is somehow caching an invalid assembly (or all of them) and not letting them go after the upload is complete. In that case I get the dependency error, but no notice that the process cannot access a file. No secondary error message.
This is a new behavior for me. The uploads are relatively small (about 500k total) so the upload is pretty quick usually and I've been donig these sort of live updates forever. Even so I often used to get failures while files were copying but never once the files were on the server. This seems to be common now and it's thrown a small kink in my typical update routine. <shrug>
App_Offline.htm
So over the last couple of days I started using App_Offline.htm to put the application on hold when updating assemblies, which is a bit of a hassle. In case you don't know about App_Offline.htm: It's a file you can drop into the root of your ASP.NET Virtual which causes ASP.NET to basically hold requests and show the content of this HTML page. ASP.NET displays the content of this file - oddly as text and not HTML - for any URL that would normally access an ASP.NET based URL.
I've never been enamored with this approach of having to have an external file that you either have to copy or rename on the server which is a bit annoying to automate. It'd be nicer if there was some mechanism to do this from within ASP.NET so it can be driven from within an application, but I suppose it makes sense: The idea is that no .NET assemblies should be loaded while App_Offline.htm is in place. If the idea is to make a clean update App_Offline.htm is the only way short of shutting down the Virtual Application or Web Server.
Since I've used the update with App_Offline.htm I've had no more problems as expected. I manage this process through my FTP upload process (which I usually do manually since I tend to update only a few files at a time rather than doing a full file dump). So my normal process is to logon jump to bin and update my assemblies, then update any pages that need to be updated.
With App_Offline.htm I keep the page in the site root and I rename it over FTP as needed. Rename to make it active, then copy all assemblies, then rename it back to something else.
What about you? How are you managing small updates to live sites? Have you seen the kind of issues i've mentioned above and have they gotten more pronounced recently (maybe something's changed in the runtimes?). And how do you manage your file updates and if you use App_Offline.htm how do you use it?
Other Posts you might also like
The Voices of Reason
# re: Server Errors when updating live Web Applications online
public static class HttpRuntimeUtil
{
private static readonly log4net.ILog log = log4net.LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
public static void LogApplicationShutdownInfo()
{
HttpRuntime runtime = (HttpRuntime)typeof(HttpRuntime).InvokeMember(
"_theRuntime",
BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.GetField,
null, null, null);
if (runtime != null)
{
string shutDownMessage = (string)runtime.GetType().InvokeMember(
"_shutDownMessage",
BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField,
null, runtime, null);
string shutDownStack = (string)runtime.GetType().InvokeMember(
"_shutDownStack",
BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField,
null, runtime, null);
string message = String.Format(
"{0}{0}HttpRuntime -> Application Shutdown{0}{0}_shutDownMessage:{0}{1}{0}{0}_shutDownStack:{0}{2}",
Environment.NewLine,
shutDownMessage,
shutDownStack);
log.Info(message);
}
}
}
# re: Server Errors when updating live Web Applications online
# re: Server Errors when updating live Web Applications online
Did you say that the App_Offline.htm file just outputs the text and not the HTML for the browser to render. I have had a similar experience where this file is delivered like a text file instead of an HTML file. This became very annoying, because my procedure is much like yours, rename it through FTP, and upload. I have been trying to find a work around, but I think I am just going to need to create a Module to deliver the file correctly, because for some reason the IIS team could manage.
Nick
# re: Server Errors when updating live Web Applications online
# re: Server Errors when updating live Web Applications online
Almost always if I copied new content over old content I had this problem.
Small sites though, newer got to using the App_Offline feature, but I guess it does the same as deleting te content in the binfolder.
# re: Server Errors when updating live Web Applications online
Did I mention that I haven't tried this yet?
# re: Server Errors when updating live Web Applications online
As far as the App_Offline it depends on what you use to upload the site. We use SyncBack which can handle this with little trick (I wish it did this in more straightforward way, but ohh well)
# re: Server Errors when updating live Web Applications online
I also use the App_Offline.htm approach when updating the site. My only concern is that if something goes bad during the update then I cannot visit my site to check what happened this is because App_Offline.htm is in place. My App_Offline.htm contains text saying "Please wait the site is being updated". I think there should be some way so that admin can view the site even though the App_Offline is in place.
# re: Server Errors when updating live Web Applications online
Btw, I use exactly this process for updating my live sites.
# re: Server Errors when updating live Web Applications online
The app_offline.htm i have in my live project displays HTML content just fine.... maybe it's the before mentioned size requirement throwing that off?
# re: Server Errors when updating live Web Applications online
# re: Server Errors when updating live Web Applications online
- Our application loads some DLLs after start up based on configuration. With shadow copying on the bin directory enabled, this could cause quite fantastic yellow screens of death as the processes managed to load new persistence layer DLLs while still running the older versions of business logic and presentation DLLs. So we disabled shadow copying by adding shadowCopyBinAssemblies="false" in our Web.config.
The [dis]advantage is that this locks the DLLs, so a simple copy and paste from Explorer won't work. We have to find a way to unload the application domain....
- We could use app_offline.htm, but app_offline.htm simply Does Not Work when using Web Gardens. (Since each worker process doesn't seem to check for the file until it's awakened by the next request, you essentially have no good way to force all of the currently active AppDomains to check for the file and spin down, short of visiting your site a whole bunch of times rapidly in succession.) Also, this file returns HTTP 404, which is dumb.
- Our solution is to activate an "offline Web site" that handles all ASP.NET requests with a "We're updating blah blah" page that returns an HTTP 503 Service Unavailable code, stop the AppPool in the real Web site, and update/replace the files in the bin directory. Then deactivate the updating site and reactivate the real site.
I'd love to hear of a better way, but this method seems to cause the least heartburn.
# re: Server Errors when updating live Web Applications online
# re: Server Errors when updating live Web Applications online
# re: Server Errors when updating live Web Applications online
This ensures that the app_pool is not starting to reload, and then misses any changes to the bin folder/web.config file, this is done from a shadow directory on the live server, to minimize the time it takes to copy the files, because doing it with FTP will sometimes cause the app_pool to not reload all dll's and web.config file :-(
I have also experimented with haveing 2 IIS websites that are working as a master/slave, this can help the app_pool to "warm" up, so that the update would be 100% transparent for the end user, but it also causes other problems like caching
So far the best method seems to be the robocopy
# re: Server Errors when updating live Web Applications online
# re: Server Errors when updating live Web Applications online
I found a great solution for uploading a new version of my site. I use the following tools:
* Gene6 FTP server on the web server (has MTDM support)
* WebDrive on my development PC (sets up an FTP connection as a drive letter)
* ViceVersa file sync tool.
WebDrive and Gene6 FTP server effectively make a folder on the web server act just like a local folder. I then use ViceVersa to do a basic file sync. Some notes about this solution:
* It's important for the FTP server to have MTDM support so that uploaded files have the right last modified date/time set for future file syncs.
* ViceVersa allows include/exclude filters, so that I only upload the necessary files. For example, I upload .aspx, .dll and .config, but I don't upload .cs.
* I have a batch file which pushes App_offline.htm up to the server. I then make any database schema changes (using the excellent RedGate SqlCompare). Finally, I run the file sync which also wipes out the App_offline.htm.
* ViceVersa can also run a script before and after the file sync. I have it run a script after the file sync to open the homepage, so that the first user won't have to wait during the first request.
Hope this helps.
Ben
# re: Server Errors when updating live Web Applications online
1) I only keep the assemblies unlikely to change in bin (i.e. components, third-party libs etc.). I put my app's assemblies into a sub-folder of bin and name the folder according to the date (for example: bin\20080801)
2) My web.config has the following:
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<probing privatePath="bin;bin\20080801;" />
</assemblyBinding>
</runtime>
When I need to update assemblies, I just upload them into a new sub-folder in bin, again with a name corresponding to the current date (bin\20080812). After the upload is complete, I upload web.config with a change to the sub-folder name. The app re-starts and picks up the new assemblies and forgets about the old sub-folder assemblies which I can leave or delete.
Seems to work and has the added benefit of speeding up the app start since the sub-folder assemblies are ignored until needed.
# re: Server Errors when updating live Web Applications online
I have no idea if yours is the best solution, but from reading it I like it a lot. This would make rolling back to a previous version pretty easy.
Nice!
# re: Server Errors when updating live Web Applications online
Response Headers:
Server Microsoft-IIS/7.0
X-Powered-By ASP.NET
Date Fri, 15 Aug 2008 02:14:59 GMT
Content-Length 25
# re: Server Errors when updating live Web Applications online
I use App_Offline.htm whenever I'm doing site changes that involve CSS, Script, and other stuff that I can't bang into the site in one quick stroke. I keep it in the root folder with ".exclude" at the end (i.e., "App_Offline.htm.exclude") and then toggle renaming it to turn it on/off. I guess the only thing quicker would be to create a batch file, but this is simpler for me to remember.
As far as site updates are concerned (with the warning message you're getting), I pre-compile the entire site -- even the markup -- so site updates involving just the ASP.NET stuff can be banged into the site instantly, without the need for App_Offline.htm.
I upload all the bin folder updates to a temp folder, and then copy/replace everything at once in the bin folder using Windows Explorer or DOS window. It happens instantly, with no downtime to users. Also, there is never any first-use compile delays that can happen with updateable sites.
# re: Server Errors when updating live Web Applications online
You can set config values like 'waitChangeNotification', so the app won't recycle right away. Giving the file system a chance to catch up with the writes.
# re: Server Errors when updating live Web Applications online
Just forcing a pool reload isn't very elegent for visitors.
Does anyone address this?
# re: Server Errors when updating live Web Applications online
If Sessions are critical in your app you shouldn't be using InProc sessions anyway - using OutOfProc or Sql Sessions will let your sessions survive AppDomain and AppPool restarts and is a much better choice if Session data is essential to visitors of your site.
# re: Server Errors when updating live Web Applications online
# Problem App_Offline.htm with IIS 7.0
Firefox and Opera (IE show as html) show only text because server send to the browser this headers:
HTTP/1.0 404 Not Found
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET
Content-Length: 1641
Content-Type: text/plain
It is bug of IIS 7.0??? Because IIS 5.1 and IIS 6.0 send Content-Type: text/html...
# re: Server Errors when updating live Web Applications online
Is there any way to get IIS to block all types of content regardless of file type/extension when an app_offline.htm is present?
# re: Server Errors when updating live Web Applications online
You start copying files, ASP.NET notices a change, restarts.
You finish copying files, ASP.NET was busy restarting, didn't notice the last changes.
Seems most likely to me, anyway.