Odd problem with ASP.NET Caching and a CAPTCHA Control
I want to throw this out because I’ve been struggling with this particular problem for a couple of weeks off and on now. A couple of weeks ago I switched over this WebLog to a custom weblog engine that I created. For the most part things have been smooth. But I continue to have a big problem with the CAPTCHA control on the form that I use.
The control I use is Jeff Atwood’s CAPTCHA Control from CodeProject and I like the control for its interface and ease of drop and forget operation. Unfortunately I’ve run into some problems with the control not properly serving the images after the app's been runing for a while. I’ve looked over the code very carefully and frankly I don’t see anything wrong with it and of course it doesn’t fail during development even when putting it under load test.
I had the control also running with .Text for a couple of weeks or so, but I didn’t really pay close attention whether it failed or not – I just assumed it worked - it mighta been failing there too occasionally. <shrug>
Here’s what’s happening: When the App first starts everything’s fine. The Captcha displays and everything is peachy. I can access the page with the Captcha on it and it shows. However, after the App’s been running for a few hours the Captcha starts failing intermittently. It shows up most of the time, but not all.
You can click on the bottom of any post on the Add Comment button to display the Captcha and then press Post without any data (Alt-S) to see it fail. When it fails, it seems to cycle and work just about every other time which is also very odd.
I looked over Jeff's code carefully and what the control does is store some information about the image to generate into the Cache object as part of the control, which is then linked to an image that calls an HTTP Handler, which picks up the image data, generates the image and serves it. Very straight forward and I don’t see anything wrong with it.
When this first started happening I was actually able to repro this in my dev environment and what I saw happening is that the item got added to the cache, but when the handler is called the item dropped out of the cache. The cache entry returns NULL (actually nothing since this is VB code <s>).
The item goes into the cache with:
HttpContext.Current.Cache.Add(LocalGuid, _captcha, Nothing, DateTime.UtcNow.AddSeconds(25.0), System.Web.Caching.Cache.NoSlidingExpiration, Caching.CacheItemPriority.Normal, Nothing)
And I could see that this was definitely happening. The handler then retrieves with:
TCImage = app.Context.Cache.Get(strGuid)
ci = CType(TCImage, CaptchaImage)
app.Context.Cache.Remove(strGuid)
And it turns out that ci comes back nothing occasionally.
I’ve been going over this over and over varying the way things are going in the cache, changing the timeout, changing priority but I continue to see the failures on my site.
So yesterday I started looking at cache entries in the application with PerfMon on the server and the number of cache entries is very low and steady. It’s not like Cache entries are accumulating rapidly.
But it appears as if ASP.NET is simply not adding the item to the cache or adding it and quickly dumping the value back out. As I said the app doesn’t make heavy use of Cache and the memory usage looks good as well (it’s running about 40 megs steady and I moved it to its own AppPool to cut down on any memory starvation from the busy AppPool it's been running in).
The machine does not have a ton of memory and there’s a bunch of stuff running on it, but it’s not critically low on memory and certainly not low enough that I would think adding a couple of cache entries would cause removing of entries nearly instantly.
I’m running out of ideas. I can’t duplicate any longer in development mode after making a few minor changes to cache priority, but it continues to fail on the server, but only after the apps been running for a while. Restart the app and all is fine again for quite a while which does seem to indicate a memory problem. However, I find that REALLY hard to believe given this is a small object and they’re not being created in large quantities since you have to click an Add Comment button first before the Captcha shows.
Has anybody seen behavior like this before in Caching? Has anybody run into this with Jeff’s control and figured out what the issue is? I actually suspect there’s nothing wrong with Jeff’s code – I suspect my environment, but you never know. The code looks fine to me (and I went over it carefully <s>).
Incidentally I musta driven Jeff mad with my notes since I thought I had it fixed a few times after tweaking the code slightly and then turning around and going – nope still failing. Sorry dude <g>…
Other Posts you might also like
- Adding minimal OWIN Identity Authentication to an Existing ASP.NET MVC Application
- Map Physical Paths with an HttpContext.MapPath() Extension Method in ASP.NET
- Getting the Client IP Address in ASP.NET Core
- Resolving Paths To Server Relative Paths in .NET Code
- Getting the ASP.NET Core Server Hosting Urls at Startup and in Requests
The Voices of Reason
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
The reason I did this is I did not want to rely on the Viewstate, Session, nor Cache, just in case it was the cache. This approach does not need the cache. When I release the next version of Subtext, I'll write more about it. However, if you want a sneak peak, check it out in our repository.
https://svn.sourceforge.net/svnroot/subtext/branches/Release1.9/
In the Subtext.Web.Controls project.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
I do culture switching on the server, so in HttpApplication.BeginRequest the culture is swapped to the browser setting. That's the primary reason I swapped to UtcNow to ensure I get a fixed date value.
I'd be surprised if UtcNow wouldn't work though, since that's an absoute date value. Even if Cache does some additional conversion the value is a fixed one to start wtih.
But I wonder if the timing of of the culture switch may not be causing some sort of date offset problem. It's easy to check though - I'll turn off the code and see if that makes a difference.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
Question is - why would this make any difference at all? Cache has got to be writing items in Utc format so it can't be the timestamp that's problematic. Something else?
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
This line of code:
HttpContext.Current.Cache.Add(LocalGuid, _captcha, Nothing, DateTime.UtcNow.AddSeconds(25.0), System.Web.Caching.Cache.NoSlidingExpiration, Caching.CacheItemPriority.Normal, Nothing)
Is only putting the item in the cache for 25 seconds? That parameter is "Absolute Expiration" which means, after that amount of time goes by, it is removed from the cache. you don't show all the code, but once the item is put in the cache, is it expected to be there all the time? I don't think UTC matters... A DateTime object is a DateTime object. If it used DateTime.Now.AddSeconds(25.0) I think that would really be the same resulting time value.
When the code retrieves from the cache, does it check that it got a value? Priorty.Normal means that it could be removed from the cache. But, I think it is being removed after 25 seconds automatically. That doesn't really make sense.
BOb
-- And I tried to comment yesterday on something else but wasn't getting a captcha... no wonder.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
This:
---
TCImage = app.Context.Cache.Get(strGuid)
ci = CType(TCImage, CaptchaImage)
app.Context.Cache.Remove(strGuid)
And it turns out that ci comes back nothing occasionally.
---
Seems dangerous to me, espessially if the cache is using priority normal. That code should test that an image was retrieved from the cache, shoudn't it?
BOb
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
Page loads a reference to the image and the handler fires to feed the image Url . The short timeout is supposed to forestall orphaned entries (originally for search engines hitting the site since they would never pick up the image so it wouldn't get removed) so the cache doesn't bloat. It doesn't (Jeff's original code had the timeout at Session.Timeout which is way too long and easily could store a lot of content quickly).
But the item is expiring way before 25 seconds. The cache entry returns null and I get a 404 (which is the check code that follows) which tells me that the CTYPE conversion works like AS in CSharp which doesn't cause an exception if the value is null.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
BOb
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
But heck what do I know. At this point I really am drawing at straws because this seems damn wacky. This is kind of an unorthodox use of a cache entry I suppose. You should never make any assumptions about cache entries and this code obviosly requires that the value comes back on the other end or its dead.
Still it's very bizarre that the culture switching should have any effect, but it looks like that's the issue. It's been working all day today since I made the change. I'll switch it back to the culture code maybe tonight for a while to see if it starts failing again to be sure.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
I had a cache problem similar to this one a while back.
I had written some code to test if an object was in the cache and if it was not then it added itself to the cache with a 20 minute expiration.
HttpRuntime.Cache.Insert(cacheKey, someBO, dep, DateTime.Now.AddMinutes(20), Nothing)
Then later in the same code block I returned the newly added item like this:
Return HttpRuntime.Cache(cacheKey)
Oops! That doesn't work 100% of the time for the exact same reason you are seeing - the cache dumped the BO before the code moved 3 lines to this statement. It wasn't there anymore.
I re-arranged the code to create the BO outside of the Insert statment and then used that BO reference twice, once to do the insert and once to do the Return value. This works 100% of the time. The next time the cache is queried, if it happens to have the BO it returns it without re-creating it - but if not, it foloows the same steps and tries to add it again and then returns the reference to the newly created BO.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
So natch - the culture is probably an unrelated detail.
This is frustrating. I'm ready to switch to something else at this point - I've wasted too much time on this particular bit of code. Phil's idea of using an encrypted value to retrieve an item without relying on either cache or session sounds good to me at this point.
But this one really nags at me. Seeing the Cache fail like this really is a scary thing to see... especially given the light load that Cache is carrying on my machine and especially in this application.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
BOb
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
Update: Ok, I've been running with this setting for a couple of days now and it looks like it works properly!
FWIW, I hadn't looked at that as I took NotRemovable mean it never gets removed. However, it actually means that the item can not be preemptively removed from the cache and expiration still applies. This is in effect exactly what this app needs.
Still it's very unnerving to see the cache dump items so quickly in its normal mode. Apparently under memory pressure (this server is maxed out for memory) ASP.NET will not add anything to the cache at all (or immediately dump it).
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
Anyhow enough plugging (it's not even on my site yet, but hopefully it'll be there in a couple of days)
Anyhow getting to why I did a google.... ASP.NET cache NotRemovable
I'd like to use the cache in my own control, and I too do not want the items being dropped at random as it will also cause false rejections of the CAPTCHA answer.
I've found using the ASP.NET cache like this to be a bit of a nightmare in the past. I really don't think it has been designed for anything other than a cache used by your program to store data that it can easily recreate. If you can't recreate a cached item I think the ASP.NET team would tell you to use session and not the cache.
Despite that I don't want to use session for security reasons.
I am wary of using the cache as I have previously encountered the exact problem you are experiencing in ASP.NET 1.0 and 1.1. Although in ASP.NET 1.0 and 1.1 I found that the NotRemovable did help, but only by a small margin, and that items were still being dropped from the cache despite being flagged not removeable.
So you may find that this is only a half fix, and that it will rear it's ugly head every now and then on heavy days for your server.
One very ugly approach is to use the remove callback and stuff it back in when if it pops out before you'd like it to. However this doesn't work too well as when the cache decides that it's full it tends to remove items as new items are added, you end up with something akin to an old comedy sketch with your app madly racing round pushing things back into the cache only to have something else immediately pop out.
I was looking to see if this problem was still an issue in ASP.NET 2.0. Your site seems pretty popular so I guess it's a great testing ground as to whether this is still an issue or not.
If it turns out to still be an issue, the ugly alternative is to replace the cache with session state.
Let me know how it goes !
P.S :) I had to submit this twice the first one said my captcha answer was invalid... it wasn't.
You got that fix on this box yet ?
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
When you add the cache give it a callback method and log it using the ASP.NET health monitoring:
CacheItemRemovedCallback callback = new CacheItemRemovedCallback(HandleCacheItemRemovedCallback); HttpContext.Current.Cache.Add(... CacheItemPriority.NotRemovable, callback); ... private void HandleCacheItemRemovedCallback(string key, Object value, CacheItemRemovedReason reason) { if (reason != CacheItemRemovedReason.Removed) { CaptchaWebFailureAuditEvent evt = new CaptchaWebFailureAuditEvent("A CAPTCHA has been removed from cache with the reason: " + reason.ToString(), this, 20062006); evt.Raise(); } } ... // Invoked in case of events identified only by their event code. public CaptchaWebFailureAuditEvent(string msg, object eventSource, int eventCode): base(msg, eventSource, eventCode) { }
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
Then again I don't see the CAPTCHA when I post - my admin interface bypasses the captcha when I'm logged in. It's good to be King <g>...
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
Luckily there’s a workaround:
Cache.Add(CacheKey,IdValue, null,
DateTime.UtcNow.AddSeconds(30), TimeSpan.Zero,
CacheItemPriority.NotRemovable,null);
Which ensures that the cache manager doesn’t remove the item from the cache. Only an explicit Context.Cache.Remove() or a timeout will remove the item.
Thanks.
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
I've had the same problem and it's been when using Jeff's Captcha control also. The cache randomly drops for no reason. I'm going to look into whether or not the app pool is recycling at this exact time when the cache drops.
I've just recently had the problem on another site that uses cache to store user roles. It's been causing users to re-login 30 seconds after thei original login.
It's definitely not the Captcha control, but can't narrow it down completely.
- Corey
# re: Odd problem with ASP.NET Caching and a CAPTCHA Control
I noticed you are using DateTime.UtcNow.AddSeconds(25.0) as the absoluteExpiration. Note the UtcNow. Inside the Cache.Add the absoluteExpiration is again converted to Utc using DateTimeUtil.ConvertToUniversalTime(). Now I don't know if that is causing the problem but it sure doesn't sound right to me.
Maurice
BTW the first attempt to reply failed, guess why :-(