Odd problem with ASP.NET Caching and a CAPTCHA Control

October 10, 2006 • from Maui, Hawaii • 29 comments

On this page:

I want to throw this out because I’ve been struggling with this particular problem for a couple of weeks off and on now. A couple of weeks ago I switched over this WebLog to a custom weblog engine that I created. For the most part things have been smooth. But I continue to have a big problem with the CAPTCHA control on the form that I use.

The control I use is Jeff Atwood’s CAPTCHA Control from CodeProject and I like the control for its interface and ease of drop and forget operation. Unfortunately I’ve run into some problems with the control not properly serving the images after the app's been runing for a while. I’ve looked over the code very carefully and frankly I don’t see anything wrong with it and of course it doesn’t fail during development even when putting it under load test.

I had the control also running with .Text for a couple of weeks or so, but I didn’t really pay close attention whether it failed or not – I just assumed it worked - it mighta been failing there too occasionally. <shrug>

Here’s what’s happening: When the App first starts everything’s fine. The Captcha displays and everything is peachy. I can access the page with the Captcha on it and it shows. However, after the App’s been running for a few hours the Captcha starts failing intermittently. It shows up most of the time, but not all.

You can click on the bottom of any post on the Add Comment button to display the Captcha and then press Post without any data (Alt-S) to see it fail. When it fails, it seems to cycle and work just about every other time which is also very odd.

I looked over Jeff's code carefully and what the control does is store some information about the image to generate into the Cache object as part of the control, which is then linked to an image that calls an HTTP Handler, which picks up the image data, generates the image and serves it. Very straight forward and I don’t see anything wrong with it.

When this first started happening I was actually able to repro this in my dev environment and what I saw happening is that the item got added to the cache, but when the handler is called the item dropped out of the cache. The cache entry returns NULL (actually nothing since this is VB code <s>).

The item goes into the cache with:

HttpContext.Current.Cache.Add(LocalGuid, _captcha, Nothing, DateTime.UtcNow.AddSeconds(25.0), System.Web.Caching.Cache.NoSlidingExpiration, Caching.CacheItemPriority.Normal, Nothing)

And I could see that this was definitely happening. The handler then retrieves with:

TCImage = app.Context.Cache.Get(strGuid)

ci = CType(TCImage, CaptchaImage)

app.Context.Cache.Remove(strGuid)

And it turns out that ci comes back nothing occasionally.

I’ve been going over this over and over varying the way things are going in the cache, changing the timeout, changing priority but I continue to see the failures on my site.

So yesterday I started looking at cache entries in the application with PerfMon on the server and the number of cache entries is very low and steady. It’s not like Cache entries are accumulating rapidly.

But it appears as if ASP.NET is simply not adding the item to the cache or adding it and quickly dumping the value back out. As I said the app doesn’t make heavy use of Cache and the memory usage looks good as well (it’s running about 40 megs steady and I moved it to its own AppPool to cut down on any memory starvation from the busy AppPool it's been running in).

The machine does not have a ton of memory and there’s a bunch of stuff running on it, but it’s not critically low on memory and certainly not low enough that I would think adding a couple of cache entries would cause removing of entries nearly instantly.

I’m running out of ideas. I can’t duplicate any longer in development mode after making a few minor changes to cache priority, but it continues to fail on the server, but only after the apps been running for a while. Restart the app and all is fine again for quite a while which does seem to indicate a memory problem. However, I find that REALLY hard to believe given this is a small object and they’re not being created in large quantities since you have to click an Add Comment button first before the Captcha shows.

Has anybody seen behavior like this before in Caching? Has anybody run into this with Jeff’s control and figured out what the issue is? I actually suspect there’s nothing wrong with Jeff’s code – I suspect my environment, but you never know. The code looks fine to me (and I went over it carefully <s>).

Incidentally I musta driven Jeff mad with my notes since I thought I had it fixed a few times after tweaking the code slightly and then turning around and going – nope still failing. Sorry dude <g>…

The Voices of Reason

Maurice
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Hi Rick,

I noticed you are using DateTime.UtcNow.AddSeconds(25.0) as the absoluteExpiration. Note the UtcNow. Inside the Cache.Add the absoluteExpiration is again converted to Utc using DateTimeUtil.ConvertToUniversalTime(). Now I don't know if that is causing the problem but it sure doesn't sound right to me.

Maurice

BTW the first attempt to reply failed, guess why :-(

Steve from Pleasant Hill
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

You indeed might have driven him mad as he is fixating on software development methodologies and religion in his blog...

Haacked
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

I ended up rewriting his control using encryption to send the answer to the image generator via the URL based on a technique I wrote about here: http://haacked.com/archive/2006/10/02/Better_CAPTCHA_Through_Encryption.aspx

The reason I did this is I did not want to rely on the Viewstate, Session, nor Cache, just in case it was the cache. This approach does not need the cache. When I release the next version of Subtext, I'll write more about it. However, if you want a sneak peak, check it out in our repository.

https://svn.sourceforge.net/svnroot/subtext/branches/Release1.9/

In the Subtext.Web.Controls project.

Rick Strahl
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Hmmmm... Maurice you may be on to something here.

I do culture switching on the server, so in HttpApplication.BeginRequest the culture is swapped to the browser setting. That's the primary reason I swapped to UtcNow to ensure I get a fixed date value.

I'd be surprised if UtcNow wouldn't work though, since that's an absoute date value. Even if Cache does some additional conversion the value is a fixed one to start wtih.

But I wonder if the timing of of the culture switch may not be causing some sort of date offset problem. It's easy to check though - I'll turn off the code and see if that makes a difference.

Rick Strahl
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Ok, so it's been a few hours and it looks like this is now working after I turned off culture mapping...

Question is - why would this make any difference at all? Cache has got to be writing items in Utc format so it can't be the timestamp that's problematic. Something else?

Bob Archer
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Rick,

This line of code:

HttpContext.Current.Cache.Add(LocalGuid, _captcha, Nothing, DateTime.UtcNow.AddSeconds(25.0), System.Web.Caching.Cache.NoSlidingExpiration, Caching.CacheItemPriority.Normal, Nothing)

Is only putting the item in the cache for 25 seconds? That parameter is "Absolute Expiration" which means, after that amount of time goes by, it is removed from the cache. you don't show all the code, but once the item is put in the cache, is it expected to be there all the time? I don't think UTC matters... A DateTime object is a DateTime object. If it used DateTime.Now.AddSeconds(25.0) I think that would really be the same resulting time value.

When the code retrieves from the cache, does it check that it got a value? Priorty.Normal means that it could be removed from the cache. But, I think it is being removed after 25 seconds automatically. That doesn't really make sense.

BOb

-- And I tried to comment yesterday on something else but wasn't getting a captcha... no wonder.

Bob Archer
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Rick,

This:

---
TCImage = app.Context.Cache.Get(strGuid)
ci = CType(TCImage, CaptchaImage)
app.Context.Cache.Remove(strGuid)

And it turns out that ci comes back nothing occasionally.
---

Seems dangerous to me, espessially if the cache is using priority normal. That code should test that an image was retrieved from the cache, shoudn't it?

BOb

Rick Strahl
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Bob, yes the item is supposed to expire pretty quickly since it's an Asynchronously loaded image. 25 seconds should be plenty to serve the CAPTCHA image. If it takes 25 seconds to load the page I have other issues than the CAPTCHA <g>.

Page loads a reference to the image and the handler fires to feed the image Url . The short timeout is supposed to forestall orphaned entries (originally for search engines hitting the site since they would never pick up the image so it wouldn't get removed) so the cache doesn't bloat. It doesn't (Jeff's original code had the timeout at Session.Timeout which is way too long and easily could store a lot of content quickly).

But the item is expiring way before 25 seconds. The cache entry returns null and I get a 404 (which is the check code that follows) which tells me that the CTYPE conversion works like AS in CSharp which doesn't cause an exception if the value is null.

Bob Archer
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Is it possible the handler is trying to get the image from the cache before the async code has loaded the image into the cache? Perhaps turning on the auto culture switch causes this delay?

BOb

Rick Strahl
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

No. The culture is set at the very beginning of th request that's long done by the time the cache gets to it. The page is done rendering by the time the browser gets it so there can't really be a timing issue and adding things to the cache isn't asynchronous so I don't think that can be it.

But heck what do I know. At this point I really am drawing at straws because this seems damn wacky. This is kind of an unorthodox use of a cache entry I suppose. You should never make any assumptions about cache entries and this code obviosly requires that the value comes back on the other end or its dead.

Still it's very bizarre that the culture switching should have any effect, but it looks like that's the issue. It's been working all day today since I made the change. I'll switch it back to the culture code maybe tonight for a while to see if it starts failing again to be sure.

Joe Fallon
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Rick,
I had a cache problem similar to this one a while back.
I had written some code to test if an object was in the cache and if it was not then it added itself to the cache with a 20 minute expiration.

HttpRuntime.Cache.Insert(cacheKey, someBO, dep, DateTime.Now.AddMinutes(20), Nothing)

Then later in the same code block I returned the newly added item like this:
Return HttpRuntime.Cache(cacheKey)

Oops! That doesn't work 100% of the time for the exact same reason you are seeing - the cache dumped the BO before the code moved 3 lines to this statement. It wasn't there anymore.

I re-arranged the code to create the BO outside of the Insert statment and then used that BO reference twice, once to do the insert and once to do the Return value. This works 100% of the time. The next time the cache is queried, if it happens to have the BO it returns it without re-creating it - but if not, it foloows the same steps and tries to add it again and then returns the reference to the newly created BO.

Rick Strahl
October 10, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Ah damn it - the problem is back. It's taken a lot longer to come around to this - basically a very busy day on the site, but it's now starting to fail once again.

So natch - the culture is probably an unrelated detail.

This is frustrating. I'm ready to switch to something else at this point - I've wasted too much time on this particular bit of code. Phil's idea of using an encrypted value to retrieve an item without relying on either cache or session sounds good to me at this point.

But this one really nags at me. Seeing the Cache fail like this really is a scary thing to see... especially given the light load that Cache is carrying on my machine and especially in this application.

Bob Archer
October 11, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

I would recommend changing the code so that when it puts the image in the cache the priorty is set to NotRemoveable and see if that solves it.

BOb

Rick Strahl
October 11, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Rick Strahl
October 12, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Bob, not sure how I missed that. Trying it now. I've turned the locale switching back on since it seems to make the problem worse when it occurs, so we should see it fail pretty quickly if the .NotRemovable flag doesn't fix the issue.

Update: Ok, I've been running with this setting for a couple of days now and it looks like it works properly!

FWIW, I hadn't looked at that as I took NotRemovable mean it never gets removed. However, it actually means that the item can not be preemptively removed from the cache and expiration still applies. This is in effect exactly what this app needs.

Still it's very unnerving to see the cache dump items so quickly in its normal mode. Apparently under memory pressure (this server is maxed out for memory) ASP.NET will not add anything to the cache at all (or immediately dump it).

Bob Archer
October 19, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Glad I could help.

Michael Lang
October 30, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Hey I came across your blog as I'm actually developing my own CAPTCHA control which incidently does NOT use images, I'm taking another approach with accessibility in mind....

Anyhow enough plugging (it's not even on my site yet, but hopefully it'll be there in a couple of days)

Anyhow getting to why I did a google.... ASP.NET cache NotRemovable

I'd like to use the cache in my own control, and I too do not want the items being dropped at random as it will also cause false rejections of the CAPTCHA answer.

I've found using the ASP.NET cache like this to be a bit of a nightmare in the past. I really don't think it has been designed for anything other than a cache used by your program to store data that it can easily recreate. If you can't recreate a cached item I think the ASP.NET team would tell you to use session and not the cache.

Despite that I don't want to use session for security reasons.

I am wary of using the cache as I have previously encountered the exact problem you are experiencing in ASP.NET 1.0 and 1.1. Although in ASP.NET 1.0 and 1.1 I found that the NotRemovable did help, but only by a small margin, and that items were still being dropped from the cache despite being flagged not removeable.

So you may find that this is only a half fix, and that it will rear it's ugly head every now and then on heavy days for your server.

One very ugly approach is to use the remove callback and stuff it back in when if it pops out before you'd like it to. However this doesn't work too well as when the cache decides that it's full it tends to remove items as new items are added, you end up with something akin to an old comedy sketch with your app madly racing round pushing things back into the cache only to have something else immediately pop out.

I was looking to see if this problem was still an issue in ASP.NET 2.0. Your site seems pretty popular so I guess it's a great testing ground as to whether this is still an issue or not.
If it turns out to still be an issue, the ugly alternative is to replace the cache with session state.

Let me know how it goes !

P.S :) I had to submit this twice the first one said my captcha answer was invalid... it wasn't.
You got that fix on this box yet ?

Rick Strahl
October 30, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

The CAPTCHA will eat it too if the server cycles. It's about the time that the server cycles every night so it's possible that you got bumped right at that time.

Michael Lang
October 30, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Another alternative would - be write your own cache object. This would only be a 1 or 2 hour coding exercise:- write a class that holds an object dictionary and a datetime dictionary, and has a worker thread removing items that have exceeded a certain time limit then store it in application state. This is the road I'm going to head down if I get feedback from you that the NotRemovable cache flag is still not reliable.

Michael Lang
October 30, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Sorry I'm a little tired today. I should have thought of this before. There's an easy way to test if you still have a problem....

When you add the cache give it a callback method and log it using the ASP.NET health monitoring:

CacheItemRemovedCallback callback = new CacheItemRemovedCallback(HandleCacheItemRemovedCallback); 

HttpContext.Current.Cache.Add(... CacheItemPriority.NotRemovable, callback);

...

        private void HandleCacheItemRemovedCallback(string key, Object value, CacheItemRemovedReason reason)
        {
            if (reason != CacheItemRemovedReason.Removed)
            {
                CaptchaWebFailureAuditEvent evt = new CaptchaWebFailureAuditEvent("A CAPTCHA has been removed from cache with the reason: " + reason.ToString(), this, 20062006);
                evt.Raise();
            }
        }

...

// Invoked in case of events identified only by their event code.
public CaptchaWebFailureAuditEvent(string msg, object eventSource,
    int eventCode):
base(msg, eventSource, eventCode)
{
}

Rick Strahl
October 30, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Michael, NotRemovable seems to have fixed the problem here. I haven't heard or seen any broken CAPTCHA images.

Then again I don't see the CAPTCHA when I post - my admin interface bypasses the captcha when I'm logged in. It's good to be King <g>...

DanTheMan
November 02, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Having the same problem with my cache. I have been logging the cacheItemRemovedCallback events and see that the garbage collection is randomly removing items as it sees fit. As it is, I have a case open with Microsoft and I am trying to work through it, but I don't see how making it NotRemovable is a fix. This kind of pisses me off because I have been running this app in 1.1 for over 3 years with no problems, as soon as I rolled over and started testing in 2.0 I am having this issue. I will post my findings as they come.

Rick Strahl
November 02, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Dan remember that NotRemovable means that it can't be removed by resource pressure, but it will remove at the timeout. So I think if you have items you really need to have in there for the cache duration this addresses that issue.

dedy
November 29, 2006

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Have you solved this problem yet? I'm having the same issue...

jrummell
July 11, 2007

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Bob Archer you are my hero. I had the same problem and NotRemovable fixed it. Thanks!

Nardos
August 27, 2007

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

I am having the same issue, Any solution?

Nardos
August 28, 2007

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Please check this link http://www.west-wind.com/WebLog/posts/11379.aspx , it solves the problem that I had with ASP.NET Cache is not actually caching cache items, Items start disappearing out of the cache even though items are definitely written to it.

Luckily there’s a workaround:

Cache.Add(CacheKey,IdValue, null,
DateTime.UtcNow.AddSeconds(30), TimeSpan.Zero,
CacheItemPriority.NotRemovable,null);

Which ensures that the cache manager doesn’t remove the item from the cache. Only an explicit Context.Cache.Remove() or a timeout will remove the item.

Thanks.

Toomas
December 12, 2007

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Thanks Rick for bringing attention to this problem. NotRemovable made my day.

Corey
April 21, 2009

# re: Odd problem with ASP.NET Caching and a CAPTCHA Control

Rick,

I've had the same problem and it's been when using Jeff's Captcha control also. The cache randomly drops for no reason. I'm going to look into whether or not the app pool is recycling at this exact time when the cache drops.

I've just recently had the problem on another site that uses cache to store user roles. It's been causing users to re-login 30 seconds after thei original login.

It's definitely not the Captcha control, but can't narrow it down completely.

- Corey

Rick Strahl's Weblog