Smart JavaScript Caching
Like many web-based software companies, a significant part of our frontend logic is dependent on JavaScript. This allows us to add all of the neat little animations and dynamic page updates that make web apps fun. It also dramatically increases the size of the JavaScript files that need to be downloaded.
Our approach in the past has been to just cache everything and forget about it. Unlike many public websites that have to operate under the assumption that the user’s cache will be empty, we can depend on a fairly high percentage of populated caches. And, because we have a single page application design, even if they don’t have the JavaScript cached, they only have to download the files once and they’ll be available the entire time the application is being used.
Our problem has actually been that pages are cached too much. We update our software fairly often, and most of the time these changes affect the JavaScript. When someone tries to use the app with outdated files in their cache all kinds of annoying blowups happen. Development was also a pain because we had to constantly keep clearing our browser cache to test new code. It wasn’t until recently, though, that we implemented a solution to these problems.
Step 1: Force Cache Reload After Upgrade
The most serious of caching problems was that people would have to manually clear their browser cache after we upgraded the application. This was solved by dynamically sticking the application version number into the JavaScript file’s querystring. Every time the app was upgraded the version number would change, as would the querystring. And, because the browser sees a new query string variable, it downloads the file from the server. The HTML being generated ended up looking like this:
<script type="text/javascript" src="js/dojo/dojo.js?redownloadToken=1.2.17"></script>
Step 2: Make the Developers’ Lives Easier
Developers are smart folks and can usually figure out when cache needs clearing. Just because they can, however, doesn’t mean they should have to. To remedy this problem we added a “development mode” to the app. When enabled, the redownloadToken above is changed from the application’s version number to a timestamp, effectively forcing a cache-free download on every page refresh. Even better, when development mode is enabled the app automatically uses the uncompressed dojo build for maximum debugability. Here’s the JSTL code we used:
<c:choose>
<c:when test="${development}">
<script type="text/javascript" src="<c:out value='${baseUrl}'/>/js/dojo/dojo.js.uncompressed.js?redownloadToken=<c:out value='${redownloadToken}'/>"></script>
</c:when>
<c:otherwise>
<script type="text/javascript" src="<c:out value='${baseUrl}'/>/js/dojo/dojo.js?redownloadToken=<c:out value='${redownloadToken}'/>"></script>
</c:otherwise>
</c:choose>
In the end we haven’t solved all caching problems, but our customers and developers are sure happier!
February 9th, 2007 at 9:08 pm
Good stuff, thanks for sharing your approach.
Thanks also for the link to our YUI Blog; I’m glad you found that article interesting.
thanks
nate
February 11th, 2007 at 9:18 am
Pretty cool. At one job, we put the CVS tag in the place of the ‘redownloadToken’. That way we didn’t have to bother with the inevitable, someone forgetting to change the value when making a build.
February 11th, 2007 at 11:21 am
That’s a great idea, Paul. That would definitely prevent unneeded file reloads when the application was upgraded and the JavaScript stays the same.
February 12th, 2007 at 4:35 am
In our project we use the similar technique:
Where ?1155 is a build number or a svn repository version. I think that redownloadToken=value is too verbose comparing to an integer appended to a js name.
February 12th, 2007 at 4:38 am
Sorry, the previous post was escaped, so here is the proposed declaration:
February 12th, 2007 at 4:23 pm
The problem of caching
I’ve already talked once before about caching problems but I after reading an article on a similar note, javascript caching and internet applications, I thought I would bring it up once more. In my original article I was specifically talking…
February 12th, 2007 at 4:46 pm
This is really cool. I’m a CVS survivor, not an expert. Is there a straight forward way to force the version number into the file automatically? Or do you just edit the file?
February 13th, 2007 at 3:27 am
To Fred:
This cannot be completely automated, nevertheless we have a magic service which returns the current build number (which is repository version in our case):
ServerInfoService.getBuildNumber();
For more details see the following example: http://snippets.dzone.com/posts/show/3507
February 13th, 2007 at 8:39 am
Excuse my ignorance, but how does the query string to the js work? What is doing the version comparison, etc?
February 13th, 2007 at 11:39 am
All this means is that the browser sees myfile.js?x=2 as a different file than myfile.js?x=3, even though they’re both just downloading the myfile.js file. By changing the stuff after the ? we can force the browser to download the file again. Because of that we can tell the browser to cache the file forever and then change what follows the ? whenever we modify the javascript.
July 11th, 2007 at 2:18 am
Thank you for a simple solution to a problem that has baffled me for a long time.
September 4th, 2007 at 9:31 am
All well in fact, until the user’s browser has the HTML page cached and never sees the new value appended to the javascript include.
Browser cache is a rogue beast – firefox is one of the worst (but I like the browser).
I’ve explicitly set my cache size to 0mb, and yet it STILL caches.
IE is pretty bad, since their default is to “Check for new versions Automatically” and the best setting is “Every visit to this page”
September 12th, 2007 at 11:40 am
Good point Michael. One thing we forgot to mention in this post is that for this technique to work all the time, you need to make sure you are serving up all your HTML pages with special HTTP response headers that tell the browser not to cache the page. In Java you do that like this:
response.setHeader(“Pragma”, “no-cache”);
response.setHeader(“Cache-Control”, “no-cache”);
which adds this to the beginning of the HTTP response:
Pragma: no-cache
Cache-Control: no-cache
It’s a rather fine distinction, so let me summarize again when to use these 2 different techniques. For dynamically generated content, you always want to make sure you add the no-cache directives that I posted above. Otherwise, your user might end up viewing stale data. The redownloadToken technique is for static resources that you do want to be cached, but which also need to be re-downloaded when a new version of your website is published.
September 27th, 2007 at 12:45 am
I don’t know what your stack is, but the primary problem I’ve run into is that IE doesn’t respect the server response on an ‘If-Modified-Since’ conditional GET. Even if a new resource is returned with a 200 status, IE decides to “agressively” cache the resource and use cache anyway. IE (and all browsers that I’m aware of) properly handle the ‘ETag’ and ‘If-None-Match’ combination.
I just wrote up our solution tonight @ http://bcarlso.blogspot.com/.
Proper ETag handling has solved the problem for us, enabling us to remove the version number “hack”. One side benefit is that because you are accessing/updating the same URI all the time, your client’s cache doesn’t keep growing and growing.
Thanks!
Brandon
December 6th, 2007 at 5:58 pm
[...] Smart JavaScript Caching [...]
November 12th, 2009 at 7:15 am
I´ve read all this thread and find it very insteresting.
I´m facing a situation in wich I IE7 is not requesting JS files at all. I verified with fiddler that there were no request for JS files. I´ve solved the problema appending a version number in the request, but I´d like to know if there is a way to avoid this behaviour. If I browse IE temporary files, there are no cached files that match the ones that are not being requested. But if I clear the browser cache the JS files begin to be requested and everithing works fine.
Best Regards
Sebastián Streiger