HTTP cache on Rails + Nginx stack

Whats inside

AlekseyL

20 min readFeb 3, 2017

Two ways to cache the uncacheable.

Why you should not ever use expires_in inside rails to cache http request.

How Nginx calcs ETag on static files and why we don’t care about it today.

How correctly purge Nginx cache with one bash command.

How totaly purge Rack::Cache in memcache even if it shared with Rails cache.

Whats wrong with responding Cache-Control: max-age=N, where N greater than zero.

How to run fresh_when/stale? properly if @resource is absent.

Complete reference on HTTP requests cache on nginx + rails stack in one place, сonfigs for rails and nginx and more.

Intro

HTTP request caching is powerful tool wich can improve perfomance dramatically. Serving requests from nginx cache can be done 10K-times faster than rendering them in your rails application.

If you want quick results and conculsions you may skip the longread and go for final sections with brief resume and configs.

If you want to properly understand and manage HTTP caching you must know:

The meaning of ETag, Last-Modified, Cache-Control headers.
How they generated and treated along the way from rails through rack and nginx to client-browser.
The way rack and nginx cache requests, and how it can be configured and managed.
How to purge or at least simulate purging cache.
How to debug cache.

All theory parts are given with practical comments on real behaviour of stack components, so it’s useful also to read them through. I tried to make this long read as a complete answer to all possible questions on http caching on rails +nginx stack.

All results and examples are given in the context of rails 5.0.1 and nginx 1.10.1, so nuances may vary with version changes.

Cache-Control, Last-Modified, ETag headers.

Http cache managed with three main headers: Cache-Control, Last-Modified, ETag. Also there were Pragma and Expires, but they obsolete now, though for backward capability they are usually also placed in headers. Also in context of caching nginx respects a custom header X-Accel-Expires.

ETag — entity tag, it’s something like a stateful id for a given resource, it may seem natural for ETag to be some kind of resource hash, but it’s not always true. For example nginx for static files doesn’t calculate hash, and produce ETag as concatenation of file length in byte and last modification datetime. So you are getting yourself a new ETag everytime you touching assets while deploying your rails app.

ETag may go with ‘w’ prefix meaning ‘weak’ ETag, which indicates that the two representations of a given resource are semantically equivalent, not byte to byte.

ETag can be generated manually inside rails controller, automatically inside Rack, and by Nginx.

Rack skips ETag auto-creation in this three cases:

def skip_caching?(headers)
  (headers[CACHE_CONTROL] && 
     headers[CACHE_CONTROL].include?('no-cache'))||
   headers.key?(ETAG_STRING) || headers.key?('Last-Modified') 
end

Rack default ETag implementation generates a hash on a full rails response, meaning only bandwidth economy, and useless for perfomance, since you need to fully render an action to obtain ETag for comparision.

There is a complementary header to ETag for request from client browser: If-None-Match. It equals previously recieved ETag for a given URL.

Last-Modified — resource modification date. It also has a complementary header for client request: If-Modified-Since.

ETag, Last-Modified and their complementaries are used in conditional GET request, when server may respond with 304 Not Modified. Also Nginx uses them while serving static files.

X-Accel-Expires — custom header respected by nginx cache and ignored by Rack::Cache.

Cache-Contol — main header for cache controling on the way from application to a client browser. Doesn’t has complementary header and used inside request and response header, but with different meanings.

Cache-control: max-age, must-revalidate, no-cache

Responses

Cache-Control actually quite a confusing header, I especially like this qoute on it:

no-cache doesn't mean "don't cache", it means it must check (or "revalidate" as it calls it) with the server before using the cached resource. no-store tells the browser not to cache it at all. Also must-revalidate doesn't mean "must revalidate", it means the local resource can be used if it's younger than the provided max-age, otherwise it must revalidate. Yeah. I know…

Cache-Control header in responses may include:

max-age=[seconds] — max time in seconds while response considered fresh, and can be served without involving web-app. This is not an absolute datetime value, but a time relative to previous request.

If max-age=N is given than browser doesn’t send request next N seconds on refresh or second visit, except for hard refresh.

If there wasn’t max-age ( ) in a rails response than nginx can cache only if proxy_cache_valid directive is present in its config.

If there wasn’t any max-age in Cache-Control recieved by clients browser than behaviour depends on nature of a response. If its a page than no caching in browser is happend. If it’s a file, like js, css, picture e.t.c. than modern browsers (test in Chrome and Opera) will cache it with some default long enough max-age. One or two years ago if you forgot to add expiration to assets they were requested each time you open new page, with conditional GET of course, but still.

s-maxage=[secs] — like a max-age only for proxies.

public — means response can be cached within proxies. If you not setting it manually than rails responds with private by default.

private — means response must not be cached outside application and client-browser.

no-cache — response must revalidate itself on server every time it’s served.

no-store — response must not be cached everywhere including browser.

must-revalidate — means it must always revalidate when max-age exceeded.

proxy-revalidate — like must-revalidate, but for proxies.

Is must-revalidate, max-age=0 different from no-cache? Yes but it’s quite confusing.

In requests

In contrast to ETag and Last-Modified, Cache-Control doesn’t have complementary header and it used inside requests with different meaning than in a response. Only max-age=0 and no-cache are available for request from client browser. Cache-Control: max-age=0 is for refresh, and Cache-Control: no-cache is for hard refresh (with a shift).

Actually Nginx cache ignores hardness of such refresh and Rack::Cache respect it only with proper config. So thats the reason I’m not sure you need to dig them deeper inside standard and again in other part of the standard.

Cache-Control live from web.

medium.com:

github.com

default rails public

default rails when logged in

Cache Patterns

Two solid cache patterns exists based on described headers ( I’ll quote original text with some additions ):

Pattern 1: Immutable content + long max-age

Cache-Control: max-age=31536000

The content at this URL never changes, therefore…
The browser/CDN can cache this resource for a year without a problem
Cached content younger than max-age seconds can be used without consulting the server

It’s the ideal pattern for all of the assets since they are finger-printed.

Mutable resource usually cannot be served with max-age other than 0. For instance imagine authorization process with max-age=N greater than zero: you already logged in but still N second browser will serve you locally cached guest version of main-page.

Pattern 2: Mutable content, always server-revalidated

Cache-Control: no-cache

The content at this URL may change, therefore…
Any locally cached version isn’t trusted without the server’s say-so

In this pattern you can add an ETag or Last-Modified date header to the response. Next time the client fetches the resource, it echoes the value for the content it already has via If-None-Match and If-Modified-Since respectively, allowing the server to say "Just use what you've already got, it's up to date", or as it spells it, "HTTP 304".

If sending ETag/Last-Modified isn't possible, and Rack is configured properly, than the server always sends the full content.

This pattern always involves a network fetch, so it isn’t as good as pattern 1 which can bypass the network entirely.

Mutable resource and fast expiration

Is it possible to cache mutable resource with expiration?

It can be done with maximum caution, above I gave an example when you changing your state from guest to authorized user, which shows 100% incorrect behaviour with max-age > 0. But when resource representation is impersonal, and it’s not so crucial for it to be absolutely-fresh max-age can be greater than 0.

For example github api:

Second correct option: cache on nginx level with very short expiration, and serving Cache-Control without any max-age.

How short expiration can be?

It can be just one second, and this is quite many, since nginx can serve more than 30K request per second from cache in average with application rendering or revalidation. See “Nginx perfomance max” section.

It can be even “zero seconds” see section “Fine tuning of nginx cache”

Managing headers inside Nginx

Affects Nginx

proxy_ignore_headers instruct nginx to ignore recieved headers.

Affects rails

proxy_set_header changes header before send it to rails, also can be used to manage cache, for example imitating hard-refresh.

Affects client-browser

add_header adds header to response, in this point we can set Cache-Control header affecting only clients cache. It’s a good place to set Cache-Control withno-cache and/or max-age=0, must-revalidate.

proxy_hide_header removes header from response.

expires sets expiration headers Cache-Control: max-age, Expires, it can be useful for assets files.

If you need to change response header inside nginx you can subsequently call proxy_hide_header and add_header. If you do this for Cache-Control, new value will be ignored by nginx cache, but will be respected by client-browser or other proxy on the way to client.

Enabling/Disabling nginx cache base

If you need intro on nginx caching, you can read it here.

Base enabling directives are: proxy_cache_path, proxy_cache, proxy_cache_key.

You can disable caching with proxy_no_cache directive.

Example:

proxy_cache_path /path/to/cache levels= keys_zone=rails_cache:10m max_size=10g use_temp_path=off inactive=120m;# levels= - means that nginx cache will not use subdirectories, and will place all caches directly in /path/to/cache wich may be useful for debuglocation / {
  proxy_cache rails_cache
}

Nginx cache managing through headers.

X-Accel-Expires, Expires, Cache-Control, Set-Cookie, and Vary set the parameters of response caching

NGINX respects the Cache-Control header values: public, private, max-age, no-cache, no-store, unless it’s configured not to.

It means when nginx cache enabled with proxy_cache_pathand proxy_cache and recieved Cache-Control: public, max-age=60, Nginx will cache response for one minute.

Nginx also respects Set-Cookie header by skipping cache.

X-Accel-Expires over Cache-Control

Since X-Accel-Expires have priority on Cache-Control we can manage nginx expiration cache from rails, bypassing Rack::Cache.

#somewhere in controller
response.headers['X-Accel-Expires'] = 10 # nginx will cache for 10s

Can Cache-Control be ignored?

Nginx guide says: “Yes, it can!”

Lets quote an example:

location /images/ {
  proxy_cache my_cache;
  proxy_ignore_headers Cache-Control;
  proxy_cache_valid any 30m;
  ...
}

NGINX ignores the Cache‑Control header for everything under /images/. The proxy_cache_validdirective enforces an expiration for the cached data and is required if ignoring Cache‑Control headers. NGINX does not cache files that have no expiration.

Be aware of that proxy_ignore_headers doesn’t hides Cache-Control and proxy_cache_valid doesn’t change it, so browser may recieve and use it to manage cache.

Fine tuning Nginx cache ( from nginx cache guide )

As a guide states:

proxy_cache_revalidate instructs NGINX to use conditional GET requests when refreshing content from the origin servers. If a client requests an item that is cached but expired as defined by the cache control headers, NGINX includes the If‑Modified‑Since field in the header of the GET request it sends to the origin server. This saves on bandwidth, because the server sends the full item only if it has been modified since the time recorded in the Last‑Modified header attached to the file when NGINX originally cached it.

Rem: Actually NGINX uses for revalidation only what it got from rails, if rails send ETag and didn’t send Last-Modified than cache will be revalidated with ETag and If-None-Match.

proxy_cache_min_uses sets the number of times an item must be requested by clients before NGINX caches it. This is useful if the cache is constantly filling up, as it ensures that only the most frequently accessed items are added to the cache. By default proxy_cache_min_uses is set to 1.
The updating parameter to the proxy_cache_use_stale directive instructs NGINX to deliver stale content when clients request an item while an update to it is being downloaded from the origin server, instead of forwarding repeated requests to the server. The first user to request a stale file has to wait for it to be updated from the origin server. The stale file is returned for all subsequent requests until the updated file is fully downloaded.
With proxy_cache_lock enabled, if multiple clients request a file that is not current in the cache (a MISS), only the first of those requests is allowed through to the origin server. The remaining requests wait for that request to be satisfied and then pull the file from the cache. Without proxy_cache_lock enabled, all requests that result in cache misses go straight to the origin server.

Cache only while rails is busy with request

One of the possible way to cache uncacheable is to set this combination of directives:

proxy_cache_lock on;
proxy_cache_valid any 0s;
proxy_use_stale updating;

also proxy_cache_revalidate can be added to this configuration.

Even though expiration time set to zero you can get UPDATING¹ and even HIT if you ping hard.

¹For meaning of HIT and UPDATING read “Debug cache” section and Reference on $upstream_cache_status values.

Static files in NGINX

You can skip this part if you delivering your assets through CDN.

Static files served based on second caching pattern: using ETag and Last-Modified. So it’s directly depends on how Nginx calcs them:

ngx_sprintf(
  etag->value.data,
  "\"%xT-%xO\"", 
  headers_out.last_modified_time, 
  headers_out.content_length_n
)

This code sample shows that ETag depends on Last-Modified and not on the file content. Older browser (may be till 2015) didn’t cache static files at all without additional config from nginx, every time rerequesting them from server. Yes it was conditional GET, but still you download yourself ‘new’ assets on every deploy with a bonus of couple 304 request for every page. Nowdays it’s gone and even if max-age is absent in response current browsers ( checked in Chrome and Opera) will cache files at least for a couple of days. If you want to be sure you may configure nginx with expires directive.

One more directive will be very useful for rails assets: gzip_static. It permits Nginx to serve static files using pre-compressed files without live compression of original files at the time of serving. Since gzipping on assets precompilation is back to rails we can use gzip_static .

Nginx conf example for assets:

location /assets {
  expires 1y;
  gzip_static on; 
}# without precompressed files:location /assets {
  expires 1y;
  gzip on; 
  gzip_types *;
}

If you want to control ETag you may use this conf:

location /assets {
  if ( $http_if_none_match = $uri ) {
    return 304;
  }  
  proxy_hide_header ETag;
  add_header ETag $uri; # since assets are finger-printed its OK
  
  expires 1y;
  gzip_static on;
}

Rack::Cache reference

Rack::Cache is a middleware which implements caching of HTTP requests on Rack level, i.e. conditional GET and expiration caches. More about Rack::Cache: general info, settings, storages.

Two things you better give more attention: storages concept and little less interesting is a config.

About storages you should know three things:

they may be shared between processes and even servers
they may be persistent on application restart and update
you can use memcache as a storage ( or even implement your own superfast storage)

As for settings you may pay attention to allow_reload and allow_revalidate, even if you’ll probably skip them:

allow_reloadis a boolean specifying whether reload requests sent by the client should be honored by the cache. When this option is enabled (rack-cache.allow_reload is true), requests that include a Cache-Control: no-cache header cause the cache to discard anything it has stored for the request and ask that the response be fully generated.
Most browsers include a Cache-Control: no-cache header when the user performs a “hard refresh” (e.g., holding Shift while clicking the “Refresh” button).
allow_revalidateis a boolean specifying whether revalidate requests sent by the client should be honored by the cache. When this option is enabled (rack-cache.allow_revalidate is true), requests that include a Cache-Control: max-age=0 header cause the cache to assume its copy of the response is stale, resulting in a conditional GET / validation request to be sent to the server.
Most browsers include a Cache-Control: max-age=0 header when the user performs a refresh (e.g., clicking the “Refresh” button).
IMPORTANT: Enabling this options globally allows all clients to break your cache.

Nginx can construct or block Cache-Controle while passing it to application.

Cache purge, bypass and debug.

Nginx cache purge

If you are using free version of Nginx and suffering of any special purging abilities of Nginx+, you can purge you nginx cache by clearing /path/to/cache directory referenced in proxy_cache_path, except for temp directory ( even if you configured your cache with use_temp_path=off ), otherwise you will need to restart nginx or reload config!

It can be done with bash command:

rm -r -f /path/to/cache/!(temp)

Rack::Cache purge

Guides states that impossible to purge content manually, but actually you can purge on key by key basis, but not by namespace. Memcache also doesn’t provide namespaced purging. So what you could do with purging when you using something more complicated than HeapStore, which is purged on application restart.

You can emulate purging by changing namespace for your stores.

Namespaces can be constructed automatically, for instance like this:

Bypass

proxy_cache_bypass — is a directive for bypassing NGINX cache.

Bypassing Rack::Cache is a matter only when you using expires_in which you should not. But in case you do, it can be bypassed in two steps: enable allow_validate/allow_reload and emulating Cache-Control: no-cache/max-age=0 in nginx conf with set_proxy_header directive.

Debug Cache

Suitable way to debug cache is to use headers. What you should check when something went wrong:

Set-Cookie, if set Nginx doesn’t cache, unless Set-Cookie ignored.
Cache-Control: private, no-cache, no-store — Nginx doesn’t cache unless Cache-Control ignored.
If you overriding them with proxy_hide_header+add_header you better inspect previous headers values, see example below.
You definetly need to output $upstream_cache_status and check for nginx cache status.
Inspect X-Rack-Cache header

Example:

# output cache status
add_header X-Cache-Status $upstream_cache_status;# output Cache-Control recieved from rails
add_header X-Cache-Control $upstream_http_cache_control;

$upstream_cache_status returns² HIT, STALE, UPDATING, REVALIDATED, EXPIRED, BYPASS, MISS

HIT, STALE, UPDATING — means content delivered from nginx cache.

REVALIDATED — a conditional GET where passed to the rails and a cached content is fresh now.

EXPIRED, BYPASS, MISS — request reached rails and response was some how served. Either rendered by rails or served by Rack::Cache. To be sure what happens look for X-Rack-Cache header.

X-Rack-Cache return value example:

X-Rack-Cache:stale, valid, store

²More on $upstream_cache_status read inside reference part.

Inside Rails

Rule out the session-ecosystem

If you want to use nginx caching with rails you must prevent session cookie from being set, since ignoring Set-Cookie at nginx levels and caching response when Set-Cookie being used is THE WORST idea.

Preventing session from being set you will also brake forgery protection, flash logic and devise gem in your app, so you cannot ignore session globally, but you can ignore them using before_action or after_action callback with any condition you need.

after_action skip_session_for_public_cachedef skip_session_for_public_cache
 if !keep_session? 
  request.session_options[:skip] = true
  #if you using cookie_store you need to delete session cookie!
  cookies.delete('_rails_app_session')
 end
end

Upd: Full case and workarounds for session ecosystem I strongly recommend you to read here.

expires_in, fresh_when, stale?

Detailed description in heroku tutorial ( by the way they give very poor example on stale? method usage, the right one is given below and of course in rails code ), description in rails api-docs, and also raw code on github.

Shortly:

expires_in — sets Cache-Control headers with max-age, public/private values.

fresh_when — sets ETag + Last-Modified and public/private in Cache-Control. fresh_when can be called in callbacks or in controller action, be aware that fresh_when called inside before_action can stop request processing down the event chain and prevent controller action from execution, if request is fresh.

stale? — inside its just two subsequent calls: fresh_when and !request.fresh?(response). There are lots of misleading examples on this method, real example when its useful are given in rails code and rails docs:

if stale?(@resource)
 @statistics = @resource.really_expensive_call # the only use case!
  respond_to do |format|
   # all the supported formats
  end
end#useless way to use it
if stale?(@resource)
 respond_to do |format|
   # all the supported formats
  end
end

Pay attention to the way fresh_when works when you want ETag based on templates and not the resource i.e. when action is resourceless or resource is nil:

fresh_when( nil ) # wrong way to do, ETag will be generated inside Rack with default implementation, regardless of action template. Never hit a conditional GET without full render of actionfresh_when( nil, template: "#{controller_path}/#{action_name}" ) # right way to do, ETag will be created based on view and dependent partials!

Never use Expires_in.

Why you should never use expires_in inside rails + Nginx stack? In two words it makes very hard to bypass all caches and involve application logic .

Lets review an example:

before_action :cache_with_expiration, only: [:home]def cache_with_expiration
  expires_in( 1.minute, public: true )
end

What do we got here? First of all we got Cache-Control: max-age=60, public. Now both Rack::Cache and Nginx will cache this response for a minute.

What you should do to bypass them and involve rails logic:

First of all you’ll need to hide in nginx Cache-Control: max-age=60, public with hide_header directive, otherwise browser will serve page from cache for a minute.
In Rack::Cache config you must set allow_reload or allow_revalidate to true, or Rack::Cache will serve its copy of content.
Inside nginx we must add bypass_cache with desired condition
If request doesn’t contain Cache-Control wich can go through allow_reload or allow_revalidate you must use set_proxy_header to add missing value ( either max-age=0, or no-cache ).
You will need to purge Rack::Cache when deploying new version of application, this is not easily done, see “Rack::Cache purge” section.

Keeping expiration cache in two places is a baaad idea. Nginx cache in my opinion is much better place for it, than Rack.

Additional rails gotchas when caching HTTP-requests: CSRF-token and cookies.

By default Rails will add CSRF-token in application layout, it means problem with posting forms from cached pages. If we want them to work properly we need to skip CSRF-token verification.

skip_before_action :verify_authenticity_token, only: [:some_action]

Second gotcha is a cookies. If you set cookie in the action, Nginx cache will skip caching! You must controll it not only in your code but in gems also³. Look for Set-Cookie header in response to be sure that this is not the reason nginx delivers you a MISS instead of HIT.

³rack-mini-profiler sets cookie for instance.

Nginx perfomance max

Here lies nice pdf on NGINX scalability and perfomace, it states max perfomance of NGINX for 45К requests per second per core. Thats quite a number!

For comparision Rack::Cache returns response in ~5ms it means ~ 200 request per second. And rails render speed is hard to predict simple page can be rendered in 20–50ms, complex in 200ms and higher.

Lets say we want to cache just for 1 second, here some numbers :

If rails renders in 0.2s ~ 5 requests per second, than with nginx cache stack can serve ~37К⁴ requests per second average.
If rails renders in 0.33s ~ 3 requests per second, than with nginx cache stack can serve ~33К requests per second average.
If rails renders in 0.4s ~ 2.5 request per second, than with nginx cache stack can serve ~32К requests per second average.

Of course this is unreacheable numbers but they show the four degree difference between serving requests from Nginx cache and rendering them with rails. So even one second caching can do the work.

⁴ (45K + 1)/(1 + 0.2)s ~ 37K per second.

Finally how you cache the uncacheable?

If resource is impersonal than you can use example with zero seconds in proxy_cache_valid, if you need to cache request with user specific data than you need to use Server Side Includes module.

Resume

Nginx can serve up to 45К requests per second it means that 1 second of nginx-cache can save one and a half hour of rails rendering.
Combination of short and even ‘zero’ expiration cache on nginx and Etag-based cache inside Rails looks like the best approach for HTTP-request caching.
If you decide to use cache be aware of session ecosystem problems and workarounds.
Never use expires_in in rails
When callingfresh_when or stale?, don’t forget nuances of them in resourceless case, in before_action and the one real usecase for stale?.
If you want to control nginx cache expiration time from rails you can use
X-Accel-Expires header.
There are two ways to cache the uncacheable: impersonal request with this config:

proxy_cache_valid any 0s;     
proxy_cache_use_stale updating;        
proxy_cache_lock on;

or Server Side Includes module, if request contains user dependent data.

8. You must know whats you are doing when serving Cache-Control: max-age=N with N > 0 otherwise you’ll struck with browser cache, look for examples in Cache patterns section. In all other cases serve to clientCache-Control: no-cache, or something like it.

9. Compress static assets and use gzip_static on;

10. Nginx cache may be purged with bash: rm -r -f /path/to/cache/!(temp)

11. Use memcache for Rack::Cache storages.

12. Use custom headers for cache debug. You definitely must output to custom headers$upstream_cache_status. And remember:HIT, STALE, UPDATING — served from cache. REVALIDATED — verified through conditional GET, all others — goes through rails.

Configs and code examples

Rails assets:

Rails

config.assets.gzip = true # must be true or default.

Nginx

location /assets/ {
  expires 1y;     # if you want to be sure
  gzip_static on; # it's much better than live-gzip
}

For pages

Rack

Configure Rack::Cache to use memcache as store. Add gem ‘dalli’ to Gemfile, and than config storage:

config.middleware.use Rack::Cache,
  :metastore   => 'memcached://localhost:11211/meta',
  :entitystore => 'memcached://localhost:11211/body'# orconfig.action_dispatch.rack_cache = {
 metastore: 'memcached://localhost:11211/meta',
 entitystore: 'memcached://localhost:11211/body'
}

Config storage for heroku:

client = Dalli::Client.new((ENV["MEMCACHIER_SERVERS"] || "").split(","), :username => ENV["MEMCACHIER_USERNAME"],
   :password => ENV["MEMCACHIER_PASSWORD"],
   :failover => true,
   :socket_timeout => 1.5,
   :socket_failure_delay => 0.2,
   :value_max_bytes => 10485760)config.action_dispatch.rack_cache = {
  :metastore    => client,
  :entitystore  => client
}

If you need direct access to store inside rails:

Rack::Cache::Storage
   .instance
   .resolve_metastore_uri('memcached://localhost:11211/meta')

Rails