Mixlr — the best live audio platform out there. Check it out

[Devblog] Improve your beta-testing with Nginx, Lua and Redis

At Mixlr, we routinely want to test new and sometimes complex features on a small subset of our user base. Until now, we had two options. We could merge the new code into our master branch, and somehow separate beta-testers from non-testers at the application level. Alternatively we could keep the new code separate, and deploy it to a subdomain like beta.mixlr.com.

Both approaches are problematic. It is often undesirable to merge complex changes into master while they are part-finished and unproven on real users. And sending beta-testers to a separate website introduces too much friction, and may introduce bias into a user's experience. We need a system which can route between multiple web applications and runs transparently on mixlr.com.

This post describes how Mixlr has used Nginx, Lua and Redis to create a lightweight, extensible and powerful system which allows us to transparently route users to different pools of backend web servers based on some arbitrary criteria.

Mixlr's routing system

Based on the first segment of the URL requested — in our case, a user's slug — the visitor will be served pages from either our master or beta branch, as appropriate. We will also see how it is trivial to change this to any other criteria we choose.

The post starts by describing the product and business requirements which led us to design and build the system. This allows us to evaluate the technical choices we have made in greater context. We go on to describe the technical implementation in detail, including example Nginx configurations.

Background

Mixlr is a social live audio platform which is widely used by podcasters, musicians and other broadcasters. Unlike Turntable.fm, Mixlr gives an individual broadcaster total control of their room - there is no gamification or fighting to get on the decks. We also enable real-time streaming from arbitrary sources such as a microphone, rather than just playing pre-recorded music MP3s.

Mixlr crowd page

Naturally, we also give our users the opportunity to record and publish their live shows, but until now they were only available for on-demand playback. We've never felt that on-demand playback is a good match for Mixlr's philosophy of creating a real-time, shared listening experience on each page.

Showreel Radio is our answer to this problem. It will create a 24/7 live stream of a user's recorded broadcasts. In future, visitors to a Mixlr profile page will hear that user's Showreel Radio by default.

This is a complex feature which represents a big change to our users' Mixlr pages, so we need to handle the roll-out carefully. In particular, we want to turn on Showreel Radio for a few users at a time, until we are completely satisfied that it works perfectly.

So our routing system:

How we built our routing system.

At Mixlr we use Nginx as a front-end web server. Our Nginx config looks something like this:

# Our backend Rails servers, running our master branch:
upstream mixlr {
  server localhost:8000;
  server localhost:8001;
  # etc
}

# Our backend Rails servers, running our beta branch:
upstream beta {
  server localhost:8020;
  server localhost:8021;
  # etc
}
 
server {
  server_name mixlr.com; # You might want to change this
  listen 80;

  location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
 
    # just serve static files like images, without bothering Rails
    if (-f $request_filename) { 
      break; 
    }
 
    # .. some caching configuration 

    proxy_pass http://mixlr;
  }
}

The important bit is the last line: proxy_pass sends any request which can't be matched to a static asset file, to one of the Rails processes defined in the mixlr upstream block.

We want to extend this so that any visitors to http://mixlr.com/user1, who is not beta-enabled, are routed to servers running our master branch. And that visitors to http://mixlr.com/user2, who is beta-enabled, are routed to servers running our beta branch.

Unfortunately, without separating the requests using something like a different subdomain, Nginx does not have the capability to perform this task out of the box.

To solve it we will use the Nginx Lua module. Lua is a lightweight, general-purpose programming language. It is also designed to be embedded: for us, having Lua within Nginx means that we can create write our own Nginx extensions which enable us to handle the above routing in any way we choose.

In this example, we also make use of the Nginx redis2 module and the Nginx-set-misc module.

Installing the Nginx modules

None of the custom modules made use of by our routing system are included in Nginx core distribution.

You can get going quickly by installing the openresty web app server, which bundles Nginx with a number of third-party plugins, including all the ones we need.

Basic Nginx configuration

First we need something to route to. Let's assume we have some Rails servers running our master branch on ports 8000 and 8001, and some more running our beta branch on ports 8020 and 8021.

Next, set up some Redis configuration. We will use Redis to allow us to determine which Mixlr users are part of the beta test. If a user's slug is contained in a certain Redis set, all visitors to their Mixlr page will be routed to the beta site.

If you don't use Redis in your own app, it should be fairly easy to switch this out for some other logic (the non-blocking MySQL Nginx module, for example).

We add this directly underneath the new upstream block.

# Tell Nginx about where our Redis is.
upstream redis {
  server localhost:6379; # Shrink to fit
  keepalive 1024 single;
}

We also need to set up some Nginx location blocks which define the particular Redis queries we are going to make. Inside the server block:

# Here we define the Redis query we will use to check if a particular user has enabled
# beta features for their account:
location /redis_check_for_beta {
  internal; 
  set_unescape_uri $username $arg_username;
 
  # Is this username in the Redis set "beta:enabled"?
  redis2_query sismember beta:enabled $username;
 
  redis2_connect_timeout 200ms;
  redis2_send_timeout 200ms;
  redis2_read_timeout 200ms;
  redis2_pass redis;
        
  error_page 500 501 502 503 504 505 @redis_error;
}
 
# If a Redis connection error occurs, this will just disable Showreel Radio 
# instead of causing a 500 error for the user:
location @redis_error {
  internal;
  content_by_lua 'ngx.print("ignore_this_error");';
}

Now, we're ready to do some Lua coding.

Writing Lua inside an Nginx config file

The Lua module allows us to write snippets of Lua code right inside our Nginx configuration. I believe the easiest way to explain it is to comment it thoroughly, so here we go.

# We want to match any request which starts with a username. The brackets capture the username automatically in the Nginx variable $1:
location ~ ^/([a-zA-Z0-9_]+)/ {
  # First, let's give the username a more memorable variable name.
  set $username $1;

  # Set a default root: this is the path to our old Rails branch.
  set $root "/www/mixlr.com/current/public";
  # Set a default backend: again, the Rails servers we defined running our old branch.
  set $backend "http://mixlr";
 
  # Any proxy configuration we need, and other Nginx config if required. 
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;

  # Now comes the embedded Lua code. :)
  rewrite_by_lua '
    # Inside this block we have an entirely new syntax: Lua.
    # Start by grabbing the Nginx username value into a local Lua variable.
    local username = ngx.var["username"];

    # Here we hit Redis using the location block we defined before. 
    local result = ngx.location.capture("/redis_check_for_beta", { args = { username = username }});

    # Redis server returns a string :1 followed by a carriage return, if this user is beta-enabled:
    if result.body == ":1\\r\\n" then
      ngx.log(ngx.NOTICE, "Detected beta:enabled for user: ", username, ", will now route to beta site.");
      
      # Now, override the backend server. This matches the upstream block defined right at the top:
      ngx.var.backend = "http://beta";
      # And change our document root to match:
      ngx.var.root = "/www/beta.mixlr.com/current/public";
    end
  ';
  
  # Now, $root and $backend are definitely set correctly and we can route to one backend or another :)
  root $root;
  if (!-f $request_filename ) {
    proxy_pass $backend;
  }
}

Now just restart Nginx.

Convenience methods

We will make life easier for ourselves and add some convenience methods to our User class in Rails.

class User < ActiveRecord::Base
  # ...

  def beta?
    $redis.sismember("beta:enabled", to_param)
  end

  def add_to_beta
    $redis.sadd("beta:enabled", to_param)
  end

  def remove_from_beta
    $redis.srem("beta:enabled", to_param)
  end
end

We can use these methods to allow our dev team, or our users themselves, to add or remove themselves from the beta at any time.

Conclusion

This project matched everything we look for in a one-day project at Mixlr. It was fun to learn about and build, as well as technically challenging.

More importantly, the result has a great amount of business value. Our new routing system will help our users adjust to a challenging new feature, increase the amount of control they have over their own experience on Mixlr, and additionally protect them from technical problems as we transition to new releases.

It also opens up a world of possible future extensions: we can imagine an A/B-testing framework implemented within Nginx using similar principles.

We'd love to hear your feedback on our choice of technologies, implementation and product decisions. Contribute below, or on Hacker News.

If you liked this article, then you'll love following the Mixlr dev team on Twitter.

26 June 2012