TheWebFellas - Ruby on Rails Development Specialists UK - Blog

Amazon launches their CDN - CloudFront

2008-11-18T11:23:00Z

Amazon have now launched their Content Delivery Network - CloudFront

In line with the other offerings from Amazon Web Services then the product is based around Usage Based Charging - with the prices varying between Edge locations

There are Edge servers located in the United States, Europe, Hong Kong and Japan.

What is it?

From the Amazon blurb:

Amazon CloudFront is a web service that gives businesses and software developers an easy and cost effective way to distribute popular content with low latency and high data transfer speeds. Like all AWS Infrastructure Services, Amazon CloudFront is a self-service, pay-per-use offering, requiring no long term commitments or minimum fees. With CloudFront, your files are delivered to end-users using a global network of edge locations. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3), which durably stores the original, definitive versions of your files.

How does it work?

The CloudFront service is backed by usage of S3. You store your content in an S3 bucket and register the bucket with the CloudFront service. This creates a 'distribution' of the content you have registered. A distribution has a unique domain name associated with it you can access your cached content - you can also map your own name to this via a CNAME record.

When users request content via this distribution, they will be automatically routed to the nearest Edge location to retrieve the content.

The API

As with the other AWS tools then you will need to add it to your account.

The Getting Started Guide and Developer Guide give you all the details you need to use the service via the API.

Goodbye attachment_fu, hello Paperclip

2008-11-02T10:40:00Z

For well over a year now attachment_fu has been my plugin of choice for adding file uploads to our Rails applications, but recently my fellow WebFellas have been raving about Paperclip from the clever guys at thoughtbot. As I’ve just started yet another new project I figured it was time to take Paperclip for a spin.

The obligatory special instructions for Windows

If, like me, you’re a long-suffering Rails developer using Windows (those new Macbooks are looking very tempting!) then you won’t be surprised to learn that getting Paperclip up and running needs a few special steps.

Install the plugin

You can install Paperclip as a regular Rails plugin from GitHub from a command prompt window like this:
```
ruby script/plugin install git://github.com/thoughtbot/paperclip.git
```
You’ll obviously need a working copy of Git: thoughtbot switched off their SVN repository this week and the gemified version of Paperclip on RubyForge is quite old now and probably shouldn’t be used (gems via GitHub are not currently available).
Download ImageMagick

Paperclip uses ImageMagick to perform image processing, which is thankfully much easier to install on Windows than ImageScience was for attachment_fu!

You’ve got two options:
- Use the RMagick gem
  
  Although Paperclip doesn’t use the RMagick gem, if you think you’re ever likely to then you should download the latest gem from RubyForge and use the version of ImageMagick that is bundled with it (see the Readme.html file included in the download for installation instructions).
- Download the latest version of ImageMagick
  
  If you’re not bothered about RMagick support then download and install the ImageMagick installer (I used the 16 bits per pixel dynamic library without any problems).
Whichever installation method you choose make sure you select the ‘Update executable search path’ option in the ImageMagick installer.
Patch the Tempfile class

Important: after looking into this a bit more closely I’m no longer 100% convinced that patching Tempfile is necessary for Paperclip. I’d recommend you first try using it without this patch and if you start to see problems then try the patch (and it’d also be great if you could post a comment to let me know that you needed the patch).

Paperclip makes extensive use of temporary files via the Ruby Tempfile class. Unfortunately this can cause some strangeness in Windows where the file size is incorrectly reported as zero bytes (attachment_fu suffered with similar problems). The solution is to monkey-patch Tempfile like this:
```
require 'tempfile'
class Tempfile
  def size
    if @tmpfile
      @tmpfile.fsync
      @tmpfile.flush
      @tmpfile.stat.size
    else
      0
    end
  end
end
```
This code comes from Emmanuel Pirsch and has worked well for me in several projects. I typically drop it into my lib/patches folder and require it from an initializer.
Shutdown, close, reopen and restart

Make sure you shutdown any running servers, close any console windows and then reopen and restart as needed: it’s important that your consoles and servers pick up the updated PATH environment variable set by the installer as Paperclip calls ImageMagick using the command line (which is why it doesn’t need RMagick).

Where to begin

At this point you should be ready to start using the plugin. I won’t cover the basics of using Paperclip with Rails as they’ve already been explained on the official project page as well as in Jim Neath’s excellent tutorial and more recently in a RailsCast from Ryan Bates. However there are some newer features and tips that you should know about to get the best out of Paperclip.

Protect your attributes

If you’ve used attachment_fu then you’re probably aware that older versions didn’t like attr_protected. The good news is that not only has attachment_fu recently been fixed, but Paperclip also plays nicely with both attr_protected and attr_accessible too. So to play it safe you should protect the file_name, content_type and size attributes in your models, for example:

has_attached_file :logo
attr_protected :logo_file_name, :logo_content_type, :logo_size

Attaching in migrations or the Rails console

Attaching a file to a model in a migration or when using the Rails console is much easier with Paperclip than it is with attachment_fu. All you need to do is open a File object and assign it to the attachment attribute of the model:

user = User.first
File.open('path/to/image.png') { |photo_file| user.photo = photo_file }
user.save

Of course that works on pretty much every operating system in the world except Windows, where the assignment will cause an error like this to be output in the console window:

identify: no decode delegate for this image format `C:/DOCUME~1/ROB~1.CML/LOCALS~1/Temp/stream.2692.0'.

In addition to this cryptic message, the file isn’t attached properly and no thumbnails are generated. Thankfully the solution is simple (although it did take me a few moments of head scratching to figure it out): set binary mode on the file to stop Windows treating it like a text file:

user = User.first
File.open('path/to/image.png', 'rb') { |photo_file| user.photo = photo_file }
user.save

Discarding an uploaded image

When an image file is uploaded Paperclip stores it using the original style name. While it isn’t possible to tell Paperclip to simply discard the original image, it is possible to define your own original style which Paperclip will then use. As an example, if your model contains the following:

has_attached_file :photo, 
                  :styles => { :original => '250x250>', 
                               :small => '50x50' }

Then even if a huge photo (say 1600x1600) is uploaded, it won’t be stored on your server (or S3 bucket). You will however still have an original file but it will have been resized to 250x250, helping to reduce the amount of storage space needed for attachments.

Styles can be Procs too

Most of the time you’ll probably be passing geometry strings for the has_attached_file :styles option, but a very nice feature is the ability to also pass a Proc to provide dynamic geometry string generation. For example, you might have a model that has photo_width and photo_height attributes that can be specified by the user and you want to generate a ‘custom’ thumbnail that uses these dimensions. Paperclip allows you to do this easily:

has_attached_file :photo, 
                  :styles => { :original => '250x250>', 
                               :small => '50x50', 
                               :custom => Proc.new { |instance| "#{instance.photo_width}x#{instance.photo_height}>" } }

As you can see the Proc receives the object that the attachment is part of as a parameter and returns a geometry string generated using the photo_width and photo_height attributes. Adding a call to the reprocess! method of the attachment object in your model’s before_save callback also allows you to regenerate the thumbnails if either of these attributes are updated.

Rename files on upload

The has_attached_file :url and :path options can be used to customise the name of your attachments: :url defines the URL that will be used by things like image_tag to access your image and allows you to either provide direct access to the image or to route access through a controller (to provide permission checking for example) and :path determines where the image file is stored either on your server (for file system storage) or in an S3 bucket.

Both options use interpolation, allowing you to use special tags that will be replaced with actual values at runtime (just like regular Ruby string interpolation). The default interpolations provided by the plugin are:

<dl>

:rails_root

The path to the Rails application.

:rails_env

The current environment (e.g. development, production)

:class

The class name of the model that the attachment is part of, underscored and pluralised for your convenience.

:basename

The name of the originally uploaded file without its extension.

:extension

The file extension of the originally uploaded file.

:id

The ID of the model that the attachment is part of.

:id_partition

The same as :id but formatted as a string using ID partitioning.

:attachment

The name of the attachment attribute (defined in the call to has_attached_file) downcased and pluralised for your enjoyment.

:style

The current style of the attachment file being processed (e.g. in the ‘discarding an uploaded image‘ example above the :style would be one of ‘original’ or ‘small’)

</dl>

The default :url and :path options for has_attached_file are:

:url => "/:attachment/:id/:style/:basename.:extension"
:path => ":rails_root/public/:attachment/:id/:style/:basename.:extension"

Let’s say you’d prefer your users’ photos to be stored in a single ‘photos’ subdirectory of the public/images folder on your server using the user ID and style as the base file name. Your model would need to contain something like this:

has_attached_file :photo,
                  :url => "/images/:attachment/:id_:style.:extension",
                  :path => ":rails_root/public/images/:attachment/:id_:style.:extension"

If you want to hide images behind a controller do something like this:

has_attached_file :photo,
                  :url => "/:class/:id/:attachment/:style.:extension",
                  :path => ":rails_root/attachments/:class/:id/:attachment/:style.:extension"

In this example the URL points to a PhotosController nested within the User resource (an example URL would be /users/2/photos/small.png) and the attachment files are stored outside of the public root in a subdirectory of an attachments folder (e.g. RAILS_ROOT/attachments/users/2/photos/small.png). The show action of the PhotosController would be responsible for returning the binary data of the appropriate file using the :style, :extension and :user_id parameters.

Custom interpolations

In addition to the predefined interpolations described above, Paperclip makes it very easy to define your own. For example one of my models has a symbol attribute that I want to use in the file name of images attached to it, so in a Rails initializer I add the following code:

Paperclip::Attachment.interpolations[:symbol] = proc do |attachment, style|
  attachment.instance.symbol.downcase
end

Interpolations are Procs that take two parameters, the attachment object and the current style, and it is possible to access the model that the attachment is part of using the instance attribute of the attachment object.

After adding my custom interpolation I can then use it like this:

has_attached_file :logo,
                  :url => '/:attachment/:symbol/:style/:basename.:extension',
                  :path => ':rails_root/public/:attachment/:symbol/:style/:basename.:extension '

Deleting an existing attachment

Deleting an existing attachment from a model is as simple as setting the attachment attribute to nil. In a RESTful world you could do this from the destroy action of a controller that maps to the attachment (for example using a DELETE request on /users/1/photos).

You can also quite easily replace an existing attachment by POSTing a new file to your update action. Things get a little trickier if you want to be able to delete an existing attachment without replacing it using an update action.

The approach I’ve used is to add a checkbox to the edit form that when checked causes any existing attachment to be removed unless a new file has also been selected in the file upload box. Here’s the view code:

Chris has pointed out that this would be even cleaner if I used an instance variable with accessors, so I’ve updated the code to use this approach.

<% form_for(user, :html => { :multipart => true }) do |f| %>
  <%# lots of exciting form fields %>
  <div>
    <%= f.label(:photo, 'Upload photo') %>
    <%= f.file_field(:photo) %>
  </div>
  <%- unless user.new_record? || !user.photo? -%>
    <div>
      <%= f.label(:delete_photo, 'Delete photo') %>
      <%= image_tag(user.photo.url, :alt => 'Photo', :title => 'Current photo') %>
      <%= f.check_box(:delete_photo) %>
    </div>
  <%- end -%>
  <%# lots more exciting form fields %>
<% end %>

This adds a checkbox, using an instance variable to track its value, if the user isn’t new and already has a photo. The instance variable is added to the model like this:


  def delete_photo=(value)
    @delete_photo = !value.to_i.zero?
  end
  
  def delete_photo
    !!@delete_photo
  end
  alias_method :delete_photo?, :delete_photo

Then the update action in the controller looks like this:

def update
  params[:user][:photo] = nil if params[:user][:photo].blank? && delete_photo?

  if @user.update_attributes(params[:user])
    flash_and_redirect_to('User profile was saved successfully.', :notice, users_path(@user))
  else
    page_title.unshift('Edit user')
    render(:action => 'edit')
  end
end

The first line forces the value of the photo attribute in the params hash to nil if no photo file has been uploaded and the checkbox was ticked. The call to update_attributes then works as normal, updating the photo attribute as appropriate, with Paperclip taking care of the rest.

When I was trawling the interwebs to see if anybody else had some code to do this I didn’t find anything, so please let me know if you’ve come up with a better idea but haven’t yet had chance to share it with the world!

Getting clever with validations

I’ve added this section thanks to DMitry’s comment.

Paperclip allows you to validate the presence, content type and size of an attachment file using the validates_attachment_presence, validates_attachment_content_type and validates_attachment_size methods.

But what if you want to do something more advanced? For example let’s say we have a Track model that represents uploaded MP3 files and, because we want to preserve the musical integrity of our site, we want to prevent files from certain ‘artists’ being uploaded. To do this we can use a custom validation method:

class Track < ActiveRecord::Base

  has_attached_file :mp3

  validates_attachment_presence :mp3
  validates_attachment_content_type :mp3, :content_type => [ 'application/mp3', 'application/x-mp3', 'audio/mpeg', 'audio/mp3' ]
  validates_attachment_size :mp3, :less_than => 10.megabytes

  validate :must_have_valid_artist_tag

  protected

    def must_have_valid_artist_tag
      Mp3Info.open(mp3.to_file.path) do |mp3info|
        errors.add(:mp3, 'must not be a Michael Bolton song (what are you thinking?!)') if mp3info.tag.artist == 'Michael Bolton'
      end if mp3?
    rescue Mp3InfoError => e
      errors.add(:mp3, "unable to process file (#{e.message})")
    end

end

This model makes use of the ruby-mp3info gem to access the ID3 tags: you’ll need to install it and set up the necessary config.gem line in your environment.rb file to get this example working.

The first four lines of the model are straight calls to Paperclip methods: they setup the attachment and ensure it is validated for presence, content type and size. The model then contains a validate callback for the must_have_valid_artist_tag method: this is where the good stuff happens.

Here’s a line-by-line breakdown:

If a new MP3 file has been attached then at the time of validation it won’t have been written to the correct location on the server (or S3 bucket). Fortunately the to_file method means we don’t have to worry about this: it returns a File object that will either refer to an existing attachment file or the new, temporary, uploaded file. The first line of the validation method passes the path name of this file to the Mp3Info class so that it can process it.
In this simple example the validation checks if the artist tag of the MP3 file is set to a particular artist and flags an error as appropriate.
Before doing any of the above it’s a good idea to check that a file was actually attached: Paperclip provides a simple way to do this by calling the mp3? method which returns true if a file has been attached. The name of this method is based on the name of your attachment, so for example if your model contains has_attached_file :photo then the photo? method will be used check for an attached file.
The Mp3Info class will raise an Mp3InfoException if something goes wrong. We need to rescue it in case an invalid MP3 file is uploaded.
To keep this example simple, if an exception occurs an error is added to the mp3 attribute containing the exception message: you could naturally do something more impressive here.

By providing you with direct access to the attachment file using the to_file method, Paperclip enables you to do pretty much anything with the attachment, either in a validation like the one shown above or in a different model callback such as before_save.

What’s next?

Being hosted on GitHub naturally means that other developers are forking Paperclip and changing it to suit their needs. We’re no exception and Chris has recently been going forking crazy, adding support for thumbnailing of video attachments amongst other things, so I’ll try and nudge him into blogging about his changes.

Even though I’ve only been using Paperclip for a couple of days I really like its straightforward, flexible approach and I’ll definitely be using it for future projects (that is until something newer and shinier comes along). However my old friend attachment_fu is far from dead and so I’ll be keeping watch on its development too.

Making a date with JavaScript and JSON

2008-09-13T17:15:00Z

I’ve always enjoyed a kind of love/hate relationship with JavaScript, something which I think stems from my early exposure to using it on Microsoft platforms in the late 90’s, using half-baked debuggers, inconsistent browsers and a huge degree of trial and error. It’s not that I can’t write clever things in JavaScript, just that it’s not my most favourite job: if I can delegate to someone else I usually will!

Last week I was left with no choice but to roll up my sleeves and get coding: I was putting together a financial trading application prototype for one of our new clients and wanted to use JavaScript and JSON to periodically update pricing information.

I decided to stick with the Rails default Prototype library and use script.aculo.us for some snazzy effects. Our trendy UI designer friends up in London have a fondness for MooTools, so we’ve used that on the last couple of projects they’ve worked with us on. I figured it was time to brush up on the latest changes in Prototype as it’s been about 6 months since I last used it.

After a couple of hours I was actually starting to feel better about the whole JavaScript thing: Prototype makes it a lot nicer to work with, the Firebug extension for Firefox is fandabidozi, and Rails makes JSON output trivial. My price information was updating nicely without me getting even slightly stressed. And then it came to the task of showing the date and time of the last price update…

My JSON output included the updated_at date nicely formatted using ISO 8601: “no problem” I naively thought, “I’m sure JS can handle that”. I figured the Date class would parse it and then I could format it appropriately for display on the page. I soon discovered that it couldn’t: the Date class just sat there mocking me, Prototype didn’t come to my aid and my attempt to find any mention in the documentation of how dates in JSON data should be handled ended in failure. My old hatred of all things JavaScript started to return.

And then, just as I was about to give up, a random Google search stumbled upon Datejs. Either I’ve not been reading the right blogs or this library is the best kept secret in the JavaScript world. Behold the simplicity it brings to JSON date handling:

date = Date.parse(json_data.updated_at);

ISO 8601 is just one of the many date formats it can handle, but parsing dates is naturally not all it can do: comparison, validation, addition and subtraction and, perhaps most useful from the presentation point of view, formatting. The core toString method provides a large number of format specifiers, for example:

date.toString("MMMM dd, yyyy HH:mm"); /* September 12, 2008 11:23 */
date.toString("M/d/yyyy");            /* 12-Sep-2008 */

And if that’s not enough the extras library provides a format method that works just like the Ruby or PHP strftime methods.

To get started, take a look at the introductory blog post, download the code (ignore the Alpha 1.0 release as it is out of date), read the documentation and enjoy another step towards completely painless JavaScript development.

Creating a new EC2 AMI from within VMware or from VMDK files

2008-09-01T14:16:00Z

I've used VMware for many years to allow me to test and develop various server configurations and distributions. It's where I played with Linux-VServer, User Mode Linux (UML), Kernel Based Virtual Machine (KVM) and Xen. The relevent part to EC2 here is obviously that EC2 is also Xen based.

So far when using EC2 I've stood on the shoulders of giants and taken existing AMIs and used them as they are or with modification. Always the control freak, I wanted to create my own AMI from scratch. Having slimmed down my hardware inventory (I got complaints about the various half built towers lying around!) then my only option was to either create an image in a loopback device on EC2 or return to my trusty steed of VMware.

Creating the AMI from VMware was easier than I thought - but just in case anyone else is looking to do it then I thought it worthy of a blog post.

There are two options for migration - one is to create the image from within the running VMware machine, the other is to convert the VMDK file into a RAW image.

I plan to create some base 'clean' images from which I can then create specific appliances - for example a base Centos i386 and x64 install which I will then layer additional software on to create specific appliance images.

My preferred method is to create the AMI from within the running VMware machine - the resulting AMI is much smaller in size (as I think that during conversion of a VMDK the resulting RAW image is the size of the volume rather than the data - although you can probably override this with options). I've given instructions for both methods so you can make your own mind up.

Converting VMDK files for use on EC2

Create your machine in VMware as usual and when ready shut it down. Then, grab yourself a copy of QEMU as we need to use it to convert the VMDK files.

Basic usage is:

qemu-img convert -O raw input.vmdk output.raw

The VMDK file will then be converted into a RAW file - we've specified the output format with the -O tag and qemu-img is intelligent enough to figure out the input format.

Your image may span several VMDKs in which case you'll need to run a simple shell loop, for example:

for file in `ls *[0-9].vmdk`; do
  qemu-img convert -O raw $file $file.raw
done
cat *.raw >> output.raw

Depending on the number of files you have for your VMware image you might need to modify the ls - my example is purely by way of demonstration.

Either way, you should now have your VMware disk image in RAW format in the file output.raw. Given this is a RAW image it should also be bootable by a local Xen, QEMU or KVM installation.

The next step is to bundle this into an AMI using ec2-bundle-image:

ec2-bundle-image -i output.raw -r x86_64 -c ~/.amazon/cert.pem -k ~/.amazon/pk.pem --user <user id>

The exact values for the -c, -k and --user parameters will depend on the location of your certificate, private key and Amazon AWS account number respectively.

You might also want to look closely at the documentation for the --block-device-mapping parameter to ensure you get a bootable machine with a valid fstab.

After the image has finished bundling it will (by default) be in your /tmp directory, so the next step is to upload it to an S3 bucket:

ec2-upload-bundle -b <bucket name> -m /tmp/image.manifest.xml -a <access id> -s <secret access id>

With the bundled image in S3 (it will take a while depending on your connection) then you can register the image:

ec2-register <bucket name>/image.manifest.xml

The AMI should then appear when you request a list of your images:

ec2-describe-images

The only remaining thing then is to start up an instance and give it a try!

Creating the EC2 AMI from within VMware

As mentioned, creating the AMI in this way seems to result in a smaller file - I prefer it as it's also quicker to complete as you don't need to convert and create bundle - you can just create the bundle.

You'll need to boot your VMware machine and make sure you have the pre-requisites installed to run the EC2 and AMI tools - i.e. ruby, java > 1.5, etc.

For CentOS I needed to disable the MAC address being specified for my Ethernet device by commenting out the HWADDRESS line of the network script.

Check the amount of disk space you're using - for me it was 1.2Gb for my basic Centos 5.2 install.

You're then ready to bundle your volume, making sure you specify a size big enough to fit your volumes into along with any software you might want to install in future:

ec2-bundle-vol -c ~/.amazon/cert.pem -k ~/.amazon/pk.pem --user <user id> -d /image -e /image -r x86_64 -s 4096 --no-inherit

Depending on your install you may also need to specify an alternative fstab to use via the --fstab option. The -s parameter specifies the size - 4Gb in this example - you can set this up to 10Gb which is the current limit imposed by EC2. The --no-inherit parameter is required because we're not bundling from within an EC2 instance.

After the image has finished bundling it will be in your /image directory (because we specified this in our command) - so the next step is to upload it to an S3 bucket:

ec2-upload-bundle -b <bucket name> -m /image/image.manifest.xml -a <access id> -s <secret access id>

With the bundled image in S3 (it will take a while depending on your connection) then you can register the image:

ec2-register <bucket name>/image.manifest.xml

The AMI should then appear when you request a list of your images:

ec2-describe-images

You can then launch an instance using your new AMI (the AMI id will have been provided on completion of the registration, or from the output of the describe command).

Giving composed_of some much needed lovin’

2008-08-24T10:11:00Z

Did you know ActiveRecord includes support for aggregations? If you did, have you ever used them? Despite being part of Rails from the start composed_of tends to lurk in the shadows while newer features like named_scope steal the limelight. It’s time to give composed_of some love again!

Aggregation helps to make your models richer: for example if you’re dealing with money why deal with integer and decimal attributes when you can use something like the Money class? Thanks to various patches the shortcomings of the original composed_of implementation have been addressed, but there is still one that for some reason remains to be fixed: aggregate class constructors.

The problem is simple: composed_of assumes that the aggregate class can be instantiated using a simple new constructor that accepts arguments in the order defined by the :mapping option. In many cases this works, but if your aggregate class doesn’t conform then you’re left to manually initialise your aggregations (in an after_initialize callback for example). Not very elegant.

Here is an example from a recent project I’ve been working on:

class Visitor < ActiveRecord::Base

  composed_of :ip_addr,
              :class_name => 'IPAddr',
              :mapping => %w(ip to_i)

end

Here the composed_of aggregation maps the ip attribute (an integer) to the IPAddr class from the Ruby standard library. Much of the code that uses this attribute wants to treat it as an IP address rather than a raw integer so it makes sense to represent it as one in the model. Unfortunately this won’t work properly as the IPAddr constructor requires a second parameter to be passed to the constructor as shown below:

IPAddr.new(1024768100)
=> ArgumentError: unsupported address family

IPAddr.new(1024768100, Socket::AF_INET)
=> #<IPAddr: IPv4:61.20.184.100/255.255.255.255>

Sadly previous attempts to allow composed_of to be more flexible with constructors have failed to make it into core. Personally I don’t think that too few people requesting an enhancement is a good enough reason not to apply it anyway. Perhaps if composed_of was more flexible in the first place more people would use it!

Hopefully though it’ll be third time lucky as I’ve just submitted my own patch to improve composed_of. If the core team still don’t feel motivated to apply it then I guess I’ll just release it as a plugin instead. It takes a slightly different approach then the previous patches:

It adds a new :constructor option that takes a symbol or a proc that will be used to instantiate the aggregate object. It is optional and if not used then the existing behaviour of calling new with the mapped attributes will be used.
It deprecates the use of a block to provide a method of converting values assigned to the aggregate attribute. The use of a block didn’t feel particularly consistent with the rest of Rails where typically symbols or procs are passed as options (a good example being the :if and :unless options for validations). Of course passing a block will still function, so existing code won’t break, but it will raise a deprecation warning. This change leads on to…
It adds a new :converter option that also takes a symbol or proc that will be used when the aggregate attribute is assigned a value that is not an instance of the aggregate class. This replaces the old block syntax and makes it easier to do things like calling composed_of from within a with_options block. If both the :converter option and a block are passed to composed_of then the :converter option will take precedence.

Here’s the IP address example from earlier updated to take advantage of the new options:

class Visitor < ActiveRecord::Base

  composed_of :ip_addr,
              :class_name => 'IPAddr',
              :mapping => %w(ip to_i),
              :constructor => Proc.new { |value| IPAddr.new(value, Socket::AF_INET) },
              :converter => Proc.new { |value| value.is_a?(Integer) ? IPAddr.new(value, Socket::AF_INET) : IPAddr.new(value.to_s) }

end

Passing a symbol to either the :constructor or :converter options provides an easy way to use a class where the constructor is not named new but still expects arguments in the order defined by the :mapping option. For example if your aggregate class uses a parse method for instantiation you could do this:

class MyClass < ActiveRecord::Base

  composed_of :aggregate_value,
              :class_name => 'AggregateClass',
              :mapping => %w(value to_s),
              :constructor => :parse,
              :converter => :parse

end

I think these simple changes make aggregation with composed_of much more powerful without breaking existing code. If you’d like to see composed_of get the love it deserves, take a look at my patch and give it your support.

It's arrived - Amazon Elastic Block Store

2008-08-21T14:27:00Z

The feature I've been waiting for on EC2 has arrived - persistent storage is now available on the public beta as per an email from the EC2 team:

We are pleased to announce the release of a significant new Amazon EC2 feature, Amazon Elastic Block Store (EBS), which provides persistent storage for your Amazon EC2 instances. With Amazon EBS, storage volumes can be programmatically created, attached to Amazon EC2 instances, and if even more durability is desired, can be backed with a snapshot to the Amazon Simple Storage Service (Amazon S3).

At a high level the features include:

Volumes from 1Gb to 1Tb
Multiple volumes can be mounted from one AMI
Implemented as block devices
Live in an availability zone
Automatically replicated within the availability zone
Support snapshots
Multiple volumes can be used and striped across to improve I/O
$0.10/gb allocated per month and $0.10 per 1 million I/O requests
Sample cost based on 100Gb storage and 100 I/O per second is $36/month

I'm going to get playing with it and it will be interesting to see how it performs and also compare it on the cost front to some of the options I outlined in a previous post - in the meantime you can read more on the Amazon Web Services site.

Fixed! Image uploads to Flickr with flickr_fu

2008-08-11T12:37:00Z

One of the projects we’re working on needs to be able to publish images to Flickr. Rather than re-invent the wheel a quick search on RubyForge and GitHub turned up a number existing Flickr libraries for Ruby. After considering future requirements, we decided to go with flickr_fu.

There was just one teeny tiny problem: uploads to Flickr within flickr_fu were broken, as we discovered when we looked at the comments in the Flickr::Uploader class:

# NOT WORKING ... FILE UPLOADS IN NET::HTTP IS TEH SUCK

A fork, a clone and a few tweaks later and we’ve got uploads working. The changes are currently in our fork but I’ll also be submitting them upstream after a little more testing.

And our survey says…

2008-08-06T15:36:00Z

There are currently a couple of web-based surveys being carried out that you may want to take a look at. First up is the survey for people who make websites 2008 aimed at anyone involved in web development from designers to Ruby coders from the guys at A List Apart.

Nearly 33,000 people took part in 2007 and it will be interesting to see what has changed and what has remained the same in terms of salary, experience, education and working conditions over the last year.

The second comes from HAML creator Hampton Catlin in the form of the more Ruby-centric Hampton’s Ruby Survey 2008. At the time of writing over 900 people have responded, so if you haven’t already, now’s the time to get your entry in.

The survey covers Ruby and Javascript frameworks, testing, interpreters and in a bizarre twist ends with religious beliefs. It will certainly be interesting to discover if the Ruby community is big on new-age spirituality or is just a godless bunch of coders!

Roll your own pagination links with will_paginate

2008-08-03T15:10:00Z

Mislav’s will_paginate plugin (and Gem) has become the de facto standard for pagination in Rails, replacing the often derided classic pagination from the dark days before Rails 2.0. If you haven’t used will_paginate before then Ryan Bates’ RailsCast is a good introduction, although be warned that it is just over a year old and there have been a number of changes to the plugin in that time (the current version is ~~2.3.2~~ 2.3.6).

The standard view helper

The plugin includes a helper, unsurprisingly named will_paginate that makes it easy to add pagination links to your views. Here’s an example of its use:

# In the controller
@users = User.paginate(:page => params[:page])

# In the view
<%= will_paginate(@users) %>

This will produce the following HTML markup:

<div class="pagination"><span class="disabled prev_page">« Previous</span> <span class="current">1</span> <a href="/users?page=2" rel="next">2</a> <a href="/users?page=3">3</a> <a href="/users?page=2" class="next_page" rel="next">Next »</a></div>

As you can see the helper uses the fairly common markup of a <div> containing links or spans for the page numbers along with previous and next page links. The plugin includes a number of CSS examples that allow the markup to be styled like digg or flickr (take a look here for more) and the helper accepts a hash of options that allow basic customisation.

<dl>

:container

Determines whether or not the container <div> is included in the generated markup (defaults to true).

:class

Allows the CSS class of the container <div> to be specified (defaults to pagination).

:previous_label

:next_label

Customises the previous and next page link labels.

:page_links

Determines whether or not the individual page links are included in the generated markup (defaults to true). If false then only previous and next page links are generated.

:inner_window

:outer_window

Customises the number of links displayed at either side of the current page (:inner_window) and at the start and end of the page links (:outer_window).

:separator

Specifies the text inserted between the individual HTML elements generated by the helper (defaults to a single space).

</dl>

Doing it your way

The standard helper provides customisation through CSS styling and the options listed above, but what do you do if the UI designer on your project insists that the pagination markup must look like this?

<ul class="pagination">
  <li class="previous"><a href="/users?page=1">« Previous</a></li>
  <li><a href="/users?page=1">1</a></li>
  <li class="current">2</li>
  <li><a href="/users?page=3">3</a></li>
  <li class="next"><a href="/users?page=3">Next »</a></li>
</ul>

You have a couple of options: either you beat the designer with a stick until they agree that the default markup is acceptable or, if like us you prefer a non-violent approach to web development, you can take advantage of will_paginate’s support for custom link renderers.

Here’s an example that will generate the above markup:

class PaginationListLinkRenderer < WillPaginate::LinkRenderer

  def to_html
    links = @options[:page_links] ? windowed_links : []

    links.unshift(page_link_or_span(@collection.previous_page, 'previous', @options[:previous_label]))
    links.push(page_link_or_span(@collection.next_page, 'next', @options[:next_label]))

    html = links.join(@options[:separator])
    @options[:container] ? @template.content_tag(:ul, html, html_attributes) : html
  end

protected

  def windowed_links
    visible_page_numbers.map { |n| page_link_or_span(n, (n == current_page ? 'current' : nil)) }
  end

  def page_link_or_span(page, span_class, text = nil)
    text ||= page.to_s
    if page && page != current_page
      page_link(page, text, :class => span_class)
    else
      page_span(page, text, :class => span_class)
    end
  end

  def page_link(page, text, attributes = {})
    @template.content_tag(:li, @template.link_to(text, url_for(page)), attributes)
  end

  def page_span(page, text, attributes = {})
    @template.content_tag(:li, text, attributes)
  end

end

As you can see the link renderer is a subclass of WillPaginate::LinkRenderer the main method of which is to_html: this is the method that is used by the view helper to generate pagination markup. In this example I’ve chosen to preserve support for all of the standard options (:container, :previous_label, etc.) however you do not have to do so in your own link renderers if you don’t want to. I’ve also chosen to override the four protected methods that are called upon by to_html to actually perform the HTML generation:

<dl>

windowed_links

Generates the individual page number links using a call to visible_page_numbers to determine the page numbers that should be included based on the inner and outer window options.

page_link_or_span

Used by windowed_links to generate the appropriate HTML for a page: this will be a link for all but the current page.

page_link

Generates a list item containing a link to a specific page.

page_span

Generates a list item containing text rather than a link.

</dl>

Of course you don’t have to override these methods if you don’t want to: you could choose to do all of your rendering in your custom to_html method or you could choose to define your own protected methods for performing specific tasks, the choice is yours. In this example I chose to use the existing method names for two reasons: it makes it easier for somebody else to maintain the code as it follows the same naming as the standard link renderer, and I couldn’t think of any better names!

The WillPaginate::LinkRenderer also has initialize and prepare methods that you can extend in your own renderer, although in most cases you won’t need to.

<dl>

initialize

The constructor initialises the gap_marker attribute used by the windowed_links method: this is the HTML that is used to fill in the gaps in page numbers (caused by the inner and outer window settings) and defaults to <span class="gap">…</span>. The gap marker is not used in the example renderer above.

prepare

This method is called each time you use the will_paginate helper in your view and is passed the collection being paginated, the options hash and view template.

</dl>

If you need to do one-time initialisation when your renderer is created then override the initialize method, and if you need to perform initialisation each time the renderer is called override prepare.

Having created a custom renderer you need to tell will_paginate to use it. There are two ways to do this, the first is to pass the :renderer option like this:

<%= will_paginate(@users, :renderer => PaginationListLinkRenderer) %>

The :renderer option accepts a class name string, a class or an instance of a link renderer. If you want your custom link renderer to be the default for all pagination you don’t have to add a :renderer option to all of your will_paginate calls, instead you can specify the default in a Rails initializer like this:

WillPaginate::ViewHelpers.pagination_options[:renderer] = 'PaginationListLinkRenderer'

Your view can then contain <%= will_paginate(@users) %> and your link renderer will be used automatically.

Custom link renderers are an excellent, if underused, feature of will_paginate that give you the freedom to markup your pagination links in whatever creative way you can come up with. Go forth and paginate!

Finally: decent documentation for Rails developers

2008-07-20T10:30:00Z

Last week Jeremy McAnally talked about the continuing decline in Rails blogging. Like many Rails developers I’ve learned a lot from blogs over the last couple of years: in the early days it was pretty much the only way to find out about how Rails worked as the official documentation was poor to say the least.

In my opinion though, the future for Rails documentation is not as bleak as Jeremy suggests. Less blogging could simply indicate a move towards better quality postings: I’d rather have a choice from a handful of really well written blogs than the hundreds of “OMG!!! Look how awesome Rails is…” style posts that I used to have to trawl through – less hype and more substance is definitely better.

The official documentation is, at last, being improved too: as announced in May there is now a docrails branch on Github and it has already seen a huge number of changes committed.

While I don’t particularly miss coding in PHP it came as quite a shock when I made the move to Rails to find that there wasn’t anything that came close to its documentation. Not only is there a language reference, full documentation for the standard extensions (and with the crazy lack of naming conventions or namespaces in PHP this is really useful!) there is also the invaluable comments section where other PHP developers can contribute extra documentation to each and every part of the language.

This is what Rails has needed for a long time. The great news is that thanks to the hard work of the guys at Nodeta it has finally arrived: behold Rails-doc! The site reached version 2.0 last week and now it has pretty much everything you could ask for:

Full documentation of the Rails API including different versions of the API.
User comments.
Easy to use search, as well as a Firefox search bar extension.
A simple, clear user interface that is far better to navigate than the official API documentation.

Now, perhaps more than ever before, we as a community can get involved to improve Rails documentation: we can of course continue with our blogs, but we can also submit patches to docrails and post comments to Rails-doc. These things combined should ensure that we don’t reach the point where, to quote Jeremy, <q cite="http://www.omgbloglol.com/?p=34">Rails becomes an esoteric project doomed to eventual abandonment because everyone has to shell out $40 for a book to learn how to use it.</q>

A tamper-proof cookie jar for Rails

2008-07-14T11:52:00Z

When the default session store for Rails was changed to use cookies last year it caused quite a furore: "It’s not secret! It’s not safe!" The truth is it was never meant to be: in most circumstances you shouldn’t really be storing data in cookies that need to be protected so strongly. But I don’t really want to re-open that can of worms! Instead I want to look at a different cookie-related situation that arose during some optimisation work I’ve been doing on an existing Rails application.

The scenario is straightforward enough: the application stores a numeric record ID in a persistent cookie that, while perfectly safe to be seen by users, shouldn’t be changed by them. The solution currently used in the application is equally straightforward: the data is encrypted (using the EzCryto gem and AES encryption) before being written to the cookie, and then decrypted when the data is read back in.

In situations where cookie contents really shouldn’t be seen by users then encryption is the way to go (unless of course you can find a way of not using a cookie at all), however for a simple numeric ID where tampering is the only thing we need to protect against then it seems like overkill. A better solution is to take inspiration from the Rails cookie session store and use a HMAC.

Enter the TamperProofCookieJar

Here’s the code (if it seems familiar it’s because it borrows heavily from the Rails cookie session store):

require 'openssl'

module ActionController
  class TamperProofCookieJar < ActionController::CookieJar

    class TamperedWithCookie < StandardError;
      attr_reader :cookie_name
      def initialize(cookie_name)
        @cookie_name = cookie_name.to_s
      end
    end

    def initialize(controller, secret, digest = 'SHA1')
      super(controller)
      @secret = secret
      @digest = OpenSSL::Digest::Digest.new(digest)
    end

    def [](name, protect_from_forgery = false)
      if (value = super(name)) && protect_from_forgery
        value, hmac = value.to_s.split('--')
        unless hmac == generate_digest(value) || hmac == generate_digest(value = CGI.unescape(value))
          raise TamperedWithCookie.new(name)
        end
      end

      value
    end

    def []=(name, options)
      if options.is_a?(Hash)
        options = options.stringify_keys
        options['name'] = name.to_s
        options['value'] = options['protect_from_forgery'] == true ? "#{options['value']}--#{generate_digest(options['value'])}" : options['value'].to_s
      else
        options = { "name" => name.to_s, "value" => options.to_s }
      end
      set_cookie(options)
    end

    protected

      def generate_digest(value)
        OpenSSL::HMAC.hexdigest(@digest, @secret, value.to_s)
      end

  end
end

The code makes use of the Ruby OpenSSL library to generate a HMAC using the SHA-1 hashing algorithm. A few notes about the code:

The TamperedWithCookie exception will be raised on reading a cookie value if it is deemed to have been tampered with. The exception provides a cookie_name attribute that can be used to get the name of the offending cookie.
The TamperProofCookieJar class is a subclass of the regular Rails CookieJar and as such is a drop-in replacement for it with one exception: the cookie value is converted to a string before being written to the cookie - this means if you’re not storing a simple string or number then you’ll need to ensure you have marshalled the data correctly before trying to write it to the cookie (again in most situations you shouldn’t be storing any complex objects in a cookie anyway).

Using the TamperProofCookieJar is easy:

# Write a regular cookie (by default HMAC protection is NOT used - so this is just like a regular Rails cookie)
cookies[:test_unprotected] = { :value => user.id, :expires => Time.zone.now + 30.days }

# Write a HMAC protected cookie
# user.id = 100
cookies[:test_protected] = { :value => user.id, :protect_from_forgery => true, :expires => Time.zone.now + 30.days }

# Read a regular cookie (again no protection is the default)
user_id = cookies[:test_unprotected]
=> "100"

# Read a HMAC protected cookie (the boolean parameter turns on HMAC protection)
# Raises TamperProofCookieJar::TamperedWithCookie if the cookie appears to have been changed
user_id = cookies[:test_protected, true]
=> "100"

# Read a raw HMAC protected cookie
# This returns the actual contents of the cookie, including the HMAC, and does not perform tamper checking
user_id = cookies[:test_protected]
=> "100--2157acbbe09b9f412cee3b97ad2e2c2136d3f5e1"

Replacing the Rails CookieJar

If you want to replace the Rails CookieJar with the TamperProofCookieJar in your application you can override the cookies method in your ApplicationController like so:

class ApplicationController < ActionController::Base
  def cookies
    ActionController::TamperProofCookieJar.new(self, 'this should be a long secret key')
  end
end

The second parameter should be a secret key that is hard to guess, just like the one used by the Rails cookie session store.

Performance improvements

As I said earlier, the TamperProofCookieJar was created as part of an optimisation project, so how does its performance compare with the original encryption solution? Using a simple benchmark on 20,000 encryptions versus 20,000 HMACs gives the following results:

<dl>

Encryption total time

2.918 seconds

HMAC total time

0.854 seconds

</dl>

This gives a 3.4x performance increase which on a high volume site such as the one I’m working on is a worthy saving!

To patch or plugin?

My original intention was to submit this as a patch to Rails however I don’t think it will be seen as “useful” enough for the wider Rails community to be included in core. Instead I’m leaning towards making it a plugin: so please let me know if you have any problems with the code or have any suggestions for how it can be improved and when time allows I’ll put it all together.

The Wedding of the Year

2008-06-30T14:48:00Z

In the week when TheWebFellas became famous for five minutes thanks to Ryan Bates, Antonio Cangiano and Gregg & Jason at Rails Envy, I was away enjoying a relaxing holiday in the beautiful Garfagnana region of Italy. But taking a break from the fast-paced world of Rails development wasn’t the only motivation for the trip: my brother and partner in crime at TheWebFellas was getting married to his girlfriend of many years Samantha.

Resisting offers from Hello magazine (although the wedding did feature in the equally popular Barga News) the highly exclusive ceremony was held in the Barga Commune with only 22 members of immediate family and close friends in attendance. The evening was spent in the grounds of the 18th Century “I Cedri” villa in nearby Albiano where we enjoyed a huge Tuscan feast that went on well into the night. I’ve put a few of my photos up on Flickr: incredibly the official photographer managed to take about 600 photos!

For the remainder of the week I was able to relax by the pool, drive on some of the greatest roads in Europe in my trusty Micra convertible and enjoy some excellent food in various restaurants and pasticcerias: if you ever find yourself in Barga you must visit Bar Lucchesi in the new part of town for some genuine Italian atmosphere, pastries and the best cappuccino ever.

Sadly while Chris and Sam are still enjoying a leisurely trip back through Europe I’m now back in the UK, working hard and missing my swimming pool with a view!

EC2 Persistent Storage for the Impatient?

2008-06-12T08:00:00Z

As I previously posted Amazon have announced that persistent storage is on it's way for EC2. Sadly the public launch date has not yet been disclosed - it's "coming later in the year". In the meantime this leaves the question of what to do when you need data to persist?

There are a number of options, especially when you start to consider scalability and fault tolerance. I won't dare claim I've considered all the options out there - I've simply started to look at what the immediate options are for persistent storage.

Without further ado, then, on to the technology. I've found a number of choices - rather than this being a 'how to' then it is more about the solutions I have found so far and that are on my list for consideration. Hopefully, if one of the options fits my needs then I will provide a guide at a later date!

At a high level the solutions can be split into two categories - file system interfaces to S3 or architecture based replication and backup. There are likely to be other options that allow you to use persistent storage outside the Amazon cloud - for the time being I'm trying to stay within the bubble. There is also a cross-over between the two strategies - i.e. using a particular architecture with a file system interface.

File System Interfaces to Amazon S3

The file system interfaces to S3 are typically based on either the FUSE library or implemented as an NBD. FUSE allows developers to create interfaces (in user space) that ultimately surface to the user as a standard file system, while NBD allows interfaces that appear as a raw, unformatted physical disk (allowing the user to format the disk as they please).

FUSE Interfaces

<dl>

s3fs (Open Source)

s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files). Maximum file size is 5G.

PersistentFS (Closed Source, Current Development Version is Free)

PersistentFS is a fast and efficient POSIX-compliant file system that provides unlimited online storage in the Amazon Web Services (AWS) storage cloud.

InfiniteFTP (Closed Source, Paid Service) over CurlFtpFs (Open Source)

InfiniteFTP provides an FTP interface to Amazon S3 - while this isn't a file system approach then you could add a FUSE layer over the top to make it appear as one.

CurlFtpFS is a filesystem for accessing FTP hosts based on FUSE and libcurl.

</dl>

NBD Interfaces

<dl>

ElasticDrive (Closed Source, Paid Service)

ElasticDrive is a Distributed Remote Storage Application that allows you to access your data regardless of location with the assurance your data is safe. It makes it possible for a remote storage resource, such as Amazon's Simple Storage Service (S3), Nirvanix & Xdrive to behave like a local hard drive.

</dl>

Architecture Considerations

There are a number of options for architecture based persistence that immediately spring to mind - i.e. various replication strategies such as DRDB or distributed filesystems perhaps backed by backups to an external service - the lag of such an approach would need to be considered. There are also options such as mirrored RAID (potentially for read speed and consistency) although RAID isn't an option on FUSE based devices.

Cost Considerations

I've only considered the options I could find for storage backed by S3 - my main in head reason for this was to avoid transfer charges. There will still be charges for the GET, PUT and LIST requests so I also need to consider the cost implications of using S3 to assist with persistent storage.

What Next?

The next step is to take a deeper look at the options on offer and try to discover if there is a workable solution for persistent storage already available. I also want to run benchmarks against some sample configurations to see if the options perform under real world requirements. Finally, I will run some numbers to check if the costs stack up. By the time I've finished, the official Amazon offering may well be available - if not I shall document my findings!

Rails 2.1: now with better integrated caching

2008-06-09T17:00:00Z

Rails 2.1 has just been out a week and so far something that seems to have passed most people by is that it now includes much better caching capabilities, including built-in support for memcached.

Last week I reached the point with an application where I needed to cache some models in memory to get a performance boost and decided to check out the current status of plugins like cache_fu and CachedModel to make sure they’d work with Rails 2.1. It was completely by accident that I stumbled across this innocent looking commit by DHH from start of this year and realised that Rails already had everything I needed!

Making sense of my notes

Unfortunately the documentation for this new caching support seems quite scarce and unusually Google couldn’t help much either as there hasn’t been much blogging about it yet. So, not for the first time, I spent a bit of time digging into the source code to find out how it all fits together. What follows is a write-up of my hastily scribbled notes.

The basics

The code lives in the ActiveSupport::Cache module and it defines a base Store class which provides the basic interface for a caching store with methods like: read, write, delete and exist? (why oh why isn’t it exists? like ActiveRecord?). There is also a ThreadSafety module that wraps read/write operations in a Mutex#synchronize block. Enabling thread safety on a cache store is a simple matter of calling the threadsafe! method.

Rails includes five cache stores:

File store (ActiveSupport::Cache::FileStore) - uses individual files to store cached values.
Memory store (ActiveSupport::Cache::MemoryStore) - uses a simple hash to store values.
DRb store (ActiveSupport::Cache::DRbStore) - works just like the MemoryStore but uses a DRb server to store values.
Memcached store (ActiveSupport::Cache::MemCacheStore) - uses memcache and memcache-client (version 1.5.0 is bundled with Rails so there’s no need to install the Gem separately) to store values in a shared memory cache.
Compressed memcached store (ActiveSupport::Cache::CompressedMemCacheStore) - works just like the regular MemCacheStore but uses GZip to decompress/compress on read/write.

Rails automatically creates a global cache during initialisation and you can access this in your code either using the RAILS_CACHE global variable or the (in my opinion nicer looking) Rails.cache. Here are a few simple examples:

# Write a string to the cache
Rails.cache.write('test_key', 'test_value')
=> true

# Read it back in
Rails.cache.read('test_key')
=> 'test_value'

# Delete the item
Rails.cache.delete('test_key')
=> true

By default if the tmp/cache directory exists in your application root then Rails will use a FileStore otherwise it’ll use a MemoryStore as the global cache. Either of these options should be fine for development and testing, but for production you’ll almost certainly want to consider using the MemCacheStore. One of the big advantages of this new code is that for the most part it allows you to switch cache store without having to change any of your code.

Important! As Peter points out in the comments, this behaviour is not currently implemented (the code is there but not used) - I’ve submitted a bug report

Changing the global cache store

Rails provides the config.cache_store option in environment.rb to allow you to change the cache store. It accepts either a class name or symbol for the required store and an array of options used when initialising the store. Some examples:

# Use the memory store - this store has no options
config.cache_store = :memory_store

# Use the file store with a custom storage path (if the directory doesn’t already exist it will be created)
config.cache_store = :file_store, '/my_cache_path'

# Use the memcached store with default options (localhost, TCP port 11211)
config.cache_store = :mem_cache_store

# Use the memcached store with an options hash
config.cache_store = :mem_cache_store, 'localhost', '192.168.1.1:1001', { :namespace => 'test' }

Behind the scenes this makes use of the ActiveSupport::Cache.lookup_store method, which you can also call directly to create a cache of your own:

# Create a separate memory cache in addition to the global Rails cache
my_cache = ActiveSupport::Cache.lookup_store(:memory_cache)

Data keys

All of these cache stores provide what is essentially a Hash that can be read from and written to using a key. Just like with a regular Hash object, this key needs to be unique and not just within the scope of your application but also, in the case of shared caches like memcached, across all applications that are using the cache. To make cache key generation a little easier Rails does two things:

Adds a cache_key method to ActiveRecord::Base that generates a key using the class name, record ID and updated_at timestamp (if available).
Provides the ActiveSupport::Cache.expand_cache_key method that builds a key using an optional namespace passed as a parameter to the method, the RAILS_CACHE_ID or RAILS_APP_VERSION environment variables (if they are set) and the result of a call to the cache_key or to_param method of the object being cached (if it responds to either of them) or in the case of an Array the expand_cache_key method is called for each element of the array and the resulting keys joined to form a single (potentially very long!) key.

MemCacheStore specifics

The MemCacheStore accepts an array of server addresses in the form hostname[:port][:weight]. For example, if you have two servers on different ports you could specify them like so:

config.cache_store = :mem_cache_store, '192.168.1.1:11000', '192.168.1.2:11001'

It also accepts a hash of additional options:

:namespace - specifies a string that will automatically be prepended to keys when accessing the memcached store.
:readonly - a boolean value that when set to true will make the store read-only, with an error raised on any attempt to write.
:multithread - a boolean value that adds thread safety to read/write operations - it is unlikely you’ll need to use this option as the Rails threadsafe! method offers the same functionality.

The read and write methods of the MemCacheStore accept an options hash too. When reading you can specify :raw => true to prevent the object being marshaled (by default this is false which means the raw value in the cache is passed to Marshal.load before being returned to you.

When writing to the cache it is also possible to use the :raw option which when false (the default) means the value is passed to Marshal.dump before being stored in the cache. The write method also accepts an :unless_exist flag which determines whether the memcached add (when true) or set (when false) method is used to store the item in the cache and an :expires_in option that specifies the time-to-live for the cached item in seconds. A few examples:

# Write a raw value to the cache
Rails.cache.write('test_key', 1, :raw => true)

# Read a raw value from the cache
Rails.cache.read('test_key', :raw => true)

# Write a value using the add method and setting expiry to 15 minutes
Rails.cache.write('test_key', 'test_value', :unless_exist => true, :expires_in => 15.minutes)

Increment and decrement

As caches are often used to store counters all of the cache stores provide increment and decrement methods. For example:

# Store an initial counter value in the cache
Rails.cache.write('number_of_cakes_eaten', 1)

# Increment the counter by 5
Rails.cache.increment('number_of_cakes_eaten', 5)

This encapsulates the process of retrieving a value, incrementing or decrementing it and then storing the modified value back in the cache. In addition, the MemCacheStore takes advantage of memcached atomic increment/decrement functionality to perform this action in a single call to the cache.

Custom caches

You now know enough about Rails caching to make good use of it in your own code. If you want to take things a step further and implement your own custom cache store you might want to take a look at Ryan Daigle’s example - his blog entry also demonstrates how to configure ActionController caching in Rails 2.1.

And finally…running memcached on Windows (groan!)

As a long suffering Windows developer you may well have read all of this, gotten quite excited about using memcached, and then realised that you’ve got to get it running on Windows. It’s normally at this point that I end up getting annoyed and resolve to buy a Mac, but I’m pleased to say on this occasion I was pleasantly surprised. I simply did the following:

Downloaded this Win32 port of memcached by Kenneth Dalgleish.
Extracted the files to C:\Program Files\memcached
Opened a command prompt in the folder and installed memcached as a service using memcached -d install
Started the service using net start "memcached Server"

And, shockingly, it just worked!

TheWebFellas - Ruby on Rails Development Specialists UK - Blog

Amazon launches their CDN - CloudFront

What is it?

How does it work?

The API

Goodbye attachment_fu, hello Paperclip

The obligatory special instructions for Windows

Install the plugin

Download ImageMagick

Use the RMagick gem

Download the latest version of ImageMagick

Patch the Tempfile class

Shutdown, close, reopen and restart

Where to begin

Protect your attributes

Attaching in migrations or the Rails console

Discarding an uploaded image

Styles can be Procs too

Rename files on upload

Custom interpolations

Deleting an existing attachment

Getting clever with validations

What’s next?

Making a date with JavaScript and JSON

Creating a new EC2 AMI from within VMware or from VMDK files

Converting VMDK files for use on EC2

Creating the EC2 AMI from within VMware

Giving composed_of some much needed lovin’

It's arrived - Amazon Elastic Block Store

Fixed! Image uploads to Flickr with flickr_fu

And our survey says…

Roll your own pagination links with will_paginate

The standard view helper

Doing it your way

Finally: decent documentation for Rails developers

A tamper-proof cookie jar for Rails

Enter the TamperProofCookieJar

Replacing the Rails CookieJar

Performance improvements

To patch or plugin?

The Wedding of the Year

EC2 Persistent Storage for the Impatient?

File System Interfaces to Amazon S3

FUSE Interfaces

NBD Interfaces

Architecture Considerations

Cost Considerations

What Next?

Rails 2.1: now with better integrated caching

Making sense of my notes

The basics

Changing the global cache store

Data keys

MemCacheStore specifics

Increment and decrement

Custom caches

And finally…running memcached on Windows (groan!)