I've recently discovered that my current favourite stub/mock framework, Brian Takita's RR, can do test-spies too!

Huh?

What's this "spy" business about? Well, when mocking, before triggering the behaviour you're testing, you set up expectations that a certain methods of collaborating objects will be invoked, with the specified parameters. Like so:

describe TransferEverything do

  before do
    @account1 = Account.new
    @account2 = Account.new
    @transfer = TransferEverything.new(:from => @account1, :to => @account2)
  end

  describe "#execute" do

    it "moves all funds from one account to the other" do

      all_the_money = 1.42
      stub(@account1).balance { all_the_money }

      mock(@account1).withdraw(all_the_money)   # <= set expectations
      mock(@account2).deposit(all_the_money)

      @transfer.execute                         # <= execute
      
    end                                         # <= verify expectations

  end

end

The expectations are typically verified auto-magically, by the mocking framework, at the end of your test.

The spy alternative

Setting up expectations before a call always feels clumsy. Using a test spy makes tests flow more naturally:

1. Stub out collaborators, setting up canned responses where required.
2. Execute the code you're testing.
3. Verify the results, including both:
• the outputs (return values, or resulting state)
• the interactions (ie. the method-invocations you expected your fake collaborators to receive).

Fur egg-sample:

describe TransferEverything do

  # ...

  describe "#execute" do

    it "moves all funds from one account to the other" do

      all_the_money = 1.42
      stub(@account1).balance { all_the_money }
      stub(@account1).withdraw
      stub(@account2).deposit

      @transfer.execute

      @account1.should have_received.withdraw(all_the_money)
      @account2.should have_received.deposit(all_the_money)

    end

  end

end

One thing I find particularly useful about this technique is the ability to execute code in a setup block, then verify the various aspects of it's behaviour in separate test-cases.

describe TransferEverything do

  # ...

  describe "#execute" do

    before do
      @all_the_money = 1.42
      stub(@account1).balance { all_the_money }
      stub(@account1).withdraw
      stub(@account2).deposit
      @transfer.execute
    end

    it "withdraws all funds from source account" do
      @account1.should have_received.withdraw(all_the_money)
    end

    it "deposits funds in receiving account" do
      @account2.should have_received.deposit(all_the_money)
    end

  end

end

This results in smaller, more coherent test-cases.

Using RR test-spies in RSpec

If you're using RSpec, you'll need to use the adapter class that comes with RR, rather than the one that comes with RSpec. That is, in your spec_helper.rb, do this, which provides access to the have_received matcher.

require 'rr'
Spec::Runners.configure do |config|
  config.mock_with RR::Adapters::Rspec
end

Spying on Java

Honourable mention: if you're lucky (*cough*) enough to be coding Java, I HIGHLY recommended Mockito, which also implements test-spies, and is easily the best Java mocking/stubbing library around.

There I ran a session on "Attacking slow-running CI builds". It was a small group, but an interesting discussion, I think. Here are my (rough, unedited) notes:

WHAT is the impact of a slow build?

• fewer checkins
• more waiting
• context switching
• discourages integration
• discourages writing of additional tests
• more chance of overlapping checkins
• more build breakages
• more time required to get the build fixed
• reduced productivity
• WASTE!

WHY is the build slow?

• slow tests (particularly acceptance tests)
• over-testing (testing the same code-paths repeatedly)
• expensive set-up and tear-down
• too much testing via the user-interface
• tests that pause, sleep, or poll (e.g. to deal with AJAX)
• too much I/O!
• use of slow infrastructure components (database servers, application servers, etc.)
• slow hardware

HOW can we make it faster?

• faster hardware
• run tests in parallel
• distribute tests
• fail fast
• selective testing: run tests most likely to fail first
• could use dependency-analysis to identify which tests were affected by recent commits
• refactor story-based acceptance tests into scenario-based tests
• bigger tests, with more assertions, offsets set-up/tear-down costs
• but makes tests more complex
• share test fixtures between a group of tests
• but breaks test isolation
• avoid I/O
• in-memory database
• in-memory file-store (RAM disk?)
• stub out infrastructure components
• avoid testing these components by side-effect
• populate the database directly, rather than using the user-interface to set-up for a test
• separate your system into components that can be tested independently

Thinking about this later ...

There are two types ...

The suggestions for improving build times seemed to fall into two categories:

1. optimise the build/tests
2. throw additional hardware at the problem

My problem with the "throw hardware at it" approach is that it typically only helps for the build-server machine; the poor old developers are still left with a slow-running build, and therefore many of the productivity issues still exist.

Another idea

It occurs to me now that we missed a fairly fundamental trick to improve test times: improve the performance of the system-under-test itself. It's a great excuse to start thinking about performance earlier in the project.

"Customer Acceptance Test" does not need to mean end-to-end

On all the projects I've been on in recent years, we've ended up with the majority of the tests being either "developer unit tests", which run super-fast, or "customer acceptance tests" which test end-to-end (browser-to-database) and run super-slow.

Methinks it should be less black-and-white. If we can demonstrate functionality that the customer cares about by calling the underlying logic directly (i.e. at unit-test level), rather than by exercising the user-interface, then what's wrong with that? (We just need one test to prove that the underlying logic has been properly integrated into the UI.)

<% form_for :customer, :url => customers_path() do |customer_form| %>
  <p>
    <label>Name:</label> 
    <%= customer_form.text_field :first_name, :size => 15 %>
    <%= customer_form.text_field :last_name, :size => 20 %>
  </p>
  ... etc ...
<% end %>

and you get

<form action="/customers" method="post">
  <p>
    <label>Name:</label> 
    <input id="customer_first_name" name="customer[first_name]" size="15" type="text" />
    <input id="customer_last_name" name="customer[last_name]" size="20" type="text" value="" />
  </p>
  ... etc ...
</form>

Rails generates sensible field names and ids for you, and slurps existing values out of the model object. So far, so good.

Lately, I've taken to using the same trick when presenting data, not just when editing it. So, whereas before I might have written:

  <p>
    <label>Name:</label> 
    <span id="customer_first_name"><%= h @customer.first_name %></span>
    <span id="customer_last_name"><%= h @customer.last_name %></span>
  </p>
  ... etc ...

I'll now code it up as:

<% fields_for :customer, :builder => ReadOnlyFormBuilder do |customer_form| %>
  <p>
    <label>Name:</label> 
    <%= customer_form.text_field :first_name, :size => 15 %>
    <%= customer_form.text_field :last_name, :size => 20 %>
  </p>
  ... etc ...
<% end %>

and get the same output. (In case you're wondering, the ids are there to help with automated testing).

Note the similarity between the last code snippet and the first one on this page; apart from the first line they're indentical. Usually, I'll put the field-declarations themselves in a partial that's shared between "new", "edit" and "show" actions. That way, your "show" page automatically gets identical layout to the others, just with raw values in place of editable fields.

The ReadOnlyFormBuilder class itself it fairly straightforward - I'm planning to wrap it up into a plugin sometime soon. In the meantime, the implementation of text_field looks something like this:

def text_field(attribute, options={})
  content_tag("span", html_escape(value_of(attribute)), :id => "#{@object_name}_#{attribute}")
end

def value_of(attribute)
  value = model.send(attribute)
end

def model
  @object || @template.instance_variable_get("@#{@object_name}")
end

module Rake
  class Task
    def execute_with_timestamps(*args)
      start = Time.now
      execute_without_timestamps(*args)
      execution_time_in_seconds = Time.now - start
      printf("** %s took %.1f seconds\n", name, execution_time_in_seconds)
    end
    
    alias :execute_without_timestamps :execute
    alias :execute :execute_with_timestamps 
  end
end

These ideas may seem blindingly obvious to some readers, but I suspect they'll appear new-and-strange, and perhaps even bad-and-wrong, to others. In any event, I hope to trigger some thought/discussion.

Principles

My build approach is based on two simple principles:

• Efficiency - don't rebuild up-to-date outputs
• Safety - do rebuild out-of-date outputs

(By "output", I mean some artifact produced by the build. I'm avoiding the word "target" here, since it has specific meaning in Ant.)

Efficiency - DON'T rebuild up-to-date outputs

Quick builds, and rapid feedback, are important for developer productivity. Using a build system that recreates everything from scratch after even a minor change is a great way to kill productivity.

Re-executing a single build step is typically not the end of the world, but many outputs are also inputs to other build steps, so unnecessarily rebuilding an output early on during the build can trigger rework all the way through.

Safety - DO rebuild out-of-date outputs

On the flip side, when a key input DOES change, you need to ensure that all the derived outputs are rebuilt, or at least revalidated. Otherwise, your build becomes "flaky" and unpredictable.

A flaky build forces developers to compensate somehow, e.g. by explicitly running "clean" builds every time, whch impacts productivity.

Tips

Explicitly declare dependencies between your targets

Some people are reluctant to declare dependencies, because declaring them introduces overhead. But not doing so is unsafe, because it opens the door to build steps being executed with stale inputs, resulting in confusing, frustrating, non-deterministic build behaviour.

If you've followed the "Don't rebuild up-to-date outputs" rule, then dependencies should be safe/cheap, ie. there's minimal overhead, and no reason not to declare them.

Targets should be Nouns, not Verbs

Typically, programmers name Ant targets by what they do, e.g. "compile", "test". However, this tends to produce very procedural builds.

So instead, I recommend choosing names describing what the target produces, e.g. "classes", "test/report". Perhaps it's just because I spent so many years automating builds using make, but I find that such noun-ish targets help in various ways:

• it's easier to understand what outputs each target produces (for obvious reasons)
• intermediate targets tend to become useful in their own right
• dependencies become clearer, as it makes more sense to depend on a concrete input, rather than a process

If you've read this far, go read Martin Fowler's "OutputBuildTarget" article; he explores the subject more eloquently than I'm capable of.

Some targets might not produce a concrete artifact (or the artifact might not be the main point of the target). In such cases, I'll sometimes name them based on the condition they produce, or ensure. For example, a target using Simian to check for duplication might be called "minimal-duplication" (as opposed to "simian").

Use <uptodate> to avoid unnecessary rework

Most Ant tasks include dependency-checking based on file timestamps, and will avoid rework. But some tasks aren't so clever. For instance, the <junit> task will happily re-run all your tests, even if they all passed last time, and neither code not tests have changed.

The <uptodate> task can help fill the gap. It compares the timestamps of specified input and output files, and sets a property indicating that work can be avoided.

Here's an example where <uptodate> is used to avoid unnecessary re-generation of XML-mapping code:

<target name="xml-module/check"
        depends="properties">
    <uptodate property="xml-module.uptodate"
              targetfile="${xml-module.jar}">
        <srcfiles dir="spec" includes="**/*.xsd"/>
    </uptodate>
</target>

<target name="xml-module"
        depends="xml-module/check, xmlbean/taskdef"
        unless="xml-module.uptodate">
    <xmlbean destfile="${xml-module.jar}"
             classpathref="xmlbeans.classpath">
        <fileset dir="spec" includes="**/*.xsd"/>
    </xmlbean>
</target>

Use <touch> to record a completed task

Although it's unusual, some build steps have no output: they are simply processes that must be executed, e.g. validating the format of a file, or verifying adherence to coding standards (Checkstyle, Simian). Other build steps can produce many outputs, e.g. code-generation tools.

In these cases, where there's no identifiable primary output, it can be useful to invent a placeholder output-file using Ant's <touch> task. The resulting file is empty, but it's timestamp can be used for dependency-checking, to determine if/when the build step needs to be re-run.

<touch> is most useful in conjunction with <uptodate>, as in the following example:

<target name="libs/check">
    <uptodate property="libs.uptodate">
        <srcfiles dir="." includes="ivy.xml"/>
        <mapper type="merge" to="lib/.done"/>
    </uptodate>
</target>

<target name="libs" description="retrieve dependencies with ivy"
        depends="libs/check" unless="libs.uptodate">
    <ivy:retrieve pattern="lib/[conf]/[artifact].[ext]" />
    <touch file="lib/.done" />
</target>    

Here we're using Ivy to download third-party libraries. After download, we create a touch-file to mark the job as done. On subsequent runs, the library resolution and download process will be skipped, unless the "ivy.xml" control-file has been changed.

As I alluded to earlier, I have also used the combination of <touch> and <uptodate> to:

• skip code-style checks when code hasn't changed
• skip tests when neither code nor tests have changed

Use <dependset> to remove out-of-date outputs

When Ant is not clever enough to determine when something needs re-doing, the <dependset> task is useful for mopping up stale outputs.

Pitfalls

Avoid "private" targets

Many builds include "private" or "hidden" targets, that are unsafe to call directly. A common convention in the Ant world is name these targets starting with '-', since that makes them inaccessible from the command-line.

I think private targets are a smell: they indicate that implicit dependencies are present in the build. Hiding the unsafe targets makes sense, in a way ... but I much prefer to make the dependencies explicit, as described above, at which point it's safe to let every target be called directly (which often comes in handy when testing some aspect of the build process).

Avoid targets depending on "clean"

Having popular targets depend on "clean" is a bad smell. You DO need to avoid using artifacts from previous builds which have passed their use-by date, but starting the whole build from scratch is overkill, when proper dependencies and careful timestamp-checking can ensure that just the stale stuff is rebuilt.

Avoid <copy overwrite="true">

An anti-pattern I often encounter (and a pet peeve) is:

<copy overwrite="true" ...>
    ...
    <filterset>
        <filter token="PASSWORD" value="${db.password}"/>
        ...
    </filterset>
</copy>

The "overwrite" attribute causes Ant to copy files every time, ignoring the usual timestamp-checking that prevents re-generation of up-to-date files. Using "overwrite" can easily cause most of your jars/wars/ears/etc to be updated with every build.

Instead, use <dependset> to invalidate the outputs in the case that ${db.password} has changed.

• Symbol#to_proc
• Nat Pryce's articles on Higher-Order Messaging in Ruby
• Groovy's implied "it" closure-parameter

I've cooked up a shortcut for generating simple blocks, meaning that rather than

people.select { |x| x.name.length > 10 }

I can write such things as:

people.select(&its.name.length > 10)

Disclaimer: I think this is more "cool hack" than useful tool; it's probably too much of an alien artifact to be useful in real life. And it's not generally applicable, like "it" in Groovy. And really, it's not that much more verbose to use a block. Aaaaaanyway ...

The trick is that the above is parsed as

people.select(&(its.name.length.>(10)))

The "its" method creates a MessageBuffer object, which records the messages (method invocations) sent it's way:

irb(main):001:0> require 'message_buffer'
=> true
irb(main):002:0> its
=> #<MessageBuffer:0x6b40b44 @messages=[]>
irb(main):003:0> its.name.length < 10
=> #<MessageBuffer:0x6b3e678 @messages=[[:name], [:length], [:<, 10]]>

Now, the "&" operator coerces it's argument to a Proc, and MessageBuffer#to_proc generates a Proc that replays all the recorded messages. Q.E.D.

The full source-code is fairly short, so I'll include it inline:

class MessageBuffer 

  instance_methods.each do |m|
    undef_method m unless m =~ /^(__|respond_to|inspect)/ 
  end
  
  def initialize
    @messages = []
  end

  def method_missing(*message)
    @messages << message        # record the message
    self                        # return self so we can keep recording
  end
  
  def __replay_all_messages__(obj)
    @messages.inject(obj) do |obj, message|
      obj.__send__(*message)
    end 
  end
  
  def to_proc
    proc { |x| __replay_all_messages__(x) }
  end

end

def its
  MessageBuffer.new
end

Update: Florian Gross suggested a better way to replay recorded messages, using inject, and I've updated the code accordingly.

• Ruby@EJA

It contains a few examples showing how expressive Ruby can be, when compared to Java.

A software framework offers to solve 80% of my problem, but usually without understanding what my problem actually is.

A toolkit is collection of tools. I can pick them up and use them as I see fit. I can use individual tools/components, without needing to adopt them all. I can use them in conjunction with other tools I have, without voiding any warranties.

Grumble.

I wanted to be able to trace the conversation with the IMAP server (or at least, Ruby's IMAP API), to see what was going on. Initially, I started sprinkling tracing statements throughout my code, until I realised that it was going to be easier to define a simple "tracing proxy", and wrap it around the object I wanted to trace:

imap_handle = TracingProxy.new(imap_handle, $stderr)

# ... do stuff with imap_handle ...

It turned out to be straightforward to implement:

class TracingProxy

  def initialize(obj, dest) 
    @obj = obj
    @dest = dest
  end

  def method_missing(symbol, *args)
    arglist = args.map { |a| a.inspect }.join(', ')
    @dest.puts "#{symbol}(#{arglist})"
    rval = @obj.send(symbol, *args)
    @dest.puts ">> #{rval.inspect}"
    rval
  end
  
end

method_missing is a fallback method invoked when the called method isn't found - it's great for implementing dynamic proxies. There's nothing particularly ground-breaking going on here - this kind of trick is fairly common in Ruby-land.

My point is: implementing a dynamic-proxy for tracing was so easy in Ruby that I actually did it. I could have done something similar in Java, using java.lang.reflect.Proxy, or cglib - but I most likely wouldn't have bothered.

In Ruby, implementing the proxy made my life easier, not harder. Ruby encourages me to produce better designs.

Such features depend quite heavily on gleaning data-type information from the code, which is fine for languages like Java and C#. But in dynamically-typed languages like Ruby, we don't have that type info, so things like method-name completion and automated renaming become impossible. (Or so I thought).

Stealing a trick from SmallTalk

It's been puzzling me that there isn't better refactoring support for Ruby, given that the whole concept of refactoring grew out of the SmallTalk community, in the first place. Or more accurately, I've been confused about how automated refactoring could be possible in a dynamic language like SmallTalk.

Then, recently, I stumbled across a paper describing "A Refactoring Tool for Smalltalk", which contains the following explanation:

The Refactoring Browser uses method wrappers to collect runtime information. These wrappers are activated when the wrapped method is called and when it returns. The wrapper can execute an arbitrary block of Smalltalk code. To perform the rename method refactoring dynamically, the Refactoring Browser renames the initial method and then puts a method wrapper on the original method. As the program runs, the wrapper detects sites that call the original method. Whenever a call to the old method is detected, the method wrapper suspends execution of the program, goes up the call stack to the sender and changes the source code to refer to the new, renamed method. Therefore, as the program is exercised, it converges towards a correctly refactored program.

Ah-ha! Cunning.

The Ruby version

As it turns out, we can do much the same thing in Ruby ... leaving aside the "go up the call stack and change the source code" part.

Here's the supporting code:

def method_renamed(h)
  old_name = h.keys[0].to_sym
  new_name = h.values[0].to_sym
  define_method(old_name) { |*args|
    file, line = caller[1].split(':')
    warning = "##{old_name} renamed to ##{new_name}"
    $stderr.puts "#{file}:#{line}: #{warning}"
    send(new_name, *args)
  }
end

Okay, here's a method I want to rename:

class LinkPanel

  def render
    # ... 
  end

end

When I rename it, I also record the change using method_renamed:

class LinkPanel

  method_renamed :render => :to_html

  def to_html
    # ... 
  end

end

Now, I run my tests, and calls to the renamed method result in warnings:

/home/mikew/eyaw/sidebar.rb:229: #render renamed to #to_html

With a single key-chord in my Ruby IDE, I can jump directly to the source-code in question, and fix up the call. I imagine that an ever-so-slightly-more intelligent IDE could complete the refactoring, applying the rename to the call-site automatically! Later on, when I'm confident that everything has been cleaned up, I'll go back and remove that method_renamed alias.

There's more to refactoring than just renaming stuff, of course. I think the "dynamic analysis" trick would be useful to support other refactorings, too ... though I haven't tried it yet.

Proviso: this approach relies on actually running the code, preferably from tests. As the original paper says:

.. the refactoring is only as good as your test suite. If there are pieces of code that are not executed, they will never be analyzed, and the refactoring will not be completed ...