Of Dreams unfulfilled, JMock, and Expectations

I just had an observation about the pattern of setting up expectations of our jmock tests. So here's a typical code block for setting it up:

final List<UserGroup> groups = this.groups;
final List<User> users = this.users;
final User user = users.get(0);

final UserService userService = this.userService;
this.jmockCtx.checking(new Expectations()
{{
  // first, rehydrate
  allowing(userService).findAllGroups();
  will(returnValue(groups));
  allowing(userService).findAllUsers();
  will(returnValue(users));
  will(returnValue(true));
  // now, save
  one(userService).createUser(user);
  one(userService).findAllUsers();
  will(returnValue(users));
}});

I was thinking, hey, why not just use this.groups, this.users, etc, directly in the expectations, instead of making new references to them? It might look like:

this.jmockCtx.checking(new Expectations()
{{
  // first, rehydrate
  allowing(this.userService).findAllGroups();
  ...
}});

This ends up being an error because 'userService' cannot be resolved or is not a field. Unexpected behavior, right? Almost.

A little background about what this code chunk is actually doing: when we do new Expectations() {}, we're actually creating an anonymous subclass. At that point, 'this' would refer to the Expecations subclass, not the test we're in.

When we do the second set of {}s, we're adding an initializer block. The best way of thinking about this is that basically, it gets invoked when you instantiate an instance of an object, similar, but different to the normal contstructor. You may be more familiar with static initializer blocks, doing something like:

public class Something {
  private static Map map = new HashMap();
  static {
    map.put("key","value");
  }
  ...
}

So my dream of using this.users and pals in my Expecations, can only be that, a dream.

Permalink Edit Destroy

Our evaluation of Selenium for web testing

Like the good little code monkeys we are, it is time for us to adopt some form of web testing. There is only so much unit and integration testing can help you, before you need a fuller solution.

Given its maturity, Selenium seemed like the obvious choice. Here's what I found:

The Good

Here's some things we liked:

  • An IDE, as a Firefox Extension, for building unit tests
  • Support for testing in a many languages, such as Java, Ruby, their own Selenese (basically glorified HTML)
  • The IDE generates Selenese, which can easily be converted to one of the above mentioned languages
  • Support for running in multiple browsers across multiple platforms
  • Support for running selenese tests using ant

All in all, pretty cool stuff.

The Bad

However, when we went down to our particulars for our project, things fell a little short. As a brief background, we use ant for building, junit for testing, and bamboo for continuous testing.

You can convert from Selenese to your language choice, but once you do, you can't go back. This is kind of expected, as you can do all sorts of trickery to the code after converting it from Selenese. The API for interacting with Selenium closely follows the Selenese. Here's an example of Selenese, and then converted Java.

<tr>
   <td>open</td>
   <td>/</td>
   <td></td>
</tr>
<tr>
   <td>clickAndWait</td>
   <td>link=diabetic nephropathy, schizophrenia, and bipolar disorder</td>
   <td></td>
</tr>
<tr>
   <td>clickAndWait</td>
   <td>link=Steering Committee</td>
   <td></td>
</tr> 

public class NewTest extends SeleneseTestCase {
   public void testNew() throws Exception {
       selenium.open("/");
       selenium.click("link=diabetic nephropathy, schizophrenia, and bipolar disorder");
       selenium.waitForPageToLoad("30000");
       selenium.click("link=Steering Committee");
       selenium.waitForPageToLoad("30000");
   }
}

The code here produced isn't the cleanest, because the API is closely following Selenium conventions, rather than using the language's conventions. For example, Maps should be used, instead of something like "link=Steering Committee", and Integers should be used instead of number in a String.

Because you can't go back to Selenese from code, we decided we should always work with Selenese, so we can maintain tests easily using the IDE. This means we need to use the 'selenese' ant tasks for running our tests.

This is how our invokation looks:

<selenese suite="${dir.webtest}/selenium_test_suite.html" 
  results="${dir.selenium}/selenium_test_results.html"
  browser="*firefox"
  timeoutInSeconds="90"
  startURL="${webapp.url}/" />

The first issue is that you have to specify a test suite. You can't just give it a directory, or fileset, that includes all your tests. This is one extra thing to maintain, and it becomes easy to forget to add new tests to it.

The other issue is the output. It's just an HTML. We want to be able to generate junit-like XML, so a report can be generated along with all our other tests.

Coming to terms

Since this is open source, the obvious solution is to get hacking, and so I started delving into the source tree.

I had two primary impressions of the codebase. The first is that there aren't suitable layers of abstraction. For example, instead of having a class representing a suite of tests, and a class for representing tests, java.io.File instances get passed around and grokked as their needed. Second was that it isn't sufficiently extensible for. There weren't any places we could hook in, or classes to extend, to change the results ouput, or even get at it.

On-the-fly suites

The issue here is that the code is expecting a File to be passed in, which contains all the tests. I had hoped it had some abstraction which would parse through the suite, so if you wanted, you could programattically build up a suite.

So here, we can get around this by building up a temporary File, which contains the appropriate HTML for Selenium to understand it as a suite.

JUnit reports

When looking at the code, I had hoped to see like a TestResult class, that you could get the results out of, and then have a way for outputting the results as you see fit.

Since we can't do that, the next best thing would be to parse through the results' HTML, and spit out XML laid out in a way that JUnit's reporting mechanism would be able to understand.

Success!

After a day or two of hacking around, I was able to implement the workarounds I mentioned here.

Unfortunately, the implementations turned out to be a bit too closely tied our environment, so I can't readily turn it out to the rest of the world.

Hopefully, I will have the time to do the necessary refactorings to be able to do so.

Permalink Edit Destroy

Cover your butt with rcov

We all know that test coverage is a good thing, right? Right. In Ruby land, the most common way to do this is to use rcov. I'll be covering getting rcov installed, and how to start using it with your Rails project.

The first thing to do is install it. On Gentoo, this is as simple as:

$ sudo emerge rcov

If you aren't fortunate enough to be on Gentoo, or if you just prefer using gems directly, you can just do:

$ sudo gem install rcov

Now you'll want to install the Rails plugin.

I highly recommend using piston to do this. Among other things, it provides a nice compromise between using svn:externals and just checking in plugins into your own repo. Piston is emergable, or as a gem.

If you go this path, you can do

$ cd vendor/plugins
$ piston import http://svn.codahale.com/rails_rcov

Otherwise, use the plugin script with svn:externals...

$ ./script/plugin install -x http://svn.codahale.com/rails_rcov

or without svn:externals...

$ ./script/plugin install http://svn.codahale.com/rails_rcov

We should be in business now.

Try running everything:

$ rake test:test:rcov

Behold, you now have coverage reports! Check them out at coverage/test/index.html

You should be all good now. Take a chance to look around 'rake -T' to see what other tasks are available for you.

How is my coverage looking? Er, well, turns out I have less than 50% coverage for unit and funcitonal tests, and 0% for integration... looks someone has some tests to write...

Permalink Edit Destroy

Shoulda macros allows you to embrace your inner slacker

So I think I've discovered Shoulda's secret sauce, the reason why you should use it.

Macros.

It's nice and all to have a decent syntax for doing nested contexts, and declaring tests. But the real power comes from having macros that define tests for you.

Here's a fully shoulda-ified model test (from an example on the Shoulda website):

class PostTest < Test::Unit::TestCase
  load_all_fixtures

  should_belong_to :user
  should_have_many :tags, :through => :taggings

  should_require_unique_attributes :title
  should_require_attributes :body, :message => /wtf/
  should_require_attributes :title
  should_only_allow_numeric_values_for :user_id
end

Imagine how much more code you'd have if you were to do this by hand.

It's not magic though. It's pretty easy to write your own. To start with, you can just define them in your test_helper.rb.

Here's the jist of what you do:

  • Define a class method, usually like should_something_something, or maybe something_should_something.
  • In this method, declare a should block. You'd probably want to name this dynamically based on the parameters given to your method.
  • Assert stuff.
  • ...
  • PROFIT!

As an example, here's something I wrote recently:

class Test::Unit::TestCase
  #snip
  def self.should_build_request_path(path)
    should "build request path of #{path}" do
      assert {path == @api_call.build_request_path}
    end
  end
  # snip
end

Then in your tests, you can use it like:

class GetTagsCallTest < Test::Unit::TestCase
  # snip
  should_build_request_path '/v1/tags/get'
  # snip
end

It's pretty straightforward stuff.

If you start to get a lot of these in your test_helper.rb, you can always move them into module, include/extend them. If they are general enough, who knows, maybe release it as a gem.

Permalink Edit Destroy

Boston.rb Hackfest 5/07 Post Mortem

By now, I think it's a running joke that we'd work on the recommendable plugin we started a month or so ago.

Instead, we worked on the Factory stuff. This article captures most of the ideas of using factories for testing and reasons for using them instead of fixtures.

The crew tonight:

Overview

As the article mentions, someone did implement a plugin which captures most of these ideas.

There were a few reasons (and some more which I don't remember) why we didn't like this plugin.

  • Plugin, not a gem, so unusable outside of rails
  • Pollutes the namespace of Object
  • No tests

So, the journey for better Factories begins. The basic ideas/goals we settled on is this:

  • Define your factories in test/test_helper.rb
  • Be able to define a factory for a class with default options
  • Be able to define alternate factories for a class, for example, to create an admin user versus a normal user
  • Flexibility for defining default attributes, ie be able to call other methods
  • Choice of making new instance, persisting a new instance, stubbing an instance, or getting valid attributes
  • Flexible API for actually calling the factories

The API

So we started out with a basic API that Dan Croak had outlined:

Factory.define do |factory|
  factory.add :project do |project|
    project.name {"factoryfu"}
    project.homepage_url {"http://factoryfu.rubyforge.org"}
  end

  factory.add :event, :meeting do |event|
    event.event_type {"Meeting"}
    event.date {Time.now}
    event.description {"an sick meeting"}
  end

  factory.add :event, :hackfest do |event|
    event.event_type {"Hackfest"}
    event.date {Time.now}
    event.description {"a cool hackfest"}
  end
end

As for usage of the factories, we had 3 possibilities:

Factory.create :project, :name => 'ambitious_sphinx'
Factory.create_project :name => 'ambitious_sphinx'
create_project :name => 'ambitious_sphinx'

I'm not going to say much about these, aside from the first being the simplest to implement, the second being the most explicit about what you are doing, and the last is the most concise.

Because it seemed like the simplest, we started implementing the first. I believe that the other two syntaxes can actually be emulated using it.

To simulate the second, as we register factories, we can create the appropriate methods, ie create_event, new_event, valid_event_attributes.

To simulate the third, we can build on the second, and probably do some magic to delegate from TestCase to our Factory.

Implementing it

For simplicity's sake, we started with our work from last week's hackfest on bostonrb.org.

We added the snippet for defining factories into our test_helper.rb, made a new model unit test which used the factory methods, and started a Factory class.

We basically ran the test, watched it blow up horribly, and added some code, until eventually the test passed.

You can see the work we ended up in the hackfest repository under 'boston_rb'

Of particular interest are:

So, I'm just go over a few of the more interesting bits of the implementation:

Factory.define

This is the entrance into defining our factories.

The implementation isn't very interesting. We just call the block on Factory. We were originally making new instances of Factory, and passing that to the block, but we then realized that this made it a little trickier to have factory methods be class-level methods.

Factory.add

This is what you use to define how you the factory behaves.

It takes a block which defines how to create new attributes. If you look at the snippet:

event.event_type {"Hackfest"}
event.date {Time.now}

Notice that event.event_type and event.date are actually taking blocks. This allows us to do Time.now, and have this not actually be invoked until object creation.

Factory.add actually passes your block a magical Hash which captures this attribute/attribute creation data. We'll talk about this has in a little bit.

After we get back from the block, we store in @@model_templates, keyed by the model and kind of the factory.

The Factory methods

We provide three factory methods:

  • valid_attributes: a Hash of valid attributes for the object
  • new: a new unpersisted instance, using valid_attributes
  • create: a persisted instance, using new

You invoke them like:

Factory.valid_attributes :project
Factory.new :project, :name => 'factoryfu', :homepage_url => 'http://somewhere.com'
Factory.create :event, :hackfest, :date => 3.days.ago, :description => 'your looking at it'

The way we parse the arguments is a little tricky. Initially, we only supported the first 2 lines. This meant our method signature looked like:

def self.new(model, attributes = {})

When we went to support the 3rd line, we thought to go like:

def self.new(model, kind = 'default', attributes = {})

By doing so, you break the 2nd line, because kind would get set to our attributes.

activesupport provides a solution: Array#extract_options. Basically, it looks at the Array, and pops the last element off the end if it's a Hash. We use it like:

def self.valid_attributes(model, *args)
  kind, attributes = parse_args(*args)
  # ...
end

protected

def self.parse_args(*args)
  attributes = args.extract_options!
  kind = args.first || 'default'
  [kind, attributes]
end

BlockfulHash

We made a subclass of Hash which lets you define keys like:

event.date {Time.now}
event.description {"This is going to be SICK!"}

Such that it's equivalent to:

event[:date] = {Time.now}
event[:description] = {"This is going to be SICK!"}

Here's the implementation:

class BlockfulHash < Hash
  def method_missing(method, *args)    
    # When Proc.new is given no arguments,
    # it returns a Proc for the block passed to the current method
    # Proc.new... yeah, it's officially a hackfest.
    self[method.to_sym] = Proc.new
  end
end

This BlockfulHash is what gets passed to the block given to Factory.add.

What's next

So, we have something workable here:

  • Uses the syntax we laid out for defining factory
  • Supports the first syntax for using factories (Factory.create :event)

As far as what is left to do:

  • Extract it out of the boston_rb code base
  • More tests
  • Gemify
  • Support Factory.create_event syntax
  • ...
  • PROFIT!!!

Addendum

Some links and other ideas tossed around over the course of the night:

Permalink Edit Destroy