Technical Pickles

When you're on your production machine, you probably want to always use the production environment. I've found myself often launching script/console only to accidentally start using it while in the development environment.

Here's one approach to ensuring this doesn't happen on Gentoo.

technicalpickles ~ # echo "RAILS_ENV=production" >> /etc/env.d/99railsenv
technicalpickles ~ # env-update
>>> Regenerating /etc/ld.so.cache...
technicalpickles ~ # source /etc/profile
technicalpickles ~ # echo ${RAILS_ENV}
production
technicalpickles ~ # cd /u/apps/technical_pickles/current/
technicalpickles current # script/console 
Loading production environment.
>>

And there you go.

So I'd like Pidgin to report idle time sanely on my Gentoo box.

For the longest time on this install, I've only been able to choose 'Never' or 'From last message sent.' Neither of those seem to give an accurate sense of your idleness.

For me, idleness really is based on your being or not being at the computer, typing or mousing away.

After grumbling about it long enough, I finally started look for information about it, googling for 'gentoo pidgin idle'. The first result was from the Ubuntu forums mentioning how you needed xscreensaver, and to recompile, to get the desired behavior.

Sure enough, pidgin hadn't been compiled with the xscreensaver USE flag. Remerging with it enabled did the trick.

I really wouldn't have guessed that xscreensaver would have provided this. It almost makes me think that just having these global use flag descriptions isn't enough; it'd be great to be able to know how a USE flag affects a particular package.

Java has more standardized APIs than you can shake a stick at. Servlets, JSP, EJB, JPA, and the number is always growing as people put together new JSRs. There's even more defacto standards, like Spring, Ant, Maven, etc.

What you don't see is standardized practices for common things you have to do when you maintain an application. I'm not talking about code here. I'm talking about things like getting a development environment going, or maintenance of configuration and database schema, or deploying applications to staging areas or production.

Time and time again, it seems like different companies / developers / etc roll their own ad hoc solutions to these problems.

I'll illustrate with the example of building a developer environment.

Developer environment

Setting up developer environments typically sucks. Does this sound familar?

• Download a JDK, install it somewhere
• Download Tomcat, extract it somewhere else
• Download Eclipse, extract it somewhere random
• Download Ant, extract it somewhere completely different
• Hack together a .bash_profile, and update the PATH to point at all these random paths you've put together

If you are lucky, there might be scripts to automate this, or maybe a tarball with everything you need and the appropriate shell files to get a sane shell environment.

If you are even luckier, you are running Java on Gentoo, and can just do 'emerge www-servers/tomcat dev-util/eclipse-sdk dev-java/ant'.

How can we right this wrong?

So what made building a dev environment less sucky? Tools.

In the first scenario, there were none. You were left to manually do everything yourself. Need to upgrade something? Have fun figuring out where you should drop the files and ensuring that your shell scripts are pointing at the right locations. Accidentally threw your machine out the window and have to build a new one from scratch? Good luck remembering all the intricacate steps you took to get to your last good state. The result: Pain to reproduce, and annoying to maintain.

In the next scenario, you do have some tools for getting things an initial environment going. But there probably aren't tools for making sure you're up to date. So while you can reproduce environments readily enough, you can't necessarily maintain them easily over time, and now you have the additional burden of maintaining these tools.

In the last scenario, we're entirely leveraging tools that are already out there. But what if it doesn't quite fit your needs? Well, it's all open source, so you have all the access you need to improve things. And for bonus points, you can improve things in a general enough way so that you can send your improvements back upstream so that everyone using the tools can benefit.

So what exactly are the tools doing here?

• Making your life easier
• Creating de facto standards

Summary

Build a better future through superior tools. Yeah, I like the ring of that.

For some time now, there has been a file collision between rails 1.1.x and rails 1.2.x, because it is a) slotted so both can be installed at the same time and b) both install /usr/bin/rails.

This issue has single handedly kept rails-1.2.3 from being stablized until now.

Since they are battling over /usr/bin/rails, it made sense for each slot to install versioned rails script, like /usr/bin/rails-1.2.3, and then use eselect to manage what /usr/bin/rails points at. This is a pretty common use case for using eselect.

Hans de Graaff, a fellow Gentoo Rubyist, did the initial work getting a new eselect module going. After reviewing and discussing it, we found a few issues that needed to be addressed. I took the torch and ran, and made sure most of the upgrade paths would worked properly.

So without further ado, I present you with dev-ruby/rails-1.1.6-r2 and dev-ruby/rails-1.2.3-r1.

These were barely committed, it might take a few hours to initially hit the mirrors.

Just 'emerge --u rails', and you'll get the updates.

Note the use of '-u'. This will ensure that both slots get updated at the same time (if they are installed) without an annoying blocker.

Thanks goes out to Hans de Graaff for his initial work and subsequent feedback, and Zack Medico for his initial post about sane upgrade paths with blockers, and subsequent help with figuring out how to get it to work with rails and eselect-rails.

Capistrano 2.0.0 has been released for just over a month, but there hadn't been much activity for it on the Gentoo front.

The reason for this is that we had 1.4.x in the tree as SLOT=0, and the prelease 1.99.x also in the tree (but package.masked) as SLOT=0 as well. The real problem is that they were actually intended by upstream to be installable at the same time, so you could have projects on the same machine either version. The logical action would be to slot it accordingly, but we would run into the same issues we had with rails because of a file collision with both slots wanting to control /usr/bin/cap.

We had originally thought to do another eselect module again, but it turns out that upstream was one step ahead of us. /usr/bin/cap actually allows for running either version. Check it:

nichoj@metaverse ~ $ cap --version
Capistrano v2.0.0
nichoj@metaverse ~ $ cap _1.4.1_ --version
Capistrano v1.4.1
nichoj@metaverse ~ $ cap _2.0.0_ --version
Capistrano v2.0.0

What we ended up having to do was pull out /usr/bin/cap into a separate package, capistrano-launcher. We used appropriately blockers again to have a sane upgrade path.

I also had to do a slotmove of the existing versions, putting 1.4.x in SLOT=1 and 1.99.x into SLOT=2. This was as easy as updating the ebuilds, and adding the following to ${PORTDIR}/profiles/updates/3Q-2007:

slotmove =dev-ruby/capistrano-1.4* 0 1
slotmove =dev-ruby/capistrano-1.9* 0 2

Without further ado, make with the emerging.

emerge -atv =capistrano-1* =capistrano-2* # get both versions
emerge -atv capistrano # or just the latest

I wrote earlier about our ebuilds for Rails being file collision free.

To get around having a blocker exposed to the user, rails had a post dependency on eselect-rails, meaning that eselect-rails would get merged AFTER rails. What I failed to remember, though, was that the eselect module was used during the merge of rails, so it had to be a build-time dependency, meaning it'd be pulled in before rails was merged.

This resulted in bug #191015.

Just changing it to DEPEND, and leaving the blockers in eselect-rails would have made annoying blockers, that would probably be a bit confusing for end-users.

So I did a bit of thinking about what really should block what. Versions less than 1.1.6-r2 are in SLOT=1 and install /usr/bin/rails. Versions 1.2.0, 1.2.1, 1.2.2, and 1.2.3 are in SLOT=2 and install /usr/bin/rails. So why not just make those versions block each other?

For less than 1.1.6-r2, we have:

DEPEND=">=dev-lang/ruby-1.8.2
        -snip-
        !~dev-ruby/rails-1.2.0
        !~dev-ruby/rails-1.2.1
        !=dev-ruby/rails-1.2.3"

And in 1.2.0, 1.2.1, 1.2.2, and 1.2.3, we have:

DEPEND=">=dev-lang/ruby-1.8.5
        -snip-
        !<dev-ruby/rails-1.1.6-r1"

I then removed the blockers in eselect-rails, so now it just depends on eselect.

So here are some use cases, and what happens you try to do emerge -u rails.

If you don't care about these, you can just skip down to the summary.

rails-1.1.6-r1 installed

/usr/bin/rails is from rails-1.1.6-r1

metaverse ~ # emerge -ptvu rails

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild     U ] dev-ruby/rails-1.1.6-r2 [1.1.6-r1] USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild  NS   ] dev-ruby/rails-1.2.3-r1  USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild  N    ]  app-admin/eselect-rails-0.10  0 kB 

Total: 3 packages (1 upgrade, 1 new, 1 in new slot), Size of downloads: 0 kB

The version in SLOT=1 gets updated to the latest, and the version in the new slot gets brought in.

Version 1.2.3-r1 gets merged first. During its postinstall, it tries to update the /usr/bin/rails symlink, but it's a file at this point, so it ignores it.

Version 1.1.6-r2 gets merged next, then 1.1.6-r1 gets unmerged because it's in the same slot. This results in there being no /usr/bin/rails. When the postinstall runs, it sees this, and populates it to point use this version.

/usr/bin/rails is a symlink managed by eselect-rails, and points at /usr/bin/rails-1.1.6

rails-1.2.3 installed

/usr/bin/rails is from rails-1.2.3.

metaverse ~ # emerge -ptvu rails
These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild     U ] dev-ruby/rails-1.2.3-r1 [1.2.3] USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild  N    ]  app-admin/eselect-rails-0.10  0 kB 

Total: 2 packages (1 upgrade, 1 new), Size of downloads: 0 kB

The version in SLOT=2 gets updated to the latest.

Version 1.2.3-r1 gets merged next, then 1.2.3 gets unmerged because it's in the same slot. This results in there being no /usr/bin/rails. When the postinstall runs, it sees this, and populates it to point use this version.

/usr/bin/rails is a symlink managed by eselect-rails, and points at /usr/bin/rails-1.2.3

rails-1.1.6-r1 and rails-1.2.3 installed

This can only happen if you don't have FEATURES="collision-protect" in your make.conf.

/usr/bin/rails ends up being from whichever version was last installed.

metaverse ~ # emerge -ptvu rails
These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild     U ] dev-ruby/rails-1.1.6-r2 [1.1.6-r1] USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild     U ] dev-ruby/rails-1.2.3-r1 [1.2.3] USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild  N    ]  app-admin/eselect-rails-0.10  0 kB 
[blocks B     ] =dev-ruby/rails-1.2.3 (is blocking dev-ruby/rails-1.1.6-r1)

Total: 3 packages (2 upgrades, 1 new, 1 block), Size of downloads: 0 kB

Now we have a blocker. What does it mean? It's saying that rails-1.2.3 is in the way of rails-1.1.6-r1. This is a little weird, because you'd think that since both versions are getting updated in such a way that would negate the blocker. It seems a bug has been filed about this very thing.

In the mean time, this is how you resolve this:

emerge -C =dev-ruby/rails-1.2.3

At this point, we only have rails-1.1.6-r1 installed, and it goes along like the first use case.

/usr/bin/rails is a symlink managed by eselect-rails, and points at /usr/bin/rails-1.1.6

If the old reversions have a file-collision, and now block each other, how did you test these use cases?

I disabled collision-protect on the command line, like:

FEATURES="-collision-protect" emerge ...

And I used the --nodeps command line argument for emerge, in order to ignore the dependencies, hence getting around the blockers.

The result is...

FEATURES="-collision-protect" emerge --nodeps =rails-1.2.3 =rails-1.1.6-r1

IMPORTANT This is not how you should get around blockers / file collisions. It really should only be used for testing and development.

Summary

The upgrade path ended up being a bit foobared for 1.1.6-r2 and 1.2.3-r1.

If you were seeing behavior described in bug #191015, I suggest doing:

metaverse ~ # emerge -C rails eselect-rails
metaverse ~ # emerge rails

If you see a blocker, that might look like:

Calculating dependencies... done!
[ebuild     U ] dev-ruby/rails-1.1.6-r2 [1.1.6-r1] USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild     U ] dev-ruby/rails-1.2.3-r1 [1.2.3] USE="fastcgi mysql -doc -postgres -sqlite -sqlite3" 0 kB 
[ebuild  N    ]  app-admin/eselect-rails-0.10  0 kB 
[blocks B     ] =dev-ruby/rails-1.2.3 (is blocking dev-ruby/rails-1.1.6-r1)

You need to...

emerge -C =rails-1.2.3

We all know that test coverage is a good thing, right? Right. In Ruby land, the most common way to do this is to use rcov. I'll be covering getting rcov installed, and how to start using it with your Rails project.

The first thing to do is install it. On Gentoo, this is as simple as:

$ sudo emerge rcov

If you aren't fortunate enough to be on Gentoo, or if you just prefer using gems directly, you can just do:

$ sudo gem install rcov

Now you'll want to install the Rails plugin.

I highly recommend using piston to do this. Among other things, it provides a nice compromise between using svn:externals and just checking in plugins into your own repo. Piston is emergable, or as a gem.

If you go this path, you can do

$ cd vendor/plugins
$ piston import http://svn.codahale.com/rails_rcov

Otherwise, use the plugin script with svn:externals...

$ ./script/plugin install -x http://svn.codahale.com/rails_rcov

or without svn:externals...

$ ./script/plugin install http://svn.codahale.com/rails_rcov

We should be in business now.

Try running everything:

$ rake test:test:rcov

Behold, you now have coverage reports! Check them out at coverage/test/index.html

You should be all good now. Take a chance to look around 'rake -T' to see what other tasks are available for you.

How is my coverage looking? Er, well, turns out I have less than 50% coverage for unit and funcitonal tests, and 0% for integration... looks someone has some tests to write...

In my last post, I was making Git out to be kind of a big deal. One of the links talked about setting up gitosis for managing repositories. It's an excellent walkthrough, but it's pretty generic. This post hopes to provide a Gentooified version of it.

The scenario

I'm making a few assumptions:

• You have a workstation with a normal user, with hostname 'urworkstation' and user 'you'.
• You've installed dev-util/git on urworkstation.
• You have a server with root access, with hostname 'urserver'.
• You will be making a repository called 'gitftw'

Installing gitosis

Gitosis is a fairly recent addition to portage. If you're not using ~arch for your ACCEPT_KEYWORDS, you will need to add an entry to /etc/portage/package.keywords.

root@urserver # echo "dev-util/gitosis" >> /etc/portage/package.keywords

Now you should be able to emerge it.

root@urserver # emerge -atv dev-util/gitosis

Note: -t shows a dependency tree of what will be merged, -v includes a little more verbose output, and -a shows you what will be merged, and asks you to continue.

This should (hopefully!) merge without any issues. It installs all the gitosis files you need, as well as create a 'git' user and group, and a place for your repositories to live in at /var/spool/gitosis/repositories.

You now are ready to start configuring gitosis.

Configuring

You'll need to generate an ssh key, if you don't already have one. I also recommend using keychain for managing your keys/ssh-agents.

you@urworkstation $ ssh-keygen -t rsa

You will be prompted for a location to save the key, but I will assume you choose ~/.ssh/id_rsa. This will create ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. Now you want to toss id_rsa.pub up on urserver using scp.

you@urworkstation $ scp ~/.ssh/id_rsa.pub root@urserver:

After you get id_rsa.pub there, the following command switches to the git user, and kicks off gitosis-init using your ssh public key.

root@urserver #  sudo -H -u git gitosis-init < id_rsa.pub

Note: -H makes sudo switch the user's home directory. -u git says to switch to the git user.

This creates the gitosis-admin repository which you will use to further setup gitosis.

Now you want to clone it:

you@urworkstation $  git clone git@urserver:gitosis-admin.git

Creating new repositories and adding users

The creating of repositories and adding of users isn't really Gentoo specific, so I refer you to the "Creating new repositories" and "Adding users" sections of the gitosis article.

I assume you'll create a repository called 'gitftw', and give yourself access to it.

Making publically accessible using git-daemon

At this point, you have your 'gitftw' repository, and you're able to clone it by doing:

you@workstation $ git clone git@urserver:gitftw.git

This is all well and good if you want only users you've setup to be able to access the repository. But what about if you want to provide anonymous access to the repository? One answer is to use git-daemon which comes as part of git.

The Gentoo package for dev-util/git provides an init script for you, so you just need to tweak the accompanying conf file, and add it to the default runlevel.

There are two ways you can choose to export your repositories. The first is to individually choose which repositories to export, or choose to export everything by default. I think the latter isn't the best idea though, since this includes gitosis-admin repository.

Selectively export

Open up /etc/conf.d/git-daemon, and update it to have:

GITDAEMON_OPTS="--syslog --base-path=/var/spool/gitosis/repositories/"

Now we touch a file 'git-daemon-export-ok' in each repository we want exported.

root@urserver # touch /var/spool/gitosis/repositories/gitftw.git/git-daemon-export-ok

Export everything

Open up /etc/conf.d/git-daemon, and update it to have:

GITDAEMON_OPTS="--syslog --base-path=/var/spool/gitosis/repositories/ --export-all"

Starting it

You're all set to actually start it up now.

root@urserver # /etc/init.d/git-daemon start

Now we try checking it out using the git protocol:

you@urworkstation $ git clone git://urserver:gitftw.git

Assuming all is well, now we can add it to the default runlevel.

root@urserver # rc-update add git-daemon default

Conclusion

Congratulations, you are now all set to manage and play with git on urserver until the cows come home.

Update Jan 22, 2008

Robin Johnson aka robbat2, who maintains the ebuilds for gitosis, posted some important notes about our gitosis package. Mostly, there are some differences between our package and the upstream version of gitosis.

You've probably noticed that there have been a lot more announcements lately if you subscribe to gentoo.org's or planet.gentoo.org's feed. You can largely thank Donnie for this improvement.

Announcements now carry a 'Discuss this!' link. Announcements have historically been one-way communications, of us pushing information to you. These links will hopefully encourage communication in both directions.

We have also been more mindful of how the announcements are written and presented.

• Tables and lists
• Lots of links
• Emphasize important text with bold

This only touches some of the things we have been thinking about when posting.

In the name of openness, I started a announcement guide. This covers in depth how an announcement is made, and consideration that should be taken when writing it. It's still in draft, but it's a decent starting point.

I don't think many people are aware of it, but Gentoo Prefix is a valid alternative to MacPorts and Fink for installing unix-y type software on Mac OS X.

Personally, I just use it to install all those command line tools and utilities I miss from Linux.

Here's my development-centric laundry list of things I've used it to install:

• version control
  
  • cvs
  • darcs
  • git
• Java stuff (yep, it supports Java using Apple's SDK)
  • ant
  • maven
• Others
  • nmap
  • keychain

So remember kiddies, Gentoo works on Mac OS X too!

So one day I was bored, and decided to some self googling. After writing my previous post on Git on Gentoo, I decided poking searching for 'nichoj git'.

Imagine my surprise when I saw it reproduced on the unofficial Gentoo wiki. Sure, I was cited, but there wasn't a link back to the origin source.

Being a wiki, I was just able to just include the link. I got to thinking, and realized I didn't really have anyway to safeguard against this kind of thing.

Being the open source lover that I am, I know there are ways to do this by appropriately licensing your code. (Here's a decent resource selecting one: part 1, part 2)

I was only vaguely aware of the Creative Commons, so I figured it was past time to read up on it.

All my needs really are:

• I want to be attributed if you use my stuff
• I don't want you making money off my stuff without my permission

The Creative Commons website actually has some tools for figuring out what license you should use.

According to this, I should be using:

• Creative Commons Attribution-Noncommercial 3.0 Unported

You can get a snippet to include the license on the page so people can know what is what. I opted to use a slightly altered version in my footer, to better fit in with the style of my site.

And with that, my content is licensed.

Considering the hype about nginx in the Rails community, I figured it was about time to try it out.

Being the slacker that I am, I didn't really want to have to learn how to use it. Fortunately, Ezra gives us a good starting point for a configuration.

Then Chris Wanstrath comes along and creates nginxconfiggenerator to make it even easier, by providing a generator around this configuration.

One kink though. Running on Gentoo and using the latest nginx package. Default template has paths not suitable for this. Bummer.

Being the clever fellow he is, Chris's generator does provide a way around this, by letting you specify your own template to use.

I took a little time to tweak his original template for Gentoo consumption. Behold!

Here's how you'd use it, assuming you've gone through the steps already mentioned in the post.

$ wget http://p.caboo.se/182778.txt -O gentoo_nginx.conf.erb
$ generate_nginx_config config.yml nginx.conf \
  --template gentoo_nginx.conf.erb

You should be all set now.

One last note, you need to set fair: false for your sites. The fair directive doesn't seem to work with the current version we have on Gentoo. I'm not positive, but I think it's related to this post by Ezra

For Ruby on Gentoo, it has been asked how one should go about managing RubyGems. You have a few options:

• Use Gentoo packages exclusively
• Use gem exclusively
• Mix and match and pray

Mix and match

Offhand, mixing and matching seems like a bad idea.

Doing so, you might install a gem using portage, but then you could use gem update to update it. Then you update it with portage...

Overall, seems like it could be problematic in the long run.

Gentoo packages exclusively

Being a Gentoo packager, my first take is that using packages is the way to go.

Some pluses:

• One stop for installing/updating packages, ruby or otherwise
• For gems with native extensions, can have dependencies on the things the native extensions use

Of course, there are a number of disadvantages.

• There are tons of gems out there
• Some update frequently
• Lot of packages to be keeping up with stable keywords
• Despite gem ebuilds being pretty straightforward, there is a bit of grunt work involved on to verify dependencies are accurate
• Lots of gems are unpackaged, because they are usually packaged as requested or as developers need them
• Despite being able to install multiple gems at once, packages usually aren't slotted that way

Gems exclusively

If you're a Ruby developer, the most natural way would be using RubyGem directly. Some nice things include:

• Most direct way to have the latest and greatest gems
• Can have as many versions installed as you want
• Zero maintenance for Gentoo packagers (sweet!)

Of course, not without their drawbacks:

• If a gem uses native extensions, it can be tricky to determine what the it's dependencies are
• Problematic when other (non-ruby) packages need to depend on gems

My current practices

So where does that bring us?

Here are the practices I've taken to using for my Ruby and Rails development and deployment needs.

In development

On my development machine (4 MacBook running Leopard with Gentoo/Prefix), I've using RubyGems directly.

If I'm working on a Rails app, and it needs any gems, I've taken to vendoring everything. This type of thing will be supported out of the box for Rails 2.1.

Of course, this wouldn't work so well for things with native extensions, like hpricot.

In production

I avoid installing any gems in production as much as possible. If an app needs something, it should be vendored, like I just described.

For gems needed by my apps that have native extensions, or are otherwise needed in production (like rake and the mysql gem), again, I'm using RubyGem directly.

One thing that kind of annoyed me about Gentoo Prefix is that you always needed to do a little work to enter the prefix:

$ ./bootstrap-prefix.sh /path/to/prefix startscript

Personally, I just want to always live in a prefixed shell. Amazingly enough, this is really easy to do.

1. First open up /etc/shells as root
2. Add a line like /path/to/prefix/bin/bash and save
3. As your user, run: chsh -s /path/to/prefix/bin/bash
4. Enter your password
5. Done

Now, everytime you open a terminal, it will be running a bash gloriously built by Gentoo

Since I've been less active packaging for Gentoo recently, I've been trying to at least lurk in #gentoo-dev-help and give a helping hand with packaging. If I can't help them directly, at least I try to give them a gentle shove towards something that can.

As a result, I've been collecting various bits of documentation about how to package stuff. This post is a result of this, as well as some other digging around I did to flesh it out.

General development documentation

• Developer Handbook
• Devmanual
• Bugzilla

Channels of communication

Lots of projects and herds within Gentoo have their own mailing lists and irc channels.

• Gentoo IRC channels
• Mailing Lists

Language-specific

Packages written in different languages often have their own way of dealing with packaging:

• Perl
• Python
• Java
• Ada
• PHP

Herds

Same goes for herds.

• Webapps
• Games
• Apache
  • depend.apache.eclass
  • apache-module.eclass
  • apache-2.eclass

Other stuff

These don't fall into any other categories, but are still useful.

• init.d scripts
• autotools
• Writing eselect modules
• Writing GuideXML documentation
• Dealing Automagic dependencies
• Dealing with --as-needed
• Getting Meaningful Backtraces