Chief Technology Officer, July 2012 through April 2013

• I have extensive knowledge of the ‘Rails Machine Stack’, in the sense of all the tools and services that enable day-to-day work to happen:

  • I built everythingisawful, an application to help periodically review & audit the monitoring systems. I use this to find alerts that have been outstanding to track why they were left unattended and get them fixed, as well as used it as a source of inspiration to refactor & improve what we monitored, and the process we use to handle them
  • I periodically reviewed the status of support tickets. I added some heuristics to identify when tickets are in a ‘weird’ state, such as missing a deadline, missing a description, older than a week. For open tickets, etc. I look for ways to close the loops, unblock tickets that have been languishing open. I also have been slowly but steadily improving how we use HelpSpot, including adding and removing filters as they become apparent, customizing statuses, adding custom fields, adding automated responses, and generally improving the day to day experience of HelpSpot.
  • I periodically reviewed backups, including fixing backups on hosts that that are failing, calculating capacity, reallocating, and improving visibility.
  • I was the primary maintainer and contributor of all Rails Machine open source projects.
  • I made massive improvements to moonshine & many plugins, to significantly reduce the amount of manual work that has to happen for managed deploys.
  • I built beanmachine, an application to help automate billing. It has logic to automatically calculate Credit Hours from Application Management fees to apply if they haven’t been used this billing period. It pulls time entries from HelpSpot tickets to add as line items to invoices. For BaseCamp-managed projects, it reduces work even more, with only a few clicks needed to invoice for the previous month’s logged time across all BaseCamp projects. A side effect of this work is that OurCustomers now tracks FreshBook and Ubersmith client IDs, BaseCamp project IDs, and Authorize.net Ids, which could be used for the foundation of further billing improvements.
  • I overhauled how we track & allocate IP addresses. It previously was managed via DNS of managedmachine.com for the ‘managed subnets’, and plain text files for the self-managed subnets. The problem was we ran out of IPs in the ‘managed subnets’, that led to double-allocating IPs, and in addition to generally being time consuming. IPD already modeled IP addresses, but they were treated as artifacts of servers, rather than a first class object themselves. I refactored this modeling such that we model all our Subnets, which has many IP addresses, and a Guest has many IP address assignments, and allow multiple guests to be intentionally assigned the same IP address (ie heartbeat or mmm). As part of rolling this out, I audited our IP assignments to make sure there weren’t duplicates, and that any IP address that was pingable is assigned to a server.
  • I added support to IPD for tracking bare metal hosts. Previously, our HelpSpot/OurCustomers livelookup service wasn’t able to work for bare metal hosts, because they simply didn’t exist. This addition meant that livelookup would work as expected, and baremetal servers could be treated like any other server, in the context of using our internal tools.
  • I automated large parts of the weekly report we generate for our VIP customers. Among other things, it puts together a list of tickets, abuse reports, monitoring alerts, and open GitHub pull requests. The work to do it manually wasn’t hard, but it was time consuming and easy to miss something. This made it trivial to do each week, and also opened the door for applying this to other customers. A side effect of this work is that OurCustomers now tracks GitHub information about account/applications/users.
  • Most day-to-day work relies heavily on systems and tools I’ve built, managed, or improved.

• I implemented ChatOps. As part of that, I dove into learning CoffeeScript and understanding how Hubot worked. I re-implemented our bot using hubot, and re-implemented most of our campfire functionality in hubot. I also began making improvements to hubot and hubot-scripts, and soon was asked to be a hubot maintainer. Since then, I have continued to keep an eye out for things that can either be done directly from Campfire with ChatOps, or at the very least provide additional context for getting work done. Some examples of improvements I’ve made in this vein:

  • claptrap alerts for showing monitoring alerts in helpspot for a specific guest (used for striking)
  • claptrap pmta for showing metrics from PMTA, and claptrap autopmta for displaying those metrics every N minutes
  • claptrap backup critical for showing hosts needing backups fixed
  • claptrap billing for linking to correct billing system for a particular guest/account
  • claptrap basecamp for linking to the basecamp project for a particular guest/account
  • claptrap authorize.net for linking to the authorize.net page for a particular guest/account
  • claptrap dns for linking to the DNS Admin interface for a specific guest
  • claptrap free ip for finding list of anassigned IP addresses
  • claptrap guests for finding guests in IPD
  • claptrap github for linking to the github repo for a guest’s application
  • claptrap hostip for geocoding IP addresses/domains, for cases when a guest is getting slammed by specific IPs, to find out if they are coming from somewhere suspicious
  • claptrap hosts for finding hosts in IPD
  • claptrap moonshine me for linking to moonshine & plugins
  • claptrap newrelic for showing all accounts with NewRelic, or linking to newrelic for a particualr guest, which is super useful while handling extended outages.
  • claptrap page for paging specific people easily from Campfire
  • claptrap papertrail for dumping logs from papertrail into campfire
  • claptrap puppet for linking to Puppet’s Type Reference for a particualr resource, ie file. saves googling
  • claptrap raid for showing hosts with raid warnings
  • claptrap redis for linking to redis command reference
  • claptrap scout for linking to the scout page for a particular guest
  • claptrap teach me for accumulating links for referenec, such as documentation about Rails Machine’s inner workings
  • claptrap who's on call for showing who is on call, also announces it on shift changes

• I spoke at Rocky Mountain Ruby about ChatOps, and received a surprising amount of good feedback on the process I used, as well as the story I told on the road to ChatOps. I also gave the same talk at Geekend, but I don’t think they were quite as ready for it.

• I acted as the Curator of the Rails Machine Way of Support, and work to improve the team’s quality of customer interactions. This spans a lot of things, including:

  • optimizing ticket language for readability
  • wordsmithing ticket & responses to create the right tone of awesomeness
  • rectify the differences between what a customer says, and what a customer wants, and giving them that
  • close ticket loops, and trending ticket towards done
  • practicing inbox zero
  • optimizing handling of tickets to provide great support, while minimizing the number of back & forth responses with customer

• I provided expert operational consulting for our customers customers. Based on some rough scripting, easily paid for my salary in consulting alone. In addition, this consulting has supported a 1000% growth of monthly revenue for a VIP customer, as well as moderate growth for a number of our other customers

• I provided support to a VIP customer for the months leading up to their peak traffic for the year, allowing them to have smooth sailing during their most important time of year, further building their confidence in us.

• I acted as an escalation points for for difficult, critical, and extended outages. Some of the types of issues I’ve worked with in this case:

  • help customers deal with a large influx of traffic
  • diagnose guests not booting
  • diagnose & fix mysql replication failing
  • diagnose & fix heartbeat suffering split brain
  • fix application errors after a ruby upgrade
  • debug large memory increases in application servers
  • debug request queue backing up due to riak issues
  • investigate & fix 503 errors caused by slow customer backup processes
  • blocked evil IP addresses flooding a server with traffic
  • investigate mmm processes with large memory
  • investigate & mitigate application performance issues during peak traffic loads

• I’ve written and collected extensive documentation covering a wide range of topics of topics. This includes the tools we use day to day, to details on the operation of different tools & services, to information about the company. Some examples:

  • haproxy
  • heartbeat
  • mysql
  • mysql mmm
  • memcached
  • redis
  • mongodb
  • xfs
  • caching
  • lvm
  • sidekiq
  • ntp
  • sphinx
  • pagerduty
  • postfix
  • ssl
  • postgresql
  • denyhosts
  • delayed_job
  • solr
  • xsendfile
  • screen
  • resque

• I was a training resources for onboarding new hires. This includes generating documentation, syllabuses, and schedules, as well as directly teaching, pairing, and providing feedback on code & support.