Rails Review: Fiveruns

Bonanzle has twice tried to implement Fiveruns software for our site, so I thought that some might find it handy to hear our impressions of it. Here you are!

Quick Overview

Fiveruns is a fairly new, Rails-centric company that develops a full system monitoring stack for Rails application. They claim to monitor everything from Apache to MySQL to Rails to your Linux OS, all from within one web-based console. I believe they monitor other systems as well, but those were the ones we tried. There aren’t too many other players in this space yet, so the promise of being able to comprehensively monitor, both for errors and performance, the full Rails stack is an idea that should sound intoxicatingly wonderful to any busy Rails developer.

At press time, they were charging something like $40/month per box and $10/month for each additional Rails app… or something along those lines. Short story: if you have a full stack on one or two boxes, you’d pay something between $50-$100 month to monitor it.

Installation

The first time we installed Fiveruns, in late 2007, was something of a mess. Though their web site has (and had) great presentation, the actual technical meaty details describing what to download and what to do with it were pretty lacking. Installing the Rails plugin itself was pretty straightforward (just a script/plugin install), but much configuration was needed afterwards before we had it up and running.

The second time we installed it, right after Railsconf in early May, was better, but still pretty painful. They now have a gem-based install for the plugin and OS process you’ll need, and better (though not great) instructions on how to setup the gem. But after spending an hour or two following their instructions as best I could, we were 1 for 4 in systems that were up and monitored (Linux monitoring was up, Rails/Mysql/Apache notsomuch).

The Fiveruns support team was very helpful. I dealt mostly with Rachel. Rachel and I spent a few days going back and forth about different possibilities as to why our various systems were not working. We dealt with user/directory permissions problems, Apache conf issues, some settings in the Fiveruns web console, and a variety of other issues. Though many of our problems were technically “user error,” I would prefer to characterize them as “user error brought about by lack of specific instructions.”

In total, it took somewhere around 10 hours to get three of our four applications (all but Apache, which I decided I didn’t really care about) registering as “active” in the Fiveruns console.

Features and Usage

I was interested in Fiveruns specifically because I wanted to know how often our Rails actions were being called, and how long those actions were taking to process over time. This should be Fiveruns bread and butter, but we never realized the promise of this during our month with Fiveruns.

I’m not sure why. In the Fiveruns console, data from a Bonanzle Rails application was being received, but only a small fraction of our overall actions were being reported in the console. After running for two weeks, I think it had registered maybe 100 total action calls (vs. a few thousand action calls that had actually occurred). The action completion times it had registered seemed suspiciously incongruent with the times reported by our ApacheBench testing. Also suspicious was that the Fiveruns console reported us as having a “single Rails app” but not a Mongrel cluster, when in fact our application is a Mongrel cluster. It made me wonder if somehow the data Fiveruns was receiving was from one of our debug machines?

The UI for the statistics that we did accumulate was nice to look at, but difficult to use effectively. All metrics for Fiveruns are reported via Flash time graphs that go back something like 12-24 hours. They looked good, but the problem was that, even after hunting around their UI for 15 minutes, I could not find out how to change the time range to go back more than 24 hours. I also couldn’t figure out how to aggregate statistics. I can’t imagine that there isn’t a way to change the time range and see the big picture of how performance is changing over time, but as Don’t Make Me Think would preach — the responsibility for making users feel smart lies not with the user, but with the designer. I felt dumb trying to learn what I wanted to learn from Fiveruns statistic reporting interface.

Other Concerns

I have no way to verify that this definitely resulted from Fiveruns, but for the weeks we ran the Fiveruns software on Rails 2.0.x (and not before, and not so far after), once or twice a day we would receive nil exceptions from fields in records that were not nil. This meant that sometimes items that had been posted for months would, very, very occasionally, say their price field was nil. Obviously these errors were impossible to track down or definitively attribute to any cause, but the fact that we have not gotten one of them in the two weeks since uninstalling Fiveruns, while we were getting a couple per day with Fiveruns installed, suggests a strong causal probability.

The other concern I had with the software was privacy. Obviously the source code for a Rails project is its crown jewel, but looking through Fiveruns logs (and from the details provided by the Fiveruns team), it seems that Fiveruns could look into or actually send to Fiveruns any or all of our Rails source code. I don’t like to be paranoid, but when we’re talking about the source code for our site, the very basis of our existence, this possibility made me feel uneasy.

Fiveruns Response

After spending more than a day to just get Fiveruns up and running in the first place, I’ll admit that I was less than enthused about the possibility of reporting our problems and working through them with Fiveruns, taking still more of our time. There is hardly a feature on our site that took more than 10 hours for me to implement, and the fact that we dropped that much time just to get Fiveruns installed (poorly), left me reluctant to invest still more time working with them to debug our problems.

After a few emails from their sales coordinator asking me if I was going to sign up or not, I did eventually email them a list of my issues. I suggested that, as much as I want Fiveruns to be the solution, at some point their engineers need to step up their game and deliver a product that “just works” for the very useful purposes it purports to. I sent that mail a few weeks ago, and have not heard back from them since.

Final Analysis

Unlike the BackgroundRb review I’m going to write in the next few days, I have hope for Fiveruns. Their product makes wonderful promises, their support team was extremely friendly, and they have enough millions invested in them that I think they will ultimately be able to sort out their problems. Obviously, my experience with them is the experience of but one person, so it is also possible that we just had the worst batch of luck possible. This possibility seems somewhat far-fetched to me, since we are running a pretty vanilla stack of Apache/Mongrel/Gutsy, but if everyone had as many problems as we’ve had, Fiveruns wouldn’t be in business.

My hope is that someone at Fiveruns will either read this, or hear from users that read this, and will circle up the wagons to capture the unrealized potential with this service. Failing that, we will be looking for alternate services that can provide an easy way for us to monitor the performance of our Rails stack, preferably without them having unfettered access to our entire source tree.

If anybody has any alternate service suggestions, please comment below!