Relentless Simplicity

Update: This plugin now exists! It’s under active development, and has some pretty cool features. Read more about it through the link.

There comes a time in most every Rails applications’ life where its owner wants to up the “Web 2.0” factor and create a blog presence. Luckily, this is Rails, a framework ideally suited for creating a blog. Unluckily, there doesn’t seem to be a de facto Rails blog plugin floating around the community yet.

Now I know what you’re thinking… Mephisto! No thanks. The same factors that drove me away from Beast drive me away from Mephisto as well. Who wants to manage multiple subdomains, multiple app repositories, two sets of Mongrels, cross-subdomain authentication issues, and figuring out how to share resources (like views and images) between your Mephisto app and your standard app?

Upon discovering that no good blog plugin existed, my first instinct was to see if I could bring to life a sort of “Savage Mephisto.” Unfortunately, whereas I had a little bit of time to squeeze in Savage Beast development during Bonanzle’s final run to launch, I have virtually no time whatsoever to work on a project like this, now that Bonanzle is live and doubling in traffic every few weeks. Nevertheless, I did at least give a half-hearted attempt to make Mephisto into a plugin before realizing that as vast and deep as Beast is, Mephisto is probably twice as complicated, and thus, half as plugin-able.

So I turned my attention to creating a blog plugin from scratch. You can get a sense of how far I’ve gotten on the other Bonanzle blog. Like SB, I wrote it using the Engines plugin, so it could theoretically be extracted to a repository for download with relatively little effort. However, I’m not sure what the demand is like for a plugin like this would be? I mean, you could write your own fairly functional blog in a day or less, and you could conquer the cross-domain Mephisto issues, or you could try out one of the not-well-rated blog plugin attempts on Agile Web Development.

Anyway, I’ve decided to use the responses to this post to gauge how much enthusiasm there is for an easy-to-setup Rails blog plugin. If there’s lots, I’ll try to make the time frame to launch short. If there’s little, it probably remains a Bonanzle-only plugin.

Usage: server [options]
-p, –port=port Runs Rails on the specified port. Default: 3000
-b, –binding=ip Binds Rails to the specified ip. Default: 0.0.0.0
-d, –daemon Make server run as a Daemon.
-u, –debugger Enable ruby-debugging for the server.
-e, –environment=name Specifies the environment to run this server under (test/development/production). Default: development
-s, –config-script=path Uses the specified mongrel config script.

-h, –help Show this help message.

Bonanzle has twice tried to implement Fiveruns software for our site, so I thought that some might find it handy to hear our impressions of it. Here you are!

Quick Overview

Fiveruns is a fairly new, Rails-centric company that develops a full system monitoring stack for Rails application. They claim to monitor everything from Apache to MySQL to Rails to your Linux OS, all from within one web-based console. I believe they monitor other systems as well, but those were the ones we tried. There aren’t too many other players in this space yet, so the promise of being able to comprehensively monitor, both for errors and performance, the full Rails stack is an idea that should sound intoxicatingly wonderful to any busy Rails developer.

At press time, they were charging something like $40/month per box and $10/month for each additional Rails app… or something along those lines. Short story: if you have a full stack on one or two boxes, you’d pay something between $50-$100 month to monitor it.

Installation

The first time we installed Fiveruns, in late 2007, was something of a mess. Though their web site has (and had) great presentation, the actual technical meaty details describing what to download and what to do with it were pretty lacking. Installing the Rails plugin itself was pretty straightforward (just a script/plugin install), but much configuration was needed afterwards before we had it up and running.

The second time we installed it, right after Railsconf in early May, was better, but still pretty painful. They now have a gem-based install for the plugin and OS process you’ll need, and better (though not great) instructions on how to setup the gem. But after spending an hour or two following their instructions as best I could, we were 1 for 4 in systems that were up and monitored (Linux monitoring was up, Rails/Mysql/Apache notsomuch).

The Fiveruns support team was very helpful. I dealt mostly with Rachel. Rachel and I spent a few days going back and forth about different possibilities as to why our various systems were not working. We dealt with user/directory permissions problems, Apache conf issues, some settings in the Fiveruns web console, and a variety of other issues. Though many of our problems were technically “user error,” I would prefer to characterize them as “user error brought about by lack of specific instructions.”

In total, it took somewhere around 10 hours to get three of our four applications (all but Apache, which I decided I didn’t really care about) registering as “active” in the Fiveruns console.

Features and Usage

I was interested in Fiveruns specifically because I wanted to know how often our Rails actions were being called, and how long those actions were taking to process over time. This should be Fiveruns bread and butter, but we never realized the promise of this during our month with Fiveruns.

I’m not sure why. In the Fiveruns console, data from a Bonanzle Rails application was being received, but only a small fraction of our overall actions were being reported in the console. After running for two weeks, I think it had registered maybe 100 total action calls (vs. a few thousand action calls that had actually occurred). The action completion times it had registered seemed suspiciously incongruent with the times reported by our ApacheBench testing. Also suspicious was that the Fiveruns console reported us as having a “single Rails app” but not a Mongrel cluster, when in fact our application is a Mongrel cluster. It made me wonder if somehow the data Fiveruns was receiving was from one of our debug machines?

The UI for the statistics that we did accumulate was nice to look at, but difficult to use effectively. All metrics for Fiveruns are reported via Flash time graphs that go back something like 12-24 hours. They looked good, but the problem was that, even after hunting around their UI for 15 minutes, I could not find out how to change the time range to go back more than 24 hours. I also couldn’t figure out how to aggregate statistics. I can’t imagine that there isn’t a way to change the time range and see the big picture of how performance is changing over time, but as Don’t Make Me Think would preach — the responsibility for making users feel smart lies not with the user, but with the designer. I felt dumb trying to learn what I wanted to learn from Fiveruns statistic reporting interface.

Other Concerns

I have no way to verify that this definitely resulted from Fiveruns, but for the weeks we ran the Fiveruns software on Rails 2.0.x (and not before, and not so far after), once or twice a day we would receive nil exceptions from fields in records that were not nil. This meant that sometimes items that had been posted for months would, very, very occasionally, say their price field was nil. Obviously these errors were impossible to track down or definitively attribute to any cause, but the fact that we have not gotten one of them in the two weeks since uninstalling Fiveruns, while we were getting a couple per day with Fiveruns installed, suggests a strong causal probability.

The other concern I had with the software was privacy. Obviously the source code for a Rails project is its crown jewel, but looking through Fiveruns logs (and from the details provided by the Fiveruns team), it seems that Fiveruns could look into or actually send to Fiveruns any or all of our Rails source code. I don’t like to be paranoid, but when we’re talking about the source code for our site, the very basis of our existence, this possibility made me feel uneasy.

Fiveruns Response

After spending more than a day to just get Fiveruns up and running in the first place, I’ll admit that I was less than enthused about the possibility of reporting our problems and working through them with Fiveruns, taking still more of our time. There is hardly a feature on our site that took more than 10 hours for me to implement, and the fact that we dropped that much time just to get Fiveruns installed (poorly), left me reluctant to invest still more time working with them to debug our problems.

After a few emails from their sales coordinator asking me if I was going to sign up or not, I did eventually email them a list of my issues. I suggested that, as much as I want Fiveruns to be the solution, at some point their engineers need to step up their game and deliver a product that “just works” for the very useful purposes it purports to. I sent that mail a few weeks ago, and have not heard back from them since.

Final Analysis

Unlike the BackgroundRb review I’m going to write in the next few days, I have hope for Fiveruns. Their product makes wonderful promises, their support team was extremely friendly, and they have enough millions invested in them that I think they will ultimately be able to sort out their problems. Obviously, my experience with them is the experience of but one person, so it is also possible that we just had the worst batch of luck possible. This possibility seems somewhat far-fetched to me, since we are running a pretty vanilla stack of Apache/Mongrel/Gutsy, but if everyone had as many problems as we’ve had, Fiveruns wouldn’t be in business.

My hope is that someone at Fiveruns will either read this, or hear from users that read this, and will circle up the wagons to capture the unrealized potential with this service. Failing that, we will be looking for alternate services that can provide an easy way for us to monitor the performance of our Rails stack, preferably without them having unfettered access to our entire source tree.

If anybody has any alternate service suggestions, please comment below!

I was able to chat with Blaine at Railsconf for awhile about the status of Starling. Super nice fellow, and very illuminating explanations to my main questions about Starling. Here’s what I learned:

get() removes entries from the queue by design. Thus, Starling isn’t ideally suited for tasks where one would query the task for status. However, Blaine mentioned a clever trick he’s used in a few cases to be able to figure out when a Starling task is done: after adding a Starling task to the queue, the caller can check the queue for a given key that the Starling task will add to the queue when it’s done. For example, say you enqueue a task to grab a user’s contacts from Gmail, and your rake task (or however you’re processing the message) will set a key called “user100_contacts_grabbed”. You can then have your caller periodically look for whether that key is in the queue. Once it is, you know the job is done.

Also “by design” is that queues are not immediately deleted from disk, even after they’ve been removed from the queue via get(). The idea here is that to actually find the key you got and remove it from disk would be expensive. The way Starling is setup, those entries on disk are just logfiles that get concatenated whenever you get or set from a queue. If Starling gets shut down for whatever reason, upon restarting, it will read through the entire log file to determine what keys have yet to be processed, and will start with those keys still in your queue. Pretty clever, and very fast. Also, the log files are automatically trimmed after they reach a certain size, so you don’t have to worry about them growing to infinity (don’t remember what he said the size was, but it seemed fine).

Blaine mentioned that a new release of Starling would probably be out soon, and despite the relative lack of noise about it lately, it will remain an active product. Documentation for it remains an important goal. This is all promising news for a product with such great potential.

What else did he say…that Starling is all about speed, and the average get/set is something like a millisecond or less…that if your task fails and you want to try it again you need to remember to requeue it…I feel like I might have learned some other things as well, I’ll update this post later if/when I remember them. As always, feel free to post any specific questions here.

UPDATE 5/24/08: There was a recent APB security bulletin for all those running Debian-based OSes (including Debian) with OpenSSL 0.9.8c (released 2006) onward. You can read about it here. Long story short: if you’re running a flavor of Debian, you should run “sudo apt-get update” and “sudo apt-get upgrade openssl” before you start these instructions, to ensure that you’re using the patched version of SSL. We now resume our regularly scheduled programming…

If you have a database in one place and some Rails stuff in another place (be it your Rails app, or an asynchronous module that interacts with the DB), and if you’re running on a hosted server (i.e., you can’t just setup a hardware firewall for your entire server network), chances are you have thought or should be thinking about setting up your database to accept SSL connections. This ensures that malicious third parties can’t read the network packets being transmitted between your remote server and your database. Here are some of the sites and notes that I used to get us setup doing this:

For detailed but not too-detailed instructions on generating the SSL keys with MySQL, the MySQL documentation on SSL is great. This documentation describes not only how to generate your SSL keys, but also how to tell your DB who to accept connections from, and whether SSL is required when interacting with those remote IPs.

Important Note 1: to get the shell script on the MySql page working, you’ll need to change the directory in the script so that your openssl.cnf file is at /etc/ssl/openssl.cnf (if you’re running Gutsy).

Important Note 2: The “common name” field in your client and server keys must be different, or your key generation will fail. You’ll know you got it wrong if you run the shell script and it doesn’t ask you if you want to sign the certificate.

A couple other points not specifically called out in the MySQL documentation:

1) to tell MySQL to load with your server certificates, “sudo vi /etc/mysql/my.cnf”. There are a couple lines near the bottom of the file (under the mysqld section) that you can uncomment and change to point at location where your certificates reside.

2) To stop and restart your MySQL server on Ubuntu (necessary so the my.cnf file is reloaded), run “/etc/init.d/mysql stop” or “/etc/init.d/mysql start”. Yes, it won’t work unless you include the path (at least, it wouldn’t for me, and the other Google results I found).

After you think your SSL certificates are legit, you can double check your work by following the instructions here. The link has the commands for verifying that your server and client certificates are setup correctly.

Next, when you have the server certificates setup, you’ve granted access to your remote box (as specified in the MySQL instructions) and you’ve copied the client certificates to your remote box, I recommend testing your mysql connection from the command line on the remote box to verify that everything is kosher. Something like this should do the trick:

mysql -u [username] -p[password] -h [mysql box address] --ssl-capath=[path to client certificates] --ssl-ca=[path to ca-cert]/ca-cert.pem --ssl-cert=[path to client cert]/client-cert.pem --ssl-key=[path to client-key.pem]/client-key.pem

This assumes that you’ve retained the default filenames (ca-cert.pem, client-cert.pem, client-key.pem) mentioned in the MySQL documentation.

If all is well, you should get connected to your MySQL server.

Next up is to setup your database.yml so that it can do this stuff for you. I was somewhat surprised to find that database.yml actually already has (almost completely undocumented) options for SSL security. Do a find all in the Rails source for “:sslkey” and you’ll find all of the options that you can pass to database.yml. Make the database.yml options point to the correct address/filesnames, and you have yourself a more secure connection between remote boxes and your DB.

If you don’t feel like doing the find all, here are the relevant SSL options to put in your database.yml:

sslkey: /path/to/client-key.pem
sslcert: /path/to/client-cert.pem
sslca: /path/to/ca-cert.pem
sslcapath: /path/to/certificates

After spending too many frustrating hours and days fighting BackgrounDRb to connect to multiple servers, I decided to try option #2, the Starling/Workling combo, as of yesterday evening. So far, I’ve been really pleased with most of the setup. I had it working preliminarily in about an hour (compare to about 2 full days for one of our developers to get BackgroundRb setup) locally. However, if there is one thing that’d made Starling difficult so far, though, it’s the lack of useful results I get when I Google anything I can think to Google about Starling documentation. Thus, I am going to try to keep updating this blog as I get Starling tweaked and working, with all of the information I can gather to try to make future Starlingers have a bit easier proposition.

Installation

On your Starling server(s), run “sudo gem install starling” to install it.

On your clients, you talk with Starling via memcached, so you’ll need the memcached client installed. “sudo gem install memcache-client” should do the trick.

Basic Usage Example

I found this morsel in the readme.txt for the Starling gem:

# Start the Starling server as a daemonized process:
# Note by Bill -- I believe that starling will only run as root currently
sudo starling -h 192.168.1.1 -d

# Put messages onto a queue:
require 'memcache'
starling = MemCache.new('192.168.1.1:22122')
starling.set('my_queue', 12345)

# Get messages from the queue:
require 'memcache'
starling = MemCache.new('192.168.1.1:22122')
loop { puts starling.get('my_queue') }

The first line starts a starling that listens on 192.168.1.1, i.e., on your local network. The next lines actually connect to that Starling and test it. They can be run from within irb to verify that your Starling setup is working as expected.

Other options

In general, you can see memcached methods to figure out what Starling is capable of (well documented here). The key difference between Starling and Memcached, from my testing, is that Starling’s get() returns the hash value and deletes it from the cache. Memcached’s get() just returns the value, which is still in the cache until you call delete().

Otherwise, Starling is pretty similar to Memcached. For example (from Starling tests):

starling = MemCache.new('192.168.1.1:22122')
starling.set('test_set_with_expiry', 5 + 2, now)
starling.set('test_set_with_expiry', 5)
sleep(now + 1 - Time.now.to_f)
starling.get('test_set_with_expiry') # returns 5


Like memcached, if you want to get stats on your Starling, such as number of items in the queue, bytes used, starling version, log size, cache hits/misses, total connections, stuff that’s currently cached, and more, you can run stats() on your starling connection object, like so:

starling = MemCache.new('192.168.1.1:22122')
starling.stats # returns hash of statistics on the starling connection


Bugs and Caveats

When you’re setting your Starling up on a remote server, you’ll want to remember to change the address you bind to when you change Starling, e.g., “sudo starling -h my_ip_address -d”. If you’re not root, you’ll probably need the sudo, since Starling binds to network ports.

A couple more apparent bugs I’ve experienced with Starling (running gem version 0.9.3 on Gutsy):

• Even if I specify a log file on the command line, no log file is generated
• Though when you get() a message it is removed from the queue, it still exists in the /var/spool/starling directory, from what I can tell, indefinitely – Blaine explains this in my Starling update

If anyone who has been a longer-time Starling user has seen and gotten past these issues, I’d sure like to hear them.

Update

I was able to talk with Blaine at Railsconf about some of my specific questions about Starling, and about the future of Starling in general. Read about it here.