Author: Bill

We had been racking our brains on this one for a couple weeks. We have monit looking over our Mongrels, which usually keeps everything on the up and up. But every so often, our server would go bananas and the nginx error log would flood with the message:

939#0: accept() failed (24: Too many open files) while accepting new connection

Usually the problem automatically resolved itself, but last night it didn’t. Taking the error at face value, our server guy started looking at the number of open files on our system and the maximum files that could be opened (it’s confusing… “ulimit -a” reports one limit while “cat/proc/sys/fs/file-max” reports another. I think that the former might be for actual file system files opened and the latter might be for file handles (which also includes open IP connections and such)). But even after upping the limit and rebooting repeatedly, the problem persisted.

After server guy (literally) fell asleep on the keyboard around 2 AM, I figured out what had really been happening: any time a new visitor came to our site, we were geocoding their IP with a service that had gone AWOL. About a week earlier I’d noticed a similar slowdown of about 1-2 seconds with actions that created sessions, but I assumed it was the session creation itself that was causing the slowdown, when in fact it was the geocoding that happened alongside the session creation that was responsible for the lag.

Long story short, when nginx gives this error, what it really seems to mean is that it is holding too many open connections, and usually that is happening because you are using round robin dispatching (bad, I know, but we have our reasons) and one or more of the Mongrels is stuck and forcing the Mongrel queue to skyrocket.

The other lesson here is an obvious one that I’ve read many times before but have been slow to actually act on: making remote API calls without timeouts is asking for trouble. Here is a fine article if you’re interested in solving that problem in your own site before it is your ruin.

Update: An answer from the heavens descends! On the same day of my post, no less. Gotta love an active framework.

Like most of the Rails community, I’ve been swept up in the fascination lately over easier deployment via Passenger. After getting 2.2 deployed, I had hoped that Bonanzle could try to make the leap to Passenger and become yet another success story of a multi million monthly page viewed site proving that the hype behind Passenger is legit.

But then I remembered our Mongrel handlers, specifically, that said handlers are currently satisfying about 50% (or more) of the requests for our entire site (mostly through chat and instant message reads + posts). As such, upgrading to Phusion puts me ill at ease, because I would think that if our Mongrel handlers became standard Rails actions, we would 1) require twice as many Rails instances to run our site, possibly requiring more memory with Phusion instead of less and 2) incur a much greater overhead per request, when Rails runs its routing and other pricey (and in this case, unnecessary) default behaviors on our requests.

Normally I might pose this question to a forum, but given the esotericness of the problem, I suspect that such a question would assuredly be greeted with crickets. If anyone else has actually tried this, or seen any stories comparing the two, I’d love to hear them and post them to this blog to make it easier for others to find.

If all else fails, I plan to eventually setup a test case to actually benchmark and get some numbers, but that would figure to be at leeast a 1-2 day ordeal by the time sufficient reliable data could be gathered.

A seeming sensible choice was made in 2.2 to change the log format to show times in milliseconds instead of seconds. I can understand why a Ruby programmer would find that decimal-less output “beautiful,” and I certainly wasn’t objecting.

That is, until I used my new favorite tool, pl_analyze, to check my log times, and remembered that pl_analyze was written in the early oughts, when only seconds would do. Once the times change to milliseconds, the formatting of pl_analyze becomes fairly unparsable.

Faced with this challenge, I had to pick between the better solution (patch pl_analyze) or the faster solution (patch Rails log). Guess which choice the one-full-time-programmer-company took?

I’ve attached ugly Rails patch if anyone else wants to save a couple minutes figuring this out themselves. It makes the Rails 2.2 logging format compatible with pl_analyze again.

benchmarking.rb

Last updated June 8th 2009: Bloggity version 1.0 has landed! This blog has been completely revised for the occasion.

With a spec (and funding) courtesy of Goldstar, I finally made time to incorporate many of the features I’d been hoping to add for the last year. Here is the current featureset:

• Rails 2.3 compatible (also should be compatible with Rails 2.0-2.2, if you use the Engines plugin and copy the routes into your routes file)
• Drop-in installation, including rake tasks to automate creating the necessary database tables, and moving the necessary assets into your existing project.
• WYSIWYG editor for writing blogs
• Support for AJAX uploading of images and/or assets of any kind with your blog post
• URLs formatted using blog titles for good SEO (e.g., http://mysite.com/blog/founder/welcome_to_my_blog)
• Create and maintain one or more blogs in a Rails app (e.g., a main blog, CEO blog, and user blogs)
• User commenting, built in support for gravatars
• Separate, configurable persmissions for writing, commenting on, and moderating comments for blogs
• RSS 2.0 feeds (with built in customizability for Feedburner)
• Ability to use custom stylesheets on a per-blog basis (i.e., the Founder’s blog can use different styling than the user blogs)
• Ability to tag blog posts, and filter by tag
• Ability to categorize blog posts
• Blogs can be saved in draft mode before publishing
• Model & controller test suite
• Docs and an active git project!

Requirements, Dependencies

Bloggity 1.0 has been tested with Rails 2.3. In Rails 2.3, it shouldn’t need any plugins besides will_paginate. It should also work in Rails 2.0-2.2, but you’ll need to copy the routes into your routes file, since routes were extracted from plugins in 2.3. Other monkeying around might also be needed, if you have issues installing on pre-2.3 Rails, please drop a line in the comments.

FCKEditor is used for the WYSIWYG editor. This is included with the plugin, and shouldn’t affect your existing Rails install (plugin puts its assets in a separate directory). For the image uploading, jQuery is used (since it works via AJAX). This is also included with the plugin, and doesn’t need to be used elsewhere on your site.

Installation Instructions

1. Run “script/plugin install git@github.com:wbharding/bloggity.git” to grab Bloggity and put it in vendor/plugins/bloggity
2. Run “rake bloggity:bootstrap_db” from your application’s base directory
3. Run “rake bloggity:bootstrap_bloggity_assets” to copy the images and stylesheets used with bloggity into your apps /public/stylesheets/bloggity and /public/images/bloggity directories.
4. Run “rake bloggity:bootstrap_third_party_assets” to copy FCKEditor and jQuery into your “/public/javascripts/bloggity” directory. This is not required, but the WYSIWYG editor and asset uploading won’t work without it.
5. Take a gander at bloggity/lib/bloggity/user_init.rb. This shows the methods that bloggity will use to interface with your user model (for blog permissions and such). Implement these in a sensible way in your user model. Don’t forget to copy the blog associations into your user. (If you prefer, you could also copy the UserInit into a lib directory of yours, fill in the methods, and include it in your User model directly)
6. There are two methods in bloggity/lib/bloggity/bloggity_application.rb: current_user and login_required. Implement these in your application_controller.rb if you haven’t already.

And you’re done!

In the Wild

An earlier version of Bloggity can be seen in the wild at the Bonanzle blog. Bloggity 1.0, with its support for tagging, categorization, and gravatars, actually exceeds the features you’ll see at Bonanzle.

Screenshots

Here are a couple shots of Bloggity in action with its default styles.

Looking at the main blog.

Writing a blog.

Uploading assets for a blog.

Miscellaneous

I haven’t built in support yet for upgrading from pre-1.0 versions of Bloggity, because, well, just about everything changed with the release of 1.0, not the least of which is the names for several models. However, when Bonanzle moves from Bloggity 0.5 to 1.0, I’ll post how we went about it here.

Please add your bug reports to the comments section and I’ll commit changes as you find them.

We regularly copy images around on our site — for cropping, for duplicating items, and for many other purposes. For the last six months, we’ve been using the code that I found on Ruby forum. It has been passable, but as I’ve been stamping out the bottlenecks in our performance, I’ve found that image duplication has been one of our slower movers. Copying a single image with its two thumbnails was taking about 5-10 seconds per image. Given that all it should be doing, functionally, is making a file copy and a new database entry, this didn’t seem right. I did some research into it today and figured out why.

The reason is that the code on Ruby forum still relies on the main image creating its thumbnails from scratch. That is, the main loop of thumbnail creation in their method isn’t actually saving the thumbnails. The thumbnails get saved on the c.save line, via the standard attachment_fu after_save callback.

I just finished an updated monkey patch that I’m sharing here that should let you copy images without the costly thumbnail re-processing. You can grab attachment_fu mixin.rb or copy the ugly WordPress version below.


module Technoweenie # :nodoc:
module AttachmentFu # :nodoc:
module InstanceMethods
attr_writer :skip_thumbnail_processing

# Added by WBH from http://groups.google.com/group/rubyonrails-talk/browse_thread/thread/e55260596398bdb6/4f75166df026672b?lnk=gst&q=attachment_fu+copy&rnum=3#4f75166df026672b
# This is intended to let us make copies of attachment_fu objects
# Note: This makes copies of the main image AND each of its prescribed thumbnails


def create_clone
c = self.clone

self.thumbnails.each { |t|
n = t.clone
img = t.create_temp_file
n.temp_path = img #img.path -- Commented so that img wo'nt get garbage collected before c is saved, see the thread, above.
n.save_to_storage
c.thumbnails<<n
}

img = self.create_temp_file

# We'll skip processing (resizing, etc.) the thumbnails, unless the thumbnails array was empty. If the array is empty,
# it's possible that the thumbnails for the main image might have been deleted or otherwise messed up, so we want to regenerate
# them from the main image
c.skip_thumbnail_processing = !self.thumbnails.empty?
c.temp_path = img #img.path
c.save_to_storage

c.save
return c
end

protected

# Cleans up after processing. Thumbnails are created, the attachment is stored to the backend, and the temp_paths are cleared.
def after_process_attachment
if @saved_attachment
if respond_to?(:process_attachment_with_processing) && thumbnailable? && !@skip_thumbnail_processing && !attachment_options[:thumbnails].blank? && parent_id.nil?
temp_file = temp_path || create_temp_file
attachment_options[:thumbnails].each { |suffix, size| create_or_update_thumbnail(temp_file, suffix, *size) }
end
save_to_storage
@temp_paths.clear
@saved_attachment = nil
callback :after_attachment_saved
end
end

Include it in your environment.rb and you should be golden.

Notice that it is nearly identical to the Ruby forum code, except that it adds a new member variable to the object being copied so it will just save, not re-process, the thumbnails.

I’ve tested it on my dev machine for a few hours this afternoon and it all seems working, I’ll post back later if I encounter any problems with it. You can do the ame.

Continuing my series of reviews of the plugins and products that have made Bonanzle great, today I’ll talk about Thinking Sphinx: How we’ve used it, what it’s done, and why it’s a dandy.

What It Is Bro, What It Is

What it is is a full text search Rails plugin that uses the Sphinx search engine to allow you to search big tables for data that would take a long-assed time (and a lot of custom application code) to find if you used MySql full text searching.

What Are Your Other Options

In the space of legitimate Rails full text plugins, the commonly mentioned choices are Sphinx (via Thinking Sphinx or Ultra Sphinx), Xapian (via acts_as_xapian), solr (via acts_as_solr) and (shudder) Ferret (via acts_as_ferret).

Jim Mulholland does a great job of covering the various choices at a glance, so if you’d like a good overview, starts with his blog about the choices.

To his commentary, I would add that Solr looks complicated to get running, appears to have been abandoned by its creator, and hasn’t been updated in quite awhile. It should also be mentioned that if you were to choose solr, every time you wished to talk about it online you’d have the burdensome task of backspacing the “a” out of the name that your fingers were intent on typing in.

Xapian seems alright, but the documentation on it seemed lacking and not a little arcane. Despite Jim’s post on how to use it, it seemed like the Xapian Rails community was pretty sparse. My impression was that if it didn’t “just work,” it would be I alone who would have to figure out why. Also, from what I can tell in Jim’s post, it sounds like one has to stop Xapian from serving search requests to run the index? Update: the FUD patrol informs me that you can concurrently index and serve, oh what joy!

Ferret sucks. We tried it in our early days. It caused mysterious indexing exceptions left and right, whenever we changed our models or migrated. The day we expunged it from our system was the day I started programming our site and stopped worrying about what broke Ferret today.

Ultra Sphinx looks OK, but as you can read here, it’s ease of use leaves something to be desired compared to the star of our blog post, who has now entered the building. Ladies and gentlemen, may I present to you, hailing from Australia and weighing in at many thousand lines of code:

Thinking Sphinx!

There’s a lot to like about Thinking Sphinx: it has easy to read docs with examples, it has an extremely active Google Group behind it, and it supports useful features like location-based searches and delta indexing (e.g., search updates in real time).

But if there is one reason that I would recommend Thinking Sphinx above your other choices, it’s that you probably don’t care a hell of a lot about full text searching. Because I didn’t. I care about writing my website. This is where Thinking Sphinx really shines. With the tutorials and Railscasts that exist for Thinking Sphinx, you can write an index for your model and actually be serving results from Thinking Sphinx within a couple hours time. That doesn’t mean its an oversimplified app though. Its feature list is long (most of the features we don’t yet use), but smarts defaults are assumed, and its super easy to get rolling with a basic setup, allowing you to hone the parameters of the search as your situation dictates.

Also extremely important in choosing a full text search system is reliability. Programming a full text engine (and its interface into your application) is rocket science, as far as I’m concerned. I don’t want to spend my time interpreting esoteric error messages from my full text search engine. It must work. Consistently. All the time. Without me knowing anything about it. Thinking Sphinx has done just that for us. In more than a month since we started using it, it’s been a solid, reliable champ.

A final, if somewhat lesser, consideration in my recommendation of TS is who you’ll be dealing with if something goes wrong. Being open source, my usual expectation is that if me and Google can’t solve a particular problem, it will be a long wait for a response from a random, ever-so-slightly-more-experienced-than-me user of the system in question who will hopefully, eventually answer my question in a forum. Thinking Sphinxes creator Pat Allen blows away this expectation by tirelessly answering almost all questions on Thinking Sphinx in its Google Group. From what I can tell, he does this practically every night. This is a man possessed. I don’t claim to know or understand what’s in the punch he’s drinking (probably not beginner’s enthusiasm, since TS has been around for quite some time now), but whatever’s driving him, I would recommend you take advantage of his expertise soon — before he becomes jaded and sour like the rest of us.

What About the Performance and Results?

Performance: great. Our usual TS query is returned in a fraction of a second from a table of more than 200,000 rows indexed on numerous attributes. Indexing the table currently takes about 1-2 minutes and doesn’t lock the database. Nevertheless, we recently moved our indexing to a remote server, since it did bog down the system somewhat to have it constantly running. I plan to describe in the next couple days how we got the remote indexing working, but suffice to say, it wasn’t very hard (especially with Pat’s guidance on the Google Group).

Results: fine. I don’t know what the pertinent metrics are here, but you can use weighting for your results and search on any number of criteria. Our users are happy enough with the search results they’re getting with TS out of the box, and when we do go to get more customized with our search weighting, I have little doubt that TS will be up to the task, and it’ll probably be easy to setup.

Final Analysis

If you want to do full text searching on a Rails model, do yourself a favor and join the bandwagon enjoying Thinking Sphinx. It’s one of the best written and supported plugin/systems I’ve stumbled across so far in the creation of Bonanzle.

I’m Bill Harding, and I approved of this message.