Starling Update from the Horses’ Mouth

I was able to chat with Blaine at Railsconf for awhile about the status of Starling. Super nice fellow, and very illuminating explanations to my main questions about Starling. Here’s what I learned:

get() removes entries from the queue by design. Thus, Starling isn’t ideally suited for tasks where one would query the task for status. However, Blaine mentioned a clever trick he’s used in a few cases to be able to figure out when a Starling task is done: after adding a Starling task to the queue, the caller can check the queue for a given key that the Starling task will add to the queue when it’s done. For example, say you enqueue a task to grab a user’s contacts from Gmail, and your rake task (or however you’re processing the message) will set a key called “user100_contacts_grabbed”. You can then have your caller periodically look for whether that key is in the queue. Once it is, you know the job is done.

Also “by design” is that queues are not immediately deleted from disk, even after they’ve been removed from the queue via get(). The idea here is that to actually find the key you got and remove it from disk would be expensive. The way Starling is setup, those entries on disk are just logfiles that get concatenated whenever you get or set from a queue. If Starling gets shut down for whatever reason, upon restarting, it will read through the entire log file to determine what keys have yet to be processed, and will start with those keys still in your queue. Pretty clever, and very fast. Also, the log files are automatically trimmed after they reach a certain size, so you don’t have to worry about them growing to infinity (don’t remember what he said the size was, but it seemed fine).

Blaine mentioned that a new release of Starling would probably be out soon, and despite the relative lack of noise about it lately, it will remain an active product. Documentation for it remains an important goal. This is all promising news for a product with such great potential.

What else did he say…that Starling is all about speed, and the average get/set is something like a millisecond or less…that if your task fails and you want to try it again you need to remember to requeue it…I feel like I might have learned some other things as well, I’ll update this post later if/when I remember them. As always, feel free to post any specific questions here.