Best of Rails GUI, Performance, and other Utilities

I’m all about putting order to “best of” and “worst of” lists, so why not give some brief props to the tools, plugins, and utilities that make life on Rails a wonderous thing to behold?

5. Phusion Passenger. OK, this would probably be first on the list, but it’s already been around so long that I think it’s officially time to start taking it for granted. But before we completely take it for granted, would anyone care to take a moment to remember what life was like in a world of round-robin balanced Mongrels web servers? You wouldn’t? Yah, me neither. But no matter how I try, I cannot expunge memories of repeatedly waking up to the site alarm at 7am to discover somebody had jammed up all the Mongrels with their stinking store update and now I’ve got to figure out some way to get them to stop.

4. jQuery / jRails. This probably deserves to score higher, as the difference between jQuery and Prototype is comparable to the difference between Rails and PHP. But since it’s not really Rails-specific, I’m going to slot it at four and give major props to John Resig for being such an attentive and meticulous creator. Without jQuery and jQuery UI, the entire web would be at least 1-2 years behind where it is in terms of interactivity, and I don’t think that’s hyperbole. (It even takes the non-stop frown off my face when I’m writing Javascript. With jQuery, it’s merely an intermittent frown mixed with ambivalence!)

3. Sphinx / Thinking Sphinx. There’s a reason that, within about six months time, Thinking Sphinx usurped the crown of “most used full text search” utility from “UltraSphinx.” And the reason is that it takes something (full text search) that is extremely complicated, and it makes it stupidly easy. And not just easy, but extremely flexible. Bonanzle has bent Sphinx (0.9.8) into doing acrobatics that I would have never guessed would be possible, like updating it in nearly-real time as users log in and log out. Not to mention the fact it can search full text data from 4 million records in scant milliseconds.

Sphinx itself is a great tool, too, though if I were going to be greedy I would wish that 0.9.9 didn’t reduce performance over 0.9.8 by around 50% in our testing, and I would wish that it got updated more often than once or twice per year. But in the absence of credible competition, it’s a great search solution, and rock solidly stable.

2. New Relic. OK, I’ll admit that I’ve had my ups and downs with New Relic, and with the amount of time I’ve spent complaining to their team about the UI changes from v1 to v2, they probably have no idea that it still ranks second in my list, ahead of Sphinx, as most terrific Rails tools. But it does, because, like all the members of this list, 1) the “next best” choice is so far back that it might as well not exist (parsing logs with pl-analyze? Crude and barely useful. Scout? Nice creator, but the product is still a tot. Fiveruns? Oh wait, Fiveruns doesn’t exist anymore. Thank goodness) and 2) it is perhaps the most essential tool for running a production-quality Rails site. Every time I visit an ASP site and get the infinite spinner of doom when I submit a form, I think to myself, “they must not know that every time I submit a form it takes 60 seconds to return. That would suck.” On a daily basis, I probably only use 10% of the functionality in New Relic, but without that 10%, the time I’d spend tracking logs and calculating metrics would make my life unfathomably less fun.

1. Rubymine. The team that created this product is insane. Every time that I hit CTRL-O and I type in “or/n” and it pops up all files in the “offers_resolution” folder starting with the letter “n,” I know they are insane, because having that much attention to productivity is beyond what sane developers do. Again, for context’s sake, one has to consider the “next best” choice, which, for Linux (or Windows) is arguably a plain text editor (unless you don’t mind waiting for Eclipse to load and crash a few times per day). But, instead of programming like cavemen, we have a tool that provides killer function/file lookup; impeccable code highlighting and error detection (I had missed that, working in a non-compiled language); a working visual debugger; and, oh yeah, a better Git GUI than any of five standalone tools that were built specifically to be Git GUIs.

Perhaps as importantly as what Rubymine does is what it doesn’t do. It barely ever slows down, it doesn’t make me manage/update my project (automatically detecting new files and automatically adding new files to Git when I create them from within Rubymine), and it handles tabs/spaces like a nimble sorcerer (something that proved to be exceedingly rare in my quest for a usable IDE).

Like New Relic, I probably end up using only a fraction of the features it has available, but I simply can’t think of anything short of writing my code for me that Rubymine could do that it doesn’t already handle like a champ. Two thumbs up with a mutated third thumb up if I had one.

Conclusion

Yes, it is a list of apples and oranges, but the commonality is that all five-ish of the lists members stand apart from the “second best” solution in their domain by a factor of more than 2x. All of them make me feel powerful when I use them. And all of them, except arguably New Relic, are free or bargain priced. Hooray for life after Microsoft. Oh how doomed are we all.

Rails Performance Analysis and Monitoring? Free and Easy with pl_analyze

Since writing about our lackluster experience with Fiveruns awhile back, I had been keeping my eyes open for a good way to get reliable, comprehensive, and easy-to-interpret performance metrics for Bonanzle. Prior to our transition away from shared hosting in November, I would browse through our production logs and regularly find actions taking 10 seconds or more, but I had no way to get a good overall sense of what actions were taking the longest, how long those actions were taking, and how it was changing from day to day.

Now we have a solution, and it’s a dandy.

Many have heard of pl_analyze, but I think that few may realize how truly easy it is to get up and running on any Rails 2.X application (we’re now on 2.2). To start, Geoff has written a strikingly excellent blog about how to change your log format to be compatible with pl_analyze by adding a single file to your lib directory and changing a couple lines in environment.rb.

After you follow his instructions, here are a couple more key points that I found relevant to polish the rough edges:

* By default, when we were running 2.0 (don’t know if it’s the case with 2.2, since we were pre-patched when we upgraded), logging of DB times was disabled. This monkey patch fixes that.

* By default, when we moved to 2.2, the log format changed and broke pl_analyze. This fixes that.

* After enabling the Hodel logger as described by Geoff, our production log became extremely verbose. One of the helpful commenters on Geoff’s blog pointed out the solution to this: add the line “config.logger.level = Logger::INFO” to your production.rb

After that, you’re good to go. Took me less than five minutes to get it set up, after which I was able to get priceless info like the following:

Request Times Summary: Count Avg Std Dev Min Max
ALL REQUESTS: 1576 0.242 0.229 0.000 2.487

ItemsController#show: 685 0.328 0.141 0.008 0.751
ChatsController#service_user_list: 286 0.022 0.037 0.005 0.301
HomeController#index: 158 0.256 0.308 0.000 1.461

Well, doesn’t look so pretty on my crap blog (shouldn’t I have upgraded this by now?), but it’ll look like a champ in your terminal window.

After getting our basic setup established a couple weeks ago, I started experimenting today with archiving our daily performance info into the database so I can track how action times are changing from day to day. Geoff provides a great start for this as well, with his pl_analyze Rails plugin, which you can install by running

./script/plugin install http://topfunky.net/svn/plugins/mint

After you’ve installed the plugin, run

rake mint:create_analyze_table

to create your table, and then

rake mint:analyze PRODUCTION_LOG=/path/to/production.log

to actually extract and store the day’s data. Be sure to read my comment in Geoff’s blog (it’s about the 30th comment) for a slight modification you may need to make to the analyze task if you get an error that says something to the effect that “0 parameteres were expected, got 1” or somesuch when you run mint:analyze.

Also note that this will store everything in your production log, so if you haven’t already setup log rotating on your production log, you’ll want to do that (it can be as simple as this).

After you’re archiving performance metrics to your database, you are in a performance analysis wonderland — it’s just a matter of applying creativity to data. For starters, I plan to create reports that list the top 10 daily time users (= count of action * average time per action), and the top 10 biggest daily deltas in time taken for various actions. I plan to use the latter to help triangulate where code changes affect performance. It would be trivial to create a model that actually allowed daily notes to be added to a report, so if the items#show action starts running 20% slower one day, I could demarcate the code changes that were made on the day that happened (though our changelist makes such a demarcation unnecessary).

I’m not sure what other tools are being used by the community to keep an eye on performance, but with the quickness and reliability (= stable results reported for weeks) of getting a pl_analyze solution running, it gets high recommends from ’round these parts. I can’t imagine returning to a day where we didn’t know if our average request time was 300 ms, 3000 ms, or 30000 ms (yes, we had some very dark days before our server move).

Change ACL of Amazon S3 Files in Recursive Batch

We’re in the process of moving our images to be served off of S3, and wanted to share a quick recommend I came across this evening when trying to change our presently-private S3 image files to be public. The answer is Bucket Explorer. All things being equal, you certainly won’t mistake it for a high budget piece of UI mastery, but it is surprisingly capable of doing many things that have been troublesome for me with the Firefox S3 plugin (which is a major pain to even get working with Firefox 2 (which is a major pain to upgrade to Firefox 3… I upgraded for a month, spent 5 hours trying to figure out why some pages seemed to randomly freeze indefinitely before giving up and downgrading (my best guess was it seemed to be Flash-related))), or the AWS-S3 gem, or the other free S3 browsing web service I found somewhere or another.

In addition to providing a capable, FTP-like interface for one’s S3 buckets, it can also get stats on directories, do the aforementioned recursive batch permission setting, delete buckets (S3 gem won’t let me, even with the :force => true option), a bunch of other features, and probably most importantly (for me) — it works! Tra-lee!

It’s $50 to buy, but once it finishes changing batch permissions for about 20,000 of our images files (as its currently in the process of) I would seriously consider paying it. For the time being, I’m on a fully functional 30 day trial.

Compare Rails IDEs for Windows

Jordan said it, and I couldn’t agree more: one of the real weaknesses of Rails development in Windows is the lack of a killer IDE. I never realized how good I had it when I was developing C++ with Visual Studio and Workspace Whiz, but after a few months of developing in Rails, where I’ve gone through about an IDE per month, I have sussed out my baseline requirements for a usable IDE. Most of these things are not what I’d consider “fancy” features, but if they aren’t in my IDE I get very annoyed very quickly:

  1. IDE keeps a project workspace that can be updated to include new files with somewhere between “little” and “no” effort
  2. When I press “CTRL-F” on a word, that word has automatically populated the find dialog box
  3. There exists the means to define a key combination for invoking a “find in files” dialog box,
  4. And that find in files dialog can search for regular expressions
  5. I can open any file in the project in a couple keystrokes without using my mouse
  6. Syntax highlighting exists for HTML, CSS, JS and Ruby
  7. The IDE does not fight with me about tabbing
  8. The IDE doesn’t crash
  9. The IDE doesn’t pause and stop me from what I’m doing. Ever.
  10. Intellisense (Not essential, but highly desirable)
  11. Built-in debugging (Not essential, but highly desirable)

I guess they aren’t trivial requirements, but it doesn’t seem to me like I’m asking for a ton here.

So far, I have over the last six months spent some time using (in chronological order): EasyEclipse, Ultraedit, PSPad, NetBeans, and Ruby in Steel.

I don’t have the time to do a wordy comparison of the bunch of them at present, but the following spreadsheet details my experience with them in regard to my 11 essential (and almost-essential) attributes:

As you can see, some IDEs got evaluated more than others, because some of them got on my nerves almost instantly (UltraEdit, Netbeans), and some stayed in good favor for long enough to evaluate them thoroughly (PSPad, Ruby in Steel). My biggest beef with Ruby in Steel had been that I could not figure out a way to get a file opened with a single keystroke. But after exhaustive Googling, I learned today about the magic of CTRL-ALT-A, which brings up a command window where I can type “open [filename]” to immediately open any file in the project. Eureka! At $50, Ruby in Steel is currently my decisive winner in the contest for least-annoying Rails IDE.

— UPDATE —

After some cajoling, I persuaded Josh Jensen, creator of Workspace Whiz, to create a version of his world-class Visual Studio plugin that is compatible with Ruby in Steel. What does it mean? It means that you can get instantaneous one-key lookup of any file or tag (=method or class name in Javascript or Rails) in your entire project. That is, you can hit CTRL-D, type “message” and, as you are typing in “message”, Workspace Whiz will be interactively telling you every Rails or Javascript function/method in your entire project that begin with the word “message”, similar to Google Desktop searching, if any of you use that. Or you can just press CTRL-space on any method and have Workspace Whiz jump to the method (again, Javascript or Rails). CTRL-left arrow will then return you to where you started.

Pretty damn cool stuff. Curious if RadRails has anything like it?

The only question is when Josh will release this version to the public. If you post a comment that you want it below, I can find a way to make it happen.

Assorted Plugs

Today begins week two of my “un-“, er, “self-“, employment. Week #1 was spent in large part taking another pass at my processes to see what further enhancements I could make to the way I get stuff done. Here’s the cheat sheet of my favorites.

TODO List: Who knows how many different incarnations I’ve had of TODO lists? The one I’ve kept going back to is just a running document that sits in my Gmail “Drafts” folder. But I think I’ve finally found a piece of software that can beat the elegance of nothing: rememberthemilk.com. I found the site after Googling “best todo software” and finding a web poll where people rated about 10 different pieces of online TODO software. RTM beat the others by a factor of many. After using it, I can see why. It’s interface is dirt-simple, but it does everything I hoped it would, without bells and whistles getting in the way. Which is not to say it doesn’t also have bells and whistles: it took about 2 minutes to set up my Google Calendar to show which TODO items I have scheduled on which day. I consider RTM a must-have for any person who makes their own schedule.

Park on a Downhill Slope: The idea, which I read in Life Hacker (Confession in order: I got almost all these ideas from Life Hacker), says that the last thing you do before you leave should be to leave a folder on your desktop that describes in specifics what your first task will be for the next day. Then, when you get in, you’ll get straight to work and set up your momentum for the rest of the day. This goes hand-in-hand with a similar idea: don’t check your email for the first hour of the day (until you finish your first task). It sets the stage for distraction and not getting tasks done.

Close Thine Email: I’ve heard this recommendation countless times, but have finally started taking it. What I’ve noticed is that even though having Gmail open doesn’t seem like a big distraction (it doesn’t pop up or make noises when I receive an email, like some of my friends’ Outlook configurations do), it serves as an escape valve that my brain uses whenever hard problems arise. My tendency had been to double-check whether there might be any email to read instead of solving said problem. Bad, brain, bad!

Ban You from Your Most Visited Sites: Another Life Hacker special — here’s a script that you can use along with GreaseMonkey to keep yourself from visiting certain, customizable, web sites at customizable times of day. I scoffed at this one when I first saw it (isn’t that what self-discipline is for?), but it is handy to have the computer be strong for me when I am weak.

Time > Money: I came up with this one myself — when one is a programmer, and one has a lot to get done in not a lot of time, many hardware expenses, most software expenses, and pretty much all book expenses are worth the cost if they can make you work faster. When you figure that the average Rails contractor makes $75-$150 an hour, if there is a book that could cumulatively save you one hour, it is worth whatever it costs. It was this same stream of logic that emboldened me to get my fancy new Quad Core machine last week, which is not only going to minimize my idle time, but improve morale when I get more done with the same amount of brain power.

Close Call

Holy smokes, folks! I avoided catastrpohe by the skin of my teeth this evening. I feel like I should learn something from it, or at least help somebody else learn something from it. Geek advisory forthcoming: this catastrophe was computer-centric and I’m about to get a little (or a lot) geeky in explaniing it.

explosion-22.JPGThe trouble started when I got home from work this evening and rebooted my computer per Windows XP’ insistence. Upon rebooting, I was dismayed to find my computer booting to its old partition (the half-installed Vista partition that came with this laptop before I installed XP). I had tried to remove this partition previously, but it’s been hanging around like mustachioed hotshots at the roller rink, reliably causing a BSOD whenever the laptop chooses to boot that partition. Unfortunately, as of this evening, the still-unidentified laptop demons decided that this evil partition would become the only bootable partition available on my disk.

I started the built-in Windows XP Recovery Tool (which I highly recommend…if you like fake DOS clones that serve no purpose), and it indicated that I had a block of unpartitioned space about the size of what my hard drive had been. Not good. I tried making it into a partition to see if I could boot from it, but no dice: the partition that the Recovery Tool created was an extended partition that couldn’t be booted from.

Long story short, I proceeded through the Dell MediaDirect Repair disk, Cute Partition Manager, and Partition-Recover, before finally finding the freeware app Test Disk. Now, I should be clear in my Test Disk review that it is basically a text-only app that is sparse on instruction and generally pretty black-boxy, as opposed to Partition-Recover, which has a reasonably-well laid out interface (for a DOS app) and a clear path to follow. But Partition Recover costs greater than $0, whereas Test Disk costs exactly $0. So Test Disk won. And finally, after about four hours of trial and error, I have fully recovered my deleted partition after about five button presses with Test Disk.
tortoise.png
This was an experience that made me rethink what I would lose if my primary computer got wiped with no prior notice. Of the many potential backup options to reduce the risk of this happening, my favorite one so far is Tortoise Subversion. Subversion is a free CVS-like source control application. Tortoise offers a Windows Explorer skin to the source control. As you can hopefully, sort of see in the picture at right, Tortoise provides a graphical indication on each indexed file showing whether it is up to date or not.

But here’s the most awesome part of all: you can create your own source control repository from scratch in less than five minutes. Right click on a folder, there is literally a “Create Repository” option within the Tortoise choices, fill out a couple fields, and you have both a ready-made backup system and source control. I’ve been using it to keep my three home computers in sync, and to get revision history for my documents that I might want to look back at later on. I can’t imagine a much easier solution. It’s only real drawback as a backup solution is that you have to transfer files over your home network, which probably won’t have the throughput of a USB 2.0 flash drive. But it can always run in the background. I’m doing my backup as I write this.

Troublemaker

For better or worse, I think I am probably perceived by some at my day job to be a “troublemaker.” I like to consider myself the good kind of troublemaker who makes trouble that leads to meaningful improvements. Because I do bring about a number of those. But trouble is trouble. Given that causing positive trouble has probably cost me far more in effort and energy than it has netted me in recognition and praise, I think one could reasonably ask why I persist in doing it.

I remembered the answer today, when I stayed a couple hours late to write a Perl script that would track the cumulative time that myself and our team of 15 people spend building our project every day. I’ve been wired to derive a feeling of exhilaration from finding solvable problems and figuring out the steps to get those problems solved. So when the day comes in a couple weeks that I can assemble 15 people worth of build data and present my interpretation of what the cost of those times is, it won’t matter to me what kind of trouble I’m causing, because ultimately, a problem will be one big step closer to being solved.

The shareable observation to make here is that, though the pleasure I took from writing this script was tremendous, there were still a few days that I prioritized it lower than my everyday tasks, because technically, it was optional. But once I finished it this evening, I realized that this is what makes me really happy. If you’ve got something that sounds like a good idea to you but is going to take a couple hours to get done, listen to how you feel once you finish that thing. If it is “good” or “great”, hopefully you will join me and be smarter about what gets precedence next time you’re prioritizing standard responsibilities, distractions, and whatever makes you feel like a champ.

The other point is that every company needs some quality troublemakers.

Hours + More Hours = ?

productivityaccount.pngIt’s baffling that Mr. Productivity Crusader myself didn’t figure this one out earlier: it pays to organize and catalogue where my Bonanzle hours are being spent. I’ve done it at my day job for the last six months or so. I keep an open Excel spreadsheet that provides me a minute-by-minute breakdown of where my time is being spent during a given day, week, or milestone. Using Excel’s Pivotcharts, I can then aggregate results from my spreadsheet to reveal meta-patterns such as “Mondays are usually administrative-heavy” and “Milestones where I have the time to program go more smoothly.”

The other simple-stupid and supremely insightful fact from these spreadsheets? How much time I really spend each week working. Since I’ve held myself strictly accountable for my minutes, I’ve reduced my garbage Internet time from about a half hour a day to basically none.

But I hadn’t thought to apply any of these techniques to Bonanzle. Partly because there is some energy-overhead in being that responsible. Partly because I have had too many things to do to be able to precisely define what I ought to be doing and what I am doing. But that changes now. Partner or none, there are enough different areas that require my attention at this point that I’m going to start maintaining a simple list of what I plan to do for the week, and how much time I’d like to allocate to it. And what do ya know? Blogging got two hours this week!

I’m looking forward to determining just how much time I am dedicating to this site on a weekly basis. From my rough (and somewhat conservative) projections for this week, it looks like I’ll be spending about 30 hours on it. Add that to my 40 hour work week, and that goes a long way toward explaining some of those rough mornings. It also goes a long way toward explaining how I know that this site will work out to the extent I can be disciplined to spend my hours smartly; particularly since my hours are matched by those invested by the growing Bonanzle cadre.

Best of: Productivity Tools

Hi my name is Bill and my passion is productivity.

These are three tools I’ve found myself repeatedly using lately to stay maximally productive:

1. Google Desktop. God help you if you are still searching for files on your computer using Windows XP search. I can distinctly recall that when running DOS 3.1, about 15 years ago, I could recursively search for a file in about half the time XP search takes (and don’t even get me started on the 30 seconds XP takes to delete a file (must…avoid…turning blog entry into tirade… deep breath and)). Google Desktop indexes every email, document, and file on your computer to provide search results across your entire computer as you type words into it. Being as that you are currently visiting a high tech “blog” I assume you are probably tech-savvy enough to already have Google Desktop installed, but if not, godspeed to Google.

2. Firefox Bookmarks Synchronizer. I first saw this on Craig Babcock’s blog, and have since seen it many other places, which is no surprise, because it’s that good. In literally less than one minute, you can download it, install it, and setup an account that will henceforth keep the Firefox bookmarks for all of your computers in sync. I have about four computers I use regularly, and I am certain this tool is a Very Good Thing for people like me.

OneNote
3. Office OneNote. No download link because you can’t download this because it ain’t free. But it is handy. It’ll let you do productive stuff like hold “Windows Key-S” and drag a box around any image on your page. This cropped image will immediately be pasted into a OneNote document (even if OneNote wasn’t open), where it can be labeled and organized into a page of your choosing (see example of the screenshot+note in image at right). It also has a lot of tools that make creating tables easy, as well as generally promoting order in an habit (note taking) prone to chaos. If it’s sitting idle in an installation of Office on your computer, open it up and give it a shot.