Measuring developer productivity in 2019

Until I spent the better part of 50 hours writing this guide about measuring developer productivity, I had little appreciation for how far we’ve come in the past four years. If you’re an engineering manager, there’s a growing body of data that suggests you can make a blanket increase in engineering efficacy. If you’re a great developer, you now stand to earn your due. These are exciting times to be programming.

Additional topics covered in the (admittedly a bit too epic) guide:

If you manage developers, would love to hear what you think of this effort? Is it relevant to you? Anything that could make it better?

Alternatives to Xmarks (now discontinued)

Xmarks (nee Firemarks) has been a reliable companion to address my cross-platform bookmarking needs over the past 10 years. As such, I was saddened to receive word that parent company Lastpass has chosen to discontinue Xmarks as of mid-2018. This sent me on a quest to find the best Xmarks alternative, but the first Google result I was presented (alternative.to) when searching “Xmarks alternatives” contained a spate of services that had precious little relevance to users like myself who simply want the means to save bookmarks in a cross-browser compatible extension, with as few other bells and whistles as possible.

After considerable Googling, here are the top three recommendations I’d present to other Xmarks users being forced to abandon ship:

Eversync

There’s good reason that this is the most-cited service you’ll find (alongside Xmarks) when Googling terms like “cross browser bookmarks.” It supports all the major browsers/platforms (including my current laptop, running Chromium on Arch Linux). It has been doing this long enough to have a quaint (read: “embarrassingly outdated”) little website that imputes the difficulty of building a business with via bookmarking extension.

Eversync’s web site welcomes you to the 90s

Most importantly, it’s got impeccable ratings. As of March 2018, it maintains a 4.5 star rating on the Chrome app store with about 3,000 ratings. I consider this an incredible feat, given that greater volume of ratings typically drive a product’s average toward the 3 stars.

The biggest gripe I have with Eversync is that its creators package a collection of junk like “Speed Dial.” If you’re a power user with more than 500 bookmarks, Nimbus (who appears to have purchased the extension from Everhelper at some point) will try to charge you $45/year for the extension (plus Nimbus bloat). Woe that it does cost money to maintain a web service.

Chrome sync

Ok, technically this isn’t a cross-browser solution. But if you’re like me and you only stray outside of Chrome on occasion (usually to test a page I’m developing on another browser), then the simplest way to sync bookmarks is to simply use the browser’s built-in capability. No extensions to download, no potential that you’re going to have to change managers again when software maker abandons their product.

If you choose this option, you will most likely want to visit your browser’s Settings -> Sync settings and disable the properties that you don’t want or need synced on all of your workstations:

Raindrop.io

This option “only” manages 4 stars on 300 ratings in the Chrome app store, so it’s a half-step below Eversync in terms of its user reputation. I’m including it here anyways because its home page inspires me to believe that its developers are actively working on it, and thus its functionality may be more likely to improve compared to Eversync. Further, it’s list of features including “Duplicate finder” and “Broken link finder” indicate a level of product dogfooding (creator using own product = creation of features to maximize user satisfaction) that was largely absent in Xmarks. For what it’s worth, the Pro version of Raindrop is also a few bucks cheaper than the Pro version of Eversync ($5/month for Eversync vs $3/month for Raindrop).

If anyone has firsthand experience with Raindrop, I’d be much obliged to learn your satisfaction level in the comment section below.

Why’s “open link in new tab” not working in Safari + iOS 10 + iPhone 7?

From the files of “endlessly frustrating problems that Google wouldn’t help me with.”

When I bought my iPhone 7, I couldn’t open links in new tabs in Safari for the first six months I owned it. Whenever I tried to hold a link to open it, Safari would sometimes ignore me, sometimes it would start showing a preview of the destination for that link. Which behavior it chose to do seemed to depend on the whether Mercury was in retrograde.

After much searching, I eventually learned this behavior isn’t a “bug.” According to Apple, it is the “feature” that they have long been marketing as “3d touch.” That is, your iPhone is now paying attention to how hard your touch your Safari links. If you hold a link “hard,” then, instead of popping up the context menu that allows you to open the link in a new tab, you instead get the obnoxious popup preview of the next page.

The solution to being able to pop the standard iOS Safari context menu? Hold on a link in Safari, but do it lightly, tenderly, as if you were stroking a baby dove.

This is what “intuitive UI” has become on the new iPhone.

Turning off 3d touch

If, like me, you find 3d touch to be more of a nuisance than a feature, you can disable it as follows: open Settings, search for “Touch,” and pick what should be the first option in the list:

Then click into the 3d touch option, which can be turned off for a 32% less aggravating mobile experience:

PS bonus fix! Have you had trouble moving your iPhone’s icons around since upgrading to iOS 10.1? Guess why. That’s right, 3d touch. As with the above, to move around app icons on a 3d-touch device, you have to touch them ever so lightly.

I will continue waiting for this to feel like a “feature.”

Evernote vs Quip: A note taker’s detailed comparison

Over the past 5 years, I’ve probably spent more time in Evernote than most any other productivity app. I’ve created thousands of notes across Evernote’s OS X, iOS and web clients, becoming intimately familiar with the limitations of each. Over the years, I’ve periodically tested alternatives, mostly when Evernote obliterates my changes via merge conflicts with unsynched notes. I had high hopes for Onenote when I gave it a few weeks’ test run recently, but ultimately determined it was even more fatally flawed than Evernote (not to mention how difficult they make it to import past work from Evernote).

A couple days ago I discovered Quip. It is a multiplatform note-taking app that apparently started as a collaboration platform, and was acquired by Salesforce for $750m in 2016. It sports a reasonably effective built-in Evernote importer, plus an impressive suite of free features. Based on my initial impressions, I’m giving it even odds to replace Evernote as my note-taking app of choice. Yet, like Evernote, it too comes with a set of warts.

Given how powerful-yet-flawed both products are, I’m going to attempt to enumerate, from a note-taker’s perspective, where each shines. Hopefully this list (ordered roughly by impact on my everyday productivity) helps others decide which set of drawbacks is most forgivable to them.

Evernote Advantages

  • Notes can be displayed in grid while ordered by recency of update or other criteria (Quip only allows user-defined sorting in list view…and only on desktop clients at that. C’mon dudes, allowing notes ordered by recency of update is utterly essential feature)
  • Can navigate from note to note with single click, since list of notes remains visible in left pane even when a note is being edited
  • Better UI for text formatting: resides permanently at top of each note (in Quip, text formatting options are hidden behind a sometimes temperamental popout in left gutter)
  • Support for searching phrases (not just individual words)
  • Essentially instant search
  • Not as aggressive about trying to spam all my contacts during install (unlike Quip — forcing me to manually uncheck every Google contact during iOS install. Not a great approach to build goodwill with a new user, Quip)

Quip Advantages

  • Notes are saved in real time as they’re created (!). The note’s text is automatically synchronized to all devices that have the note open. Not only does this fix Evernote’s most fatal shortcoming (merge conflicts), it also allows multiple people to edit the same document concurrently, a la Google Docs
  • Less janky checkboxes for todo lists. They don’t slow down the note when there are lots of them, they can be copy-pasted without introducing weirdness, and their line gets crossed out when you check off your todo item.
  • Spreadsheets with full Excel-like functionality (functions, etc) are natively supported, and can be inserted inline into any note
  • Code blocks can be inserted into any note (!)
  • Variety of attractive pre-built font themes can be selected and set as default for notes
  • Can use across more than two devices for free (Evernote now charges for this functionality)
  • Note’s edit history can be viewed without paying monthly fee
  • More robust feature set for collaboration (collaboration seems to have been the initial purpose behind Quip’s existence) than Evernote. Can add inline comments to any line of any note.
  • More intuitive interface for working with multi-level lists (arrows to move back and forth between list level, rather than needing to use tab+shift)
  • Can organize notes into subfolders within a notebook
  • No upsells as yet in free version

Both apps allow chatting (because everyone wants to chat in their note taking app…right?), and both allow some degree of collaboration on notes (Quip much stronger here). It’s my goal to use Quip as my primary app for the next couple weeks before I make a final call on the victor. If you have experience with either app, would be delighted to hear your power user tips in the comment section below!

Evernote QA Report: Merge & Sync

Hi Evernote,

Your sync and merge has long been broken, and I hope to provide you enough detail to get it fixed here. All the below should be reproducible, let me know if you find otherwise.
List Fruits 1

It starts simple. Create a note in Evernote webview. List some fruits. If you’re anything like me, you probably make a couple lists like this per day. All seems well. Later that day, you remember this list, and have a burning need to add “Starfruit” to it. All looks well?

2015-11-05 16.06.47
But things are not so well, because when we return to the webview, there’s some bad news awaiting:
List Fruits Double Note               ^– Now two notes, instead of one

That’s right, my fruit list has split itself in two lists with edits from desktop and mobile. Similar problems persist as I try to use the document.

Unless one is fastidious enough to sync every note when finished, and then manually sync when resuming on another platform, you are liable (at rate inversely proportional to speed of your internet connection) to soon find yourself with something like this:

Two Syncs Later

Every time I switch devices, I must remember “ensure save,” then pre-sync on the new device (and remember to confirm sync finished). Granted, there are a number of trigger points for syncs, but it’s impossible to be sure which, if any, have triggered/completed. One too-fast move, and I’m back in the ghetto of Mergetown.

I usually don’t even know I’ve been losing list items (to duplicated versions), until I notice the note lags in load/save time. And then I realize there are 10 copies of the list in my note, below the (already long) version I was treating as the “real list.”

I sympathize that getting sync to work reliably across disparate platforms is a hard problem [2]. So why not, instead of fully fixing the sync, just spend a few days fixing the merge?

If you could automatically add new lines to both documents without creating duplicate copies, that would be a good start. If you kept the note differences in the area they were found (not bottom of document), that would help. Source control tools have spent 20+ years building great systems to merge two documents. You can probably crib most of your implementation from a plugin/library in that domain? Or at least mark “duplicate documents” so I know I’ll need to hand merge them?

Thanks for considering these improvements. I take the time to write this because I still believe you’re the best solution in your space. If you could just address this fundamental stuff, I’d reap benefits greater than the (sizeable) time it took to create this report.

[1] Lists featuring fruits.
[2] Albeit the hard problem you ostensibly built your business to solve

Fix it: Ubuntu CTRL-SHIFT-V Won’t Paste Into Terminal

Ugh, just spent an hour traveling from forum to forum trying to figure out why I couldn’t CTRL-C in Rubymine and CTRL-SHIFT-V into terminal. As many forum posters were eager to point out, it is possible to use CTRL-SHIFT-INSERT or middle click to paste into terminal, just not CTRL-SHIFT-V. Unfortunately, those workarounds were not OK since my insert key is in the arctic circle of my keyboard, and I don’t want to touch the mouse.

Luckily, the folks at Rubymine helped me figure out the answer where Google couldn’t.

The problem is that when running Rubymine via OpenJDK, the clilpboard used is not compatible with the clipboard used in terminal. The solution was to run Rubymine with Oracle Java instead. In Rubymine, this is a matter of first installing Oracle Java (I’ll let you Google that one, you have to add a repository) and then adding the following lines to the beginning of your Rubymine startup script:

export JAVA_HOME=/usr/lib/jvm/java-6-sun
export JDK_HOME=$JAVA_HOME
export RUBYMINE_JDK=$JAVA_HOME

After that you should be golden. In my hour of Googling I read that many other IDEs (Netbeans in particular) seem to be subject to the same problem with CTRL-SHIFT-V not working. I’d reckon that if these users were to change their application to use Oracle Java it would probably resolve the problem in other IDEs as well.

Traits & Qualities of Best Developers on a Deserted Island

What is programming perfection? While the qualities that comprise “the most productive” developer will vary to some extent based on industry and role, I believe there are a number of similarities that productive developers tend to share. Understanding these similarities is a key to improving oneself (if oneself == developer) or differentiating from a pool of similar-seeming developer prospects. In my assorted experience (full time jobs in Pascal, C, C++, C#, and now Ruby (damn I sound old)), I have repeatedly observed developers who vary in productivity by a factor of 10x, and I believe it is a worthwhile exercise to try to understand the specifics behind this vast difference.

<OBLIGATORY DISCLAIMER>

I do not profess to be kid-tested or mother-approved (let alone academically rigorous or scientifically proven) when it comes to prescribing what exactly these qualities are. But I do have a blog, an opinion, and an eagerness to discuss this topic with others. I realize this is a highly subjective, but that’s part of what makes it fun to try to quantify.

</ OBLIGATORY DISCLAIMER>

As a starting point for discussion, I hereby submit the following chart which approximates the general progression I’ve observed in developers that move from being marginally productive to absurdly productive. It works from the bottom (stuff good junior programmers do) to the top (stuff superstars do).

Levels of Developer Productivity

And here is my rationale, starting from the bottom:
mario_1

Level 1: Qualities possessed by an effective newbie

Comments on something, somewhere. A starting point for effective code commenting is that you’re not too lazy or obstinate to do it in principle.

Self-confident. Programmers that lack self-confidence can never be extremely effective, because they spend so much time analyzing their own code and questioning their lead about their code. That said, if you’re learning, it’s far better to know what you don’t know than to think you do know when you really don’t. So this one can be a difficult balance, particularly as coders grow in experience and become less willing to ask questions.

Abides by existing conventions. This is something that new coders actually tend to have working in their favor over veterans. On balance, they seem to be more willing to adapt to the conventions of existing code, rather than turning a five person programming project into a spaghetti codebase with five different styles. A codebase with multiple disparate conventions is a subtle and insidious technical debt. Usually programmers starting out are pretty good at avoiding this, even if their reason is simply that they don’t have their own sense for style.

Doesn’t stay stuck on a problem without calling for backup. This ties into the aforementioned danger with being self-confident. This is another area where young programmers tend to do pretty well, while more experienced coders can sometimes let themselves get trapped more frequently. But if you can avoid this when you’re starting, you’re getting off on the right foot.

mario_2

Level 2: Qualities possessed by an effective intermediate programmer

Understand the significance of a method’s contract. Also known as “writes methods that don’t have unexpected side effects.” This basically means that the developer is good at naming their functions/methods, such that the function/method does not affect the objects that are passed to it in a way that isn’t implied by its name. For example, when I first started coding, I would write functions with names like “inflictDamage(player)” that might reduce a player’s hitpoints, change their AI state, and change the AI state of the enemies around the player. As I became more experienced, I learned that “superfunctions” like this were not only impossible to adapt, but they were very confusing when read by another programmer from their calling point: “I thought it was just supposed to inflict damage, why did it change the AI state of the enemies?”

Figures out missing steps when given tasks. This is a key difference between a developer that is a net asset or liability. Often times, a level 0 developer will appear to be getting a lot done, but their productivity depends on having more advanced programmers that they consult with whenever they confront a non-trivial problem. As a lead developer, I would attempt to move my more junior developers up the hierarchy by asking “Well, what do you think?” or “What information would you need to be able to answer that question?” This was usually followed by a question like, “so what breakpoint could you set in your debugger to be able to get the information you need?” By helping them figure it out themselves, it builds confidence, while implicitly teaching that it is not more efficient to break their teammates’ “mental context” for problems that they have the power to solve themselves.

Consistently focused, on task. OK, this isn’t really one that people “figure out” at level two, so much as it is a quality that usually accounts for a 2x difference in productivity between those that have it and those that don’t. I don’t know how you teach this, though, and I’d estimate that about half of the programmers I worked with at my past jobs ended up spending 25-50% of their day randomly browsing tech sites (justified in their head as “research?”). Woe is our GDP.

Handy with a debugger. Slow: figure out how a complicated system works on paper or in your head. Fast: run a complicated system and see what it does. Slow: carefully consider how to adapt a system to make it do something tricky. Fast: change it, put in a breakpoint, does it work? What variables cause it not to? Caveat: You’ve still got to think about edge cases that didn’t naturally occur in your debugging state.

mario_3

Level 3: Hey, you’re pretty good!

Thinks through edge cases / good bug spotter. When I worked at my first job, I often felt that programmers should be judged equally on the number of their bugs they fixed and the number of other programmers bugs’ they found. Of course, there are those that will make the case that bug testing is the job of QA. And yes, QA is good. But QA could never be as effective as a developer that naturally has a sense for the edge cases that could affect their code so don’t write those bugs in the first place. And getting QA to try to reproduce intermittent bugs is usually no less work than just examining the code and thinking about what might be broke.

Unwilling to accept anomalous behavior. Good programmers learn that there is a serious cost to “code folklore” — the mysterious behaviors that have been vaguely attributed to a system without understanding the full root cause. Code folklore eventually leads to code paranoia, which can cause inability to refactor, or reluctance to touch that one thing that is so ugly, yet so mysterious and fragile.

Understands foreign code quickly. If this post is supper, here’s the steak. Weak developers need code to be their own to be able to work with it. And proficient coders are proficient because, for the 75% of the time that you are not working in code that you recently wrote, you can figure out what’s going on without hours or days of rumination. More advanced versions of this trait are featured in level four (“Can adapt foreign code quickly”) and level five (“Accepts foreign code as own”). Stay tuned for the thrilling conclusion on what the experts can do with code that isn’t theirs.

Doesn’t write the same code twice. OK, I’ve now burned through at least a thousand words without being a language zealot, so please allow me this brief lapse into why I heart introspective languages: duplicated code is the root of great evil. The obvious drawback of writing the same thing twice is that it took longer to write it, and it will take longer to adapt it. The more sinister implications are that, if the same method is implemented in three different ways, paralysis often sets in and developers become unwilling to consolidate the methods or figure out which is the best one to call. In a worst case scenario, they write their own method, and the spiral into madness is fully underway.
mario_4

Level 4: You’re one of the best programmers in your company

Comments consistently on code goals/purpose/gotchyas. Bonus points for examples. You notice that I haven’t mentioned code commenting since level one? It is out of my begrudging respect for the Crazed Rubyists who espouse the viewpoint that “good code is self-documenting.” In an expressive language, I will buy that to an extent. But to my mind, there is no question that large systems necessarily have complicated parts, and no matter how brilliantly you implement those parts, the coders that follow you will take longer to assimilate them if the docs are thin. Think about how you feel when you find a plugin or gem that has great documentation. Do you get that satisfying, “I know what’s going on and am going to implement this immediately”-sort of feeling? Or do you get the “Oh God this plugin looks like exactly what I need but it’s going to take four hours to figure out how to use the damned thing.” An extra hour spent by the original programmer to throw you a bone would have saved you (and countless others) that time. Now do you sympathize with their viewpoint that their code is “self-documenting?”

Can adapt foreign code quickly. This is the next level of “understands foreign code quickly.” Not only do you understand it, but you know how to change it without breaking stuff or changing its style unnecessarily. Go get ’em.

Doesn’t write the same concept twice. And this is the next level of “doesn’t write the same code twice.” In a nutshell, this is the superpower that good system architects possess: a knack for seeming patterns and similarities across a system, and knowing how to conceptualize that pattern into a digestible unit that is modular, and thus maintainable.
mario_5

Level 5: Have I mentioned to you that we’re hiring?

Constant experimenting for incremental gains. The best of the best feel an uncomfortable churn in their stomach if they have to use “find all” to get to a method’s definition. They feel a thrill of victory if they can type in “hpl/s” to open a file in the “hand_picked_lists” directory called “show” (thank you Rubymine!). They don’t settle for a slow development environment, build process, or test suite. They cause trouble if they don’t have the best possible tools to do their job effectively. Each little thing the programming expert does might only increase their overall productivity by 1% or less, but since developer productivity is a bell curve, those last few percent ratchet the expert developer from the 95th to 99th percentile.

Accepts foreign code as their own. OK, I admit that this is a weird thing to put at the top of my pyramid, but it’s so extremely rare and valuable that I figure it will stand as a worthwhile challenge to developers, if nothing else. Whereas good developers understand others’ code and great developers can adapt it, truly extraordinary developers like foreign code just as much as they like their own. In a trivial case, their boss loves them because rather than complaining that code isn’t right, they will make just the right number of revisions to improve what needs to be improved (and, pursuant to level 3, they will have thought through edge cases before changing). In a more interesting example, they might take the time to grok and extensively revise a plugin that most developers would just throw away, because the expert developer has ascertained that it will be 10% faster to make heavy revisions than to rewrite from scratch. In a nutshell, whereas most developers tolerate the imperfections in code that is not their own, experts empathize with how those imperfections came about, and they have a knack for figuring out the shortest path to making the code in question more usable. And as a bonus, they often “just do it” sans the lamentations of their less productive counterparts.

This isn’t to say they won’t revise other people’s crappy code. But they’re just as likely to revise the crappy code of others as they are their own crappy code (hey, it happens). The trick that these experts pull is that they weigh the merits of their own code against the code of others with only one objective: what will get the job done best?

Teach me better

What qualities do you think are shared by the most effective developers? What do you think are the rarest and most desirable qualities to find? Would love to hear from a couple meta-thinkers to compare notes on the similarities you’ve observed amongst your most astoundingly productive.

Rails tests: One line to seriously pump up the speed

Alright, I admit it: I didn’t seriously write tests for Bonanzle until a couple months ago. But I had my reasons, and I think they were good ones.

Reason #1 was that I hated everything about fixtures. I hated creating them, I hated updating them every time we migrated, and I hated remembering which fixture corresponded to which record. Factory Girl was the panacea for this woe.

Reason #2 was that it took an eon to run even a single test. When trying to iterate tests and fixes, this meant that I ended up spending my time 10 parts waiting to one part coding. After much digging, I eventually determined that 90% of our test load time was attributable to caching all our classes in advance. Of course, my first inclination was just not not cache classes in our test environment, which actually worked reasonably well to speed tests the hell up, until I started writing integration tests, and found our models getting undefined and unusable over the course of multiple requests. Then, I found the answer:

config.eager_load_paths.clear

This line basically says, even if you set config.cache_classes = true, Rails should not try to pre-load all models (which, in our case is more than 100).

Adding this line allows us to cache classes in test (which fixes the integration test problems), while at the same time getting the benefits of a configuration that doesn’t take 2 minutes to load.

(Of course, also key was configuring our test rakefile such that we could run single tests, rather than being obligated to run the entire suite of tests at once. If anyone needs finds this post and doesn’t yet know how to invoke a single test, post a comment and I’ll get unlazy and post the code for that)

Best of Rails GUI, Performance, and other Utilities

I’m all about putting order to “best of” and “worst of” lists, so why not give some brief props to the tools, plugins, and utilities that make life on Rails a wonderous thing to behold?

5. Phusion Passenger. OK, this would probably be first on the list, but it’s already been around so long that I think it’s officially time to start taking it for granted. But before we completely take it for granted, would anyone care to take a moment to remember what life was like in a world of round-robin balanced Mongrels web servers? You wouldn’t? Yah, me neither. But no matter how I try, I cannot expunge memories of repeatedly waking up to the site alarm at 7am to discover somebody had jammed up all the Mongrels with their stinking store update and now I’ve got to figure out some way to get them to stop.

4. jQuery / jRails. This probably deserves to score higher, as the difference between jQuery and Prototype is comparable to the difference between Rails and PHP. But since it’s not really Rails-specific, I’m going to slot it at four and give major props to John Resig for being such an attentive and meticulous creator. Without jQuery and jQuery UI, the entire web would be at least 1-2 years behind where it is in terms of interactivity, and I don’t think that’s hyperbole. (It even takes the non-stop frown off my face when I’m writing Javascript. With jQuery, it’s merely an intermittent frown mixed with ambivalence!)

3. Sphinx / Thinking Sphinx. There’s a reason that, within about six months time, Thinking Sphinx usurped the crown of “most used full text search” utility from “UltraSphinx.” And the reason is that it takes something (full text search) that is extremely complicated, and it makes it stupidly easy. And not just easy, but extremely flexible. Bonanzle has bent Sphinx (0.9.8) into doing acrobatics that I would have never guessed would be possible, like updating it in nearly-real time as users log in and log out. Not to mention the fact it can search full text data from 4 million records in scant milliseconds.

Sphinx itself is a great tool, too, though if I were going to be greedy I would wish that 0.9.9 didn’t reduce performance over 0.9.8 by around 50% in our testing, and I would wish that it got updated more often than once or twice per year. But in the absence of credible competition, it’s a great search solution, and rock solidly stable.

2. New Relic. OK, I’ll admit that I’ve had my ups and downs with New Relic, and with the amount of time I’ve spent complaining to their team about the UI changes from v1 to v2, they probably have no idea that it still ranks second in my list, ahead of Sphinx, as most terrific Rails tools. But it does, because, like all the members of this list, 1) the “next best” choice is so far back that it might as well not exist (parsing logs with pl-analyze? Crude and barely useful. Scout? Nice creator, but the product is still a tot. Fiveruns? Oh wait, Fiveruns doesn’t exist anymore. Thank goodness) and 2) it is perhaps the most essential tool for running a production-quality Rails site. Every time I visit an ASP site and get the infinite spinner of doom when I submit a form, I think to myself, “they must not know that every time I submit a form it takes 60 seconds to return. That would suck.” On a daily basis, I probably only use 10% of the functionality in New Relic, but without that 10%, the time I’d spend tracking logs and calculating metrics would make my life unfathomably less fun.

1. Rubymine. The team that created this product is insane. Every time that I hit CTRL-O and I type in “or/n” and it pops up all files in the “offers_resolution” folder starting with the letter “n,” I know they are insane, because having that much attention to productivity is beyond what sane developers do. Again, for context’s sake, one has to consider the “next best” choice, which, for Linux (or Windows) is arguably a plain text editor (unless you don’t mind waiting for Eclipse to load and crash a few times per day). But, instead of programming like cavemen, we have a tool that provides killer function/file lookup; impeccable code highlighting and error detection (I had missed that, working in a non-compiled language); a working visual debugger; and, oh yeah, a better Git GUI than any of five standalone tools that were built specifically to be Git GUIs.

Perhaps as importantly as what Rubymine does is what it doesn’t do. It barely ever slows down, it doesn’t make me manage/update my project (automatically detecting new files and automatically adding new files to Git when I create them from within Rubymine), and it handles tabs/spaces like a nimble sorcerer (something that proved to be exceedingly rare in my quest for a usable IDE).

Like New Relic, I probably end up using only a fraction of the features it has available, but I simply can’t think of anything short of writing my code for me that Rubymine could do that it doesn’t already handle like a champ. Two thumbs up with a mutated third thumb up if I had one.

Conclusion

Yes, it is a list of apples and oranges, but the commonality is that all five-ish of the lists members stand apart from the “second best” solution in their domain by a factor of more than 2x. All of them make me feel powerful when I use them. And all of them, except arguably New Relic, are free or bargain priced. Hooray for life after Microsoft. Oh how doomed are we all.

Rails Performance Analysis and Monitoring? Free and Easy with pl_analyze

Since writing about our lackluster experience with Fiveruns awhile back, I had been keeping my eyes open for a good way to get reliable, comprehensive, and easy-to-interpret performance metrics for Bonanzle. Prior to our transition away from shared hosting in November, I would browse through our production logs and regularly find actions taking 10 seconds or more, but I had no way to get a good overall sense of what actions were taking the longest, how long those actions were taking, and how it was changing from day to day.

Now we have a solution, and it’s a dandy.

Many have heard of pl_analyze, but I think that few may realize how truly easy it is to get up and running on any Rails 2.X application (we’re now on 2.2). To start, Geoff has written a strikingly excellent blog about how to change your log format to be compatible with pl_analyze by adding a single file to your lib directory and changing a couple lines in environment.rb.

After you follow his instructions, here are a couple more key points that I found relevant to polish the rough edges:

* By default, when we were running 2.0 (don’t know if it’s the case with 2.2, since we were pre-patched when we upgraded), logging of DB times was disabled. This monkey patch fixes that.

* By default, when we moved to 2.2, the log format changed and broke pl_analyze. This fixes that.

* After enabling the Hodel logger as described by Geoff, our production log became extremely verbose. One of the helpful commenters on Geoff’s blog pointed out the solution to this: add the line “config.logger.level = Logger::INFO” to your production.rb

After that, you’re good to go. Took me less than five minutes to get it set up, after which I was able to get priceless info like the following:

Request Times Summary: Count Avg Std Dev Min Max
ALL REQUESTS: 1576 0.242 0.229 0.000 2.487

ItemsController#show: 685 0.328 0.141 0.008 0.751
ChatsController#service_user_list: 286 0.022 0.037 0.005 0.301
HomeController#index: 158 0.256 0.308 0.000 1.461

Well, doesn’t look so pretty on my crap blog (shouldn’t I have upgraded this by now?), but it’ll look like a champ in your terminal window.

After getting our basic setup established a couple weeks ago, I started experimenting today with archiving our daily performance info into the database so I can track how action times are changing from day to day. Geoff provides a great start for this as well, with his pl_analyze Rails plugin, which you can install by running

./script/plugin install http://topfunky.net/svn/plugins/mint

After you’ve installed the plugin, run

rake mint:create_analyze_table

to create your table, and then

rake mint:analyze PRODUCTION_LOG=/path/to/production.log

to actually extract and store the day’s data. Be sure to read my comment in Geoff’s blog (it’s about the 30th comment) for a slight modification you may need to make to the analyze task if you get an error that says something to the effect that “0 parameteres were expected, got 1” or somesuch when you run mint:analyze.

Also note that this will store everything in your production log, so if you haven’t already setup log rotating on your production log, you’ll want to do that (it can be as simple as this).

After you’re archiving performance metrics to your database, you are in a performance analysis wonderland — it’s just a matter of applying creativity to data. For starters, I plan to create reports that list the top 10 daily time users (= count of action * average time per action), and the top 10 biggest daily deltas in time taken for various actions. I plan to use the latter to help triangulate where code changes affect performance. It would be trivial to create a model that actually allowed daily notes to be added to a report, so if the items#show action starts running 20% slower one day, I could demarcate the code changes that were made on the day that happened (though our changelist makes such a demarcation unnecessary).

I’m not sure what other tools are being used by the community to keep an eye on performance, but with the quickness and reliability (= stable results reported for weeks) of getting a pl_analyze solution running, it gets high recommends from ’round these parts. I can’t imagine returning to a day where we didn’t know if our average request time was 300 ms, 3000 ms, or 30000 ms (yes, we had some very dark days before our server move).