Ubuntu/Nautilus Script to Upload to Application Servers

Here’s a fun one. If you’re running Nautilus (which you probably are, if you’re running vanilla Ubuntu), this will allow you to copy one or more assets to one or more app servers with a right click.

Firstly, you need to open the script:

vi ~/.gnome2/nautilus-scripts/"Upload to app servers"

Then paste:


file_count=0
dir="`echo $NAUTILUS_SCRIPT_CURRENT_URI | sed 's/file:\/\///'`"
for arg
do
full_file_or_dir="$dir/$arg"
relative_file_or_dir="`echo "$dir/$arg" | sed 's/\/path\/to\/local\/root\/directory\///'`"
rsync $full_file_or_dir loginuser@appserver.com:/path/to/remote/root/directory/$relative_file_or_dir
# Add other app servers here if you want...
file_count=$((file_count+1))
done
zenity --info --text "Uploaded $file_count files to app servers"

After saving the above example, when you right clicked a file in Nautilus, you’ll see a “Scripts” menu option, which, when expanded, would have an option called “Upload to app servers”.

Upon clicking that script, the relative path of the file in your /path/to/local/root/directory will be copied to the corresponding location in /path/to/remote/root/directory on your appserver.com server. You can select multiple files and upload all at once. After uploading, a zenity message will pop up telling you how many files were successfully copied (requires the zenity package, apt-get install zenity).

So for example, if you right clicked the file /path/to/local/root/directory/myfile.txt, and picked “Upload to app servers,” the file would automagically be copied to appserver.com as /path/to/remote/root/directory/myfile.txt.

This script has proved very helpful for uploading post-deploy fixes to multiple app servers without redeploying the whole lot of them.

Bonanzle: A Year One Retrospective, Part I

I just realized that Bonanzle’s one year anniversary (since beta launch) passed by a month ago without me taking a minute to chronicle the madness of it all. But I’ve never been much of one for hazy-eyed generalities (e.g., “starting a business is hard work,” “business plans are useless,” etc.), which is most likely what this blog would become if written in free form.

I much prefer hard facts and examples, the sort of thing that gets drawn out through the efforts of a skilled interviewer. Unfortunately, since Brian Williams isn’t returning my calls, we’re going to have to settle for a purely fictional interviewer, who I will name Patsy Brown, on account of the fact that she’s a brownnosing patsy that tosses me softballs to make me sound as smart as possible. Thanks a bunch, Patsy. Mind if I call you Pat? Oh, you’d love it if I called you that? Well, great! Let’s great started.

Pat: So, Bill, it has been an incredible year one for Bonanzle. It’s really something how, with no investment money, this creation has garnered more traction than sites with far more money and far more experience. Even more improbably, it’s been done with a team of two full timers and an ever-changing legion of contractors when comparable sites field teams of 10+ employees. How has this first year compared to your expectations?

Bill: Well thank you, Pat. That’s really nice of you to say. It has been pretty wild, hasn’t it? To be honest, there were very few moments in the early evolution of Bonanzle where I thought I knew what would happen next in terms of traffic, sales, or revenues (early interview tangent: I firmly believe the founders of Twitter had no idea what they were building at first either. I still remember in 2007 when they were advertising it as a means to let your friends know what you were doing on a minute-by-minute basis, which was a pretty dumb premise for a business, if you asked me or the other 3 people that were using it at the time. I am already dreading the inevitable declarations of genius that revisionist historians will bestow upon those founders in the months to come. Anyway, we now return to your regularly scheduled interview. And yes, I have no business interrupting this interview before the first paragraph has been finished). I mean, we spent more than a year building Bonanzle as a site that would compete with Craigslist, and it wasn’t until a couple months after we launched (in beta) that we realized that there was simply no market for a better Craigslist.

Once we figured that out and re-geared ourselves as a utopia for small sellers, the first few months were pretty unreal — growing 10-15% larger with every passing week. That was incredibly tough to manage, because at the time we’re increasing the load on our servers by a factor of about 2x-3x monthly, and I was still only learning how to program web sites. If I had set expectations, these early months would have certainly blown them out of the water.

Pat: Can you give us a story to get a sense of just how hectic those early months were?

Bill: Sure, Pat. One memorable Bonanzle moment for me was a week back in October, when I was housesitting for some friends. This was at a time when our traffic was starting to push into the hundreds of thousands of unique users, and our servers were in what I think could best be termed “perpetual meltdown mode.” I remember one particular night where I was up until 4 AM, fiendishly working on some improvements to our search so that it wouldn’t crash. The Olympics were on TV in the room, and I felt like I had an intimate bond with the athletes — I mean, it takes a certain type of insanity to workout for thousands of hours to become the best athlete in the world, and it takes a similar type of insanity to lock oneself up in a room for 12-14 hours per day and try to scale a site up to hundreds of thousands of visitors with no prior experience. Generally, a team of many experienced programmers is going to be required for that amount of traffic, but, being on the brink of going broke, that wasn’t an option. So I pried my eyelids open until I finished the search upgrades, and wearily made my way back to bed to get up early and repeat.

It turned out that that “getting up early” in this case was about three hours after I went to sleep, when I received a then-common automated phone call from our server monitoring center that Bonanzle was down. I dragged myself out of bed, slogged down to the hot computer room, and spent another couple hours figuring out what had gone wrong. When it was fixed, I turned the Olympics back on and basked in our shared absurdity at 8 AM that morning.

Pat: What were some of the key lessons you learned during those months?

Bill: Well, other than technical lessons, the most salient lesson was that, when you find a way to solve a legitimate pain, amazing things can happen. In our case, by building a marketplace as easy as Craigslist with a featureset that rivaled eBay’s, we had what seemed like every seller online telling us how relieved & empowered they felt to have discovered us. Then they told their friends via blogs and forums. It was heady stuff. Our item count rocketed in a way that we were told had never been seen amongst online marketplace not named “eBay,” as we shot from 0 listings to one million within our first few months.

Pat: But with great success comes great responsibility, right? Tell me about how you dealt with managing the community you suddenly found at your (virtual) doorstep.

Bill: That was a real challenge, but something that was really important to us to get right. I have frequently said that, being a Northwest company, one of my foremost goals is for us to live up to the legacy of the great customer-centric companies that have come from this region, like Costco, Amazon, and Nordstrom. Customer service is a company’s best opportunity to get to know its users and develop meaningful trust. So as we started to appreciate the amount of time and effort that’s required to keep thousands of users satisfied, we knew that it was going to become a full time job, so that’s when Mark was anointed the full time Community Overseer.

Pat: Tell me about your relationship with Mark and what he has been to Bonanzle.

The pairing of Mark and I is the sort of thing you read about in “How to Build a Business”-books. Our personalities are in many ways diametrically opposed, but in a perfectly compatible way for a business that requires multiple unrelated skillsets. Mark is patient, I am impatient. Mark is happy dealing with people for hours on end, I am happy dealing with computers for hours on end. Mark is content to be on an amazing ride, I am neverever satisfied with what we have and constantly looking forward.

Fortunately for us, there are also a few qualities that we have in common. We are both OK with constant chaos around us, which is assured in any startup (though I’d say that goes doubly for any community-driven startup). We both enjoy what we do, so we don’t mind 10-12 hour days 6 days per week (right, Mark? :)). And I think we both are generally pretty good at seeing things in other people’s shoes, painful though that sometimes can be when we can’t get something exactly the way we want it due to resource constraints.

Pat: I hear “community” as a common theme amongst your answers. Tell me about the makeup of the Bonanzle community and what their role has been in the building of Bonanzle.

Bill: Well I think it’s pretty obvious within a click or two of visiting the site that Bonanzle is the community. Almost all of the key features that differentiate us from other marketplaces revolve around letting the many talents of our community shine through. It starts from the home page, which is comprised of catchy groups of items curated by our numerous users with an eye for art/uniformity. Real time interaction, another Bonanzle cornerstone, relies on our sellers’ communication talents. And the traffic growth of the site has been largely driven by the efforts of our sellers to find innovative ways to get people engaged with Bonanzle: from writing to editors about us (it was actually the efforts of a single user that got us a feature story in Business Week), to organizing numerous on-site sales (Christmas in July, etc.) that drive buyers from far and wide.

I think that, from a management standpoint, it’s our responsibility to strive to keep out of the way of our sellers. In so doing, the embarrassment of riches we have in member talent can continue to build Bonanzle in ways that we’d have never even considered.

Pat: If you’re just joining us, I’m talking with Bill Harding, Founder of Bonanzle.com, about the experience of his first year of running Bonanzle. Please stay tuned — when we return, I’m going to talk to Bill about what he sees in today’s Bonanzle, and what he predicts for the future of Bonanzle. With any luck, I’ll even get him to answer the eternal question of which is tastier between pizza, nachos, and pie. But for now, a word from our sponsor:

Nice find, Arlene!

Determine location of ruby gems

Not hard:

gem environment

You can even get free bonus information about where ruby and rubygems are… but only if you order now. For a limited time, we’ll even throw in the command that got us this gem of an answer: “gem help commands”

Copy aptitude packages between Linux (Ubuntu) Computers/Systems/Installations

Surprising this isn’t better Google-documented. Here’s how I do it.

Make a list of your existing gems in a text file (run from machine with gems already installed):

sudo dpkg --get-selections | awk '{print $1}' > installedpackages

Copy the file to your new system. Then run:

cat installedpackages | xargs sudo aptitude install -y

It’s run through each line of your gemlist file and run “sudo aptitude install -y” on it.

Set/Increase Memory Available in Rubymine

I did not find anything to love about the results I was getting searching for queries like “Increase rubymine memory” and “set rubymine memory”. Here’s the location of the file that dictates Rubymine memory usage in my Ubuntu install:

[RUBYMINE_DIRECTORY]/bin/rubymine.vmoptions

Inside you can specify lines like:

-Xms800m
-Xmx1200m
-XX:MaxPermSize=1000m
-ea

These declare maximum memory usage, maximum sustained memory usage, and, uh, some other stuff.

[Hops onto soapbox]
I love Rubymine, but I sometimes wish that instead of adding a ridiculous number of new features, they’d just make the damn thing run consistently fast, and not make me worry about memory usage. I know that with my web site I always have a preference toward adding cool new stuff, but responsible thing to do is often to perfect the pieces that are already there. Almost everything negative I’ve heard about Rubymine (which isn’t all that much) is along this same vein, but I’ve never heard the Rubymine team admit that it wishes that the product ran faster or its memory management were better.
[Hops back down]

For my money, still definitely the best choice for Linux IDEs.

Is it just me?

Or do you notice that every time you visit a website or a submit a form, and you get the indefinite spinner of doom, that the url always seems to end in .aspx?

Hmmmm!

How Much Do We Want A Decent Version of Git for Windows? T-H-I-S M-U-C-H.

We want it bad. Reeeeeal bad.

Bonanzle is still running on Subversion (via Tortoise), because comparing its UI to the UI for the de facto git standard (msysgit) is like comparing Rails to ASP on Visual Basic. Yes, the difference is that big. Msysgit is ugly as Pauly Shore, most of its windows can’t be resized, it crashes regularly, and trying to decipher the intent of its UI is like reading dead sea scrolls.

Yes, yes, I know: if you don’t like a piece of open source software, you should shut up and fix it. Unfortunately, I’m sort of single-handedly maintaining this web site that has grown to about 150k uniques this month, where two months ago we had about 10k uniques. I can not end world hunger and rescue my cat stuck in a tree.

But if there is any intelligent life out there that has spare programming cycles and the desire to make a huge difference in the world, this is my personal plea that they will consider giving some love to the woebegone msysgit… or maybe just start their own Windows git client, I can’t imagine a real hacker would take more than a week or two to match the featureset of the existing msysgit.

I’d really like to move Savage Beast to github, and I’d really like to collaborate on the other projects that are happening on there, but it just doesn’t make sense to go from a slick, error free, decipherable UI like Tortoise to the meager helpings of msysgit.

I’d happily donate to a project like better Windows git.

Preemptive note to smart alecks: No, I’m not moving to Mac (or Linux) now. There are plenty of reasons why, I’ll tell you all about it some other time. Incidentally, what is the preeminent GUI for git on Mac these days? From what I understand, many of the real hackers are perfectly content using git from the command line…? Shudder to think of reading the diffs and histories.

MySql Use “Distinct” and “Order by” with Multiple Columns AKA Apply “Order by” before “Group”

I’ve had a devil of a time trying to get Google to tell me how to write a Mysql query that allows us to somehow perform a MySql query that 1) filters rows on a distinct column 2) returns other columns in the query besides the distinct column and 3) allows us to order by a column. In our case, we (and you, if you’re running Savage Beast!) have a list of most recent forum posts on the site — currently, if you list all recent posts, the search will just find all posts and order by date of creation, but this makes for some dumb-looking output since you often end up with a list where 10 of the 20 posts are all from the same forum topic. All the user really wants to know is what topics have a new post in them, and to get a brief glimpse as to what that new post might be.

Thus, we want to create a query that returns the new posts, ordered by date of creation, that have a distinct topic_id.

Here’s the SQL that can make it happen:

Post.find_by_sql(“select posts.* from posts LEFT JOIN posts t2 on posts.topic_id=t2.topic_id AND posts.created_at < t2.created_at WHERE t2.topic_id IS NULL ORDER BY posts.created_at DESC”)

Hope that Google sees fit to lead other people here instead of struggling to get GROUP BY to order results beforehand (GROUP BY ‘posts.topic_id’ works, but it returns the first post in each distinct topic, rather than the last post as we desire), or get SELECT DISTINCT to return more than one column, as many forum posters unhelpfully suggested in all the results I was getting.

Update 11/26/08 – A Word of Caution

I finally got around to setting up some profiling for our site yesterday and was surprised to discover that the above query was taking longer per execution than almost anything else on our entire site. The SQL Explain was not too helpful to explain why, but it showed three joins, the join on the topics table involving every row of the table (which is presently almost 10,000).

Takeaway: for this query to work, it seems to consider every distinct topic in the table, rather than being smart and stopping when it hits the per-page paginated limit. Since I already determined that “group” and “distinct” were non-starters for being able to pick the newest post in each topic, I ended up revising the way the logic was done to an easier to manage and far-more-DB-efficient way:

We now track in each topic the newest post_id within that topic. While this adds a bit of overhead to keeping the topic updated when new posts are made to it, it allows us to do a far simpler query where we just select the more recent topics, joining to the most recent post in each topic, and then ordered by the age of those posts.

If you have the ability to create an analogous situation to solve your problem, your database will thank you for it. The above query starts getting extremely slow with more than a few thousand rows. Yet, I defy you to find an alternative to it that works at all using “group” or “distinct.”