Category: Rails

Ran into an interesting bug a few days ago that I’ve been meaning to document for anyone who has written their own truncating function in Rails. If you have, I’d guess it probably looked something like this:

def truncate(str, length)
	return '' if str.blank?
	truncated = str.size > length
	(str[0..(truncated ? length - 3 : length)] + (truncated ? "..." : ''))
end

Fine enough. Unless you happen to be truncating user-given strings.

If you are letting your users enter in the information that gets truncated, chances are that some of them are entering unicode characters for accents, quotation marks, etc. Because unicode characters are 2-4 bytes long, the above truncate will split characters in half and cause general headaches if it truncates text that has unicode characters in it.

Split-in-half characters are bad news. They will cause errors like “illegal JSON output,” which is how I originally spotted this as a problem with our truncate method.

The solution is to take a page from Rails’ own truncate, and use #mb_chars. So a hand-written truncate that works for unicode would be:

def truncate(str, length)
	return '' if str.blank?
	truncated = str.size > length
	(str.mb_chars[0..(truncated ? length - 3 : length)] + (truncated ? "..." : '')).to_s
end

You’re welcome.

This was a vexing problem that probably affects 0.1% of all Thinking Sphinx users, but for those select few, you can benefit from my pain.

We have a model with the following associations:

has_many :merch_match_data_tags, :class_name => "MerchMatchItemData", :dependent => :delete_all
has_one :mm_bizarre, :class_name => "MerchMatchItemData", :conditions => { :data_tag_id => MerchMatchItemData::BIZARRE }
has_one :mm_good_picture, :class_name => "MerchMatchItemData", :conditions => { :data_tag_id => MerchMatchItemData::NICE_PICTURE }
has_one :mm_funny, :class_name => "MerchMatchItemData", :conditions => { :data_tag_id => MerchMatchItemData::FUNNY }

Intending to add these to a Sphinx index, we used the following code:

has mm_good_picture.tag_count, :as => :good_picture_points
has mm_bizarre.tag_count, :as => :bizarre_points
has mm_funny.tag_count, :as => :funny_points

What perplexed me after trying this was that while the “good_picture_points” could be queried and sorted, bizarre_points and funny_points returned no Sphinx results. Looking into the output generated by thinking_sphinx:configure, I discovered why:

...
LEFT OUTER JOIN `merch_match_item_datas` ON merch_match_item_datas.item_id = items.id AND `merch_match_item_datas`.`data_tag_id` = 0   LEFT OUTER JOIN `merch_match_item_datas` mm_bizarres_items ON mm_bizarres_items.item_id = items.id AND `merch_match_item_datas`.`data_tag_id` = 2   LEFT OUTER JOIN `merch_match_item_datas` mm_good_values_items ON mm_good_values_items.item_id = items.id AND `merch_match_item_datas`.`data_tag_id` = 7   LEFT OUTER JOIN `merch_match_item_datas` mm_funnies_items ON mm_funnies_items.item_id = items.id AND `merch_match_item_datas`.`data_tag_id` = 4  
...

The problem was that, in determining the SQL to build, Thinking Sphinx uses the first association it comes across as the default set of conditions for all future joins to the table. So, in this case, anything that joined the merch_match_item_datas table was going to be joining that table with the data_tag_id = 0 condition of our first declared association (mm_good_picture_tag). That is, mm_bizarre now was looking for data_tag_id=0 and data_tag_id=[id of bizarre tag]. So, nothing was returned.

After a bit of head scratching, I came up with the following workaround for this:

has merch_match_data_tags.tag_count
has mm_good_picture.tag_count, :as => :good_picture_points
has mm_bizarre.tag_count, :as => :bizarre_points
has mm_good_value.tag_count, :as => :good_value_points

Basically, just make the first association that Thinking Sphinx encounter be an unqualified, unfiltered association to the merch_match_data_table. This ensures that the proper join structure is setup, so all of the subsequent has attributes function as they should.

Hope that I’m not the only one ever to find this useful.

From the results Google gives on this, it seems that about three people in the world are using integration tests in Rails, and two of them stopped programming in 2007.

My goal: to get at session data from within an integration test.

Bad news: I don’t know any way to do this without first calling a controller action from within your integration test.

Good news: I have example code on how to get at it after making a request.

def add_to_cart_integration_test
  s = open_session
  s.post url_for(:controller => 'shopping_cart', :action => :add_item, :item_id => 1)
  s.session.data # a populated session hash
  s.flash # your flash data
end

And there you have it. Here is a more detailed examples that may be of use, derived from working integration tests in our codebase:

open_session do |s|
	item_1 = FactoryHelper::ItemFactory.create_sellable_item
	add_item_to_cart(item_1, s)
	user = Factory(:activated_user, {:password => USER_PASSWORD})
	login_user(user, USER_PASSWORD, s)

	s.post url_for(:controller => 'offers', :action => :cart_summary)
	all_offers = s.assigns(:all_offers_by_booth)
	assert_equal all_offers.size, 1
	the_offer = all_offers.first

	# Trial 1:  Simulate dragging item out of cart
	s.post url_for(:controller => 'lootbins', :action => :remove_from_lootbin_draggable, :item => item_1.id)

	s.post url_for(:controller => 'offers', :action => :cart_summary)
	all_offers = s.assigns(:all_offers_by_booth)
	assert all_offers.empty?
	assert !Offer.exists?(the_offer)

	# Trial 2:  Simulate removing item from cart on cart page
	add_item_to_cart(item_1, s)
	s.post url_for(:controller => 'offers', :action => :cart_summary)
	all_offers = s.assigns(:all_offers_by_booth)
	assert_equal all_offers.size, 1
	the_offer = all_offers.first

	s.put url_for(:controller => 'offers', :action => :remove_from_offer, :item_id => item_1.id, :id => the_offer.id)
	s.post url_for(:controller => 'offers', :action => :cart_summary)
	all_offers = s.assigns(:all_offers_by_booth)
	assert all_offers.empty?
	assert !Offer.exists?(the_offer)
end

And here are the helpers involved:

def add_item_to_cart(item, os)
	items_in_bin = begin
		lootbin = Lootbin.new(os.session.data) # equivalent to ApplicationController#my_lootbin
		lootbin.offer_items.size
	rescue
		0
	end
	os.post url_for(:controller => 'lootbins', :action => :add_to_lootbin_draggable, :item => item[:id])
	lootbin = Lootbin.new(os.session.data) # equivalent to ApplicationController#my_lootbin
	assert lootbin
	assert lootbin.has_items?
	assert_equal lootbin.offer_items.size, items_in_bin+1
end

def login_user(user, password, integration_session)
	integration_session.https!
	integration_session.post url_for(:controller => 'sessions', :action => :create, :username => user.user_name, :password => password)
		
	assert_equal integration_session.session.data[:user_id], user.id
end

The title of this post is a Google query that yielded no good results, but plenty of puzzled users trying to figure out how to make it work. I’ve only been playing with it for about half an hour, but this code is getting Paypal to tell me that the request is successful:

def self.send_money(to_email, how_much_in_cents, options = {})
	credentials = {
	  "USER" => API_USERNAME,
	  "PWD" => API_PASSWORD,
	  "SIGNATURE" => API_SIGNATURE,
	}

	params = {
	  "METHOD" => "MassPay",
	  "CURRENCYCODE" => "USD",
	  "RECEIVERTYPE" => "EmailAddress",
	  "L_EMAIL0" => to_email,
	  "L_AMT0" => ((how_much_in_cents.to_i)/100.to_f).to_s,
	  "VERSION" => "51.0"
	}

	endpoint = RAILS_ENV == 'production' ? "https://api-3t.paypal.com" : "https://api-3t.sandbox.paypal.com"
	url = URI.parse(endpoint)
	http = Net::HTTP.new(url.host, url.port)
	http.use_ssl = true
	all_params = credentials.merge(params)
	stringified_params = all_params.collect { |tuple| "#{tuple.first}=#{CGI.escape(tuple.last)}" }.join("&")

	response = http.post("/nvp", stringified_params)
end

Certainly not the tersest solution, but I’ve kept it a bit verbose to make it clearer what’s happening.

One point of note is that you’ll need separate credentials when submitting to sandbox vs. production. You can sign up for a sandbox account by clicking “Sign up” from this page. After you have your account, login, click on “profile,” then get your API access credentials.

Here is the PHP example I based my code on and here is the Paypal Masspay NVP documentation that was marginally helpful in figuring out what params to pass.

The Problem

After adding our third app server a couple days ago, the appeal of digging through three separate production.log files when things go awry on Bonanzle was officially over.

Like many Rails developers in this situation, I Googled numerous terms in search of a solution, and most of these terms sent me to my good friend Jesse Proudman’s blog on using Syslog-ng to unify Rails logfiles. So we installed it (well, to be specific, we had Blue Box set it up for us, because it looked complicated), and determined that it was not what we were looking for. Installation issues aside (of which there were a few), the real killer when using Syslogger with Rails is that you lose the buffered production log output you have come to know and love, leaving your production logfiles a stew of mishmashed lines from numerous Passenger process in numerous states of processing. In short, if you get an appreciable amount of traffic (and I’d imagine you do if you’re reading this in the first place), and you are a human being, you will not be able to read an unbuffered Rails log without considerable time and frustration.

The Solution

Exists on Github here.

Since it looked like Syslog was the only game in town currently for merging application server logs, I decided to spend the afternoon writing a plugin that would allow us to take an arbitrary number of production logfiles, from an arbitrary number of hosts, and merge them together into one file without changing the formatting of the production logs or affecting performance on the app servers.

The basic mechanism of this plugin is that it uses rsync to grab your production logs, then it boils those production logs down into hashes of { :time => action_time, :text => action text }. It then outputs the actions from all of your app servers into a single file in chronological order.

As a bonus, it also lets you specify the maximum size of your unified log file, and handles keeping the logfiles broken into bite-sized chunks, so that you can actually read the output afterwards (rather than ending up with a 5GB log file). This functionality is built in, and can be configured via an included YML file.

The remainder of this post will just quote from the Github project, which does a pretty fine job of explaining what’s going on:

Savage Beast 2.0 has been the de facto solution for those looking to add a message forum to their existing Rails site, but it was created more than a year ago, and had many aspects that tied it to Rails 2.0. Also, it relied on the Engines plugin, which is not the most lightweight plugin. Although Engines doesn’t seem to affect performance, it did rub some people the wrong way.

After a year’s worth of promises that an update was “coming soon,” an update has finally arrived and is now available at Github.

Detailed instructions on getting it rolling with Rails 2.3 follow.

Installation

Currently, the following is necessary to use the Savage Beast plugin:

1. The Savage Beast 2.3 plugin. Go to your application’s root directory and:

   script/plugin install git://github.com/wbharding/savage-beast.git

2. Most of the stuff you need to run Beast…
   • Redcloth: gem install Redcloth. Make sure you add “config.gem 'RedCloth'” inside your environment.rb, so that it gets included.
   • A bunch of plugins (white_list, white_list_formatted_content, acts_as_list, gibberish, will_paginate). If you’re using Rails 2.2 or earlier, you’ll need the Engines plugin, if you’re on Rails 2.3, you don’t need Engines. The easiest way to install these en masse is just to copy the contents of savage_beast/tested_plugins to your standard Rails plugin directory (/vendor/plugins). If you already have versions of these plugins, you can just choose not to overwrite those versions

3. Go to your application’s root directory and run “rake savage_beast:bootstrap_db” to create the database tables used by Savage Beast. If it happens you already have tables in your project with the names Savage Beast wants to use, your tables won’t be overwritten (though obviously SB won’t work without its tables). To see the tables Savage Beast uses, look in lib/tasks/savage_beast.rake in your Savage Beast plugin folder.
4. Next run “rake savage_beast:bootstrap_assets” to copy Savage Beast stylesheets and images to savage_beast asset subdirectories within your public directory.
5. Implement in your User model the four methods in plugins/savage_beast/lib/savage_beast/user_init that are marked as "#implement in your user model“
6. Add the line “include SavageBeast::UserInit” to your User model. Location shouldn’t matter unless you intend to override it.
7. Add the line “include SavageBeast::ApplicationHelper” to ApplicationHelper within your application_helper.rb file.
8. Implement versions of the methods in SavageBeast::AuthenticationSystem (located in /plugins/savage_beast/lib) in your application controller if they aren’t already there (note: technically, I believe only “login_required” and “current_user” are necessary, the others give you more functionality). Helpful commenter Adam says that if you have the “helper :all” line in your application controller, be sure to add the “SavageBeast::AuthenticationSystem” line after that.

If you’re using Rails 2.0-2.2, and thus using the Engines plugin, you’ll need a couple extra steps:

1. Add this line to the top of your environment.rb, right after the require of boot: require File.join(File.dirname(__FILE__), '../vendor/plugins/engines/boot')
2. Move the routes.rb file from the “savage-beast/config” directory to the root (“savage-beast”) directory of the plugin. Then add the line “map.from_plugin :savage_beast” to your routes.rb. Location shouldn’t matter unless you intend to override it.

And off you go! When you visit your_site/forums something should happen. I’ve been creating new forums by visiting /forums/new. There’s probably a hidden admin view somewhere.

Implementing Your Own Views and Controllers

Just create a new file in your /controllers or /views directories with the same name as the file you want to override in Savage Beast. If you just want to override a particular method in a controller, you can do that piecemeal if you just leave your XController empty except for the method you wanted to override (Note: I know this piecemeal method adding works with the Engines plugin installed, but haven’t tested it without).

If you’re integrating this into an existing site, I’d recommend you start by creating a forums layout page (/app/views/layouts/forums.html.erb). This will give you a taste of how easy it is to selectively override files from the plugin.

Demo

You can check out a (slightly-but-not-too-modified) version of Savage Beast online at Bonanzle. The differences between our version and the version checked into Subversion are 1) addition of topic tagging (users can tag topics to get them removed, etc) 2) recent post list shows posts in unique topics, rather than showing posts from the same topic repeatedly (there’s another blog on here about the SQL I used to do that) and 3) skinning. None of those changes feel intrinsic to what SB is “supposed to do,” which is why they aren’t checked in.

Conclusion

Comments are most welcome. I’ll be checking in changes to the project as I find bugs and improvements in using it, but this is admittedly something I don’t have a lot of spare time to closely follow (see my other entries on the wonders of entrepreneurship). Hopefully others can contribute patches as they find time. If you like the plugin, feel free to stop by Agile Development and give it a rating so that others can find it in the future.