As detailed here: , a bunch of people have problems with Hpricot core dumping with this sig:
/Library/Ruby/Gems/1.8/gems/hpricot-0.8.1/lib/hpricot/parse.rb:33: [BUG] Bus Error ruby 1.8.6 (2007-09-24) [universal-darwin9.0]
The problem is exactly that the fetched page is 16384 bytes long, and so one workaround is the following:
f = open(url)
f.write(" ") if f.size == 16384
doc = Hpricot(f)
Whee. Cheezits are us, but it works.
The other thing that works is to increase the buffer size to a point where Hpricot doesn’t care:
Hpricot.buffer_size = 262144