2009/01/16

Twitter on Merb : Merb flat app Walkthrough





Description


Motivation

I’m a newbie in merb community. Since “the merger”, I needed correct and reliable reference and so was other newbies in the community. I started to translate merb-book in Korean. But it was not enough for us .I thought that learning merb would be easier if I build an app what I really need such as tracking recent changes. It was easy since there are good tutorials and tools. And scraping nightly version of README file could be helpful too. Because merb core team members are too busy to write guides and tutorials.

I laid aside scraping the README files for now. I just aggregated merb commits rss and send them to twitter. I use TwitterFox as twitter client on my desktop. Whenever a new message comes, It shows the message in a brief moment. It's really handy.



Generating a flat app



This is what I did at first time. As you see, It is really small and shows basic structure how merb works. If you want to learn more about internals of merb, visit here(http://github.com/michaelklishin/merb-internals-handbook)


Using Model in a flat app

I edited config/framework.rb file. Because I could not use twitter_search gem to verify the message is unique or not. This is same as twitter timeline api. It seems that they use cache. Therefore I had to add a model to check the message I'm sending is unique or not.



First, I added one line :model => Merb.root / "models" in config/frameworks.rb file. And then, wrote dependencies in init.rb file.

$merb-gen model tweet



$rake db:automigrate


Scraping with nokogiri

Since merb flat app has a single controller file, It was easy to start. But the twitter api and nokogiri took some time to use. I was not familiar with nokogiri, I googled and found this tutorial. (http://www.robertsosinski.com/2008/12/08/scraping-pages-made-easy-with-ruby-and-nokogiri/).
And I better use a well designed web site template. I picked one of the recommended templates from samshmagazine.(http://www.smashingmagazine.com/2008/12/01/100-free-high-quality-xhtmlcss-templates/)

There were 3 or 4 ruby libraries for twitter. And twitter4r has more documents than the others. It took much time to discover that timeline api and search api use cache. I just forgot about the cache. Anyway I installed necessary gems.

$sudo gem install twitter4r twitter nokogiri shorturl

And then I eddited index.html.erb file. In the controller, it only needs render method. I just wanted to use feedtools and feedupdater gem. But there were not enough document to reference and as merb-book says, I better not use codes that I don't understand. Scraping pages with nokogiri is easy but my app would be fragile.



I know the codes above is not that beautiful. But I hope this code help another rubist who want a scraping example through nokogiri. As you see above, k.content which was return by nokogiri's search method was Hash. This is why i used to_s method.


Scheduling background works



And posting a message to twitter was tricky for me. Because of increasing popularity of twitter, posting throuth api was too fast for twitter sometimes. Posting messages takes much time and that code block needs to sleep. In the initial codes, I wrapped the block with run_later. But I found that I need to schedule the tasks. There would be several way to acheive scheduling the daemons. In version 0.0.1.4(Twitter-on-Merb), I used crontab and rake. But the codes were dirty and it did not work as I expected. I talked about this with lakteck via tiwtter, and I remembered that I've seen rufus-scheduler at igvita.com[link]. It was really easy to implement scheduler with rufus-scheduler gem. The scheduler scrapes web sites on every 10 minutes. And I made a starling consumer daemon. I referenced "Advanced Rails Recipes #42"

confing/init.rb


lib/daemons/starling_consumer.rb file


lib/daemons/starling_consumer_ctl.rb file


config/consumer.god file. Daemons are started after "cap:deploy".


Cache

My flat app was too slow, and it was time to implement cache. I referenced,
Here's quick introduction [original gist document link]

== Quick intro
With fragment caching, you can mix dynamic and static content.

With action caching, the whole template is cached
but the before filters are still processed.

With page caching, the whole template is put in html files in a special
directory in order to be handled directly without triggering Merb.

by Dan Kubb on July 22, 2008




But my init file was wrong at that time and throws erros whenever I run "rake db:automigrate" since I set up the cache(http://gist.github.com/44702). I really didn't know how to fix this. At first, I guessed that datamapper version could be the reason why causes it. After updating the gems, error were still there. Fortunately I have known Guillaume Maury(giom) recently via twitter. He helped to get out of this. This code was not familiar with me. I thought I knew the way to setup the cache.
The codes were already in his article (http://gom-jabbar.org/articles/2008/12/14/example-nginx-configuration-for-merb-with-page-caching-using-the-file-store). I run my apps on passenger so I simply skipped it. :) I highly recommend his blog if you are in Merb2 or Rails3. Here is the snippet written by him. (The initial flat app with plain dependency line did not work.)


And I referenced here.
http://blog.evanweaver.com/files/doc/fauna/memcached/files/README.html

$ wget http://download.tangent.org/libmemcached-0.25.tar.gz
$ tar -xzvf libmemcached-0.25.tar.gz
$ cd libmemcached-0.25.tar.gz
$ ./configure
$ make && sudo make install
$ sudo gem install memcached
$ irb
> require 'rubygems'
> require 'memcached'


Deploy

$cap deploy:setup
$scp -r database.yml to server
$scp -r twitter.yml to server

$rake db:create
$rake db:automigrate
$cap deploy

this is my deploy.rb file.

tip

And if you have problems after your deployment, check if config.ru(must be 'config.ru', not 'Config.ru') file exists and the permissions are correct(chmod 755 -R).
at 1:02 PM Labels:
blog comments powered by Disqus