Re: [Rails] Re: Threads and net/http: am I missing something?

jhaagmans a écrit, le 11/19/2009 03:34 PM :
> And yes, I'm still using BRB, I really can't think of a way to avoid
> using BRB. I need to query a few thousand hosts every hour, every day.
>
>

This translates to a few hosts each second. If I had this kind of load I
wouldn't use a background job scheduler but a queue manager, several
processes picking the requests from the queue and a custom-built
scheduler. In fact I'm doing this for several projects and I use
ActiveMessaging.

> I really regret choosing Ruby/Rails for this particular application at
> this point.
>

Ruby/Rails has not much to do with your problems. Choosing the right
tool for the job is the issue.
I've a server running a web spider hand-coded in Ruby using the tools I
described above and it successfully make 10s of thousands of HTTP
queries per hour. The only limit I hit is the RAM available as I use
several Ruby processes (which, by the way, internally use threads to
handle simultaneous HTTP HEAD/GET requests efficiently). If I had say
32G i could probably hit 100k requests per hour. Given I use a queue
manager, I could also add a second server and run queue processors on it
to get more capacity...

Lionel

--

You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=.