[Rails] Re: Parallelizing Tasks in the Background

Hi Phil,

Sidekiq sounds like the gem you need. It does background processing using threads so you get true parallelism. You would need an implementation of Ruby though that can support true parallelism like Rubinius or JRuby. Ruby MRI has the GIL so it can't do true multithreaded processing. 

Here are links for Sidekiq:

1. http://railscasts.com/episodes/366-sidekiq

2. https://github.com/mperham/sidekiq

Hope it helps!

Cheers,

Gabe

On Saturday, October 5, 2013 12:16:48 PM UTC+8, Phil wrote:

I'm experimenting with optimizing some areas that involve long processing times on some SQL selects (or any background task, really).  This is in Rails 4.0.0 with Ruby 2.0.0p247, PostgreSQL 9.2.4.

In this case, I can do intelligent dividing of selects to even the workload amongst N processes for the DB backend.

I currently am trying Spawnling, and this works for exactly one hit before I have to restart the web service:

  spawns = []

  ABC.each do # criteria for selects to divide load

   spawns << Spawnling.new do

     # portion of big SQL select here via ActiveRecord, and store in memcache

   end

  end

  # wait for all N blocks of code to finish running

  Spawnling.wait(spawns)

  # assemble result from memcache and deliver

  render ...

Works for one hit and then future hits log correctly but fail to deliver any content to the browser. There seems to be a number of issues with Spawnling anyways as it seems to be in disrepair.

How are folks parallelizing tasks these days for lots of CPUs per hit?  What's the Rails 4/Ruby 2 way to do it?

Thanks for any thoughts.

Phil

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/93705f7a-ce5f-444d-8574-7a5b9b56c9b5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.