Re: How many workers do you run on one machine using django celery?

RabbitMQ with Celery is a distributed asynchronous code execution service. Keyword Distributed. The best thing to do in run several instances per machine of djcelery behind a supervisord. When you start encountering performance hits start by adding more services on the machine with supervisord. You will hit a limit at some point where you wont see any increase in performance from this. It is at this point that i suggest you scale down the amount of services on that machine and add another  machine to the rabbit stack with the same configuration. RabbitMQ should handle the load balancing itself. However it is at this point i believe that logging will become more difficult. So be sure to have a solution in place for handling that. 

You should also look into batching your processes among tasks, if you are going to be running so many tasks, than cant a task hold the last 10 sms messages to be sent out and send all 10 of them. This will improve scalability as well.

On Saturday, April 20, 2013 7:28:55 PM UTC-4, sparky wrote:

Quick question, just so I can compare, I would really like to hear other devs experience.

How many workers do you run on one machine using django celery?

I'm running 3 workers on an EC2 small instance. It takes around 5 seconds to complete 1 task running all 3 workers, does this sound right to you?

My issue is I could have 100,000 tasks very soon... scale wise I'm unsure what I'm going to need to do this. Bigger CPU, RAM and X workers, 5 seconds is far too long for me. All the task is doing is sending a SMS message HTTP thats it.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.