Re: Designing the email processing system. What path should I follow?

On Fri, Feb 15, 2013 at 12:57 PM, Andre Lopes <lopes80andre@gmail.com> wrote:
> Hi all,
>
> I'm building a Django website. Currently I'm searching for the best
> way to design the email processing system.
>
> The website is a Ads site. I need to send emails in a regular basis to
> the users. I'm looking for 20 or 30 e-mails hour for now.
>
> In other projects, I've been using a database table to handle the
> e-mail processing. But when I started to develop in Django I have read
> some articles about Celery/RabbitMQ. What I want to know is if adding
> Celery/RabbitMQ to the e-mail processing system is a must or I adding
> this I will only add another layer of complexity.

The problem is that if you don't use Celery, what you end up doing is…

>
> The solution that I've designed in another project uses a database
> table and a cronjob running every 5 or 10 minutes to handle this. The
> Database table is very simple. Looks like this:
>
> [code]
> CREATE TABLE "atem_emails_envios" (
> "id_email_envio" int4 NOT NULL,
> "id_email_msg" varchar(20) NOT NULL,
> "dat_inserted" timestamp NOT NULL,
> "dat_sended" timestamp,
> "try_number" int4,
> "max_tries" int4 NOT NULL,
> "email_from" varchar(500) NOT NULL,
> "email_to" varchar(500) NOT NULL,
> "email_cc" varchar(500),
> "email_bcc" varchar(500),
> "email_subject" varchar(500) NOT NULL,
> "email_msg" text NOT NULL,
> "error_msg" text,
> "i_started" timestamp,
> "pid" int4,
> "coment" varchar(2000),
> "id_utiliz_ins" varchar(45),
> "id_utiliz_upd" varchar(45),
> "data_ult_actual" timestamp,
> PRIMARY KEY("id_email_envio"),
> CONSTRAINT "Ref_atem_emails_envios_to_atem_mensagens_email" FOREIGN
> KEY ("id_email_msg")
> REFERENCES "atem_mensagens_email"("id_email_msg")
> MATCH SIMPLE
> ON DELETE NO ACTION
> ON UPDATE NO ACTION
> NOT DEFERRABLE
> );
> [/code]
>
> When an e-mail is being processed I just store the PID to the table to
> avoid collisions.

… this.

>
> My question goes in this direction. I've been using this table to
> process e-mails in a website with low traffic, and works well. What
> advantages I have using a Queue manager like Celery and a broker like
> RabbitMQ? What benefits I will gain using a solution like
> Celery/RabbitMQ?
>

You get a robust queuing system for processing tasks asynchronously
that will be understood by anyone who has used celery. If they haven't
used celery, then they can learn it, and then on the next project that
requires asynchronous handling of tasks, they don't need to learn the
quirks and deficiencies of yet another hand rolled queueing system.

You get tight integration into Django.

You get tools for monitoring and maintaining your queues.

You get the comfort that hundreds of billions of messages have
successfully passed through the same setup.

You don't have to write and maintain a queueing system. You don't have
to subsequently expand it to do multiprocessing. You don't have to fix
the inevitable bugs!

You get a proper AMQP stack which heterogeneous components can
communicate with each other through, if you have the need. For
instance, actions on your django site could send AMQP messages to a
node.js instance instructing it to push data to a client over
websockets.

(I swear there is an awesome blog post that explains all of this
better than I have, but my google-fu is weak today apparently!)

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.