[Rails] Reducing downtime

We have our RoR apps on cloud servers that can be blown away and rebuilt with the aid of chef. The only volatile data on the server is the database which we backup periodically and pull off to our backup server.

If we backup the database once a day then should things go wrong and we restore the database from backup we will be, on average, 12 hours out of date. At worst (if the system crashes just before the backup is made) we could be 24 hours out of date.

To reduce the time we lose when we restore a backup we could of course just take more frequent backups. But this would start to become a performance issue, not to mention pulling the backups off the server will start to eat into our bandwidth.

The final additional problem is that we are running >25 different applications, each with their own database (some small, some huge). We tried replication but we seemed to need a real monster of a server to replicate to than we could afford and we can't afford to run >25 duplicate servers either.

Has anyone tackled this problem?

I was imaging an extension to active record that created a journal / log file of changes such as:

UPDATE Event#1234 start_time="2013-07-03"

UPDATE Event#3453 status="completed", finished_at="2013-07-03 12:15:00"

DELETE Ticket#2222

CREATE Ticket#3453 status="opened", owner="fred"

Which could be rerun once the database had been updated to catch up with the changes that happened since the backup was made.

I am assuming that any idea that I have someone else has already had and probably implemented (unless it's completely insane - in which case some indication as to why it wont work would be nice).

Any thoughts?

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CALxYQy5hQxdqqFXEKSi0%2BnyF2PvDMisJr14H8Q-DeTwBQNp-QQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.