Re: django cron job - stops after reading some portion of huge file - why is this?

yes, db code is doin all these calls in single transaction, i mean, i am not using transactions, may be this is the reason? 

this is my cron code: http://pastebin.com/Lrym1z8E i know, very ugly code, it is saving at least some objects into db

this is url where i am reading my xml file from: http://www.bookatable.com/feed.aspx?channel=524&secret=10F70A84-76D7-C56C-2350-779A1A849819&language=de-DE&type=restaurants

also i noticed now that in db, there are objects whose some fields are not fully filled out even if the xml file does have those information. it means, this is a transaction issue, right? 

could you please take a look at the code? would transaction solve this issue? 

On Tuesday, October 15, 2013 3:40:42 PM UTC+2, ke1g wrote:

One possibility is that your code keeps all that is read (or something derived from it) in memory, and you are running out.

E.g.; Is your database code trying to do all this in a single transaction?

Another possibility is that something in the file at that spot triggers a but in your code that contains an infinite loop.

There are other possibilities.  But there's no diagnosing it with the information you've given.

Can you, in python, read through the file, doing nothing with the data?  E.g.:

    f = open('your/file/path/here')

    n = 0

    s = True

    while s:

        s = f.read(1024*1024)

        n += len(s)

        print n

    print 'done'

That should work.  If not, does your O/S not correctly handle files that big?

Bill

On Tue, Oct 15, 2013 at 6:55 AM, doniyor <doniy...@gmail.com> wrote:

I am reading file from url and parsing it and saving some information out of this file into db - using cron job. 

i am testing now in my local dev. 

the problem is: job is reading file and saving into db without any problem but after some time, since file is very huge approx. >8GB, job doesnot do anything and freezes, without giving any error, 

i am using django 1.4, python 2.7 and postgresql. is there any limit for writing into db? why is it freezing? 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b567f273-508b-4f14-9d40-197af8c8a079%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b61fec61-7481-4113-ab8c-31b7143df3f5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.