Re: django cron job - stops after reading some portion of huge file - why is this?

One possibility is that your code keeps all that is read (or something derived from it) in memory, and you are running out.

E.g.; Is your database code trying to do all this in a single transaction?

Another possibility is that something in the file at that spot triggers a but in your code that contains an infinite loop.

There are other possibilities.  But there's no diagnosing it with the information you've given.

Can you, in python, read through the file, doing nothing with the data?  E.g.:

    f = open('your/file/path/here')

    n = 0

    s = True

    while s:

        s = f.read(1024*1024)

        n += len(s)

        print n

    print 'done'

That should work.  If not, does your O/S not correctly handle files that big?

Bill

On Tue, Oct 15, 2013 at 6:55 AM, doniyor <doniyor.v.j@gmail.com> wrote:

I am reading file from url and parsing it and saving some information out of this file into db - using cron job. 

i am testing now in my local dev. 

the problem is: job is reading file and saving into db without any problem but after some time, since file is very huge approx. >8GB, job doesnot do anything and freezes, without giving any error, 

i am using django 1.4, python 2.7 and postgresql. is there any limit for writing into db? why is it freezing? 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b567f273-508b-4f14-9d40-197af8c8a079%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAB%2BAj0u8Z8fRDJEhMmZkX8EnZaEz-%3Dwpw4Qk5FyFsSWeAQ70yg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.