[Rails] Re: out of memory (java heap space) on zip creation (jruby)

On Wednesday, May 9, 2012 6:21:39 PM UTC-3, Jedrin wrote:

> The heap contains all the objects created for the application.. In this
> case, it looks like your file is still too big
>
> > Error occurred during initialization of VM
> > Could not reserve enough space for object heap
> > JVM creation failed
>
> This means that you tried to allocate more than is available on the machine
>
> Are you doing this for a single load, or will it be an application that
> will commonly receive large files?
>
> If it's the latter, I'd probably try to redesign the code you're using to
> load the files.  Sounds like this is part of a third party gem?  If that's
> the case, maybe they have some mechanism for handling larger files?
>
> --
> Greg Akinshttp://twitter.com/akinsgre

 What I do is create a csv file from the database. I had some memory
problems there, but using active record find_in_batches() seemed to
solve that.

The CSV file has 1.4 million rows. It gets created successfully. I
then use rubyzip gem to create a zip file that just contains that CSV
file. I just used examples I found from google searches on how to
create the zip file which are shown earlier up in the thread. I looked
at the class info on the web for rubyzip and didn't see an obvious way
to stream data into the zip file. Tomorrow I can look at perhaps some
other way to create a zip file using a different gem or some such ..

As I mentioned in my previous reply and similar to the problem you had when creating the file: you're trying to load the whole thing.

There are two options for this:

A) You stream the contents of your CSV file, reading by chunks into a ZipStream

or

B) You zip the file from outside Ruby (shelling out to gzip for example)

--

Luis Lavena

 

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/rubyonrails-talk/-/mwyK5VTPabEJ.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.