Re: Loading XSD file into memory ONCE

On Sun, Dec 9, 2012 at 8:05 AM, Dwayne Ghant <dghant1024@gmail.com> wrote:
> Hello All,
>
> I will be short and sweet. I have a simple (well at least I think it's
> simple) question. First let me explain, I'm writing a RESTful webservice
> that uses validates xml submissions using an xsd (440kb in size),
> pre-defined, schema. I would, idealistically, like to bootstrap the schema
> into memory so that it's not being requested every time the web service is
> requested. What's the best way to do this? A brief google search yielded
> the results below
> (http://stackoverflow.com/questions/11159077/python-load-2gb-of-text-file-to-memory):
>
> import mmap
>
> with open('dump.xml', 'rb') as f:
> # Size 0 will read the ENTIRE file into memory!
> m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) #File is open read-only
>
> # Proceed with your code here -- note the file is already in memory
> # so "readine" here will be as fast as could be
> data = m.readline()
> while data:
> # Do stuff
> data = m.readline()
>
>
> Will this work? Will this make the file only load once into memory?
>

No - not if you run that code on every (applicable) request. Ignore
the cost of reading it from disk - if you read a file from disk
repeatedly, even the worst OS should start to cache that file - the
actual cost is repeatedly re-parsing the XML document.

What you want to do is cache the resulting object, the parsed XML
document. An easy way to do this is by using a global:

__xsd_cache = None
def get_xsd():
global __xsd_cache
if __xsd_cache is None:
__xsd_cache = parse_xsd()
return __xsd_cache

def parse_xsd():
...

The only problem with caching is that eventually, what is cached is no
longer correct. You will need to think about how and when you will/can
invalidate the cache.

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.