Monday, January 31, 2011

Storing *big* texts in db fields

I've created an application to manage texts, storing the content in a
TextField, along with metadata in other fields. Now that I've
completed the model and started actually adding the texts, the admin
is getting verrrry slow. The app is just for the use of me and my
team, so the slowness is not a deal-breaker, but it's annoying to work
with and I still have a lot of texts to add to the corpus.

Although I may be adding a large amount of smaller texts in the
future, the texts that I have now are large, mostly in the tens of
thousands of words, with the largest currently at 101,399 words.
(Which I know because I added a method to the model to calculate the
wordcount, and have it displayed in the admin list. Which gives me no
end of pleasure.)

So, is it a bad idea to be storing texts this large in a database
field? I really hope not, because when I first started this project
(granted, before I started using Django), I was reading the data from
files and running into constant encoding/decoding problems. (These
texts I'm collecting are in Arabic.)

If it's not a totally horrible idea to do this like I'm doing, is
there anything I can do to improve performance? I tried implementing
caching and it didn't make any difference.

Thanks,
Karen

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

0 Comments:

Post a Comment

Subscribe to Post Comments [Atom]

<< Home


Real Estate