twitter: comp.lang.python - 26 new messages in 11 topics

comp.lang.python
http://groups.google.com/group/comp.lang.python?hl=en

comp.lang.python@googlegroups.com

Today's topics:

==============================================================================
TOPIC: Japanese (speaking) developer needed for a bit of regex magic
http://groups.google.com/group/comp.lang.python/t/1aa8c6029c8c3a1b?hl=en
==============================================================================

== 1 of 2 ==
Date: Wed, Apr 21 2010 12:07 am
From: Chris Rebert

On Tue, Apr 20, 2010 at 9:52 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
> Sebastian <basti@redtoad.de> writes:
>> All locales return error messages in English. Only the Japanese uses
>> Japanese which my regular expressions cannot handle at the moment.
>
> What exactly are you expecting to happen, and what exactly happens
> instead?
>
> General advice with character sets in Python apply: always explicitly
> declare the encoding of input, then decode to Unicode interally as early
> as possible, and process all text that way. Only fix into an encoding
> when it's time to output.

I think he has more of a *literal* language problem: He doesn't know
Japanese and thus can't read the Japanese error message in order to
develop a regex for it. I assume there's some reason he can't just do
a blind equality test on the error message string(s).

Cheers,
Chris
--
http://blog.rebertia.com

== 2 of 2 ==
Date: Wed, Apr 21 2010 2:31 am
From: Sebastian

> General advice with character sets in Python apply: always explicitly
> declare the encoding of input, then decode to Unicode interally as early
> as possible, and process all text that way. Only fix into an encoding
> when it's time to output.

Maybe I was too vague when describing my problem. As Chris correctly
guessed, I have a literal language problem.

> > All locales return error messages in English. Only the Japanese uses
> > Japanese which my regular expressions cannot handle at the moment.
>
> What exactly are you expecting to happen, and what exactly happens
> instead?

My regular expressions turn the Amazon error messages into Python
exceptions.

This works fine as long as they are in English: "??? is not a valid
value for BrowseNodeId. Please change this value and retry your
request.", for instance, will raise an InvalidParameterValue
exception. However, the Japanese version returns the error message "???
は、BrowseNodeIdの値として無効です。値を変更してから、再度リクエストを実行して
ください。" which will not be
successfully handled.

This renders the my module pretty much useless for Japanese users.

I'm was therefore wondering if someone with more knowledge of Japanese
than me can have a look at my expressions. Maybe the Japanese messages
are completely different...

I have a collection of sample messages here (all files *-jp-*.xml):
http://bitbucket.org/basti/python-amazon-product-api/src/tip/tests/2009-11-01/

Any help is appreciated!

Cheers,
Sebastian

==============================================================================
TOPIC: python glibc crypt() function
http://groups.google.com/group/comp.lang.python/t/02278fbd2dd1968a?hl=en
==============================================================================

== 1 of 1 ==
Date: Wed, Apr 21 2010 12:31 am
From: Peter Otten <__peter__@web.de>

luca72 wrote:

> On 20 Apr, 19:38, Peter Otten <__pete...@web.de> wrote:
>> luca72 wrote:
>> > Hello i have to do this :
>> > glibc crypt() function, using salt $1$abcdefgh$
>>
>> > cryptPw = crypt(plainPw, "$1$abcdefgh$")
> Thanks
> The result is correct i obtain the same with ctypes and crypt module,
> so i think that is better to use the crypt module is correct?

Yes, use the crypt module.

==============================================================================
TOPIC: how does a queue stop the thread?
http://groups.google.com/group/comp.lang.python/t/f8b231182c70bb92?hl=en
==============================================================================

== 1 of 5 ==
Date: Wed, Apr 21 2010 1:08 am
From: kaiix

A simple thread pool example. My question is, since *MyThread.run*
will loop endless, how does the thread know the time to quit? how does
the *queue* notify the thread? is there any shared variables, like a
*lock*?

When I set daemon false, it stays in the loop, not quit any more.
what's the role does the daemon state plays?

code:
----------------------------------------------------------------------------------------
import Queue
import threading

def do_some_thing(x):
print int(x)

class MyThread(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue

def run(self):
while True:
params = self.queue.get()
do_some_thing(params)
self.queue.task_done()

q = Queue.Queue()

for i in range(1, 5):
t = MyThread(q)
t.setDaemon(True)
t.start()

for x in range(10):
q.put(x)

q.join()

== 2 of 5 ==
Date: Wed, Apr 21 2010 1:20 am
From: Kushal Kumaran

On Wed, Apr 21, 2010 at 1:38 PM, kaiix <kvn.hou@gmail.com> wrote:
> A simple thread pool example. My question is, since *MyThread.run*
> will loop endless, how does the thread know the time to quit? how does
> the *queue* notify the thread? is there any shared variables, like a
> *lock*?
>
> When I set daemon false, it stays in the loop, not quit any more.
> what's the role does the daemon state plays?
>

Take a look at the documentation for the threading module here:
http://docs.python.org/release/2.6/library/threading.html, and the
discussion on daemon vs non-daemon threads here:
http://blog.doughellmann.com/2008/01/pymotw-threading_13.html

Basically, as long as at least one non-daemon thread is still running,
your program will not exit.

> <snip code>

--
regards,
kushal

== 3 of 5 ==
Date: Wed, Apr 21 2010 1:23 am
From: Kushal Kumaran

On Wed, Apr 21, 2010 at 1:50 PM, Kushal Kumaran
<kushal.kumaran+python@gmail.com> wrote:
> On Wed, Apr 21, 2010 at 1:38 PM, kaiix <kvn.hou@gmail.com> wrote:
>> A simple thread pool example. My question is, since *MyThread.run*
>> will loop endless, how does the thread know the time to quit? how does
>> the *queue* notify the thread? is there any shared variables, like a
>> *lock*?
>>
>> When I set daemon false, it stays in the loop, not quit any more.
>> what's the role does the daemon state plays?
>>
>
> Take a look at the documentation for the threading module here:
> http://docs.python.org/release/2.6/library/threading.html, and the
> discussion on daemon vs non-daemon threads here:
> http://blog.doughellmann.com/2008/01/pymotw-threading_13.html
>
> Basically, as long as at least one non-daemon thread is still running,
> your program will not exit.
>
>> <snip code>
>

And I should have read your mail better. Take a look at the
documentation of the task_done method:
http://docs.python.org/library/queue.html

--
regards,
kushal

== 4 of 5 ==
Date: Wed, Apr 21 2010 2:36 am
From: kaiix

@kushal, thanks for your replies.

before i wrote the email, i've already read the python docs carefully.
i need the proof from code, i mean python source code. i tried to
prove some of my assumptions that lead the loop quit, and i traced
back to Queue.py, threading.py, dummy_thread.py, now i need some hints
to help me understanding the sample from python source code.

== 5 of 5 ==
Date: Wed, Apr 21 2010 3:00 am
From: James Mills

On Wed, Apr 21, 2010 at 7:36 PM, kaiix <kvn.hou@gmail.com> wrote:
> before i wrote the email, i've already read the python docs carefully.
> i need the proof from code, i mean python source code. i tried to
> prove some of my assumptions that lead the loop quit, and i traced
> back to Queue.py, threading.py, dummy_thread.py, now i need some hints
> to help me understanding the sample from python source code.

You would need to look into the source code of
python's builtin 'thread' module which is the low-level
threading implementation used by python's 'threading'
module (which is a wrapper atop this).

'threading' aside from everything else it does, really in the
end calls thread.start_new_thread(...)

cheers
James

==============================================================================
TOPIC: error when printing a UTF-8 string (python 2.6.2)
http://groups.google.com/group/comp.lang.python/t/fddf1ad516252069?hl=en
==============================================================================

== 1 of 4 ==
Date: Wed, Apr 21 2010 1:09 am
From:

Hello.

I read a string from an utf-8 file:

fichierLaTeX = codecs.open(sys.argv[1], "r", "utf-8")
s = fichierLaTeX.read()
fichierLaTeX.close()

I can then print the string without error with 'print s'.

Next I parse this string:

def parser(s):
i = 0
while i < len(s):
if s[i:i+1] == '\\':
i += 1
if s[i:i+1] == '\\':
print "backslash"
elif s[i:i+1] == '%':
print "pourcentage"
else:
if estUnCaractere(s[i:i+1]):
motcle = ""
while estUnCaractere(s[i:i+1]):
motcle += s[i:i+1]
i += 1
print "mot-clé '"+motcle+"'"

but when I run this code, I get this error:

Traceback (most recent call last):
File "./versOO.py", line 115, in <module>
parser(s)
File "./versOO.py", line 105, in parser
print "mot-clé '"+motcle+"'"
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in
position 6: ordinal not in range(128)

What must I do to solve this?

Thanks!

--
Fabrice DELENTE

== 2 of 4 ==
Date: Wed, Apr 21 2010 1:38 am
From: Peter Otten <__peter__@web.de>

fab@slick.airforce-one.org wrote:

> Hello.
>
> I read a string from an utf-8 file:
>
> fichierLaTeX = codecs.open(sys.argv[1], "r", "utf-8")
> s = fichierLaTeX.read()
> fichierLaTeX.close()
>
> I can then print the string without error with 'print s'.
>
> Next I parse this string:
>
> def parser(s):
> i = 0
> while i < len(s):
> if s[i:i+1] == '\\':
> i += 1
> if s[i:i+1] == '\\':
> print "backslash"
> elif s[i:i+1] == '%':
> print "pourcentage"
> else:
> if estUnCaractere(s[i:i+1]):
> motcle = ""
> while estUnCaractere(s[i:i+1]):
> motcle += s[i:i+1]
> i += 1
> print "mot-clé '"+motcle+"'"
>
> but when I run this code, I get this error:
>
> Traceback (most recent call last):
> File "./versOO.py", line 115, in <module>
> parser(s)
> File "./versOO.py", line 105, in parser
> print "mot-clé '"+motcle+"'"
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in
> position 6: ordinal not in range(128)
>
> What must I do to solve this?

>>> "mot-clé" + "mot-clé"
'mot-cl\xc3\xa9mot-cl\xc3\xa9'

>>> u"mot-clé" + u"mot-clé"
u'mot-cl\xe9mot-cl\xe9'

>>> "mot-clé" + u"mot-clé"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6:
ordinal not in range(128)

codecs.open().read() returns unicode, but your literals are all bytestrings.
When you are mixing unicode and str Python tries to convert the bytestring
to unicode using the ascii codec, and of course fails for non-ascii
characters.

Change your string literals to unicode by adding the u-prefix and you should
be OK.

Peter

== 3 of 4 ==
Date: Wed, Apr 21 2010 2:29 am
From:

> Change your string literals to unicode by adding the u-prefix and you should
> be OK.

Thanks, it solved the problem... for a while!

I need now to know if s[i] gives the next byte or the next character,
when I scan the string s. I've googled pages about python and unicode,
but didn't find a solution to that. I scan the string read from the
file char by char to construct s, but now get the same error when just
trying 'print s'.

Is there a way to tell python that all strings and characters are to
be treated as UTF-8? I have LC_ALL=en_GB.utf-8 in my shell, but python
does'nt seem to use this variable?

Thanks!

--
Fabrice DELENTE

== 4 of 4 ==
Date: Wed, Apr 21 2010 2:37 am
From: Chris Rebert

On Wed, Apr 21, 2010 at 2:29 AM, <fab@slick.airforce-one.org> wrote:
>> Change your string literals to unicode by adding the u-prefix and you should
>> be OK.
>
> Thanks, it solved the problem... for a while!
>
> I need now to know if s[i] gives the next byte or the next character,
> when I scan the string s. I've googled pages about python and unicode,
> but didn't find a solution to that. I scan the string read from the
> file char by char to construct s, but now get the same error when just
> trying 'print s'.

Assuming s = fichierLaTeX.read() as from your code snippet, the next
character. When in doubt, check what `type(s)` is; if it's <type
'str'>, indices are in bytes; if it's <type 'unicode'>, indices are in
code points.

Please give the full stack traceback for your error.

Cheers,
Chris
--
http://blog.rebertia.com

==============================================================================
TOPIC: Operations on sparse matrices
http://groups.google.com/group/comp.lang.python/t/d6a3158ab1d4aba3?hl=en
==============================================================================

== 1 of 1 ==
Date: Wed, Apr 21 2010 1:14 am
From: Helmut Jarausch

On 04/19/10 08:03, pp wrote:
> I am currently dealing with sparse matrices and have doubts on whether
> we can use
> 1.) dot (for matrix multiplication) and inv (inverse) operations of
> numpy on sparse matrices of CSR format.
>

I don't know of any use of the inverse of a sparse matrix.
Note, in nearly all cases the inverse of a sparse matrix is a full matrix.
Instead of inverting a matrix solve a linear system with that matrix.
What do you need the inverse for?

Helmut.

--
Helmut Jarausch

Lehrstuhl fuer Numerische Mathematik
RWTH - Aachen University
D 52056 Aachen, Germany

==============================================================================
TOPIC: Write web apps in Python?
http://groups.google.com/group/comp.lang.python/t/7caab317a1d56d6e?hl=en
==============================================================================

== 1 of 3 ==
Date: Wed, Apr 21 2010 1:28 am
From: Bruno Desthuilliers

Bryan a écrit :
>
> I think I see what you mean

Err...

> -- correct me if I'm wrong:

You are, sorry !-)

> You want to
> keep complex application data structures around between requests.

Nope. I want to keep all my settings parsed, my librairies loaded, all
my connections opened etc. That is, all the time consuming stuff at app
startup - which, with PHP, mostly happens for each and every request.

>>> PHP frameworks generally allow and encourage application code
>>> to be independent of the underlying plumbing.
>> This is debatable at best. PHP code (except cli PHP code of course) is
>> written without any care for persistent global state, concurrency
>> issues, race conditions etc - because it's written with the idea that
>> the code serving a request will be runned in total isolation. CGI
>> heritage here, obviously.
>
> No, that's good web-app structure, regardless of language and server
> interface. If we keep persistent global state in a shared database
> rather than program variables,

Err... Did you really read what you're answering too ???

Also, I never said this execution model was necessarily bad - just that
it had pros *and* cons.

>> And please note I'm not criticizing this
>> design- just pointing one of it's consequences.
>>
>>> Many large,
>>> sophisticated, high-volume web apps are in PHP.
>> Did anyone pretend otherwise ?
>
> How about this howler: "The PHP execution model (mostly based on CGI
> FWIW) tends to be a bit unpractical for non-trivial applications".

"tends to be a bit unpractical" != "doesn't work".

Many large, sopĥisticated etc applications are written in C. Does that
make C a practical application programming language ?

Now I'm sorry to say that for quite a few "sophisticated" PHP apps I've
seen (and eventually had to work on), the "startup" part - parsing the
include files, configuration, establishing connections etc - took a good
part of the total processing time.

== 2 of 3 ==
Date: Wed, Apr 21 2010 2:33 am
From: Adam Tauno Williams

On Wed, 2010-04-21 at 10:28 +0200, Bruno Desthuilliers wrote:
> Bryan a écrit :
> >
> > I think I see what you mean
>
> Err...
>
> > -- correct me if I'm wrong:
>
> You are, sorry !-)
>
> > You want to
> > keep complex application data structures around between requests.
>
> Nope. I want to keep all my settings parsed,

Store them in the session.
> my librairies loaded,

Enable APC

> all my connections opened etc.

If you are talking about RDBMS connections, use persistent connections.

Then you have everything you've asked for.

> Now I'm sorry to say that for quite a few "sophisticated" PHP apps I've
> seen (and eventually had to work on), the "startup" part - parsing the
> include files, configuration, establishing connections etc - took a good
> part of the total processing time.

== 3 of 3 ==
Date: Wed, Apr 21 2010 2:40 am
From: Paul Rubin

Bruno Desthuilliers <bruno.42.desthuilliers@websiteburo.invalid> writes:
> Nope. I want to keep all my settings parsed, my librairies loaded, all
> my connections opened etc. That is, all the time consuming stuff at
> app startup - which, with PHP, mostly happens for each and every
> request.

I thought we have WSGI for this? Nothing stops a Python app from
working like PHP. PHP has an advantage when you want to run mutually
hostile apps in the same process (relevant if you're running ultra-cheap
shared hosting and you want to put 1000 customers' virtual hosts in the
same mod_php instance), but I don't think you're looking to do that.

==============================================================================
TOPIC: Is it better to extend a class, or do something repetitious in the main
part of a program?
http://groups.google.com/group/comp.lang.python/t/b737c577429b451a?hl=en
==============================================================================

== 1 of 1 ==
Date: Wed, Apr 21 2010 1:34 am
From: Vinay Sajip

On Apr 19, 3:45 pm, J <dreadpiratej...@gmail.com> wrote:
> First, before I get farther,
>
> Is there a way for theloggingmodule to natively handle lists and
> dict objects whenlogging?
>
> e.g. take this {'key1':'val1','key2':'val2'} and have it logged like this:
>
> INFO: key1: val1
> INFO: key2: val2
>
> If I pass the dict or list directly to the logger, it is logged the
> same as if you simply did this:
>
> mydict={1:1, 2:2}
> mylist=[1,2,3]
>
> print mydict
> print mylist
>
> >>> {1:1,2:2}
> >>> [1,2,3]
>
> It came up that I wanted to haveloggingpresent command line options
> from optparse if the log level was set to debug... so they'd look
> something like this in the output:
>
> debug: True
> sleep_time: 30
> log_file: /tmp/testlog
>
> So the options I've managed to work out are to either parse the list
> or dict object item by item and feed those items one at a time into
> the logger:
>
> for i in mylist:
> logging.info(i)
>
> Or to extend the StreamHandler class to handle this by creating a new
> report.msg...
>
> Then the discussion came up: which is better? To parse a dictionary
> or list in the main code and pass each item to the logger one at a
> time, or to extend the logger to handle it natively?
>
> Any thoughts on which is the more proper way to handle cases like this?

Since you want to have a specific output format for the logged
information, the best way to go would be to write your own Formatter
subclass, and check in its format() method whether you have lists/
dicts and then format them however you want into a string, and return
the appropriately formatted string from that method.

Regards,

Vinay Sajip

==============================================================================
TOPIC: rfind bug ?
http://groups.google.com/group/comp.lang.python/t/d03fbdcacfbf5794?hl=en
==============================================================================

== 1 of 6 ==
Date: Wed, Apr 21 2010 1:56 am
From: Chris Rebert

On Wed, Apr 21, 2010 at 1:51 AM, Stef Mientki <stef.mientki@gmail.com> wrote:
> With the following code, I would expect a result of 5 !!
>
>>>> a= 'word1 word2 word3'
>>>> a.rfind(' ',7)
> 11
>
> Is this a bug ?

No. Don't you think someone would have found such an obvious bug by now?

You want regular str.find(), which searches from left-to-right:
>>>> a= 'word1 word2 word3'
>>> a.find(' ')
5

str.rfind() is a variant of str.find() that searches from
right-to-left. The "r" is for "right".

Cheers,
Chris
--
http://blog.rebertia.com

== 2 of 6 ==
Date: Wed, Apr 21 2010 2:06 am
From: James Mills

On Wed, Apr 21, 2010 at 6:51 PM, Stef Mientki <stef.mientki@gmail.com> wrote:
>
> With the following code, I would expect a result of 5 !!
>
>>>> a= 'word1 word2 word3'
>>>> a.rfind(' ',7)
> 11
>
> Is this a bug ?

Python's documentation states:

"Return the highest index in S"

I haven't looked at python's source code for the
str object, but perhaps this is exactly what it's
algorithm does!

cheers
James

== 3 of 6 ==
Date: Wed, Apr 21 2010 2:15 am
From: Peter Otten <__peter__@web.de>

Chris Rebert wrote:

[didn't see the original message]

> On Wed, Apr 21, 2010 at 1:51 AM, Stef Mientki <stef.mientki@gmail.com>
> wrote:
>> With the following code, I would expect a result of 5 !!
>>
>>>>> a= 'word1 word2 word3'
>>>>> a.rfind(' ',7)
>> 11
>>
>> Is this a bug ?
>
> No. Don't you think someone would have found such an obvious bug by now?

Indeed.

OP: you may be looking for

>>> a = "a bb ccc"
>>> a[::-1].find(" ")
3

Peter

== 4 of 6 ==
Date: Wed, Apr 21 2010 2:42 am
From: Paul Rudin

Peter Otten <__peter__@web.de> writes:

> OP: you may be looking for
>
>>>> a = "a bb ccc"
>>>> a[::-1].find(" ")
> 3

But you should be aware of the effeciency implications of doing
this. a[::-1] constructs a new list. It's probably faster to do e.g.:
len(a) - a.rfind(..) - 1

== 5 of 6 ==
Date: Wed, Apr 21 2010 2:54 am
From: Peter Otten <__peter__@web.de>

Paul Rudin wrote:

> Peter Otten <__peter__@web.de> writes:
>
>
>> OP: you may be looking for
>>
>>>>> a = "a bb ccc"
>>>>> a[::-1].find(" ")
>> 3
>
>
> But you should be aware of the effeciency implications of doing
> this. a[::-1] constructs a new list.

A new string, yes.

> It's probably faster to do e.g.:
> len(a) - a.rfind(..) - 1

Yes, especially if the string is "long":

$ python -m timeit -s'a = "word1 word2 word3"' 'a[::-1].rfind(" ")'
1000000 loops, best of 3: 0.834 usec per loop
$ python -m timeit -s'a = "word1 word2 word3"*100' 'a[::-1].rfind(" ")'
100000 loops, best of 3: 5.04 usec per loop

$ python -m timeit -s'a = "word1 word2 word3"' 'len(a)-a.rfind(" ")-1'
1000000 loops, best of 3: 0.587 usec per loop
$ python -m timeit -s'a = "word1 word2 word3"*100' 'len(a)-a.rfind(" ")-1'
1000000 loops, best of 3: 0.592 usec per loop

But be aware of the following difference:

>>> a[::-1].rfind("not there")
-1

>>> len(a) - a.rfind("not there") -1
8

Peter

== 6 of 6 ==
Date: Wed, Apr 21 2010 2:59 am
From: Stef Mientki

On 21-04-2010 10:56, Chris Rebert wrote:
> On Wed, Apr 21, 2010 at 1:51 AM, Stef Mientki <stef.mientki@gmail.com> wrote:
>
>> With the following code, I would expect a result of 5 !!
>>
>>
>>>>> a= 'word1 word2 word3'
>>>>> a.rfind(' ',7)
>>>>>
>> 11
>>
>> Is this a bug ?
>>
> No. Don't you think someone would have found such an obvious bug by now?
>
if it's not a bug,
then the start index has no meaning ...
... and some would call that a bug.

cheers,
Stef

==============================================================================
TOPIC: Code redundancy
http://groups.google.com/group/comp.lang.python/t/00286d4686672e21?hl=en
==============================================================================

== 1 of 1 ==
Date: Wed, Apr 21 2010 2:38 am
From: Jean-Michel Pichavant

Alan Harris-Reid wrote:
> Jean-Michel Pichavant wrote:
>> Alan Harris-Reid wrote:
>>> Hi,
>>>
>>> During my Python (3.1) programming I often find myself having to
>>> repeat code such as...
>>>
>>> class1.attr1 = 1
>>> class1.attr2 = 2
>>> class1.attr3 = 3
>>> class1.attr4 = 4
>>> etc.
>>>
>>> Is there any way to achieve the same result without having to repeat
>>> the class1 prefix? Before Python my previous main language was
>>> Visual Foxpro, which had the syntax...
>>>
>>> with class1
>>> .attr1 = 1
>>> .attr2 = 2
>>> .attr3 = 3
>>> .attr4 = 4
>>> etc.
>>> endwith
>>>
>>> Is there any equivalent to this in Python?
>>>
>>> Any help would be appreciated.
>>>
>>> Alan Harris-Reid
>> Hello,
>>
>> Use an effective text editor, repeating stuff should not be a
>> problem. In a more general manner, avoid trying to speed your writing
>> while you should care speeding the reading.
>> Most of the tricks you could use will confuse the reader (unless the
>> reader is familiar with Visual foxpro).
>>
>> Anyway,
>>
>> for attrName, value in [
>> ('attr1', 1),
>> ('attr2', 2),
>> ('attr3', 3),
>> ]:
>> setattr(class1, attrName, value)
>>
>> or
>>
>> class Foo:
>> def __init__(self):
>> self.attr1=None
>> self.attr2=None
>> self.attr3=None
>>
>> def set(self, *args, **kwargs):
>> for k in kwargs:
>> if hasattr(self, k):
>> setattr(self, k, kwargs[k])
>> else:
>> raise AttributeError('%s instance has no attribute "%s"' %
>> (self.__class__.__name__, k))
>>
>> f = Foo()
>> f.set(attr1=25)
>> print f.__dict__
>> f.set(attr3=4, attr2=89)
>> print f.__dict__
>> f.set(bar= 8)
>>
>> output:
>> {'attr2': None, 'attr3': None, 'attr1': 25}
>> {'attr2': 89, 'attr3': 4, 'attr1': 25}
>> AttributeError: Foo instance has no attribute "bar"
>>
>>
>> JM
>>
> Hi Jean-Michel,
>
> Interesting solutions, but I think for the effort involved (and
> readability) I'll stick to repeating the class.
>
> Regards,
> Alan
that's the way to go :)

==============================================================================
TOPIC: deleting objects present in a list
http://groups.google.com/group/comp.lang.python/t/9fb32392d01aa581?hl=en
==============================================================================

== 1 of 1 ==
Date: Wed, Apr 21 2010 2:48 am
From: Dave Angel

(For some reason you posted your response before the message you were
replying to. That's called Top-posting, and is bad form on these
mailing lists)

Sandy wrote:
> Thanks for the replies.
>
> Terry,
> What does 'immediately' mean? I did a small test and here are the
> results.
>
> import psutil
>
> def testing():
> class Object():
> pass
>
> l =}
> apm =sutil.avail_phymem()/(1024*1024)
> print 'Before creating objs: ' + repr(apm)
>
> for i in xrange(500000):
> l.update({Object():1})
>
> apm =sutil.avail_phymem()/(1024*1024)
> print 'After creating objs: ' + repr(apm)
> return l
>
> def hello():
> myl =esting()
>
> apm =sutil.avail_phymem()/(1024*1024)
> print 'Before deleting: ' + repr(apm)
>
> del myl
>
> # Here I want to delete the objects in the list
> # deleting myl doesn't seem to change the memory
>
> apm =sutil.avail_phymem()/(1024*1024)
> print 'After deleting: ' + repr(apm)
>
>
> if __name__ ='__main__':
> hello()
>
> OUTPUT:
> Before creating objs: 2516L
> After creating objs: 2418L
> Before deleting: 2418L
> After deleting: 2430L
>
> In my original case the memory is not getting released even after long
> time.
>
> - dksr
>
>
> On Apr 20, 8:44 pm, Terry Reedy <tjre...@udel.edu> wrote:
>
>> On 4/20/2010 3:21 PM, Sandy wrote:
>>
>>
>>> Hi all,
>>> I have large number of objects created and to handle them properly, I
>>> store them in a list. How can I delete all of these objects (delete I
>>> mean here is to remove the object from memory not just from list)?
>>> I cannot use the list to iterate through the objects to delete them.
>>> Because 'del' only reduces the reference count and as it is present in
>>> the list it is not deleted. I cannot delete the list because I loose
>>> control over the objects.
>>>
>> Deleting the list is the best you can do. If that deletes the last
>> reference, then the interpreter will delete the object when it feels
>> like it. For *current* CPython, this will be immediately. For other
>> implementations, whenever.
>>
>
>
First, you're using some 3rd party library for measuring some kind of
memory usage. I'd guess you're probably using
http://code.google.com/p/psutil/ Since I'm not familiar with
how they derive these numbers, I can only make a good guess as to how
valid they are. And if avail_phymem refers to what its name implies,
it has little to do with Python. Python doesn't control physical
memory, only virtual.

So let's skip those numbers and talk about what CPython actually does.
As others have pointed out, other implementations will be different.

When you delete a large number of objects (which you can do with
myl=None), CPython may keep some of the memory under its own control for
reuse. A future object of the same size will fit nicely in the hole,
and that may be faster than calling free() and malloc() again.

When CPython calls free(), the C runtime library almost certainly keeps
the memory for reuse by subsequent calls to malloc(). Operating system
calls for allocating and freeing memory are (usually) done in larger,
fixed-size blocks. In Windows for example, the granularity is 4k or 64k
for the more efficient methods of memory allocation. Swapfile
allocation, for example, is always in 4k multiples. See function call
VirtualAlloc(). Anyway, if there's even a single allocated byte in a
block, it can't release the block. And searching for such blocks is slow.

When the operating system is told to free something, it usually does not
"free" physical memory immediately. In the case of Windows, it marks
the block as available, and eventually a daemon task will zero it. But
it could very well be "charged to" the current process until some other
process needs the physical memory. What it does do is free it from
virtual memory. But notice that Virtual memory is represented by a
swapfile on disk, and Windows doesn't support a file with gaps in it
(sparse allocation). So unless this particular allocation is at the end
of the file, the size isn't likely to go down.

If you really want to "observe" that the memory has been "released,"
I'd suggest calling a similar testing() function a second time, with
objects of the same size, and see whether the numbers get any worse.
I'd say they won't, at least not by much, assuming there's any validity
to this avail_phymem() function.

I also have to point out that using "del myl" is not necessary for
freeing up the memory. All that's necessary is for the myl name to stop
referring to the list, and the list will go away. So myl=42 will work
just as well.

HTH
DaveA

==============================================================================
TOPIC: shove does not store data as expected
http://groups.google.com/group/comp.lang.python/t/0f10299bbb0fb5e8?hl=en
==============================================================================

== 1 of 1 ==
Date: Wed, Apr 21 2010 2:51 am
From: Alex

Dear all,
I'm trying to use the shove module (http://pypi.python.org/pypi/shove)
for a simple script. The script read a CSV file ad store the data.
When I check the content of the "store" object (instance of Shove)
*before* I close it, the data are all there but when I close and re-
open it some data are lost. How it is possible? There is something
wrong in my code or I didn't understand how shove works?

Thanks in advance.

Here is a sample of my code:

DBNAME = "file://store.db"

csv_file ="test.csv"
cities = csv.reader(open(csv_file), delimiter=";")
store = Shove(DBNAME, compress=True)
for city,region,code in cities:
entry_list = store.setdefault(region, [])
data = 'just a sample'
entry = {city:data}
entry_list.append(entry)

print "----Before closing----"
for k,v in store.iteritems():
print "Key->", k
print "Value->", v
print "----"
store.close()

store = Shove(DBNAME, compress=True)
print "----After closing and re-opening----"
for k,v in store.iteritems():
print "Key->", k
print "Value->", v
print "----"

Here is the output

----Before closing----
Key-> Marche
Value-> [{'Ancona': 'just a sample'}, {'Ascoli Piceno': 'just a
sample'}]
Key-> Piemonte
Value-> [{'Alessandria': 'just a sample'}, {'Asti': 'just a sample'}]
Key-> Puglia
Value-> [{'Bari': 'just a sample'}, {'Barletta-Andria-Trani': 'just a
sample'}]
Key-> Valle d'Aosta
Value-> [{'Aosta': 'just a sample'}]
----
----After closing and re-opening----
Key-> Marche
Value-> [{'Ancona': 'just a sample'}]
Key-> Piemonte
Value-> [{'Alessandria': 'just a sample'}]
Key-> Puglia
Value-> [{'Bari': 'just a sample'}, {'Barletta-Andria-Trani': 'just a
sample'}]
Key-> Valle d'Aosta
Value-> [{'Aosta': 'just a sample'}]
----

Here is the content of the CSV file:

Alessandria;Piemonte;AL
Ancona;Marche;AN
Aosta;Valle d'Aosta;AO
Ascoli Piceno;Marche;AP
Asti;Piemonte;AT
Bari;Puglia;BA
Barletta-Andria-Trani;Puglia;BT

==============================================================================

You received this message because you are subscribed to the Google Groups "comp.lang.python"
group.

To post to this group, visit http://groups.google.com/group/comp.lang.python?hl=en

To unsubscribe from this group, send email to comp.lang.python+unsubscribe@googlegroups.com

To change the way you get mail from this group, visit:
http://groups.google.com/group/comp.lang.python/subscribe?hl=en

To report abuse, send email explaining the problem to abuse@googlegroups.com

==============================================================================
Google Groups: http://groups.google.com/?hl=en

twitter

Wednesday, April 21, 2010

comp.lang.python - 26 new messages in 11 topics - digest

0 Comments:

Post a Comment

About Me

Previous Posts