twitter: comp.lang.python - 25 new messages in 12 topics

comp.lang.python
http://groups.google.com/group/comp.lang.python?hl=en

comp.lang.python@googlegroups.com

Today's topics:

==============================================================================
TOPIC: shelve.open generates (22, 'Invalid argument') Os X 10.5 with Python 2.
5
http://groups.google.com/group/comp.lang.python/t/b415a047f9b8c570?hl=en
==============================================================================

== 1 of 1 ==
Date: Sun, Feb 14 2010 10:21 am
From: aahz@pythoncraft.com (Aahz)

In article <7a9d26a8-0a9f-4bf3-bf50-0ac5e337f482@r24g2000yqd.googlegroups.com>,
seth <hklasky@gmail.com> wrote:
>
>We have a code that creates a simple Python shelve database. We are
>able to serialize objects and store them in the dbm file. This seem to
>work the same on Windows XP Python 2.5, Ubuntu 9.1 with Python 2.6,
>but on Os X 10.5 with Python 2.5 the database filename is changed by
>the operating system by attaching the .db extension to it. Does an one
>know why is that?

Nope -- any reason you can't change the filename?
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

"At Resolver we've found it useful to short-circuit any doubt and just
refer to comments in code as 'lies'. :-)"

==============================================================================
TOPIC: Please help with MemoryError
http://groups.google.com/group/comp.lang.python/t/6ed1981bc9821443?hl=en
==============================================================================

== 1 of 3 ==
Date: Sun, Feb 14 2010 10:32 am
From: Steve Holden

rantingrick wrote:
> On Feb 12, 4:10 pm, Steve Holden <st...@holdenweb.com> wrote:
>> Antoine Pitrou wrote:
>>> Le Fri, 12 Feb 2010 17:14:57 +0000, Steven D'Aprano a écrit :
>
> On Feb 12, 4:10 pm, Steve Holden <st...@holdenweb.com> wrote:
>> Antoine Pitrou wrote:
>>> Le Fri, 12 Feb 2010 17:14:57 +0000, Steven D'Aprano a écrit :
>
> Steve,
> Why do so many of your posts come in as doubles and triples. Is this a
> case of "studdering click finger" of some fault of your newsreader?
>
> -- concerned fellow pythonista...

I suspect it's because I am being insufficiently disciplined about using
"reply" instead of "reply all". On gmane they only appear once, so I
suspect some other component of the mail/news loop is less assiduous
about de-duping the content.

How do you read the list?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/

== 2 of 3 ==
Date: Sun, Feb 14 2010 10:48 am
From: Steve Howell

On Feb 14, 10:32 am, Steve Holden <st...@holdenweb.com> wrote:
> rantingrick wrote:
> > On Feb 12, 4:10 pm, Steve Holden <st...@holdenweb.com> wrote:
> >> Antoine Pitrou wrote:
> >>> Le Fri, 12 Feb 2010 17:14:57 +0000, Steven D'Aprano a écrit :
>
> > On Feb 12, 4:10 pm, Steve Holden <st...@holdenweb.com> wrote:
> >> Antoine Pitrou wrote:
> >>> Le Fri, 12 Feb 2010 17:14:57 +0000, Steven D'Aprano a écrit :
>
> > Steve,
> > Why do so many of your posts come in as doubles and triples. Is this a
> > case of "studdering click finger" of some fault of your newsreader?
>
> > -- concerned fellow pythonista...
>
> I suspect it's because I am being insufficiently disciplined about using
> "reply" instead of "reply all". On gmane they only appear once, so I
> suspect some other component of the mail/news loop is less assiduous
> about de-duping the content.
>
> How do you read the list?
>

Hi Steve, I mostly read the list through Google groups, and I have
seen occasional dups from you. Of course, the readers themselves
could be a little smarter about recognizing duplication, but that does
not appear to be the case with Google. On the other hand, I am not
seeing dups from you in a quick skim of this thread, so I wonder if it
just takes Google a little while to weed out the dups.

== 3 of 3 ==
Date: Sun, Feb 14 2010 11:04 am
From: Steve Howell

On Feb 11, 5:50 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
> On Thu, 11 Feb 2010 15:39:09 -0800, Jeremy wrote:
> > My Python program now consumes over 2 GB of memory and then I get a
> > MemoryError. I know I am reading lots of files into memory, but not 2GB
> > worth.

> > 2. When do I need
> > to manually allocate/deallocate memory and when can I trust Python to
> > take care of it?
>
> You never need to manually allocate memory.
>
> You *may* need to deallocate memory if you make "reference loops", where
> one object refers to itself:
>
> l = [] # make an empty list
> l.append(l) # add the list l to itself
>
> Python can break such simple reference loops itself, but for more
> complicated ones, you may need to break them yourself:
>
> a = []
> b = {2: a}
> c = (None, b)
> d = [1, 'z', c]
> a.append(d) # a reference loop
>
> Python will deallocate objects when they are no longer in use. They are
> always considered in use any time you have them assigned to a name, or in
> a list or dict or other structure which is in use.
>
> You can explicitly remove a name with the del command. For example:
>
> x = ['my', 'data']
> del x
>
> After deleting the name x, the list object itself is no longer in use
> anywhere and Python will deallocate it. But consider:
>
> x = ['my', 'data']
> y = x # y now refers to THE SAME list object
> del x
>
> Although you have deleted the name x, the list object is still bound to
> the name y, and so Python will *not* deallocate the list.
>
> Likewise:
>
> x = ['my', 'data']
> y = [None, 1, x, 'hello world']
> del x
>
> Although now the list isn't bound to a name, it is inside another list,
> and so Python will not deallocate it.
>

Another technique that comes up some time is that you have a list of
objects that you are processing:

x = [obj1, obj2, obj3, obj4]

When you are done processing obj1, you want to remove the reference to
it, but you do not necessarily want to change the list itself. You
can break the reference by saying "x[0] = None" when you are done
handling obj1. Of course, if you can avoid creating the list in the
first place, as some people have suggested, then you really get a
savings.

The setting-to-None technique is also occasionally useful with
objects, where you can say foo.bar = None when you are done with "bar"
but not with "foo." Of course, the need to use such a technique often
points out a deeper code smell with Foo itself, but I've seen it come
up.

Steven's examples of a reference loop are deliberately simplified, of
course, but the chain of references in a real world program can get
quite long, and there's often great savings to be reaped if you can
break the "keystone" reference, so to speak. In other words, breaking
just one reference often allows other references to fall down like
dominos.

==============================================================================
TOPIC: Python Optimization
http://groups.google.com/group/comp.lang.python/t/df0c3e1a94d5bd65?hl=en
==============================================================================

== 1 of 4 ==
Date: Sun, Feb 14 2010 10:41 am
From: Steve Howell

On Feb 14, 9:48 am, Mark Dickinson <dicki...@gmail.com> wrote:
> On Feb 14, 4:53 pm, mukesh tiwari <mukeshtiwari.ii...@gmail.com>
> wrote:
>
> > Hello everyone. I am new to python and previously i did programming in
> > c/c++.Could some one please help me to improve the run time for this
> > python program as i don't have idea how to optimized this code.
> > [...]
>
> How much of a speedup do you need? Are we talking about orders of
> magnitude (in which case you might want to consider using something
> like the Multiple Polynomial Quadratic Sieve method instead, or as
> well), or just a few percent?
>
> (1) Have you profiled the code to see where it's spending most of its
> time? This is an essential first step.
>

I ditto the profiling recommendation.

http://docs.python.org/library/profile.html

It might also be useful to time your algorithm for n=10, 100, 1000,
10000, etc., to get a better sense of how the overall algorithm
behaves.

== 2 of 4 ==
Date: Sun, Feb 14 2010 11:35 am
From: Mark Dickinson

On Feb 14, 6:03 pm, a...@pythoncraft.com (Aahz) wrote:
> In article <363498c7-3575-4f1e-ad53-d9cd10c8d...@q16g2000yqq.googlegroups.com>,
> Mark Dickinson <dicki...@gmail.com> wrote:
>
> >(2) Obvious things: use range rather than xrange in your loops.
>
> Um, what? You meant the reverse, surely?

Er, yes I did. Thanks!

--
Mark

== 3 of 4 ==
Date: Sun, Feb 14 2010 11:52 am
From: Mark Dickinson

On Feb 14, 4:53 pm, mukesh tiwari <mukeshtiwari.ii...@gmail.com>
wrote:
> Hello everyone. I am new to python and previously i did programming in
> c/c++.Could some one please help me to improve the run time for this
> python program as i don't have idea how to optimized this code.This
> code also seems to be more unpythonic so how to make it look like more
> pythonic . I am trying for this problem(https://www.spoj.pl/problems/
> FACT1/).
> Thank you

One other thing: in the 'brent' function, you're setting m to
randrange(1, n). What's the purpose of this? It looks to me as
though m controls the number of Pollard-Rho iterations that are
clumped together at one time (before doing a gcd operation), and using
a random number for this doesn't make a lot of sense to me.

--
Mark

== 4 of 4 ==
Date: Sun, Feb 14 2010 12:07 pm
From: Steve Howell

On Feb 14, 11:52 am, Mark Dickinson <dicki...@gmail.com> wrote:
> On Feb 14, 4:53 pm, mukesh tiwari <mukeshtiwari.ii...@gmail.com>
> wrote:
>
> > Hello everyone. I am new to python and previously i did programming in
> > c/c++.Could some one please help me to improve the run time for this
> > python program as i don't have idea how to optimized this code.This
> > code also seems to be more unpythonic so how to make it look like more
> > pythonic . I am trying for this problem(https://www.spoj.pl/problems/
> > FACT1/).
> > Thank you
>
> One other thing: in the 'brent' function, you're setting m to
> randrange(1, n). What's the purpose of this? It looks to me as
> though m controls the number of Pollard-Rho iterations that are
> clumped together at one time (before doing a gcd operation), and using
> a random number for this doesn't make a lot of sense to me.
>

The randomness also makes the algorithm a little buggy.

I tried this input:

37^5 41^5

Usually I get the right answer. Other times I get this:

37^5 41^3 1681^1

And occasionally it appears to get into an infinite loop, which
definitely could be a cause of "slowness." :)

==============================================================================
TOPIC: equivalent of Ruby's Pathname?
http://groups.google.com/group/comp.lang.python/t/f580fb3763208425?hl=en
==============================================================================

== 1 of 1 ==
Date: Sun, Feb 14 2010 12:39 pm
From: aahz@pythoncraft.com (Aahz)

In article <8fc356e0-f3ed-4a67-9b37-f21561cef4a5@p13g2000pre.googlegroups.com>,
Sean DiZazzo <half.italian@gmail.com> wrote:
>On Feb 8, 2:36=A0pm, a...@pythoncraft.com (Aahz) wrote:
>> In article <dcace5fc-5ae9-4756-942d-6da7da2f6...@s36g2000prh.googlegroups=
>.com>,
>> Sean DiZazzo =A0<half.ital...@gmail.com> wrote:
>>>
>>>Why did Path() get rejected? Is it the idea itself, or just the
>>>approach that was used? What are the complaints?
>>
>> You should search for the discussiona around it.
>
>I read the discussion, and there was definitely some going back and
>forth on whether it should be sub-classed from string, but the
>conversation just seemed to stop abruptly with no decision one way of
>the other. Maybe I missed a thread.
>
>I guess being dropped without a final go-ahead is just as good as a
>formal no anyway.

Not quite: because it was not rejected, someone who wants to shepherd the
process forward would likely be welcomed (even if it ends up with formal
rejection). I suggest starting by writing your own summary of the
previous discussion and see if the people involved agree that your
summary is reasonably accurate. Also check to see if the original PEP
writer wants to be involved or whether zie is willing to have you take
over.

Another good (and related) starting point would be to create a reasoning
favoring one side or the other that was not brought up in previous
discussion.
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

"At Resolver we've found it useful to short-circuit any doubt and just
refer to comments in code as 'lies'. :-)"

==============================================================================
TOPIC: Fighting with subprocess.Popen
http://groups.google.com/group/comp.lang.python/t/85c2a874044f1a04?hl=en
==============================================================================

== 1 of 1 ==
Date: Sun, Feb 14 2010 12:43 pm
From: Christian Heimes

Mr.John wrote:
> Below is a test case that demonstrates this. Tests 1 & 2 concatenate the
> command and the argument, Tests 3 & 4 mimic the python docs and use the form
> Popen(["mycmd", "myarg"], ...), which never seems to work.

It doesn't work with shell=True because the shell is not able to
interpret the list form. It's recommended that you don't use shell=True
unless you need a shell.

Christian

==============================================================================
TOPIC: python crash on windows but not on linux
http://groups.google.com/group/comp.lang.python/t/7d380bb24fe5e8b8?hl=en
==============================================================================

== 1 of 1 ==
Date: Sun, Feb 14 2010 12:46 pm
From: Roel Schroeven

Op 2010-02-13 13:14, Alf P. Steinbach schreef:
> * hjebbers:
>> I enlarged the windows page file from 750Kb to 1.5Gb .
>> The crash still happens.
>> btw, the crash does not happen at a peak memory usage.
>> According to windows task manager, at the moment of crash mem usage of
>> my program is 669kb, peak memory usage is 1.136kb
>>
>> henk-jan
>
> Probably you meant to write "M", not "k" or "K"?
>
> I've never managed to make much sense of the displays in Windows' Task Manager,
> if that's what you're using, but I think the mem usage it's displaying by
> default is the process' working set, or something very similar to that measure.
>
> You can display additional columns in Task Manager, and one useful one is how
> much virtual memory is allocated,.
>
> And perhaps then (if that's not what you're looking it already) it'll be closer
> to 2 GiB?

Note that the memory measurements in Task Manager are pretty limited
and. Comparing Task Manager and Process Explorer:

Task Manager - Process Explorer
Mem Usage - Working Set
VM Size - Private Bytes
n/a - Virtual Size

I tend to trust Process Explorer a lot more than Task Manager. Note that
what Task Manager calls VM Size is not the size of the virtual memory as
might be expected (if Process Explorer is to be trusted), and that Task
Manager doesn't show the virtual memory size (at least on Windows XP).

--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov

Roel Schroeven

==============================================================================
TOPIC: Function attributes [OT]
http://groups.google.com/group/comp.lang.python/t/73a7dac9c913c91c?hl=en
==============================================================================

== 1 of 2 ==
Date: Sun, Feb 14 2010 1:23 pm
From: Terry Reedy

On 2/13/2010 8:14 AM, Steve Holden wrote:

> Gmane is primarily a news (NNTP-based) service, though for reasons best
> know to the organizers they choose to rename the standard newsgroups to
> fit their own idea of how the universe should be organized, so there the
> group becomes gmane.comp.python.general.

As I understand the site, the last I read a year or two ago, gmane is a
technical mailinglist-to-news gateway. It does not mirror newsgroups,
standard or not. It only mirrors lists such as python-list, not
newsgroups, such comp.lang.python. In doing so, it organizes the lists
into categories like g.comp.python, which has about 200 subgroups
mirroring 200 python lists.

== 2 of 2 ==
Date: Sun, Feb 14 2010 1:32 pm
From: Steve Holden

Terry Reedy wrote:
> On 2/13/2010 8:14 AM, Steve Holden wrote:
>
>> Gmane is primarily a news (NNTP-based) service, though for reasons best
>> know to the organizers they choose to rename the standard newsgroups to
>> fit their own idea of how the universe should be organized, so there the
>> group becomes gmane.comp.python.general.
>
> As I understand the site, the last I read a year or two ago, gmane is a
> technical mailinglist-to-news gateway. It does not mirror newsgroups,
> standard or not. It only mirrors lists such as python-list, not
> newsgroups, such comp.lang.python. In doing so, it organizes the lists
> into categories like g.comp.python, which has about 200 subgroups
> mirroring 200 python lists.
>
That would make more sense. I know that the python.org infrastructure
acts as a gateway between the mailing list and the newsgroup.

==============================================================================
TOPIC: problem with floats and calculations
http://groups.google.com/group/comp.lang.python/t/0159b8f2f69399dc?hl=en
==============================================================================

== 1 of 1 ==
Date: Sun, Feb 14 2010 1:54 pm
From: Dennis Lee Bieber

On Sun, 14 Feb 2010 10:33:54 +0100, Karsten Goen
<karsten.goen@googlemail.com> declaimed the following in
gmane.comp.python.general:

> Maybe anyone can help me with this problem, I don't want to generate for
> every possible user input a single formula. And also it should be possible
> for a computer, my calculator at home does the same and is much smaller and
> slower.
>
Most ALL calculators that I've encountered use packed BCD internally
for numeric data. That is -- two decimal digits (0-9) and a few special
codes (mantissa sign, exponent sign, decimal point location, and
exponent start). A lot of them (though strangely not HP) included "guard
digits" -- they would display, say, 8 significant digits for results,
but kept 10 digits internally, so that rounding errors wouldn't
accumulate as rapidly. M$ Excel/Access/VisualBasic "money" data type
carries four decimal places, on the assumption that only two are
significant, and the last two are guards.

Most ALL computers are using IEEE floating point (and the exceptions
are still a binary floating point, not a decimal representation).

The practice, which /used to be/ taught, is that one does not
compare floating numbers for equality, but rather one compares the
difference between floating numbers to be less than some epsilon value;
epsilon chosen depending upon the significance needed.

Rather than comparing

a = b

one uses

abs(a - b) < epsilon

Since Decimal() also has "infinite" series values (1.0/3.0), an
epsilon comparison may also be called for.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/

==============================================================================
TOPIC: Modifying Class Object
http://groups.google.com/group/comp.lang.python/t/fd36962c4970ac48?hl=en
==============================================================================

== 1 of 3 ==
Date: Sun, Feb 14 2010 1:54 pm
From: Dennis Lee Bieber

On Sun, 14 Feb 2010 00:50:04 -0800, Stephen Hansen
<apt.shansen@gmail.com> declaimed the following in
gmane.comp.python.general:

Hodges' Harbrace College Handbook 10th Ed.

16a Use quotation marks for direct quotations. ...
16b Use quotation marks for minor titles (short stories, ...

16c Words used in a special or an ironic sense are sometimes
enclosed in quotation marks

{We'll skip "e" and "f" which tend to cover overuse and other
punctuation with the quotes -- as "American" usage of punctuation
adjacent to quotes is pretty much at total odds with syntax of
programming languages <G>}
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/

== 2 of 3 ==
Date: Sun, Feb 14 2010 3:41 pm
From: Ethan Furman

Steve Howell wrote:
> On Feb 14, 7:11 am, Steven D'Aprano <st...@REMOVE-THIS-
> cybersource.com.au> wrote:
>
>>On Sat, 13 Feb 2010 23:45:47 -0800, Steve Howell wrote:
>>
>>>The term "pointer" is very abstract. Please give me a concrete
>>>definition of a pointer.
>>
>>A programming language data type whose value directly specifies (or
>>"points to") another value which is stored elsewhere in the computer
>>memory.
>>
>>I quote from Wikipedia:
>>
>>http://en.wikipedia.org/wiki/Pointer_(computing)
>>
>> [quote]
>> A pointer is a simple, less abstracted implementation of the
>> more abstracted reference data type
>> [end quote]
>>
>>And later:
>>
>> [quote]
>> While "pointer" has been used to refer to references in
>> general, it more properly applies to data structures whose
>> interface explicitly allows the pointer to be manipulated
>> (arithmetically via pointer arithmetic) as a memory
>> address...
>> [end quote]
>>
>>And again:
>>
>> [quote]
>> A memory pointer (or just pointer) is a primitive, the value
>> of which is intended to be used as a memory address; it is said
>> that a pointer points to a memory address. It is also said that
>> a pointer points to a datum [in memory] when the pointer's value
>> is the datum's memory address.
>>
>> More generally, a pointer is a kind of reference, and it is said
>> that a pointer references a datum stored somewhere in memory; to
>> obtain that datum is to dereference the pointer. The feature that
>> separates pointers from other kinds of reference is that a
>> pointer's value is meant to be interpreted as a memory address,
>> which is a rather 'low-level' concept.
>> [end quote]
>>
>>
>>>A curly brace is one of these: { }
>>
>>>Pretty concrete, I hope.
>>
>>But { and } are glyphs in some typeface. Chances are that what you see,
>>and what I see, are quite different, and whatever pixels we see, the
>>compiler sees something radically different: two abstract characters
>>implemented in some concrete fashion, but that concrete fashion is a mere
>>implementation detail. They could be implemented as bytes x7b and x7d, or
>>as four-byte sequences x0000007b and x0000007d for UTF-32, or who knows
>>what in some other system. So the *concrete* representation of the curly
>>brace varies according to the system.
>>
>>From that, it's not a difficult leap to say that Pascal's BEGIN and END
>>key words are mere alternate spellings of the abstract "open curly brace"
>>and "close curly brace" with different concrete representations, and from
>>that it's a small step to say that the INDENT and DEDENT tokens seen by
>>the Python compiler (but absent from Python source code!) are too.
>>
>
>
> Thanks. It's a useful analogy; I think I understand your point
> better. I've been bouncing around between Python and Javascript a lot
> lately, so your analogy resonates with me. There are many times when
> I find myself simply respelling things like begin/end, and those
> respellings to me almost make me think of Python and Javascript as
> different dialects of an underlying language. Of course, there are
> other places where the languages differ more substantively, too.
>
> Going back to pointers vs. references, I think the key distinction
> being made is that pointers allow specific memory manipulation,
> although I think even there you're really just dealing with
> abstractions. The address 0x78F394D2 is a little bit closer to the
> machine than, say, the 42nd element of a Python list, but they are
> both just abstractions on top of underlying machines, whether the
> machines are virtual, electronic circuits, vacuum tubes, whatever.
> You can add 6 to 42 and get the 48th object, but its Python's
> convention not to call the 48th object a memory address or expose a
> reference to it as a pointer. If I want to pass along the reference
> to the 48th element of a list as the slot to be updated (i.e. with the
> intention to actually mutate the list itself), then I need a tuple
> like (lst, 48).
>

I think that's the key right there -- if 48 was really a pointer, you
wouldn't need to pass lst in as 48 would in fact be the memory address
of the object you wanted to manipulate.

~Ethan~

== 3 of 3 ==
Date: Sun, Feb 14 2010 4:41 pm
From: "Alf P. Steinbach"

* Ethan Furman:
> Steve Howell wrote:
>>
>> Going back to pointers vs. references, I think the key distinction
>> being made is that pointers allow specific memory manipulation,
>> although I think even there you're really just dealing with
>> abstractions. The address 0x78F394D2 is a little bit closer to the
>> machine than, say, the 42nd element of a Python list, but they are
>> both just abstractions on top of underlying machines, whether the
>> machines are virtual, electronic circuits, vacuum tubes, whatever.
>> You can add 6 to 42 and get the 48th object, but its Python's
>> convention not to call the 48th object a memory address or expose a
>> reference to it as a pointer. If I want to pass along the reference
>> to the 48th element of a list as the slot to be updated (i.e. with the
>> intention to actually mutate the list itself), then I need a tuple
>> like (lst, 48).
>>
>
> I think that's the key right there -- if 48 was really a pointer, you
> wouldn't need to pass lst in as 48 would in fact be the memory address
> of the object you wanted to manipulate.

The generalization is known as a "based pointer".

Except where it's a fundamental abstraction in a programming language, where it
might be called anything.

For example, in C++ some so called "member pointers" are logically based
pointers. They have pointer syntax (as do C++ iterators, which are not
necessarily pointers), but member pointers are not pointers in the C++
standard's sense; in particular, dereferencing a C++ member pointer yields a
typeless entity, which is not the case for a normal pointer, although that
standard confusingly calls also member pointers pointers in some places, and in
other places uses the pointer term only about basic pointers.

So, delving into the details of that terminology means traveling into a pretty
chaotic territory. But on the other hand, going for the more abstract it gets
cleaner and simpler. The Wikipedia article is about in the middle somewhere.

It is perhaps not confusing that it is confusing to many. :-)

Cheers & hth.,

- Alf

==============================================================================
TOPIC: MemoryError, can I use more?
http://groups.google.com/group/comp.lang.python/t/0944e954b29aa311?hl=en
==============================================================================

== 1 of 5 ==
Date: Sun, Feb 14 2010 2:05 pm
From: Ross Ridge

"Diez B. Roggisch" <deets@nospam.web.de> writes:
> No, PAE can be used to access much more memory than 4GB - albeit
> through paging. AFAIK up to 2^36 Bytes.

Anssi Saari <as@sci.fi> wrote:
>That too. I admit, after checking, that you can't go above 3 GiB per
>process even in server Windows. But for Linux there exists (or
>existed, since it seems it hasn't been updated since 2004) a kernel
>patch which provides a "4GB/4GB" address split. Kernel is in one
>segment, userland in another and hence a process can access full 4GB.

Windows has a similar feature that allows 32-bit applications running on
64-bit versions of Windows to have a nearly 4Gb virtual address space.
Windows also allows 32-bit applications to use more than 4GB of physical
memory through a paging mechanism called "Address Windowing Extensions".
Also 32-bit applications can effectively use more than 4GB of RAM through
indirect means like multiple processes, the disk cache or video card RAM.

Ross Ridge

--
l/ // Ross Ridge -- The Great HTMU
[oo][oo] rridge@csclub.uwaterloo.ca
-()-/()/ http://www.csclub.uwaterloo.ca/~rridge/
db //

== 2 of 5 ==
Date: Sun, Feb 14 2010 3:20 pm
From: "Diez B. Roggisch"

Am 14.02.10 12:28, schrieb Laszlo Nagy:
> 2010.02.13. 17:40 keltez�ssel, Diez B. Roggisch �rta:
>> Am 13.02.10 17:18, schrieb Anssi Saari:
>>> Nobody<nobody@nowhere.com> writes:
>>>
>>>> A single process can't use much more than 2GiB of RAM without a
>>>> 64-bit CPU
>>>> and OS.
>>>
>>> That's not really true. Even Windows XP has the /3GB boot option to
>>> allow 3 GiB per process. On PCs, free operating systems and server
>>> Windows can use PAE to give access to full 4 GB per process.
>>
>> No, PAE can be used to access much more memory than 4GB - albeit
>> through paging. AFAIK up to 2^36 Bytes.
> PAE is for the kernel. Yes, you can use much more memory with 32 bit
> kernel + PAE. But the ~3G per 32bit process limit still applies. It is
> because the PAE extension allows the kernel to distribute the same
> virtual address ranges between different processes, but map them to
> different physical memory addresses. However, the process that was
> compiled and is running in 32 bit mode, cannot access more than ~3GB
> simply because it is not able to address more memory locations with 32
> bit addresses. (minus ~1G is because it needs virtual addresses for I/O
> devices as well, and those addresses cannot be mapped to physical memory).

No. It can access more, through paging, similar like mmap. Specialized
software like databases do that to keep large data in memory. And this
has nothing to do with compilation.

Of course this doesn't help for python, at least not for python-objects,
as these need a flat memory model. But it's not true that only the
kernels benefit from the PAE.

Diez

== 3 of 5 ==
Date: Sun, Feb 14 2010 4:19 pm
From: "Echavarria Gregory, Maria Angelica"

I use the physical and kernel memory boxes in the windows task manager under the performance tab... in that way I can see the exact RAM that just OS and idle processes occupy before I run my app, and then also the limit at which my app pushes the memory...

M. Angelica Echavarria-Gregory, M.Sc., E.I.
Ph.D Candidate
University of Miami
Phone 305 284-3611
________________________________________
From: ssteinerX@gmail.com [ssteinerx@gmail.com]
Sent: Friday, February 12, 2010 7:58 PM
To: Echavarria Gregory, Maria Angelica
Cc: python-list@python.org
Subject: Re: MemoryError, can I use more?

On Feb 12, 2010, at 7:21 PM, Echavarria Gregory, Maria Angelica wrote:

> Dear group:
>
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
>
> Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...

How are you determining that it has occupied "exactly only 2.2GB?"

== 4 of 5 ==
Date: Sun, Feb 14 2010 4:20 pm
From: "Echavarria Gregory, Maria Angelica"

Dear Chris,

One of the machines I tested my app in is 64 bit and happened the same. The RAM consumed by the OS and other processes is already included in the 2.2 I'm telling... my app enters to work when the RAM is already consumed in ~600 MB in the 3- 32 bit machines ... in the 64 bit machine was exactly the same only that it started a little bit higher because it has windows 7... so should I understand that there is nothing I can do for my app to use up more RAM?

Thanks for your time and answer,
Angelica.

M. Angelica Echavarria-Gregory, M.Sc., E.I.
Ph.D Candidate
University of Miami
Phone 305 284-3611

________________________________
From: Chris Kaynor [ckaynor@zindagigames.com]
Sent: Friday, February 12, 2010 7:44 PM
To: Echavarria Gregory, Maria Angelica
Cc: python-list@python.org
Subject: Re: MemoryError, can I use more?

A 32 bit app can only use 4 GB of memory itself (regardless of the amount of system ram), the OS claims some of this for the system, dlls occupy some of it, etc. As such, the app can only really use a smaller subset (generally between 2 to 3 GB, depending upon the app and the OS).

Chris

On Fri, Feb 12, 2010 at 4:21 PM, Echavarria Gregory, Maria Angelica <m.echavarriagregory@umiami.edu<mailto:m.echavarriagregory@umiami.edu>> wrote:
Dear group:

I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.

Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...

Thanks,
Angelica.
--
http://mail.python.org/mailman/listinfo/python-list

== 5 of 5 ==
Date: Sun, Feb 14 2010 7:16 pm
From: Dave Angel

Echavarria Gregory, Maria Angelica wrote:
> Dear Chris,
>
> One of the machines I tested my app in is 64 bit and happened the same. The RAM consumed by the OS and other processes is already included in the 2.2 I'm telling... my app enters to work when the RAM is already consumed in ~600 MB in the 3- 32 bit machines ... in the 64 bit machine was exactly the same only that it started a little bit higher because it has windows 7... so should I understand that there is nothing I can do for my app to use up more RAM?
>
> Thanks for your time and answer,
> Angelica.
>
>
> M. Angelica Echavarria-Gregory, M.Sc., E.I.
> Ph.D Candidate
> University of Miami
> Phone 305 284-3611
>
> ________________________________
> From: Chris Kaynor [ckaynor@zindagigames.com]
> Sent: Friday, February 12, 2010 7:44 PM
> To: Echavarria Gregory, Maria Angelica
> Cc: python-list@python.org
> Subject: Re: MemoryError, can I use more?
>
> A 32 bit app can only use 4 GB of memory itself (regardless of the amount of system ram), the OS claims some of this for the system, dlls occupy some of it, etc. As such, the app can only really use a smaller subset (generally between 2 to 3 GB, depending upon the app and the OS).
>
> Chris
>
>
> On Fri, Feb 12, 2010 at 4:21 PM, Echavarria Gregory, Maria Angelica <m.echavarriagregory@umiami.edu<mailto:m.echavarriagregory@umiami.edu>> wrote:
> Dear group:
>
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
>
> Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...
>
> Thanks,
> Angelica.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
>
There are three different limits at play here. Since you're still not
saying how you're "measuring" usage, we've all been guessing just which
one you're hitting. There's physical RAM, virtual address space, and
swappable space (swapfile on disk). Each reaches some limit in
different ways.

And there are probably a dozen different ways to measure "memory use,"
that get a dozen different answers. If you say which one you're using,
that gives "exactly 2.2 GB," maybe someone will be familiar with that
particular approach, and its meaning.

DaveA

==============================================================================
TOPIC: new python install
http://groups.google.com/group/comp.lang.python/t/fb26d04ac43d9712?hl=en
==============================================================================

== 1 of 2 ==
Date: Sun, Feb 14 2010 2:42 pm
From: monkeys paw

Upon invoking python, it hangs
until Ctrl^C is typed, and then the
>>> interactive shell begins.

Like so:

joemoney% python
Python 2.4.6 (#1, Dec 13 2009, 23:45:11) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
# Hangs ^^^ at this point until ^C is typed
^C

>>>

I've seen this in other new installs and wondered if there is
a common problem that would cause this? It's on a sun os box

== 2 of 2 ==
Date: Sun, Feb 14 2010 6:28 pm
From: Tim Chase

monkeys paw wrote:
> Upon invoking python, it hangs
> until Ctrl^C is typed, and then the
> >>> interactive shell begins.
>
> Like so:
>
> joemoney% python
> Python 2.4.6 (#1, Dec 13 2009, 23:45:11) [C] on sunos5
> Type "help", "copyright", "credits" or "license" for more information.
> # Hangs ^^^ at this point until ^C is typed
> ^C
>
> >>>
>
> I've seen this in other new installs and wondered if there is
> a common problem that would cause this? It's on a sun os box

Though I've never seen such live, perhaps you have some sort of
site-wide or user-specific config file such as ~/.pythonrc.py or
a site-wide site.py file. You can read up a bit at

http://docs.python.org/library/site.html
http://docs.python.org/library/user.html

I believe the way to test this is to start Python with either the
-s or -S option (or both) to disable looking for user or site
modules. Also check if you have some crazy value set for
$PYTHONSTARTUP.

http://docs.python.org/using/cmdline.html#cmdoption-S

My guess is that something in one of these places is triggering
the hang until you kill that with a ^C after which you get the
requested prompt.

Hope this helps,

-tkc

==============================================================================
TOPIC: hausrat und haftpflicht versicherung, nürnberger hausratversicherung,
hausratversicherung raub, ammerländer versicherung, hausratversicherung im
ausland, hausrat ammerländer,
http://groups.google.com/group/comp.lang.python/t/b4800f9ad742c943?hl=en
==============================================================================

== 1 of 1 ==
Date: Sun, Feb 14 2010 6:17 pm
From: cheng goethe

hausrat und haftpflicht versicherung, nürnberger hausratversicherung,
hausratversicherung raub, ammerländer versicherung,
hausratversicherung im ausland, hausrat ammerländer,

+
+
+
HAUSRAT VERSICHERUNG +++ HAUSRATVERSICHERUNG BILLIG +++ BILLIGE
HAUSRATSVERSICHERUNG
http://WWW.U2V.NL
http://WWW.U2V.NL
http://WWW.U2V.NL
http://WWW.U2V.NL
http://WWW.U2V.NL
http://WWW.U2V.NL
http://WWW.U2V.NL
http://WWW.U2V.NL
+
+
+
+

http://r-v-hausratversicherung.U2V.NL
http://hausratversicherung-was-ist-versichert.U2V.NL
http://vergleich-von-hausratversicherungen.U2V.NL
http://guenstige-versicherungen.U2V.NL

einbruch günstigste hausratsversicherung in Passau
privathaftpflicht hausratversicherung mit glas in Libyen
hausratversicherung umzug rechtsschutzversicherungen in Osterode
wer braucht eine hausratversicherung haft und hausratversicherung in
Greiz
kündigung hausratversicherung umzug hausratversicherung tresor in
Bayern
hausratversicherung leistungsumfang hausratversicherung kinder in
Sambia
hausrat online günstige versicherungen in Iran
wertsachen versicherung was beinhaltet die hausratversicherung in
Somalia
blitzschlag hausratversicherung da in Belize

http://suche.aol.de/aol/search?query=hausratsversicherung+notwendig+inurl%3Afor-um&invocationType=no.omittWeb&filter=false

==============================================================================

You received this message because you are subscribed to the Google Groups "comp.lang.python"
group.

To post to this group, visit http://groups.google.com/group/comp.lang.python?hl=en

To unsubscribe from this group, send email to comp.lang.python+unsubscribe@googlegroups.com

To change the way you get mail from this group, visit:
http://groups.google.com/group/comp.lang.python/subscribe?hl=en

To report abuse, send email explaining the problem to abuse@googlegroups.com

==============================================================================
Google Groups: http://groups.google.com/?hl=en

twitter

Sunday, February 14, 2010

comp.lang.python - 25 new messages in 12 topics - digest

0 Comments:

Post a Comment

About Me

Previous Posts