comp.lang.python - 26 new messages in 8 topics - digest

comp.lang.python
http://groups.google.com/group/comp.lang.python?hl=en

comp.lang.python@googlegroups.com

Today's topics:

* Is it possible to protect python source code by compiling it to .pyc or .pyo?
- 1 messages, 1 author
http://groups.google.com/group/comp.lang.python/t/72a8ca674ee4bb63?hl=en
* Python 3.x adoption - 10 messages, 8 authors
http://groups.google.com/group/comp.lang.python/t/3d5defc6634ef3b4?hl=en
* How to write this as a list comprehension? - 5 messages, 4 authors
http://groups.google.com/group/comp.lang.python/t/0b63055a0d321622?hl=en
* setup.py issue - some files are included as intended, but one is not - 1
messages, 1 author
http://groups.google.com/group/comp.lang.python/t/615fb6822e01f928?hl=en
* numpy.where() and multiple comparisons - 4 messages, 3 authors
http://groups.google.com/group/comp.lang.python/t/252e25bda74254c2?hl=en
* Python Scalability TCP Server + Background Game - 2 messages, 2 authors
http://groups.google.com/group/comp.lang.python/t/8234bf03949a8164?hl=en
* doctests compatibility for python 2 & python 3 - 1 messages, 1 author
http://groups.google.com/group/comp.lang.python/t/e83b94c8f7e04717?hl=en
* Guessing the encoding from a BOM - 2 messages, 2 authors
http://groups.google.com/group/comp.lang.python/t/1b4501ed3e9e2af1?hl=en

==============================================================================
TOPIC: Is it possible to protect python source code by compiling it to .pyc or
.pyo?
http://groups.google.com/group/comp.lang.python/t/72a8ca674ee4bb63?hl=en
==============================================================================

== 1 of 1 ==
Date: Fri, Jan 17 2014 2:02 pm
From: Tim Delaney


On 18 January 2014 08:31, Joshua Landau <joshua@landau.ws> wrote:

> On 17 January 2014 00:58, Sam <lightaiyee@gmail.com> wrote:
> > I would like to protect my python source code. It need not be foolproof
> as long as it adds inconvenience to pirates.
> >
> > Is it possible to protect python source code by compiling it to .pyc or
> .pyo? Does .pyo offer better protection?
>
> If you're worried about something akin to corporate espionage or
> some-such, I don't know of a better way than ShedSkin or Cython. Both
> of those will be far harder to snatch the source of. Cython will be
> particularly easy to use as it is largely compatible with Python
> codebases.
>

Indeed - I've only had one time someone absolutely insisted that this be
done (for trade secret reasons - there needed to be a good-faith attempt to
prevent others from trivially getting the source). I pointed them at Pyrex
(this was before Cython, or at least before it was dominant). They fully
understood that it wouldn't stop a determined attacker - this was a place
where a large number of the developers were used to working on bare metal.

If you're going to do this, I strongly suggest only using Cython on code
that needs to be obscured (and if applicable, performance-critical
sections). I'm currently working with a system which works this way - edge
scripts in uncompiled .py files, and inner code as compiled extensions. The
.py files have been really useful for interoperability purposes e.g. I was
able to verify yesterday that one of the scripts had a bug in its
command-line parsing and I wasn't going insane after all.

Also, remember that any extension can be imported and poked at (e.g. in the
interactive interpreter). You'd be surprised just how much information you
can get that way just using help, dir, print and some experimentation. The
output I was parsing from one of the scripts was ambiguous, and it was one
where most of the work was done in an extension. I was able to poke around
using the interactive interpreter understand what it was doing and obtain
the data in an unambiguous manner to verify against my parser.

The only way to truly protect code is to not ship any version of it
(compiled or otherwise), but have the important parts hosted remotely under
your control (and do your best to ensure it doesn't become compromised).

Tim Delaney





==============================================================================
TOPIC: Python 3.x adoption
http://groups.google.com/group/comp.lang.python/t/3d5defc6634ef3b4?hl=en
==============================================================================

== 1 of 10 ==
Date: Fri, Jan 17 2014 2:10 pm
From: Terry Reedy


On 1/17/2014 10:27 AM, Grant Edwards wrote:
> On 2014-01-14, Staszek <noreply@eisenbits.com> wrote:
>
>> What's the problem with Python 3.x?
>
> The problem with Python 3.x is Python 2.7. ;)

Cute.

>> What's wrong?...
>
> Python 2.7 still does everything 99% of us need to do, and we're too
> lazy to switch.

While '99' is rhetorical, the statement is accurate for many. The core
devs do not expect such people* to switch until they think they would
benefit, and even then, only to the version that has enough goodies.

* except for authors of widely used libraries ;-), and even then, it
ends up being a matter of whether such authors think they will benefit
from having 3.x users.

--
Terry Jan Reedy





== 2 of 10 ==
Date: Fri, Jan 17 2014 2:16 pm
From: beliavsky@aol.com


On Tuesday, January 14, 2014 2:38:29 PM UTC-5, Skip Montanaro wrote:
> > What's the problem with Python 3.x? It was first released in 2008, but
>
> > web hosting companies still seem to offer Python 2.x rather.
>
> >
>
> > For example, Google App Engine only offers Python 2.7.
>
> >
>
> > What's wrong?...
>
>
>
> What makes you think anything's wrong? Major changes to any
>
> established piece of software takes a fairly long while to infiltrate.
>
> Lots of COBOL and Fortran 77 still running out there.

I don't think the Fortran analogy is valid.

The Fortran standards after F77 are almost complete supersets of F77, and Fortran compiler vendors handle even the deleted parts of F77, knowing their customer base. Therefore you do not need to rewrite old Fortran code to use it with Fortran 95 or 2003 compilers, and you can easily mix old-style and modern Fortran. Later Fortran standards did not invalidate basic syntax such as print statements, as Python 3 did. Python 2 and 3 are incompatible in ways that do not apply to Fortran standards pre- and post- F77.




== 3 of 10 ==
Date: Fri, Jan 17 2014 2:51 pm
From: Mark Lawrence



On 17/01/2014 22:16, beliavsky@aol.com wrote:
> On Tuesday, January 14, 2014 2:38:29 PM UTC-5, Skip Montanaro wrote:
>>> What's the problem with Python 3.x? It was first released in 2008, but
>>
>>> web hosting companies still seem to offer Python 2.x rather.
>>
>>>
>>
>>> For example, Google App Engine only offers Python 2.7.
>>
>>>
>>
>>> What's wrong?...
>>
>>
>>
>> What makes you think anything's wrong? Major changes to any
>>
>> established piece of software takes a fairly long while to infiltrate.
>>
>> Lots of COBOL and Fortran 77 still running out there.
>
> I don't think the Fortran analogy is valid.
>
> Later Fortran standards did not invalidate basic syntax such as print statements, as Python 3 did. Python 2 and 3 are incompatible in ways that do not apply to Fortran standards pre- and post- F77.
>

A good choice to make, the capability to use "from __future__ import
print_function", or whatever the actual thing is, has been available for
years. 2to3 has been available for years, six was released at the end
of June 2010 and there's now future, see http://python-future.org/
Admittedly there's a problem with the porting of code which mixes bytes
and strings, but that's being addressed right now via PEPs 460, 461 and
possibly others.

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence





== 4 of 10 ==
Date: Fri, Jan 17 2014 3:03 pm
From: Terry Reedy


On 1/17/2014 5:16 PM, beliavsky@aol.com wrote:

> I don't think the Fortran analogy is valid.

The appropriate analogy for the changes between Python 2.x and 3.x,
which started about 1 and 2 decades after the original Python, are the
changes between Fortran IV/66 and Fortran 77, also about 1 and 2 decades
after the original Fortran. The latter two have a comparable number of
differences. "In this revision of the standard [F77], a number of
features were removed or altered in a manner that might invalidate
previously standard-conforming programs.
https://en.wikipedia.org/wiki/Fortran
Not mentioned in the wiki article is the change in calling convention
from call by value to call by reference (or maybe the opposite). I
remember a program crashing because of this when I tried it with F77.

Overall, there was more churn in Fortran up to F77 than there was in
Python up to 3.0.

> The Fortran standards after F77 are almost complete supersets of F77, and Fortran compiler vendors handle even the deleted parts of F77, knowing their customer base. Therefore you do not need to rewrite old Fortran code to use it with Fortran 95 or 2003 compilers, and you can easily mix old-style and modern Fortran. Later Fortran standards did not invalidate basic syntax such as print statements, as Python 3 did. Python 2 and 3 are incompatible in ways that do not apply to Fortran standards pre- and post- F77.

Since 3.0, we have added new syntax ('yield from', u'' for instance) but
I do not believe we have deleted or changed any syntax (I might have
forgotten something minor) and I do not know of any proposal to do so
(except to re-delete u'', which should only be used as a temporary
crutch for 2&3 code).

> Python 2 and 3 are incompatible in ways that do not apply to Fortran
> standards pre- and post- F77.

As stated above, I disagree with respect to pre-F77 and F77. Did you
actually program in both, as I did?

--
Terry Jan Reedy





== 5 of 10 ==
Date: Fri, Jan 17 2014 3:12 pm
From: MRAB


On 2014-01-17 23:03, Terry Reedy wrote:
[snip]
> Since 3.0, we have added new syntax ('yield from', u'' for instance) but
> I do not believe we have deleted or changed any syntax (I might have
> forgotten something minor) and I do not know of any proposal to do so
> (except to re-delete u'', which should only be used as a temporary
> crutch for 2&3 code).
>
There was the removal of backticks.





== 6 of 10 ==
Date: Fri, Jan 17 2014 3:17 pm
From: Chris Angelico


On Sat, Jan 18, 2014 at 10:12 AM, MRAB <python@mrabarnett.plus.com> wrote:
> On 2014-01-17 23:03, Terry Reedy wrote:
> [snip]
>
>> Since 3.0, we have added new syntax ('yield from', u'' for instance) but
>> I do not believe we have deleted or changed any syntax (I might have
>> forgotten something minor) and I do not know of any proposal to do so
>> (except to re-delete u'', which should only be used as a temporary
>> crutch for 2&3 code).
>>
> There was the removal of backticks.

Wasn't that removed _in_, not _since_, 3.0?

ChrisA




== 7 of 10 ==
Date: Fri, Jan 17 2014 5:01 pm
From: Dennis Lee Bieber


On Fri, 17 Jan 2014 18:03:45 -0500, Terry Reedy <tjreedy@udel.edu>
declaimed the following:


>Not mentioned in the wiki article is the change in calling convention
>from call by value to call by reference (or maybe the opposite). I
>remember a program crashing because of this when I tried it with F77.
>
Maybe because there was NO change in calling convention. FORTRAN was
call by reference in all versions I've worked with. DEC did add special
"functions" to force a parameter to be passed as "value", "reference", or
"descriptor" -- mostly to interface with other languages (descriptor is
what was used for character strings -- basically a structure containing the
length of the string/buffer, and the address of said buffer).

However, one may have encountered an implementation that did some
checking for duplicated parameter addresses -- or may have behaved
internally as a value/return system, wherein the by-reference parameters
were copied to local/stack storage, manipulated, and only written back to
the reference address on exit. That would result in differences:

a = 1
call xyz(a, b, a)

subroutine xyz(x, y, z)

x = x + 1
z = z * 2

...
In a true by-reference system, "a" would turn into 2 for the first
assignment, but then turn into 4 after the second.

In value/return, the second assignment still sees "1" and results in
"2", and thereby "a" becomes "2", not "4"...

I'd still consider a value/return scheme to be a misbehaving FORTRAN.
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/





== 8 of 10 ==
Date: Fri, Jan 17 2014 5:18 pm
From: Ben Finney


Terry Reedy <tjreedy@udel.edu> writes:

> Since 3.0, we have added new syntax ('yield from', u'' for instance)
> but I do not believe we have deleted or changed any syntax (I might
> have forgotten something minor)

I'm aware of the removal of '`foo`' (use 'repr(foo)' instead), and
removal of 'except ExcClass, exc_instance' (use 'except ExcClass as
exc_instance' instead).

--
\ "For my birthday I got a humidifier and a de-humidifier. I put |
`\ them in the same room and let them fight it out." --Steven Wright |
_o__) |
Ben Finney





== 9 of 10 ==
Date: Fri, Jan 17 2014 5:27 pm
From: Ben Finney


Ben Finney <ben+python@benfinney.id.au> writes:

> Terry Reedy <tjreedy@udel.edu> writes:
>
> > Since 3.0, we have added new syntax ('yield from', u'' for instance)
> > but I do not believe we have deleted or changed any syntax (I might
> > have forgotten something minor)
>
> I'm aware of the removal of '`foo`' (use 'repr(foo)' instead), and
> removal of 'except ExcClass, exc_instance' (use 'except ExcClass as
> exc_instance' instead).

Ah, you meant "deleted or changed any Python 3 syntax". No, I'm not
aware of any such changes.

--
\ "I have never imputed to Nature a purpose or a goal, or |
`\ anything that could be understood as anthropomorphic." --Albert |
_o__) Einstein, unsent letter, 1955 |
Ben Finney





== 10 of 10 ==
Date: Fri, Jan 17 2014 6:49 pm
From: Roy Smith


In article <lbc296$itl$1@reader1.panix.com>,
Grant Edwards <invalid@invalid.invalid> wrote:

> On 2014-01-17, Tim Chase <python.list@tim.thechases.com> wrote:
> > On 2014-01-17 15:27, Grant Edwards wrote:
> >> > What's wrong?...
> >>
> >> Python 2.7 still does everything 99% of us need to do, and we're too
> >> lazy to switch.
> >
> > And in most distros, typing "python" invokes 2.x, and explicitly
> > typing "python3" is almost 17% longer. We're a lazy bunch! :-)
>
> And my touch typing accuracy/speed drops pretty noticably when I have
> to use the top row of keys...

This is why shells have the ability to create aliases, and also why
command-line autocomplete was invented :-)





==============================================================================
TOPIC: How to write this as a list comprehension?
http://groups.google.com/group/comp.lang.python/t/0b63055a0d321622?hl=en
==============================================================================

== 1 of 5 ==
Date: Fri, Jan 17 2014 3:19 pm
From: Piet van Oostrum


Hi,

I am looking for an elegant way to write the following code as a list
comprehension:

labels = []
for then, name in mylist:
_, mn, dy, _, _, _, wd, _, _ = localtime(then)
labels.append(somefunc(mn, day, wd, name))

So mylist is a list of tuples, the first member of the tuple is a time
(as epoch offset) and I neeed to apply a function on some fields of the
localtime of it.

I could define a auxiliary function like:

def auxfunc(then, name):
_, mn, dy, _, _, _, wd, _, _ = localtime(then)
return somefunc(mn, day, wd, name)

and then use
[auxfunc(then, name) for then, name in mylist]

or even
[auxfunc(*tup) for tup in mylist]

But defining the auxfunc takes away the elegance of a list comprehension. I would like to integrate the unpacking of localtime() and calling somefunc within the list comprehension, but I don't see a simple way to do that.

somefunc(mn, day, wd, name) for _, mn, dy, _, _, _, wd, _, _ in [localtime(then)]
(i.e. using a list comprehension on a one element list to do the variable shuffling)
works but I don't find that very elegant.

labels = [somefunc(mn, day, wd, name)
for then, name in mylist
for _, mn, dy, _, _, _, wd, _, _ in [localtime(then)]]

Python misses a 'where' or 'let'-like construction as in Haskell.

Anybody has a more elegant solution?
--
Piet van Oostrum <piet@vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]




== 2 of 5 ==
Date: Fri, Jan 17 2014 3:49 pm
From: Dan Stromberg


On Fri, Jan 17, 2014 at 3:19 PM, Piet van Oostrum <piet@vanoostrum.org> wrote:
> Hi,
>
> I am looking for an elegant way to write the following code as a list
> comprehension:
>
> labels = []
> for then, name in mylist:
> _, mn, dy, _, _, _, wd, _, _ = localtime(then)
> labels.append(somefunc(mn, day, wd, name))

My recomendation: Don't use a list comprehension. List comprehensions
and generator expressions are great for quick little things, but
become less readable when you have to string them over multiple
physical lines.

> labels = [somefunc(mn, day, wd, name)
> for then, name in mylist
> for _, mn, dy, _, _, _, wd, _, _ in [localtime(then)]]




== 3 of 5 ==
Date: Fri, Jan 17 2014 7:25 pm
From: Rustom Mody


On Saturday, January 18, 2014 4:49:55 AM UTC+5:30, Piet van Oostrum wrote:
> Hi,

> I am looking for an elegant way to write the following code as a list
> comprehension:

> labels = []
> for then, name in mylist:
> _, mn, dy, _, _, _, wd, _, _ = localtime(then)
> labels.append(somefunc(mn, day, wd, name))

> So mylist is a list of tuples, the first member of the tuple is a time
> (as epoch offset) and I neeed to apply a function on some fields of the
> localtime of it.

> I could define a auxiliary function like:

> def auxfunc(then, name):
> _, mn, dy, _, _, _, wd, _, _ = localtime(then)
> return somefunc(mn, day, wd, name)

> and then use
> [auxfunc(then, name) for then, name in mylist]

> or even
> [auxfunc(*tup) for tup in mylist]

> But defining the auxfunc takes away the elegance of a list comprehension. I would like to integrate the unpacking of localtime() and calling somefunc within the list comprehension, but I don't see a simple way to do that.

> somefunc(mn, day, wd, name) for _, mn, dy, _, _, _, wd, _, _ in [localtime(then)]
> (i.e. using a list comprehension on a one element list to do the variable shuffling)
> works but I don't find that very elegant.

> labels = [somefunc(mn, day, wd, name)
> for then, name in mylist
> for _, mn, dy, _, _, _, wd, _, _ in [localtime(then)]]

> Python misses a 'where' or 'let'-like construction as in Haskell.

+1
Yes Ive often been bitten by the lack of a 'comprehension-let'

Something like this is possible??


[somefunc(mn,day,wd,name) for (_, mn,dy,_,_,_,wd,_,_), name) in [localtime(then), name for then, name in mylist]]

Some debugging of the structure will be necessary (if at all possible)
I dont have your functions so cant do it






== 4 of 5 ==
Date: Sat, Jan 18 2014 12:36 am
From: Peter Otten <__peter__@web.de>


Piet van Oostrum wrote:

> Hi,
>
> I am looking for an elegant way to write the following code as a list
> comprehension:
>
> labels = []
> for then, name in mylist:
> _, mn, dy, _, _, _, wd, _, _ = localtime(then)
> labels.append(somefunc(mn, day, wd, name))
>
> So mylist is a list of tuples, the first member of the tuple is a time
> (as epoch offset) and I neeed to apply a function on some fields of the
> localtime of it.
>
> I could define a auxiliary function like:
>
> def auxfunc(then, name):
> _, mn, dy, _, _, _, wd, _, _ = localtime(then)
> return somefunc(mn, day, wd, name)
>
> and then use
> [auxfunc(then, name) for then, name in mylist]
>
> or even
> [auxfunc(*tup) for tup in mylist]
>
> But defining the auxfunc takes away the elegance of a list comprehension.
> I would like to integrate the unpacking of localtime() and calling
> somefunc within the list comprehension, but I don't see a simple way to do
> that.
>
> somefunc(mn, day, wd, name) for _, mn, dy, _, _, _, wd, _, _ in
> [localtime(then)] (i.e. using a list comprehension on a one element list
> to do the variable shuffling) works but I don't find that very elegant.
>
> labels = [somefunc(mn, day, wd, name)
> for then, name in mylist
> for _, mn, dy, _, _, _, wd, _, _ in [localtime(then)]]
>
> Python misses a 'where' or 'let'-like construction as in Haskell.
>
> Anybody has a more elegant solution?

Options I can think of:

You could do it in two steps...

time_name_pairs = ((localtime(then), name) for then, name in mylist)
labels = [somefunc(t.tm_mon, t.tm_mday, t.tm_wday, name)
for t, name in time_name_pairs]

...or you could inline the helper function...

mon_mday_wday = operator.attrgetter("tm_mon", "tm_day", "tm_wday")
labels = [somefunc(*mon_mday_wday(localtime(then)), name=name)
for then, name in mylist]

-- but both seem less readable than the classical for-loop.

What would a list-comp with `let` or `where` look like? Would it win the
beauty contest against the loop?





== 5 of 5 ==
Date: Sat, Jan 18 2014 2:00 am
From: Piet van Oostrum


Rustom Mody <rustompmody@gmail.com> writes:

> On Saturday, January 18, 2014 4:49:55 AM UTC+5:30, Piet van Oostrum wrote:
[...]
>
>> Python misses a 'where' or 'let'-like construction as in Haskell.
>
> +1
> Yes Ive often been bitten by the lack of a 'comprehension-let'

If it used only in a comprehension as in my example you can write instead of 'where vars = expr':
for vars in [expr], unnecessarily construction a one element list.
If there would be a syntax like:
for vars = expr
this could be avoided.
>
> Something like this is possible??
>
>
> [somefunc(mn,day,wd,name) for (_, mn,dy,_,_,_,wd,_,_), name) in
> [localtime(then), name for then, name in mylist]]
It works modulo some corrections in the parentheses:

[somefunc(mn,day,wd,name) for (_, mn,dy,_,_,_,wd,_,_), name in
[(localtime(then), name) for then, name in mylist]]
but I find that hardly more elegant.
--
Piet van Oostrum <piet@vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]





==============================================================================
TOPIC: setup.py issue - some files are included as intended, but one is not
http://groups.google.com/group/comp.lang.python/t/615fb6822e01f928?hl=en
==============================================================================

== 1 of 1 ==
Date: Fri, Jan 17 2014 4:43 pm
From: Dan Stromberg


On Wed, Jan 15, 2014 at 7:18 AM, Piet van Oostrum <piet@vanoostrum.org> wrote:
> Dan Stromberg <drsalists@gmail.com> writes:
>
>> On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg <drsalists@gmail.com> wrote:
>>> Hi folks.
>>>
>>> I have a setup.py problem that's driving me nuts.
>>
>> Anyone? I've received 0 responses.
>
> I can't even install your code because there's a bug in it.
>
> m4_treap.m4 contains this instruction twice:
>
> ifdef(/*pyx*/,cp)if current is None:
> ifdef(/*pyx*/,cp)raise KeyError
>
> Which when generating pyx_treap.pyx (with *pyx* defined) expands to the syntactically incorrect
>
> cpif current is None:
> cpraise KeyError
Apologies. I'd fixed that in my source tree, but not checked it in.

It's checked in now.





==============================================================================
TOPIC: numpy.where() and multiple comparisons
http://groups.google.com/group/comp.lang.python/t/252e25bda74254c2?hl=en
==============================================================================

== 1 of 4 ==
Date: Fri, Jan 17 2014 5:51 pm
From: John Ladasky


Hi folks,

I am awaiting my approval to join the numpy-discussion mailing list, at scipy.org. I realize that would be the best place to ask my question. However, numpy is so widely used, I figure that someone here would be able to help.

I like to use numpy.where() to select parts of arrays. I have encountered what I would consider to be a bug when you try to use where() in conjunction with the multiple comparison syntax of Python. Here's a minimal example:

Python 3.3.2+ (default, Oct 9 2013, 14:50:09)
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> a = np.arange(10)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> b = np.where(a < 5)
>>> b
(array([0, 1, 2, 3, 4]),)
>>> c = np.where(2 < a < 7)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Defining b works as I want and expect. The array contains the indices (not the values) of a where a < 5.

For my definition of c, I expect (array([3, 4, 5, 6]),). As you can see, I get a ValueError instead. I have seen the error message about "the truth value of an array with more than one element" before, and generally I understand how I (accidentally) provoke it. This time, I don't see it. In defining c, I expect to be stepping through a, one element at a time, just as I did when defining b.

Does anyone understand why this happens? Is there a smart work-around? Thanks.




== 2 of 4 ==
Date: Fri, Jan 17 2014 6:16 pm
From: duncan smith


On 18/01/14 01:51, John Ladasky wrote:
> Hi folks,
>
> I am awaiting my approval to join the numpy-discussion mailing list, at scipy.org. I realize that would be the best place to ask my question. However, numpy is so widely used, I figure that someone here would be able to help.
>
> I like to use numpy.where() to select parts of arrays. I have encountered what I would consider to be a bug when you try to use where() in conjunction with the multiple comparison syntax of Python. Here's a minimal example:
>
> Python 3.3.2+ (default, Oct 9 2013, 14:50:09)
> [GCC 4.8.1] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import numpy as np
>>>> a = np.arange(10)
>>>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>>> b = np.where(a < 5)
>>>> b
> (array([0, 1, 2, 3, 4]),)
>>>> c = np.where(2 < a < 7)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
>
> Defining b works as I want and expect. The array contains the indices (not the values) of a where a < 5.
>
> For my definition of c, I expect (array([3, 4, 5, 6]),). As you can see, I get a ValueError instead. I have seen the error message about "the truth value of an array with more than one element" before, and generally I understand how I (accidentally) provoke it. This time, I don't see it. In defining c, I expect to be stepping through a, one element at a time, just as I did when defining b.
>
> Does anyone understand why this happens? Is there a smart work-around? Thanks.
>


>>> a = np.arange(10)
>>> c = np.where((2 < a) & (a < 7))
>>> c
(array([3, 4, 5, 6]),)
>>>

Duncan




== 3 of 4 ==
Date: Fri, Jan 17 2014 8:00 pm
From: John Ladasky


On Friday, January 17, 2014 6:16:28 PM UTC-8, duncan smith wrote:

> >>> a = np.arange(10)
> >>> c = np.where((2 < a) & (a < 7))
> >>> c
> (array([3, 4, 5, 6]),)

Nice! Thanks!

Now, why does the multiple comparison fail, if you happen to know?





== 4 of 4 ==
Date: Sat, Jan 18 2014 12:50 am
From: Peter Otten <__peter__@web.de>


John Ladasky wrote:

> On Friday, January 17, 2014 6:16:28 PM UTC-8, duncan smith wrote:
>
>> >>> a = np.arange(10)
>> >>> c = np.where((2 < a) & (a < 7))
>> >>> c
>> (array([3, 4, 5, 6]),)
>
> Nice! Thanks!
>
> Now, why does the multiple comparison fail, if you happen to know?

2 < a < 7

is equivalent to

2 < a and a < 7

Unlike `&` `and` cannot be overridden (*), so the above implies that the
boolean value bool(2 < a) is evaluated. That triggers the error because the
numpy authors refused to guess -- and rightly so, as both implementable
options would be wrong in a common case like yours.

(*) I assume overriding would collide with short-cutting of boolean
expressions.






==============================================================================
TOPIC: Python Scalability TCP Server + Background Game
http://groups.google.com/group/comp.lang.python/t/8234bf03949a8164?hl=en
==============================================================================

== 1 of 2 ==
Date: Fri, Jan 17 2014 11:44 pm
From: phiwer@gmail.com


> Quick smoke test. How big are your requests/responses? You mention
>
> REST, which implies they're going to be based on HTTP. I would expect
>
> you would have some idea of the rough size. Multiply that by 50,000,
>
> and see whether your connection can handle it. For instance, if you
>
> have a 100Mbit/s uplink, supporting 50K requests/sec means your
>
> requests and responses have to fit within about 256 bytes each,
>
> including all overhead. You'll need a gigabit uplink to be able to
>
> handle a 2KB request or response, and that's assuming perfect
>
> throughput. And is 2KB enough for you?
>
>
>
> ChrisA

My assumption is that there will be mostly reads and some writes; maybe in the order of 80-20%. There is a time element in the game, which forces player's entity to update on-demand. This is part of the reason why I wanted the server to be able to handle so many reques, so that it could handle the read part without having any caching layer.

Let me explain a bit more about the architecture, and possible remedies, to give you an idea:

* On top are the web servers exposing a REST api.

* At the bottom is the game.

* Communication betweeen these layers is handled by a simple text protocol using TCP.

The game has a tick function every now and then, which forwards the game's time. If a player enters the game, a message is sent to the game server (querying for player details), and if the game's tick is greater than the cached version of the player details, then the game updates the player details (and caches it).

This design obviously has its flaws. One being that both reads/writes has to pass through the game server. One way to remedy this is by using a new layer, on top of the game, which would hold the cache. But then another problem arises, that of invalidating the cache when a new tick has been made. I'm leaning towards letting the cache layer check the current tick every now and then, and if new tick is available, update a local variable in the cache (which each new connection would check against). Any thoughts regarding this?

There are some periods in the game, where many people will be online during the same tick, which could potentially cause the game to become slow at times, but maybe this should be accepted for the pleasure of making the game in python... :D

A follow-up question (which is more to the point really): How does other python game development frameworks solve this issue? Do they not use greenlets for the network layer, to be able to use the shared Queue from multiprocess? Do they only use one process for both network and game operations?

On a side note, I'm considering using 0MQ for the message layering between services (web-server <-> cache <-> game) on the back-end. Besides being a great message protocol, it also has built in queues which might be able to remedy the situation when many clients are requesting data. Anyone with experience with regards to this?

(This problem can be boggled down multiple producer, single consumer, and then back to producer again).

Thanks for all the replies.

/Phil




== 2 of 2 ==
Date: Sat, Jan 18 2014 12:01 am
From: Chris Angelico


On Sat, Jan 18, 2014 at 6:44 PM, <phiwer@gmail.com> wrote:
>> Quick smoke test. How big are your requests/responses? You mention
>>
>> REST, which implies they're going to be based on HTTP. I would expect
>>
>> you would have some idea of the rough size. Multiply that by 50,000,
>>
>> and see whether your connection can handle it. For instance, if you
>>
>> have a 100Mbit/s uplink, supporting 50K requests/sec means your
>>
>> requests and responses have to fit within about 256 bytes each,
>>
>> including all overhead. You'll need a gigabit uplink to be able to
>>
>> handle a 2KB request or response, and that's assuming perfect
>>
>> throughput. And is 2KB enough for you?
>>
>>
>>
>> ChrisA
>
> My assumption is that there will be mostly reads and some writes; maybe in the order of 80-20%. There is a time element in the game, which forces player's entity to update on-demand. This is part of the reason why I wanted the server to be able to handle so many reques, so that it could handle the read part without having any caching layer.
>

(You're using Google Groups, which means your replies are
double-spaced and your new text is extremely long lines. Please fix
this, either by the fairly manual job of fixing every post you make,
or the simple method of switching to a better client. Thanks.)

My point was just about the REST API, nothing else. You have to handle
a request and a response for every API call. Whether they're reads or
writes, you need to receive an HTTP request and send an HTTP response
for each one. In order to support the 50k requests per second you hope
for, you would have to handle 50k requests coming in and 50k responses
going out. To do that, you would need - at a very VERY rough estimate
- a maximum request size of 2KB and a gigabit internet connection
(which is expensive). No further calculations are worth pursuing if
you can't handle those requests and responses.

(And small requests tend to be more expensive than large ones. Since
you'll need a minimum of >SYN, <SYN/ACK, >ACK, >DATA, <DATA, >FIN,
<FIN/ACK, >ACK in order to transmit and receive one single packet's
worth of data each way, you're looking at roughly 8 packets minimum
for a one-byte message and one-byte response. But at a very very rough
estimate, 2KB each direction is the most you can sustain.)

ChrisA





==============================================================================
TOPIC: doctests compatibility for python 2 & python 3
http://groups.google.com/group/comp.lang.python/t/e83b94c8f7e04717?hl=en
==============================================================================

== 1 of 1 ==
Date: Sat, Jan 18 2014 12:39 am
From: Albert-Jan Roskam


--------------------------------------------
On Fri, 1/17/14, Terry Reedy <tjreedy@udel.edu> wrote:

Subject: Re: doctests compatibility for python 2 & python 3
To: python-list@python.org
Date: Friday, January 17, 2014, 10:10 PM

On 1/17/2014 7:14 AM, Robin Becker
wrote:

> I tried this approach with a few more complicated
outcomes and they fail
> in python2 or 3 depending on how I try to render the
result in the doctest.

I never got how you are using doctests. There were certainly
not meant for heavy-duty unit testing, but for testing
combined with explanation. Section 26.2.3.7. (in 3.3)
Warnings warns that they are fragile to even single char
changes and suggests == as a workaround, as 'True' and
'False' will not change. So I would not reject that option.


=====> I used doctests in .txt files and I converted ALL of them when I wanted to make my code work for both Python 2 and 3. I tried to fix something like a dozen of them so they'd work in Python 2.7 and 3,3. but I found it just too cumbersome and time consuming. The idea of doctest is super elegant, but it is really only meant for testable documentation (maybe with sphinx). If you'd put all the (often boring, e.g. edge cases) test cases in docstrings, the .py file will look very cluttered. One thing that I missed in unittest was Ellipsis, but: https://pypi.python.org/pypi/gocept.testing/1.6.0 offers assertEllipsis and other useful stuff.

Albert-Jan







==============================================================================
TOPIC: Guessing the encoding from a BOM
http://groups.google.com/group/comp.lang.python/t/1b4501ed3e9e2af1?hl=en
==============================================================================

== 1 of 2 ==
Date: Sat, Jan 18 2014 1:41 am
From: Gregory Ewing


Chris Angelico wrote:
> On Fri, Jan 17, 2014 at 8:10 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
>
>Every time I see it I picture Inspector
>>Clouseau, "A BOM!!!" :)
>
> Special delivery, a berm! Were you expecting one?

A berm? Is that anything like a shrubbery?

--
Greg




== 2 of 2 ==
Date: Sat, Jan 18 2014 2:10 am
From: Chris Angelico


On Sat, Jan 18, 2014 at 8:41 PM, Gregory Ewing
<greg.ewing@canterbury.ac.nz> wrote:
> Chris Angelico wrote:
>>
>> On Fri, Jan 17, 2014 at 8:10 PM, Mark Lawrence <breamoreboy@yahoo.co.uk>
>> wrote:
>>
>> Every time I see it I picture Inspector
>>>
>>> Clouseau, "A BOM!!!" :)
>>
>>
>> Special delivery, a berm! Were you expecting one?
>
>
> A berm? Is that anything like a shrubbery?

http://www.youtube.com/watch?v=GLDvtpSC_98

ChrisA




==============================================================================

You received this message because you are subscribed to the Google Groups "comp.lang.python"
group.

To post to this group, visit http://groups.google.com/group/comp.lang.python?hl=en

To unsubscribe from this group, send email to comp.lang.python+unsubscribe@googlegroups.com

To change the way you get mail from this group, visit:
http://groups.google.com/group/comp.lang.python/subscribe?hl=en

To report abuse, send email explaining the problem to abuse@googlegroups.com

==============================================================================
Google Groups: http://groups.google.com/?hl=en