python 3 niggle: None < 1 raises TypeError

[Chris Withers]

Hi All,

Sending this to python-dev as I'm wondering if this was considered when
the choice to have objects of different types raise a TypeError when
ordered...

So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that
can either be None or, for sake of argument, a numeric value.

To implement __lt__ in Python 2, I could do:

def __lt__(self, other):
if not isinstance(other, Range):
return True
return ((self._lower, self._upper, self._bounds) <
(other._lower, other._upper, other._bounds))

Because None < 1 raises a TypeError, in Python 3 I have to do:

def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented
for attr in '_lower', '_upper', '_bounds':
self_value = getattr(self, attr)
other_value = getattr(other, attr)
if self_value == other_value:
pass
elif self_value is None:
return True
elif other_value is None:
return False
else:
return self_value < other_value
return False

Am I missing something? How can I get this method down to a sane size?

cheers,

Chris

[Nick Coghlan]

Post by Chris Withers
Am I missing something? How can I get this method down to a sane size?

The easiest way is usually to normalise the attributes to a sensible
numeric value depending on where you want "None" to sort (positive and
negative infinity floating point values often work well in the case of
numeric data, but a custom AlwaysLess or AlwaysGreater type works for
arbitrary data). You can either do that dynamically, or else cache the
normalised values when the attributes are set.

Python 2 used to guess, Python 3 makes developers decide how they want
None to be handled in the context of ordering operations.

Cheers,
Nick.

[Chris Barker]

Post by Nick Coghlan

Post by Chris Withers
Am I missing something? How can I get this method down to a sane size?

The easiest way is usually to normalise the attributes to a sensible
numeric value depending on where you want "None" to sort (positive and
negative infinity floating point values often work well in the case of
numeric data, but a custom AlwaysLess or AlwaysGreater type works for
arbitrary data).

This is actually a pretty common case -- the question here is: Is this
really None? or does None have a special meaning. It sounds like this is a
wrapper for a PostgreSQL range object, in which case None isn't really
right, there must be a +-infinity concept there -- how does postgress
handle it? (is this a generic range, or a datetime range? I'm not up on it).
If this means what I think it does, None is really a hack, both because of
al lteh specia case code required for sorting, comparing, etc, but also
because it's missing information -- the two ends of the range should really
have different meanings.

I've dealt with this for Python datetime via a pari fo custom InfDateTime
objects one for plu infiinity, one for minus. Then we can use these in
ranges and the reest of teh code can jsut do simple comparisons, sorting
etc.

Of course, floating point numbers have inf and -inf, which is really useful.

I hadn't though of Nick's generic AlwaysLess or AlwaysGreater type -- if
that means what I think it does -- i,e, it would compare appropriately to
ANY other object, that would actually be a nice addition to
the standard library. I had been thinking that the standard
library datetime could use these, but why not just use generic ones?

(though it could get a bit tricky -- what would AlwaysGreater >
float('inf") evaluate to?

-Chris
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Angelico]

(though it could get a bit tricky -- what would AlwaysGreater > float('inf")
evaluate to?

It'd be true. AlwaysGreater is greater than infinity. It is greater
than float("nan"). It is greater than the entire concept of floating
point. It is greater than everything imaginable... except for a nice
MLT - a mutton, lettuce, and tomato sandwich... oh.

Where things get tricky is when you compare two AlwaysGreater objects.
Or even compare one against itself. It has to be greater.

Great.

ChrisA

[Terry Reedy]

The idea of top and bottom objects, by whatever name, has be proposed,
discussed, and rejected on python-ideas list (which is where this
discussion really belongs if continued).

Post by Chris Angelico

(though it could get a bit tricky -- what would AlwaysGreater > float('inf")
evaluate to?

It'd be true. AlwaysGreater is greater than infinity. It is greater
than float("nan"). It is greater than the entire concept of floating
point. It is greater than everything imaginable... except for a nice
MLT - a mutton, lettuce, and tomato sandwich... oh.
Where things get tricky is when you compare two AlwaysGreater objects.
Or even compare one against itself. It has to be greater.

equal, and less than itself. Or not, depending on exactly how the
methods are defined and coded. Which, as I remember, is part of why the
idea of a *generic* class was rejected. Real cases are better served by
custom classes that meet the custom need.

[Nick Coghlan]

Post by Terry Reedy
The idea of top and bottom objects, by whatever name, has be proposed,
discussed, and rejected on python-ideas list (which is where this discussion
really belongs if continued).

A fair point, but my recollection of those discussions is that we
completely missed the use case of being able to sensibly map SQL NULL
handling in ORDER BY clauses to the sorting of Python 3 containers.
Having a concrete use case with widespread implications makes a big
difference :)

Cheers,
Nick.

[Isaac Morland]

Post by Chris Barker
This is actually a pretty common case -- the question here is: Is this
really None? or does None have a special meaning. It sounds like this is a
wrapper for a PostgreSQL range object, in which case None isn't really
right, there must be a +-infinity concept there -- how does postgress handle
it? (is this a generic range, or a datetime range? I'm not up on it).
If this means what I think it does, None is really a hack, both because of
al lteh specia case code required for sorting, comparing, etc, but also
because it's missing information -- the two ends of the range should really
have different meanings.

Postgres range types are a separate kind of type which can be created
using CREATE TYPE:

http://www.postgresql.org/docs/current/interactive/sql-createtype.html

Several handy range types are built in, although not as many as one might
expect.

Postgres ranges can be made infinite by specifying NULL as a bound.
There isn't a separate special value for this. This has somewhat strange
effects when the base type of the range has its own "infinity" value:

=> select upper_inf (daterange (current_date, 'infinity'));
upper_inf
-----------
f
(1 row)

=> select isfinite ('infinity'::date);
isfinite
----------
f
(1 row)

=> select upper_inf (daterange (current_date, null));
upper_inf
-----------
t
(1 row)

In other words, a range with non-NULL bounds is finite according to the
range functions even if the bound is the infinity value of the base type.

Isaac Morland CSCF Web Guru
DC 2619, x36650 WWW Software Specialist

[Chris Angelico]

Post by Chris Withers
return True
return ((self._lower, self._upper, self._bounds) <
(other._lower, other._upper, other._bounds))
return NotImplemented
self_value = getattr(self, attr)
other_value = getattr(other, attr)
pass
return True
return False
return self_value < other_value
return False
Am I missing something? How can I get this method down to a sane size?

Can you be certain that all your values are either None or positive
integers? If so, try this:

def __lt__(self, other):
if not isinstance(other, Range):
return True # or NotImplemented, not sure why this change
return ((self._lower or 0, self._upper or 0, self._bounds or 0) <
(other._lower or 0, other._upper or 0, other._bounds or 0))

That'll treat all Nones as 0, and compare them accordingly. If you
can't depend on them being positive (eg if you need None to be less
than 0 rather than equal - even more so if you need it to be less than
negative numbers), you'll need to more explicitly check. But this is
nice and tidy, if it works.

ChrisA

[Lennart Regebro]

Post by Chris Withers
Hi All,
Sending this to python-dev as I'm wondering if this was considered when the
choice to have objects of different types raise a TypeError when ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that can
either be None or, for sake of argument, a numeric value.
return True
return ((self._lower, self._upper, self._bounds) <
(other._lower, other._upper, other._bounds))
return NotImplemented
self_value = getattr(self, attr)
other_value = getattr(other, attr)
pass
return True
return False
return self_value < other_value
return False
Am I missing something? How can I get this method down to a sane size?

It was considered. It's not obvious where you want "None" to appear in
your ordering, so you will have to implement this by yourself. I can't
come up with anything obviously shorter.

Or, well, this, but it's not exactly pretty:

def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented
ninf = float('-inf') # So None is lower than anything else.
return ((self._lower if self._lower is not None else ninf,
self._upper if self._upper is not None else ninf,
self._bounds if self._bounds is not None else ninf) < (
other._lower if other._lower is not None else ninf,
other._upper if other._upper is not None else ninf,
other._bounds if other._bounds is not None else ninf))

Or maybe:

def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented
s = (self._lower, self._upper, self._bounds)
if None is s:
return True
o = (other._lower, other._upper, other._bounds)
if None in o:
return False
return s < o

[Antoine Pitrou]

On Fri, 14 Feb 2014 10:46:50 +0100

Post by Lennart Regebro

Post by Chris Withers
Sending this to python-dev as I'm wondering if this was considered when the
choice to have objects of different types raise a TypeError when ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that can
either be None or, for sake of argument, a numeric value.

[...]

Post by Lennart Regebro
It was considered. It's not obvious where you want "None" to appear in
your ordering, so you will have to implement this by yourself. I can't
come up with anything obviously shorter.

I have to agree with Lennart. The fact that SQL defines an order for
NULL and other values doesn't mean it's obvious or right in any way (I
never remember which way it goes).

Regards

Antoine.

[Nick Coghlan]

Received: from localhost (HELO mail.python.org) (127.0.0.1)
by albatross.python.org with SMTP; 14 Feb 2014 11:13:50 +0100
Received: from mail-qa0-x22a.google.com (unknown
[IPv6:2607:f8b0:400d:c00::22a])
(using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
(No client certificate requested)
by mail.python.org (Postfix) with ESMTPS
for <python-***@python.org>; Fri, 14 Feb 2014 11:13:49 +0100 (CET)
Received: by mail-qa0-f42.google.com with SMTP id k4so18289723qaq.29
for <python-***@python.org>; Fri, 14 Feb 2014 02:13:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:cc:content-type;
bh=xYVqaLieqsDBNMv5XC7GI61o3+NmgLnpgXH0GQn7Ooo=;
b=aso9L8bjES0luEKJzOGnnTpXuEcnisTHDoNqWgHXgVC4jpKhqdDYgZVemHPnGCchfy
Bp4DMayOOobIKa2iBL18JNi+IjQTX4jLvB8UvEdrn5j0wbdjB4+qh9Lr9xf/qvRjmLUd
4yhEgFzNRAhhEpDcO/im1SZyAGaWyXMtflX/ST/a5DEX26/0Pg11b386SNUC5N1cD3vQ
QRVq/TxIZpR5d4Rf77sZYrnv3lDCppMGyBB+NJfRDkXSYNGBVQHUOwKBSWqUdg2CZYdt
8RvphdXZVf/2nwlXxcpVXSxotJOFSAKXdxjZyXaWEDSMm5FuOxKeq9e/tKloeKQ9bM+b
KoAw==
X-Received: by 10.224.156.68 with SMTP id v4mr11399653qaw.66.1392372823191;
Fri, 14 Feb 2014 02:13:43 -0800 (PST)
Received: by 10.224.95.136 with HTTP; Fri, 14 Feb 2014 02:13:43 -0800 (PST)
In-Reply-To: <***@fsol>
X-BeenThere: python-***@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Python core developers <python-dev.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-dev>,
<mailto:python-dev-***@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-dev/>
List-Post: <mailto:python-***@python.org>
List-Help: <mailto:python-dev-***@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-dev>,
<mailto:python-dev-***@python.org?subject=subscribe>
Errors-To: python-dev-bounces+python-python-dev=***@python.org
Sender: "Python-Dev"
<python-dev-bounces+python-python-dev=***@python.org>
Archived-At: <http://permalink.gmane.org/gmane.comp.python.devel/145510>

Post by Antoine Pitrou
On Fri, 14 Feb 2014 10:46:50 +0100

Post by Lennart Regebro

Post by Chris Withers
Sending this to python-dev as I'm wondering if this was considered when the
choice to have objects of different types raise a TypeError when ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that can
either be None or, for sake of argument, a numeric value.

[...]

Post by Lennart Regebro
It was considered. It's not obvious where you want "None" to appear in
your ordering, so you will have to implement this by yourself. I can't
come up with anything obviously shorter.

I have to agree with Lennart. The fact that SQL defines an order for
NULL and other values doesn't mean it's obvious or right in any way (I
never remember which way it goes).

SQL doesn't define an order for NULL, it's more like a "quiet NaN" -
if either operand is NULL, the result is also NULL. (I ran into this
recently, in the context of "NULL == value" and "NULL != value" both
being effectively false).

We could theoretically do something similar, but the NULL handling in
SQL is a frequent source of bugs in filter definitions, so that really
doesn't sound like a good idea.

Cheers,
Nick.

[Antoine Pitrou]

On Fri, 14 Feb 2014 20:13:43 +1000

Post by Nick Coghlan

Post by Antoine Pitrou
On Fri, 14 Feb 2014 10:46:50 +0100

Post by Lennart Regebro

Post by Chris Withers
Sending this to python-dev as I'm wondering if this was considered when the
choice to have objects of different types raise a TypeError when ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that can
either be None or, for sake of argument, a numeric value.

[...]

Post by Lennart Regebro
It was considered. It's not obvious where you want "None" to appear in
your ordering, so you will have to implement this by yourself. I can't
come up with anything obviously shorter.

I have to agree with Lennart. The fact that SQL defines an order for
NULL and other values doesn't mean it's obvious or right in any way (I
never remember which way it goes).

SQL doesn't define an order for NULL, it's more like a "quiet NaN" -
if either operand is NULL, the result is also NULL. (I ran into this
recently, in the context of "NULL == value" and "NULL != value" both
being effectively false).

Hmm, it seems you're right, but I'm quite sure some DBMSes have a
consistent way of ordering NULLs when using ORDER BY on a nullable
column.

Regards

Antoine.

[Paul Moore]

Post by Antoine Pitrou
Hmm, it seems you're right, but I'm quite sure some DBMSes have a
consistent way of ordering NULLs when using ORDER BY on a nullable
column.

ORDER BY xxx [NULLS FIRST|LAST] is the syntax in Oracle, with (IIRC)
NULLS LAST as default. But I agree, this is not an argument in favour
of doing the same in Python.

Paul

[Nick Coghlan]

Post by Paul Moore

Post by Antoine Pitrou
Hmm, it seems you're right, but I'm quite sure some DBMSes have a
consistent way of ordering NULLs when using ORDER BY on a nullable
column.

ORDER BY xxx [NULLS FIRST|LAST] is the syntax in Oracle, with (IIRC)
NULLS LAST as default. But I agree, this is not an argument in favour
of doing the same in Python.

IIRC, MySQL and PostgreSQL sort them in the opposite order from each
other (or it's possibly just Teiid's PostgreSQL emulation that is the
opposite of MySQL).

Either way, it nicely illustrates *why* we didn't grant None an
exemption from the "no implicit cross-type ordering operations" in
3.x. It does make *adapting* to other models like SQL a bit more
painful though. That's the main reason I still occasionally wonder if
"AlwaysMin" and "AlwaysMax" singletons might make sense, although that
approach has problems of its own (specifically, it's hard to actually
use without breaking the clean NULL -> None mapping and in many cases
+/- infinity already work fine).

Cheers,
Nick.

[Chris Angelico]

Post by Nick Coghlan
IIRC, MySQL and PostgreSQL sort them in the opposite order from each
other

Ouch! You're right:

http://dev.mysql.com/doc/refman/5.7/en/working-with-null.html

"""When doing an ORDER BY, NULL values are presented first if you do
ORDER BY ... ASC and last if you do ORDER BY ... DESC."""

Not configurable in MySQL. It's apparently not part of the standard,
according to Wikipedia:

http://en.wikipedia.org/wiki/Order_by

So definitely SQL's handling of NULL should not be any sort of guide
as regards Python's treatment of None.

ChrisA

[Oleg Broytman]

Post by Chris Angelico
So definitely SQL's handling of NULL should not be any sort of guide
as regards Python's treatment of None.

Why not? Just make the order different for CPython and PyPy, Cython
and Jython. ;-)

Oleg.

[yoav glazner]

Post by Oleg Broytman

Post by Chris Angelico
So definitely SQL's handling of NULL should not be any sort of guide
as regards Python's treatment of None.

Why not? Just make the order different for CPython and PyPy, Cython
and Jython. ;-)
Oleg.
--
Programmers don't die, they just GOSUB without RETURN.

Unicode + bytes explodes in order to get the error sooner.i.e
Not waiting for the killer 8bit string.

So None<1 is the opposite !
sort(somelist) is a ticking bomb in py3

[Chris Barker]

Post by Nick Coghlan
That's the main reason I still occasionally wonder if
"AlwaysMin" and "AlwaysMax" singletons might make sense, although that
approach has problems of its own (specifically, it's hard to actually
use without breaking the clean NULL -> None mapping and in many cases
+/- infinity already work fine).

only for floating point -- it would be nice to have them for other types:
integers, datetimes, any others that make sense.... ( In know I've wanted
+-inf and NaN for integers forever, though what I really want is hardware
support)

But do you do that with a generic type or a special one for each type?

-Chris
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Chris Angelico]

Post by Antoine Pitrou
Hmm, it seems you're right, but I'm quite sure some DBMSes have a
consistent way of ordering NULLs when using ORDER BY on a nullable
column.

Yes, and I believe it's part of the SQL-92 spec. Certainly here's
PostgreSQL's take on the matter:

http://www.postgresql.org/docs/9.3/static/sql-select.html#SQL-ORDERBY

In short, NULL is by default considered greater than anything else
(last in an ascending sort, first in a descending sort), but this can
be inverted. Oddly enough, the syntax is NULLS FIRST vs NULLS LAST,
not NULLS LOW vs NULLS HIGH.

ChrisA

[M.-A. Lemburg]

Post by Antoine Pitrou
On Fri, 14 Feb 2014 20:13:43 +1000

Post by Nick Coghlan

Post by Antoine Pitrou
On Fri, 14 Feb 2014 10:46:50 +0100

Post by Lennart Regebro

Post by Chris Withers
Sending this to python-dev as I'm wondering if this was considered when the
choice to have objects of different types raise a TypeError when ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that can
either be None or, for sake of argument, a numeric value.

[...]

Post by Lennart Regebro
It was considered. It's not obvious where you want "None" to appear in
your ordering, so you will have to implement this by yourself. I can't
come up with anything obviously shorter.

I have to agree with Lennart. The fact that SQL defines an order for
NULL and other values doesn't mean it's obvious or right in any way (I
never remember which way it goes).

SQL doesn't define an order for NULL, it's more like a "quiet NaN" -
if either operand is NULL, the result is also NULL. (I ran into this
recently, in the context of "NULL == value" and "NULL != value" both
being effectively false).

Hmm, it seems you're right, but I'm quite sure some DBMSes have a
consistent way of ordering NULLs when using ORDER BY on a nullable
column.

They do, but it's not consistent:

MS SQL Server: orders NULLs first (in table order; stable sort)
Sybase ASE: orders NULLs first (in arbitrary order)
Oracle: orders NULLs last (in arbitrary order)
PostgreSQL: orders NULLs last (in arbitrary order)
IBM DB2: orders NULLs last (in table order; stable sort)

Reference: tests done with actual databases.

A note about consistency: None is always ordered first in Python 2,
so there is a precedent. And since Python's list.sort() is stable,
Python 2 is in the same camp as MS SQL Server.
/* None is smaller than anything */
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;

IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

Post by Antoine Pitrou

Post by Nick Coghlan

Post by Antoine Pitrou
l = [1,2,None,4,5,None,6]
l.sort()
l

[None, None, 1, 2, 4, 5, 6]

Post by Antoine Pitrou

Post by Nick Coghlan

Post by Antoine Pitrou
l = [1,2,None,4,5,None,6]
l.sort()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: NoneType() < int()
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 14 2014)

Post by Antoine Pitrou

Post by Nick Coghlan

Post by Antoine Pitrou
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Peter Otten]

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

Post by Antoine Pitrou
l = [1,2,None,4,5,None,6]
l.sort()
l

[None, None, 1, 2, 4, 5, 6]

Post by Antoine Pitrou
l = [1,2,None,4,5,None,6]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: NoneType() < int()

While it is trivial to fix

Post by M.-A. Lemburg

sorted([1,2,None,4,5,None,6],

... key=lambda x: (x is None, x))
[1, 2, 4, 5, 6, None, None]

maybe the key should be given a name like functools.none_first/none_last in
order to offer a standard approach?

On the other hand I'm not sure I like

none_last(x) < none_last(y)

[Nick Coghlan]

Post by Peter Otten

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

Post by Antoine Pitrou
l = [1,2,None,4,5,None,6]
l.sort()
l

[None, None, 1, 2, 4, 5, 6]

Post by Antoine Pitrou
l = [1,2,None,4,5,None,6]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: NoneType() < int()

While it is trivial to fix

Post by M.-A. Lemburg

sorted([1,2,None,4,5,None,6],

... key=lambda x: (x is None, x))
[1, 2, 4, 5, 6, None, None]
maybe the key should be given a name like functools.none_first/none_last in
order to offer a standard approach?

Yeah, I was thinking something similar. I posted an RFE if anyone
wants to run with the idea for Python 3.5:
http://bugs.python.org/issue20630 (it's also the kind of thing that is
quite amenable to inclusion in libraries like six and future for
Python 2/3 compatible code bases - in Python 2, it can just be a pass
through, while in Python 3 it can actually alter the sort operation to
allow None values).

At a broaded scale, this thread made me realise it may be worth
defining a __key__ protocol so custom comparisons are easier to
implement correctly: http://bugs.python.org/issue20632

That would need a PEP, but currently, there are lots of things you
need to deal with around hashability, comparison with other types, etc
when attempting to define a custom ordering. A new __key__ protocol
could provide sensible default behaviour for all of those, such that
you only needed to define the other methods if you wanted to override
the defaults for some reason. (As noted in the issue, this is also
amenable to being implemented as a third party module prior to
proposing it as a language feature)

Cheers,
Nick.

[Lennart Regebro]

What I think is the biggest nitpick here is that you can't make a NULL
object so that

NULL is None

evaluates as True. If you could, you could then easily make this NULL
object evaluate as less than everything except itself and None, and
use that consistently. But since this NULL is not None, that breaks
any backwards compatibility if your SQL library mapped NULL to None in
earlier versions.

[Stephen J. Turnbull]

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.

The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

[M.-A. Lemburg]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

Python 3 breaks this notion by always raising an exception when
using None in an ordered comparison, making it pretty much useless
for the above purpose.

Yes, there are ways around this, but none of them are intuitive.

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Gustavo Carneiro]

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.
Python 3 breaks this notion by always raising an exception when
using None in an ordered comparison, making it pretty much useless
for the above purpose.
Yes, there are ways around this, but none of them are intuitive.

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()

Maybe Python 3 should have a couple of None-like objects that compare the
way you want: AlwaysComparesLess and AlwaysComparesGreater, but with better
names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None alone,
please.

Post by M.-A. Lemburg
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com

[M.-A. Lemburg]

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.
Python 3 breaks this notion by always raising an exception when
using None in an ordered comparison, making it pretty much useless
for the above purpose.
Yes, there are ways around this, but none of them are intuitive.

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()

Maybe Python 3 should have a couple of None-like objects that compare the
way you want: AlwaysComparesLess and AlwaysComparesGreater, but with better
names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None alone,
please.

This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,
based on the intuition that nothing is less than anything :-)

FWIW, I don't think we need to invent a new name for it, just add
an appropriate tp_richcompare slot to the PyNoneType or readd the
special case to Object/object.c. This would also aid in porting
existing Python 2 code to Python 3.
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Stephen J. Turnbull
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Gustavo Carneiro]

Post by Terry Reedy

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.
Python 3 breaks this notion by always raising an exception when
using None in an ordered comparison, making it pretty much useless
for the above purpose.
Yes, there are ways around this, but none of them are intuitive.

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()

Maybe Python 3 should have a couple of None-like objects that compare the
way you want: AlwaysComparesLess and AlwaysComparesGreater, but with

better

Post by Gustavo Carneiro
names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None

alone,

Post by Gustavo Carneiro
please.

This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,
based on the intuition that nothing is less than anything :-)
I still think that relying on your intuition is not the right way for

Python. Refuse the temptation to guess.

Post by Terry Reedy
FWIW, I don't think we need to invent a new name for it, just add
an appropriate tp_richcompare slot to the PyNoneType or readd the
special case to Object/object.c. This would also aid in porting
existing Python 2 code to Python 3.

Based on your comment, SortsFirst and SortsLast sound like good names ;-)

These would be "universal sortable objects", that could be compared to any
other type.

Post by Terry Reedy
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Stephen J. Turnbull
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Paul Moore]

Post by Gustavo Carneiro

Post by M.-A. Lemburg
FWIW, I don't think we need to invent a new name for it, just add
an appropriate tp_richcompare slot to the PyNoneType or readd the
special case to Object/object.c. This would also aid in porting
existing Python 2 code to Python 3.

Based on your comment, SortsFirst and SortsLast sound like good names ;-)
These would be "universal sortable objects", that could be compared to any
other type.

I think that having both is over-engineered. For the use cases I know
of, all that is wanted is *one* object that doesn't raise a TypeError
when compared with any other object, and compares predictably (always
before or always after, doesn't matter which).

Whether that object is None, or whether it should be a new singleton,
is not obvious. The advantage of None is that it's Python 2 compatible
(which aids porting as noted). The advantage of a new object is that
not all uses of None need universal sortability, and indeed in some
cases universal sortability will hide bugs.

Paul

[M.-A. Lemburg]

Post by Terry Reedy

Post by Terry Reedy

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.
Python 3 breaks this notion by always raising an exception when
using None in an ordered comparison, making it pretty much useless
for the above purpose.
Yes, there are ways around this, but none of them are intuitive.

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

Post by Stephen J. Turnbull

Post by M.-A. Lemburg

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()

Maybe Python 3 should have a couple of None-like objects that compare the
way you want: AlwaysComparesLess and AlwaysComparesGreater, but with

better

Post by Gustavo Carneiro
names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None

alone,

Post by Gustavo Carneiro
please.

This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,
based on the intuition that nothing is less than anything :-)

I still think that relying on your intuition is not the right way for
Python. Refuse the temptation to guess.

Hmm, are you sure ?

There should be one-- and preferably only one --obvious way to do it.

and then

Although that way may not be obvious at first unless you're Dutch.

which directly relates to:

changeset: 16123:f997ded4e219
branch: legacy-trunk
user: Guido van Rossum <***@python.org>
date: Mon Jan 22 19:28:09 2001 +0000
summary: New special case in comparisons: None is smaller than any other object

:-)

Post by Terry Reedy

Post by Terry Reedy
FWIW, I don't think we need to invent a new name for it, just add
an appropriate tp_richcompare slot to the PyNoneType or readd the
special case to Object/object.c. This would also aid in porting
existing Python 2 code to Python 3.

Based on your comment, SortsFirst and SortsLast sound like good names ;-)
These would be "universal sortable objects", that could be compared to any
other type.

Of course, it's easy to add a new type for this, but a lot of Python 2
code relies on None behaving this way, esp. code that reads data from
databases, since None is the Python mapping for SQL NULL.

[Serhiy Storchaka]

Post by M.-A. Lemburg
Of course, it's easy to add a new type for this, but a lot of Python 2
code relies on None behaving this way, esp. code that reads data from
databases, since None is the Python mapping for SQL NULL.

At the same time a lot of Python 2 relies on the assumption that any two
values are comparable. I suppose that I see a code that try to sort a
heterogeneous list more often than a code which want to sort a list
containing only numbers and Nones.

[Serhiy Storchaka]

Post by Serhiy Storchaka

Post by M.-A. Lemburg
Of course, it's easy to add a new type for this, but a lot of Python 2
code relies on None behaving this way, esp. code that reads data from
databases, since None is the Python mapping for SQL NULL.

At the same time a lot of Python 2 relies on the assumption that any two
values are comparable. I suppose that I see a code that try to sort a
heterogeneous list more often than a code which want to sort a list
containing only numbers and Nones.

Sorry, I press "Send" too early.

If we consider the idea to make None universally comparable, we should
consider the idea to make all other values universally comparable too.
And consider again the reasons which caused these changes in Python 3.

[Terry Reedy]

Post by Serhiy Storchaka

Post by M.-A. Lemburg
Of course, it's easy to add a new type for this, but a lot of Python 2
code relies on None behaving this way, esp. code that reads data from
databases, since None is the Python mapping for SQL NULL.

At the same time a lot of Python 2 relies on the assumption that any two
values are comparable.

That assumption was very intentionally broken and abandoned by Guido
when complex numbers were added over a decade ago. It was further
abandoned, intentionally, when datetimes were added. I think it was
decided then to finish the job in 3.0.

I suppose that I see a code that try to sort a

Post by Serhiy Storchaka
heterogeneous list more often than a code which want to sort a list
containing only numbers and Nones.

[Tim Peters]

The behavior of None in comparisons is intentional in Python 3. You
can agitate to change it, but it will probably die when Guido gets
wind of it ;-)

The catch-all mixed-type comparison rules in Pythons 1 and 2 were only
intended to be "arbitrary but consistent". Of course each specific
release picked some specific scheme, but these schemes weren't
documented, and intentionally not.

For "a long time" CPython's default for mixed-typed comparisons was to
compare the names of the types (as strings). And for just as long,
nobody had noticed that this _wasn't_ always "consistent".

For example, 3L < 4 must be true. But [] < 3L was also true (because
"list" < "long"), and 4 < [] was true too (because "int" < "list").
So, in all,

[] < 3L < 4 < []

was true, implying [] < [].

Part of fixing that was removing some cases of "compare objects of
different types by comparing the type name strings". Guido & I were
both in the office at the time, and one said to the other: "what
about None? Comparing that to other types via comparing the string
'None' doesn't make much sense." "Ya, OK ... how about changing None
to - by default - comparing 'less than' objects of other types?"
"Don't see why not - sure." "OK, done!".

No more than 2 minutes of thought went into it. There was no intent
to cater to any real use case here - the only intent was to pick some
arbitrary-but-consistent rule that didn't suck _quite_ as badly as
pretending None was the string "None" ;-)

Guido wanted to drop all the "arbitrary but consistent" mixed-type
comparison crud for Python 3. Nothing special about None in that. As
already noted, the various `datetime` types were the first to
experiment with implementing full blown Python3-ish mixed-type
comparison rules. After the first two times that caught actual bugs
in code using the new types, there was no turning back. It's not so
much that Python 3 finished the job as that Python 2 started it ;-)

much-ado-about-nonething-ly y'rs - tim

[M.-A. Lemburg]

Post by Tim Peters
[...]
Guido wanted to drop all the "arbitrary but consistent" mixed-type
comparison crud for Python 3. Nothing special about None in that. As
already noted, the various `datetime` types were the first to
experiment with implementing full blown Python3-ish mixed-type
comparison rules. After the first two times that caught actual bugs
in code using the new types, there was no turning back. It's not so
much that Python 3 finished the job as that Python 2 started it ;-)

Well, I guess it depends on how you look at it.

None worked as "compares less than all other objects" simply due
to the fact that None is a singleton and doesn't implement the
comparison slots (which for all objects not implementing rich
comparisons, meant that the fallback code triggered in Python 2).

In Python 3 the special casing was dropped and because None
still doesn't implement the comparison slot (tp_richcompare this time),
it doesn't support ordering comparisons anymore.

Now, the choice to have None compare less than all other objects
may have been arbitrary, but IMO it was a good, consistent and
useful choice.

So why not bring it back and perhaps this time in a way that
actually does work consistently for all Python objects by
implementing the tp_richcompare slot on PyNoneType objects
and documenting it ?!
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Tim Peters

Post by Antoine Pitrou
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Tim Peters]

[M.-A. Lemburg]

Post by M.-A. Lemburg
...
None worked as "compares less than all other objects" simply due
to the fact that None is a singleton and doesn't implement the
comparison slots (which for all objects not implementing rich
comparisons, meant that the fallback code triggered in Python 2).

And the fallback code - which you've already found - went on to
special-case the snot out of None to make "less than" work. It's not
like it was "a natural" outcome: it was forced.

Post by M.-A. Lemburg
In Python 3 the special casing was dropped and because None
still doesn't implement the comparison slot (tp_richcompare this time),
it doesn't support ordering comparisons anymore.

Which makes mixed-type non-numeric default comparisons involving None
work the same way all other mixed-type non-numeric default comparisons
work in Python 3. This was the result of a deliberate design decision
about how comparison _should_ work in Python 3, not an accidental
consequence of the None type not defining tp_richcompare.

IOW, this is about design, not about implementation. The differing
implementations you see are consequences of the differing designs.
Staring at the implementations is irrelevant to the design decisions.

Post by M.-A. Lemburg
Now, the choice to have None compare less than all other objects
may have been arbitrary, but IMO it was a good, consistent and
useful choice.

Possibly useful for some apps, sure. Not for my apps. For example,
when I initialize an object attribute to None in Python 3, I _expect_
I'll get an exception if I try to use that attribute in most contexts.
It makes no more sense to ask whether that attribute is, say, less
than 3, than it does to add 3 to it. The exception is most useful
then. More often than not, it was annoying to me that Python 2
_didn't_ whine about trying to compare None.

Post by M.-A. Lemburg
So why not bring it back

A huge barrier is (or should be) that Python 3 is over 5 years old
now. Fiddling with the semantics of basic builtin types was possible
- and even welcome - 6 years ago. Now they really shouldn't be
touched in the absence of a critical bug, or a wholly
backward-compatible (with Python 3) addition.

Post by M.-A. Lemburg
and perhaps this time in a way that actually does work consistently for
all Python objects by implementing the tp_richcompare slot on
PyNoneType objects and documenting it ?!

Something to suggest for Python 4, in which case I'll only be -0 ;-)

[M.-A. Lemburg]

Post by Tim Peters
[M.-A. Lemburg]

Post by M.-A. Lemburg
Now, the choice to have None compare less than all other objects
may have been arbitrary, but IMO it was a good, consistent and
useful choice.

Possibly useful for some apps, sure. Not for my apps. For example,
when I initialize an object attribute to None in Python 3, I _expect_
I'll get an exception if I try to use that attribute in most contexts.
It makes no more sense to ask whether that attribute is, say, less
than 3, than it does to add 3 to it. The exception is most useful
then. More often than not, it was annoying to me that Python 2
_didn't_ whine about trying to compare None.

Yes, I see your point, but please also consider that None is
used in database applications as natural mapping for SQL NULL
and in other data processing applications to mean "no value".

This is not garbage data, it's just missing information for
a particular field and you can still happily process such data
without having the information for these fields as values
(indeed in some cases, the information whether a field has a
value or not is important, e.g. for data validations and to cross
check entries).

You do still want to be able to sort such data, but as it stands,
Python 3 doesn't allow you to, without adding an extra layer
of protection against None values deep inside the structures
you're sorting.

Post by Tim Peters

Post by M.-A. Lemburg
So why not bring it back

A huge barrier is (or should be) that Python 3 is over 5 years old
now. Fiddling with the semantics of basic builtin types was possible
- and even welcome - 6 years ago. Now they really shouldn't be
touched in the absence of a critical bug, or a wholly
backward-compatible (with Python 3) addition.

Post by M.-A. Lemburg
and perhaps this time in a way that actually does work consistently for
all Python objects by implementing the tp_richcompare slot on
PyNoneType objects and documenting it ?!

Something to suggest for Python 4, in which case I'll only be -0 ;-)

Well, like I said: we'd be making something work again that has
worked before and only remove a failure case. The barrier for
entry should be lower in such a case.

This is similar to bringing back the u"" literals. Those used
to cause a SyntaxError and now they no longer do.

Even better: we have a chance to properly document the behavior
this time around and make it consistent all the way.

The alternative would be adding a new singleton to mean mostly
the same thing as None, but having the property of comparing
less than all other objects and then recommend its use in the
DB-API for Python 3 applications... which would break a lot
of existing DB-API based apps of course... which is the reason
why I'm advocating for changing None instead :-)
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 18 2014)

Post by Tim Peters

Post by M.-A. Lemburg

Post by Antoine Pitrou
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Brett Cannon]

I'll reply inline, but tl;dr: I'm with Tim on this.

Post by M.-A. Lemburg

Post by Tim Peters
[M.-A. Lemburg]

Post by M.-A. Lemburg
Now, the choice to have None compare less than all other objects
may have been arbitrary, but IMO it was a good, consistent and
useful choice.

Possibly useful for some apps, sure. Not for my apps. For example,
when I initialize an object attribute to None in Python 3, I _expect_
I'll get an exception if I try to use that attribute in most contexts.
It makes no more sense to ask whether that attribute is, say, less
than 3, than it does to add 3 to it. The exception is most useful
then. More often than not, it was annoying to me that Python 2
_didn't_ whine about trying to compare None.

Yes, I see your point, but please also consider that None is
used in database applications as natural mapping for SQL NULL
and in other data processing applications to mean "no value".

Fine, but not everything uses a database. =)

Post by M.-A. Lemburg
This is not garbage data, it's just missing information for
a particular field and you can still happily process such data
without having the information for these fields as values
(indeed in some cases, the information whether a field has a
value or not is important, e.g. for data validations and to cross
check entries).
You do still want to be able to sort such data, but as it stands,
Python 3 doesn't allow you to, without adding an extra layer
of protection against None values deep inside the structures
you're sorting.

Which in my opinion is fine as the boilerplate is typically minimal. No one
has said any solution is going to take even 10 lines of code to work around
this.

Post by M.-A. Lemburg

Post by Tim Peters

Post by M.-A. Lemburg
So why not bring it back

A huge barrier is (or should be) that Python 3 is over 5 years old
now. Fiddling with the semantics of basic builtin types was possible
- and even welcome - 6 years ago. Now they really shouldn't be
touched in the absence of a critical bug, or a wholly
backward-compatible (with Python 3) addition.

Post by M.-A. Lemburg
and perhaps this time in a way that actually does work consistently for
all Python objects by implementing the tp_richcompare slot on
PyNoneType objects and documenting it ?!

Something to suggest for Python 4, in which case I'll only be -0 ;-)

Well, like I said: we'd be making something work again that has
worked before and only remove a failure case. The barrier for
entry should be lower in such a case.

But this is still a semantic change. Python 3 users could very well be
relying on the fact that comparing against None raises an exception as a
way to detect when erroneous data has leaked into some data structure. What
is being suggested is a semantic change to a core data type within a major
release. We waited until Python 3 to make comparisons against disparate
types illegal for a reason.

Post by M.-A. Lemburg
This is similar to bringing back the u"" literals. Those used
to cause a SyntaxError and now they no longer do.

I don't think they are similar at all. Allowing the u prefix on strings
basically made something that was a typo or accidental hold-over from
Python 2 just not throw an error. There was no semantic shift in what u
meant in Python 3.2 vs. 3.3 which could break some code. Basically we just
made some syntactic fluff not upset the parser which in no way changed the
semantic meaning of the string literal it was attached to. Making None sort
a specific way is beyond just adding some syntactic fluff.

Post by M.-A. Lemburg
Even better: we have a chance to properly document the behavior
this time around and make it consistent all the way.
The alternative would be adding a new singleton to mean mostly
the same thing as None, but having the property of comparing
less than all other objects and then recommend its use in the
DB-API for Python 3 applications... which would break a lot
of existing DB-API based apps of course... which is the reason
why I'm advocating for changing None instead :-)

Or DB-API stuff needs to be updated to use some None-that-sorts singleton
if they want to use None to represent NULL but sort in a specific way.

As Tim has said, 3.0 has been out for five years, so we are talking about
breaking code for this over what can be viewed as a minor inconvenience for
DB code. The current situation is not insurmountable in any way for those
that want None to sort.

I should also mention I view None as representing nothing, which includes
not sharing a type with anything, which is why it can't be compared against
any other type in Python 3. But this suggestion of having None sort as the
lowest thing no matter what in way says None is implicitly every type when
it comes to sorting which breaks that mental model of None representing the
void of information by being void of value, but everything when it comes to
its type for comparison purposes.

[Glenn Linderman]

Post by M.-A. Lemburg
Yes, I see your point, but please also consider that None is
used in database applications as natural mapping for SQL NULL
and in other data processing applications to mean "no value".
Fine, but not everything uses a database. =)

Python None and SQL NULL share some properties, which makes it seem like
it is a natural mapping. However, they also have some differing
properties, which means that really it isn't a natural mapping.

A big mistake in most Python/SQL marriages is the failure to implement
an SQL NULL class that fully matches SQL semantics.

Of course it is easier to map SQL NULL to Python None and then complain
about the semantic differences.

[Lele Gaifax]

Post by Brett Cannon

Post by M.-A. Lemburg
Yes, I see your point, but please also consider that None is
used in database applications as natural mapping for SQL NULL
and in other data processing applications to mean "no value".

Fine, but not everything uses a database. =)

Right, and even if everything did, as already said not all databases
share the same default ordering rules, or have the same syntax to impose
one specific direction.

But more to the point, no database I know let you say "WHERE somecolumn
< 1" and expect that when "somecolumn IS NULL" the condition is true...

Paradoxically, if Python goes back to give a meaning to "None < 1", it
would have an even more different semantic than most database engines
out there.

ciao, lele.

[Ethan Furman]

Post by M.-A. Lemburg
This is not garbage data, it's just missing information for
a particular field [...]

I think the basic problem is that None was over-used (or even mis-used?) to the point where 3.0 had to make a choice to
bring consistency back to the language.

It seems to me what we really need is either a Null singleton to represent a data point with no known value but that can
still be sorted, or have every data type able to represent a no-value state (datetime, I'm looking at you!).

--
~Ethan~

[Serhiy Storchaka]

Post by Ethan Furman
It seems to me what we really need is either a Null singleton to
represent a data point with no known value but that can still be sorted,
or have every data type able to represent a no-value state (datetime,
I'm looking at you!).

A Never singleton?

[Greg Ewing]

Post by M.-A. Lemburg
The alternative would be adding a new singleton to mean mostly
the same thing as None, but having the property of comparing
less than all other objects and then recommend its use in the
DB-API for Python 3 applications...

Which I think would be a *really bad* idea, because
there would then no longer be One Obvious Way to
represent and process null values.

E.g. you would no longer be able to write
'value is None' and trust it to work on all kinds
of null value you're likely to get.

[Glenn Linderman]

Post by Greg Ewing

Post by M.-A. Lemburg
The alternative would be adding a new singleton to mean mostly
the same thing as None, but having the property of comparing
less than all other objects and then recommend its use in the
DB-API for Python 3 applications...

Which I think would be a *really bad* idea, because
there would then no longer be One Obvious Way to
represent and process null values.
E.g. you would no longer be able to write
'value is None' and trust it to work on all kinds
of null value you're likely to get.

Of course you couldn't... value is None would never work full Null
values, only for None, like it should.

[Greg Ewing]

Post by Tim Peters
Guido wanted to drop all the "arbitrary but consistent" mixed-type
comparison crud for Python 3.

Nobody is asking for a return to the arbitrary-but-
[in]consistent mess of Python 2, only to bring
back *one* special case, i.e. None comparing less
than everything else.

I think there is a reasonable argument to be made
in favour of that. Like it or not, None does have
a special place in Python as the one obvious way
to represent a null or missing value, and often
one wants to sort a collection of objects having
keys that can take on null values. Refusing to
make that easy seems like allowing purity to beat
practicality.

[Tim Peters]

[Tim]

Post by Greg Ewing

Post by Tim Peters
Guido wanted to drop all the "arbitrary but consistent" mixed-type
comparison crud for Python 3.

[Greg Ewing]

Post by Greg Ewing
Nobody is asking for a return to the arbitrary-but-
[in]consistent mess of Python 2, only to bring
back *one* special case, i.e. None comparing less
than everything else.

Of course. My point was that dropping None comparisons wasn't
specifically aimed at None comparisons.

Post by Greg Ewing
I think there is a reasonable argument to be made
in favour of that.

There may have been 6 years ago, but Python 3 isn't a thought
experiment anymore. It was first released over 5 years ago, and has
its own compatibility (with Python 3, not Python 2) constraints now.

Post by Greg Ewing
Like it or not, None does have
a special place in Python as the one obvious way
to represent a null or missing value, and often
one wants to sort a collection of objects having
keys that can take on null values.

Perhaps that's often true of your code, but it's never been true of mine.

Post by Greg Ewing
Refusing to make that easy seems like allowing purity to beat
practicality.

Since I've never had a need for it, I can't say for sure whether or
not it's easy - but the .sort() interface is rich enough that I almost
always find even truly complex key transformations "easy enough".

Perhaps a real, concrete use case would help. I have a hard time
imagining why I'd _want_ to sort a list of objects with "null or
missing" keys, instead of filtering such garbage out of the list first
and sorting what remains.

But even with a compelling use case, I'd still think it's years too
late to change this in Python 3.

[Greg Ewing]

Post by Tim Peters
[Greg Ewing]

Post by Greg Ewing
often
one wants to sort a collection of objects having
keys that can take on null values.

Perhaps that's often true of your code, but it's never been true of mine.

It's fairly common in accounting circles. I have a
collection of invoices, each of which can have a due
date specified, but doesn't have to. I want to sort
the invoices by due date. It's not particularly
important whether the missing dates sort first or
last, but it *is* important that the sort *not blow
up*.

Dates seem to be a particularly irksome case. Here's
one suggestion from StackOverflow for dealing with
the problem:

import datetime
mindate = datetime.date(datetime.MINYEAR, 1, 1)

def getaccountingdate(x):
return x['accountingdate'] or mindate

results = sorted(data, key=getaccountingdate)

That's a ridiculous amount of boilerplate for doing
something that ought to be straightforward.

If you don't want to touch comparison in general,
how about an option to sort()?

results = sorted(invoices, key=attrgetter('duedate'), none='first')

Post by Tim Peters
I have a hard time
imagining why I'd _want_ to sort a list of objects with "null or
missing" keys, instead of filtering such garbage out of the list first

A piece of data is *not* garbage just because an optional
part of it is not specified.

[Paul Moore]

Post by Greg Ewing
If you don't want to touch comparison in general,
how about an option to sort()?
results = sorted(invoices, key=attrgetter('duedate'), none='first')

Or alternatively, a "default on None" function - Oracle SQL calls this
nvl, so I will too:

def nvl(x, dflt):
return dflt if x is None else x

results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1))

Admittedly the key function is starting to get complex, but I find
that key functions often do - they are the one big area where Python
uses a lot of functional-style constructs. The existence of
itemgetter/attrgetter are a testament to this fact.

PS isn't this python-ideas territory by now?

Paul

[Serhiy Storchaka]

Post by Paul Moore
Or alternatively, a "default on None" function - Oracle SQL calls this
return dflt if x is None else x
results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1))

Or, as was proposed above:

results = sorted(invoices,
key=lambda x: (x.duedate is not None, x.duedate))

[MRAB]

Post by Greg Ewing

Post by Paul Moore
Or alternatively, a "default on None" function - Oracle SQL calls this
return dflt if x is None else x
results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1))

results = sorted(invoices,
key=lambda x: (x.duedate is not None, x.duedate))

That makes me wonder.

Why is:

None < None

unorderable and not False but:

(None, ) < (None, )

orderable?

[Robert Kern]

Post by MRAB

Post by Greg Ewing

Post by Paul Moore
Or alternatively, a "default on None" function - Oracle SQL calls this
return dflt if x is None else x
results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1))

results = sorted(invoices,
key=lambda x: (x.duedate is not None, x.duedate))

That makes me wonder.
None < None
(None, ) < (None, )
orderable?

tuple's rich comparison uses PyObject_RichCompareBool(x, y, Py_EQ) to find the
first pair of items that is unequal. Then it will test the order of any
remaining elements.

http://hg.python.org/cpython/file/79e5bb0d9b8e/Objects/tupleobject.c#l591

PyObject_RichCompareBool(x, y, Py_EQ) treats identical objects as equal.

http://hg.python.org/cpython/file/79e5bb0d9b8e/Objects/object.c#l716

[Lennart Regebro]

Post by Greg Ewing
If you don't want to touch comparison in general,
how about an option to sort()?
results = sorted(invoices, key=attrgetter('duedate'), none='first')

I think this is a much better option.

//Lennart

[Georg Brandl]

Post by Greg Ewing

Post by Tim Peters
[Greg Ewing]

Post by Greg Ewing
often
one wants to sort a collection of objects having
keys that can take on null values.

Perhaps that's often true of your code, but it's never been true of mine.

It's fairly common in accounting circles. I have a
collection of invoices, each of which can have a due
date specified, but doesn't have to. I want to sort
the invoices by due date. It's not particularly
important whether the missing dates sort first or
last, but it *is* important that the sort *not blow
up*.
Dates seem to be a particularly irksome case. Here's
one suggestion from StackOverflow for dealing with
import datetime
mindate = datetime.date(datetime.MINYEAR, 1, 1)
return x['accountingdate'] or mindate
results = sorted(data, key=getaccountingdate)
That's a ridiculous amount of boilerplate for doing
something that ought to be straightforward.

Seeing how you need a key function in any case for this sort to work,
it's only the "or mindate" added, which I can't recognize as "ridiculous
amount of boilerplate". Much more so since you can put the key function
in a shared module and import it from there anywhere you need it.

Georg

[Greg Ewing]

Post by Georg Brandl
Seeing how you need a key function in any case for this sort to work,
it's only the "or mindate" added, which I can't recognize as "ridiculous
amount of boilerplate".

Well, I don't much like having to construct a key
function in the first place for something as common
as sorting on an attribute.

Also, in the particular case of dates, there's the
annoying fact that the datetime module doesn't
provide any kind of null object that can be compared
with datetimes, so you have to make a fake one
yourself.

All of these small annoyances add up to what is, for
me, a fairly big annoyance.

What I'd *really* like to be able to write is:

sort(invoices, keyattr = 'duedate', none = 'first')

[Paul Moore]

Post by Greg Ewing
sort(invoices, keyattr = 'duedate', none = 'first')

Which is of course a pretty trivial utility function to write. But I
understand your point - these "trivial helpers" add up over time into
a bit of a burden. But fixing that (if anyone feels it's worth doing
so) can be handled many ways, and changing the semantics of None is
far from the simplest or most obvious.

Pul

[Nick Coghlan]

Post by Paul Moore

Post by Greg Ewing
sort(invoices, keyattr = 'duedate', none = 'first')

Which is of course a pretty trivial utility function to write. But I
understand your point - these "trivial helpers" add up over time into
a bit of a burden. But fixing that (if anyone feels it's worth doing
so) can be handled many ways, and changing the semantics of None is
far from the simplest or most obvious.

So perhaps the real answer is for someone to write a sorting helper module
and put it on PyPI. Things like sort_byattr (implicitly uses attrgetter),
sort_byitem (implicitly uses itemgetter), a flag argument defaulting to
"none_first=True", SortsLow and SortsHigh singletons, etc.

Those don't need to be builtins, but collating them into a clean helper API
would not only allow that module to be used directly, but also act as a
reference for anyone wanting to implement the behaviour themselves.

(I know, I know, we're way into python-ideas territory - it's just that so
much if the thread *was* on topic for python-dev, it seems relatively
pointless to split this iteration of the discussion. The *next* thread
about it should definitely be on python-ideas, though)

Cheers,
Nick.

Post by Paul Moore
Pul
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

[Terry Reedy]

Post by Greg Ewing
results = sorted(invoices, key=attrgetter('duedate'), none='first')

I think this is the best idea on the thread. As a pure enhancement, it
could be added in 3.5. The only tricky part of the implementation is
maintaining stability of the sort. The obvious implementation of
swapping Nones with objects at one end would break that. Instead, a scan
listing the positions of Nones should be followed by a series of block
moves of objects (pointers) between the Nones. It would still be O(n).

[Oscar Benjamin]

Post by Greg Ewing
results = sorted(invoices, key=attrgetter('duedate'), none='first')

I think this is the best idea on the thread. As a pure enhancement, it could
be added in 3.5. The only tricky part of the implementation is maintaining
stability of the sort. The obvious implementation of swapping Nones with
objects at one end would break that. Instead, a scan listing the positions
of Nones should be followed by a series of block moves of objects (pointers)
between the Nones. It would still be O(n).

This only works if the list entry is a simple None. If the None is

Post by Greg Ewing

(1, 2, 3) < (1, 2, None)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()


Oscar

[Terry Reedy]

Post by Greg Ewing
Nobody is asking for a return to the arbitrary-but-
[in]consistent mess of Python 2, only to bring
back *one* special case, i.e. None comparing less
than everything else.

For a < None, that is only the fallback rule if a does not handle the
comparison. The result is a mess, including a possible inconsistency
between direct comparison and cmp. See my previous posts.

'Bringing back' what was or an improved version would be a semantic
change that could break code and would require a two-version deprecation
period.

[Mark Lawrence]

Post by Terry Reedy

Post by Greg Ewing
Nobody is asking for a return to the arbitrary-but-
[in]consistent mess of Python 2, only to bring
back *one* special case, i.e. None comparing less
than everything else.

For a < None, that is only the fallback rule if a does not handle the
comparison. The result is a mess, including a possible inconsistency
between direct comparison and cmp. See my previous posts.
'Bringing back' what was or an improved version would be a semantic
change that could break code and would require a two-version deprecation
period.

Sorry if this has already been suggested, but why not introduce a new
singleton to make the database people happier if not happy? To avoid
confusion call it dbnull? A reasonable compromise or complete cobblers? :)

[Chris Angelico]

Post by Mark Lawrence
Sorry if this has already been suggested, but why not introduce a new
singleton to make the database people happier if not happy? To avoid
confusion call it dbnull? A reasonable compromise or complete cobblers? :)

That would be a major change to the DB API. Possibly the best
solution, though. Start off by having the default be to return None
(as now) and have a flag that can be set "please return sys.dbnull
instead" (does dbnull belong in sys?), and let that settle for a few
years. Recommend that all applications explicitly set the flag, either
to True or to False. Then eventually, with the full deprecation
warnings, change the default to True. (Or maybe make it an error to
not set it.) Then, after another long round of deprecations, drop the
None behaviour from the spec altogether.

But even in its early steps, that could solve the problem. Anyone who
wants to sort a list of tuples that came from the database can simply
set the flag (assuming their back-end supports that flag) and happily
use dbnull.

ChrisA

[Lennart Regebro]

Post by Mark Lawrence
Sorry if this has already been suggested, but why not introduce a new
singleton to make the database people happier if not happy? To avoid
confusion call it dbnull? A reasonable compromise or complete cobblers? :)

I think this is possible already, for the database people. The problem
is that it will not pass the is None test, which at the very least is
not backwards compatible with how they have used it before.

[Glenn Linderman]

Post by Lennart Regebro

Post by Mark Lawrence
Sorry if this has already been suggested, but why not introduce a new
singleton to make the database people happier if not happy? To avoid
confusion call it dbnull? A reasonable compromise or complete cobblers? :)

I think this is possible already, for the database people. The problem
is that it will not pass the is None test, which at the very least is
not backwards compatible with how they have used it before.

The new singleton will be called something else, likely with Null in the
name, so that's what I'll call it here... Null.

So when switching from None to Null, you must also switch from "is None"
to "is Null".

Of course it is not backwards compatible... but once all the "database
related None usage" is switched to "Null usage" it should work the same
as before, but with proper (for some database's definition of proper)
semantics.

[Greg Ewing]

Of course it is not backwards compatible... but once all the "database related
None usage" is switched to "Null usage" it should work the same as before,

My problem with this is that there is no clear distinction
between "database-related None usage" and None usage in general.

Data retrieved from a database is often passed to other code
that has nothing to do with databases, and data received from
other code is inserted into databases. Somewhere in between
someone is going to have to be careful to translate back and
forth between Nones and Nulls. This is a recipe for chaos.

[Stephen J. Turnbull]

Post by Greg Ewing
Data retrieved from a database is often passed to other code
that has nothing to do with databases, and data received from
other code is inserted into databases. Somewhere in between
someone is going to have to be careful to translate back and
forth between Nones and Nulls.

Sure. The suggestion is to assign responsibility for such careful
translation to implementers of the DB API.

Post by Greg Ewing
This is a recipe for chaos.

If it is, then chaos is already upon us because you can't be careful
even if you want to.

I don't know if fixing it is worth the work and confusion involved in
the fixing process, but fixing it will conserve (and maybe reduce)
chaos, not create it.

[Jon Ribbens]

Post by M.-A. Lemburg
This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,

No you can't. See http://bugs.python.org/issue1673405 .

According to Tim Peters, the "None is less than everything" rule
never existed.

[M.-A. Lemburg]

Post by Jon Ribbens

Post by M.-A. Lemburg
This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,

No you can't. See http://bugs.python.org/issue1673405 .
According to Tim Peters, the "None is less than everything" rule
never existed.

Well, then Tim probably didn't read the code in object.c :-)

Seriously, the datetime module types were the first types to experiment
with the new mixed type operation mechanism added at the time:

http://www.python.org/dev/peps/pep-0207/
http://www.python.org/dev/peps/pep-0208/

Objects implementing the rich comparison slot can indeed override
these defaults and also raise exceptions in comparison operations,
so you're right: None comparisons can still be made to raise
exceptions, even in Python 2.7.

Still, None comparisons work just fine for most types in Python 2.x
and people have written code assuming that it works for many years.
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Jon Ribbens

Post by M.-A. Lemburg

Post by Antoine Pitrou
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Terry Reedy]

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,

No you can't. See http://bugs.python.org/issue1673405 .
According to Tim Peters, the "None is less than everything" rule
never existed.

Tim is correct. Copying from my other response (posted after you wrote this)

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
class Bottom(object): # get same results below without 'object'

def __lt__(self, other):
return True

# the following two results are consistent and
# contradict the claim that 'None is smaller than anything'

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
Bottom() < None

True

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
cmp(Bottom(), None)

-1

# the following two results are not consistent with the
# definition of cmp, so 1 of the 2 is buggy

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
None < Bottom()

True

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
cmp(None, Bottom())

1

Post by M.-A. Lemburg
Well, then Tim probably didn't read the code in object.c :-)

I did, as I suspect Time has also. Function default_3way_compare is a
'final fallback'. The comment within, besides being a code comment and
not the doc, is wrong unless 'anything' is qualified.

[Terry Reedy]

Post by Terry Reedy

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,

No you can't. See http://bugs.python.org/issue1673405 .

This issue was about the fact that datetimes do not compare with None,
which means that even in 2.x, putting None in a list of datetimes (to
mean no datetime) means that the list cannot be sorted.

Post by Terry Reedy

Post by M.-A. Lemburg

Post by Jon Ribbens
According to Tim Peters, the "None is less than everything" rule
never existed.

Tim is correct. Copying from my other response (posted after you wrote this)

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
class Bottom(object): # get same results below without 'object'

return True
# the following two results are consistent and
# contradict the claim that 'None is smaller than anything'

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
Bottom() < None

True

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
cmp(Bottom(), None)

-1
# the following two results are not consistent with the
# definition of cmp, so 1 of the 2 is buggy

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
None < Bottom()

True

Post by M.-A. Lemburg

Post by Jon Ribbens

Post by M.-A. Lemburg
cmp(None, Bottom())

1

Post by M.-A. Lemburg
Well, then Tim probably didn't read the code in object.c :-)

I did, as I suspect Time has also. Function default_3way_compare is a
'final fallback'. The comment within, besides being a code comment and
not the doc, is wrong unless 'anything' is qualified.

FWIW: do_cmp first calls try_rich_to_3way_compare and if that fails it
calls try_3way_compare and if that fails it calls default_3way_compare.
Somewhat confusingly, try_rich_to_3way_compare first calls
try_3way_compare and if that fails, it calls default_3way_compare. So
the backup calls in do_cmp are redundant, as they will have already
failed in try_rich_to_3way_compare.

The special casing of None in default_3way_compare was added by Guido in
rev. 16123 on 2001 Jan 22. Before that, None was compared by the
typename, as still specified in the docs.

Tim was correct when he wrote "For older types, the result of inequality
comparison with None isn't defined by the language, and the outcome does
vary across CPython releases."

[Serhiy Storchaka]

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()

If you replace None to another value that cannot be compared with int
(e.g. string), you will got the same nasty case.

[M.-A. Lemburg]

l = [(1, None), (2, None)]
l.sort()
l

[(1, None), (2, None)]

l = [(1, None), (2, None), (3, 4)]
l.sort()
l

[(1, None), (2, None), (3, 4)]

l = [(1, None), (2, None), (3, 4), (2, 3)]
l.sort()

File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()

If you replace None to another value that cannot be compared with int (e.g. string), you will got
the same nasty case.

Yes, but that's not the point. Unlike strings or other mixed types that
you cannot compare, None is used as placeholder in data processing as
special value to mean "no value available".

You intentionally use such values in programming. It's not a bug to
have None in a data list or as value of a variable.

[Serhiy Storchaka]

Post by M.-A. Lemburg
Yes, but that's not the point. Unlike strings or other mixed types that
you cannot compare, None is used as placeholder in data processing as
special value to mean "no value available".

Isn't float('nan') such placeholder?

Post by M.-A. Lemburg
You intentionally use such values in programming. It's not a bug to
have None in a data list or as value of a variable.

You can't have None in array('f'), you can't add or multiply by None.
Relation operators don't looks an exception here. Applying sorted() to a
list which contains numbers and Nones makes as much sense as applying
sum() to it.

[M.-A. Lemburg]

Post by Serhiy Storchaka

Post by M.-A. Lemburg
Yes, but that's not the point. Unlike strings or other mixed types that
you cannot compare, None is used as placeholder in data processing as
special value to mean "no value available".

Isn't float('nan') such placeholder?

You can easily construct other such placeholders, but None was intended
for this purpose:

http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None

"""
The Python None object, denoting lack of value. ...
"""

Post by Serhiy Storchaka

Post by M.-A. Lemburg
You intentionally use such values in programming. It's not a bug to
have None in a data list or as value of a variable.

You can't have None in array('f'), you can't add or multiply by None. Relation operators don't looks
an exception here. Applying sorted() to a list which contains numbers and Nones makes as much sense
as applying sum() to it.

Of course, you cannot apply any operations with None - it doesn't
have a value -, but you can compare it to other objects and it provides
a consistent behavior in Python 2. Python 3 is missing such an object.
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Serhiy Storchaka

Post by M.-A. Lemburg

Post by Antoine Pitrou
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Gustavo Carneiro]

Post by M.-A. Lemburg

Post by Serhiy Storchaka

Post by M.-A. Lemburg
Yes, but that's not the point. Unlike strings or other mixed types that
you cannot compare, None is used as placeholder in data processing as
special value to mean "no value available".

Isn't float('nan') such placeholder?

You can easily construct other such placeholders, but None was intended
http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None
"""
The Python None object, denoting lack of value. ...
"""

Post by Serhiy Storchaka

Post by M.-A. Lemburg
You intentionally use such values in programming. It's not a bug to
have None in a data list or as value of a variable.

You can't have None in array('f'), you can't add or multiply by None.

Relation operators don't looks

Post by Serhiy Storchaka
an exception here. Applying sorted() to a list which contains numbers

and Nones makes as much sense

Post by Serhiy Storchaka
as applying sum() to it.

Of course, you cannot apply any operations with None - it doesn't
have a value -, but you can compare it to other objects and it provides
a consistent behavior in Python 2. Python 3 is missing such an object.

I agree with you that Python 3 could use such an object. Just don't make
it the default. Leave None as it is.

Also I agree that my previous naming suggestions are bad. What we need is
only one (or two) additional object whose main semantic meaning is still
"no value", but which also adds a meaning of "comparable". Then it's a
matter of choosing a good name for it, with lots of bikeshedding involved.
Just lets not change None only because we're too lazy to discuss a proper
alternative name. Also this use case is not _that_ common, so it's ok if
it has a slightly longer name than None.

Also think of the implications of changing None at this point. It would
allow us to write programs that work Python >= 3.5 and Python <= 2.7, but
fail mysteriously in all other versions in between. What a mess that would
be...

[M.-A. Lemburg]

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Serhiy Storchaka

Post by M.-A. Lemburg
Yes, but that's not the point. Unlike strings or other mixed types that
you cannot compare, None is used as placeholder in data processing as
special value to mean "no value available".

Isn't float('nan') such placeholder?

You can easily construct other such placeholders, but None was intended
http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None
"""
The Python None object, denoting lack of value. ...
"""

Post by Serhiy Storchaka

Post by M.-A. Lemburg
You intentionally use such values in programming. It's not a bug to
have None in a data list or as value of a variable.

You can't have None in array('f'), you can't add or multiply by None.

Relation operators don't looks

Post by Serhiy Storchaka
an exception here. Applying sorted() to a list which contains numbers

and Nones makes as much sense

Post by Serhiy Storchaka
as applying sum() to it.

Of course, you cannot apply any operations with None - it doesn't
have a value -, but you can compare it to other objects and it provides
a consistent behavior in Python 2. Python 3 is missing such an object.

I agree with you that Python 3 could use such an object. Just don't make
it the default. Leave None as it is.
Also I agree that my previous naming suggestions are bad. What we need is
only one (or two) additional object whose main semantic meaning is still
"no value", but which also adds a meaning of "comparable". Then it's a
matter of choosing a good name for it, with lots of bikeshedding involved.
Just lets not change None only because we're too lazy to discuss a proper
alternative name. Also this use case is not _that_ common, so it's ok if
it has a slightly longer name than None.
Also think of the implications of changing None at this point. It would
allow us to write programs that work Python >= 3.5 and Python <= 2.7, but
fail mysteriously in all other versions in between. What a mess that would
be...

Yes, that's unfortunately true.

If you regard this as bug in the Python 3 series, we could still fix
it for 3.3 and 3.4, though. After all, we'd just be removing an unwanted
exception and not make already working Python 3 applications fail.
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Gustavo Carneiro

Post by M.-A. Lemburg

Post by Serhiy Storchaka
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Chris Angelico]

Post by M.-A. Lemburg

Post by Gustavo Carneiro
Also think of the implications of changing None at this point. It would
allow us to write programs that work Python >= 3.5 and Python <= 2.7, but
fail mysteriously in all other versions in between. What a mess that would
be...

Yes, that's unfortunately true.

I don't know that that's in itself that much of a problem. (BTW, I
wouldn't call it "<= 2.7"; it'd be all 2.x, it's not like some of the
2.7.y versions aren't included.) There are already barriers to
supporting 2.7 and 3.1/3.2, like the u"asdf" notation for Unicode
literals. Making it easier to support 2.x and 3.x from the same
codebase is an improvement that has been done and will be done more.
It's not such a mysterious failure; it's just that you support Python
2.5+ and 3.4+ (or whatever the specific versions are).

ChrisA

[Nick Coghlan]

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

This is the first I've ever heard of that sorting behaviour being an
intentional feature, rather than just an artefact of Python 2 allowing
arbitrarily ordered comparisons between different types. Can you point me
to the relevant entry in the Python 2 language reference?

Cheers,
Nick.

[M.-A. Lemburg]

Post by Nick Coghlan

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

This is the first I've ever heard of that sorting behaviour being an
intentional feature, rather than just an artefact of Python 2 allowing
arbitrarily ordered comparisons between different types. Can you point me
to the relevant entry in the Python 2 language reference?

This is not documented anywhere in the language spec, AFAIK. It
is documented in the code (Python 2.7; Object/object.c):

default_3way_compare(PyObject *v, PyObject *w)
...
/* None is smaller than anything */
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;

Note that it's not important whether None is smaller or greater
than any other object. The important aspect is that it's sorting
order is consistent and doesn't raise a TypeError when doing an
ordered comparison with other objects.
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Nick Coghlan

Post by M.-A. Lemburg

Post by Stephen J. Turnbull
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Terry Reedy]

Received: from localhost (HELO mail.python.org) (127.0.0.1)
by albatross.python.org with SMTP; 17 Feb 2014 16:30:12 +0100
Received: from plane.gmane.org (unknown [80.91.229.3])
(using TLSv1 with cipher AES256-SHA (256/256 bits))
(No client certificate requested)
by mail.python.org (Postfix) with ESMTPS
for <python-***@python.org>; Mon, 17 Feb 2014 16:30:12 +0100 (CET)
Received: from list by plane.gmane.org with local (Exim 4.69)
(envelope-from <python-python-***@m.gmane.org>) id 1WFQ8w-0004wJ-OB
for python-***@python.org; Mon, 17 Feb 2014 16:30:02 +0100
Received: from pool-173-75-254-207.phlapa.fios.verizon.net ([173.75.254.207])
by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
id 1AlnuQ-0007hv-00
for <python-***@python.org>; Mon, 17 Feb 2014 16:30:02 +0100
Received: from tjreedy by pool-173-75-254-207.phlapa.fios.verizon.net with
local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00
for <python-***@python.org>; Mon, 17 Feb 2014 16:30:02 +0100
X-Injected-Via-Gmane: http://gmane.org/
Lines: 116
X-Complaints-To: ***@ger.gmane.org
X-Gmane-NNTP-Posting-Host: pool-173-75-254-207.phlapa.fios.verizon.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
rv:24.0) Gecko/20100101 Thunderbird/24.3.0
In-Reply-To: <***@egenix.com>
X-BeenThere: python-***@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Python core developers <python-dev.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-dev>,
<mailto:python-dev-***@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-dev/>
List-Post: <mailto:python-***@python.org>
List-Help: <mailto:python-dev-***@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-dev>,
<mailto:python-dev-***@python.org?subject=subscribe>
Errors-To: python-dev-bounces+python-python-dev=***@python.org
Sender: "Python-Dev"
<python-dev-bounces+python-python-dev=***@python.org>
Archived-At: <http://permalink.gmane.org/gmane.comp.python.devel/145596>

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

This is the first I've ever heard of that sorting behaviour being an
intentional feature, rather than just an artefact of Python 2 allowing
arbitrarily ordered comparisons between different types. Can you point me
to the relevant entry in the Python 2 language reference?

This is not documented anywhere in the language spec, AFAIK.

http://docs.python.org/2/reference/expressions.html#not-in

"The operators <, >, ==, >=, <=, and != compare the values of two
objects. The objects need not have the same type. If both are numbers,
they are converted to a common type. Otherwise, objects of different
types always compare unequal, and are ordered consistently but arbitrarily."

http://docs.python.org/2/library/stdtypes.html#comparisons

"Objects of different types, except different numeric types and
different string types, never compare equal; such objects are ordered
consistently but arbitrarily"

It goes on to note the exception for complex numbers, but none for None.
It continues

"CPython implementation detail: Objects of different types except
numbers are ordered by their type names;"

Again, there is no exception noted for None, although, since the type
name of None is 'NoneType', its behavior does not match that doc note.

I believe that CPython implementation detail was some of the arbitrary
orderings changed in CPython between 1.3 and 2.7. I do not know whether
other implementations all mimicked CPython or not, but the reference
manual does not require that.

Post by M.-A. Lemburg
default_3way_compare(PyObject *v, PyObject *w)

This is the 'final fallback' for comparisons, not the first thing tried.

Post by M.-A. Lemburg
...
/* None is smaller than anything */

Reading CPython C code is not supposed to be a requirement for
programming in Python. If that comment were true, I would regard it as
only documenting a version of CPython, not the language standard. But it
is not even true in 2.7.6. But it is not even true.

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg
class Bottom(object): # get same results below without 'object'

def __lt__(self, other):
return True

# the following two results are consistent and
# contradict the claim that 'None is smaller than anything'

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg
Bottom() < None

True

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg
cmp(Bottom(), None)

-1

# the following two results are not consistent with the
# definition of cmp, so 1 of the 2 is buggy

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg
None < Bottom()

True

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg
cmp(None, Bottom())

1

It appears that 'anything' needs to be qualified as something like
'anything that does not itself handle comparison with None or is not
given a chance to do so in a particular comparison expression'.

Post by M.-A. Lemburg
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;
Note that it's not important whether None is smaller or greater
than any other object. The important aspect is that it's sorting
order is consistent and doesn't raise a TypeError when doing an
ordered comparison with other objects.

I can agree that it might have been nice if None had been defined as a
universal bottom object, but it has not been so defined and it did not
act as such. To make it (or anything else) a true bottom object would
require a change in the way comparisons are evaluated. Comparisons with
'bottom' would have to be detected and special-cased at the beginning of
the code, not at the end as a fallback.

[Nick Coghlan]

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg

Post by Stephen J. Turnbull

Post by M.-A. Lemburg
IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

I don't understand this statement. The theory is that they *should*
fail.
The example of sort is a good one. Sometimes you want missing values
to be collected at the beginning of a list, sometimes at the end.
Sometimes you want them treated as top elements, sometimes as bottom.
And sometimes it is a real error for missing values to be present.
Not to mention that sometimes the programmer simply hasn't thought
about the appropriate policy. I don't think Python should silently
impose a policy in that case, especially given that the programmer may
have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

This is the first I've ever heard of that sorting behaviour being an
intentional feature, rather than just an artefact of Python 2 allowing
arbitrarily ordered comparisons between different types. Can you point me
to the relevant entry in the Python 2 language reference?

This is not documented anywhere in the language spec, AFAIK. It
default_3way_compare(PyObject *v, PyObject *w)
...
/* None is smaller than anything */
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;
Note that it's not important whether None is smaller or greater
than any other object. The important aspect is that it's sorting
order is consistent and doesn't raise a TypeError when doing an
ordered comparison with other objects.

Thanks, that's enough to persuade me that it is a good idea to restore that
behaviour (and make it an official part of the language spec) as part of
the "eliminate remaining barriers to migration from Python 2" effort in 3.5.

It will mean that SQL sorting involving NULL values maps cleanly again, and
it's easy enough to use any() to check that container doesn't contain None.

Cheers,
Nick.

Post by M.-A. Lemburg
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 17 2014)

Post by Nick Coghlan

Post by M.-A. Lemburg

Post by Stephen J. Turnbull
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________
2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Nick Coghlan]

Post by Nick Coghlan

Post by M.-A. Lemburg

Post by Nick Coghlan

Post by M.-A. Lemburg
None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

This is the first I've ever heard of that sorting behaviour being an
intentional feature, rather than just an artefact of Python 2 allowing
arbitrarily ordered comparisons between different types. Can you point me
to the relevant entry in the Python 2 language reference?

This is not documented anywhere in the language spec, AFAIK. It
default_3way_compare(PyObject *v, PyObject *w)
...
/* None is smaller than anything */
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;
Note that it's not important whether None is smaller or greater
than any other object. The important aspect is that it's sorting
order is consistent and doesn't raise a TypeError when doing an
ordered comparison with other objects.

Thanks, that's enough to persuade me that it is a good idea to restore

that behaviour (and make it an official part of the language spec) as part
of the "eliminate remaining barriers to migration from Python 2" effort in
3.5.

Post by Nick Coghlan
It will mean that SQL sorting involving NULL values maps cleanly again,

and it's easy enough to use any() to check that container doesn't contain
None.

Note that I sent this before reading Tim's reply regarding the origin of
the Python 2 sorting behaviour, and Terry's point that the relevant Python
2 code lived in the legacy 3 way comparison fallback that was deliberately
removed from 3.0.

MAL's perspective does still at least mean I am only -0 rather than -1 on
making None always sorts first rather than throwing an exception, but my
preferred solution remains to provide backporting-friendly support for
handling of None values in comparisons (and particularly sorting
operations): http://bugs.python.org/issue20630

Cheers,
Nick.

[Barry Warsaw]

Post by Nick Coghlan
Thanks, that's enough to persuade me that it is a good idea to restore that
behaviour (and make it an official part of the language spec) as part of
the "eliminate remaining barriers to migration from Python 2" effort in 3.5.

At the very least, it's an idea worth PEP-ifying.

-Barry

[Terry Reedy]

Post by Nick Coghlan

Post by M.-A. Lemburg
default_3way_compare(PyObject *v, PyObject *w)
...
/* None is smaller than anything */

Unless it is not, as with datetimes, perhaps other classes written
similarly, and some user class instances.

Post by Nick Coghlan

Post by M.-A. Lemburg
Note that it's not important whether None is smaller or greater
than any other object. The important aspect is that it's sorting
order is consistent and doesn't raise a TypeError when doing an
ordered comparison with other objects.

Thanks, that's enough to persuade me that it is a good idea to restore
that behaviour

Would you restore the actual sometimes inconsistent 2.x behavior or
implement something new -- what M-A claims but never was? I doubt the
former would be trivial since it was part of the now deleted cmp and
3way machinery. To make None a true bottom object, the rich comparison
methods would have to special-case None as either argument before
looking at the __rc__ special methods of either.

Regardless of how implemented, such a change would break user code that
defines a Bottom class and depends on Bottom() < None == True. Given the
python-ideas discussions about adding a top or bottom class, I expect
that such classes are in use. So a deprecation period would be needed,
pushing the change off to 3.7. It is also possible that someone took the
ending of cross-type comparisons seriously and is depending on the
TypeError.

while True:
x = f(args)
try:
if x > 10: a(x)
else: b(x)
except TypeError:
# f did not return a number
break

[Greg Ewing]

Post by Terry Reedy
To make None a true bottom object, the rich comparison
methods would have to special-case None as either argument before
looking at the __rc__ special methods of either.

I don't think it would be necessary to go that far.
It would be sufficient to put the special case *after*
giving both operations a chance to handle the operation
via the type slots.

Well-behaved objects generally wouldn't go out of their
way to defeat that, but they could do so if necessary,
e.g. to create a Bottom() that compares less than None.
Although once None becomes usable as a bottom in most
cases, there shouldn't be much need for people to
introduce their own bottoms.

[Terry Reedy]

Post by Greg Ewing

To make None a true bottom object, the rich comparison methods would
have to special-case None as either argument before looking at the
__rc__ special methods of either.

I don't think it would be necessary to go that far.

It is if you want None to be 'a true bottom object'.
If you think something less is sufficient, or even desirable, then yes, ...

Post by Greg Ewing
It would be sufficient to put the special case *after*
giving both operations a chance to handle the operation
via the type slots.

That would more or less reproduce the 2.x situation, except that a
conflict between < and cmp would not be possible.

Post by Greg Ewing
Well-behaved objects generally wouldn't go out of their
way to defeat that, but they could do so if necessary,
e.g. to create a Bottom() that compares less than None.
Although once None becomes usable as a bottom in most
cases, there shouldn't be much need for people to
introduce their own bottoms.

[MRAB]

Post by Chris Withers
Hi All,
Sending this to python-dev as I'm wondering if this was considered when
the choice to have objects of different types raise a TypeError when
ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that
can either be None or, for sake of argument, a numeric value.
return True
return ((self._lower, self._upper, self._bounds) <
(other._lower, other._upper, other._bounds))
return NotImplemented
self_value = getattr(self, attr)
other_value = getattr(other, attr)
pass
return True
return False
return self_value < other_value
return False
Am I missing something? How can I get this method down to a sane size?

How about this:

def make_key(item):
return [(x is not None, x or 0) for i in [item._lower,
item._upper, item._bounds]]

def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented

return make_key(self) < make_key(other)

It'll make None less than any number.