UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position

[Νίκος]

I just started to have this error without changing nothing

in my index.html(template) and metrites.py(which ipen the template)

[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original
exception was:
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Traceback
(most recent call last):
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] File
"/home/nikos/public_html/cgi-bin/metrites.py", line 19, in <module>
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host =
socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0:
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py


Why cant it decode the starting byte? what starting byte is that?

[Dave Angel]

Post by Νίκος
I just started to have this error without changing nothing
in my index.html(template) and metrites.py(which ipen the template)
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original
exception was: [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] Traceback (most recent call last): [Thu Jul 04
11:35:14 2013] [error] [client 108.162.229.97] File
"/home/nikos/public_html/cgi-bin/metrites.py", line 19, in <module>
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host =
socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte
0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013]
metrites.py
Why cant it decode the starting byte? what starting byte is that?

The error message means that somebody is trying to decode a byte string
into Unicode, and using the utf-8 codec for it. Only certain sequences
are legal in utf-8, and the first byte of a character may not be 0xb6.
So it gives an error. The question is where does this string come from.


Well, the message shows the source line from metrites.py:

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'

So the most likely candidate is the string in the environment named
"REMOTE_ADDR" Can you display that string? it should look like

"11.24.32.4"

or some other valid IP address.


I'm assuming Python 2.7. You should specify the python version when
starting a new thread, as we (or at least I) cannot keep track of what
version everyone's running.

[Νίκος]

Post by Dave Angel

Post by Νίκος
I just started to have this error without changing nothing
in my index.html(template) and metrites.py(which ipen the template)
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original
exception was: [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] Traceback (most recent call last): [Thu Jul 04
11:35:14 2013] [error] [client 108.162.229.97] File
"/home/nikos/public_html/cgi-bin/metrites.py", line 19, in <module>
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host =
socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte
0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013]
metrites.py
Why cant it decode the starting byte? what starting byte is that?

The error message means that somebody is trying to decode a byte string
into Unicode, and using the utf-8 codec for it. Only certain sequences
are legal in utf-8, and the first byte of a character may not be 0xb6.
So it gives an error. The question is where does this string come from.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
So the most likely candidate is the string in the environment named
"REMOTE_ADDR" Can you display that string? it should look like
"11.24.32.4"
or some other valid IP address.
I'm assuming Python 2.7. You should specify the python version when
starting a new thread, as we (or at least I) cannot keep track of what
version everyone's running.

Ima using Python v3.3.2 Dave

The host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' should not have give an error since i explicityl tell it
that if it cannot resolve dns the ip to hostname to set it as "unresolved"

The error appear ONLY when i CloudFlare superhost.gr

If i pause tthe domain form CloudFlare then my website loads properly.

[Dave Angel]

Post by Νίκος

Post by Dave Angel

Post by Νίκος
I just started to have this error without changing nothing
in my index.html(template) and metrites.py(which ipen the template)
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original
exception was: [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] Traceback (most recent call last): [Thu Jul 04
11:35:14 2013] [error] [client 108.162.229.97] File
"/home/nikos/public_html/cgi-bin/metrites.py", line 19, in <module>
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host =
socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte
0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013]
metrites.py
Why cant it decode the starting byte? what starting byte is that?

The error message means that somebody is trying to decode a byte string
into Unicode, and using the utf-8 codec for it. Only certain sequences
are legal in utf-8, and the first byte of a character may not be 0xb6.
So it gives an error. The question is where does this string come from.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
So the most likely candidate is the string in the environment named
"REMOTE_ADDR" Can you display that string? it should look like
"11.24.32.4"
or some other valid IP address.
I'm assuming Python 2.7. You should specify the python version when
starting a new thread, as we (or at least I) cannot keep track of what
version everyone's running.

Ima using Python v3.3.2 Dave
The host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' should not have give an error since i explicityl tell it
that if it cannot resolve dns the ip to hostname to set it as "unresolved"

That's not true. The 'or' doesn't get executed until after the
gethostbyaddr() call has returned. So if something's wrong with that
call, and it throws an exception, the 'or "unresolved"' won't help.

I don't know that this is the problem, and I'm not all familiar with
these api's. But I cannot see anything else that could go wrong there
to give that particular exception. Unless the hostname it's going to
return is a byte string.

Post by Νίκος
The error appear ONLY when i CloudFlare superhost.gr
If i pause tthe domain form CloudFlare then my website loads properly.

I don't really know what CloudFlare is, and have no idea what 'pausing
the form' will do. But since it has something to do with dns, perhaps
it's returning an invalid host name, one that isn't encoded in utf-8.

[Νίκος]

-------- Αρχικό μήνυμα --------
Θέμα: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in
position 0: invalid start byte
Ημερομηνία: Thu, 04 Jul 2013 06:29:25 -0400
Από: Dave Angel <***@davea.name>
Προς: python-***@python.org
Ομάδες συζήτησης: comp.lang.python

Post by Νίκος

Post by Dave Angel

Post by Νίκος
I just started to have this error without changing nothing
in my index.html(template) and metrites.py(which ipen the template)
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original
exception was: [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] Traceback (most recent call last): [Thu Jul 04
11:35:14 2013] [error] [client 108.162.229.97] File
"/home/nikos/public_html/cgi-bin/metrites.py", line 19, in <module>
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host =
socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client
108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte
0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013]
metrites.py
Why cant it decode the starting byte? what starting byte is that?

The error message means that somebody is trying to decode a byte string
into Unicode, and using the utf-8 codec for it. Only certain sequences
are legal in utf-8, and the first byte of a character may not be 0xb6.
So it gives an error. The question is where does this string come from.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
So the most likely candidate is the string in the environment named
"REMOTE_ADDR" Can you display that string? it should look like
"11.24.32.4"
or some other valid IP address.
I'm assuming Python 2.7. You should specify the python version when
starting a new thread, as we (or at least I) cannot keep track of what
version everyone's running.

Ima using Python v3.3.2 Dave
The host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnResolved' should not have give an error since i explicityl tell it
that if it cannot resolve dns the ip to hostname to set it as "unresolved"

That's not true. The 'or' doesn't get executed until after the
gethostbyaddr() call has returned. So if something's wrong with that
call, and it throws an exception, the 'or "unresolved"' won't help.

So how do you propose to write the above statement?
I was under the i impression then if for any reason the gethostbyaddr
failed then or's argument will be returned instead.

Post by Νίκος
The error appear ONLY when i CloudFlare superhost.gr
If i pause tthe domain form CloudFlare then my website loads properly.

I don't really know what CloudFlare is, and have no idea what 'pausing
the form' will do. But since it has something to do with dns, perhaps
it's returning an invalid host name, one that isn't encoded in utf-8.

I think so too.

[Ulrich Eckhardt]

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

Uli

[Νίκος]

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?

What character is 0xb6 anyways?

[Chris Angelico]

So you are also suggesting that what gesthostbyaddr() returns is not utf-8
encoded too?
What character is 0xb6 anyways?

It isn't. It's a byte. Bytes are not characters.

http://www.joelonsoftware.com/articles/Unicode.html

ChrisA

[Νίκος]

Post by Chris Angelico

So you are also suggesting that what gesthostbyaddr() returns is not utf-8
encoded too?
What character is 0xb6 anyways?

It isn't. It's a byte. Bytes are not characters.
http://www.joelonsoftware.com/articles/Unicode.html

Well in case of utf-8 encoding for the first 127 codepoing we can safely
say that a character equals a byte :)

[MRAB]

Post by Νίκος

Post by Chris Angelico

So you are also suggesting that what gesthostbyaddr() returns is not utf-8
encoded too?
What character is 0xb6 anyways?

It isn't. It's a byte. Bytes are not characters.
http://www.joelonsoftware.com/articles/Unicode.html

Well in case of utf-8 encoding for the first 127 codepoing we can safely
say that a character equals a byte :)

Equals? No. Bytes are not characters. (Strictly speaking, they're
codepoints, not characters.)

And anyway, it's the first _128_ codepoints.

[Νίκος]

Post by MRAB

Post by Νίκος

Post by Chris Angelico

So you are also suggesting that what gesthostbyaddr() returns is not utf-8
encoded too?
What character is 0xb6 anyways?

It isn't. It's a byte. Bytes are not characters.
http://www.joelonsoftware.com/articles/Unicode.html

Well in case of utf-8 encoding for the first 127 codepoing we can safely
say that a character equals a byte :)

Equals? No. Bytes are not characters. (Strictly speaking, they're
codepoints, not characters.)
And anyway, it's the first _128_ codepoints.

Yes 0-127 = 128, i knew that!

Well the relationship between characters and bytes is that:

A [0-127] Unicode codepoints(characters) need 1-byte to be stored in
utf-8 encoding.

I think its also correct to say that the byte in the above situation is
the representation of our character.

[Chris Angelico]

Post by MRAB

Post by Νίκος

Post by Chris Angelico

So you are also suggesting that what gesthostbyaddr() returns is not utf-8
encoded too?
What character is 0xb6 anyways?

It isn't. It's a byte. Bytes are not characters.
http://www.joelonsoftware.com/articles/Unicode.html

Well in case of utf-8 encoding for the first 127 codepoing we can safely
say that a character equals a byte :)

Equals? No. Bytes are not characters. (Strictly speaking, they're
codepoints, not characters.)
And anyway, it's the first _128_ codepoints.

As MRAB says, even if there's a 1:1 correspondence between bytes,
codepoints, and characters, they're still not the same thing. Plus,
0xb6 is not in the first 128, so your statement is false and your
question has no answer. Do you understand why I gave you that link? If
not, go read the page linked to.

ChrisA

[MRAB]

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)

If it's ISO-8859-7 (what you've previously referred to as "greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'

You'll need to find out where that bytestring is coming from.

[Νίκος]

Post by MRAB

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)
If it's ISO-8859-7 (what you've previously referred to as "greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'
You'll need to find out where that bytestring is coming from.

Right.
But nowhere in my script(metrites.py) i use an 'Ά' so i really have no
clue where this is coming from.

And you are right if it was a byte came from an utf-8 encoding scheme
then it would be automatically decoded.

The only thing i can say for use is that this problem a[[ear only when i
cloudflare my domain "superhost.gr"

If i un-cloudlflare it it cease to display errors.

Can you tell me hpw to write the following properly:

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'

so even if the function fails "unresolved" to be returned back?
Somehow i need to capture the error.

Or it dosnt have to do it the or operand will be returned?

[MRAB]

Post by Νίκος

Post by MRAB

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)
If it's ISO-8859-7 (what you've previously referred to as "greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'
You'll need to find out where that bytestring is coming from.

Right.
But nowhere in my script(metrites.py) i use an 'Ά' so i really have no
clue where this is coming from.
And you are right if it was a byte came from an utf-8 encoding scheme
then it would be automatically decoded.
The only thing i can say for use is that this problem a[[ear only when i
cloudflare my domain "superhost.gr"
If i un-cloudlflare it it cease to display errors.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
so even if the function fails "unresolved" to be returned back?
Somehow i need to capture the error.
Or it dosnt have to do it the or operand will be returned?

If gethostbyaddr fails, it raises socket.gaierror, (which, from Python
3.3 onwards, is a subclass of OSError), so try catching that, setting
'host' to 'UnResolved' if it's raised.

Also, try printing out ascii(os.environ['REMOTE_ADDR']).

[Νίκος]

Post by MRAB

Post by Νίκος

Post by MRAB

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)
If it's ISO-8859-7 (what you've previously referred to as "greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'
You'll need to find out where that bytestring is coming from.

Right.
But nowhere in my script(metrites.py) i use an 'Ά' so i really have no
clue where this is coming from.
And you are right if it was a byte came from an utf-8 encoding scheme
then it would be automatically decoded.
The only thing i can say for use is that this problem a[[ear only when i
cloudflare my domain "superhost.gr"
If i un-cloudlflare it it cease to display errors.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
so even if the function fails "unresolved" to be returned back?
Somehow i need to capture the error.
Or it dosnt have to do it the or operand will be returned?

If gethostbyaddr fails, it raises socket.gaierror, (which, from Python
3.3 onwards, is a subclass of OSError), so try catching that, setting
'host' to 'UnResolved' if it's raised.
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

I have followed your suggestion by trying this:

try:
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
except socket.gaierror:
host = "UnResolved"

and then re-cloudlflared "superhost.gr" domain

http://superhost.gr/ gives internal server error.

[MRAB]

Post by Dave Angel

Post by MRAB

Post by Νίκος

Post by MRAB

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)
If it's ISO-8859-7 (what you've previously referred to as "greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'
You'll need to find out where that bytestring is coming from.

Right.
But nowhere in my script(metrites.py) i use an 'Ά' so i really have no
clue where this is coming from.
And you are right if it was a byte came from an utf-8 encoding scheme
then it would be automatically decoded.
The only thing i can say for use is that this problem a[[ear only when i
cloudflare my domain "superhost.gr"
If i un-cloudlflare it it cease to display errors.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
so even if the function fails "unresolved" to be returned back?
Somehow i need to capture the error.
Or it dosnt have to do it the or operand will be returned?

If gethostbyaddr fails, it raises socket.gaierror, (which, from Python
3.3 onwards, is a subclass of OSError), so try catching that, setting
'host' to 'UnResolved' if it's raised.
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "UnResolved"
and then re-cloudlflared "superhost.gr" domain
http://superhost.gr/ gives internal server error.

Try catching OSError instead. (As I said, from Python 3.3,
socket.gaierror is a subclass of it.)

[Νίκος Γκρ33κ]

Post by MRAB

Post by Dave Angel

Post by MRAB

Post by Νίκος

Post by MRAB

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97]
Premature
end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)
If it's ISO-8859-7 (what you've previously referred to as
"greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'
You'll need to find out where that bytestring is coming from.

Right.
But nowhere in my script(metrites.py) i use an 'Ά' so i really have no
clue where this is coming from.
And you are right if it was a byte came from an utf-8 encoding scheme
then it would be automatically decoded.
The only thing i can say for use is that this problem a[[ear only when i
cloudflare my domain "superhost.gr"
If i un-cloudlflare it it cease to display errors.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
so even if the function fails "unresolved" to be returned back?
Somehow i need to capture the error.
Or it dosnt have to do it the or operand will be returned?

If gethostbyaddr fails, it raises socket.gaierror, (which, from Python
3.3 onwards, is a subclass of OSError), so try catching that, setting
'host' to 'UnResolved' if it's raised.
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "UnResolved"
and then re-cloudlflared "superhost.gr" domain
http://superhost.gr/ gives internal server error.

Try catching OSError instead. (As I said, from Python 3.3,
socket.gaierror is a subclass of it.)

At least CloudFlare doesn't give me issues:

if i try this:

try:
host = os.environ['REMOTE_ADDR'][0]
except socket.gaierror:
host = "UnResolved"

then i get no errors and a valid ip back

but the above fails.

I don't know how to catch the exception with OSError.

i know only this two:

except socket.gaierror:
except socket.herror

both fail.

[MRAB]

Post by Νίκος Γκρ33κ

Post by MRAB

Post by Dave Angel

Post by MRAB

Post by Νίκος

Post by MRAB

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
I just started to have this error without changing nothing

Well, undo the nothing that you didn't change. ;)

Post by Νίκος
invalid start byte
[Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97]
Premature
end
of script headers: metrites.py
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?
What character is 0xb6 anyways?

Well, it's from a bytestring, so you'll have to specify what encoding
you're using! (It clearly isn't UTF-8.)
If it's ISO-8859-7 (what you've previously referred to as
"greek-iso"),

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
import unicodedata
unicodedata.name(b"\xb6".decode("ISO-8859-7"))

'GREEK CAPITAL LETTER ALPHA WITH TONOS'
You'll need to find out where that bytestring is coming from.

Right.
But nowhere in my script(metrites.py) i use an 'Ά' so i really have no
clue where this is coming from.
And you are right if it was a byte came from an utf-8 encoding scheme
then it would be automatically decoded.
The only thing i can say for use is that this problem a[[ear only when i
cloudflare my domain "superhost.gr"
If i un-cloudlflare it it cease to display errors.
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved'
so even if the function fails "unresolved" to be returned back?
Somehow i need to capture the error.
Or it dosnt have to do it the or operand will be returned?

If gethostbyaddr fails, it raises socket.gaierror, (which, from Python
3.3 onwards, is a subclass of OSError), so try catching that, setting
'host' to 'UnResolved' if it's raised.
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "UnResolved"
and then re-cloudlflared "superhost.gr" domain
http://superhost.gr/ gives internal server error.

Try catching OSError instead. (As I said, from Python 3.3,
socket.gaierror is a subclass of it.)

host = os.environ['REMOTE_ADDR'][0]
host = "UnResolved"

It's pointless trying to catch a socket exception here because you're
not using a socket, you're just getting a string from an environment
variable.

Post by Νίκος Γκρ33κ
then i get no errors and a valid ip back
but the above fails.
I don't know how to catch the exception with OSError.
except socket.herror
both fail.

What do you mean "I don't know how to catch the exception with
OSError"? You've tried "except socket.gaierror" and "except
socket.herror", well just write "except OSError" instead!

[Νίκος Γκρ33κ]

Post by MRAB
What do you mean "I don't know how to catch the exception with
OSError"? You've tried "except socket.gaierror" and "except
socket.herror", well just write "except OSError" instead!

try:
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
except OSError:
host = "UnResolved"

produces also an internal server error.

Are you sure is just except OSError ?

seems very general...

[Wayne Werner]

Post by Dave Angel

Post by MRAB
What do you mean "I don't know how to catch the exception with
OSError"? You've tried "except socket.gaierror" and "except
socket.herror", well just write "except OSError" instead!

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "UnResolved"
produces also an internal server error.
Are you sure is just except OSError ?

Have you ensured that 'REMOTE_ADDR' is actually a key in os.environ? I
highly recommend using the logging module to help diagnose what the actual
exception is.

HTH,
-W

[Ferrous Cranus]

Post by Wayne Werner

Post by Dave Angel

Post by MRAB
What do you mean "I don't know how to catch the exception with
OSError"? You've tried "except socket.gaierror" and "except
socket.herror", well just write "except OSError" instead!

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "UnResolved"
produces also an internal server error.
Are you sure is just except OSError ?

Have you ensured that 'REMOTE_ADDR' is actually a key in os.environ? I
highly recommend using the logging module to help diagnose what the
actual exception is.
HTH,
-W

Yes it is a key, but the problem as i suspected was cloudflare.
i had to use os.environ['HTTP_CF_CONNECTING_IP'] that cloudflare passes
as variable i the cgi enviroment in order to retrieve the visitor's ip.


try:
gi = pygeoip.GeoIP('/usr/local/share/GeoLiteCity.dat')
city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] )
host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0]
except Exception as e:
host = repr(e)


Sometimes though iam still receiving the usual
UnicodeDecodeError('utf-8', b'\xc1\xf0\xef\xf4\xf5

but only for a few ip addresses, in moste cases it works.

[Dave Angel]

Post by Ferrous Cranus

Post by Wayne Werner

Post by Dave Angel

Post by MRAB
What do you mean "I don't know how to catch the exception with
OSError"? You've tried "except socket.gaierror" and "except
socket.herror", well just write "except OSError" instead!

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "UnResolved"
produces also an internal server error.
Are you sure is just except OSError ?

Have you ensured that 'REMOTE_ADDR' is actually a key in os.environ? I
highly recommend using the logging module to help diagnose what the
actual exception is.
HTH,
-W

Yes it is a key, but the problem as i suspected was cloudflare.
i had to use os.environ['HTTP_CF_CONNECTING_IP'] that cloudflare passes
as variable i the cgi enviroment in order to retrieve the visitor's ip.
gi = pygeoip.GeoIP('/usr/local/share/GeoLiteCity.dat')
city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] )
host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0]
host = repr(e)
Sometimes though iam still receiving the usual
UnicodeDecodeError('utf-8', b'\xc1\xf0\xef\xf4\xf5
but only for a few ip addresses, in moste cases it works.

And naturally, you now know how to debug those UnicodeDecodeError
problems. Surely, the code you post here isn't what you actually do,
because when people spend time to give you detailed advice, you actually
read it, and work at understanding it.

Chortle, snort.

[Νίκος]

Post by MRAB
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

'108.162.229.97' is the result of:

print( ascii(os.environ['REMOTE_ADDR']) )

Seems perfectly valid. and also have a PTR record, so that leaved us
clueless about the internal server error.

[MRAB]

Post by Νίκος

Post by MRAB
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

print( ascii(os.environ['REMOTE_ADDR']) )
Seems perfectly valid. and also have a PTR record, so that leaved us
clueless about the internal server error.

For me, socket.gethostbyaddr('108.162.229.97') raises socket.herror,
which is also a subclass of OSError from Python 3.3 onwards.

[Νίκος Γκρ33κ]

-------- Αρχικό μήνυμα --------
Θέμα: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in
position 0: invalid start byte
Ημερομηνία: Thu, 04 Jul 2013 14:34:42 +0100
Από: MRAB <***@mrabarnett.plus.com>
Απάντηση: python-***@python.org
Προς: python-***@python.org
Ομάδες συζήτησης: comp.lang.python

Post by Νίκος

Post by MRAB
Also, try printing out ascii(os.environ['REMOTE_ADDR']).

print( ascii(os.environ['REMOTE_ADDR']) )
Seems perfectly valid. and also have a PTR record, so that leaved us
clueless about the internal server error.

For me, socket.gethostbyaddr('108.162.229.97') raises socket.herror,
which is also a subclass of OSError from Python 3.3 onwards.

Tell me how i should write the try/except please.

[Ulrich Eckhardt]

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?

I never said that. And do some research yourself, you were given plenty
of hints.

Uli

[Νίκος Γκρ33κ]

Post by Ulrich Eckhardt

Post by Νίκος

Post by Ulrich Eckhardt

Post by Νίκος
Why cant it decode the starting byte? what starting byte is that?

It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence.
Please do some research on UTF-8, that should clear it up. You could
also search for common causes of that error.

So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?

I never said that. And do some research yourself, you were given plenty
of hints.
Uli

Yes and as you can see form my responses i have tried any suggestion so
far and the problem still remains unresolved.

What you said implied that the string returned by the function cannot be
decoded as utf-8.

[Dennis Lee Bieber]

Post by Νίκος
What character is 0xb6 anyways?

It depends on the encoding... In EBCDIC it's unassigned. It's a
paragraph mark in ISO-Latin-1 (ISO-8859-1). Apparently also a paragraph
mark in ISO-Latin-9 (ISO-8859-15).

If it is valid in UTF-8, I can't find a reference. It's not a prefix
for a multi-byte character, which implies that the previous byte should
have been something in prefix or another extended byte entry...

[Ferrous Cranus]

Post by Dennis Lee Bieber

Post by Νίκος
What character is 0xb6 anyways?

It depends on the encoding... In EBCDIC it's unassigned. It's a
paragraph mark in ISO-Latin-1 (ISO-8859-1). Apparently also a paragraph
mark in ISO-Latin-9 (ISO-8859-15).
If it is valid in UTF-8, I can't find a reference. It's not a prefix
for a multi-byte character, which implies that the previous byte should
have been something in prefix or another extended byte entry...

try:
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
except:
host = "Reverse DNS Failed"

Is there a way to write the above so i cna print the error return when
it fails?

[Lele Gaifax]

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"
Is there a way to write the above so i cna print the error return when
it fails?

Try something like

try:
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
except Exception as e:
host = "Reverse DNS Failed"
print(e)

?

ciao, lele.

[Michael Torrie]

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"
Is there a way to write the above so i cna print the error return when
it fails?

Do you know what IP address causes the failure? If so write a little
python program that does the socket.gethostbyaddr and run it on the
command line! Debugging through the CGI interface sucks.

Have you been writing python tests that you can run on the command line
during your development these last weeks?

[Nobody]

Post by Νίκος
So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?

The gethostbyaddr() OS function returns a byte string with no specified
encoding. Python 3 will doubtless try to decode that to a character string
using some (probably unspecified) encoding.

Names obtained from DNS should consist entirely of ASCII characters
(gethostbyname shouldn't attempt to decode internationalised names
which use IDN, it should return the raw data).

Names obtained by other means (e.g. /etc/hosts or Active Directory) could
contain anything, but if you use non-ASCII hostnames you're asking for
trouble.

[Ferrous Cranus]

Post by Nobody

Post by Νίκος
So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?

The gethostbyaddr() OS function returns a byte string with no specified
encoding. Python 3 will doubtless try to decode that to a character string
using some (probably unspecified) encoding.

I see, but if the function returns a byte string not inutf-8 format then
how my script is uspposes to decode this byte string?

And why only this error happens when i cloudflare my domain, while when
i un-cloudflare it are reverse DNS are being resolves without problem.
So the queston is: How come it only fails when i cloidflare the domain?

Also please comment on that:

host = gethostbyaddr(....) or "UnResolved"

This will return the first argument that define the evaluation as being
true or untrue.

if function returns false the the 2nd argument.
Nut if the function gives an exception will the condition return the 2nd
argument or will the program fail?

I was udner the impression that i could avoid error handling inside
try/excepts by utilizing "or".

[Lele Gaifax]

Post by Ferrous Cranus
host = gethostbyaddr(....) or "UnResolved"
This will return the first argument that define the evaluation as
being true or untrue.
if function returns false the the 2nd argument.
Nut if the function gives an exception will the condition return the
2nd argument or will the program fail?
I was udner the impression that i could avoid error handling inside
try/excepts by utilizing "or".

No, you had the wrong impression. Why don't you simply invoke the Python
interpreter and try things out with that??

Post by Ferrous Cranus

a = 1/0 or 100

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

Post by Ferrous Cranus

a

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined

Post by Ferrous Cranus

a = 0/1 or 100
a

100
ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
***@metapensiero.it | -- Fortunato Depero, 1929.

[Νίκος Gr33k]

Post by Lele Gaifax

Post by Ferrous Cranus
host = gethostbyaddr(....) or "UnResolved"
This will return the first argument that define the evaluation as
being true or untrue.
if function returns false the the 2nd argument.
Nut if the function gives an exception will the condition return the
2nd argument or will the program fail?
I was udner the impression that i could avoid error handling inside
try/excepts by utilizing "or".

No, you had the wrong impression. Why don't you simply invoke the Python
interpreter and try things out with that??

Post by Ferrous Cranus

a = 1/0 or 100

File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

Post by Ferrous Cranus

a

File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined

Post by Ferrous Cranus

a = 0/1 or 100
a

100

Thank you Lele, i wanted to but i had no idea how to test it.
Your devision by zero is very smart thing to test!

So it proves that a condition cannot be evaluation as truthy or falsey
if one of the operators is giving out an exception.
Thank you.

[Νίκος Gr33k]

Post by Nobody

Post by Νίκος
So you are also suggesting that what gesthostbyaddr() returns is not
utf-8 encoded too?

The gethostbyaddr() OS function returns a byte string with no specified
encoding. Python 3 will doubtless try to decode that to a character string
using some (probably unspecified) encoding.
Names obtained from DNS should consist entirely of ASCII characters
(gethostbyname shouldn't attempt to decode internationalised names
which use IDN, it should return the raw data).
Names obtained by other means (e.g. /etc/hosts or Active Directory) could
contain anything, but if you use non-ASCII hostnames you're asking for
trouble.

Please help because i just happened to noticed that after having this code:

try:
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
except Exception as e:
host = "Reverse DNS Failed"


all requests are being resolves, result to:


Reverse DNS Failed as you can see here:
http://superhost.gr/?show=log&page=index.html

How can the above code not be able to reeverse dns any more and it falls
back to the failed string?

[Lele Gaifax]

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"
How can the above code not be able to reeverse dns any more and it
falls back to the failed string?

The only way to know is actually printing out the exception, either to
stderr, or better using the logging facility, as I suggested.

FYI, your code above is (almost) exactly equivalent to the simpler

try:
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
except:
host = "Reverse DNS Failed"

ciao, lele.

[Νίκος Gr33k]

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"

Yes i uses to had it like that, until i was looking for ways to make it
hold the error

except Exception as e:
print( e )
host = e

but print( e ) in the way i used to had it doesn't print out the error,
it instead gives an internal server error on browser.

I must somehow take a look at the error to understand why every visitor
i have gets UnResolved, but how since prints fails?

[Dave Angel]

Post by Νίκος Gr33k

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"

Yes i uses to had it like that, until i was looking for ways to make it
hold the error
print( e )
host = e
but print( e ) in the way i used to had it doesn't print out the error,
it instead gives an internal server error on browser.
I must somehow take a look at the error to understand why every visitor
i have gets UnResolved, but how since prints fails?

How have you been doing it all along? Just open a console onto that
server, start the appropriate version of Python interactively, and try
the things we've been talking about. If it fails the same way as within
the cgi environmnet, you get full visibility.

Or if the problems cannot be recreated outside the cgi environment, use
the log files you've been logging other problems into. Or simply open a
text file for writing, and add a file= keyword parameter to the print
function call.

print(repr(e), file=myfile)

[Νίκος Gr33k]

Post by Dave Angel
Or if the problems cannot be recreated outside the cgi environment, use
the log files you've been logging other problems into. Or simply open a
text file for writing, and add a file= keyword parameter to the print
function call.
print(repr(e), file=myfile)

Yes you are correct, problem need to be recreated within the cgi env.

try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)


http://superhost.gr/?show=log&page=index.html shows explicitly the same
kind of error that python interpreter via cli gave me the same error!

[Benjamin Kaplan]

Post by Lele Gaifax

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"
How can the above code not be able to reeverse dns any more and it
falls back to the failed string?

The only way to know is actually printing out the exception, either to
stderr, or better using the logging facility, as I suggested.
FYI, your code above is (almost) exactly equivalent to the simpler
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"
ciao, lele.

They aren't equivalent. "except Exception" won't catch KeyboardInterrupt or
SystemExit or a few others that you really don't want to catch in a generic
error handler. You should almost never have a bare except.

[Dave Angel]

On 07/05/2013 02:51 AM, Νίκος Gr33k wrote:

<SNIP>

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]
host = "Reverse DNS Failed"

Don't ever catch a bare Exception class. Make it more specific to the
particular problem you're trying to suppress.

In particular, your previous problem with the utf-8 decoding will also
be caught by the Exception class, so it'll get all lumped together as
"Reverse DNS Failed".

Post by Dave Angel
http://superhost.gr/?show=log&page=index.html
How can the above code not be able to reeverse dns any more and it falls
back to the failed string?

Since you've not made any progress with all the other suggestions, how
about if you decompose this line into several, and see just which one is
failing? Maybe that'll tell you what's going on. In general,
suppressing an exception without knowing why it's firing is a huge mistake.

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]

refactor that to:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]

and see which one throws the exception. Then once you have that,
examine the exact parameters that might be triggering the problem. In
particular, figure out the exact types and values for remadd and tuple3.

print(type(remadd) + " : " + repr(remadd))

Of course, print itself won't work too well in a CGI environment. But
you must have solved that problem by now, either using log files or
running the program excerpt in a regular console.

[Νίκος Gr33k]

Post by Dave Angel

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]

remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
and see which one throws the exception. Then once you have that,
examine the exact parameters that might be triggering the problem. In
particular, figure out the exact types and values for remadd and tuple3.
print(type(remadd) + " : " + repr(remadd))

I'am not sure how iam supposed to write this: i just tried this:


try:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except:
host = type(remadd) + " : " + repr(remadd)


but iam getting an internal server error.

I didnt print it as you said but its the same thing host var gets
printed later on.

Now, why would this give an internal server error?

[Dave Angel]

Post by Dave Angel

Post by Dave Angel

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]

remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
and see which one throws the exception. Then once you have that,
examine the exact parameters that might be triggering the problem. In
particular, figure out the exact types and values for remadd and tuple3.
print(type(remadd) + " : " + repr(remadd))

remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
host = type(remadd) + " : " + repr(remadd)
but iam getting an internal server error.
I didnt print it as you said but its the same thing host var gets
printed later on.
Now, why would this give an internal server error?

I have no idea what causes an internal server error. It's up to you to
get the output of the expression to some location you can examine.
Easiest way is to run those 3 lines directly on the server, not in the
cgi environment.

But if you don't have any debugging tools, then STOP right now and build
some. Use logging, or redirect print, or do something that the server
folks provide as debugging aids. But just blindly guessing is
ludicrous. So also is throwing out clues by using a bare except.

Assigning that string to host makes no sense at all. And neither does
putting it in the except clause. You want to get that string to YOUR
eyes, not to the server who might get an internal server error.

[Νίκος Gr33k]

Post by Dave Angel

Post by Dave Angel

Post by Dave Angel

Post by Dave Angel
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]

remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
and see which one throws the exception. Then once you have that,
examine the exact parameters that might be triggering the problem. In
particular, figure out the exact types and values for remadd and tuple3.
print(type(remadd) + " : " + repr(remadd))

remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
host = type(remadd) + " : " + repr(remadd)
but iam getting an internal server error.
I didnt print it as you said but its the same thing host var gets
printed later on.
Now, why would this give an internal server error?

I have no idea what causes an internal server error. It's up to you to
get the output of the expression to some location you can examine.
Easiest way is to run those 3 lines directly on the server, not in the
cgi environment.
But if you don't have any debugging tools, then STOP right now and build
some. Use logging, or redirect print, or do something that the server
folks provide as debugging aids. But just blindly guessing is
ludicrous. So also is throwing out clues by using a bare except.
Assigning that string to host makes no sense at all. And neither does
putting it in the except clause. You want to get that string to YOUR
eyes, not to the server who might get an internal server error.

I don't think running it via 'cli' would help much, since its a
cgi-script and ip addr function have no meaning calling them in plain
our of a cgi environment but here it is:


Python 3.3.2 (default, Jun 3 2013, 16:18:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Post by Dave Angel

Post by Dave Angel

Post by Dave Angel
import os
remadd = os.environ('REMOVE_ADDR')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '_Environ' object is not callable

[Dave Angel]

On 07/05/2013 04:49 AM, Νίκος Gr33k wrote:
<SNIP>

Post by Νίκος Gr33k
I don't think running it via 'cli' would help much, since its a
cgi-script and ip addr function have no meaning calling them in plain

No idea how to parse "have no meaning calling them in plain our of a cgi
environment"

Post by Νίκος Gr33k
Python 3.3.2 (default, Jun 3 2013, 16:18:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Post by Dave Angel

Post by Dave Angel
import os
remadd = os.environ('REMOVE_ADDR')

File "<stdin>", line 1, in <module>
TypeError: '_Environ' object is not callable

But there were two problems with the code you faithfully copied from my
earlier post. One was already pointed out by feedthetroll, that I
accidentally changed REMOTE_ADDR to REMOVE_ADDR.

The other one is perhaps more subtle; I replaced square brackets with
parentheses.

Post by Νίκος Gr33k

Post by Dave Angel
import os
remadd = os.environ['REMOTE_ADDR']

I get an error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR'


but presumably your machine actually has such an environment variable.

Isn't that mistake something you could easily have caught? Or were you
just blindly pasting my bugs without understanding what I was trying to
do with refactoring?

Anyway, I can't see any reason why the rest of the sequence shouldn't
behave identically from a terminal as it does in CGI.

[Νίκος Gr33k]

Post by Dave Angel
<SNIP>

Post by Νίκος Gr33k
I don't think running it via 'cli' would help much, since its a
cgi-script and ip addr function have no meaning calling them in plain

No idea how to parse "have no meaning calling them in plain our of a cgi
environment"

Post by Νίκος Gr33k
Python 3.3.2 (default, Jun 3 2013, 16:18:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Post by Dave Angel

Post by Dave Angel
import os
remadd = os.environ('REMOVE_ADDR')

File "<stdin>", line 1, in <module>
TypeError: '_Environ' object is not callable

But there were two problems with the code you faithfully copied from my
earlier post. One was already pointed out by feedthetroll, that I
accidentally changed REMOTE_ADDR to REMOVE_ADDR.
The other one is perhaps more subtle; I replaced square brackets with
parentheses.

Post by Νίκος Gr33k

Post by Dave Angel
import os
remadd = os.environ['REMOTE_ADDR']

File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR'
but presumably your machine actually has such an environment variable.
Isn't that mistake something you could easily have caught? Or were you
just blindly pasting my bugs without understanding what I was trying to
do with refactoring?
Anyway, I can't see any reason why the rest of the sequence shouldn't
behave identically from a terminal as it does in CGI.

Yes i didnt see your typo and i have corrected it:


try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)


Ima still receiving the same kind of erro as i did with cli as well.

You can view the error in the very first line here:

http://superhost.gr/?show=log&page=index.html


which yields: TypeError("'_Environ' object is not callable",)

[Lele Gaifax]

Post by Νίκος Gr33k
http://superhost.gr/?show=log&page=index.html

No, visiting that page simply emit the standard Apache error page,
without details.

Post by Νίκος Gr33k
which yields: TypeError("'_Environ' object is not callable",)

Dave already told you the reason[1]. Νίκος, *read* **and** *understand*
our *whole* answers to your questions, otherwise we are wasting time,
you, and us!

ciao, lele.

[1] “… The other one is perhaps more subtle; I replaced square brackets
with parentheses.”

[Νίκος Gr33k]

Post by Lele Gaifax

Post by Νίκος Gr33k
http://superhost.gr/?show=log&page=index.html

No, visiting that page simply emit the standard Apache error page,
without details.

Post by Νίκος Gr33k
which yields: TypeError("'_Environ' object is not callable",)

Dave already told you the reason[1]. Νίκος, *read* **and** *understand*
our *whole* answers to your questions, otherwise we are wasting time,
you, and us!
ciao, lele.
[1] “… The other one is perhaps more subtle; I replaced square brackets
with parentheses.”

I read carefully all of tour answer please try to load again:

http://superhost.gr/?show=log&page=index.html

Di i miss an explanation on this?

TypeError("'_Environ' object is not callable",)

[Νίκος Gr33k]

Post by Νίκος Gr33k
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
host = repr(e)
which yields: TypeError("'_Environ' object is not callable",)

Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error?

I noticed that if i remove my domain from cloudflare the gethostbyaddr
as it uses too months now.

[Lele Gaifax]

Post by Νίκος Gr33k

Post by Νίκος Gr33k
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
host = repr(e)
which yields: TypeError("'_Environ' object is not callable",)

Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error?

Yes, I'd try to understand the error message, and eventually lookup the
documentation on os.environ to see what type of object it is bound
to. The solution should be easy.

ciao, lele.

[Νίκος Gr33k]

Post by Lele Gaifax

Post by Νίκος Gr33k

Post by Νίκος Gr33k
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
host = repr(e)
which yields: TypeError("'_Environ' object is not callable",)

Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error?

Yes, I'd try to understand the error message, and eventually lookup the
documentation on os.environ to see what type of object it is bound
to. The solution should be easy.

Looks now when i print( repr(e)) i get

UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

but what string does it try to decode and jeeps failing?

[Lele Gaifax]

Post by Νίκος Gr33k
Looks now when i print( repr(e)) i get
UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3
but what string does it try to decode and jeeps failing?

Reasonably it's the second one, as the first clearly seems the tag of
the decoder that tried to translate it to Unicode.

As already explained, your immediate goal should be trying to understand
from *where* that byte string is coming. I can't help on that, sorry.

ciao, lele.

[Νίκος Gr33k]

Post by Lele Gaifax

Post by Νίκος Gr33k
Looks now when i print( repr(e)) i get
UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3
but what string does it try to decode and jeeps failing?

Reasonably it's the second one, as the first clearly seems the tag of
the decoder that tried to translate it to Unicode.

2nd one of what? 2nd byte in order?
Can ou show me form which characters does this string consist of so we
migth have an idea of where its coming from if we know what it looks like?

Post by Lele Gaifax
As already explained, your immediate goal should be trying to understand
from *where* that byte string is coming. I can't help on that, sorry.

Thats what i'm trying to do.
If i completely remove the gethostbyaddr fucntion adds function then
there is nor problem.

The problem is recreating when the script tries to decode a hostname.

For some bizarre reason if i exclude my domain from CloudFlare then the
reverse DNS resolution of the visitors hostname would be returned properly.

I will do it right now for you to see.

[Νίκος Gr33k]

Post by Νίκος Gr33k
Thats what i'm trying to do.
If i completely remove the gethostbyaddr fucntion adds function then
there is nor problem.
The problem is recreating when the script tries to decode a hostname.
For some bizarre reason if i exclude my domain from CloudFlare then the
reverse DNS resolution of the visitors hostname would be returned properly.
I will do it right now for you to see.

Precisely as i have said it would happen:

I removed the domain form CloudFlare and your domains visting my website
appearing as usual.

Its only when i cloudflare it and the UnicodeError happens.

But why?!?! I see no reason as to why when my domain becomes Cloudflared
the gethostbyaddr() fails.

What kind of weird strings does it return back?

[Lele Gaifax]

Post by Νίκος Gr33k

Post by Lele Gaifax

Post by Νίκος Gr33k
UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3
but what string does it try to decode and jeeps failing?

Reasonably it's the second one, as the first clearly seems the tag of
the decoder that tried to translate it to Unicode.

2nd one of what? 2nd byte in order?

You asked “what string” (although you probably meant “which string”):

UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

first string-------^^^^^^^
second string---------------^^^^^^^^^^^^^^^^^^^^^^

ciao, lele.

[Νίκος Gr33k]

Post by Νίκος Gr33k
UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3
first string-------^^^^^^^
second string---------------^^^^^^^^^^^^^^^^^^^^^^

Hold on please!

From where do these dashes and carets characters come from?
Also from where do you see 2 strings?

Looking at that: UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

ic an understandn onlt that utf-8 has failsed decoding some byte stream
satrting with \xb6

I totally not follow...

[f***@gmx.de]

Post by Νίκος Gr33k

Post by Νίκος Gr33k
UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3
first string-------^^^^^^^
second string---------------^^^^^^^^^^^^^^^^^^^^^^

Hold on please!
From where do these dashes and carets characters come from?

ROTFL!!!!
Nikos, you made my day, again! Fun is back in these threads!

Post by Νίκος Gr33k
Also from where do you see 2 strings?

Look, my little dumb baby: The dashes and carets point to the strings.
The first one being 'utf-8', the second one starting with '\xb6\xe3\xed' (being a bytestring, therefore the b before the ')
Sorry, I forgot. You are not using python for your business, therefore you can't know, that strings in python can for example be identified by surrounding '.

Post by Νίκος Gr33k
ic an understandn onlt that utf-8 has failsed decoding some byte stream
satrting with \xb6

So ... where did you call utf-8() so that it could try to decode something?

[Νίκος Gr33k]

Post by Lele Gaifax
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR

Wait!
Are you saying that the ip address is being returned as a byte string
which then i have to decode with something like:

host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0]

?

[f***@gmx.de]

Post by Νίκος Gr33k
...
Wait!
Are you saying that the ip address is being returned as a byte string
host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0]

Wait!
I get a decode error when python tries to automatically decode a bytestring assuming it to be utf-8 encoded.
I am sure the error will disappear, when I try to decode it explicit using utf-8. Heureka! I got it!

Or in other words:
If a big stone falls on my foot accidently it hurts.
But I am sure it will not hurt, if take that same stone and throw it on my foot.
Heureka! I got it!

Post by Νίκος Gr33k
Also you have been shown this link and I feel you really need to read it.
http://slash7.com/2006/12/22/vampires/

[f***@gmx.de]

Post by Νίκος Gr33k
...
Wait!
Are you saying that the ip address is being returned as a byte string
host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0]

Wait!
I get a decode error when python tries to automatically decode a bytestring
assuming it to be utf-8 encoded.
I am sure the error will disappear, when I try to decode it explicit using
utf-8. Heureka! I got it!

Or in other words:
If a big stone falls on my foot accidently, it hurts.
------------------------------------------^
But I am sure it will not hurt, if take that same stone and throw it on my foot.

Heureka! I got it!

Post by Νίκος Gr33k
Also you have been shown this link and I feel you really need to read it.
http://slash7.com/2006/12/22/vampires/

[Dave Angel]

Post by Νίκος Gr33k

Post by Lele Gaifax
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR

Wait!
Are you saying that the ip address is being returned as a byte string
host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0]

Don't fix the problem till you understand it. Figure out who is dealing
with a byte string here, and where that byte string came from. Adding a
decode, especially one that's going to do the same decode as your
original error message, is very premature.

You're quoting from my error output, and that's caused because I don't
have such an environment variable. But you do. So why aren't you in
there debugging it? And why on earth are you using the complex
expression instead of a refactored one which might be simple enough for
you to figure out what's wrong with it.

There is definitely something strange going on with that os.environ
reference (NOT call). So have you yet succeeded in running the factored
lines? If you can't get them to run, at least up to the point that you
get that unicode error, then you'll make progress only by guessing.

Get to that interactive debug session, and enter the lines till you get
an error. Then at least you know which line is causing the error.

xxx = os.environ['REMOTE_HOST']
yyy = socket.gethostbyaddr(xxx)
host = yyy[0]


I'll bet the real problem is you're using some greek characters in the
name of the environment variable, rather than "REMOTE_HOST" So
everything you show us is laboriously retyped, hiding the real problems
underneath.

[Lele Gaifax]

Post by Dave Angel
You're quoting from my error output, and that's caused because I don't
have such an environment variable. But you do.

Dave, maybe you already know, but that variable is "injected" by the CGI
mechanism, is not coming from the OP shell environment.

As Νίκος discovered, when he "cloudfare" (whatever that means) his site,
the REMOTE_HOST envvar contains some (I guess) latin-greek encoded
string, and the remote address is carried by a different envvar...

ciao, lele.

[Νίκος Gr33k]

Post by Lele Gaifax

Post by Dave Angel
You're quoting from my error output, and that's caused because I don't
have such an environment variable. But you do.

Dave, maybe you already know, but that variable is "injected" by the CGI
mechanism, is not coming from the OP shell environment.
As Νίκος discovered, when he "cloudfare" (whatever that means) his site,
the REMOTE_HOST envvar contains some (I guess) latin-greek encoded
string, and the remote address is carried by a different envvar...

Exactly only when i CloudFlare(www.cloudflare.com) the domain the
hostname cannot be retrieved.

At least i managed to solve this by:

try:
host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0]
except Exception as e:
host = repr(e)


Seems like when you cloudflare a domain you can no longer have the
originates ip address of the visitor but you have to read the above
environmental variable to be bale to retrieve it!

[f***@gmx.de]

Post by Dave Angel
...
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
host = type(remadd) + " : " + repr(remadd)

Hey, if no one told you before:
You are allowed to read what other people suggest you to do, think about it and so correct obvious typos: 'REMO*V*E_ADDR'

Post by Dave Angel
...
print( ascii(os.environ['REMOTE_ADDR']) )
For me, socket.gethostbyaddr('108.162.229.97') raises socket.herror,
which is also a subclass of OSError from Python 3.3 onwards.

[Lele Gaifax]

Post by Benjamin Kaplan

Post by Lele Gaifax
FYI, your code above is (almost) exactly equivalent to the simpler

They aren't equivalent. "except Exception" won't catch KeyboardInterrupt or
SystemExit or a few others that you really don't want to catch in a generic
error handler. You should almost never have a bare except.

I know, that's why I added "(almost)", I was just trying to explain why
he wasn't able to see the problem.

Thanks for pointing out the difference,
ciao, lele.

[Νίκος Gr33k]

-------- Αρχικό ΌήΜυΌα --------
ΘέΌα: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in
position 0: invalid start byte
ΗΌεροΌηΜία: Fri, 05 Jul 2013 15:16:45 +0300

Post by f***@gmx.de

Post by Νίκος Gr33k

Post by Νίκος Gr33k
UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3
first string-------^^^^^^^
second string---------------^^^^^^^^^^^^^^^^^^^^^^

Hold on please!
From where do these dashes and carets characters come from?

ROTFL!!!!
Nikos, you made my day, again! Fun is back in these threads!

Oh my God! i though they were the actual sting not that the pointed to
the string themselves! I ebn laughted at me!

Post by f***@gmx.de

Post by Νίκος Gr33k
Also from where do you see 2 strings?

Look, my little dumb baby: The dashes and carets point to the strings.
The first one being 'utf-8', the second one starting with '\xb6\xe3\xed' (being a bytestring, therefore the b before the ')
Sorry, I forgot. You are not using python for your business, therefore you can't know, that strings in python can for example be identified by surrounding '.

Indeed, but where is the 2nd sttring coming from?

Post by f***@gmx.de

Post by Νίκος Gr33k
ic an understandn onlt that utf-8 has failsed decoding some byte stream
satrting with \xb6

So ... where did you call utf-8() so that it could try to decode something?

Thats still unknown,

At least i manages to solve this by:

try:
host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0]
except Exception as e:
host = repr(e)


Seems like when you cloudflare a domain you can no longer have the
originates ip address of the visitor but you have to read the above
environmental variable to be bale to retrieve it!

[r***@gmail.com]

shut off the HD, start the SO form an USB and shut on again the HD