SFTP and unicode file names...

Discussion:

Joseph Galbraith

2004-10-05 16:48:02 UTC

Okay, I've been forced to face up to the unfortunate truth:

Unix directories can contain files encoded in multiple
different char-sets and the server has no way to tell
what these multiple char-sets are and translate them
to UTF-8. Because the transformation is one way, once
the server has mistranslated the filename, there
is no way for the client to get back to the original
data.

So, for these file-systems, the best possible thing
to do is send the filename raw and let the client
(with help from the user decode it.)

On the other hand, maximum possible interoperability
between different language and regions is obtained
through use of UTF-8 where available.

I haven't been able to come up with a solution I
really like.

Here are some possibilities:

1. Let the server say what it is going to use,
UTF-8 or 'undefined-raw' at the beginning
of the sftp session.

pro: simple. really simple.
con: Doesn't address cases where the server
might be able to do utf-8 for some file
systems (ext-3 under fedora, I think is
one example) but not others.

2. Allow the filename to be prefixed by a flag
byte saying it is raw. For example, the
byte 0xFF is an invalid UTF-8 lead byte.
If the first byte of the filename is 0xFF,
then the 0xFF is discarded, and the rest
of string is the 'raw, undefined' filename
data.

pro: It handles the real life complexity of
being able to tell sometimes, but not
others.
con: It is a little more complex, and a bit
icky.

3. Give the filename structure. Filenames are
always specified in the following structure:

uint32 length of the structure
boolean utf-8
byte filename[length-1]

pro: This also handles being able to give
UTF-8 sometimes, but not all the time.
pro: This isn't icky.
con: This is a bit more complex.

4. Use the high order bit of the length field
to flag raw more. In practice, no file name
will ever be more than 2 gig long :-) We
can safely borrow that bit for other purposes.

pro: This also handles being able to give
UTF-8 sometimes, but not all the time.
con: Slightly icky, a little more complex.

Can anyone think of a better solution?

I think I prefer solution three, I could live
with 2 or 4; I'd really rather not go with 1.

What do others think?

Thanks,

Joseph

Jacob Nevins

2004-10-06 23:20:35 UTC

Permalink

Unix directories can contain files encoded in multiple different
char-sets and the server has no way to tell what these multiple
char-sets are and translate them to UTF-8. Because the transformation
is one way, once the server has mistranslated the filename, there is
no way for the client to get back to the original data.
So, for these file-systems, the best possible thing to do is send the
filename raw and let the client (with help from the user decode it.)

I support this. While it would be nice for all filenames to be in UTF-8
or reliably convertible to/from it, it is beyond the power of this
working group to cause this to happen, so this requirement simply won't
be implemented.

(We've pinched stuff from NFS before. Do they have any views on this
issue?)

On the other hand, maximum possible interoperability between different
language and regions is obtained through use of UTF-8 where available.
I haven't been able to come up with a solution I really like.
1. Let the server say what it is going to use, UTF-8 or
'undefined-raw' at the beginning of the sftp session.
pro: simple. really simple.
con: Doesn't address cases where the server might be able to do
utf-8 for some file systems (ext-3 under fedora, I think is
one example) but not others.

It's true that this is rather limiting. But if your directory tree has
heterogenous UTF-8-ness, don't you run into problems with having some
path components "raw" and others UTF-8? Given our approach of paths
being rather opaque things, I can't think of a way round this.

Therefore, I suspect that something like this simple approach may have
to be taken...

2. Allow the filename to be prefixed by a flag byte saying it is raw.
For example, the byte 0xFF is an invalid UTF-8 lead byte. If the
first byte of the filename is 0xFF, then the 0xFF is discarded,
and the rest of string is the 'raw, undefined' filename data.
pro: It handles the real life complexity of being able to tell
sometimes, but not others.
con: It is a little more complex, and a bit icky.
3. Give the filename structure. Filenames are always specified in the
uint32 length of the structure
boolean utf-8
byte filename[length-1]
pro: This also handles being able to give UTF-8 sometimes, but not
all the time.
pro: This isn't icky.
con: This is a bit more complex.

In particular, backwards-compatible implementations now need two sets of
filename-handling code.

4. Use the high order bit of the length field to flag raw more. In
practice, no file name will ever be more than 2 gig long :-) We
can safely borrow that bit for other purposes.
pro: This also handles being able to give UTF-8 sometimes, but not
all the time.
con: Slightly icky, a little more complex.
Can anyone think of a better solution?
I think I prefer solution three, I could live with 2 or 4; I'd really
rather not go with 1.

I think I slightly prefer 2 to 4, if the issue I mention above can be
got round.

Niels Möller

2004-10-08 11:28:45 UTC

Permalink

Post by Joseph Galbraith
1. Let the server say what it is going to use,
UTF-8 or 'undefined-raw' at the beginning
of the sftp session.
pro: simple. really simple.
con: Doesn't address cases where the server
might be able to do utf-8 for some file
systems (ext-3 under fedora, I think is
one example) but not others.

I'd much prefer the simple way. Just let the server say what character
set it uses, and do no conversions at the server end (a server that
really knows what it is doing MAY handle a file system with mixed
character sets by converting them to and from utf-8, and use utf-8
exclusively on the wire, but I don't think that is very practical).
Some possible values are "unknown", "usascii", "iso-8859-1", "utf-8",
and "utf-8-with-normalization-form-c".

(Note that utf-8 with undefined normalization is suboptimal for
filenames and for identifiers in general).

This approach should work fine for systems that uses utf-8
consistently in the filesystem (I think windows-ce does that, not sure
about more regular windows versions).

I think the typical unix filesystem uses usascii for all "system
files", and then has some home directories with iso-8859-x names, some
with utf-8 names, which all include usascii. (I admit that I don't
have any experience with asian unix installations). Then it will work
reasonably to set the advertised charset from the user's $LC_CTYPE.

There will naturally be some difficulties if different parts of the
filesystem tree uses different character sets, or if there are single
pathnames that use components (directory names) with different
character sets. However, those difficulties are precisely the same
with *local* access to the file system, so it makes absolutely no
sense to try to solve them in the sftp spec.

Best regards,
/Niels

Jeffrey Hutzelman

2004-10-08 16:05:30 UTC

Permalink

On Friday, October 08, 2004 13:28:45 +0200 Niels Möller

Post by Niels MÃ¶ller

So far, so good. Note that the appropriate set of possible values is that
contained in the IANA Character Set registry
(http://www.iana.org/assignments/character-sets). When a character set in
that registry has more than one name or alias, the alias designated as
"preferred MIME name" SHOULD be used.

But see below...

Post by Niels MÃ¶ller
(Note that utf-8 with undefined normalization is suboptimal for
filenames and for identifiers in general).

Yes. I'd normally argue that in any case where UTF-8 will be used on the
wire, it probably ought to be normalized. However, the key issue here is
not really normalization of names sent by the server but of those sent by
the client, which refer to the names in the filesystem. Unfortunately,
there are filesystems that use unnormalized Unicode, so the server would be
required to do path processing itself, normalizing all the names in each
directory until it found one matching what the user requested. That's
going to get unwieldy real fast, I think.

Post by Niels MÃ¶ller
I think the typical unix filesystem uses usascii for all "system
files", and then has some home directories with iso-8859-x names, some
with utf-8 names, which all include usascii. (I admit that I don't
have any experience with asian unix installations). Then it will work
reasonably to set the advertised charset from the user's $LC_CTYPE.

I would expect this to be more or less the case. In fact, I'd expect that
on most systems, most users will be using the _same_ character set.

Post by Niels MÃ¶ller
There will naturally be some difficulties if different parts of the
filesystem tree uses different character sets, or if there are single
pathnames that use components (directory names) with different
character sets. However, those difficulties are precisely the same
with *local* access to the file system, so it makes absolutely no
sense to try to solve them in the sftp spec.

I agree -- solving that problem is beyond our ability, and I do not think
it is worth the effort to try.

There is one thing I'd do differently. Whenever possible, I'd prefer that
the server not just advertise the character set but actually do the
conversions. The theory here is that the server is likely to have at its
disposal the tools needed to convert between Unicode and any other
character set in use on the server. The _client_, on the other hand, can
only reasonably be expected to have those tools for character sets in use
on the client. When these sets are disjoint, interoperability demands the
use on the wire of a common character set (i.e. Unicode).

Let the server advertise the character set it thinks is in use.
Let the client decide whether it wants UTF-8 or raw bytes.

-- Jeff

Niels Möller

2004-10-08 17:46:30 UTC

Permalink

Post by Jeffrey Hutzelman
On Friday, October 08, 2004 13:28:45 +0200 Niels Möller

Post by Niels MÃ¶ller
(Note that utf-8 with undefined normalization is suboptimal for
filenames and for identifiers in general).

There's actually one important use case where normalization doesn't
matter. That is if the client first requests a directory listing, and
then presents those names to the user, via some graphical file dialog,
or via TAB-completion. In this case, the only important thing is that
the client doesn't modify the coding when a name is copied from the
directory listing to the open request.

Post by Jeffrey Hutzelman
Unfortunately, there are filesystems that use unnormalized Unicode,

That's going to be painful. Examples?

Post by Jeffrey Hutzelman
There is one thing I'd do differently. Whenever possible, I'd prefer
that the server not just advertise the character set but actually do
the conversions. [...]
Let the server advertise the character set it thinks is in use.
Let the client decide whether it wants UTF-8 or raw bytes.

Good idea. I agree it makes sense to give clients that choice.

I'm not so sure I share your preference, though; I'm afraid subtle
problems may occur when file names are converted to utf-8, processed
by client ui code, and back.

One interesting case: When converting from the local charset to utf-8,
it is natural to require that the server uses normalized utf-8, right?
(With canonical normalization, not compatibility normalization).

However, if the server's local character set happens to be utf-8, and
some file names are unnormalized, then having the server normalize the
data for transmission is likely to break things.

Regards,
/Niels

Jeffrey Hutzelman

2004-10-08 18:21:37 UTC

Permalink

On Friday, October 08, 2004 19:46:30 +0200 Niels Möller

Post by Niels MÃ¶ller

Post by Jeffrey Hutzelman
On Friday, October 08, 2004 13:28:45 +0200 Niels Möller

Post by Niels MÃ¶ller
(Note that utf-8 with undefined normalization is suboptimal for
filenames and for identifiers in general).

Correct.

Post by Niels MÃ¶ller

Post by Jeffrey Hutzelman
Unfortunately, there are filesystems that use unnormalized Unicode,

That's going to be painful. Examples?

I don't believe Windows normalizes anything.

Post by Niels MÃ¶ller
Good idea. I agree it makes sense to give clients that choice.
I'm not so sure I share your preference, though; I'm afraid subtle
problems may occur when file names are converted to utf-8, processed
by client ui code, and back.
One interesting case: When converting from the local charset to utf-8,
it is natural to require that the server uses normalized utf-8, right?
(With canonical normalization, not compatibility normalization).
However, if the server's local character set happens to be utf-8, and
some file names are unnormalized, then having the server normalize the
data for transmission is likely to break things.

Yes. If the server's local character set is UTF-8, it SHOULD NOT
normalize. It it's something else, then the server should convert to UTF-8
using whatever rules it deems appropriate, and should accept any reasonable
representation from the client (particularly, it SHOULD NOT require
normalized input).

Joseph Galbraith

2004-10-09 22:48:22 UTC

Permalink

Post by Jeffrey Hutzelman
On Friday, October 08, 2004 13:28:45 +0200 Niels Möller

Post by Niels MÃ¶ller

So far, so good. Note that the appropriate set of possible values is
that contained in the IANA Character Set registry
(http://www.iana.org/assignments/character-sets). When a character set
in that registry has more than one name or alias, the alias designated
as "preferred MIME name" SHOULD be used.
But see below...

Post by Niels MÃ¶ller
(Note that utf-8 with undefined normalization is suboptimal for
filenames and for identifiers in general).

Yes. I'd normally argue that in any case where UTF-8 will be used on
the wire, it probably ought to be normalized. However, the key issue
here is not really normalization of names sent by the server but of
those sent by the client, which refer to the names in the filesystem.
Unfortunately, there are filesystems that use unnormalized Unicode, so
the server would be required to do path processing itself, normalizing
all the names in each directory until it found one matching what the
user requested. That's going to get unwieldy real fast, I think.

I would expect this to be more or less the case. In fact, I'd expect
that on most systems, most users will be using the _same_ character set.

I agree -- solving that problem is beyond our ability, and I do not
think it is worth the effort to try.
There is one thing I'd do differently. Whenever possible, I'd prefer
that the server not just advertise the character set but actually do the
conversions. The theory here is that the server is likely to have at
its disposal the tools needed to convert between Unicode and any other
character set in use on the server. The _client_, on the other hand,
can only reasonably be expected to have those tools for character sets
in use on the client. When these sets are disjoint, interoperability
demands the use on the wire of a common character set (i.e. Unicode).

I would definitely prefer to see the server do the translation
when it can... that's why we went to UTF-8 in the first place.
If the server knows the charset, it should just do the conversion.
For example, Windows systems don't know about EUC-JIS (for Japanese
encoding), they use an alternate encoding call Shift-JIS. So the server
can easily translate from EUC-JIS to UTF-8, using built-in os support
(if it is using EUC-JIS anyway.) There has to be custom code written
on the client side though to do the conversion. That is yucky. So
it really is better for the server to do the translation.

The problem is if the server doesn't (really) know the charset,
and tries to do the conversion anyway (as would happen, for
example, using the users LC_TYPE). In this case, the conversion
loses data, and can not be reversed. This means that the client
not only can't display a meaningful filename, but it can't even
open the file by sending back the garbage name (because the server
can't reverse the conversion.)

So, what I'd like is a way for the server to convert what it
can, and to give the client something that can be used to open
files even if it can't convert them.

So... how about the following--

1. A extension the server sends to advise of the charset
it thinks is most likely in use (probably from the
users LC_CTYPE under unix, windows will probably just
always send UTF-8.)

By the way, how does one get a charset out of the iana
registry from the users NLS environment under most
unixes? Is it a simple thing to do?

2. An extension the client can use to control the translation
on the server; something like translation-mask:

SSH_TRANSLATE_KNOWN 0x00000001
Translate files with a known charset. If the
translation fails, use the rawname instead.

SSH_TRANSLATE_GUESSED 0x00000002
Translate files if the server has a reasonable
idea what the charset is, but doesn't know.

(For example, using the users LC_CTYPE.)

If the translation fails (an invalid character
for example) the server should use the rawname
instead.

SSH_TRANSLATE_INCLUDE_KNOWN_RAW 0x0000004
Include the rawname as an a part of the attrib
if charset was known, and translation was successful.
(If translation was unsuccessful, the rawname is
sent as the name.)

(New attribute field SSH_FILEXFER_ATTR_RAW_NAME.)

SSH_TRANSLATE_INCLUDE_GUESSED_RAW 0x0000004
Include the rawname as an a part of the attrib
if charset was guessed, and translation was successful.

The initial mode is SSH_TRANSLATE_KNOWN|SSH_TRANSLATE_GUESSED.

3. Allow the server and the client to prepend a 0xFF to any
filename to indicate it should be passed down to the system
APIs without further processing.

The thing this does is allow most clients to continue
to operate as they do under sftp v4 & 5, except that
if the translation fails, behavior is well defined
and there is a way to open the file, even if the
filename can not be displayed correctly.

If a user runs into trouble, they can set INCLUDE_GUESSED_RAW,
or turn off TRANSLATE_GUESSED.

Okay... after all that, I've just realized that the per-file
tag for rawmode is more complicated than I thought-- it has
to be per path component. Rats.

I really wanted to be able to rely on the server as much
as possible and only punt the exceptions to the client
for processing... and the client wouldn't really process
them, because it would know it didn't know the charset
(cause if it was the one the server thought it was,
it would have worked on the server side.) It would just
use them as opaque handles to at the file contents--
or the user would have to manually intervene to display
the filename.

Argh... well, I'm going to go off an think now.

- Joseph

Niels Möller

2004-10-11 14:54:06 UTC

Permalink

Post by Joseph Galbraith
I would definitely prefer to see the server do the translation
when it can... that's why we went to UTF-8 in the first place.

I think there's one more use case that you need to consider, which I
expect is quite common:

The remote filesystem using the foo charset. The local system using
the same foo charset. Why do I think this is common? Because on both
sides, it's the same user's files, and the user is likely to use his
or her favourite charset (iso-8859-1, utf-8, euc-jis, whatever) on
most or all systems where he or she has an account.

What you call "raw mode" will work fine in this case, no matter if the
sftp implementation on server or client side knows about the foo
charset.

I like Jeffrey Hutzelman's proposal: Have two modes of operation, and
let the client select which mode it prefers,

1. Server tells client the server's best guess as to what character
set is used for filenames, and doesn't convert filenames in any
way.

2. All filenames on the wire are utf-8. Server converts filenames to
and from utf-8 on a best effort basis, according to it's best
guess of the actual charset. (What's the right thing to do if/when
conversion fails, I don't know yet).

In both cases, server can tell client what charset is used on the
server side (or "unknown") at startup. Client can select mode of
operation either as a global protocol state, or a per request-flag. I
don't think I care very much if the mode selection is global or per
request; I'd expect most clients to choose one mode and stick to that.

The reasons I like this are:

* It is fairly simple.

* The client has the final say on whether or not the server should do
any conversion.

* The use case above is easily supported, without requiring any
special server (or client) support for the obscure foo charset.

* I don't see any use cases where the above scheme fails, and a more
complicated scheme could do better.

Regards,
/Niels

Jeffrey Hutzelman

2004-10-11 15:52:41 UTC

Permalink

On Monday, October 11, 2004 16:54:06 +0200 Niels Möller

Post by Niels MÃ¶ller

Post by Joseph Galbraith
I would definitely prefer to see the server do the translation
when it can... that's why we went to UTF-8 in the first place.

I think there's one more use case that you need to consider, which I
The remote filesystem using the foo charset. The local system using
the same foo charset. Why do I think this is common? Because on both
sides, it's the same user's files, and the user is likely to use his
or her favourite charset (iso-8859-1, utf-8, euc-jis, whatever) on
most or all systems where he or she has an account.
What you call "raw mode" will work fine in this case, no matter if the
sftp implementation on server or client side knows about the foo
charset.
I like Jeffrey Hutzelman's proposal: Have two modes of operation, and
let the client select which mode it prefers,
1. Server tells client the server's best guess as to what character
set is used for filenames, and doesn't convert filenames in any
way.
2. All filenames on the wire are utf-8. Server converts filenames to
and from utf-8 on a best effort basis, according to it's best
guess of the actual charset. (What's the right thing to do if/when
conversion fails, I don't know yet).

IMHO it is important that the server identify its character set up front,
so that the client is able to use that information in deciding which mode
to use. Also, the server needs to be able to indicate that it is incapable
of performing Unicode conversion. I'd like to be able to say that the
server MUST be capable of performing the conversion, but I don't think
that's realistic.

If the server is capable of doing conversion, then conversion from the
local character set to unicode should not be able to fail. If it does,
something is very wrong.

I can think of only two cases in which the conversion from UTF-8 to the
local character set can fail. The first is when the input is somehow
invalid (bad UTF-8, contains illegal Unicode code points, etc). We could
handle this in a number of ways, up to and including terminating the
connection. :-)

The second case is when the input is valid, but contains characters not
present in the local character set. The server's mapping should be good
enough to handle cases where the same local character can be represented in
multiple ways in unicode (for example, there are at least two ways to write
Å in unicode, but only one way in iso-8859-1). For characters which are
genuinely not in the local character set, the server can return an
appropriate error depending on context. For example, trying to open a file
whose name contains an untranslateable character will always fail with
something ENOENT-like. Trying to create such a file should fail with an
illegal filename.

-- Jeff

der Mouse

2004-10-11 16:39:41 UTC

Permalink

Post by Jeffrey Hutzelman
IMHO it is important that the server identify its character set up
front, [...]

What about cases where there is no well-defined character set, such as
most Unix variants? (Suppose I have a file whose name consists of the
octets 0xc3 0xa1 0xd0 0xbf. Is that a file named LATIN SMALL LETTER A
WITH ACUTE - CYRILLIC SMALL LETTER PE (UTF-8), a file named LATIN
CAPTIAL LETTER A WITH TILDE - INVERTED EXCLAMATION MARK - LATIN CAPITAL
LETTER EDH - INVERTED QUESTION MARK (8859-1[%]), a file named GREEK
CAPITAL LETTER GAMMA - some character whose name I'm not sure of -
GREEK CAPITAL LETTER PI - GREEK CAPITAL LETTER OMEGA WITH TONOS
(8859-7[%]), a file whose name represents the integer 3173013909 rather
than any character sequence (base 254 with digits 0x01-0x2e and
0x30-0xff), or what? (Given how nonsensical the others are, the last
of those may actually be the most plausible.)

Nobody but the entities (software and/or humans) the file was created
for and used by can tell. Certainly ssh/sshd can't, not without being
told somehow.

Anything predicated on the assumption that every filename consists of
characters will run into this problem. I strongly believe there needs
to be some provision for filenames to be treated as opaque octet
sequences rather than insisting that they are character sequences.

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

[%] Names taken from Unicode 0080-00FF and 0370-03FF; I don't have a
handy reference to the names 8859 uses.

Niels Möller

2004-10-11 20:23:17 UTC

Permalink

Post by Jeffrey Hutzelman
On Monday, October 11, 2004 16:54:06 +0200 Niels Möller

Post by Niels MÃ¶ller
I like Jeffrey Hutzelman's proposal: Have two modes of operation, and
let the client select which mode it prefers,
1. Server tells client the server's best guess as to what character
set is used for filenames, and doesn't convert filenames in any
way.
2. All filenames on the wire are utf-8. Server converts filenames to
and from utf-8 on a best effort basis, according to it's best
guess of the actual charset. (What's the right thing to do if/when
conversion fails, I don't know yet).

IMHO it is important that the server identify its character set up
front, so that the client is able to use that information in deciding
which mode to use.

Agree.

Post by Jeffrey Hutzelman
Also, the server needs to be able to indicate that it is incapable
of performing Unicode conversion. I'd like to be able to say that
the server MUST be capable of performing the conversion, but I don't
think that's realistic.

The server can be incapable of doing a proper conversion in at least
two ways: Either the server has no idea whatsoever which charset to
use to interpret its local filenames. Or it knows somehow that it
should use a prticular characterset (say, euc-jis), but it hasn't been
compiled with support for that conversion.

I don't think it is possible to forbid the first failure mode; we must
allow the server to say charset "unknown" or "" in this case.

We could forbid the second case, in effect forcing the server to say
"unknown" when it in fact knows the character set, but isn't able to
convert it. But I don't think it is a good idea to do that; it is
better to at least tell the client what the charset is.

Post by Jeffrey Hutzelman
I can think of only two cases in which the conversion from UTF-8 to
the local character set can fail.

I don't think conversion utf-8 -> local is a big problem. Some
filenames can't be represented, and they must simply be treated as
non-existing files. And invalid utf-8 should be a protocol error
(under no circumstances should an implementation be allowed to say
that it's using utf-8, and then send invalid utf-8 filenames).

Conversion the other way is more difficult. The problem is, when the
server thinks it should convert from A to utf-8 (based, for example,
on examining the user's $LC_CTYPE), but in fact some file names are
stored using a different charset B, where B sequences are invalid when
interpreted as A-characters. Only example I can come up with off the
top of my head is A = utf-8 and B = latin1, but I'm pretty sure there
are some other examples where neither A nor B is utf-8.

I could think of some ways to deal with this: (i) pretend the files
don't exist, or (ii) replace unexpected characters or character
sequences with uniquely chosen private use characters. I think (i)
will be a very confusing failure mode for users, and (ii) seems quite
ugly.

Telling the client "sorry, here's some strange filename I can't
convert to utf-8, you can try again with raw filenames, if you like"
seems simpler to udnerstand. Note that this can happen even (or
perhaps it's even more likely to happen) when the local character set
is believed to be utf-8, and some file names violate this assumption.

Regards,
/Niels

Jeffrey Hutzelman

2004-10-11 21:01:55 UTC

Permalink

On Monday, October 11, 2004 22:23:17 +0200 Niels Möller

Post by Niels MÃ¶ller

The server can be incapable of doing a proper conversion in at least
two ways: Either the server has no idea whatsoever which charset to
use to interpret its local filenames. Or it knows somehow that it
should use a prticular characterset (say, euc-jis), but it hasn't been
compiled with support for that conversion.
I don't think it is possible to forbid the first failure mode; we must
allow the server to say charset "unknown" or "" in this case.
We could forbid the second case, in effect forcing the server to say
"unknown" when it in fact knows the character set, but isn't able to
convert it. But I don't think it is a good idea to do that; it is
better to at least tell the client what the charset is.

Agree.

Post by Niels MÃ¶ller

Post by Jeffrey Hutzelman
I can think of only two cases in which the conversion from UTF-8 to
the local character set can fail.

Agree.

Post by Niels MÃ¶ller
Telling the client "sorry, here's some strange filename I can't
convert to utf-8, you can try again with raw filenames, if you like"
seems simpler to udnerstand. Note that this can happen even (or
perhaps it's even more likely to happen) when the local character set
is believed to be utf-8, and some file names violate this assumption.

I think it is more likely that the local character set is believed to be
UTF-8 and actually is not, than that we will believe the local character
set to be something "flat" like iso-8859-1 and then find characters which
are invalid in that set.

In any case, I agree the behaviour you describe seems to be the most sane
thing to do in this situation.

FWIW, it is worth noting that it is possible to tell with fairly high
probability whether a particular string is UTF-8 or some flat 8-bit
character set, because valid UTF-8 is fairly well structured and the
sequences which represent non-7-bit characters are fairly unlikely to occur
in flat character sets. So when conversion to UTF-8 is being done by the
server, it is possible for an implementation to apply a heuristic to allow
support of filesystems containing both UTF-8 and an 8-bit character set,
even in separate components of the same pathname.

-- Jeff

r***@roumenpetrov.info

2004-10-12 06:21:41 UTC

Permalink

Post by Joseph Galbraith
[SNIP]
in flat character sets. So when conversion to UTF-8 is being done by the
server, it is possible for an implementation to apply a heuristic to allow

heuristic ???
This is inpossible.
As example a valid cyrillic name in UTF-8 can be converted into valid name
in more that one charset (ISO-8859-5, KOI8-R, CP1251).
Client with help from user should tell server how to convert "utf-8" <->
8-bit character set, i.e. prefered charset for filename on server(remote
host).

Post by Joseph Galbraith
support of filesystems containing both UTF-8 and an 8-bit character set,
even in separate components of the same pathname.

Niels Möller

2004-10-12 13:03:52 UTC

Permalink

Post by Jeffrey Hutzelman
I think it is more likely that the local character set is believed to
be UTF-8 and actually is not, than that we will believe the local
character set to be something "flat" like iso-8859-1 and then find
characters which are invalid in that set.

I agree, with the reservation that I don't have much experience with
character sets that use eight-bit characters, and lots of different
shift states.

Post by Jeffrey Hutzelman
FWIW, it is worth noting that it is possible to tell with fairly high
probability whether a particular string is UTF-8 or some flat 8-bit
character set,

That may be correct (at least that's an advertised property of utf-8),
but as far as I can see, that doesn't help much unless you have some
clue about *which* flat 8-bit character set was used. If you know that
a file system contains mixed latin-1 and utf-8 filenames (and no other
character sets), you can probably make the heuristics work most of the
time, but if you have a filesystem with mixed utf-8, iso-8859-1,
iso-8859-5 and koi8, it gets rather difficult.

I don't think heuristic decoding is a good idea. One problem is that
you really need to be able to convert both ways:

Say you have a directory containing filenames in utf-8 and iso-8859-1.
The client asks for a directory listing, and you give it a utf-8
listing, using some working heuristic for converting the latin-1 names
to utf-8 on the fly. Next, the user selects one file using the
client's ui, and the client tries to open it. Now the server has to
figure out if the filename to be opened should be converted to latin-1
or not (in the worst case, there exists files with both the utf-8 name
and the corresponding latin1 name). I think this gets ugly.

The hack I mentioned, to replace the sequences that are invalid utf8
with uniquely chosen private use characters, have a better chance of
surviving the roundtrip to the client, but I'm sure that has other
problems.

Regards,
/Niels

denis bider

2004-10-23 02:00:58 UTC

Permalink

Hello Joseph,

any information on when the SFTP v6 draft will be available?

I will be starting implementation of a new module soon and my design is
pending availability of the SFTP v6 specification.

What is the current progress? When can the v6 draft be anticipated?

Thanks!

denis

Joseph Galbraith

2004-10-25 19:02:20 UTC

Permalink

I made the deadline (sent to IETF at 3:00 AM this
morning my time.)

I put a big ** DO NOT IMPLEMENT ** in it though. My
plan is to spend some time @ IETF and to publish a
revision on the monday after we meet.

Here are the changes (a rough list anyway):

- Added NOFOLLOW option.
- Made TEXT_HINT an attrib of it's own. Servers that 'know'
might include this information as part of the directory
read; if the server is going to have to do lots of work to
figure it out, it should wait for a stat request expressing
interest in the data.
- Added 'filename-charset' and 'filename-translation-control'
extensions (which still need some polishing in the language.)

Also added the ability for the server to set an attrib-bits flag
if the filename translation to UTF-8 failed, and to include the
untranslated name in the attributes.
- Add 'versions' extension to allow server to express non-contigous
set of supported versions to client, and 'version' extension
from client-to-server for server to pick one. (Because the
client to server request isn't allowed to fail w/o closing
the channel, it doesn't require a round trip. The client
can fire it and move on without waiting.)
- Added link-count attribute and mime-type attribute.
- Added space available extension.
- Added DIR_NOT_EMPTY, NOT_A_DIRECTORY, INVALID_FILENAME and
LINK_LOOP error codes. Please, everyone, look at the list
of error codes and suggest errors codes that we should have
but aren't there. If something can go wrong on your OS and
we can't represent it, now is the time to have a dialog about
it.

So... draft should appear shortly, and we WILL HAVE another draft
on Monday after the IETF meets.

(I'll be available for anyone interested in helping with edits or
chatting about the draft from Monday evening through Tuesday.)

Thanks,

Joseph

Post by denis bider
Hello Joseph,
any information on when the SFTP v6 draft will be available?
I will be starting implementation of a new module soon and my design is
pending availability of the SFTP v6 specification.
What is the current progress? When can the v6 draft be anticipated?
Thanks!
denis

jason bailey

2004-10-26 15:31:47 UTC

Permalink

Would it be within scope to suggest that a future version of the SFTP
server provide a signed receipt of the file transfer(on request)?

This feature, would do much to alleviate a lot of our issues with secure
file transfers.

thanks

-Jason

Jon Bright

2004-10-27 06:16:02 UTC

Permalink

Hi,

Post by jason bailey
Would it be within scope to suggest that a future version of the SFTP
server provide a signed receipt of the file transfer(on request)?
This feature, would do much to alleviate a lot of our issues with secure
file transfers.

I'm not in a position to answer your question wrt. scope, but I'm
curious as to what exactly the signature would signify? That the server
has definitely received the file at a given time?

--
Jon Bright
Silicon Circus Ltd.
http://www.siliconcircus.com

jason bailey

2004-10-27 12:09:40 UTC

Permalink

Exactly.

A receipt provides proof that a given file was delivered at a specific
time.

Receipts usually contains information like file name, a
fingerprint(hash), and a time stamp which is then signed by the servers
private key. It's part of a non-repudiation process.

To give an example. I deal with the transfer of data from my companies
production network to a variety of third party organizations. For the
majority of file transfers SFTP fits my needs. However there are
occasions where the data is sensitive enough, or we have some sort of
time based contractual obligation, where it is necessary for us to get a
receipt of transfer.

Right now my options are to go with some heavy weight B2B protocol which
requires a team of engineers and an architect to get set up properly. Or
implement our own solution, which inevitably results in placing some
form of server (of our devising) on another companies network. This
tends to make their network people very antsy.

If I could get a receipt capability into a standard like SFTP. It would
allow me promote SFTP over all the other options. It would make my
management happy, and would make my company dealings with other
companies a lot smoother.

Jason

Post by Jon Bright
Hi,

I'm not in a position to answer your question wrt. scope, but I'm
curious as to what exactly the signature would signify? That the server
has definitely received the file at a given time?

denis bider

2004-10-27 13:54:03 UTC

Permalink

Technically speaking, an extension to the SFTP protocol could be implemented
in which the server would testify (with a digital signature) that a file of
a certain name, attributes and contents existed on the server at a certain
time.

Whether or not this file was uploaded entirely by a certain user would be a
more complex challenge because SFTP has no such concept as "uploading" or
"downloading" a whole file. You have random access and you can pretty much
scratch anywhere you want in the remote filesystem.

An extension that would produce a certificate of a file's existence at a
certain time would be fairly straightforward. Provide an extension request
type for requesting the certificate, and define the contents of a receipt.

If providing a certificate of the file's existence on the server is
insufficient, and you must really provide a receipt which includes
information about the act of uploading, this could be done, too. For
example, a file for which you require an upload receipt must be opened with
a certain flag or set of flags which signal that you're going to do
receipted-uploading. When you open the file, you are allowed to append to
the file only (like uploading in TEXT mode). When you close it, the server
sends a RECEIPT message rather than STATUS. The format of the RECEIPT
message is what needs to be defined.

Whether or not this is something for a separate Internet-Draft (documenting
the SFTP extension) or something that can be added to SFTP itself as an
optional feature is, I guess, up for the workgroup or the SFTP draft editor
to decide. In my view, the first solution type (certificate of file's
existence) would be more apt for a separate draft because it is fairly
orthogonal to existing functionality. The second solution type (certificate
of the upload act with special flags for opening the file) might be better
documented in SFTP itself because of the flag's definition.

If the 2nd solution (upload receipt) is required, a question to answer is:
what if the user needs to upload an extremely large file and the connection
breaks? Should the "upload receipt" process allow for a file to be resumed?
If so, how do you know that the file being resumed is one that was actually
uploaded in a previous session by the same user, and that it wasn't lying
there on the server belonging to someone else?

denis

-----Original Message-----
Sent: Wednesday, October 27, 2004 14:10
To: Jon Bright
Subject: Re: future SFTP version question
Exactly.
A receipt provides proof that a given file was delivered at a
specific time.
Receipts usually contains information like file name, a
fingerprint(hash), and a time stamp which is then signed by
the servers private key. It's part of a non-repudiation process.
To give an example. I deal with the transfer of data from my
companies production network to a variety of third party
organizations. For the majority of file transfers SFTP fits
my needs. However there are occasions where the data is
sensitive enough, or we have some sort of time based
contractual obligation, where it is necessary for us to get a
receipt of transfer.
Right now my options are to go with some heavy weight B2B
protocol which requires a team of engineers and an architect
to get set up properly. Or implement our own solution, which
inevitably results in placing some form of server (of our
devising) on another companies network. This tends to make
their network people very antsy.
If I could get a receipt capability into a standard like
SFTP. It would allow me promote SFTP over all the other
options. It would make my management happy, and would make my
company dealings with other companies a lot smoother.
Jason

Post by Jon Bright
Hi,

Post by jason bailey
Would it be within scope to suggest that a future version of the
SFTP server provide a signed receipt of the file transfer(on
request)?
This feature, would do much to alleviate a lot of our issues with
secure file transfers.

I'm not in a position to answer your question wrt. scope, but I'm
curious as to what exactly the signature would signify?

That the server

Post by Jon Bright
has definitely received the file at a given time?

Peter Gutmann

2004-10-27 14:24:28 UTC

Permalink

Post by denis bider
Whether or not this is something for a separate Internet-Draft (documenting
the SFTP extension) or something that can be added to SFTP itself as an
optional feature is, I guess, up for the workgroup or the SFTP draft editor
to decide.

If you're going to produce signed receipts as proof-of-delivery, you're well
into S/MIME / PGP territory. This seems to be going way beyond what SSH
should be doing (depending on how far you want to take this you'd need to
reinvent significant chunks of PGP or S/MIME), it'd be better to just define a
signed content type for one of those formats and use that.

Peter.

Jon Bright

2004-10-27 15:16:39 UTC

Permalink

Post by Peter Gutmann
If you're going to produce signed receipts as proof-of-delivery, you're well
into S/MIME / PGP territory. This seems to be going way beyond what SSH
should be doing (depending on how far you want to take this you'd need to
reinvent significant chunks of PGP or S/MIME), it'd be better to just define a
signed content type for one of those formats and use that.

I agree that the spec for the signature itself belongs elsewhere - but I
can see an argument for building the ability to ask the server to
produce such a signature into SFTP. The signature standard need only be
incorporated by reference.

As I read it, Jason's only requirement is for the server to sign to say
"file X (hash Y) was on this server at time T". This could use
signature schemes already in existence, providing that SFTP were able to
ask the server to produce the signature and transfer the produced
signature back to the client.

--
Jon Bright
Silicon Circus Ltd.
http://www.siliconcircus.com

Niels Möller

2004-10-28 07:22:26 UTC

Permalink

Post by Peter Gutmann

If you're going to produce signed receipts as proof-of-delivery, you're well
into S/MIME / PGP territory. This seems to be going way beyond what SSH
should be doing [...]

I agree this seems to be beyond what standard sftp is supposed to do.
And I also don't see why sftp extensions are crucial for supporting
the given use case. One could use plain sftp (or *any* file transfer
mechanism, for that matter) and the following convention:

* Client uploads the file "foo" into a particular directory or using some
particular naming scheme.

* When upload is complete (sftp close), the server processes the
file using the signature mechanisms of its choice (pgp, s/mime,
whatever), and writes a receipt as a new file "foo.receipt".

* The client downloads "foo.receipt". Everyone is happy.

Adding extensions to sftp to do this have the potential advantage of
letting us standardize it, but I seriously doubt it's worth the
effort; it seems too obscure and specialized.

Regards,
/Niels

jason bailey

2004-10-29 12:59:35 UTC

Permalink

Allow me to clarify some of my terminology.

When I first made the suggestion, I had in mind a possible change to the
protocol where a receipt would be returned upon successful completion of
a file transfer. Upon further thought and input I recognized that an
extension would be more then suitable.To be accurate that is no longer a
receipt, it would be a manifest. Proof of existence of a specific file
on a particular server at a particular time, with no regard to how it
got there.

Whether it is a manifest file or a receipt, both provide similar
functionality, which is an aspect of non-repudiation.

The mechanism that you describe is technically correct. However it is a
simplification of the issue and misses some of of the inherent problems.
Yes, you can write a server (utilizing any protocol) that will accept a
connection , receive a file, manifest the file and then return the
manifest. In fact my current solution for this is done over https to a
customized tomcat server.

So if I have a solution in place why am I bringing this up?

My answer relates in part to the last observation

Post by Niels MÃ¶ller
but I seriously doubt it's worth the
effort; it seems too obscure and specialized.

Non-repudiation is a growing demand in the corporate world. It is
inextricably tied into secure file transfers. In a system which requires
non-repudiation, sftp fulfills all the necessary requirements ( key
handling, secure transmission, validation ) everything, except for the
digitally signed receipt/manifest.

There are standards that implement the necessary non-repudiation. EbXML
and Rossetta are the two that come to my mind. However these are ugly,
bloated, protocols. Completely unsuitable to do something as simple and
as straight forward as moving a file from a to b.

The reason I am bringing this up is because from my point of view having
sftp adopt this as an extension would create a standard which would
result in quicker deployments, better integration with other companies,
happier customers, happier management, and costs savings that extend
into the 10's of millions of american dollars for my company alone.

I also see this as something that would be equally beneficial to the
third party companies I work with as it is to ours.

Is it obscure? Not from my perspective, nor the companies I work with.
Is it specialized? possibly.
Is it worth the effort. I really can't answer that from this groups
perspective.

I was surprised to realize that no one had brought this up before. I
don't have a problem if, at this time, the working group doesn't believe
that this is a suitable fit as an extension to the current protocol.
However I do believe it deserves more critical consideration.

-Jason

Post by Niels MÃ¶ller
I agree this seems to be beyond what standard sftp is supposed to do.
And I also don't see why sftp extensions are crucial for supporting
the given use case. One could use plain sftp (or *any* file transfer
* Client uploads the file "foo" into a particular directory or using some
particular naming scheme.
* When upload is complete (sftp close), the server processes the
file using the signature mechanisms of its choice (pgp, s/mime,
whatever), and writes a receipt as a new file "foo.receipt".
* The client downloads "foo.receipt". Everyone is happy.
Adding extensions to sftp to do this have the potential advantage of
letting us standardize it, but I seriously doubt it's worth the
effort; it seems too obscure and specialized.
Regards,
/Niels

denis bider

2004-10-29 21:33:21 UTC

Permalink

Post by jason bailey

Post by Niels MÃ¶ller
but I seriously doubt it's worth the
effort; it seems too obscure and specialized.

I agree with Jason's assessment and disagree about this extension being "too
obscure and specialized". It is so simply because there is no simple way of
doing it. If we come up with an extension that provides a simple way of
doing it, I believe there will be plenty demand. I would implement it in our
server.

Someone needs to draft an extension though. That ought to be Jason because
he has background knowledge about the problem and can likely come up with
something that will be simple enough yet will work.

I propose this be done here because we've had at least 2 positive responses
(I don't recall if there were more) and just one negative. If this is an
optional extension, people who don't want to implement will not be impacted
by it, whether it's part of SFTP or a separate draft.

What does Joseph as the SFTP draft editor think about making this an
extension in the SFTP draft, one like MD5 hashes for instance? (which I
think were an excellent idea btw)

If this is too complex to be an extension in the same draft, I think Jason
can just go get xml2rfc or a similar utility and write up an Internet-Draft
for this - am I right? How would he submit it when he writes one?

denis

Joseph Galbraith

2004-10-29 22:16:05 UTC

Permalink

Post by denis bider

Post by jason bailey

Post by Niels MÃ¶ller
but I seriously doubt it's worth the
effort; it seems too obscure and specialized.

I agree with Jason's assessment and disagree about this extension being "too
obscure and specialized". It is so simply because there is no simple way of
doing it. If we come up with an extension that provides a simple way of
doing it, I believe there will be plenty demand. I would implement it in our
server.
Someone needs to draft an extension though. That ought to be Jason because
he has background knowledge about the problem and can likely come up with
something that will be simple enough yet will work.
I propose this be done here because we've had at least 2 positive responses
(I don't recall if there were more) and just one negative. If this is an
optional extension, people who don't want to implement will not be impacted
by it, whether it's part of SFTP or a separate draft.
What does Joseph as the SFTP draft editor think about making this an
extension in the SFTP draft, one like MD5 hashes for instance? (which I
think were an excellent idea btw)
If this is too complex to be an extension in the same draft, I think Jason
can just go get xml2rfc or a similar utility and write up an Internet-Draft
for this - am I right? How would he submit it when he writes one?

Well, below is a _very_ rough version of an extension. I'd be
more than happy to have Jason's expertise to help get this
fleshed out where it will work for him (and us.)

And since non-repudiation has recently whacked me over the head,
I'd agree that I suspect for most of us implementing sftp, it
will probably become non-obscure and not-so-specialized soon, if
it hasn't already.

(One alternate proposal I had was to put together a AS profile for
running of sftp, AS4... but I think that might be _really_
heavy weight, and what Jason seems to be saying is we already
have most of what we need anyway.)

- Joseph

---

9.2 File Signing

This extension allows a client to request that the server provide a
signature proving that it has recieve a correct copy of the data.

byte SSH_FXP_EXTENDED
uint32 request-id
string "sign-filename" / "sign-handle"
string filename [UTF-8] / file-handle
string pubkey-algorithm-list [from transport draft]

byte SSH_FXP_EXTENDED_REPLY
uint32 request-id
string filename-used-in-signature [UTF8]
int64 time-of-request
attrib file-attrib
string public-key [from transport draft]
string signature [from transport draft]

filename
Used if 'sign-filename' is specified; indicates the name of the
file to use. The hash will be of the file contents as it would
appear on the wire if the file were opened with no special flags.

file-handle
Used if 'sign-handle' is specified; specifies a file handle to
read the data from. The handle MUST be a file handle, and
ACE4_READ_DATA MUST have been included in the desired-access when

Galbraith Expires April 29, 2005 [Page 46]

Internet-Draft SSH File Transfer Protocol October 2004

the file was opened.

If this file handle was opened in TEXT mode, the signature must be
made of the data as it would be sent on the wire.

pubkey-algorithm-list
The algorithm list of publickeys the client is capable of
supporting. If the server cannot provide a signature using any of
the algorithm requested, it should respond with a STATUS
response, with UNSUPPORTED error code.

In the case of x509v3 (see below), no attempt is made to negotiate
which algorithms are supported within the certificate chain.

filename-used-in-signature
time-of-request
file-attrib
The filename actually used, time-of-request, and file attribute
structure which were used to make the signature. The filename, in
particular, may be difference from that specified if the server
applied some sort of canonicalization, or may be empty if the
request was a sign-handle and the server couldn't map from a
handle to a filename.

public-key
The public-key the server used to sign the filename. This is a
data structure as specified in the secsh transport specification
[2].

This draft specifies an additional public format,
***@filexfer.secsh.ietf.org. The key data in this case is the
x509v3 certificate DER encoded. The signature algorithm used for
this key type is one of three types:

ssh-dss
Encoded as per the secsh transport specification [2].

ssh-rsa
Encoded as per the secsh transport specification [2].

***@filexfer.secsh.ietf.org
A DER encoded pkcs #7 signature with detached data, as
specified by .

The prefered signature algorithm is pkcs7.

Galbraith Expires April 29, 2005 [Page 47]

Internet-Draft SSH File Transfer Protocol October 2004

signature
The signature over the following data, formatted as specified in
the secsh transport specification [2] or as above.

string filename [UTF-8]
string username-of-signature-requestor [UTF-8]
int64 time-of-request
attrib file-attrib
string file-data

filename
The filename of the file being signed. If this is a
'sign-filename' request, the filename MUST be included,
although it MAY be canonicalized. If this is a sign-handle,
the filename SHOULD be included; however, if the server can not
map backwards from a handle to a filename, the filename may be
zero length.

username-of-signature-requestor
This is the username by which the client authenticated to the
system.

time-of-request
This is the time that the signature request was made, format as
described by file attributes. (Section 6.7)

file-attrib
File attribute structure describing the attributes of the file.
The server SHOULD include any fields that it can, even if it
would not normally retrieve the data. In particular, the
server SHOULD include owner, group, acl, permissions, and
mime-type data if it can support these field. The size and
attrib-bits fields SHOULD also be included.

file-data
The file data as an SSH string.

jason bailey

2004-11-01 15:34:48 UTC

Permalink

I'd be more then willing to work with Joseph on the draft. I'm going to
take the next couple of days to get familiar with the current
documentation and then I'll see what I can add.

Jason

On Fri, 2004-10-29 at 18:16, Joseph Galbraith wrote:
<snip>

Post by Joseph Galbraith
Well, below is a _very_ rough version of an extension. I'd be
more than happy to have Jason's expertise to help get this
fleshed out where it will work for him (and us.)
And since non-repudiation has recently whacked me over the head,
I'd agree that I suspect for most of us implementing sftp, it
will probably become non-obscure and not-so-specialized soon, if
it hasn't already.
(One alternate proposal I had was to put together a AS profile for
running of sftp, AS4... but I think that might be _really_
heavy weight, and what Jason seems to be saying is we already
have most of what we need anyway.)
- Joseph

<snip>

Niels Möller

2004-10-30 16:40:01 UTC

Permalink

Post by jason bailey
Allow me to clarify some of my terminology.

[...]

Post by jason bailey
Non-repudiation is a growing demand in the corporate world. It is
inextricably tied into secure file transfers. In a system which requires
non-repudiation, sftp fulfills all the necessary requirements ( key
handling, secure transmission, validation ) everything, except for the
digitally signed receipt/manifest.

I'm afraid, I'm not really familiar with the non-repudiation area.

Post by jason bailey
Is it obscure? Not from my perspective, nor the companies I work with.
Is it specialized? possibly.
Is it worth the effort. I really can't answer that from this groups
perspective.
I was surprised to realize that no one had brought this up before. I
don't have a problem if, at this time, the working group doesn't believe
that this is a suitable fit as an extension to the current protocol.
However I do believe it deserves more critical consideration.

Then I think the first thing you have to do is to write up the
requirements. "Non-repudiation" is a very fuzzy concept to me, and
I'll have a hard time participating in discussion of details in a
non-repudiation mechanism.

I don't know if the others in the secsh wg are familiar with
non-repudiation, but perhaps you have to seek feedback elsewhere in
order to get the requirements right.

When the requirements are clear, we'll have a better foundation for
designing or evaluating extensions to sftp.

Regards,
/Niels

der Mouse

2004-10-30 21:22:13 UTC

Permalink

Post by Niels MÃ¶ller

Non-repudiation is a growing demand in the corporate world. [...]

I'm afraid, I'm not really familiar with the non-repudiation area.

Non-repudiation is a way of arranging that it is impossible (well,
infeasible - about as impossible as anything gets in cryptography) to
later claim "no I didn't do that". (As a non-crypto example, for
example, my signature on a document gives some non-repudiable assurance
I've read the document.)

It's not clear how it's being used here: is the client trying to
protect itself against a server saying "no I never received that file"
or is the server trying to protect itself against a client saying "no I
didn't send that file"? (The mechanisms sketched appear to be designed
for the former, but that strikes me as an unlikely desire.)

When I first saw the mechanisms outlined, it appeared to me that they
were designed to give the client a certificate which could demonstrate
to a third party that the server did indeed receive the file; while
this does have non-repudiation uses, it seemed to me more designed to
allow the client to prove the fact of transmission to a third paty
without requiring the proof to involve contacting the receiving host or
its admins.

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

Peter Gutmann

2004-11-01 03:39:27 UTC

Permalink

It's a fuzzy concept to everyone, so much so that after 20-odd years of trying
the X.509 guys gave up and renamed the nonRepudiation flag in certs to
something that actually had a meaning. Calling it a "delivery receipt" would
be better, that's what S/MIME (which is the only major standard to
specifically address this) calls it. In fact you could probably lift a lot of
the S/MIME stuff, since they've looked at it in some detail.

Peter.

Ben Laurie

2004-11-01 16:41:58 UTC

Permalink

Post by Peter Gutmann

Its actually a concept with a perfectly well-defined meaning in law -
which is, it is something that is _defined by law_ to be non-repudiable
(i.e. you can deny it, but it doesn't get you anywhere). An example is
the UK Customs and Excise electronic VAT returns - they are
non-repudiable by statute.

If anyone cares, this whole debate caused me to write a paper about this
(and other things) with a lawyer: http://www.apache-ssl.org/tech-legal.pdf

Section 2.7 deals with non-repudiation.

Cheers,

Ben.

--
ApacheCon! 13-17 November! http://www.apachecon.com/

http://www.apache-ssl.org/ben.html http://www.thebunker.net/

"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff

Peter Gutmann

2004-11-02 08:23:30 UTC

Permalink

Well, that's the problem, legal nonrepudiation is well-established and well-
defined, but technical nonrepudiation isn't. The best advice I've seen (from
talking to lawyers) is (1) back everything up with paper documents and written
signatures and (2) pray you never become the test case.

Peter.

Ben Laurie

2004-11-02 12:11:56 UTC

Permalink

Post by Peter Gutmann

Technical nonrepudiation is just a stupid idea, which could be why
no-one ever agrees about it.

Bill Sommerfeld

2004-11-02 14:02:34 UTC

Permalink

Post by Peter Gutmann
Well, that's the problem, legal nonrepudiation is well-established and well-
defined, but technical nonrepudiation isn't. The best advice I've seen (from
talking to lawyers) is (1) back everything up with paper documents and written
signatures and (2) pray you never become the test case.

I think I've seen enough on this thread.

My take:

Reusing keys used for authentication and/or key management within SSH to
also sign the receipts seems like a bad idea.

Defining a receipt format is out of scope for this WG, in part because
it involves too many political and financial layer considerations.

A draft explaining how to carry someone else's receipt format as an
opaque bag of bits within ssh/sftp might potentially be in scope, if an
appropriate one which meets relevant layer-8 and layer-9 requirements
exists and can be referenced.

- Bill

(N. B. since it's been a while since I've seen those T-shirts for sale
at an IETF... I'm using the quasi-satirical IETF extension of the ISO
reference model to add layer 8 (financial) and layer 9 (political) on
top of layer 7 (application).

jason bailey

2004-10-27 16:51:07 UTC

Permalink

You've made some very good points.

Initially I had considered a flag for the transfer which would result in
a returned RECEIPT rather then a STATUS.

However, given your observations,and reviewing what I am looking for I
see no problem with it being an extension to the protocol.

I'm not certain that it is, as you put it, orthogonal to the current
functionality. There already appears to be an existing extension to
support a hashing of a file to verify the contents.

In my mind I see this as very similar. We are merely verifying the
contents and that it is on a particular server at a given time.

Jason

Post by denis bider
Technically speaking, an extension to the SFTP protocol could be implemented
in which the server would testify (with a digital signature) that a file of
a certain name, attributes and contents existed on the server at a certain
time.
Whether or not this file was uploaded entirely by a certain user would be a
more complex challenge because SFTP has no such concept as "uploading" or
"downloading" a whole file. You have random access and you can pretty much
scratch anywhere you want in the remote filesystem.
An extension that would produce a certificate of a file's existence at a
certain time would be fairly straightforward. Provide an extension request
type for requesting the certificate, and define the contents of a receipt.
If providing a certificate of the file's existence on the server is
insufficient, and you must really provide a receipt which includes
information about the act of uploading, this could be done, too. For
example, a file for which you require an upload receipt must be opened with
a certain flag or set of flags which signal that you're going to do
receipted-uploading. When you open the file, you are allowed to append to
the file only (like uploading in TEXT mode). When you close it, the server
sends a RECEIPT message rather than STATUS. The format of the RECEIPT
message is what needs to be defined.
Whether or not this is something for a separate Internet-Draft

(documenting

Post by denis bider
the SFTP extension) or something that can be added to SFTP itself as an
optional feature is, I guess, up for the workgroup or the SFTP draft editor
to decide. In my view, the first solution type (certificate of file's
existence) would be more apt for a separate draft because it is fairly
orthogonal to existing functionality. The second solution type

(certificate

Post by denis bider
of the upload act with special flags for opening the file) might be better
documented in SFTP itself because of the flag's definition.

Joseph Galbraith

2004-10-27 16:54:10 UTC

Permalink

This seems like a useful extension; I've got half
of a proposal worked up... I just haven't quite
managed to finish it yet.

Hopefully I can post something to be shot down
in a day or so.

- Joseph
(sftp draft editor)

Post by jason bailey
You've made some very good points.
Initially I had considered a flag for the transfer which would result in
a returned RECEIPT rather then a STATUS.
However, given your observations,and reviewing what I am looking for I
see no problem with it being an extension to the protocol.
I'm not certain that it is, as you put it, orthogonal to the current
functionality. There already appears to be an existing extension to
support a hashing of a file to verify the contents.
In my mind I see this as very similar. We are merely verifying the
contents and that it is on a particular server at a given time.
Jason

Post by denis bider
Technically speaking, an extension to the SFTP protocol could be

implemented

Post by denis bider
in which the server would testify (with a digital signature) that a

file of

Post by denis bider
a certain name, attributes and contents existed on the server at a

certain

Post by denis bider
time.
Whether or not this file was uploaded entirely by a certain user would

be a

Post by denis bider
more complex challenge because SFTP has no such concept as "uploading"

Post by denis bider
"downloading" a whole file. You have random access and you can pretty

much

Post by denis bider
scratch anywhere you want in the remote filesystem.
An extension that would produce a certificate of a file's existence at

Post by denis bider
certain time would be fairly straightforward. Provide an extension

request

Post by denis bider
type for requesting the certificate, and define the contents of a

receipt.

Post by denis bider
If providing a certificate of the file's existence on the server is
insufficient, and you must really provide a receipt which includes
information about the act of uploading, this could be done, too. For
example, a file for which you require an upload receipt must be opened

with

Post by denis bider
a certain flag or set of flags which signal that you're going to do
receipted-uploading. When you open the file, you are allowed to append

Post by denis bider
the file only (like uploading in TEXT mode). When you close it, the

server

Post by denis bider
sends a RECEIPT message rather than STATUS. The format of the RECEIPT
message is what needs to be defined.
Whether or not this is something for a separate Internet-Draft

(documenting

Post by denis bider
the SFTP extension) or something that can be added to SFTP itself as

Post by denis bider
optional feature is, I guess, up for the workgroup or the SFTP draft

editor

Post by denis bider
to decide. In my view, the first solution type (certificate of file's
existence) would be more apt for a separate draft because it is fairly
orthogonal to existing functionality. The second solution type

(certificate

Post by denis bider
of the upload act with special flags for opening the file) might be

better

Post by denis bider
documented in SFTP itself because of the flag's definition.

Damien Miller

2004-10-31 00:53:20 UTC

Permalink

I don't think that this is within the scope of sftp at all. It would
more appropriately be done by a higher-level protocol, e.g. one that
uses sftp for the file transfer and something else (OpenPGP, S/MIME) for
the signed receipts.

-d

Niels Möller

2004-10-28 07:41:56 UTC

Permalink

Post by Joseph Galbraith
- Add 'versions' extension to allow server to express non-contigous
set of supported versions to client, and 'version' extension
from client-to-server for server to pick one.

As I said before, I really don't think this is the right way to go.

I can accept a "fancy version negotiation"-extension only if it is
specified in such a way that implementations that don't care about
disjunct version intervals (e.g. the common case of an implementation
supporting only version x, or supporting only version x and version
x+1) don't need to implement or otherwise care about the extension.

I view the fancy-version-negotioation thing only as a temporary kludge
needed for the unfortunate version 3 -> version 6+ upgrade. It will be
totally unneeded bloat and uglyness once this is over and the protocol
specification stabilizes. (I don't see any value whatsoever in fancy
version negotiation in the longer run. Simply advertising the highest
supported version number provides for adequate backwards compatibility
for everybody else's protocols, and I think it really ought to work
also for sftp).

And I don't think it is appropriate at all to consider things like
round-trip optimization for a temoprary kludge. Just solve the problem
at hand in a simple a way as possible. Both the problem and the
solution will be forgotten a few years from now.

Regards,
/Niels

denis bider

2004-10-28 08:42:41 UTC

Permalink

Niels,

if the version extension scheme we're talking about is what I proposed, it
does seem to solve the problem at hand in the simplest complete way.

If in the future the 3->6 problem indeed becomes forgotten and obsolete,
clients will simply send "23" as their version (or whatever the highest
version exists at that time) and will bomb out if the server responds with
anything lower than "17". Fine.

If a client is concerned about the 3-6 problem though and wants to avoid the
chance that the server might reply to 6 with 4 or 5, I believe my proposal
is simple and complete.

If you don't care to implement it, it won't burden your client
implementation at all - you can continue to send the highest version you
support and that will work just as fine.

On the server side, this method will burden you only if you wish to offer
your highest version to clients which support non-contiguous version ranges.
But do you not wish your server to be well interoperable with such clients?
Is it not legitimate for clients to have a non-contiguous version
requirement?

Note that the solution has almost 0 impact on sessions where the client is
not one that supports non-contiguous version ranges.

I don't know why you label it as "fancy", because fancy is not what it is.
It just solves the problem with the smallest possible negative impact. The
only alternative with even less burden is to ignore the problem exists in
the first place.

I, for one, see myself implementing a client with non-contiguous version
support. And I would appreciate if the new servers allowed my client to do
so.

-----Original Message-----
Sent: Thursday, October 28, 2004 09:42
To: Joseph Galbraith
Subject: Re: SFTP v6?

As I said before, I really don't think this is the right way to go.
I can accept a "fancy version negotiation"-extension only if
it is specified in such a way that implementations that don't
care about disjunct version intervals (e.g. the common case
of an implementation supporting only version x, or supporting
only version x and version
x+1) don't need to implement or otherwise care about the extension.
I view the fancy-version-negotioation thing only as a
temporary kludge needed for the unfortunate version 3 ->
version 6+ upgrade. It will be totally unneeded bloat and
uglyness once this is over and the protocol specification
stabilizes. (I don't see any value whatsoever in fancy
version negotiation in the longer run. Simply advertising the
highest supported version number provides for adequate
backwards compatibility for everybody else's protocols, and I
think it really ought to work also for sftp).
And I don't think it is appropriate at all to consider things
like round-trip optimization for a temoprary kludge. Just
solve the problem at hand in a simple a way as possible. Both
the problem and the solution will be forgotten a few years from now.
Regards,
/Niels

Niels Möller

2004-10-29 09:23:19 UTC

Permalink

Post by denis bider
if the version extension scheme we're talking about is what I proposed, it
does seem to solve the problem at hand in the simplest complete way.

I think it is the need for completeness that I want to challenge. The
sftp version negotiation mechanism was decided a long time ago, and it
appears to be adequate for most other protocols. I believe it should
work well to live with that decision and its limitations, and *not*
try to retrofit a very different version negotiation scheme into the
protocol.

To make it a little more concrete, I'll make a counterproposal, and
explain the assumptions that are behind it, to see if the rest of you
will shoot it down. First, assumptions:

* sftp version 3 is widely deployed now.

* sftp version 6 will be (we hope) widely deployed soon.

* current deployment of sftp version 4 and 5 is quite marginal. (If
you want to correct me on this, please differentiate between
server and client deployment, it makes a difference).

The simple version negotiation scheme is

V_S = max { version numbers supported by the server }
V_C = max { version numbers supported by the client }
V_U = min (V_S, V_C) # version to use

If V_U is supported by both parties, negotiation is successful, and
V_U is the version that is used on the wire. Otherwise, negotiation
fails.

This scheme is used in lots of protocols, including ssh and sftp (at
least up to now).

If both server and client supports a continuous set of version
numbers, this negotiation scheme is perfect: if there is any version
that both parties supports, V_U = max { version numbers supported by
both parties }.

The limitation of this scheme, is that if client or server supports a
discontinuous set of version numbers, negotiation can fail even if
there is some version that both sides support.

Our problem case is: A supports versions { 3, 4 }, B supports { 3, 6
}. Then V_U = 4, which isn't supported by B, and hence negotiation
fails. Hence, A are B not interoperatable, while a different
negotiation scheme could come up with V_U = 3 and succeed.

My first comment is that this can happen only if the client or server
actually advertises 4 (or 5) as V_C/V_S. How much deployed code out
there actually do that?

My counterproposal is to keep the old version negotiation, and add the
following note on version negotiation:

The sftp protocol uses the traditional version negotiation mechanism
where each side advertise the highest version number it supports,
and the smallest of these two numbers are selected for use.
Communication is possible if this selected version is supported by
both sides.

This mechanism is widely used in Internet protocols, but it also has
some limitations. For example, if one side supports versions 3 and
4, and the other supports versions 3 and 6, the output of the
version negotiation is that version 4 should be used. Communication
fails, even though there is one protocol version, namely 3, that is
supported by both sides. This is expected to be a rare problem in
practice, since at any given time, deployment is usually dominated
by at most two different versions.

In order to make communication possible also in situations like the
above example, where the version negotiation fails, implementations
that support more than one version of the protocol SHOULD provide
some mechanism to manually or automatically configure which of its
supported versions it advertises. In the above example, if either
side is configured to advertise version 3 as the highest supported
version, i.e. to pretend that the support for the higher version
protocol is non-existant, then the output from the version
negotiation is version 3, which both sides support.

This proposal sacrifices trivial out-of-the-box interoperability
between implementations that support version 3 and 6, and
implementations supporting version 3 and 4 (interoperability requires
either manual configuration, or implementation kludges like sometimes
reconnecting and trying a different version). I'm willing to do that
sacrifice in order to keep the protocol nice and clean, and because
the 3,4-vs-3,6-interoperability problem is rare and temporary.

Am I alone in that?

Post by denis bider
If you don't care to implement it, it won't burden your client
implementation at all - you can continue to send the highest version you
support and that will work just as fine.

Is that really correct? If I connect to a 3,6 server, that server will
advertise version 3 (it *has* to, if it wants to interoperate with old
3,4 clients, and that is the entire point of the extended version
negotion, right?). If I'm a version 6 only client, and I don't
implement the proposal, my interpretation of this will be that the
server is an old version 3 server, and that I can't communicate with
it, so I will just display an error message and disconnect.

Post by denis bider
On the server side, this method will burden you only if you wish to offer
your highest version to clients which support non-contiguous version ranges.

As I understand it, a server or client that supports only version 6
still has to implement the proposal (i.e., advertise version 3, which
it does *not* support at all, in the initial version exchange, and
negotiate version 6 in the new second phase), or else it will not
interoperate at all with clients that support version 3 and 6. That's
why I find it so ugly.

Post by denis bider
I don't know why you label it as "fancy", because fancy is not what it is.

It's "fancy", i.e. different and more complicated, compared to the
good old way of doing version negotiation. I didn't mean any more than
that. Sorry for any confusion.

Best regards,
/Niels

denis bider

2004-10-29 11:28:37 UTC

Permalink

Niels,

I will reply with the technical aspects first.

Post by Niels MÃ¶ller

Post by denis bider
If you don't care to implement it, it won't burden your client
implementation at all - you can continue to send the
highest version you support and that will work just as fine.

If your client sends version 6 and the server supports version 6 it can
respond with version 6.

If the client supports a contiguous range of versions, we can make it OK for
the client to simply advertise the highest version it supports. It would
work fine.

Post by Niels MÃ¶ller
As I understand it, a server or client that supports only
version 6 still has to implement the proposal (i.e.,
advertise version 3, which it does *not* support at all, in
the initial version exchange, and negotiate version 6 in the
new second phase), or else it will not interoperate at all
with clients that support version 3 and 6. That's why I find
it so ugly.

This can be fixed. If the client supports a contiguous range of versions, it
should send the highest version it supports in INIT. Otherwise it should set
the highest of the first contiguous set of versions it supports, e.g. if it
supports 3,4,6, it should send 4.

The server should respond with min{advertisedClientVersion,
highestServerVersion} in VERSION. However if advertisedClientVersion is
lower than highestServerVersion, the server should include the
"higher-versions" extension (or whatever it's called) enumerating the higher
versions it supports. In this case the server MUST then accept the
subsequent "SELECTVERSION" packet from the client.

I haven't yet had the chance to read Joseph's wording in the SFTP 6 draft so
I don't know if his wording supports this. If it doesn't we can fix.

Now for the practical aspects of this.

Post by Niels MÃ¶ller
This is expected to be a rare problem in practice, since at any given
time, deployment is usually dominated by at most two different versions.

I believe this to be too optimistic. A situation like 3-6 can reoccur with a
future set of versions. Could be several years from now if the need arises
for new features requiring an incremented version. It won't hurt to have
this interoperability support in place then. Or if the past is any
predictor, something like that could even happen again now. Suppose 6 is a
major thing, then a few people implement 7, and then 8 is the next major
thing.

Note that even in a few years' time we will still likely have SFTP versions
3, 4, 5 around.

Right now our SSH server still supports SFTP version 2 only. The features of
SFTP version 2 are all that's needed in a Windows server, so we didn't yet
upgrade. We will upgrade, but right now we have a fairly considerable base
of deployed installations featuring SFTP version 2. A good portion of our
users won't bother to upgrade to the next version and will continue to
advertise SFTP version 2.

That's for how long a deployed version can be expected to persist. I expect
SFTP version 3 will continue to be around for some time to come.

Post by Niels MÃ¶ller
In order to make communication possible also in situations like the
above example, where the version negotiation fails, implementations
that support more than one version of the protocol SHOULD provide
some mechanism to manually or automatically configure which of its
supported versions it advertises.

Niels - are any of your clients and servers graphical? Textual configuration
is easy. But providing good and well working graphical configuration for
ANYTHING is much, much more difficult than extending the protocol for it to
work automatically. Adding a new option can even take more than a day some
times. It's an important consideration when implementing anything.

In my experience, if the protocol doesn't provide an easy solution to a
common problem, the result will be chaotic differences between
implementations and haphazard interoperability.

Post by Niels MÃ¶ller
This proposal sacrifices trivial out-of-the-box
interoperability between implementations that support version
3 and 6, and implementations supporting version 3 and 4
(interoperability requires either manual configuration, or
implementation kludges like sometimes reconnecting and trying
a different version). I'm willing to do that sacrifice in
order to keep the protocol nice and clean, and because the
3,4-vs-3,6-interoperability problem is rare and temporary.

I'm not willing to do that sacrifice because the burden of implementing this
non-contiguous version negotiation is in my view much smaller than
application-specific remedies that will need to be implemented otherwise,
and the user frustration and loss of interoperability that will result.

Besides - a client that supports a contiguous set of versions does not need
to be impacted in any way. Just the server needs to support this extension
to be interoperable at a higher version with clients implementing
non-contiguous versions.

Post by Niels MÃ¶ller
It's "fancy", i.e. different and more complicated, compared
to the good old way of doing version negotiation. I didn't
mean any more than that. Sorry for any confusion.

Look at it this way. If we both spent the time we are spending to discuss
this implementing this extension, we both would have had a server and client
implementation already. :-)

Best regards,

denis

Niels Möller

2004-10-30 17:00:58 UTC

Permalink

Post by denis bider
I will reply with the technical aspects first.

Thanks.

Post by denis bider

Post by Niels MÃ¶ller

Post by denis bider
If you don't care to implement it, it won't burden your client
implementation at all - you can continue to send the
highest version you support and that will work just as fine.

If your client sends version 6 and the server supports version 6 it can
respond with version 6.

I'm afraid that makes absolutely no sense to me. It is possible that I
have misunderstood the vesrion extension in some fundamental way, so I
should probably shut up until I've reread the proposal (which may take
a long time, since I'm pretty busy with work for the rest of this
year).

Anyway, I'll summarize my current understanding of the proposal.
Given:

Server S, that supports version 3 and version 6 of the protocol.

Client O (old), supporting version 3 and version 4.

Client N (new), supporting only version 6.

S wants to interoperate with O. Using the traditional version
negotiation will fail (server advertises version 6, O advertises
version 4, which leads to version 4 being selected, which S doesn't
support). Hence S *must* implement the proposed extension. It always
advertises version 3, and intends to use the proposed extension to
negotiate version 6 in the second phase.

Now N connects to S. If N uses the traditional version exchange only,
communication will fail (N advertises 6, S advertises 3, which leads
to version 3 being selected, which N doesn't support. So if N wants to
interoperate with S, it too *must* implement the extension.

Post by denis bider
Right now our SSH server still supports SFTP version 2 only. The features of
SFTP version 2 are all that's needed in a Windows server, so we didn't yet
upgrade.

When you upgrade, you will have a server that supports version 2 and
6, or version 2,3 and 6, right? I don't see why that should pose a
problem.

The problem is only implementations that *advertice version 4 and 5*.
Any body who knows about deployment of such versions, please speak up.
I'd like to see some evidence for the existence of the problem the
extension is trying to solve.

Post by denis bider
Niels - are any of your clients and servers graphical? Textual configuration
is easy. But providing good and well working graphical configuration for
ANYTHING is much, much more difficult than extending the protocol for it to
work automatically. Adding a new option can even take more than a day some
times. It's an important consideration when implementing anything.

I don't usually to GUI programming. From my limited experience, if you
have some kind of options dialog or menu, adding one more checkbox to
that dialog is about the same amount of work as adding one more
command line option to a command line tool.

Regards,
/Niels

Joseph Galbraith

2004-10-31 17:02:50 UTC

Permalink

Post by Niels MÃ¶ller
The problem is only implementations that *advertice version 4 and 5*.
Any body who knows about deployment of such versions, please speak up.
I'd like to see some evidence for the existence of the problem the
extension is trying to solve.

I have a version 4 deployment. I'm pretty sure Richard Walen
(process software) has a version 4 deployment.

And Henrick Hellström says that he and others he knows of
have a v5 deployments.

So yes, the problem does exist in the wild and we aren't just
talking about the hypothetical problem. There are known,
currently shipping implementations of every version of
sftp from 2-5.

I'm mildly surprised about the 2, by the
way. I would have thought that no one was currently
shipping version 2, though I suspected there was still a
number of v2 implementations in use.

That is one of the reasons I claim this problem isn't
quite as 'temporary' as you and I would wish.

Thanks,

- Joseph

denis bider

2004-11-01 04:04:55 UTC

Permalink

Firstly, with regard to the "file-share" method of Joseph's proposal - if in
the end that is the solution picked, I will agree.

Post by Niels MÃ¶ller
Server S, that supports version 3 and version 6 of the protocol.
Client O (old), supporting version 3 and version 4.
Client N (new), supporting only version 6.
S wants to interoperate with O. Using the traditional version
negotiation will fail (server advertises version 6, O
advertises version 4, which leads to version 4 being
selected, which S doesn't support). Hence S *must* implement
the proposed extension. It always advertises version 3, and
intends to use the proposed extension to negotiate version 6
in the second phase.
Now N connects to S. If N uses the traditional version
exchange only, communication will fail (N advertises 6, S
advertises 3, which leads to version 3 being selected, which
N doesn't support. So if N wants to interoperate with S, it
too *must* implement the extension.

Reading this it seems to me like you're not at all taking into account that
S doesn't "advertise", it responds. Unlike the SSH version string which is
sent by both parties upfront, the server's SFTP version packet is a response
sent to the client. If S prefers version 6 and N advertises 6, it would be
senseless for S to send 3. S can send 6 in response to N's 6, and still be
interoperable at highest version with client X that supports non-contiguous
versions 3 and 6 and implements extended version exchange (i.e. sending 3
and then negotiating up to version 6).

Post by Niels MÃ¶ller
I don't usually to GUI programming. From my limited
experience, if you have some kind of options dialog or menu,
adding one more checkbox to that dialog is about the same
amount of work as adding one more command line option to a
command line tool.

That depends on what your aesthetic criteria are. Certainly if you don't
care about how the program looks it can be as simple as you described. Not
so if you're already running out of space and need to rearrange things just
to make room for the new configuration option and then arrange it in a way
that is aesthetically attractive as well as clear and non-confusing to the
operator.

Then comes the work of documenting the option; implementing a user friendly
interface guiding the user towards that option when required so that they
will be intuitively led to use it; updating the configuration schema to work
correctly with the new option; interlinking all the various configuration
and program modules that now need to use that option.

And still, after all that work has been performed, the user will be
confused, will not understand why the connection is not working and will
post messages on the forum requiring help from technical support.

In the end, if there is no version negotiation scheme allowing support for
non-contiguous versions, it becomes a better solution to implement all of
the version range, rather than having to deal with the consequences of
supporting only 2 selected versions but no solid version negotiation scheme
in place.

Joseph Galbraith

2004-10-29 20:06:10 UTC

Permalink

Post by Niels MÃ¶ller

Post by denis bider
if the version extension scheme we're talking about is what I proposed, it
does seem to solve the problem at hand in the simplest complete way.

This is almost the way it works, but not quite.

V_C = max { version numbers supported by the client }
V_U = highest version supported by server <= V_C

(V_S doesn't actually exist on the wire.)

A minor difference... but...

[...]

Post by Niels MÃ¶ller

Post by denis bider
If you don't care to implement it, it won't burden your client
implementation at all - you can continue to send the highest version you
support and that will work just as fine.

No, this is not correct.

There is actually no problem with the existing version scheme for
_servers_ supporting discontigous protocol versions. This is
because the _client_ MUST send it's version first. And the
server doesn't actually send it's version, it sends the version
to use.

So, the server simply waits for the clients version, and picks
the highest protocol version that it supports that is less than
or equal to the clients version and sends that as V_U.

Post by Niels MÃ¶ller

Post by denis bider
On the server side, this method will burden you only if you wish to offer
your highest version to clients which support non-contiguous version ranges.

See above for client supporting only version 6.

A client that wishes to support discontigous ranges
will behave as follows:

1. It must send version 3 or higher during initial
exchange, because if it doesn't the server can't
send back an extension packet describing it's
extended version options.

This implies that in order to support server's
that don't implement the new version scheme, which
include v3,4, and 5 servers (of which I know of at
least on of each), it MUST actually support the version
3 protocol variant. YUCK.

2. It will look for the 'version' extension in the servers
SSH_FXP_VERSION packet. If the server doesn't send the
packet, then the server doesn't support the protocol.

This implies that a server that doesn't support the new
protocol will interoperate with this client as if it only
supported the clients lowest contiguous range of protocols.

3. If the 'version' packet is available, and contains a version
higher than that which is currently negotiated, the client
sends a 'version-select' (I like that name much better than
what I used in the draft.)

So, yes, if you implement a version 6 client, you'll need to be
able to support enough version 3 to get through the extension.
And I agree, that is gross.

So I'll make an alternate proposal:

SFTP (which stands for Secure File Transfer Protocol) is not
a file transfer protocol. If it were a file transfer protocol,
it would have operations like get and put. Instead it has
operations like open, read, write, and close.

Therefore, I suggest that we change the name of the subsystem
to 'file-share'

The 'file-share' protocol supports all versions of the 'sftp'
protocol, with the minor exception that version exchange
consists of:

Server and client both send (without waiting to receive):

FXP_INIT2
string version-list
<extension-data>

version-list is comma separated string of versions,
in order of preference. For example. "6,2,1"

The version to use is the first version on the clients
list that is also on the server list.

Now:

- People with existing implementations only need a little
bit of parameterization based on subsystem name to use
the same piece of code to support 'sftp' and 'file-share'.

Unix out-of-process implementations, might, for example,
simply hard-link the sftp server binary to sftpd and
fileshared. Or they might pass the subsystem name on
the command line.

In proc implementations have similar options available.

In fact, implementors can (and SHOULD I think) make the
switch to 'file-share' independent of implementing
the latest protocol version.

- Older clients continue to work without change (they
talk to the older 'sftp' subsystem.)

- Newer clients will have to attempt to start the
'file-share' subsystem, and if that fails, start
the 'sftp' subsystem.

This could cost one additional round trip when talking
to a downlevel server, but, when talking to newer
servers, it actually saves half a round trip because
the server no longer waits for the clients version
before sending it's own, and in addition, appropriate
client extensions can be batched up and sent during
init (something that couldn't be done before because of
compatibility problems.) Which could save even more.

- The new version negotiation scheme is:

a. familiar to all ssh implementors.
b. flexible enough to handle any future needs I can think
of.
c. more graceful than the old, especially in the way it fails.
In particular, the client can now tell the user what versions
the server _does_ support (which is really useful to me
as an implementor or sys admin try to help a befuddled
user.)

I actually think the name change is a cleaner solution.
Niels does raise a good point about a version 6 only
server having to lie and support version 3 in order to
work with discontigous clients. This does seem very
ugly to me.

However, I do think we need to do one or the other
of these proposals. I don't think simple language
change is sufficient with implementation advice is
sufficient.

The problem is indeed, temporary, for some definition
of temporary. But, I suspect we will be seeing version
3 in use for another 5 years, at least. That definitely
puts it on my mid to long range radar.

My pessimistic evaluation is that it will be more like
5-7 years after we hit RFC status before we don't have
customers that want to interoperate with a version
client or server.

I also suspect if we give people with existing
implementations a way to get to the latest version with
less pain the problem will be more temporary (i.e., newer
versions will be out there sooner for people to start
upgrading to, and the sooner they start the more temporary
our problem.)

- Joseph

denis bider

2004-10-29 21:15:25 UTC

Permalink

Post by Joseph Galbraith
A client that wishes to support discontigous ranges
1. It must send version 3 or higher during initial
exchange, because if it doesn't the server can't
send back an extension packet describing it's
extended version options.
This implies that in order to support server's
that don't implement the new version scheme, which
include v3,4, and 5 servers (of which I know of at
least on of each), it MUST actually support the version
3 protocol variant. YUCK.

I proposed in my previous email to the list the following simple change
which solves this problem:

If the client supports a contiguous range of versions, it should send the
highest version it supports in INIT. Otherwise it should send the highest of
the first contiguous set of versions it supports, e.g. if it supports 3,4,6,
it should send 4.

This means that if the client supports 4,6 it can send 4 and does not need
to implement 3. No yuck here. :)
The SFTP module can be implemented separately from the SSH server, as we do.
This proposal requires the SSH server to know about complexities of the SFTP
protocol and to invoke the SFTP module differently depending on what verb
was used to start it.

This also violates independence of SFTP from the SSH protocol. Each
underlying protocol must now think up its own rules for starting old versus
new versions of SFTP.

I think that's much uglier.

Doesn't my proposal above solve all of the gripes about people having to
advertise versions they don't actually support? It's a straightforward
change and if that's the only real problem thus far identified we have a
good solution I think (if Niels agrees).

denis

Niels Möller

2004-11-01 09:32:59 UTC

Permalink

Post by Joseph Galbraith
This is almost the way it works, but not quite.
V_C = max { version numbers supported by the client }
V_U = highest version supported by server <= V_C
(V_S doesn't actually exist on the wire.)

Thanks for the correction. Let's see if I understand what this implies
for interoperability. The failure, for traditional version exchange,
seems to be

* Client supporting version 3 and 6.
* Server supporting version 3 and 4.

=> V_U = 4 => communication failure.

Is this correct?
[...]

Post by Joseph Galbraith
Therefore, I suggest that we change the name of the subsystem
to 'file-share'
The 'file-share' protocol supports all versions of the 'sftp'
protocol, with the minor exception that version exchange
FXP_INIT2
string version-list
<extension-data>
version-list is comma separated string of versions,
in order of preference. For example. "6,2,1"

I agree this is a clean solution, in line with how the rest of the ssh
protocol works. However, let's compare how it will work in practice
during the transition period (next few years), with particular focus
on our failure case: a client supporting versions 3 and 6, and an old
server supporting versions 3 and 4.

1. Client asks SSH server for subsystem "file-share".

2. SSH server replies failure.

3. Client asks for subsystem "sftp".

4. SSH server replies success, and starts the subsystem.

5. Client sends V_C = 3 (servers supporting version 6 are supposed
to support "file-share", so there's no point of asking for
version 6 at this point).

6. The sftp server selects V_U = 3.

7. Handshaking is finished and sftp version 3 is used for the rest
of the session.

Do I understand your proposal correctly? Now compare this to the
"kludge" that can be used without any changes to the current protocol:

1. Client asks SSH server for subsystem "sftp"

2. SSH server replies success, and starts the subsystem.

3. Client sends V_C = 6.

4. Server replies V_U = 4.

5. Client terminates session, since it doesn't support version 4.

6. Client asks SSH server for subsystem "sftp"

7. SSH server replies success, and starts the subsystem.

8. Client sends V_C = 3.

9. Server replies V_U = 3.

10. Handshaking is finished and sftp version 3 is used for the rest
of the session.

In this case (which I think is the most central problematic case which
we're trying to address) I don't see a big win in introducing
"file-share". It makes things only slightly easier for the client, and
slightly more complex for the server.

And when having this choice, I usually prefer putting the complexity
in the client. The reason is that complexity always increases the risk
for bugs, and bugs usually have a much worse security impact in the
server than in the client.

Best regards,
/Niels

Joseph Galbraith

2004-11-01 16:38:56 UTC

Permalink

Post by Niels MÃ¶ller

Thanks for the correction. Let's see if I understand what this implies
for interoperability. The failure, for traditional version exchange,
seems to be
* Client supporting version 3 and 6.
* Server supporting version 3 and 4.
=> V_U = 4 => communication failure.
Is this correct?

Yes, I believe this is the case.

[...]

Post by Niels MÃ¶ller
I agree this is a clean solution, in line with how the rest of the ssh
protocol works. However, let's compare how it will work in practice
during the transition period (next few years), with particular focus
on our failure case: a client supporting versions 3 and 6, and an old
server supporting versions 3 and 4.

[...description omitted for brevitty...]

Post by Niels MÃ¶ller
Do I understand your proposal correctly?

Yes, I think you do.

Post by Niels MÃ¶ller
Now compare this to the

[...description omitted for brevitty...]

I suppose that the kludge isn't that bad.

Post by Niels MÃ¶ller
In this case (which I think is the most central problematic case which
we're trying to address) I don't see a big win in introducing
"file-share". It makes things only slightly easier for the client, and
slightly more complex for the server.
And when having this choice, I usually prefer putting the complexity
in the client. The reason is that complexity always increases the risk
for bugs, and bugs usually have a much worse security impact in the
server than in the client.

In general I agree, I'd rather have complexity in the client than the
server, however, in this case, I think the additional complexity in
the server is pretty small for 'file-share' and the additional
complexity in the client for 'kludge-mode' is significantly higher
degree than increased complexity for 'file-share'.

In fact, 'kludge-mode' requires the entire channel to go down and
be reopened. 'file-share' experiences a failure to start a subsystem,
and so requests a different one, but shouldn't require the channel
to go down.

Put another way, I have a lower layer (connection) that starts an
upper layer protocol (sftp). In kludge mode, that upper layer now
has to:

a. pass back private data to the lower, connection, layer (the server
sftp version) and a request the connection layer to restart
the sftp layer with the version as a parameter.

--or--

b. Emulate the lower layer itself to start a new channel.

With file-share, this complexity occurs along a more natural border
(at least in my software), because the channel doesn't have to go
down.

In addition, their are several reasons I feel strongly about the
'file-share' proposal.

1. I believe it is the better version negotiation protocol.

2. It works the way you thought it did :-), with client and
server both announcing their versions to the other.

As you noted, this is the way many protocols work,
and I think it is more intuitive.

3. It is familiar to SSH implementors in it's specific
algorithm (choose first item on client list also on
servers.)

4. When a version negotiation failure occurs, both sides
know why, because they both have the same information.

This leads to better error messages and easier trouble
shooting for sys admins and tech support.

In fact, with the current draft, if the server doesn't
support the clients version at all (v2-4 client, v6
only server), the server has to close the connection,
there is no way to communicate to the client why it
did so.

Until we had this discussion, I would have assumed, as
a tech support, admin, or implementor, that the server
had a bug and crashed, not that there was a version
problem.

5. It allows extension of the startup protocol on both
sides, not just the server.

6. For a brand new latest-version-only implementation,
it reduces complexity. There is one less packet type, one
less error condition on each side (the server sent me a client
init packet, or visa versa.)

So, it reduces burden on new implementations while only adding
a little complexity to existing implemenations.

This is important because one presumes there will be more
implementations after we get to RFC, that don't want / need
to be burden with all of our intermediate baggage.

7. It is faster (saves half a round trip.)

Even in compatibility mode, when the client has to fall back
to 'sftp', it is only one extra round trip, as compared
to kludge mode which requires a channel open, subsystem
request, sftp-init, and sftp-version, and channel close
(5 round trips) before it can fall back to the version
that will inter-operate.

[Note this is least significant to me, but still a property
of the proposal.]

In other words, outside of the fact that it solves the
version problem we've been discussing, I think it is
just plain a better way to go.

Thanks,

- Joseph

Niels Möller

2004-11-01 21:37:58 UTC

Permalink

Post by Joseph Galbraith
In fact, 'kludge-mode' requires the entire channel to go down and
be reopened. 'file-share' experiences a failure to start a subsystem,
and so requests a different one, but shouldn't require the channel
to go down.

At least conceptually, ssh channels are farily lightweight. Ok, a
typical implementation with sftp as a separate process will spawn one
process for each channel, which wastes some resources, but I'm not
sure optimal resource usage is important for the backwards compatibility
case.

Post by Joseph Galbraith
Put another way, I have a lower layer (connection) that starts an
upper layer protocol (sftp). In kludge mode, that upper layer now
a. pass back private data to the lower, connection, layer (the server
sftp version) and a request the connection layer to restart
the sftp layer with the version as a parameter.
--or--
b. Emulate the lower layer itself to start a new channel.

I'm not sure if you're thinking about the server or client now, but as
far as I can see, the interface is clean in both cases.

The sftp client asks the lower layer (ssh connection) for a session
channel, and asks it to start the sftp subsystem. And it sometimes
does that twice. But it's identical both times, no "private data"
involved. Then, the request for a session channel may or may not imply
a new ssh connection, but that's an implementation issue. One case
that I expect to be common when efficiency matters, is to keep an ssh
connection in the background, like the lsh gateway mode, or the
corresponding feature in recent openssh. Then *all* sftp sessions
channels to the same host will reuse an existing connection, no matter
which versions they want to use.

Post by Joseph Galbraith
1. I believe it is the better version negotiation protocol.

It's more complete than version negotiation in most protocols. Most
protocols seem to work fine without completeness, though.

I think the reason we seem to be getting into trouble is the rapid
increments to the version number, with corresponding changes that
aren't backwards compatible at all. Development of most other internet
protocols (including ssh!) pays more attention to backwards
compatibility, and bumps the protocol version number very rarely.

Post by Joseph Galbraith
3. It is familiar to SSH implementors in it's specific
algorithm (choose first item on client list also on
servers.)

Just some tangential comments: If we go this way, then the "version
numbers" need not be numbers at all, they can be arbitrary names, with
no ordering attached by the protocol. Then an alternative approach
(which I'm not advocating) is to delete version negotiation from the
sftp protocol completely, just have different subsystem names, sftp1,
sftp3, sftp6, .... The client asks for the subsystem type it wants, in
can make multiple requests in any order of its choice.

Post by Joseph Galbraith
4. When a version negotiation failure occurs, both sides
know why, because they both have the same information.
This leads to better error messages and easier trouble
shooting for sys admins and tech support.

I think this is one of the stronger arguments for your proposal.

I tend to be somewhat negative to all general protocol changes (as I'm
sure you've noticed ;-), but I guess I ought to be less conservative
with sftp than with ssh proper.

Regards,
/Niels

Joseph Galbraith

2004-11-01 22:49:22 UTC

Permalink

Post by Niels MÃ¶ller

I'm not sure if you're thinking about the server or client now, but as
far as I can see, the interface is clean in both cases.

Client...

Post by Niels MÃ¶ller
The sftp client asks the lower layer (ssh connection) for a session
channel, and asks it to start the sftp subsystem. And it sometimes
does that twice. But it's identical both times, no "private data"
involved.

But in order for kludge mode to work, the second time
it starts the sftp subsystem, it has to pass down
the server version it got from the first attempt. (So
that the second time around the client can send the
version that will have the server pick the right version.)

(In our software, it is all one process, but different objects.)

The problem is that kludge mode requires me to either
reset an an object (complex) and give that object a
way to ask the lower layer to restart it's underlying
transport, or to persist some state from the object (the
version from the sftp server) and provide a way to restart.

'file-share', at least for my implementation, works the way
you are describing, because I don't have to get into the
middle of the sftp protocol before finding out it doesn't
work. The lower layer starts one, and if it doesn't work,
it starts the other.

Post by Niels MÃ¶ller
[...]

Post by Joseph Galbraith
1. I believe it is the better version negotiation protocol.

It's more complete than version negotiation in most protocols. Most
protocols seem to work fine without completeness, though.
I think the reason we seem to be getting into trouble is the rapid
increments to the version number, with corresponding changes that
aren't backwards compatible at all. Development of most other internet
protocols (including ssh!) pays more attention to backwards
compatibility, and bumps the protocol version number very rarely.

On the other hand, it seems like there were several
occasions were we punted on a problem or choose a
sub-optimal solution because we had to maintain
backwards compatibility.

So it does swing both ways.

Post by Niels MÃ¶ller

Post by Joseph Galbraith
3. It is familiar to SSH implementors in it's specific
algorithm (choose first item on client list also on
servers.)

Just some tangential comments: If we go this way, then the "version
numbers" need not be numbers at all, they can be arbitrary names, with
no ordering attached by the protocol.

Yes; that had occurred to me. I'm not sure we'd ever take
advantage of it, but it is possible.

Post by Niels MÃ¶ller
Then an alternative approach
(which I'm not advocating) is to delete version negotiation from the
sftp protocol completely, just have different subsystem names, sftp1,
sftp3, sftp6, .... The client asks for the subsystem type it wants, in
can make multiple requests in any order of its choice.

I think this is one of the stronger arguments for your proposal.

Yes, this is a very significant point.

I tried to change the version negotiation back when
I first started editing sftp so that the server sent
back it's version (V_S rather than V_U)... which
at least gives the server something valid to send
when version negation fails, but we had to back
it out because of backwards compatibility.

The other argument that is strong to me is that
the new proposal works better than what we have
for someone implementing only the latest version.

I.e., the protocol we go to RFC with will be simpler,
not more complex, because of it. (Even if clients and
servers doing backwards compatibility do get a little
more complex.)

Post by Niels MÃ¶ller
I tend to be somewhat negative to all general protocol changes (as I'm
sure you've noticed ;-), but I guess I ought to be less conservative
with sftp than with ssh proper.

I tend to be quite negative towards protocol changes in SSH proper...
however, sftp hasn't gone to last call yet (for good reason...) and
so isn't in the same semi-frozen state as the SSH proper.

(I hope to get there soon though :-)

Joseph

Niels Möller

2004-11-02 05:46:35 UTC

Permalink

Post by Joseph Galbraith

I'd tend to include the sftp version negotiation in the sftp "layer".
The lower layer, connection, provides a way for the upper layer (sftp)
to open a channel and request a subsystem. But all data transmission
on the channel, including the sftp version exchange, is the upper
layer's (i.e. sftp's) job. The lower layer doesn't know *anything*
about the protocol used on the channel, not even the version used.

Post by Joseph Galbraith
From this point of view, I see no problem with the interface.

Post by Niels MÃ¶ller

Post by Joseph Galbraith
4. When a version negotiation failure occurs, both sides
know why, because they both have the same information.

I tried to change the version negotiation back when
I first started editing sftp so that the server sent
back it's version (V_S rather than V_U)... which
at least gives the server something valid to send
when version negation fails, but we had to back
it out because of backwards compatibility.

One simple idea: In this failure case, server version 6, client
version 4 (say), is there any good reason the server can't send
version 6 in its SSH_FXP_VERSION, before disconnecting? That would
give the client at least some clue. It could reasonably interprete
this response as "This server doesn't seem to support versions below
6", and tell that to the user.

Regards,
/Niels

Joseph Galbraith

2004-11-02 16:23:33 UTC

Permalink

Post by Niels MÃ¶ller

Post by Joseph Galbraith

Post by Joseph Galbraith
4. When a version negotiation failure occurs, both sides
know why, because they both have the same information.

Yes; unfortunately, it is hard to say waht a version 4 client
would do with that, since it isn't legal in the protocol
v4 protocol... if we'd spec'd it that way for 4 it could
work :-)

We could make the change for v6... and the implementation
would have to decide whether to take the risk that a v4
implemenation would behave better with an invalid version
packet or behave better with a silent channel close.

I think I'd prefer to solve the whole ball of wax at
one go... this is an area where we've been bandaging
the protocol for a while for backwards compatibilities
sake.

I'd like to just bite the bullet and get it over with.

- Joseph

denis bider

2004-11-02 19:24:11 UTC

Permalink

Post by Joseph Galbraith
I think I'd prefer to solve the whole ball of wax at
one go... this is an area where we've been bandaging
the protocol for a while for backwards compatibilities
sake.
I'd like to just bite the bullet and get it over with.

I support that and I'd like to see whatever real solution in place, just not
the kludge. It seems to me like this has already been discussed enough.

Joseph Galbraith

2004-11-01 18:18:14 UTC

Permalink

Post by Niels MÃ¶ller

Thanks for the correction. Let's see if I understand what this implies
for interoperability. The failure, for traditional version exchange,
seems to be
* Client supporting version 3 and 6.
* Server supporting version 3 and 4.
=> V_U = 4 => communication failure.
Is this correct?

Let me call out one additional failure-- I mention
it in my other message but I think it is worth
calling out seperately.

It is more a failure to fail gracefully.

* Client supporting versions < n
* Server supporting versions >= n+1

These two can't interoperate, but the server can't
communicate why.

The client sees a channel close as the failure mode,
which could mean many different things.

Today, in fact, a channel close probably means that
the server crashed.

Or had some other fatal startup bug/failure.

Version negotiation failure is probably a small
percentage of such cases. (Today anyway-- next year
may be different.)

I'd like an unambigous failure when the versions are
incompatible.

Thanks,

- Joseph

der Mouse

2004-10-10 05:41:06 UTC

Permalink

[...filenames...UTF-8...character set...conversions...]

I see a philosophical problem here that I suspect is at the root of
many of these difficulties.

What are file names?

On most Unix variants, file names are opaque octet sequences (with 0x00
octets not permitted and 0x2f octets permitted only as component
separators). (Some very old variants also forbid octets 0x80-0xff.)

Note that they are not character sequences. Interpreting those octets
as characters is something done only for human-interaction purposes and
is done using whatever character set the display device is using.
Thus, even _contemplating_ doing character set conversions anywhere is,
from a Unix perspective, a grave mistake.

However, other systems complicate matters. I have seen it said (with
what truth I know not - I am no Windows geek) that Windows filenames
are sequences of Unicode characters, or more precisely sequences of
Unicode codepoints, from which perspective character set conversions
are perfectly reasonable.

I see no good way to cater to both philosophies other than either to
have a bit indicating whether a given filename is an octet sequence or
(an encoding of) a character sequence, or perhaps to have a character
set indicator but with a special value reserved to indicate "raw octet
sequence".

Thoughts? Agree? Disagree?

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

r***@roumenpetrov.info

2004-10-11 08:33:02 UTC

Permalink

Post by Joseph Galbraith
Unix directories can contain files encoded in multiple
different char-sets and the server has no way to tell
what these multiple char-sets are and translate them
to UTF-8. Because the transformation is one way, once
the server has mistranslated the filename, there
is no way for the client to get back to the original
data.
So, for these file-systems, the best possible thing
to do is send the filename raw and let the client
(with help from the user decode it.)
On the other hand, maximum possible interoperability
between different language and regions is obtained
through use of UTF-8 where available.

User can use cat/type/ls/dir <FILENAME> to access a local file and it
should be able to do same with SFTP.
As example "echo 'get <FILENAME>' | sftp localhost" should get same file.
In all cases <FILENAME> should be same and is encoded in user
charmap(charset/codeset/etc.).

Post by Joseph Galbraith
I haven't been able to come up with a solution I
really like.
1. Let the server say what it is going to use,
UTF-8 or 'undefined-raw' at the beginning
of the sftp session.
[SNIP]

I'm not sure that server is responsible to do decision for encoding:
'UTF-8' or 'RAW'.
Since in a SFTP session client can request more that one file negotiation
of file name encoding should be at begining of session.
Client should request encoding from server.

I guest that a new extension "encoding" will solve problem:
1.) Client send to server list of accepted encodings and server return
prefered one or "RAW" or "UTF-8".

To do this sftp implementation MUST implement extension "encoding".
Extension should be defined in draft as "newline" is defined.

I not sure that sftp can use names like "ascii", "usascii", "C", "POSIX",
"ANSI_X3.4....", since ascii define only 7 bit charset. When SFTP server
support 7-bit encoding is should(must?) reject file names containing
symbols with code greater that 127.

When encoding is not set server should treat filenames in "raw" or "utf-8"
format.
This must be annonced in "Server Initialization".
Empty "encoding" is alias to "RAW".
When encoding is set server must convert "local filename in encoding" <->
"wire filename in utf-8".
Client may convert "wire filename in utf-8" <-> "local filename in
encoding". Note that client know name conversion on the server.

I guess that this solution is interoperable with SFTP clients version 1, 2
or 3.
For version 4(four) clients, when server support encodung it should
announce 3(three) as maximum version.
For client version 1,2,3 server must use "RAW".
Server version N(N>=5) must support "UTF-8", "RAW" and "ISO8859-1" encodings.
Server version N(N>=5) may support "ISO8859-N" encodings, where N is in
range 2-15.
Client version N(N>=5) must support the "UTF-8" and "RAW" "encoding"
extension.

P.S.:
As esample cyrillic use many encodings. Most popular are IS08859-5,
KOI8-R, CP1251.
In case of cyrillic one utf-8 file name in cyrillic can address different
files on file system and this depend of encoding.
In this case SFTP client with help from the user is responsible to select
correct encoding.
This is same case as access to local file system.

I don't have problem to adress correct file name on remote host.
My system is properly setup and I can use UTF-8, IS08859-5, KOI8-R and
CP1251 in file/directory names and in GUI termininals.
For the test in directory $HOME/tmp/cyr I have four files with name in
format f.<ENCODING>.<NAME>,
where <ENCODING> is one of mentioned above and
<NAME> is first three leter from cyrillic alphabet in uppercase followed
by same leters in lovercase in same encoding.
Content of each file is "data.<ENCODING>:<NAME>" where <ENCODING> and
<NAME> match the file name.
In four xterm for every encoding I run same command sequence.
Results are attached images in "ssh_session.UTF-8.png",
"ssh_session.ISO8859-5.png", "ssh_session.KOI8-R.png"
and "ssh_session.CP1251.png".
With command "echo get tmp/cyr/f.*.<NAME> | sftp localhost", where sftp is
openssh SFTP version 3 client, file that I get depend of my locale
charmap.

Regards,
Roumen

Joseph Galbraith

2004-10-29 22:05:56 UTC

Permalink

I proposed in my previous email to the list the following simple change
If the client supports a contiguous range of versions, it should send the
highest version it supports in INIT. Otherwise it should send the highest of
the first contiguous set of versions it supports, e.g. if it supports 3,4,6,
it should send 4.
This means that if the client supports 4,6 it can send 4 and does not need
to implement 3. No yuck here. :)

But, if the server only support 6, it must still respond
respond with 4, even though it does not support 4.

Then, if the client doesn't send a 'version-select' the
server must close the channel, because it can't speak
the protocol.

Rather than getting a nice error message about the server
not supporting the version, the user will likely see
strange behavior, because the version 4 client won't be
expecting a channel after version exchange is complete
to mean 'version unsupported.' It thought the version
stuff was already sorted out.

Ugh.

On the other side of the coin, let's say we have a 4,6
client, which advertises 4 in it's INIT packet. Now,
the server is a 3,6 server, so it responds 3. The
client now needs to negotiate down to 3.

That's not so bad, I guess. (I don't think I allowed
it in the original language, but that could be fixed :-)

The third problematic case is a a client that supports
1,2,6. Such a client can not send 2, because 2 doesn't
support extensions. So it must send 3.

Now, what if that client actually meets a version 3 server?
(A highly likely event.) It is now stuck. It can't jump
to six, nor can it fall back to 2 where it could actually
interoperate. So a client really must support, at a minimum,
version 3.

Ugh.

The SFTP module can be implemented separately from the SSH server, as we do.
This proposal requires the SSH server to know about complexities of the SFTP
protocol and to invoke the SFTP module differently depending on what verb
was used to start it.

Not really; it just requires the SSH server to pass the name
of the subsystem being started to the subsystem.

This could be as simple as invoking the sftp-server module
by a different name, or having the config file setup in a
way such that the 'file-share' subsystem invokes
'sftp-server -file-share' and the 'sftp' subsystem invokes
'sftp-server' ... or it might be more complicated
in some implementations.

But it shouldn't require your SSH server to know anything about
the SFTP protocol... it just has to support a new subsystem, and
the subsystem has to know it's name (or you have to have separate
modules.)

In all honesty, I think that the new version exchange is how
it should have been done originally. And I think that it solves
a number of outstanding problems in the protocol that we've
been reluctant to address because of backwards compatibility
issues, including:

- the version problem we've been discussing
- extensions from the client on startup (these have
been very useful on the server side... originally they
were there for the client to, but had to be removed for
backwards compatibility.)
- Better failure mode when there isn't a compatible version.
(In fact the draft is currently silent on how to deal with
this and there isn't a great way to do so with the current
scheme. With the new scheme, both sides know that the version
exchange has failed, and which versions the other supports.)

We've actually tried to make some changes for better failure
mode before too.

This also violates independence of SFTP from the SSH protocol. Each
underlying protocol must now think up its own rules for starting old versus
new versions of SFTP.

I'm not sure I follow this argument.

Since any hypothetical non-SSH implementation doesn't actually
have subsystems, it already had to think up its own rules for
starting the protocol.

I don't know of any actual non-SSH implementations.

Any new non-SSH implementation should just always use the
new version exchange, or in other words, the 'file-share'
version semantics.

It should also probably just use the latest protocol version
and not have any of this mess, since it is new and doesn't
have to worry about interoperating with old.

I think that's much uglier.
Doesn't my proposal above solve all of the gripes about people having to
advertise versions they don't actually support? It's a straightforward
change and if that's the only real problem thus far identified we have a
good solution I think (if Niels agrees).

Unfortunately, no, it really doesn't. At least I don't see
how yet.

I was initially in favor of the version re-negotiate scheme
(it was basically the solution I had in the back of mind
if I ever got a chance to solve this problem.)

But as I've thought about the possible interoperability
interactions, I've become convinced that it would actually
be pretty fragile, and could lead to non-obvious failures
(like the one I described at the top.)

Thanks,

Joseph