string.h memory functionsclarification, for n=0 (memcpy(0, 0, 0)).

[d***@gmail.com]

Arguments (array_ptr=NULL, size=0) for library char array functions (memcpy, memcpy, memmove, ...) are technically forbidden (see reference to standard in the end of message)
Seems to be missed corner case, not intentional.
GCC 4.9 starts to exploit this for optimization ( https://gcc.gnu.org/gcc-4.9/porting_to.html ), so this technical detail about undefined behaviour with {array_ptr=NULL, size=0} has become not theoretical issue, but very important in practise.

Practically, it has become anti-optimization, due redundant checks {if(size) memcpy(dst,src,size)}. This 'if(size)' already handled inside memcpy algorithms. And some projects are forced to avoid optimization about deduced null/non-nulll pointers for whole project, not only memcpy ( http://blog.mycre.ws/articles/bind-and-gcc-49/ ), so this optimization gives a contrary effect.

Sure, it is useful for optimization purposes to deduce that dst/src pointers are not NULL from algorithm {for(size_t i=0;i<size;++i)dst[i]=src[i]}, but only when size!=0. When size is 0, this conclusion is not true.
It's surprising, that algorithm written as C trivial implementation should have subtle difference from library function, and used only for additional reasoning about arguments, but not for algorithm itself.
It's surprising, that copy algorithm is tightly bound to additional effects.

Can this be issued as a defect report?

Proposed addition: allow some class of invalid pointers (one-past-the end of array, NULL pointers) to be passed to library array functions, if size of array is zero.

Current implementations of memcpy functions already likely handle this case.
I had checked glibc x86_x64 implementation, ( ./sysdeps/x86_64/memcpy.S - https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/memcpy.S;h=d6cd553a266c56c0dca3f00bf5e1dcf071b57b9f;hb=HEAD )
for memcpy(dst, src, n), rdi=dst, rsi=src, rdx=n
For n smaller than 32 byte this functions checks rdx bitpattern, and if all bits are zero, explicitly exits function ( see L(1d): / andl $0xf0, %edx / jz L(exit))

=======================================
Related excerpts from WG14/N1124 - Committee Draft - May 6, 2005 - ISO/IEC 9899:TC2:

String function conventions
7.21.1/2
Where an argument declared as size_t n specifies the length of the array for a
function, n can have the value zero on a call to that function. Unless explicitly stated
otherwise in the description of a particular function in this subclause, pointer arguments
on such a call shall still have valid values, as described in 7.1.4. On such a call, a
function that locates a character finds no occurrence, a function that compares two
character sequences returns zero, and a function that copies characters copies zero
characters


7.1.4 Use of library functions
7.1.4/1
Each of the following statements applies unless explicitly stated otherwise in the detailed
descriptions that follow: If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the address space of the program,
or a null pointer, or a pointer to non-modifiable storage when the corresponding
parameter is not const-qualified) or a type (after promotion) not expected by a function
with variable number of arguments, the behavior is undefined. If a function argument is
described as being an array .......

=======================================
May be related discussion:
https://groups.google.com/forum/m/#!searchin/comp.std.c/memcpy/comp.std.c/QkQkcvqfYKQ
https://groups.google.com/forum/m/#!searchin/comp.std.c/memcpy/comp.std.c/XLKwhmB8L5s

[Keith Thompson]

Post by d***@gmail.com
Arguments (array_ptr=NULL, size=0) for library char array functions
(memcpy, memcpy, memmove, ...) are technically forbidden (see
reference to standard in the end of message)

They're not *forbidden* as such; the standard just leaves the behavior
undefined.

Post by d***@gmail.com
Seems to be missed corner case, not intentional.

I see no evidence that this was unintentional. I think the authors of
the standard just didn't see any reason to define the behavior of
memcpy(NULL, NULL, 0) (or memcpy(non_null, NULL, 0), or memcpy(NULL,
non_null, 0).

Post by d***@gmail.com
GCC 4.9 starts to exploit this for optimization (
https://gcc.gnu.org/gcc-4.9/porting_to.html ), so this technical
detail about undefined behaviour with {array_ptr=NULL, size=0} has
become not theoretical issue, but very important in practise.

The behavior has always been undefined; programs depending on it have
always been buggy (at least since the 1989 ANSI standard).

Post by d***@gmail.com
Practically, it has become anti-optimization, due redundant checks
{if(size) memcpy(dst,src,size)}. This 'if(size)' already handled
inside memcpy algorithms. And some projects are forced to avoid
optimization about deduced null/non-nulll pointers for whole project,
not only memcpy ( http://blog.mycre.ws/articles/bind-and-gcc-49/ ), so
this optimization gives a contrary effect.
Sure, it is useful for optimization purposes to deduce that dst/src
pointers are not NULL from algorithm {for(size_t
i=0;i<size;++i)dst[i]=src[i]}, but only when size!=0. When size is 0,
this conclusion is not true.
It's surprising, that algorithm written as C trivial implementation
should have subtle difference from library function, and used only for
additional reasoning about arguments, but not for algorithm itself.
It's surprising, that copy algorithm is tightly bound to additional effects.
Can this be issued as a defect report?

Sure -- but I wouldn't expect the committee to agree with you.

Post by d***@gmail.com
Proposed addition: allow some class of invalid pointers (one-past-the
end of array, NULL pointers) to be passed to library array functions,
if size of array is zero.

By "allow", you mean making the behavior defined rather than undefined.
Presumably the defined behavior would be to do nothing. If you're going
to propose a change to the language, it's best to be as precise as
possible.

[...]
N1124 is out of date. N1256 is a draft of the C99 standard, with all
three Technical Corrigenda merged into it -- but the C99 standard is
officially obsolete, superseded by the C11 standard. N1570 is the most
current draft of the C11 standard, published shortly before the official
standard was released. (The C11 standard itself is available, but you
have to pay for it.)

I don't think either of the passages you quote has changed
significantly, but it's best to use the most current reference if your
proposing changes to the standard.

Post by d***@gmail.com
https://groups.google.com/forum/m/#!searchin/comp.std.c/memcpy/comp.std.c/QkQkcvqfYKQ
https://groups.google.com/forum/m/#!searchin/comp.std.c/memcpy/comp.std.c/XLKwhmB8L5s

Hmm. Those links don't work for me. Perhaps Google Groups has messed
up its user interface yet again.

[d***@gmail.com]

===============================================================
Yes, "do nothing" for empty array should be specified exactly for each function.
Seems, it is already specified in the 7.21.1/2 and in the each specific function description.
"... n can have the value zero ... On such a call, a function that locates a character finds no occurrence, a function that compares two character sequences returns zero, and a function that copies characters copies zero characters"



===============================================================
I suggest to extend types of pointers which can be passed in the case n=0.
I propose the following change int 7.21.1/2 :

...
n can have the value zero on a call to that function. Unless explicitly stated otherwise in the description of a particular function in this subclause, pointer arguments on such a call shall still have valid values, as described in 7.1.4
<NEW>, or to be null pointer, or to be pointer to past the end of array</NEW>
...

Affected functions:
memcpy
memcmp
memmove
strncmp
memset
strncpy

strxfrm is curious exception: it already specifies and allows NULL argument.

strncat: This this change do not affect this function. It seems current definition already cause undefined behaviour when n=0 for strncat (empty array does not allow to write mandatory '\0').

===============================================================

What actions should I do to issue defect report, and to make work of committee members easier?

===============================================================
fixed URLs to previous discussions:
https://groups.google.com/forum/#!topic/comp.std.c/XLKwhmB8L5s
https://groups.google.com/forum/#!topic/comp.std.c/QkQkcvqfYKQ

[ivan dobrokotov]

I had tested assumption about NULL-ready string.h functions is supported by at least trivial counterexamples (runtime implementation of string.h functions on volatile input).
tested compilers: gcc 4.8 (with -O0 and -O2), gcc 4.9 (with -O0 and -O2), clang 3.4, Visual C++ on x64 platform, g++ 4.8 on x86-32bit
==========================================================
#include <string.h>
#include <stdio.h>

typedef char *ptr_t;

volatile ptr_t p = NULL;
char something;
volatile ptr_t q = &something;
volatile size_t n = 0;

#define TEST(x) if (x) ; else { printf("error, %s\n", #x); }

void test_ptrs(volatile ptr_t x, volatile ptr_t y) {
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));
}

int main() {
test_ptrs(p, p);
test_ptrs(p, q);
test_ptrs(q, p);
return 0;
}
==========================================================

Post by d***@gmail.com
What actions should I do to issue defect report, and to make work of committee members easier?

[m***@yahoo.co.uk]

Post by ivan dobrokotov
I had tested assumption about NULL-ready string.h functions is
supported by at least trivial counterexamples (runtime implementation
of string.h functions on volatile input).

Given that NULL arguments to the following functions are all undefined
behaviour, I don't see what you think your test shows?

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

[James Kuyper]

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
I had tested assumption about NULL-ready string.h functions is
supported by at least trivial counterexamples (runtime implementation
of string.h functions on volatile input).

Given that NULL arguments to the following functions are all undefined
behaviour, I don't see what you think your test shows?

I think he's trying to show that it would be feasible to define the
behavior as "does nothing", because that's what many existing
implementations already do. Of course, it's hard to prove that they "do
nothing" rather than "do something (possibly a very dangerous something)
but with no obvious effect".

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

[Jakob Bohm]

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
I had tested assumption about NULL-ready string.h functions is
supported by at least trivial counterexamples (runtime implementation
of string.h functions on volatile input).

Given that NULL arguments to the following functions are all undefined
behaviour, I don't see what you think your test shows?

I think he's trying to show that it would be feasible to define the
behavior as "does nothing", because that's what many existing
implementations already do. Of course, it's hard to prove that they "do
nothing" rather than "do something (possibly a very dangerous something)
but with no obvious effect".

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

(Un)fortunately his small set of example compilers all ship with full
source code for their library implementations of those particular
functions, they do not document in much detail their inline (intrinsic)
implementations, which are part of the compiler itself, rather than a
file read by the compiler.

Things become much more interesting once going beyond the world of flat
mode x86/x86_64. In fact, one need look no further than the same CPUs
in 16 bit protected mode to see interesting side effects of loading
pointer arguments in preparation for the trivial implementations of
those particular functions using the obvious compact opcodes.

Other implementations for other CPU families might read a few input
bytes before realizing the count is 0, though one might guess that
intrinsic implementations would recognize a hard coded 0 count, but
not a variable count which happens to be 0.

It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.


Enjoy

Jakob

[James Kuyper]

...

Post by Jakob Bohm

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

(Un)fortunately his small set of example compilers all ship with full
source code for their library implementations of those particular
functions, they do not document in much detail their inline (intrinsic)
implementations, which are part of the compiler itself, rather than a
file read by the compiler.
Things become much more interesting once going beyond the world of flat
mode x86/x86_64. In fact, one need look no further than the same CPUs
in 16 bit protected mode to see interesting side effects of loading
pointer arguments in preparation for the trivial implementations of
those particular functions using the obvious compact opcodes.
Other implementations for other CPU families might read a few input
bytes before realizing the count is 0, though one might guess that
intrinsic implementations would recognize a hard coded 0 count, but
not a variable count which happens to be 0.
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Then you disagree with Martin - it's not obviously true?

[Jakob Bohm]

Post by James Kuyper
...

Post by Jakob Bohm

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

(Un)fortunately his small set of example compilers all ship with full
source code for their library implementations of those particular
functions, they do not document in much detail their inline (intrinsic)
implementations, which are part of the compiler itself, rather than a
file read by the compiler.
Things become much more interesting once going beyond the world of flat
mode x86/x86_64. In fact, one need look no further than the same CPUs
in 16 bit protected mode to see interesting side effects of loading
pointer arguments in preparation for the trivial implementations of
those particular functions using the obvious compact opcodes.
Other implementations for other CPU families might read a few input
bytes before realizing the count is 0, though one might guess that
intrinsic implementations would recognize a hard coded 0 count, but
not a variable count which happens to be 0.
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Then you disagree with Martin - it's not obviously true?

Having seen source code for mem functions in various C runtime
libraries, it is obvious that those are doing a heroic job, often in
assembler, to run as fast as absolutely possible in at least 4 cases:

- Copying/overwriting a small aligned item, such as a structure.
- Copying/overwriting a small misaligned item, such as a protocol
or file format header or a text string.
- Copying/overwriting huge aligned buffers, such as uncompressed
full screen bitmap images or freshly allocated lookup tables.
- Copying/overwriting huge misaligned buffers.
- mutiply the above by two for memset(p, 0, n) versus
memset(p, not 0, n);
- multiply the above by two for memcpy args with same/different
alignment.

Each of these cases are often special cased for various (otherwise
binary compatible) CPUs with different performance and caching
architectures. The authors of these functions have obviously
spent a lot of time agonizing over every CPU clock cycle, every
memory wait state, pipelining anomalies etc. Anything not
guaranteed by the specification is thrown aside to gain that last
500ps speedup, just in case some programs loop over thousands of
smaller memcpy calls, e.g. to assemble/split TCP streams or to
extract/fill subsets of matrix-like bitmap buffers. Sometimes
this code is even written by the CPU designers and provided to
C language implementers.

Enjoy

Jakob

[James Kuyper]

Post by Jakob Bohm

Post by James Kuyper
...

Post by Jakob Bohm

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

(Un)fortunately his small set of example compilers all ship with full
source code for their library implementations of those particular
functions, they do not document in much detail their inline (intrinsic)
implementations, which are part of the compiler itself, rather than a
file read by the compiler.
Things become much more interesting once going beyond the world of flat
mode x86/x86_64. In fact, one need look no further than the same CPUs
in 16 bit protected mode to see interesting side effects of loading
pointer arguments in preparation for the trivial implementations of
those particular functions using the obvious compact opcodes.
Other implementations for other CPU families might read a few input
bytes before realizing the count is 0, though one might guess that
intrinsic implementations would recognize a hard coded 0 count, but
not a variable count which happens to be 0.
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Then you disagree with Martin - it's not obviously true?

Having seen source code for mem functions in various C runtime
libraries, it is obvious that those are doing a heroic job, often in
- Copying/overwriting a small aligned item, such as a structure.
- Copying/overwriting a small misaligned item, such as a protocol
or file format header or a text string.
- Copying/overwriting huge aligned buffers, such as uncompressed
full screen bitmap images or freshly allocated lookup tables.
- Copying/overwriting huge misaligned buffers.
- mutiply the above by two for memset(p, 0, n) versus
memset(p, not 0, n);
- multiply the above by two for memcpy args with same/different
alignment.
Each of these cases are often special cased for various (otherwise
binary compatible) CPUs with different performance and caching
architectures. The authors of these functions have obviously
spent a lot of time agonizing over every CPU clock cycle, every
memory wait state, pipelining anomalies etc. Anything not
guaranteed by the specification is thrown aside to gain that last
500ps speedup, just in case some programs loop over thousands of
smaller memcpy calls, e.g. to assemble/split TCP streams or to
extract/fill subsets of matrix-like bitmap buffers. Sometimes
this code is even written by the CPU designers and provided to
C language implementers.

That's good to know - but it doesn't really answer my question, which
was framed as a "yes" or "no" question. You could also answer "it
doesn't make sense to restrict the answers to 'yes' and 'no' because
[reason]". However, until you've provided one of those three possible
answers, you haven't answered my question. You've no obligation to
answer it, of course, but responding to my question without clearly
answering it is odd.

If I had to guess, I'd say that you were probably indirectly implying
"yes" - but I shouldn't have to guess.

Note: while he only said it "... is obviously true" that "it is possible
to implement the functions so that if the length is zero nothing happens
even if a pointer is NULL", the context was dismissing the importance of
evidence from several different implementation that was consistent with
that possibility. The clear implication of that comment in that context
is that it's not merely possible, but also unexceptional. Are you
disagreeing with that implication?

[Jakob Bohm]

Post by James Kuyper

Post by Jakob Bohm

Post by James Kuyper
...

Post by Jakob Bohm

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

(Un)fortunately his small set of example compilers all ship with full
source code for their library implementations of those particular
functions, they do not document in much detail their inline (intrinsic)
implementations, which are part of the compiler itself, rather than a
file read by the compiler.
Things become much more interesting once going beyond the world of flat
mode x86/x86_64. In fact, one need look no further than the same CPUs
in 16 bit protected mode to see interesting side effects of loading
pointer arguments in preparation for the trivial implementations of
those particular functions using the obvious compact opcodes.
Other implementations for other CPU families might read a few input
bytes before realizing the count is 0, though one might guess that
intrinsic implementations would recognize a hard coded 0 count, but
not a variable count which happens to be 0.
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Then you disagree with Martin - it's not obviously true?

Having seen source code for mem functions in various C runtime
libraries, it is obvious that those are doing a heroic job, often in
- Copying/overwriting a small aligned item, such as a structure.
- Copying/overwriting a small misaligned item, such as a protocol
or file format header or a text string.
- Copying/overwriting huge aligned buffers, such as uncompressed
full screen bitmap images or freshly allocated lookup tables.
- Copying/overwriting huge misaligned buffers.
- mutiply the above by two for memset(p, 0, n) versus
memset(p, not 0, n);
- multiply the above by two for memcpy args with same/different
alignment.
Each of these cases are often special cased for various (otherwise
binary compatible) CPUs with different performance and caching
architectures. The authors of these functions have obviously
spent a lot of time agonizing over every CPU clock cycle, every
memory wait state, pipelining anomalies etc. Anything not
guaranteed by the specification is thrown aside to gain that last
500ps speedup, just in case some programs loop over thousands of
smaller memcpy calls, e.g. to assemble/split TCP streams or to
extract/fill subsets of matrix-like bitmap buffers. Sometimes
this code is even written by the CPU designers and provided to
C language implementers.

That's good to know - but it doesn't really answer my question, which
was framed as a "yes" or "no" question. You could also answer "it
doesn't make sense to restrict the answers to 'yes' and 'no' because
[reason]". However, until you've provided one of those three possible
answers, you haven't answered my question. You've no obligation to
answer it, of course, but responding to my question without clearly
answering it is odd.
If I had to guess, I'd say that you were probably indirectly implying
"yes" - but I shouldn't have to guess.
Note: while he only said it "... is obviously true" that "it is possible
to implement the functions so that if the length is zero nothing happens
even if a pointer is NULL", the context was dismissing the importance of
evidence from several different implementation that was consistent with
that possibility. The clear implication of that comment in that context
is that it's not merely possible, but also unexceptional. Are you
disagreeing with that implication?

It is almost tautologically true that it is *possible* to create
implementations with the desired property.

It is evidently true, that on some platforms, some of the most optimal
implementations happen to have the desired property.

It is thus evidently true, that on *some* platforms, there is zero
cost to implementing the desired property.

It is not at all obvious, and possibly false, that there is zero
cost to implementing the desired property on *all* platforms.


Enjoy

Jakob

[James Kuyper]

On 04/15/2015 10:39 AM, Jakob Bohm wrote:
...

Post by Jakob Bohm
It is almost tautologically true that it is *possible* to create
implementations with the desired property.

Agreed.

Post by Jakob Bohm
It is evidently true, that on some platforms, some of the most optimal
implementations happen to have the desired property.
It is thus evidently true, that on *some* platforms, there is zero
cost to implementing the desired property.
It is not at all obvious, and possibly false, that there is zero
cost to implementing the desired property on *all* platforms.

An example of a specific case where the cost is definitely not zero
would be far more useful than pointing out the vague possibility that
there might be such a case.

[Philip Lantz]

Post by James Kuyper
...

Post by Jakob Bohm

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

(Un)fortunately his small set of example compilers all ship with full
source code for their library implementations of those particular
functions, they do not document in much detail their inline (intrinsic)
implementations, which are part of the compiler itself, rather than a
file read by the compiler.
Things become much more interesting once going beyond the world of flat
mode x86/x86_64. In fact, one need look no further than the same CPUs
in 16 bit protected mode to see interesting side effects of loading
pointer arguments in preparation for the trivial implementations of
those particular functions using the obvious compact opcodes.
Other implementations for other CPU families might read a few input
bytes before realizing the count is 0, though one might guess that
intrinsic implementations would recognize a hard coded 0 count, but
not a variable count which happens to be 0.
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Then you disagree with Martin - it's not obviously true?

Martin said that it's obviously true that "it is possible to implement
the functions" like that. He did not say or imply that all
implementations do that, or that it's the best way, or that it should be
required.

[James Kuyper]

Post by Philip Lantz

Post by James Kuyper
...

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

[lots of detailed explanation of ways that those functions might be
implemented that would render such code problematic]

Post by Philip Lantz

Post by James Kuyper
Then you disagree with Martin - it's not obviously true?

Martin said that it's obviously true that "it is possible to implement
the functions" like that. He did not say or imply that all
implementations do that, or that it's the best way, or that it should be
required.

He only explicitly acknowledged that such implementations are possible,
which is a very week statement - it's also possible for a meteorite to
hit the Eiffel tower. However, he used the obviousness of that statement
to dismiss as unimportant evidence that such implementations might be
commonplace. That is an implicit acknowledgment that they are at least
unexceptional - a much stronger statement than merely "possible".

[m***@yahoo.co.uk]

He [Martin] only explicitly acknowledged that such implementations are
possible, which is a very weak statement

Yes (it was intended as such).

However, he used the obviousness of that statement
to dismiss as unimportant evidence that such implementations might be
commonplace.

Ah, *that* is your point! Right. I interpreted the test as evidence
that such implementations exist (not that they are common). The question
is: do other implementations exist (where the desired property does not
hold)? I don't know the answer to that.

That is an implicit acknowledgement that they are at least
unexceptional - a much stronger statement than merely "possible".

[m***@yahoo.co.uk]

Post by James Kuyper
...

Post by Jakob Bohm

Post by James Kuyper

It is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

To answer that question: The trouble is that _forcing_ implementers
to write the functions so they do nothing if length is zero may cost
optimization possibilities. C has always taken the view that it is
better to force the programmer to jump through hoops than lose a single
optimization; personally, I think this has probably gone too far and
that a bit more defined behaviour would improve the reliability of
software quite a lot.

Post by James Kuyper

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Then you disagree with Martin - it's not obviously true?

I don't think he is disagreeing with me. It is possible to write
the functions to do nothing if the length is zero - but it is also
possible to write them to invoke undefined behaviour / seg-fault if
the pointers are not valid.

As a specific example, if one was using X86 pointers with segment+offset,
one might well want to load the source and destination segment registers
_then_ check the length (because loading a segment register takes time,
and one might as well do something while the CPU gets on with it).

[Richard Kettlewell]

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn’t the whole story. Perhaps we could have
memcpy_unsafe() for “as fast as possible, but has weird constraints” and
reserve memcpy() for a safer “never accesses more than n bytes”
behavior?

Meanwhile there are legitimate pointer values that nevertheless cannot
be dereferenced:

void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

[Jakob Bohm]

Post by Richard Kettlewell

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn’t the whole story. Perhaps we could have
memcpy_unsafe() for “as fast as possible, but has weird constraints” and
reserve memcpy() for a safer “never accesses more than n bytes”
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

If not already so phrased in the standard, it would be a reasonable
interpretation, that the pointer just past the end of an array is
a valid pointer to an array of zero elements.

I am unsure if memcpy() et al. are currently defined to handle a
count of 0, or if it is allowed to do the equivalent of:

void* Maybe_memcpy(void *p, const void *q, size_t n)
{
char *p1 = p;
const char *q1 = 1;

do {
*p1++ = *q1++;
} while (n--);

return p;
}


Enjoy

Jakob

[James Kuyper]

Post by Jakob Bohm

Post by Richard Kettlewell

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn’t the whole story. Perhaps we could have
memcpy_unsafe() for “as fast as possible, but has weird constraints” and
reserve memcpy() for a safer “never accesses more than n bytes”
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

If not already so phrased in the standard, it would be a reasonable
interpretation, that the pointer just past the end of an array is
a valid pointer to an array of zero elements.
I am unsure if memcpy() et al. are currently defined to handle a
void* Maybe_memcpy(void *p, const void *q, size_t n)
{
char *p1 = p;
const char *q1 = 1;

I'll assume "= 1;" should have been "= q;"?

What purpose is served by q1? Why not use q directly?

Post by Jakob Bohm
do {
*p1++ = *q1++;
} while (n--);

[Re: <string.h>]
"Where an argument declared as size_t n specifies the length of the
array for a function, n can have the value zero on a call to that
function. Unless explicitly stated otherwise in the description of a
particular function in this subclause, pointer arguments on such a call
shall still have valid values, as described in 7.1.4. On such a call, a
function that locates a character finds no occurrence, a function that
compares two character sequences returns zero, and a function that
copies characters copies zero characters." (7.24.1p2)

If p points at modifiable storage, and q is any valid non-null pointer,
and n is 0, then none of the things that 7.1.4 says to identify invalid
pointer values applies, so (7.24.1p2) requires that no characters be
copied. Your code copies one character in that case.

Post by Jakob Bohm
return p;
}

[Philip Lantz]

Post by James Kuyper

Post by Jakob Bohm

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn?t the whole story. Perhaps we could have
memcpy_unsafe() for ?as fast as possible, but has weird constraints? and
reserve memcpy() for a safer ?never accesses more than n bytes?
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

If not already so phrased in the standard, it would be a reasonable
interpretation, that the pointer just past the end of an array is
a valid pointer to an array of zero elements.
I am unsure if memcpy() et al. are currently defined to handle a
void* Maybe_memcpy(void *p, const void *q, size_t n)
{
char *p1 = p;
const char *q1 = 1;

I'll assume "= 1;" should have been "= q;"?
What purpose is served by q1? Why not use q directly?

What should q++ do?

Post by James Kuyper

Post by Jakob Bohm
do {
*p1++ = *q1++;
} while (n--);

[James Kuyper]

...

Post by Philip Lantz

Post by James Kuyper

Post by Jakob Bohm
void* Maybe_memcpy(void *p, const void *q, size_t n)
{
char *p1 = p;
const char *q1 = 1;

I'll assume "= 1;" should have been "= q;"?
What purpose is served by q1? Why not use q directly?

What should q++ do?

Sorry - I read the code too fast - I missed the change of pointer type.
I've been missing a bit of sleep lately due to the combination of tax
season and newborn twins.

Hopefully, a sufficiently smart compiler could recognize that q1 and q
can occupy the same piece of memory (by the as-if rule).

[Philip Lantz]

Post by James Kuyper

Post by Philip Lantz

Post by James Kuyper
What purpose is served by q1? Why not use q directly?

What should q++ do?

Sorry - I read the code too fast - I missed the change of pointer type.
I've been missing a bit of sleep lately due to the combination of tax
season and newborn twins.

Congratulations! I suspected something like that.

Post by James Kuyper
Hopefully, a sufficiently smart compiler could recognize that q1 and q
can occupy the same piece of memory (by the as-if rule).

They do, in my experience. The name/type change generates no code.

[ivan dobrokotov]

========================================
I have searched http://www.open-std.org/jtc1/sc22/wg14/ , and https://groups.google.com/forum/m/#!topic/comp.std.c/yIpg6gtdOoU . But I am still not sure how it's better to start DR submission procedure (I do not want fuss, I know members of committee already have a lot of work, but obscurity is confusing).

========================================
We know at least one existing compiler, where undefined behavior have visible impact on code execution. Not "possible possibilities" and "we can imagine", but right now, here, real, target of any portable code.
It has been already mentioned, gcc 4.9
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61236

This demonstration how NULL restriction can be exploited by compiler is real demonstration of how really this can be exploited by compiler.

gcc 4.9 uses this restriction not for memcpy optimization itself (and other string.h routines), but couple it with reasoning about pointer value (even for zero array size).

========================================
My thoughts about 'Pro' and 'Cons' about prohibition to pass NULL to string.h functions with zero n (array size) argument:

Pro:
1) Vague "optimization opportunity" of memcpy itself, and more (false?) compiler reasoning about program logic.
2) Do not render existing gcc 4.9.0 implementation as invalid.

Cons:
1) This is also anti-optimization - we add additional check to algorithm which is already works fine with NULL. Compiler can remove such 'if', but it's strange to force programmer to write meaningless 'if's, and such optimization is not portable to other compilers. Hardcore code optimization persons are not happy.

2) Security: this optimization can be used by hackers on existing soft compiled in new way, to send empty input converted to {ptr=NULL, size=0} dynamic array and bypass some code validation.

3) This NULLeness pointer deduction is coupled with copying algorithm. It's important for library to be orthogonal, and do not couple two possible independent wishes (optimization hint, and byte copying).
If programmer can ensure that pointer is not null, he can write explicit hint. Standardization of such a hint, like the __assume/__expect is much orthogonal and gives programmer a lot more of possibilities.
It is better to state, that memcpy is equivalent of
{while(n--)dst[n]=src[n];},
not equivalent of something like
{ if(n || src==NULL/*why?..*/) { *src; } while(n--){dst[i]=src[i]}; }.

Hardcore code optimization person still can hint compiler, even inside MACRO-generated code.

4) If compiler can deduce that n>0 in the memcpy(dst,src,n) call, there are still reasonable hint that dst and src pointers are not NULL.

if (src) { i += 1; }
/* previous 'if'-check can be dropped, if compiler is sure
that strlen() can't return (size_t(0)-1) on target platform. */
memcpy(dst, src, strlen(src) + 1)

6) If memory allocation library makes one-element-past-end of array memory unreadable, it's unlikely that memcpy implementation tries to read/write any pointer when n=0.

(less technical reasoning, but still have large impact on the world):
5) Make gcc(6+N).0 ... more robust. May be, even gcc 4.9.* and 5.1.* next minor update.
7) Make the world better place
8) less if's and branches - better and faster software.

===================================
There is important typo in my test code:
- TEST(dst == memset(dst, 0, 0));
+ TEST(dst == memset(dst, 0, n));

Full code: https://gist.github.com/dobrokot/61959f21609abdea0960

And for future readers of mail archive, copy:
===================================
#include <string.h>
#include <stdio.h>

typedef char *p;

volatile p src = NULL;
volatile p dst = NULL;
volatile size_t n = 0;

#define TEST(x) if (x) ; else { printf("error, %s\n", #x); }

int main() {
TEST(dst == memcpy(dst, src, n));
TEST(dst == memmove(dst, src, n));
TEST(dst == strncpy(dst, src, n));
TEST(dst == memset(dst, 0, n));
TEST(0 == strncmp(src, dst, n));
TEST(0 == memcmp(src, dst, n));


/* Observable effect after observable reads from 'volatile n'.
OS are not always give explicit message
about segmentation fault/invalid read at process termination. */
printf("Still alive. Still alive.\n");
return 0;
}

===================================

[Tim Rentsch]

Post by ivan dobrokotov
I have searched http://www.open-std.org/jtc1/sc22/wg14/ , and
https://groups.google.com/forum/m/#!topic/comp.std.c/yIpg6gtdOoU.
But I am still not sure how it's better to start DR submission
procedure (I do not want fuss, I know members of committee
already have a lot of work, but obscurity is confusing).
We know at least one existing compiler, where undefined behavior
have visible impact on code execution. Not "possible
possibilities" and "we can imagine", but right now, here, real,
target of any portable code. It has been already mentioned, gcc
4.9 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61236

Go ahead and file a DR if you think that's necessary, but
don't be disappointed if the response is "what's there is
what we meant, and it won't be changed." The indications
are that the committee members knew what they were saying
and intended exactly that, ie, that null arguments to
memcpy() result in undefined behavior regardless of how
many bytes are to be copied.

Also, it's perfectly easy to sidestep the problem if someone
wants to do that, by wrapping calls to memcpy() in another
function:

static inline void *
safer_memcpy( void *a, const void *b, size_t n ){
return n ? memcpy( a, b, n ) : a;
}

Calling safer_memcpy() rather than memcpy() avoids problems
like those described in the gcc bug report.

[Jakob Bohm]

Post by James Kuyper

Post by Jakob Bohm

Post by Richard Kettlewell

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn’t the whole story. Perhaps we could have
memcpy_unsafe() for “as fast as possible, but has weird constraints” and
reserve memcpy() for a safer “never accesses more than n bytes”
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

If not already so phrased in the standard, it would be a reasonable
interpretation, that the pointer just past the end of an array is
a valid pointer to an array of zero elements.
I am unsure if memcpy() et al. are currently defined to handle a
void* Maybe_memcpy(void *p, const void *q, size_t n)
{
char *p1 = p;
const char *q1 = 1;

I'll assume "= 1;" should have been "= q;"?

Yep, typo.

Post by James Kuyper
What purpose is served by q1? Why not use q directly?

Post by Jakob Bohm
do {
*p1++ = *q1++;
} while (n--);

[Re: <string.h>]
"Where an argument declared as size_t n specifies the length of the
array for a function, n can have the value zero on a call to that
function. Unless explicitly stated otherwise in the description of a
particular function in this subclause, pointer arguments on such a call
shall still have valid values, as described in 7.1.4. On such a call, a
function that locates a character finds no occurrence, a function that
compares two character sequences returns zero, and a function that
copies characters copies zero characters." (7.24.1p2)
If p points at modifiable storage, and q is any valid non-null pointer,
and n is 0, then none of the things that 7.1.4 says to identify invalid
pointer values applies, so (7.24.1p2) requires that no characters be
copied. Your code copies one character in that case.

Thanks, I wasn't sure, don't have the final std handy.

Then the remaining issue is how preloading the pointers into memory
manager processing might fail with NULL or fully invalid pointers.

One case I know of for sure is this size-optimized implementations for
x86 16 bit protected mode:

; Save registers first
les di,[p]
mov cx,[n]
lds si,[q]
cld ; Optional if calling convention already requires this
mov dx,es
mov ax,di
rep movs es:[BYTE PTR di],[BYTE PTR si]
; restore registers and return dx:ax

The les and lds instructions evaluate the memory management data for
the "segment" portion of the pointer values, accept 0 as valid, but
will raise a fault on other invalid values.

On this architecture, it is generally required that the end of array
pointer is at most xxxx:FFFF so subtractions of the low 16 bits
produce valid pointer differences. This makes the code above work
for one-past-end pointers too.


Enjoy

Jakob

[Tim Rentsch]

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn?t the whole story. Perhaps we could have
memcpy_unsafe() for ?as fast as possible, but has weird constraints? and
reserve memcpy() for a safer ?never accesses more than n bytes?
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

My reading of the Standard is that this case is defined
behavior. More specifically, the semantics are defined
in the paragraphs pertaining to memcpy(), including the
common portions in parent section(s), and none of the
explicit circumstances of UB given in 7.1.whatever-it-is
apply (at least, not that I can see). Hence the behavior
is defined, assuming I'm not missing something.

[Richard Kettlewell]

Post by Tim Rentsch

Post by Richard Kettlewell
Of course performance isn’t the whole story. Perhaps we could have
memcpy_unsafe() for ‘as fast as possible, but has weird constraints’
and reserve memcpy() for a safer ‘never accesses more than n bytes’
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

My reading of the Standard is that this case is defined
behavior. More specifically, the semantics are defined
in the paragraphs pertaining to memcpy(), including the
common portions in parent section(s), and none of the
explicit circumstances of UB given in 7.1.whatever-it-is
apply (at least, not that I can see). Hence the behavior
is defined, assuming I'm not missing something.

That was my provisional conclusion too.

So is there is a real platform where a memcpy(0,0,0) that speculatively
dereferences 0 will not be the no-op that the programmer hoped for, but
speculatively dereferencing the address just after any array is always
safe (_and_ where the speculative dereference is actually worthwhile)?

[Tim Rentsch]

Post by Richard Kettlewell

Post by Tim Rentsch

Of course performance isn?t the whole story. Perhaps we could have
memcpy_unsafe() for ?as fast as possible, but has weird constraints?
and reserve memcpy() for a safer ?never accesses more than n bytes?
behavior?
Meanwhile there are legitimate pointer values that nevertheless
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

My reading of the Standard is that this case is defined
behavior. More specifically, the semantics are defined
in the paragraphs pertaining to memcpy(), including the
common portions in parent section(s), and none of the
explicit circumstances of UB given in 7.1.whatever-it-is
apply (at least, not that I can see). Hence the behavior
is defined, assuming I'm not missing something.

That was my provisional conclusion too.
So is there is a real platform where a memcpy(0,0,0) that
speculatively dereferences 0 will not be the no-op that the
programmer hoped for, but speculatively dereferencing the address
just after any array is always safe (_and_ where the speculative
dereference is actually worthwhile)?

The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.

The gcc bug report cited elsethread illustrates one such
scenario. Suppose we have code something like this:

. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...

This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.

However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

[Richard Kettlewell]

Post by Tim Rentsch

Post by Richard Kettlewell
So is there is a real platform where a memcpy(0,0,0) that
speculatively dereferences 0 will not be the no-op that the
programmer hoped for, but speculatively dereferencing the address
just after any array is always safe (_and_ where the speculative
dereference is actually worthwhile)?

The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.
The gcc bug report cited elsethread illustrates one such
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...
This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.
However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

I think you’ve got cause and effect backwards? The miscompilation
problem occurs (at most) where the standard permits it to occur. The
optimizer behavior doesn’t drive the standard, the standard has driven
the optimizer behavior.

The point of my question is that if there’s no such real platform then
the freedom the standard offers is simply freedom for optimizers to
miscompile code that would otherwise be perfectly reasonable.

The current situation is also one of the standard ‘having its cake and
eating it’: programmers aren’t allowed to use NULL as a ‘zero size’
array, but the implementation of malloc() is free to do just that, and
on at least one real platform actually does so, which is another cause
of bugs.

[Tim Rentsch]

Post by Tim Rentsch

Post by Richard Kettlewell
So is there is a real platform where a memcpy(0,0,0) that
speculatively dereferences 0 will not be the no-op that the
programmer hoped for, but speculatively dereferencing the address
just after any array is always safe (_and_ where the speculative
dereference is actually worthwhile)?

The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.
The gcc bug report cited elsethread illustrates one such
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...
This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.
However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

I think you've got cause and effect backwards: The miscompilation
problem occurs (at most) where the standard permits it to occur.
The optimizer behavior doesn't drive the standard, the standard
has driven the optimizer behavior.

No, I haven't got them backwards, I already understood that.
But now I see the point of your question is different from
my earlier sense of it.

The point of my question is that if there's no such real
platform then the freedom the standard offers is simply freedom
for optimizers to miscompile code that would otherwise be
perfectly reasonable.

I mostly agree. I think the question could be expanded to
ask also whether there is likely to be such a platform in
the future. But the burden of proof should be on those
who say that there is, or that there likely will be -
saying simply that there might be is not sufficient to
be convincing.

The current situation is also one of the standard 'having its
cake and eating it': programmers aren't allowed to use NULL as
a 'zero size' array, but the implementation of malloc() is free
to do just that, and on at least one real platform actually
does so, which is another cause of bugs.

Seems like a minor point, although it might be one I agree with.
What bothers me more is the general trend that behavior should be
undefined unless there is a compelling argument that it should
defined. This is backwards - the presumption is that behavior
should be defined unless there is at least some compelling
evidence that having it be undefined would produce a significant
benefit (perhaps only on some platforms).

So now that I see the point of what you were asking, I think we
are in agreement, at least as far as general principle goes.

[Richard Kettlewell]

Post by Tim Rentsch
Seems like a minor point, although it might be one I agree with.
What bothers me more is the general trend that behavior should be
undefined unless there is a compelling argument that it should
defined. This is backwards - the presumption is that behavior
should be defined unless there is at least some compelling
evidence that having it be undefined would produce a significant
benefit (perhaps only on some platforms).

I very much agree.

Post by Tim Rentsch
So now that I see the point of what you were asking, I think we
are in agreement, at least as far as general principle goes.

Yes.

[Richard Damon]

Post by Tim Rentsch
The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.
The gcc bug report cited elsethread illustrates one such
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...
This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.
However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

I would agree with the bug report, but could perhaps express it
differently. Yes, by the standard the implementation can consider that x
must not be NULL, but if it does, then shouldn't it at least give a
"Condition is always false" warning on the if statement? If it doesn't
assume it is always false, then it should do the test and execute the
code. This is an inconsistency between parts of the compiler.

[Hans-Bernhard Bröker]

Post by Richard Damon

Post by Tim Rentsch
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...

I would agree with the bug report, but could perhaps express it
differently. Yes, by the standard the implementation can consider that x
must not be NULL, but if it does, then shouldn't it at least give a
"Condition is always false" warning on the if statement?

No. For two reasons:

1) The compiler is, of course, _allowed_ to issue such a diagnostic, but
it is by no means asked, much less required to do so.

2) That particular diagnostic would be a lie. The condition isn't known
to be always false. It's only known to be irrelevant.

The right warning to issue in this case, if any, would be "possible null
pointer argument in call to memcpy()".

[Richard Damon]

Post by Hans-Bernhard Bröker

Post by Richard Damon

Post by Tim Rentsch
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...

I would agree with the bug report, but could perhaps express it
differently. Yes, by the standard the implementation can consider that x
must not be NULL, but if it does, then shouldn't it at least give a
"Condition is always false" warning on the if statement?

1) The compiler is, of course, _allowed_ to issue such a diagnostic, but
it is by no means asked, much less required to do so.
2) That particular diagnostic would be a lie. The condition isn't known
to be always false. It's only known to be irrelevant.
The right warning to issue in this case, if any, would be "possible null
pointer argument in call to memcpy()".

Of course it isn't a bug by the standard, as the warning "Condition is
always false" isn't a requirement by the standard.

But, once the implementation generates such a warning, it is a clear
inconsistency to on one hand use the fact that you conclude that the
expression must be false (or undefined behavior has occurred) to omit
the code generation, but not consider it suitable for the warning
(assuming that warning is enabled).

This is the fundamental problem behind most of the "surprising
mis-compliation" conditions. You have a compiler that historically has
been very good at pointing out questionable situations that might be
program errors, you then add an aggressive optimizer that detects these
sorts of conditions, and "improves" the code, but then doesn't generate
a warning that if it had been a more obvious condition it would have.

The compiler has thus created a bug of being inconsistent and failed to
report a condition that it is documented as reporting, even though it
HAS detected that condition. It isn't non-conformance to the "standard",
just deviation from its documented operation.

I can see no way to justify that the optimization without warning is
something that would be desired, in fact, the detection is likely to
indicate a latent problem with the code, which is what warnings are
designed to show.

[Tim Rentsch]

Post by Richard Damon

Post by Tim Rentsch
The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.
The gcc bug report cited elsethread illustrates one such
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...
This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.
However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

I would agree with the bug report, but could perhaps express it
differently. Yes, by the standard the implementation can consider
that x must not be NULL, but if it does, then shouldn't it at
least give a "Condition is always false" warning on the if
statement? If it doesn't assume it is always false, then it
should do the test and execute the code. This is an inconsistency
between parts of the compiler.

Philosophically I agree with you. However let me make a few
observations.

One, it isn't that the condition is always false, it is that the
condition may be "legally inferred" to be false. The inference
is allowed (ie, by the Standard) but that doesn't mean it must
be included in the compiler's computational logic.

Two, what I think you're interested in is not the possibility of
a consequence (that an unexpected code transformation _might_
happen) but the fact of a consequence (that an unexpected code
transformation _did_ happen). This condition is not always easy
to detect (or so I have been led to believe) because of how the
result is arrived at, namely, via the optimizer, which sometimes
makes it hard to relate cause and effect.

Three, going back to the beginning, any such assumption (eg, that
passing a pointer value to memcpy() may be used to imply that the
pointer necessarily has a non-null value) has the potential for
dire consequences, both for the process of program development
and for the drastic results that may occur as a result of running
programs affected by such compilers. My position is that any
such inference rule, which by their nature may produce an unsound
conclusion, MUST be overridable via a compiler option (preferably
individually, but _at least_ a single option that turns off all
of them). And if it is possible to disable the "suspicioius
inference rules", that provides a tool that can be used to detect
whether transformations depending on those rules will occur for
a given program upon allowing such inferences.

[Richard Damon]

Post by Tim Rentsch

Post by Richard Damon

Post by Tim Rentsch
The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.
The gcc bug report cited elsethread illustrates one such
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...
This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.
However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

I would agree with the bug report, but could perhaps express it
differently. Yes, by the standard the implementation can consider
that x must not be NULL, but if it does, then shouldn't it at
least give a "Condition is always false" warning on the if
statement? If it doesn't assume it is always false, then it
should do the test and execute the code. This is an inconsistency
between parts of the compiler.

Philosophically I agree with you. However let me make a few
observations.
One, it isn't that the condition is always false, it is that the
condition may be "legally inferred" to be false. The inference
is allowed (ie, by the Standard) but that doesn't mean it must
be included in the compiler's computational logic.
Two, what I think you're interested in is not the possibility of
a consequence (that an unexpected code transformation _might_
happen) but the fact of a consequence (that an unexpected code
transformation _did_ happen). This condition is not always easy
to detect (or so I have been led to believe) because of how the
result is arrived at, namely, via the optimizer, which sometimes
makes it hard to relate cause and effect.
Three, going back to the beginning, any such assumption (eg, that
passing a pointer value to memcpy() may be used to imply that the
pointer necessarily has a non-null value) has the potential for
dire consequences, both for the process of program development
and for the drastic results that may occur as a result of running
programs affected by such compilers. My position is that any
such inference rule, which by their nature may produce an unsound
conclusion, MUST be overridable via a compiler option (preferably
individually, but _at least_ a single option that turns off all
of them). And if it is possible to disable the "suspicioius
inference rules", that provides a tool that can be used to detect
whether transformations depending on those rules will occur for
a given program upon allowing such inferences.

I think you are missing my point. The purpose of generating warnings is
to point out potentially bad code. While we can't demand that a compiler
find every thing that could be bad code, if the compiler does determine
that a piece of code meets one of the defined criteria for a warning, to
not generate that warning (if it is enabled) is a bug, and a very user
hostile type of action (I could have helped you, but I decided not to).

It has been decided previously, that condition testing in an expression,
that we can determine from program analysis to always be false (or true)
is possibly a programming error (likely either there is a problem in the
path coming to here is wrong (we shouldn't have been able to do the
simplification), the test is wrong (we are testing for the wrong
condition).

Here we have a case that we have detected that the program thinks that
the pointer might have been null, so is testing for it, but the compiler
has determined that the only way to get here is through paths that the
pointer being null would have invoked undefined behavior, so it will
assume it isn't null. From the previous statements, it should be clear
that the compiler is being deliberately unhelpful, and based on the
documentation, it is reasonable to expect that the compiler would issue
a warning here. It then compounds the unhelpfulness by generating code
that bypasses the case that might have detected the problem in the code
(abet, after the fact). This seems to indicate a compiler writer that
has lost part of the goal of the compiler, to help the programmer create
a working program.

[Tim Rentsch]

Post by Tim Rentsch

Post by Richard Damon

Post by Tim Rentsch
The problem is more subtle than speculative dereference. Or more
specifically, there can be ramifications of undefined behavior
(in the case of passing a null pointer) even if code in memcpy()
does not dereference its arguments when the third argument has
the value zero.
The gcc bug report cited elsethread illustrates one such
. . .
memcpy( x, y, k ); // where k will sometimes have the value 0
. . .
if ( !x ) ...something...
This code can be "miscompiled". That is, a compiler is allowed
to throw away the entire if() statement, since it may be
conformingly inferred that 'x' is not null, _because of the call
to memcpy()_. So it doesn't matter if memcpy() tries to fetch
a byte or not - there can be consequences even if it doesn't.
However, these problems don't occur for pointers just past the
end of an array. Which may partly explain why such pointers
are not included in the explicit UB situations listed in 7.1.4.

I would agree with the bug report, but could perhaps express it
differently. Yes, by the standard the implementation can consider
that x must not be NULL, but if it does, then shouldn't it at
least give a "Condition is always false" warning on the if
statement? If it doesn't assume it is always false, then it
should do the test and execute the code. This is an inconsistency
between parts of the compiler.

Philosophically I agree with you. However let me make a few
observations.
One, it isn't that the condition is always false, it is that the
condition may be "legally inferred" to be false. The inference
is allowed (ie, by the Standard) but that doesn't mean it must
be included in the compiler's computational logic.
Two, what I think you're interested in is not the possibility of
a consequence (that an unexpected code transformation _might_
happen) but the fact of a consequence (that an unexpected code
transformation _did_ happen). This condition is not always easy
to detect (or so I have been led to believe) because of how the
result is arrived at, namely, via the optimizer, which sometimes
makes it hard to relate cause and effect.
Three, going back to the beginning, any such assumption (eg, that
passing a pointer value to memcpy() may be used to imply that the
pointer necessarily has a non-null value) has the potential for
dire consequences, both for the process of program development
and for the drastic results that may occur as a result of running
programs affected by such compilers. My position is that any
such inference rule, which by their nature may produce an unsound
conclusion, MUST be overridable via a compiler option (preferably
individually, but _at least_ a single option that turns off all
of them). And if it is possible to disable the "suspicioius
inference rules", that provides a tool that can be used to detect
whether transformations depending on those rules will occur for
a given program upon allowing such inferences.

I think you are missing my point. The purpose of generating
warnings is to point out potentially bad code. While we can't
demand that a compiler find every thing that could be bad code, if
the compiler does determine that a piece of code meets one of the
defined criteria for a warning, to not generate that warning (if
it is enabled) is a bug, and a very user hostile type of action (I
could have helped you, but I decided not to).
It has been decided previously, that condition testing in an
expression, that we can determine from program analysis to always
be false (or true) is possibly a programming error (likely either
there is a problem in the path coming to here is wrong (we
shouldn't have been able to do the simplification), the test is
wrong (we are testing for the wrong condition).
Here we have a case that we have detected that the program thinks
that the pointer might have been null, so is testing for it, but
the compiler has determined that the only way to get here is
through paths that the pointer being null would have invoked
undefined behavior, so it will assume it isn't null. From the
previous statements, it should be clear that the compiler is being
deliberately unhelpful, and based on the documentation, it is
reasonable to expect that the compiler would issue a warning here.
It then compounds the unhelpfulness by generating code that
bypasses the case that might have detected the problem in the code
(abet, after the fact). This seems to indicate a compiler writer
that has lost part of the goal of the compiler, to help the
programmer create a working program.

I believe I did (and do) understand your point, but I guess I
didn't do a good job of communicating that. Let me try again,
starting with summarizing/paraphrasing your comments.

(1) What happens in the memcpy()/if(x) case is, or appears to be,
the same situation as one that gets a warning for "condition is
always false";

(2) Because the compiler gives a warning for the second case but
not the first, it appears to users that the compiler writers are,
at best, being capricious and arbitrary, and possibly much worse
than that (eg, hostile);

(3) Whether the compiler writer has been unhelpful deliberately
or just accidentally, the result merits criticism because an
important aspect of compiler functionality is being neglected
(it might be malfeasance or it might be just misfeasance, but
in either case the attitude and behavior is below what should
be considered acceptable).

Is this a reasonable summary (perhaps somewhat milder, but
still roughly similar) as what you were saying? If it isn't
then I am confused.

If it is a reasonable summary, then I would offer some additional
points as follows.

(1) I agree the two cases appear to be the same situation, but
they aren't really the same.

(2) The detection mechanisms needed in the two cases are quite
different from each other. I understand that the two situations
appear similar, but appearances are deceiving in this instance.

(3) In some cases like this one (I'm not sure about the
particular case here), it isn't easy to detect what has happened
well enough to be able to put in the same category as "condition
is always false". I agree that it would be helpful to get a
diagnostic along those lines, but the nature of the problem
sometimes makes that very difficult, or even totally impractical.

Now, assuming (3) applies in this particular case, what can be
done? A workable solution to the problem must observe the needs
of both parties (ie, developers and compiler writers). It
doesn't help to insist the compiler writers do something if that
something isn't in the set of things they can practically do. If
they can't do what we want, what can they do that will helps us
achieve our goal? This question is what I was trying to address
in my earlier comments (ie, previous posting).

Did this help at all?

[Phil Carmody]

Post by Tim Rentsch

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn?t the whole story. Perhaps we could have
memcpy_unsafe() for ?as fast as possible, but has weird constraints? and
reserve memcpy() for a safer ?never accesses more than n bytes?
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

My reading of the Standard is that this case is defined
behavior. More specifically, the semantics are defined
in the paragraphs pertaining to memcpy(), including the
common portions in parent section(s), and none of the
explicit circumstances of UB given in 7.1.whatever-it-is
apply (at least, not that I can see). Hence the behavior
is defined, assuming I'm not missing something.

Yup, concur. There are other corner cases:

char *f(size_t n) {
char *p = malloc(n);
memset(p, 0, n);
return p;
}
//...
char *p = f(0); /* defined behaviour or not? */

Phil

[Tim Rentsch]

Post by Phil Carmody

Post by Tim Rentsch

Post by Jakob Bohm
It is plausible that some implementations for some CPUs will trigger
a NULL pointer exception before detecting that the iteration count is
0. It is also worth noting that these particular functions are some
of the most speed-optimized in C run time libraries, and every
constraint imposed on the corner cases is likely to slow down some
implementations due to lost optimization possibilities. For example,
this is why memcpy() is rarely an alias for memmove() and bzero() is
rarely an alias for memset(,0,), because those few extra features have
a performance cost.

Of course performance isn?t the whole story. Perhaps we could have
memcpy_unsafe() for ?as fast as possible, but has weird constraints? and
reserve memcpy() for a safer ?never accesses more than n bytes?
behavior?
Meanwhile there are legitimate pointer values that nevertheless cannot
void f(void) {
static char a[1], b[1];
memcpy(&a[1], &b[1], 0); /* defined behavior or not? */
}

My reading of the Standard is that this case is defined
behavior. More specifically, the semantics are defined
in the paragraphs pertaining to memcpy(), including the
common portions in parent section(s), and none of the
explicit circumstances of UB given in 7.1.whatever-it-is
apply (at least, not that I can see). Hence the behavior
is defined, assuming I'm not missing something.

char *f(size_t n) {
char *p = malloc(n);
memset(p, 0, n);
return p;
}
//...
char *p = f(0); /* defined behaviour or not? */

Assuming the return value of malloc(n) is not null, I
believe this case is also defined behavior, for the same
reasons as the previous one. If malloc(n) returns a null
pointer then the behavior is undefined of course (and
remember that malloc(0) can be implementation-defined to be
null always). Besides the question about malloc(n) return
value, I don't see any distinction between this case and
the previous one.

[Philip Lantz]

Post by James Kuyper

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
I had tested assumption about NULL-ready string.h functions is
supported by at least trivial counterexamples (runtime implementation
of string.h functions on volatile input).

Given that NULL arguments to the following functions are all undefined
behaviour, I don't see what you think your test shows?

I think he's trying to show that it would be feasible to define the
behavior as "does nothing", because that's what many existing
implementations already do. Of course, it's hard to prove that they "do
nothing" rather than "do something (possibly a very dangerous something)
but with no obvious effect".

Post by m***@yahoo.co.uk

Post by ivan dobrokotov
TEST(x == memcpy(x, y, n));
TEST(x == memmove(x, y, n));
TEST(x == strncpy(x, y, n));
TEST(x == memset(x, 0, 0));
TEST(0 == strncmp(x, y, n));
TEST(0 == memcmp(x, y, n));

Perhaps that it is possible to implement the functions so that if
the length is zero nothing happens even if a pointer is NULL? If so,
yes, that is obviously true - but not very interesting.

If you consider it obviously true, then you shouldn't object to making
the behavior defined. Do you?

Just because it is obviously true that it is *possible* to implement the
functions like that, doesn't in any way imply that it should be
*required* to implement them like that!