RFT: Please help testing the llvm/clang 3.5.0 import

[Dimitry Andric]

Hi,

We're working on updating llvm, clang and lldb to 3.5.0 in head. This
is quite a big update again, and any help with testing is appreciated.

To try this out, ensure you have good backups or snapshots, then build
world and kernel from the projects/clang350-import branch [1]. Please
use a Subversion mirror [2], if you are able to.

The status of this project branch is as follows:

* Since llvm/clang 3.5.0 requires C++11 support, you can only build it
when your current installation has both clang (>= 3.3) and libc++
installed. E.g., FreeBSD 10.x and later should work out of the box,
but for FreeBSD 9.x you should first build libc++ with clang, and
install it. Older versions of FreeBSD will not work.
* Both the i386 and amd64 arches are expected to work completely, e.g.
they should build, install and run without any problems. For some
less-used parts of world and kernel, you might encounter warnings
that are not fixed yet. To ignore those, you can use NO_WERROR, but
please create bug reports for them.
* The different ARM builds still need work, any help would be greatly
appreciated there.
* PowerPC (32 and 64 bit) will most likely not work yet, until we can
figure out how to build parts of the tree with clang, other parts with
gcc.
* Sparc64 might work at least partially, but has not been tested on
real hardware.
* A ports exp-run has been requested [3].

The tentative goal is to be able to import this new version before the
end of the year, hopefully before Christmas.

If you encounter issues, please report them in FreeBSD Bugzilla [4],
unless you think it is better discussed on one of the appropriate
mailing lists.

-Dimitry

[1] svn://svn.freebsd.org/base/projects/clang350-import
[2] https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html#svn-mirrors
[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195480
[4] https://bugs.freebsd.org/submit/

[Dmitry Marakasov]

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head.
This is quite a big update again, and any help with testing is
appreciated.

Well, of 4 error logs from exp-run I've checked (one my port and 3
unmaintained ports) two had basically the same problem and it seems
to be libc++ related, so I ask: was new version of libc++ imported
along with clang/llvm? Past experience show that libc++ should be
updated along with clang, as it may have bugs new clang versions
are not tolerable to.

I've prepared the minimal test program to demonstrate the problem:

--- test.cc begins here ---
#include <functional>
#include <iostream>

class Foo {
public:
void Const() const { std::cerr << "OK" << std::endl; }
};

int main() {
Foo foo;
auto f = std::bind(&Foo::Const, foo);
f();

return 0;
}
--- test.cc ends here ---

Note that the method std::bind is used on is const (non-const methods
produce no problems).

Here's how it works:

--- test.log begins here ---
% for CC in c++ g++48 g++49 g++5 clang++33 clang++34 clang++35; do echo "===> ${CC}"; ${CC} -std=c++11 test.cc && ./a.out; done
===> c++
OK
===> g++48
OK
===> g++49
OK
===> g++5
OK
===> clang++33
OK
===> clang++34
OK
===> clang++35
In file included from test.cc:1:
In file included from /usr/include/c++/v1/functional:474:
/usr/include/c++/v1/type_traits:433:76: error: no matching function for call to '__source'
: public integral_constant<bool, sizeof(__is_function_imp::__test<_Tp>(__is_function_imp::__source<_Tp>())) == 1>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/v1/type_traits:438:14: note: in instantiation of template class 'std::__1::__libcpp_is_function<void () const, false>' requested here
: public __libcpp_is_function<_Tp> {};
^
/usr/include/c++/v1/type_traits:443:97: note: in instantiation of template class 'std::__1::is_function<void () const>' requested here
template <class _Tp, class _Up> struct __libcpp_is_member_function_pointer<_Tp _Up::*> : public is_function<_Tp> {};
^
/usr/include/c++/v1/type_traits:446:14: note: in instantiation of template class 'std::__1::__libcpp_is_member_function_pointer<void (Foo::*)() const>' requested here
: public __libcpp_is_member_function_pointer<typename remove_cv<_Tp>::type> {};
^
/usr/include/c++/v1/type_traits:460:38: note: in instantiation of template class 'std::__1::is_member_function_pointer<void (Foo::*)() const>' requested here
!is_member_function_pointer<_Tp>::value> {};
^
/usr/include/c++/v1/type_traits:3093:17: note: in instantiation of template class 'std::__1::is_member_object_pointer<void (Foo::*)() const>' requested here
is_member_object_pointer<typename remove_reference<_Fp>::type>::value &&
^
/usr/include/c++/v1/type_traits:3100:1: note: (skipping 2 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
__invoke(_Fp&& __f, _A0&& __a0)
^
/usr/include/c++/v1/type_traits:3126:11: note: in instantiation of template class 'std::__1::__invokable_imp<void (Foo::*)() const, Foo &>' requested here
__invokable_imp<_Fp, _Args...>::value>
^
/usr/include/c++/v1/functional:1973:31: note: in instantiation of template class 'std::__1::__invokable<void (Foo::*)() const, Foo &>' requested here
static const bool value = __invokable<_Fp,
^
/usr/include/c++/v1/functional:1985:18: note: in instantiation of template class 'std::__1::_is_valid_bind_return<void (Foo::*)() const, std::__1::tuple<Foo>, std::__1::tuple<> >' requested here
bool = _is_valid_bind_return<_Fp, _BoundArgs, _TupleUj>::value>
^
/usr/include/c++/v1/functional:2082:18: note: in instantiation of default argument for '__bind_return<void (Foo::*)() const, std::__1::tuple<Foo>, std::__1::tuple<> >' required here
typename __bind_return<_Fd, _Td, tuple<_Args&&...> >::type
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test.cc:12:3: note: while substituting deduced template arguments into function template 'operator()' [with _Args = <>]
f();
^
/usr/include/c++/v1/type_traits:424:28: note: candidate template ignored: substitution failure [with _Tp = void () const]: reference to function type 'void () const' cannot have 'const' qualifier
template <class _Tp> _Tp& __source();
~~~ ^
test.cc:12:2: error: no matching function for call to object of type 'std::__1::__bind<void (Foo::*)() const, Foo &>'
f();
^
/usr/include/c++/v1/functional:2083:9: note: candidate template ignored: substitution failure [with _Args = <>]: implicit instantiation of undefined template 'std::__1::__bind_return<void (Foo::*)() const, std::__1::tuple<Foo>, std::__1::tuple<>, false>'
operator()(_Args&& ...__args)
^
/usr/include/c++/v1/functional:2092:9: note: candidate template ignored: substitution failure [with _Args = <>]: implicit instantiation of undefined template 'std::__1::__bind_return<void (Foo::*const)() const, const std::__1::tuple<Foo>, std::__1::tuple<>, false>'
operator()(_Args&& ...__args) const
^
2 errors generated.
--- test.log ends here ---

I think this should be fixed in the first place.

[Dimitry Andric]

Post by Dmitry Marakasov

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head.
This is quite a big update again, and any help with testing is
appreciated.

Well, of 4 error logs from exp-run I've checked (one my port and 3
unmaintained ports) two had basically the same problem and it seems
to be libc++ related, so I ask: was new version of libc++ imported
along with clang/llvm?

No, I really prefer to do this after the 3.5.0 import. This is already
a very big import job, and I'd rather like to avoid importing too many
different components at once.

Post by Dmitry Marakasov
Past experience show that libc++ should be
updated along with clang, as it may have bugs new clang versions
are not tolerable to.

In this case, there is a fairly simple fix:
http://llvm.org/viewvc/llvm-project?view=revision&revision=209785

I have pulled this into head in r275366, and also merged it to the
clang350-import project branch in r275367. Please try again after that
revision. It should be enough to just rebuild lib/libc++ and install
it.

-Dimitry

[Dmitry Marakasov]

Post by Dimitry Andric

Post by Dmitry Marakasov

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head.
This is quite a big update again, and any help with testing is
appreciated.

Well, of 4 error logs from exp-run I've checked (one my port and 3
unmaintained ports) two had basically the same problem and it seems
to be libc++ related, so I ask: was new version of libc++ imported
along with clang/llvm?

No, I really prefer to do this after the 3.5.0 import. This is already
a very big import job, and I'd rather like to avoid importing too many
different components at once.

Post by Dmitry Marakasov
Past experience show that libc++ should be
updated along with clang, as it may have bugs new clang versions
are not tolerable to.

http://llvm.org/viewvc/llvm-project?view=revision&revision=209785
I have pulled this into head in r275366, and also merged it to the
clang350-import project branch in r275367. Please try again after that
revision. It should be enough to just rebuild lib/libc++ and install
it.

Sorry, I haven't tested the branch myself, only seen exp-run results.
Would be nice to have another exp-run.

Btw, is it possible to merge the patch into stable/10 as well?
It will make it possible to use clang35 there.

[Dimitry Andric]

Post by Dmitry Marakasov

Post by Dimitry Andric

Post by Dmitry Marakasov

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head.
This is quite a big update again, and any help with testing is
appreciated.

Well, of 4 error logs from exp-run I've checked (one my port and 3
unmaintained ports) two had basically the same problem and it seems
to be libc++ related, so I ask: was new version of libc++ imported
along with clang/llvm?

No, I really prefer to do this after the 3.5.0 import. This is already
a very big import job, and I'd rather like to avoid importing too many
different components at once.

Post by Dmitry Marakasov
Past experience show that libc++ should be
updated along with clang, as it may have bugs new clang versions
are not tolerable to.

http://llvm.org/viewvc/llvm-project?view=revision&revision=209785
I have pulled this into head in r275366, and also merged it to the
clang350-import project branch in r275367. Please try again after that
revision. It should be enough to just rebuild lib/libc++ and install
it.

Sorry, I haven't tested the branch myself, only seen exp-run results.
Would be nice to have another exp-run.

Yes, but we first need to fix another issue, which is more important:
several of the lang/gcc ports don't work properly, e.g. bootstrap stage
comparison fails.

There is also something fishy going on with gcc in base, which may or
may not be related: building the devel/binutils ports with it causes
cc1plus to segfault while compiling gold's archive.cc.

I am still searching for the root cause; any help in this area would be
greatly appreciated, as the maintainer has not responded yet.

Post by Dmitry Marakasov
Btw, is it possible to merge the patch into stable/10 as well?
It will make it possible to use clang35 there.

Yes, this is also why I prefer to cherry-pick; I have set an MFC timeout
of 3 days.

-Dimitry

[Dimitry Andric]

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head. This
is quite a big update again, and any help with testing is appreciated.
To try this out, ensure you have good backups or snapshots, then build
world and kernel from the projects/clang350-import branch [1]. Please
use a Subversion mirror [2], if you are able to.

Here are some updates about the status of the 3.5.0 import.

* i386 and amd64 have been tested through make universe, and everything
should compile and run.
* Little-endian ARM builds should now compile and run, thanks to Andrew
Turner for putting in lots of work.
* Big-endian ARM is apparently supposed to work, but I'm not sure if
Andrew managed to test it on real hardware.
* PowerPC64 should mostly work, thanks to Justin Hibbits.
* PowerPC32 might start working soon; it really needs some backporting
of fixes to clang 3.4.1, which is now in head, so there is an easier
upgrade path for PowerPC users.
* Sparc64 still does not work, and I don't see any quick solutions to it
for now. It should probably stay with gcc.
* Mips will only have a chance with the upcoming clang 3.6.0, but that
is way too late for this import. It will probably require external
toolchain support to get it working.
* Another ports exp-run was done [3], after fixing the problem with
lang/gcc, which lead to many skipped dependent ports.
* The second exp-run had much better results: the failure with the
highest number of dependencies is devel/mingw32-gcc, but this seems
to be due to a problem with makeinfo, not clang. The next highest on
the list is java/openjdk6, for which ports r374780 [4] was very
recently committed.

I would really like to merge this branch to head in about a week,
pending portmgr approvall; I don't expect the base system (outside of
llvm/clang) to need any further updates.

Lastly, to clear things up about the requirements for this branch (and
thus for head, in a while); to build it, you need to have:
* A C++11 capable "host" compiler, e.g. clang >= 3.3 or later, or gcc

Post by Dimitry Andric
= 4.8 (I'm not 100% sure if gcc 4.7 will work, reports welcome)

* A C++11 standard library, e.g. libc++, or libstdc++ from gcc >= 4.8.

So from any earlier standard 10.x or 11.x installation, you should be
good, unless you explicitly disabled clang or libc++. In that case,
you must build and install both of those first.

On a 9.x installation, you will have clang by default, but not libc++,
so libc++ should be built and installed first, before attempting to
build the clang350-import branch.

On 8.x an earlier, you need to upgrade to at least 9.x first, follow
the previous instruction.

As for MFC'ing, I plan on merging clang 3.5.x to 10.x in a while
(roughly a month), but this will cause upgrades from 9.x to 10.x to
start requiring the build of libc++, as described above. I don't think
we can merge clang 3.5.x to 9.x, unless clang becomes the default
compiler there (but that is very unlikely).

-Dimitry

[1] svn://svn.freebsd.org/base/projects/clang350-import
[2] https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html#svn-mirrors
[3] http://pb2.nyi.freebsd.org/build.html?mastername=head-amd64-PR195480-default&build=2014-12-12_23h17m02s
[4] https://svnweb.freebsd.org/changeset/ports/374780

[Jung-uk Kim]

Post by Dimitry Andric
* The second exp-run had much better results: the failure with the
highest number of dependencies is devel/mingw32-gcc, but this
seems to be due to a problem with makeinfo, not clang. The next
highest on the list is java/openjdk6, for which ports r374780 [4]
was very recently committed.

Unfortunately, r374780 was not enough. Instead, I just turned off
"-Werror" for now (r374824).

https://svnweb.freebsd.org/changeset/ports/374824

Jung-uk Kim

[Warner Losh]

This is excellent news Dimitry!

Post by Dimitry Andric

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head. This
is quite a big update again, and any help with testing is appreciated.
To try this out, ensure you have good backups or snapshots, then build
world and kernel from the projects/clang350-import branch [1]. Please
use a Subversion mirror [2], if you are able to.

Here are some updates about the status of the 3.5.0 import.
* i386 and amd64 have been tested through make universe, and everything
should compile and run.
* Little-endian ARM builds should now compile and run, thanks to Andrew
Turner for putting in lots of work.
* Big-endian ARM is apparently supposed to work, but I'm not sure if
Andrew managed to test it on real hardware.

I know Andrew doesn’t have the right arm gear to do this test, and emulation
environments that run FreeBSD have had poor big-endian support for arm.

Post by Dimitry Andric
* PowerPC64 should mostly work, thanks to Justin Hibbits.
* PowerPC32 might start working soon; it really needs some backporting
of fixes to clang 3.4.1, which is now in head, so there is an easier
upgrade path for PowerPC users.
* Sparc64 still does not work, and I don't see any quick solutions to it
for now. It should probably stay with gcc.
* Mips will only have a chance with the upcoming clang 3.6.0, but that
is way too late for this import. It will probably require external
toolchain support to get it working.

For native builds yes. For cross builds, clang 3.6 can be built on an
x86 host.

Post by Dimitry Andric
* Another ports exp-run was done [3], after fixing the problem with
lang/gcc, which lead to many skipped dependent ports.
* The second exp-run had much better results: the failure with the
highest number of dependencies is devel/mingw32-gcc, but this seems
to be due to a problem with makeinfo, not clang. The next highest on
the list is java/openjdk6, for which ports r374780 [4] was very
recently committed.

Will users of our stable branch have code similar to the code that caused
problems? One warning flag about your upgrade to the stable branch
would be if there’s a significant number of user-written programs that
suddenly become uncompilable with the new clang using the environment
that they have today. We know of some items that are issues, so careful
attention here is needed. Unless we go proactively looking for these,
there’s a good chance we won’t find them until users hit them and start
to complain (by which point it is likely too late). Could you post a summary
of the issues that ports have hit and the fixes necessary? We may need
to have that in the release notes and/or UPDATING file to help prepare
our users for the bumps and give them solutions over them.

Post by Dimitry Andric
I would really like to merge this branch to head in about a week,
pending portmgr approvall; I don't expect the base system (outside of
llvm/clang) to need any further updates.

I think there’s good reason to do this, but we should chat about the
build issues below before doing it. They are minor, but an important
detail. I’ll see if I can find a few minutes to pull the branch and send
patches.

Post by Dimitry Andric
Lastly, to clear things up about the requirements for this branch (and
* A C++11 capable "host" compiler, e.g. clang >= 3.3 or later, or gcc

Post by Dimitry Andric
= 4.8 (I'm not 100% sure if gcc 4.7 will work, reports welcome)

* A C++11 standard library, e.g. libc++, or libstdc++ from gcc >= 4.8.
So from any earlier standard 10.x or 11.x installation, you should be
good, unless you explicitly disabled clang or libc++. In that case,
you must build and install both of those first.

This is true only on i386, amd64, and arm hosts. Given that some people
do try to do weird things, tightening up how you present this will get the
word out a little better.

Post by Dimitry Andric
On a 9.x installation, you will have clang by default, but not libc++,
so libc++ should be built and installed first, before attempting to
build the clang350-import branch.

Can you make sure that the UPDATING entry you are writing for this
contains explicit instructions.

Post by Dimitry Andric
On 8.x an earlier, you need to upgrade to at least 9.x first, follow
the previous instruction.

We should remove building on 8 support then, unless there external
toolchain stuff is up to the task (e.g. build gcc 4.9, libstc++, etc).

Post by Dimitry Andric
As for MFC'ing, I plan on merging clang 3.5.x to 10.x in a while
(roughly a month), but this will cause upgrades from 9.x to 10.x to
start requiring the build of libc++, as described above. I don't think
we can merge clang 3.5.x to 9.x, unless clang becomes the default
compiler there (but that is very unlikely).

Let’s see how it goes, and what the upgrade issues wind up being
before doing this merge back. New “major” compilers on stable branches
traditionally haven’t been done, but if clang is better about being ABIly
identical to prior releases than gcc was, it might not have the same issues
associated with it.

Warner

[Dimitry Andric]

On 18 Dec 2014, at 15:47, Warner Losh <***@bsdimp.com> wrote:
...

Post by Warner Losh

Post by Dimitry Andric
* Mips will only have a chance with the upcoming clang 3.6.0, but that
is way too late for this import. It will probably require external
toolchain support to get it working.

For native builds yes. For cross builds, clang 3.6 can be built on an
x86 host.

Yes, and it could even be one of the ports, if that is easier to use.

Post by Warner Losh
* Another ports exp-run was done [3], after fixing the problem with

Post by Dimitry Andric
lang/gcc, which lead to many skipped dependent ports.
* The second exp-run had much better results: the failure with the
highest number of dependencies is devel/mingw32-gcc, but this seems
to be due to a problem with makeinfo, not clang. The next highest on
the list is java/openjdk6, for which ports r374780 [4] was very
recently committed.

Will users of our stable branch have code similar to the code that caused
problems?

I'm not sure which code you are referring to here, the openjdk6 code?
The code itself is basically fine, but for reasons unknown to me, the
port is compiled with -Werror (which is not the case for the other
openjdk ports, apparently). Since clang 3.5.0 adds a few new warnings
for shaky C++ constructions, these appear during the openjdk6 build, but
they are easily suppressed, if upstream does not fix them, or does not
care to fix them.

I already sent Jung-uk an alternative fix for openjkd6, similar to the
one used for www/squid, where warnings are suppressed based on the
COMPILER_VERSION variable provided the ports infrastructure. In my
opinion it would still be easier to just to turn off -Werror for any
third-party code, if we don't feel like modifying it (with all the risks
involved).

Post by Warner Losh
One warning flag about your upgrade to the stable branch
would be if there’s a significant number of user-written programs that
suddenly become uncompilable with the new clang using the environment
that they have today. We know of some items that are issues, so careful
attention here is needed. Unless we go proactively looking for these,
there’s a good chance we won’t find them until users hit them and start
to complain (by which point it is likely too late). Could you post a summary
of the issues that ports have hit and the fixes necessary? We may need
to have that in the release notes and/or UPDATING file to help prepare
our users for the bumps and give them solutions over them.

The base system is already completely free of warnings, as far as I know
of, so no action is needed there. For ports, the number of failures
introduced by new warnings are quite small, as far as I can see, and
mostly for ports that are compiled with -Werror.

The most encountered new warnings are, off the top of my head:

-Wabsolute-value

This warns in two cases, for both C and C++:
* When the code is trying to take the absolute value of an unsigned
quantity, which is effectively a no-op, and almost never what was
intended. The code should be fixed, if at all possible.
* When the code is trying to take the absolute value, but the called
abs() variant is of the wrong type, which may lead to truncation.
If the warning is turned off, better make sure any truncation does
not lead to unwanted side-effects.

-Wtautological-undefined-compare and
-Wundefined-bool-conversion

These warn when C++ code is trying to compare 'this' against NULL, while
'this' should never be NULL in well-defined C++ code. However, there is
some legacy (pre C++11) code out there, which actively abuses this
feature, which was less strictly defined in previous C++ versions.

Squid does this, and apparently openjdk too. The warning can be turned
off for C++98 and earlier, but compiling the code in C++11 mode might
result in unexpected behavior, for example the unreachable parts of the
program could be optimized away.

Post by Warner Losh
I would really like to merge this branch to head in about a week,

Post by Dimitry Andric
pending portmgr approvall; I don't expect the base system (outside of
llvm/clang) to need any further updates.

I think there’s good reason to do this, but we should chat about the
build issues below before doing it. They are minor, but an important
detail. I’ll see if I can find a few minutes to pull the branch and send
patches.

Post by Dimitry Andric
Lastly, to clear things up about the requirements for this branch (and
* A C++11 capable "host" compiler, e.g. clang >= 3.3 or later, or gcc

Post by Dimitry Andric
= 4.8 (I'm not 100% sure if gcc 4.7 will work, reports welcome)

* A C++11 standard library, e.g. libc++, or libstdc++ from gcc >= 4.8.
So from any earlier standard 10.x or 11.x installation, you should be
good, unless you explicitly disabled clang or libc++. In that case,
you must build and install both of those first.

This is true only on i386, amd64, and arm hosts. Given that some people
do try to do weird things, tightening up how you present this will get the
word out a little better.

Post by Dimitry Andric
On a 9.x installation, you will have clang by default, but not libc++,
so libc++ should be built and installed first, before attempting to
build the clang350-import branch.

Can you make sure that the UPDATING entry you are writing for this
contains explicit instructions.

I'm quite bad at writing UPDATING entries, so any help there is much
appreciated. :-)

Post by Warner Losh
On 8.x an earlier, you need to upgrade to at least 9.x first, follow

Post by Dimitry Andric
the previous instruction.

We should remove building on 8 support then, unless there external
toolchain stuff is up to the task (e.g. build gcc 4.9, libstc++, etc).

The problem with 8.x is that it still has the old binutils 2.15, and
neither clang nor libc++. It would really require some externally
supplied parts. Maybe this could be done with ports, but I'm not sure
how long ports still supports the 8.x branch?

Post by Warner Losh

Post by Dimitry Andric
As for MFC'ing, I plan on merging clang 3.5.x to 10.x in a while
(roughly a month), but this will cause upgrades from 9.x to 10.x to
start requiring the build of libc++, as described above. I don't think
we can merge clang 3.5.x to 9.x, unless clang becomes the default
compiler there (but that is very unlikely).

Let’s see how it goes, and what the upgrade issues wind up being
before doing this merge back. New “major” compilers on stable branches
traditionally haven’t been done, but if clang is better about being ABIly
identical to prior releases than gcc was, it might not have the same issues
associated with it.

We don't really use llvm or clang's own ABI for anything at the moment,
just the resulting compiler executable, which is actually one big binary
(and it is even statically linked, by default).

The code output by clang 3.4.1 or 3.5.0 is not different in any "ABI"
sense of the word. Of course it will be different in absolute sense,
since optimizations were improved, and so on.

The only real issue is how to bootstrap the compiler itself, since it
requires working C++11 support. In 10.x, we provide that by default,
but not in earlier releases.

-Dimitry

[Warner Losh]

Post by Dimitry Andric
...

Post by Warner Losh

Post by Dimitry Andric
* Mips will only have a chance with the upcoming clang 3.6.0, but that
is way too late for this import. It will probably require external
toolchain support to get it working.

For native builds yes. For cross builds, clang 3.6 can be built on an
x86 host.

Yes, and it could even be one of the ports, if that is easier to use.

Post by Warner Losh
* Another ports exp-run was done [3], after fixing the problem with

Post by Dimitry Andric
lang/gcc, which lead to many skipped dependent ports.
* The second exp-run had much better results: the failure with the
highest number of dependencies is devel/mingw32-gcc, but this seems
to be due to a problem with makeinfo, not clang. The next highest on
the list is java/openjdk6, for which ports r374780 [4] was very
recently committed.

Will users of our stable branch have code similar to the code that caused
problems?

I'm not sure which code you are referring to here, the openjdk6 code?
The code itself is basically fine, but for reasons unknown to me, the
port is compiled with -Werror (which is not the case for the other
openjdk ports, apparently). Since clang 3.5.0 adds a few new warnings
for shaky C++ constructions, these appear during the openjdk6 build, but
they are easily suppressed, if upstream does not fix them, or does not
care to fix them.

I meant “similar code to what’s causing problems” with the build run in their
code they build on FreeBSD. If it is a few new warnings for obscure things,
we can advice to the release notes about what to avoid and how to mitigate
things.

Post by Dimitry Andric
I already sent Jung-uk an alternative fix for openjkd6, similar to the
one used for www/squid, where warnings are suppressed based on the
COMPILER_VERSION variable provided the ports infrastructure. In my
opinion it would still be easier to just to turn off -Werror for any
third-party code, if we don't feel like modifying it (with all the risks
involved).

Yea, we can sort out the code in src and ports. I’m more worried about
what to tell our users that may be compiling their own code that we don’t
control. If these new warnings are ubiquitous, then that could be a problem
for adoption (since many shops mandate -Werror as much as possible, and
to comply with that mandate would require additional resources when trying
to upgrade). If there are a few, then we could just document them and move on.

Post by Dimitry Andric

Post by Warner Losh
One warning flag about your upgrade to the stable branch
would be if there’s a significant number of user-written programs that
suddenly become uncompilable with the new clang using the environment
that they have today. We know of some items that are issues, so careful
attention here is needed. Unless we go proactively looking for these,
there’s a good chance we won’t find them until users hit them and start
to complain (by which point it is likely too late). Could you post a summary
of the issues that ports have hit and the fixes necessary? We may need
to have that in the release notes and/or UPDATING file to help prepare
our users for the bumps and give them solutions over them.

The base system is already completely free of warnings, as far as I know
of, so no action is needed there. For ports, the number of failures
introduced by new warnings are quite small, as far as I can see, and
mostly for ports that are compiled with -Werror.

Yea, I wasn’t too worried about this aspect of things.

Post by Dimitry Andric
-Wabsolute-value
* When the code is trying to take the absolute value of an unsigned
quantity, which is effectively a no-op, and almost never what was
intended. The code should be fixed, if at all possible.
* When the code is trying to take the absolute value, but the called
abs() variant is of the wrong type, which may lead to truncation.
If the warning is turned off, better make sure any truncation does
not lead to unwanted side-effects.
-Wtautological-undefined-compare and
-Wundefined-bool-conversion
These warn when C++ code is trying to compare 'this' against NULL, while
'this' should never be NULL in well-defined C++ code. However, there is
some legacy (pre C++11) code out there, which actively abuses this
feature, which was less strictly defined in previous C++ versions.
Squid does this, and apparently openjdk too. The warning can be turned
off for C++98 and earlier, but compiling the code in C++11 mode might
result in unexpected behavior, for example the unreachable parts of the
program could be optimized away.

This is the kind of information I was talking about. Do we have a process
to make sure this gets into the release notes?

Post by Dimitry Andric

Post by Warner Losh
I would really like to merge this branch to head in about a week,

Post by Dimitry Andric
pending portmgr approvall; I don't expect the base system (outside of
llvm/clang) to need any further updates.

I think there’s good reason to do this, but we should chat about the
build issues below before doing it. They are minor, but an important
detail. I’ll see if I can find a few minutes to pull the branch and send
patches.

Post by Dimitry Andric
Lastly, to clear things up about the requirements for this branch (and
* A C++11 capable "host" compiler, e.g. clang >= 3.3 or later, or gcc

Post by Dimitry Andric
= 4.8 (I'm not 100% sure if gcc 4.7 will work, reports welcome)

* A C++11 standard library, e.g. libc++, or libstdc++ from gcc >= 4.8.
So from any earlier standard 10.x or 11.x installation, you should be
good, unless you explicitly disabled clang or libc++. In that case,
you must build and install both of those first.

This is true only on i386, amd64, and arm hosts. Given that some people
do try to do weird things, tightening up how you present this will get the
word out a little better.

Post by Dimitry Andric
On a 9.x installation, you will have clang by default, but not libc++,
so libc++ should be built and installed first, before attempting to
build the clang350-import branch.

Can you make sure that the UPDATING entry you are writing for this
contains explicit instructions.

I'm quite bad at writing UPDATING entries, so any help there is much
appreciated. :-)

I’m happy to help with this, but I work best from a rough draft ...

Post by Dimitry Andric

Post by Warner Losh
On 8.x an earlier, you need to upgrade to at least 9.x first, follow

Post by Dimitry Andric
the previous instruction.

We should remove building on 8 support then, unless there external
toolchain stuff is up to the task (e.g. build gcc 4.9, libstc++, etc).

The problem with 8.x is that it still has the old binutils 2.15, and
neither clang nor libc++. It would really require some externally
supplied parts. Maybe this could be done with ports, but I'm not sure
how long ports still supports the 8.x branch?

If we can’t bootstrap from an 8.x system and have it work, we need to
remove support for doing that from Makefile.inc1 and friends. If we think
we can still support it with the external tool chain stuff, or with the ‘end
goal’ for the external toolchain stuff, we should leave it in. At this point,
I think it would be best to add a .warning for 8.x saying this isn’t supported
without an external toolchain, and then a month out from the branch point
we can decide what to do.

8.x is supported through the summer, iirc, on ports.

Right now, for example, we say if the host is < 8.0<mumble> then we don’t
support it. I’ll add something that says if the host is < 9<mumble> then bootstrapping
clang won’t work, but I won’t make it an error just yet and send it to you for
your branch.

Post by Dimitry Andric

Post by Warner Losh

Post by Dimitry Andric
As for MFC'ing, I plan on merging clang 3.5.x to 10.x in a while
(roughly a month), but this will cause upgrades from 9.x to 10.x to
start requiring the build of libc++, as described above. I don't think
we can merge clang 3.5.x to 9.x, unless clang becomes the default
compiler there (but that is very unlikely).

Let’s see how it goes, and what the upgrade issues wind up being
before doing this merge back. New “major” compilers on stable branches
traditionally haven’t been done, but if clang is better about being ABIly
identical to prior releases than gcc was, it might not have the same issues
associated with it.

We don't really use llvm or clang's own ABI for anything at the moment,
just the resulting compiler executable, which is actually one big binary
(and it is even statically linked, by default).

Right, this isn’t what I’m worried about...

Post by Dimitry Andric
The code output by clang 3.4.1 or 3.5.0 is not different in any "ABI"
sense of the word. Of course it will be different in absolute sense,
since optimizations were improved, and so on.

This is what I’m worried about given our past experiences with updating
compilers. If we believe there’s no issues here, we can tick that box
but it was traumatic enough the last time we tried this (admittedly with
gcc) that I have to ask. Similar representations were made, though
it turned out people forgot to ask “including C++” until people noticed
that it failed in some cases and started asking


Post by Dimitry Andric
The only real issue is how to bootstrap the compiler itself, since it
requires working C++11 support. In 10.x, we provide that by default,
but not in earlier releases.

Yea, that’s something we can document with release notes.

Cool! Thanks for taking the time to address my concerns.

Warner

[John-Mark Gurney]

Post by Warner Losh
This is excellent news Dimitry!

Post by Dimitry Andric

Post by Dimitry Andric
We're working on updating llvm, clang and lldb to 3.5.0 in head. This
is quite a big update again, and any help with testing is appreciated.
To try this out, ensure you have good backups or snapshots, then build
world and kernel from the projects/clang350-import branch [1]. Please
use a Subversion mirror [2], if you are able to.

Here are some updates about the status of the 3.5.0 import.
* i386 and amd64 have been tested through make universe, and everything
should compile and run.
* Little-endian ARM builds should now compile and run, thanks to Andrew
Turner for putting in lots of work.
* Big-endian ARM is apparently supposed to work, but I'm not sure if
Andrew managed to test it on real hardware.

I know Andrew doesn???t have the right arm gear to do this test, and emulation
environments that run FreeBSD have had poor big-endian support for arm.

I have a board that I plan to test on shortly... If Andrew would like,
I know Jim Thompson has a standing offer to send board(s) to people who
will test it...

He provided me w/ the board I will be testing on soon...

[John-Mark Gurney]

Post by Dimitry Andric
* Big-endian ARM is apparently supposed to work, but I'm not sure if
Andrew managed to test it on real hardware.

hmmm... I can't get it to compile... Maybe I'm missing something... I
tried to do:
# make buildworld TARGET_ARCH=armeb WITH_BOOTSTRAP_CLANG= WITH_CLANG= WITHOUT_GCC= WITHOUT_BOOTSTRAP_GCC=

This is from an amd64 host, though it is a month or two out of date...

But it ended w/:
c++ -O -pipe -I/a/src/usr.bin/clang/clang/../../../contrib/llvm/include -I/a/src/usr.bin/clang/clang/../../../contrib/llvm/tools/clang/include -I/a/src/usr.bin/clang/clang/../../../contrib/llvm/tools/clang/tools/driver -I. -I/a/src/usr.bin/clang/clang/../../../contrib/llvm/../../lib/clang/include -DLLVM_ON_UNIX -DLLVM_ON_FREEBSD -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -fno-strict-aliasing -DLLVM_DEFAULT_TARGET_TRIPLE=\"armeb-gnueabi-freebsd11.0\" -DLLVM_HOST_TRIPLE=\"armeb-unknown-freebsd11.0\" -DDEFAULT_SYSROOT=\"\" -fno-exceptions -fno-rtti -static -o clang cc1_main.o cc1as_main.o driver.o /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangfrontendtool/libclangfrontendtool.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangfrontend/libclangfrontend.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangdriver/libclangdriver.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangserializati
on/libclangserialization.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangcodegen/libclangcodegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangparse/libclangparse.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangsema/libclangsema.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclanganalysis/libclanganalysis.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangedit/libclangedit.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangast/libclangast.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclangbasic/libclangbasic.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libclanglex/libclanglex.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmoption/libllvmoption.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmlinker/libllvmlinker.a /usr/obj/arm.armeb/a/src/usr.bin
/clang/clang/../../../lib/clang/libllvmirreader/libllvmirreader.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmipo/libllvmipo.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmvectorize/libllvmvectorize.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvminstrumentation/libllvminstrumentation.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmbitwriter/libllvmbitwriter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmbitreader/libllvmbitreader.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmasmparser/libllvmasmparser.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmarmdisassembler/libllvmarmdisassembler.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmarmcodegen/libllvmarmcodegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmarmasmparser/libllvmarmasmparser.a /usr/obj/ar
m.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmarmdesc/libllvmarmdesc.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmarminfo/libllvmarminfo.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmarminstprinter/libllvmarminstprinter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmipsdisassembler/libllvmmipsdisassembler.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmipscodegen/libllvmmipscodegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmipsasmparser/libllvmmipsasmparser.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmipsdesc/libllvmmipsdesc.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmipsinfo/libllvmmipsinfo.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmipsinstprinter/libllvmmipsinstprinter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmp
owerpccodegen/libllvmpowerpccodegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmpowerpcasmparser/libllvmpowerpcasmparser.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmpowerpcdesc/libllvmpowerpcdesc.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmpowerpcinfo/libllvmpowerpcinfo.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmpowerpcinstprinter/libllvmpowerpcinstprinter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsparcdisassembler/libllvmsparcdisassembler.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsparccodegen/libllvmsparccodegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsparcasmparser/libllvmsparcasmparser.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsparcdesc/libllvmsparcdesc.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsparcinfo/libllv
msparcinfo.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsparcinstprinter/libllvmsparcinstprinter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86disassembler/libllvmx86disassembler.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86asmparser/libllvmx86asmparser.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86codegen/libllvmx86codegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmselectiondag/libllvmselectiondag.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmasmprinter/libllvmasmprinter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmcparser/libllvmmcparser.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmcodegen/libllvmcodegen.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmobjcarcopts/libllvmobjcarcopts.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang
/../../../lib/clang/libllvmscalaropts/libllvmscalaropts.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvminstcombine/libllvminstcombine.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmtransformutils/libllvmtransformutils.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmipa/libllvmipa.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmanalysis/libllvmanalysis.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86desc/libllvmx86desc.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86info/libllvmx86info.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmtarget/libllvmtarget.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86instprinter/libllvmx86instprinter.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmmc/libllvmmc.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib
/clang/libllvmobject/libllvmobject.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmx86utils/libllvmx86utils.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmcore/libllvmcore.a /usr/obj/arm.armeb/a/src/usr.bin/clang/clang/../../../lib/clang/libllvmsupport/libllvmsupport.a -lncursesw
/usr/obj/arm.armeb/a/src/tmp/usr/lib/crt1.o: In function `__start':
crt1.c:(.text+0xb4): relocation truncated to fit: R_ARM_CALL against symbol `atexit' defined in .text section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libc.a(atexit.o)
crt1.c:(.text+0xbc): relocation truncated to fit: R_ARM_CALL against symbol `_init_tls' defined in .text section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libc.a(tls.o)
crt1.c:(.text+0xc4): relocation truncated to fit: R_ARM_CALL against symbol `atexit' defined in .text section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libc.a(atexit.o)
crt1.c:(.text+0x164): relocation truncated to fit: R_ARM_CALL against symbol `exit' defined in .text section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libc.a(exit.o)
/usr/obj/arm.armeb/a/src/tmp/usr/lib/crt1.o: In function `finalizer':
crt1.c:(.text+0x1d4): relocation truncated to fit: R_ARM_CALL against symbol `_fini' defined in .fini section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/crti.o
cc1_main.o: In function `__static_initialization_and_destruction_0(int, int)':
cc1_main.cpp:(.text+0xdc): relocation truncated to fit: R_ARM_CALL against symbol `getenv' defined in .text section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libc.a(getenv.o)
cc1_main.cpp:(.text+0x2c4): relocation truncated to fit: R_ARM_CALL against symbol `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)' defined in .text._ZNSsC1EPKcRKSaIcE[_ZNSsC1EPKcRKSaIcE] section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libstdc++.a(string-inst.o)
cc1_main.cpp:(.text+0x374): relocation truncated to fit: R_ARM_JUMP24 against symbol `__gnu_cxx::__exchange_and_add(int volatile*, int)' defined in .text._ZN9__gnu_cxx18__exchange_and_addEPVii section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libstdc++.a(atomicity.o)
cc1_main.cpp:(.text+0x388): relocation truncated to fit: R_ARM_JUMP24 against symbol `std::string::_Rep::_M_destroy(std::allocator<char> const&)' defined in .text._ZNSs4_Rep10_M_destroyERKSaIcE[_ZNSs4_Rep10_M_destroyERKSaIcE] section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libstdc++.a(string-inst.o)
cc1_main.cpp:(.text+0x3a0): relocation truncated to fit: R_ARM_CALL against symbol `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)' defined in .text._ZNSsC1EPKcRKSaIcE[_ZNSsC1EPKcRKSaIcE] section in /usr/obj/arm.armeb/a/src/tmp/usr/lib/libstdc++.a(string-inst.o)
cc1_main.cpp:(.text+0x44c): additional relocation overflows omitted from the output
*** Error code 1

Stop.
make[5]: stopped in /a/src/usr.bin/clang/clang
*** Error code 1

Stop.
make[4]: stopped in /a/src/usr.bin/clang
*** Error code 1

Stop.
make[3]: stopped in /a/src/usr.bin
*** Error code 1

Stop.
make[2]: stopped in /a/src
*** Error code 1

Stop.
make[1]: stopped in /a/src
*** Error code 1

Stop.
make: stopped in /a/src

[NGie Cooper]

Hi,

...

Hi Dimitry,
As a request to speed up the build process further,
- Would it be [easily] possible in the clang35 branch to bootstrap
the compiler for a specific architecture? The bootstrap / cross
compiler for instance always builds N targets instead of building just
the desired TARGET/TARGET_ARCH combo.
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?
I made a lot of progress on my faster-build branch (
https://github.com/yaneurabeya/freebsd/tree/faster-build ), but got
mired down in the minutiae of how this needs to be implemented (it
worked up until I ran make tinderbox, of course :)..), and had to work
on other things...
Thanks!

[Dimitry Andric]

...

Post by NGie Cooper
As a request to speed up the build process further,
- Would it be [easily] possible in the clang35 branch to bootstrap
the compiler for a specific architecture? The bootstrap / cross
compiler for instance always builds N targets instead of building just
the desired TARGET/TARGET_ARCH combo.

It's not very easy, at least not without breaking various parts of our
fragile build system, but I surely want to put something like this on
the TODO list for *after* the import has completed.

The branch is making progress right now, and I would not want to
complicate matters further by introducing yet another tricky feature. :)

Post by NGie Cooper
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?

I would be fine with something like this, as long as it is turned off by
default, or if it is only used for the bootstrap stages. It is actually
an extremely useful feature of clang that you can target multiple
architectures with one compiler binary.

A more interesting case would be to remodel the build system so it can
use one toolchain (external, or pkg-ng'd, maybe?) for building an entire
universe. With clang, that should be relatively easy to do.

-Dimitry

[Warner Losh]

Post by Dimitry Andric

Post by NGie Cooper
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?

I would be fine with something like this, as long as it is turned off by
default, or if it is only used for the bootstrap stages. It is actually
an extremely useful feature of clang that you can target multiple
architectures with one compiler binary.

Point of information: this seems useful for developers, and
(almost entirely) useless for everyone else. Are there other
cohorts that want this badly?
If that's correct, and there's a simple switch for
configuration ... why should this default to what's useful for the
(much?) smaller number of people? About what am I ignorant?

Only people working on a single binary of clang to build all architectures
are interested, which is a vanishingly small number. There’s little point
to build this stuff even for hard-core developers.

Warner

[Dimitry Andric]

Post by Dimitry Andric

Post by NGie Cooper
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?

I would be fine with something like this, as long as it is turned off by
default, or if it is only used for the bootstrap stages. It is actually
an extremely useful feature of clang that you can target multiple
architectures with one compiler binary.

Point of information: this seems useful for developers, and
(almost entirely) useless for everyone else. Are there other
cohorts that want this badly?
If that's correct, and there's a simple switch for
configuration ... why should this default to what's useful for the
(much?) smaller number of people? About what am I ignorant?

It's not a simple switch, at least not now. If you use the upstream
build system for llvm, e.g. autoconf or CMake, it has an option to
select all the architectures that are supported. Several config files
are then generated differently, and parts of the target support
subdirectories are selectively enabled or disabled.

In fact, we already build just a subset of the available architectures,
since FreeBSD only supports about 5 of them. We can probably arrange
for a more minimal configuration in our build system, but since the
build time saved is quite small, I don't think it makes much sense in
complicating our build system even further.

If people are really so interested in shaving off a little, for more
complication, that is fine with me. But unfortunately, I have too many
tasks on my plate right now, and I cannot work on it. Besides, doing
such a new feature now would interfere with the current branch work.

Also, after the 3.5.0 import, there are much more interesting fish to
fry, in my opinion. For example, importing newer versions of libc++ and
compiler-rt, which can bring address sanitizer support, etc.

-Dimitry

[Warner Losh]

Post by Dimitry Andric

Post by Dimitry Andric

Post by NGie Cooper
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?

I would be fine with something like this, as long as it is turned off by
default, or if it is only used for the bootstrap stages. It is actually
an extremely useful feature of clang that you can target multiple
architectures with one compiler binary.

Point of information: this seems useful for developers, and
(almost entirely) useless for everyone else. Are there other
cohorts that want this badly?
If that's correct, and there's a simple switch for
configuration ... why should this default to what's useful for the
(much?) smaller number of people? About what am I ignorant?

It's not a simple switch, at least not now. If you use the upstream
build system for llvm, e.g. autoconf or CMake, it has an option to
select all the architectures that are supported. Several config files
are then generated differently, and parts of the target support
subdirectories are selectively enabled or disabled.
In fact, we already build just a subset of the available architectures,
since FreeBSD only supports about 5 of them. We can probably arrange
for a more minimal configuration in our build system, but since the
build time saved is quite small, I don't think it makes much sense in
complicating our build system even further.
If people are really so interested in shaving off a little, for more
complication, that is fine with me. But unfortunately, I have too many
tasks on my plate right now, and I cannot work on it. Besides, doing
such a new feature now would interfere with the current branch work.

With the recent parallelism work, the is true. It might save a couple percent
off the build time. Before those changes, though, disabling all non target
arches saved about 10% of the buildworld time.

Creating a hack to do this is easy (which is how I measured it). But Dimitry
is right that creating a robust solution is hard. Even harder if you want it
to be completely clean.

Post by Dimitry Andric
Also, after the 3.5.0 import, there are much more interesting fish to
fry, in my opinion. For example, importing newer versions of libc++ and
compiler-rt, which can bring address sanitizer support, etc.

I tend to agree. IMHO, supporting the work going on to bring the
meta-mode stuff will pay far higher dividends than optimizing this
corner of the build.

Warner

[Garrett Cooper]

Post by Warner Losh
With the recent parallelism work, the is true. It might save a couple percent
off the build time. Before those changes, though, disabling all non target
arches saved about 10% of the buildworld time.

I’m curious. How much is 10% in terms of minutes and with what -j value?

Post by Warner Losh
Creating a hack to do this is easy (which is how I measured it). But Dimitry
is right that creating a robust solution is hard. Even harder if you want it
to be completely clean.

It didn’t seem incredibly hard — it just required a bit more “generated files” in clang AFAICT. I’ll hang ten until clang35 is in so I can re-asses what’s going on with building it.

Post by Warner Losh
I tend to agree. IMHO, supporting the work going on to bring the
meta-mode stuff will pay far higher dividends than optimizing this
corner of the build.

True… probably will!

[Warner Losh]

Post by Garrett Cooper

Post by Warner Losh
With the recent parallelism work, the is true. It might save a couple percent
off the build time. Before those changes, though, disabling all non target
arches saved about 10% of the buildworld time.

I’m curious. How much is 10% in terms of minutes and with what -j value?

That depends on how long the build takes. For my 20 minute builds it was about
2 minutes faster. At the time, -j didn’t really effect build times once you got north
of 4 because parallelism really sucked. Now it doesn’t suck and it scales much
better and I suspect that the time savings would be tiny because it would be done
at the same time as other things anyway, but I’ve not measured it directly.

Post by Garrett Cooper

Post by Warner Losh
Creating a hack to do this is easy (which is how I measured it). But Dimitry
is right that creating a robust solution is hard. Even harder if you want it
to be completely clean.

It didn’t seem incredibly hard — it just required a bit more “generated files” in clang AFAICT. I’ll hang ten until clang35 is in so I can re-asses what’s going on with building it.

Yea, and that file generation is a pita, or I’d have committed my
changes a while ago...

Post by Garrett Cooper

Post by Warner Losh
I tend to agree. IMHO, supporting the work going on to bring the
meta-mode stuff will pay far higher dividends than optimizing this
corner of the build.

True… probably will!

Yea, this isn’t a problem worth solving today.

Warner

[Warner Losh]

Post by Dimitry Andric
...

Post by NGie Cooper
As a request to speed up the build process further,
- Would it be [easily] possible in the clang35 branch to bootstrap
the compiler for a specific architecture? The bootstrap / cross
compiler for instance always builds N targets instead of building just
the desired TARGET/TARGET_ARCH combo.

It's not very easy, at least not without breaking various parts of our
fragile build system, but I surely want to put something like this on
the TODO list for *after* the import has completed.
The branch is making progress right now, and I would not want to
complicate matters further by introducing yet another tricky feature. :)

The build system isn’t so much the issue, but you wind up with
files that refer to all the architectures.

But this is a request for a new feature, not quite in scope for a compiler
upgrade.

Post by Dimitry Andric

Post by NGie Cooper
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?

I would be fine with something like this, as long as it is turned off by
default, or if it is only used for the bootstrap stages. It is actually
an extremely useful feature of clang that you can target multiple
architectures with one compiler binary.

This is a new feature. Various people have tried in the past to implement
it and compiling just the mips files on mips is straight forward. However,
convincing clang to not reference the other architectures requires more
sophistication than we currently have in the clang build process.

Post by Dimitry Andric
A more interesting case would be to remodel the build system so it can
use one toolchain (external, or pkg-ng'd, maybe?) for building an entire
universe. With clang, that should be relatively easy to do.

Another useful new feature. The hard part with this is getting all the fiddly
bits in the tree that depend on default CC producing proper binaries to
cooperate.. Doable, but that’s a lot of universe builds. And today it isn’t
very practical because sparc64 and mips are broken...

Warner

[Garrett Cooper]

Post by Dimitry Andric
...

Post by NGie Cooper
As a request to speed up the build process further,
- Would it be [easily] possible in the clang35 branch to bootstrap
the compiler for a specific architecture? The bootstrap / cross
compiler for instance always builds N targets instead of building just
the desired TARGET/TARGET_ARCH combo.

It's not very easy, at least not without breaking various parts of our
fragile build system, but I surely want to put something like this on
the TODO list for *after* the import has completed.
The branch is making progress right now, and I would not want to
complicate matters further by introducing yet another tricky feature. :)

Fair enough :).

Post by Dimitry Andric

Post by NGie Cooper
- Could a "MK_CLANG_ALL_TARGETS" or something similar option be
added to src.opts.mk to fine tune this process for those of us who
don't want to build a cross-compile toolchain every iteration for our
target MACHINE/MACHINE_ARCH?

I would be fine with something like this, as long as it is turned off by
default, or if it is only used for the bootstrap stages. It is actually
an extremely useful feature of clang that you can target multiple
architectures with one compiler binary.

Yes. If make tinderbox could use this it would be useful, otherwise, for most folks it seems like a less interesting feature.

Post by Dimitry Andric
A more interesting case would be to remodel the build system so it can
use one toolchain (external, or pkg-ng'd, maybe?) for building an entire
universe. With clang, that should be relatively easy to do.

Agreed. bdrewery is working on something similar to that internally for Isilon. Building the same toolchain N times internally when building the system and your upstream revision of FreeBSD doesn’t change is like testing your sanity — not much changes with the bootstrap compiler/toolchain then!

Thanks for the reply :)!