Discussion:
Discussion about merging Go frontend
Ian Lance Taylor
2010-10-24 05:40:46 UTC
Permalink
The Go frontend was approved for inclusion with gcc by the steering
committee a while back: http://gcc.gnu.org/ml/gcc/2010-01/msg00500.html .

Assuming it is OK with the release managers, I would like to get it into
the gcc 4.6 release. I have been running behind my intended schedule,
for which I apologize. However, the code is almost ready to merge, in
the sense that the only remaining changes are Go specific and should not
affect other languages. The only thing I hope to clean up further
before the merge is additional separation between the Go frontend proper
and the gcc-specific interface. I'm not going to have time to do the
full planned separation, which I will continue to work on, but I hope to
have the proper framework in place for future work.

There are three new source code directories: gcc/go, libgo, and elfcpp.
The last is currently part of the src repository, where it is used by
gold. I propose moving the master copy of elfcpp to gcc, and handling
it like libiberty.

There are three new testsuite directories: gcc/testsuite/go.dg,
gcc/testsuite/go.go-torture, gcc/testsuite/go.test. The last is mostly
a copy of the testsuite for the gc compiler.

The following other files are changed:

gcc/opts.c
gcc/toplev.c
gcc/debug.h
gcc/flag-types.h
gcc/c-family/c-lex.c
gcc/Makefile.in
Add support for -ggo, used when building runtime library.

gcc/common.opt
Add -ggo and also -static-libgo.

gcc/gcc.c
Recognize .go files.

gcc/configure.ac
Check whether -static-libstdc++ is supported.

gcc/testsuite/lib/go-dg.exp
gcc/testsuite/lib/go.exp
gcc/testsuite/lib/go-torture.exp
Go testsuite support.

gcc/testsuite/lib/target-supports.exp
Recognize Go code.

configure.ac
Add libgo. If building Go, build C++ as a boot language. If building
libgo, build libffi.

Makefile.def
Makefile.tpl
Add libgo and Go environment variables.

config-ml.in
Pass down some Go environment variables.

libtool.m4
ltmain.sh
Add Go support; patch sent upstream.

gcc/acinclude.m4
config/go.m4
Add Go support; patch sent upstream.

configure
Makefile.in
gcc/configure
Regenerate.

Ian
Joseph S. Myers
2010-10-24 14:58:55 UTC
Permalink
Post by Ian Lance Taylor
affect other languages. The only thing I hope to clean up further
before the merge is additional separation between the Go frontend proper
and the gcc-specific interface. I'm not going to have time to do the
full planned separation, which I will continue to work on, but I hope to
have the proper framework in place for future work.
Will the front end use its own text domain for i18n of messages, or the
"gcc" domain? Whatever the answer, do you need changes to po/exgettext or
similar to cause all the messages to be properly extracted?
Post by Ian Lance Taylor
There are three new source code directories: gcc/go, libgo, and elfcpp.
The last is currently part of the src repository, where it is used by
gold. I propose moving the master copy of elfcpp to gcc, and handling
it like libiberty.
How is elfcpp used? Is this front end in some way restricted to ELF
targets, or more functional with them? (In general, to what extent are
the front end and library portable to different targets - how much work,
if any, is needed to get them building and working for a new target
(architecture or OS)? Would the sort of target changes needed be risky
for Stage 3? What about host portability?)
You don't mention documentation. Does the front end have its own manual?
Whether or not it does, the new-front-end checklist (Front End in
sourcebuild.texi) lists places in the documentation that need updating for
a new front end (install.texi, contrib.texi, frontends.texi,
standards.texi, invoke.texi, sourcebuild.texi - plus web page etc. updates
for merging to trunk).
Post by Ian Lance Taylor
gcc/opts.c
gcc/toplev.c
gcc/debug.h
gcc/flag-types.h
gcc/c-family/c-lex.c
gcc/Makefile.in
Add support for -ggo, used when building runtime library.
Note that all -g* options are also handled in a case statement in
java/jvspec.c, though that might well better be handled through specs, and
I don't know if -ggo makes sense at all for Java.

I think appropriate coding standards documentation for this front end is
needed. Possibly in the form of a description of how the coding standards
differ from the general conventions for C++ in existing parts of GCC at
<http://gcc.gnu.org/wiki/CppConventions>, though those would need merging
into codingconventions.html before being used as a basis for conventions
actually used on GCC trunk.

The front end's langhooks don't seem to have been updated for the
separation of the init_options and init_options_struct hooks.

I presume you will be posting the front end and other changes for review
(or, I suppose, given the size of the changes, explicitly stating that you
propose for review the diffs between particular revisions of trunk and a
branch given by a particular svn diff command) as indicated in
<http://gcc.gnu.org/ml/gcc/2010-01/msg00504.html>.
--
Joseph S. Myers
***@codesourcery.com
Ian Lance Taylor
2010-11-02 21:39:00 UTC
Permalink
Post by Joseph S. Myers
Post by Ian Lance Taylor
affect other languages. The only thing I hope to clean up further
before the merge is additional separation between the Go frontend proper
and the gcc-specific interface. I'm not going to have time to do the
full planned separation, which I will continue to work on, but I hope to
have the proper framework in place for future work.
Will the front end use its own text domain for i18n of messages, or the
"gcc" domain? Whatever the answer, do you need changes to po/exgettext or
similar to cause all the messages to be properly extracted?
Good questions but I simply don't know. If anybody has any advice I
would be happy to hear it.

The current Go frontend source code marks text strings which require
localization just as gcc code does: with _(), except when directly
calling error or similar functions. I suppose my inclination is to
simply continue doing that, and leaving the details of localization out
of the frontend. In which case the front end will not use its own text
domain for internationalization. It seems to me that this is a decision
which can be changed later with only mild stress.

Looking at po/exgettext, a script whose existence I had not previously
suspected, it appears that it will need some minor changes to look at
C++ files. I will tackle that at some point after the merge.
Post by Joseph S. Myers
Post by Ian Lance Taylor
There are three new source code directories: gcc/go, libgo, and elfcpp.
The last is currently part of the src repository, where it is used by
gold. I propose moving the master copy of elfcpp to gcc, and handling
it like libiberty.
How is elfcpp used? Is this front end in some way restricted to ELF
targets, or more functional with them? (In general, to what extent are
the front end and library portable to different targets - how much work,
if any, is needed to get them building and working for a new target
(architecture or OS)? Would the sort of target changes needed be risky
for Stage 3? What about host portability?)
The simple_object work I committed this morning means that elfcpp is no
longer required.

The front end should be portable to new hosts and targets; I don't know
of any issues there (well, I know of one minor issue with a Mach-O
target which I intend to fix). The library is not too hard to port--it
has been ported to RTEMS already, though I'm sure a Windows port would
have some difficulties. The library does currently use the pthreads
interface directly; I don't know how hard it would be to change to use
gcc's gthread interface.
Post by Joseph S. Myers
You don't mention documentation. Does the front end have its own manual?
Whether or not it does, the new-front-end checklist (Front End in
sourcebuild.texi) lists places in the documentation that need updating for
a new front end (install.texi, contrib.texi, frontends.texi,
standards.texi, invoke.texi, sourcebuild.texi - plus web page etc. updates
for merging to trunk).
Good point. I have not dealt with doc changes yet.
Post by Joseph S. Myers
Post by Ian Lance Taylor
gcc/opts.c
gcc/toplev.c
gcc/debug.h
gcc/flag-types.h
gcc/c-family/c-lex.c
gcc/Makefile.in
Add support for -ggo, used when building runtime library.
Note that all -g* options are also handled in a case statement in
java/jvspec.c, though that might well better be handled through specs, and
I don't know if -ggo makes sense at all for Java.
Thanks. I doubt -ggo would ever make sense for Java but that seems easy
enough to handle.
Post by Joseph S. Myers
I think appropriate coding standards documentation for this front end is
needed. Possibly in the form of a description of how the coding standards
differ from the general conventions for C++ in existing parts of GCC at
<http://gcc.gnu.org/wiki/CppConventions>, though those would need merging
into codingconventions.html before being used as a basis for conventions
actually used on GCC trunk.
I think the conventions I used are the same as the ones in
CppConventions, but, yes, point taken.
Post by Joseph S. Myers
The front end's langhooks don't seem to have been updated for the
separation of the init_options and init_options_struct hooks.
Thanks, will fix.
Post by Joseph S. Myers
I presume you will be posting the front end and other changes for review
(or, I suppose, given the size of the changes, explicitly stating that you
propose for review the diffs between particular revisions of trunk and a
branch given by a particular svn diff command) as indicated in
<http://gcc.gnu.org/ml/gcc/2010-01/msg00504.html>.
I will post all changes outside of the gcc/go and libgo directories for
review, certainly. I would also appreciate review for the code in
gcc/go and libgo on the branch. Whether review is required for that
code is an interesting question. I have been assuming that I would
become the initial maintainer for the gcc/go and libgo directories. As
such, I would be permitted to make changes to those directories without
review. So is review required for the initial checkin? I don't know.

Ian
Joseph S. Myers
2010-11-03 21:06:57 UTC
Permalink
Post by Ian Lance Taylor
Post by Joseph S. Myers
I presume you will be posting the front end and other changes for review
(or, I suppose, given the size of the changes, explicitly stating that you
propose for review the diffs between particular revisions of trunk and a
branch given by a particular svn diff command) as indicated in
<http://gcc.gnu.org/ml/gcc/2010-01/msg00504.html>.
I will post all changes outside of the gcc/go and libgo directories for
review, certainly. I would also appreciate review for the code in
gcc/go and libgo on the branch. Whether review is required for that
code is an interesting question. I have been assuming that I would
become the initial maintainer for the gcc/go and libgo directories. As
such, I would be permitted to make changes to those directories without
review. So is review required for the initial checkin? I don't know.
I think new front ends should be reviewed for general style, coding
conventions, use of deprecated interfaces, unportabilities etc., just as
new back ends should be reviewed.
--
Joseph S. Myers
***@codesourcery.com
Ian Lance Taylor
2010-11-03 21:15:53 UTC
Permalink
Post by Joseph S. Myers
I think new front ends should be reviewed for general style, coding
conventions, use of deprecated interfaces, unportabilities etc., just as
new back ends should be reviewed.
Would anybody care to volunteer to review the Go frontend on these
grounds? The code can be found on the gccgo branch in the gcc/go
subdirectory.

There is also the libgo directory. The contents of libgo/go are a copy
of the standard Go library and I don't think a review of that would be
useful. But it would be reasonable to review the contents of libgo
outside of libgo/go. This code can found on the gccgo branch in the
libgo subdirectory.

And there is the testsuite, which I suppose I will post separately.

Ian
Joseph S. Myers
2010-11-06 22:41:17 UTC
Permalink
Post by Ian Lance Taylor
Post by Joseph S. Myers
I think new front ends should be reviewed for general style, coding
conventions, use of deprecated interfaces, unportabilities etc., just as
new back ends should be reviewed.
Would anybody care to volunteer to review the Go frontend on these
grounds? The code can be found on the gccgo branch in the gcc/go
subdirectory.
I have now posted reviews of this code for various issues, though I think
it could do with more eyes on it, from front-end maintainers, build-system
maintainers and others. I did not go exhaustively through all the code in
gofrontend/.

http://gcc.gnu.org/ml/gcc-patches/2010-11/msg00640.html
http://gcc.gnu.org/ml/gcc-patches/2010-11/msg00654.html
http://gcc.gnu.org/ml/gcc-patches/2010-11/msg00656.html
Post by Ian Lance Taylor
There is also the libgo directory. The contents of libgo/go are a copy
of the standard Go library and I don't think a review of that would be
useful. But it would be reasonable to review the contents of libgo
outside of libgo/go. This code can found on the gccgo branch in the
libgo subdirectory.
I don't propose to review this. A build-system maintainer might be most
useful there.
--
Joseph S. Myers
***@codesourcery.com
Ralf Wildenhues
2010-11-13 07:46:30 UTC
Permalink
Hello,
Post by Joseph S. Myers
Post by Ian Lance Taylor
There is also the libgo directory. The contents of libgo/go are a copy
of the standard Go library and I don't think a review of that would be
useful. But it would be reasonable to review the contents of libgo
outside of libgo/go. This code can found on the gccgo branch in the
libgo subdirectory.
I don't propose to review this. A build-system maintainer might be most
useful there.
I casually looked at libgo/{Makefile.am,configure.ac}.

You don't need to list .c files in the BUILT_SOURCES variable if they
are also listed in some <prog>_SOURCES variable (and are not
undocumented prerequisites for other targets). make will know that it
needs to create these files. (Omitting BUILT_SOURCES saves a recursive
make invocation in practice.)
Post by Joseph S. Myers
if LIBGO_IS_386
GOARCH = 386
else
if LIBGO_IS_X86_64
GOARCH = amd64
else
if LIBGO_IS_ARM
GOARCH = arm
else
GOARCH = unknown
endif
endif
endif
I wouldn't write Automake conditionals deeply nested, it makes for long
lines and quadratic number of characters in Makefile.in when the number
of conditionals increases. But who am I telling that. ;-)
You could just AC_SUBST([GOARCH]) and GOOS from configure.ac; but this
is a really minor stylistic issue.

Several %/check targets use 'mkdir -p'. While that is quite portable
nowadays, there are still mkdir implementations that are racy when run
in parallel. You can use $(MKDIR_P) instead.

I suppose portability to systems where OBJEXT is .obj is not a question
at this point.

configure.ac didn't really show up much at a glance.

Cheers,
Ralf
Ian Lance Taylor
2010-11-18 00:17:41 UTC
Permalink
Post by Ralf Wildenhues
I casually looked at libgo/{Makefile.am,configure.ac}.
Thanks for the review.
Post by Ralf Wildenhues
You don't need to list .c files in the BUILT_SOURCES variable if they
are also listed in some <prog>_SOURCES variable (and are not
undocumented prerequisites for other targets). make will know that it
needs to create these files. (Omitting BUILT_SOURCES saves a recursive
make invocation in practice.)
Fixed.
Post by Ralf Wildenhues
Post by Joseph S. Myers
if LIBGO_IS_386
GOARCH = 386
else
if LIBGO_IS_X86_64
GOARCH = amd64
else
if LIBGO_IS_ARM
GOARCH = arm
else
GOARCH = unknown
endif
endif
endif
I wouldn't write Automake conditionals deeply nested, it makes for long
lines and quadratic number of characters in Makefile.in when the number
of conditionals increases. But who am I telling that. ;-)
You could just AC_SUBST([GOARCH]) and GOOS from configure.ac; but this
is a really minor stylistic issue.
It's ugly but it puts the value where it is used, rather than splitting
it between configure.ac and Makefile.am. This makes it easier to
understand what is happening.

If only the person who implemented automake conditionals had added elif.
What a maroon.
Post by Ralf Wildenhues
Several %/check targets use 'mkdir -p'. While that is quite portable
nowadays, there are still mkdir implementations that are racy when run
in parallel. You can use $(MKDIR_P) instead.
Fixed.
Post by Ralf Wildenhues
I suppose portability to systems where OBJEXT is .obj is not a question
at this point.
Not really, but I changed .o to .$(OBJEXT) anyhow.

All patches committed to gccgo branch. Thanks again for the review.

Ian

Dave Korn
2010-10-24 15:38:42 UTC
Permalink
Post by Ian Lance Taylor
There are three new source code directories: gcc/go, libgo, and elfcpp.
The last is currently part of the src repository, where it is used by
gold. I propose moving the master copy of elfcpp to gcc, and handling
it like libiberty.
What about non-ELF platforms? Will go not be buildable for them?

This is the second ELF-specific object reader library getting dragged into
the compiler build. Maybe we should give some consideration to just merging
the whole of src/ and gcc/ and be done with it, so that all the non-ELF
targets can get BFD support?

cheers,
DaveK
Ian Lance Taylor
2010-10-25 02:52:40 UTC
Permalink
Post by Dave Korn
Post by Ian Lance Taylor
There are three new source code directories: gcc/go, libgo, and elfcpp.
The last is currently part of the src repository, where it is used by
gold. I propose moving the master copy of elfcpp to gcc, and handling
it like libiberty.
What about non-ELF platforms? Will go not be buildable for them?
At present, gccgo is most conveniently used on ELF platforms. The issue
is how to handle importing a package. All Go code lives in a package.
When you compile a package, gccgo puts export data into the object file,
in the .go_export section. This is done using the usual get_section
function. Support for named sections is required at present, but
nothing else.

However, when you import a package, gccgo needs to locate that data. In
gccgo I used the elfcpp library to read the data out of an ELF file.
Gccgo can also read the data out of a plain text file, so it can work
with a non-ELF format although some other mechanism, such as objcopy,
would be needed to extract the data from the .go_export section into a
text file.

Of course it would be best if gccgo simply learned how to read other
object file formats. It would not be particularly difficult, but I
don't have any current plans for working on it.

The dependence on elfcpp is not particularly deep. It's confined to a
single file, import-elf.cc. On the other hand elfcpp is not very large,
4500 lines.
Post by Dave Korn
This is the second ELF-specific object reader library getting dragged into
the compiler build. Maybe we should give some consideration to just merging
the whole of src/ and gcc/ and be done with it, so that all the non-ELF
targets can get BFD support?
It's hard for me to believe that BFD is the correct answer. It's poorly
designed for the kinds of things the compiler needs to do. Any program
which links against BFD effectively links in the GNU linker.

Ian
Andrew Pinski
2010-10-25 03:20:27 UTC
Permalink
Why can't gccgo handle this like gnu objective-c runtime handles this?
Why does it need to be designed such it is in a specific section
instead of registering at runtime? Seems like this a bad design of
registering modules really.
Post by Ian Lance Taylor
Post by Dave Korn
Post by Ian Lance Taylor
There are three new source code directories: gcc/go, libgo, and elfcpp.
The last is currently part of the src repository, where it is used by
gold. I propose moving the master copy of elfcpp to gcc, and handling
it like libiberty.
What about non-ELF platforms? Will go not be buildable for them?
At present, gccgo is most conveniently used on ELF platforms. The issue
is how to handle importing a package. All Go code lives in a package.
When you compile a package, gccgo puts export data into the object file,
in the .go_export section. This is done using the usual get_section
function. Support for named sections is required at present, but
nothing else.
However, when you import a package, gccgo needs to locate that
data. In
gccgo I used the elfcpp library to read the data out of an ELF file.
Gccgo can also read the data out of a plain text file, so it can work
with a non-ELF format although some other mechanism, such as objcopy,
would be needed to extract the data from the .go_export section into a
text file.
Of course it would be best if gccgo simply learned how to read other
object file formats. It would not be particularly difficult, but I
don't have any current plans for working on it.
The dependence on elfcpp is not particularly deep. It's confined to a
single file, import-elf.cc. On the other hand elfcpp is not very large,
4500 lines.
Post by Dave Korn
This is the second ELF-specific object reader library getting dragged into
the compiler build. Maybe we should give some consideration to just merging
the whole of src/ and gcc/ and be done with it, so that all the non-
ELF
targets can get BFD support?
It's hard for me to believe that BFD is the correct answer. It's poorly
designed for the kinds of things the compiler needs to do. Any program
which links against BFD effectively links in the GNU linker.
Ian
Ian Lance Taylor
2010-10-25 03:49:13 UTC
Permalink
Post by Andrew Pinski
Why can't gccgo handle this like gnu objective-c runtime handles this?
Why does it need to be designed such it is in a specific section
instead of registering at runtime? Seems like this a bad design of
registering modules really.
This is data generated by the compiler when it compiles a package and
read by the compiler when it imports a package.

There is nothing happening at runtime here.

Ian
Mark Mitchell
2010-10-25 15:15:42 UTC
Permalink
Post by Ian Lance Taylor
It's hard for me to believe that BFD is the correct answer. It's poorly
designed for the kinds of things the compiler needs to do. Any program
which links against BFD effectively links in the GNU linker.
It sounded from your mail like all the compiler needs to do is to read
the binary contents of a named section. Isn't that something that BFD
does well?
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Ian Lance Taylor
2010-10-25 18:15:45 UTC
Permalink
Post by Mark Mitchell
Post by Ian Lance Taylor
It's hard for me to believe that BFD is the correct answer. It's poorly
designed for the kinds of things the compiler needs to do. Any program
which links against BFD effectively links in the GNU linker.
It sounded from your mail like all the compiler needs to do is to read
the binary contents of a named section. Isn't that something that BFD
does well?
BFD will get the job done. But I don't think it's a good choice for
releasing a program like gcc.

BFD is in effect an internal library for the linker and the GNU
binutils, and it's also used by the assembler. It doesn't really
maintain source compatibility across releases, and it definitely doesn't
maintain binary compatibility. As I mentioned above, when you link
against BFD you effectively pull in the linker.

Reading a section from an object file is not hard. Linking against BFD
to do it is massive overkill. If we were already linking against BFD,
then sure. But introducing BFD for this will give us and the binutils
developers some long-term maintenance pain for limited reward.

At least, that is how I see it.

Ian
Andrew Pinski
2010-10-25 18:35:40 UTC
Permalink
Post by Ian Lance Taylor
At least, that is how I see it.
Why not require libelf just like for LTO? That seems like a time to
reduce what we depend on. For an example if we compile with lto and
go, GCC will use two different elf libraries. This seems dumb really.

Thanks,
Andrew Pinski
Andi Kleen
2010-10-25 18:43:02 UTC
Permalink
Post by Andrew Pinski
Post by Ian Lance Taylor
At least, that is how I see it.
Why not require libelf just like for LTO? That seems like a time to
reduce what we depend on. For an example if we compile with lto and
go, GCC will use two different elf libraries. This seems dumb really.
libelf is rather awkward and has different implementations with
different bugs and also usually needs to be installed explicitely on
Linux.

It would be better to make LTO use Ian's library (but then it's C++ I
believe, not C)

-Andi
--
***@linux.intel.com -- Speaking for myself only.
Dave Korn
2010-10-25 23:01:50 UTC
Permalink
Post by Andi Kleen
Post by Andrew Pinski
Post by Ian Lance Taylor
At least, that is how I see it.
Why not require libelf just like for LTO? That seems like a time to
reduce what we depend on. For an example if we compile with lto and
go, GCC will use two different elf libraries. This seems dumb really.
libelf is rather awkward and has different implementations with
different bugs and also usually needs to be installed explicitely on
Linux.
It would be better to make LTO use Ian's library (but then it's C++ I
believe, not C)
-Andi
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.

That could be orthogonal to plugging elfcpp into the role currently occupied
by libelf in that reader.

(As to needing c++, that's just a matter of enabling c++ as a stage1
language and living with the minor limitation that go can't be a stage1
language unless you already have an installed c++ compiler, no?)

cheers,
DaveK
Mark Mitchell
2010-10-25 22:49:31 UTC
Permalink
Post by Dave Korn
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
That could be orthogonal to plugging elfcpp into the role currently occupied
by libelf in that reader.
I think it's reasonable to argue that GCC should, going forward, be an
ELF-only toolchain -- with postprocessing tools for generating PE/COFF,
Symbian DLLs, Mach-O or what have you. But, we haven't made that
decision. So, I don't think we should get there by half-measures.

Either we should decide that's what we want to do, or we should try to
keep the compiler independent of the object file format -- as we have up
until now. I understand Ian's distaste for BFD, but it is the
format-independent object file reader we have, so it seems a natural
choice. And libelf, which we already rely on seems more natural than
elfcpp, if we're willing to go ELF-only -- unless we're going to replace
the use of libelf in LTO with elfcpp as well.

In any case, I think we should avoid a single compiler build requiring
multiple object-file reading libraries.
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Dave Korn
2010-10-26 02:07:52 UTC
Permalink
Post by Mark Mitchell
Post by Dave Korn
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
That could be orthogonal to plugging elfcpp into the role currently occupied
by libelf in that reader.
I think it's reasonable to argue that GCC should, going forward, be an
ELF-only toolchain -- with postprocessing tools for generating PE/COFF,
Symbian DLLs, Mach-O or what have you. But, we haven't made that
decision. So, I don't think we should get there by half-measures.
I'll probably be on the other side of that argument, when it comes, for a
combination of political and engineering reasons. But, like you say, let's
not get side-tracked.
Post by Mark Mitchell
Either we should decide that's what we want to do, or we should try to
keep the compiler independent of the object file format -- as we have up
until now.
Ian could fairly point out that LTO was accepted into the compiler before it
was format-agnostic. I would counter that, until such a decision as you
contemplate is actually made, it would have been preferable if it was
format-agnostic from the start. However, we are where we are, and don't want
to let the perfect be the enemy of the good.
Post by Mark Mitchell
I understand Ian's distaste for BFD, but it is the
format-independent object file reader we have, so it seems a natural
choice. And libelf, which we already rely on seems more natural than
elfcpp, if we're willing to go ELF-only -- unless we're going to replace
the use of libelf in LTO with elfcpp as well.
Well, TBH, I suggested BFD as a devil's-advocate position. It does indeed
work, but it is kind of clunky, and top-heavy for what the compiler's
requirements actually amount to.

From Ian's description, gccgo has the exact same requirements as LTO: be
able to parse an object file, get a list of sections, and get raw binary
access to the data contained within a named section. This is a problem which
we already have solved. (And indeed LTO's solution also has object writing
capabilities that gccgo doesn't need.)
Post by Mark Mitchell
In any case, I think we should avoid a single compiler build requiring
multiple object-file reading libraries.
Code re-use FTW! As far as I can see, we're not going to need anything
significantly more complex than what the LTO-FE already needs, until and
unless we get to a point of integrating the (assembler and) linker into the
compiler itself, which is a long way off for now.

That being the case, I think a reasonable plan would be:

- integrate gccgo, with elfcpp
- then common out the file-reading stuff from gcc/lto/ up to gcc/ so that all
the FEs can share it
- then convert it to use elfcpp (with a bit of file I/O added) and stop using
libelf altogether
- then switch gccgo over to using it

... of which I think all but the first step would even be stage3-friendly.

cheers,
DaveK
Mark Mitchell
2010-10-26 02:22:45 UTC
Permalink
Post by Dave Korn
- integrate gccgo, with elfcpp
- then common out the file-reading stuff from gcc/lto/ up to gcc/ so that all
the FEs can share it
- then convert it to use elfcpp (with a bit of file I/O added) and stop using
libelf altogether
- then switch gccgo over to using it
I think that's a reasonable plan. It makes things no less object-file
netural than they are now, which is OK. (And, before someone else
points it out, I believe it was I who started using libelf in the LTO
prototype, so I know full well how we got here!) I certainly have no
problem with using elfcpp over libelf.
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Frank Ch. Eigler
2010-10-26 02:34:51 UTC
Permalink
[...] From Ian's description, gccgo has the exact same requirements
as LTO: be able to parse an object file, get a list of sections, and
get raw binary access to the data contained within a named section.
This is a problem which we already have solved. (And indeed LTO's
solution also has object writing capabilities that gccgo doesn't
need.) [...]
By the way, is there some necessity in accomplishing this by means of
a linked library, as opposed to via a spawned objcopy process?

- FChE
Mark Mitchell
2010-10-26 02:35:54 UTC
Permalink
Post by Frank Ch. Eigler
By the way, is there some necessity in accomplishing this by means of
a linked library, as opposed to via a spawned objcopy process?
Probably none in theory, but it certainly seems messy and likely to be
slow in practice. Is there a reason that this would be desirable?
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Frank Ch. Eigler
2010-10-26 02:39:13 UTC
Permalink
Hi -
Post by Mark Mitchell
Post by Frank Ch. Eigler
By the way, is there some necessity in accomplishing this by means of
a linked library, as opposed to via a spawned objcopy process?
Probably none in theory, but it certainly seems messy and likely to
be slow in practice.
Yes, maybe.
Post by Mark Mitchell
Is there a reason that this would be desirable?
It would seem to moot the present discussion about competing elf
consumer libraries. "none of the above" is a possible answer.

- FChE
Mark Mitchell
2010-10-26 02:41:20 UTC
Permalink
Post by Frank Ch. Eigler
It would seem to moot the present discussion about competing elf
consumer libraries. "none of the above" is a possible answer.
True. It seems that LTO and Go need a very simple interface; presumably
we can abstract that in the compiler and then we can implement that
interface as we please. I agree that a fallback to an external objcopy
is plausible, as is linking with BFD.
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Joseph S. Myers
2010-10-26 14:41:56 UTC
Permalink
Post by Mark Mitchell
Post by Frank Ch. Eigler
By the way, is there some necessity in accomplishing this by means of
a linked library, as opposed to via a spawned objcopy process?
(elfcpp isn't a *linked* library; it's a C++ template library consisting
entirely of headers, with no makefile code or .o or .a files.)
Post by Mark Mitchell
Probably none in theory, but it certainly seems messy and likely to be
slow in practice. Is there a reason that this would be desirable?
Well, slow on hosts where process creation is slow (just like the separate
gcc/cc1/as/collect2/ld/lto-wrapper/... processes). The separate process
design was probably based on process creation being fast - although
separate processes do have security and potentially parallelism advantages
over using libraries for everything. (It probably wouldn't be hard to
support linking more of the separate programs into one for hosts where
this helps; cf. the past MVS discussions.)
--
Joseph S. Myers
***@codesourcery.com
Mark Mitchell
2010-10-26 15:51:20 UTC
Permalink
Post by Joseph S. Myers
Well, slow on hosts where process creation is slow (just like the separate
gcc/cc1/as/collect2/ld/lto-wrapper/... processes). The separate process
design was probably based on process creation being fast
A lot of this is also historical; gcc/cc1/as/ld mirror typical UNIX
compilers of the era at which GCC was built. collect2 was presumably
necessary because of dependence on proprietary ld; if we could assume
GNU ld (or GOLD) everywhere, we could fold that functionality directly
into the linker.
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Ian Lance Taylor
2010-10-26 16:05:41 UTC
Permalink
Post by Mark Mitchell
A lot of this is also historical; gcc/cc1/as/ld mirror typical UNIX
compilers of the era at which GCC was built. collect2 was presumably
necessary because of dependence on proprietary ld; if we could assume
GNU ld (or GOLD) everywhere, we could fold that functionality directly
into the linker.
For what it's worth, all of the functionality except for -frepo is
already folded into both. And I suspect that -frepo support could be
handled via a linker plugin.

Ian
Andi Kleen
2010-10-26 16:39:21 UTC
Permalink
Post by Mark Mitchell
Post by Joseph S. Myers
Well, slow on hosts where process creation is slow (just like the separate
gcc/cc1/as/collect2/ld/lto-wrapper/... processes). The separate process
design was probably based on process creation being fast
A lot of this is also historical; gcc/cc1/as/ld mirror typical UNIX
compilers of the era at which GCC was built. collect2 was presumably
necessary because of dependence on proprietary ld; if we could assume
GNU ld (or GOLD) everywhere, we could fold that functionality directly
into the linker.
Right now it is definitely required to set up all the LTO
environment for the backend linker.

However that could probably move into the gcc driver.

-Andi
--
***@linux.intel.com -- Speaking for myself only.
Ian Lance Taylor
2010-10-29 01:31:33 UTC
Permalink
Post by Dave Korn
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
How about this?

This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.

With this patch, libelf is no longer needed.

I've bootstrapped this on x86_64-unknown-linux-gnu, and I've run the LTO
testsuite.

This patch puts the code in libiberty, but it could equally well go in
gcc. Anybody want to make an argument one way or another?

Does the general interface look OK?

This patch requires approval from the LTO maintainers. I don't need
approval for the libiberty changes (if the code stays in libiberty) but
of course I would appreciate it if somebody could look it over. I think
the configure and Makefile changes are sufficiently obvious given the
other changes as to not require approval.

If this patch is accepted, then gccgo will not require elfcpp.

Ian


include/ChangeLog:

2010-10-28 Ian Lance Taylor <***@google.com>

* objfile.h: New file.

libiberty/ChangeLog:

2010-10-28 Ian Lance Taylor <***@google.com>

* objfile.c: New file.
* objfile-common.h: New file.
* objfile-elf.c: New file.
* objfile-mach-o.c: New file.
* objfile-coff.c: New file.
* configure.ac: Add AC_TYPE_SSIZE_T.
* Makefile.in: Rebuild dependencies.
(CFILES): Add objfile.c, objfile-coff, objfile-elf.c,
objfile-mach-o.c.
(REQUIRED_OFILES): Add corresponding object files.
* configure: Rebuild.
* config.in: Rebuild.

gcc/ChangeLog:

2010-10-28 Ian Lance Taylor <***@google.com>

* configure.ac: Remove elf_getshdrstrndx test. Don't substitute
LTO_BINARY_READER or LTO_USE_LIBELF. Remove LIBELFLIBS and
LIBELFINC. Remove HAVE_libelf.
* gcc/config.gcc: Don't set lto_binary_reader.
* gcc/Makefile.in (LIBELFLIBS, LIBELFINC): Remove variables.
(LTO_BINARY_READER, LTO_USE_LIBELF): Remove variables.
(LIBS): Remove $(LIBELFLIBS).
(INCLUDES): Remove $(LIBELFINC).
* doc/install.texi (Prerequisites): Remove libelf paragraphs.
(Configuration): Mention --disable-lto. Remove --with-libelf
paragraph.
* configure: Rebuild.
* config.in: Rebuild.

gcc/lto/ChangeLog:

2010-10-28 Ian Lance Taylor <***@google.com>

* lto-objfile.c: New file.
* lto-elf.c: Remove file.
* lto-macho.c: Remove file.
* lto-macho.h: Remove file.
* lto-coff.c: Remove file.
* lto-coff.h: Remove file.
* Make-lang.in (LTO_OBJS): Change lto/$(LTO_BINARY_READER).o to
lto/lto-objfile.o.
($(LTO_EXE)): Remove $(LTO_USE_LIBELF)
(lto/lto-objfile.o): New target.
(lto/lto-elf.o, lto/lto-coff.o, lto/lto-macho.o): Remove targets.

./ChangeLog:

2010-10-28 Ian Lance Taylor <***@google.com>

* configure.ac: Don't set default_enable_lto. Remove libelf tests.
* configure: Rebuild.
Dave Korn
2010-10-29 08:56:02 UTC
Permalink
Post by Ian Lance Taylor
Post by Dave Korn
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
How about this?
That looks excellent, thank you!
Post by Ian Lance Taylor
This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
I'm right here :) Can't help with Darwin but hopefully Jack/Iain will be
available.
Post by Ian Lance Taylor
With this patch, libelf is no longer needed.
I've bootstrapped this on x86_64-unknown-linux-gnu, and I've run the LTO
testsuite.
This patch puts the code in libiberty, but it could equally well go in
gcc. Anybody want to make an argument one way or another?
Libiberty is better for sharing with lto-plugin, I think. I sent a patch
that commoned out some of the COFF-reader code into gcc/, but I only put it
there because it was a single header file with no associated object file. Now
there's actual code to be linked in, I think libiberty is a better choice than
lto-plugin trying to share .o files from the ../gcc/ objdir.
Post by Ian Lance Taylor
Does the general interface look OK?
This patch requires approval from the LTO maintainers.
Also, I have a patch outstanding to COFF-ize the lto-plugin(*), so if we can
get an early "OK in principle" I'll get cracking on respinning it to use this
new interface.

cheers,
DaveK
--
(*) - http://gcc.gnu.org/ml/gcc-patches/2010-10/msg02176.html
IainS
2010-10-29 11:09:59 UTC
Permalink
Post by Dave Korn
Post by Ian Lance Taylor
This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
I'm right here :) Can't help with Darwin but hopefully Jack/Iain will be
available.
I tried with 166058 on ppc-darwin9 (my other machines are tied up
right now)

-flto fails on link with missing _main.
e.g:
Executing on host: /Volumes/ScratchCS/gcc-4-6-trunk-build/gcc/xgcc -B/
Volumes/ScratchCS/gcc-4-6-trunk-build/gcc/ c_lto_20081024_0.o -
O0 -flto -m32 -o gcc-dg-lto-20081024-21 (timeout = 60)
Undefined symbols:
"_main", referenced from:
start in crt1.10.5.o

-whopr ices with:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
0x007f8e18 in objfile_mach_o_find_sections (objfile=0x40e078b0,
pfn=0x1d7ec <lto_obj_add_section>, data=0xbffff19c, err=0xbffff198)
at /GCC/gcc-live-trunk/libiberty/objfile-mach-o.c:469
469 memcpy (name, sechdr + sectname_offset, MACH_O_NAME_LEN);
(gdb) bt
#0 0x007f8e18 in objfile_mach_o_find_sections (objfile=0x40e078b0,
pfn=0x1d7ec <lto_obj_add_section>, data=0xbffff19c, err=0xbffff198)
at /GCC/gcc-live-trunk/libiberty/objfile-mach-o.c:469
#1 0x007f5dbc in objfile_find_sections (objfile=<value temporarily
unavailable, due to optimizations>, pfn=<value temporarily
unavailable, due to optimizations>, data=<value temporarily
unavailable, due to optimizations>, err=<value temporarily
unavailable, due to optimizations>) at /GCC/gcc-live-trunk/libiberty/
objfile.c:173
#2 0x0001dc60 in lto_obj_build_section_table (lto_file=<value
temporarily unavailable, due to optimizations>) at /GCC/gcc-live-trunk/
gcc/lto/lto-objfile.c:270
#3 0x0001af80 in lto_read_all_file_options () at /GCC/gcc-live-trunk/
gcc/lto/lto.c:2052
#4 0x00003960 in lto_post_options (pfilename=<value temporarily
unavailable, due to optimizations>) at /GCC/gcc-live-trunk/gcc/lto/lto-
lang.c:709
#5 0x004f0eec in toplev_main (argc=15, argv=Cannot access memory at
address 0x1c
) at /GCC/gcc-live-trunk/gcc/toplev.c:1778
#6 0x00001944 in start ()

if you need more analysis - ping me with what and I'll try and do sth
-- or maybe Jack can help with a user on his machine.

cheers,
Iain
Jack Howarth
2010-10-29 13:18:53 UTC
Permalink
Post by Dave Korn
Post by Ian Lance Taylor
Post by Dave Korn
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
How about this?
That looks excellent, thank you!
Post by Ian Lance Taylor
This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
I'm right here :) Can't help with Darwin but hopefully Jack/Iain will be
available.
Dave,
Doesn't the go compiler require functional split stack support? Mike Stump
left me with the impression that split stack support would require additional
linker support on darwin.
Jack
Post by Dave Korn
Post by Ian Lance Taylor
With this patch, libelf is no longer needed.
I've bootstrapped this on x86_64-unknown-linux-gnu, and I've run the LTO
testsuite.
This patch puts the code in libiberty, but it could equally well go in
gcc. Anybody want to make an argument one way or another?
Libiberty is better for sharing with lto-plugin, I think. I sent a patch
that commoned out some of the COFF-reader code into gcc/, but I only put it
there because it was a single header file with no associated object file. Now
there's actual code to be linked in, I think libiberty is a better choice than
lto-plugin trying to share .o files from the ../gcc/ objdir.
Post by Ian Lance Taylor
Does the general interface look OK?
This patch requires approval from the LTO maintainers.
Also, I have a patch outstanding to COFF-ize the lto-plugin(*), so if we can
get an early "OK in principle" I'll get cracking on respinning it to use this
new interface.
cheers,
DaveK
--
(*) - http://gcc.gnu.org/ml/gcc-patches/2010-10/msg02176.html
Dave Korn
2010-10-29 14:08:51 UTC
Permalink
Post by Jack Howarth
Post by Dave Korn
Post by Ian Lance Taylor
Post by Dave Korn
What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
How about this?
That looks excellent, thank you!
Post by Ian Lance Taylor
This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
I'm right here :) Can't help with Darwin but hopefully Jack/Iain will be
available.
Dave,
Doesn't the go compiler require functional split stack support?
Ian will have to answer that, I don't know.
Post by Jack Howarth
Mike Stump
left me with the impression that split stack support would require additional
linker support on darwin.
Well, this also affects LTO, since it refactors the object file support
underlying that. As Iain has discovered...

cheers,
DaveK
Ian Lance Taylor
2010-10-29 13:49:51 UTC
Permalink
Post by Jack Howarth
Doesn't the go compiler require functional split stack support? Mike Stump
left me with the impression that split stack support would require additional
linker support on darwin.
The Go compiler can work without split stack support. The effect is
that you are limited in the number of goroutines you can create,
particularly on a 32-bit system. And you are also limited in the depth
of recursion and size of local variables you can create. But you can
write working Go programs.

The objfile patch, however, is not really about Go, although gccgo will
use it. It's really about LTO. If it works for LTO, it will work for
gccgo.

Ian
Jack Howarth
2010-10-29 14:33:59 UTC
Permalink
Post by Ian Lance Taylor
Post by Jack Howarth
Doesn't the go compiler require functional split stack support? Mike Stump
left me with the impression that split stack support would require additional
linker support on darwin.
The Go compiler can work without split stack support. The effect is
that you are limited in the number of goroutines you can create,
particularly on a 32-bit system. And you are also limited in the depth
of recursion and size of local variables you can create. But you can
write working Go programs.
Ian,
Is split stack support unique to the go compiler or might it eventually
be leveraged in the other compilers as well? We could submit a radar for
the addition of split stack support for the linker in Xcode 4.0 or later
but it would helpful if the eventual usage was greater than just the go
compiler.
Jack
Post by Ian Lance Taylor
The objfile patch, however, is not really about Go, although gccgo will
use it. It's really about LTO. If it works for LTO, it will work for
gccgo.
Ian
Ian Lance Taylor
2010-10-29 15:00:28 UTC
Permalink
Post by Jack Howarth
Is split stack support unique to the go compiler or might it eventually
be leveraged in the other compilers as well? We could submit a radar for
the addition of split stack support for the linker in Xcode 4.0 or later
but it would helpful if the eventual usage was greater than just the go
compiler.
Split stack support is now in mainline and is available for any language
via the -fsplit-stack option. The only thing unique to the Go frontend
is that the Go frontend turns it on by default.

Using it on Darwin will require some porting work of the assembly code
in libgcc, which currently uses ELF pseudo-ops. Object files which are
compiled with -fsplit-stack need to be marked in some way. What the
linker needs to do is frob functions in object files compiled with
-fsplit-stack which call functoins defined in object files compiled
without -fsplit-stack.

Ian
Richard Guenther
2010-10-29 13:31:15 UTC
Permalink
Post by Ian Lance Taylor
  What would be even nicer would be if we could share the same code-reader
interface between lto and go (and the lto-plugin), thereby getting object
format independence equally everywhere for no extra cost.
How about this?
This implements an object file reader/writer which does everything
required by LTO and gccgo.  The ELF code works.  I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
With this patch, libelf is no longer needed.
I've bootstrapped this on x86_64-unknown-linux-gnu, and I've run the LTO
testsuite.
This patch puts the code in libiberty, but it could equally well go in
gcc.  Anybody want to make an argument one way or another?
Does the general interface look OK?
This patch requires approval from the LTO maintainers.  I don't need
approval for the libiberty changes (if the code stays in libiberty) but
of course I would appreciate it if somebody could look it over.  I think
the configure and Makefile changes are sufficiently obvious given the
other changes as to not require approval.
If this patch is accepted, then gccgo will not require elfcpp.
Nice!

The LTO changes are ok (I suppose you'll be around in helping people
debug eventual problems).

Richard.
Post by Ian Lance Taylor
Ian
       * objfile.h: New file.
       * objfile.c: New file.
       * objfile-common.h: New file.
       * objfile-elf.c: New file.
       * objfile-mach-o.c: New file.
       * objfile-coff.c: New file.
       * configure.ac: Add AC_TYPE_SSIZE_T.
       * Makefile.in: Rebuild dependencies.
       (CFILES): Add objfile.c, objfile-coff, objfile-elf.c,
       objfile-mach-o.c.
       (REQUIRED_OFILES): Add corresponding object files.
       * configure: Rebuild.
       * config.in: Rebuild.
       * configure.ac: Remove elf_getshdrstrndx test.  Don't substitute
       LTO_BINARY_READER or LTO_USE_LIBELF.  Remove LIBELFLIBS and
       LIBELFINC.  Remove HAVE_libelf.
       * gcc/config.gcc: Don't set lto_binary_reader.
       * gcc/Makefile.in (LIBELFLIBS, LIBELFINC): Remove variables.
       (LTO_BINARY_READER, LTO_USE_LIBELF): Remove variables.
       (LIBS): Remove $(LIBELFLIBS).
       (INCLUDES): Remove $(LIBELFINC).
       * doc/install.texi (Prerequisites): Remove libelf paragraphs.
       (Configuration): Mention --disable-lto.  Remove --with-libelf
       paragraph.
       * configure: Rebuild.
       * config.in: Rebuild.
       * lto-objfile.c: New file.
       * lto-elf.c: Remove file.
       * lto-macho.c: Remove file.
       * lto-macho.h: Remove file.
       * lto-coff.c: Remove file.
       * lto-coff.h: Remove file.
       * Make-lang.in (LTO_OBJS): Change lto/$(LTO_BINARY_READER).o to
       lto/lto-objfile.o.
       ($(LTO_EXE)): Remove $(LTO_USE_LIBELF)
       (lto/lto-objfile.o): New target.
       (lto/lto-elf.o, lto/lto-coff.o, lto/lto-macho.o): Remove targets.
       * configure.ac: Don't set default_enable_lto.  Remove libelf tests.
       * configure: Rebuild.
Dave Korn
2010-10-29 14:07:18 UTC
Permalink
Post by Richard Guenther
Post by Ian Lance Taylor
This patch requires approval from the LTO maintainers. I don't need
approval for the libiberty changes (if the code stays in libiberty) but
of course I would appreciate it if somebody could look it over. I think
the configure and Makefile changes are sufficiently obvious given the
other changes as to not require approval.
If this patch is accepted, then gccgo will not require elfcpp.
Nice!
The LTO changes are ok (I suppose you'll be around in helping people
debug eventual problems).
I'll start work on porting the lto-plugin to use the new interface. It'll
subsume my plugin-for-coff patch.

cheers,
DaveK
Mark Mitchell
2010-10-29 14:50:03 UTC
Permalink
Post by Ian Lance Taylor
This patch requires approval from the LTO maintainers. I don't need
approval for the libiberty changes (if the code stays in libiberty) but
of course I would appreciate it if somebody could look it over. I think
the configure and Makefile changes are sufficiently obvious given the
other changes as to not require approval.
This all looks good to me, and seems like a reasonable solution. I
think libiberty is as good a place as any for the routines, FWIW.

Thanks,
--
Mark Mitchell
CodeSourcery
***@codesourcery.com
(650) 331-3385 x713
Richard Henderson
2010-10-30 00:23:11 UTC
Permalink
+extern objfile_read *
+objfile_open_read (int descriptor, off_t offset, const char *segment_name,
+ const char **errmsg, int *err);
...
+extern objfile_write *
+objfile_start_write (objfile_attributes *ATTRS, const char *segment_name,
+ const char **errmsg, int *err);
Mismatch in naming conventions? I like the use of "release"
to clearly indicate non-closure of the FD.
+ shdr_size = (cl == ELFCLASS32
+ ? sizeof (Elf32_External_Shdr)
+ : sizeof (Elf64_External_Shdr));
+ memset (buf, 0, shdr_size);
Constant size memset is easier to optimize. You might as well
zero all of BUF, even if we're not going to use it all. The
slop between 32 and 64 is minimal.
+ if (!objfile_internal_read (objfile->descriptor,
+ objfile->offset + eor->shoff + shdr_size,
+ shdrs,
+ shdr_size * (shnum - 1),
+ &errmsg, err))
Do we really want to keep re-reading section data for every section
lookup we do? Can't we do this in objfile_open_read?
+ set_32 (hdr + offsetof (struct external_scnhdr, s_flags),
+ (STYP_DATA | IMAGE_SCN_ALIGN_1BYTES | IMAGE_SCN_MEM_DISCARDABLE
+ | IMAGE_SCN_MEM_SHARED | IMAGE_SCN_MEM_READ));
You're not recording alignment in the coff object file?
IMAGE_SCN_ALIGN_<N>BYTES, 1 <= N <= 8192, are all defined
with a simple function in that nibble.


r~
Dave Korn
2010-10-30 08:16:21 UTC
Permalink
Post by Richard Henderson
+ if (!objfile_internal_read (objfile->descriptor,
+ objfile->offset + eor->shoff + shdr_size,
+ shdrs,
+ shdr_size * (shnum - 1),
+ &errmsg, err))
Do we really want to keep re-reading section data for every section
lookup we do? Can't we do this in objfile_open_read?
It should only be necessary to do one section lookup per object file anyway.
Keep extra data hanging around in memory in the backend just so that we can
write algorithmically inefficient code in the client? Seems like a bad
tradeoff to me!
Post by Richard Henderson
+ set_32 (hdr + offsetof (struct external_scnhdr, s_flags),
+ (STYP_DATA | IMAGE_SCN_ALIGN_1BYTES | IMAGE_SCN_MEM_DISCARDABLE
+ | IMAGE_SCN_MEM_SHARED | IMAGE_SCN_MEM_READ));
You're not recording alignment in the coff object file?
It looks to me like he's recording an alignment of 1 byte.
Post by Richard Henderson
IMAGE_SCN_ALIGN_<N>BYTES, 1 <= N <= 8192, are all defined
with a simple function in that nibble.
The line you quoted uses IMAGE_SCN_ALIGN_1BYTES. That's all we'll ever need
for LTO sections.

cheers,
DaveK
Richard Henderson
2010-10-30 17:57:56 UTC
Permalink
Post by Dave Korn
Post by Richard Henderson
Do we really want to keep re-reading section data for every section
lookup we do? Can't we do this in objfile_open_read?
It should only be necessary to do one section lookup per object file anyway.
Keep extra data hanging around in memory in the backend just so that we can
write algorithmically inefficient code in the client? Seems like a bad
tradeoff to me!
Uh, really? I thought there were like a half-dozen lto sections...
Post by Dave Korn
The line you quoted uses IMAGE_SCN_ALIGN_1BYTES. That's all we'll ever need
for LTO sections.
Perhaps, but there's an argument to the create_section function that
specifies the alignment. If we only need 1-byte alignment that's fine,
but we should either assert that's true, omit the argument, or properly
support it.


r~
Dave Korn
2010-10-30 18:37:51 UTC
Permalink
Post by Richard Henderson
Post by Dave Korn
Post by Richard Henderson
Do we really want to keep re-reading section data for every section
lookup we do? Can't we do this in objfile_open_read?
It should only be necessary to do one section lookup per object file anyway.
Keep extra data hanging around in memory in the backend just so that we can
write algorithmically inefficient code in the client? Seems like a bad
tradeoff to me!
Uh, really? I thought there were like a half-dozen lto sections...
Which we iterate over just once, and record them all in a hash table from
the per-section callback, unless I've missed something.
Post by Richard Henderson
Post by Dave Korn
The line you quoted uses IMAGE_SCN_ALIGN_1BYTES. That's all we'll ever need
for LTO sections.
Perhaps, but there's an argument to the create_section function that
specifies the alignment. If we only need 1-byte alignment that's fine,
but we should either assert that's true, omit the argument, or properly
support it.
Fair point.

cheers,
DaveK
Richard Henderson
2010-10-30 18:24:08 UTC
Permalink
Post by Dave Korn
Post by Richard Henderson
Uh, really? I thought there were like a half-dozen lto sections...
Which we iterate over just once, and record them all in a hash table from
the per-section callback, unless I've missed something.
Oh, right. Nevermind then.


r~
Dave Korn
2010-10-30 18:58:51 UTC
Permalink
Post by Richard Henderson
Post by Dave Korn
Post by Richard Henderson
Uh, really? I thought there were like a half-dozen lto sections...
Which we iterate over just once, and record them all in a hash table from
the per-section callback, unless I've missed something.
Oh, right. Nevermind then.
r~
It'd be worth putting a warning comment on the find_section (no "s")
function saying that it's pretty inefficient and that it is best practice to
call find_sections just once and record details rather than make multiple
calls to find_section. (Both lto FE and lto-plugin are already architected
that way.)

cheers,
DaveK
Ian Lance Taylor
2010-10-31 17:55:12 UTC
Permalink
Post by Dave Korn
Post by Richard Henderson
Post by Dave Korn
Post by Richard Henderson
Uh, really? I thought there were like a half-dozen lto sections...
Which we iterate over just once, and record them all in a hash table from
the per-section callback, unless I've missed something.
Oh, right. Nevermind then.
r~
It'd be worth putting a warning comment on the find_section (no "s")
function saying that it's pretty inefficient and that it is best practice to
call find_sections just once and record details rather than make multiple
calls to find_section. (Both lto FE and lto-plugin are already architected
that way.)
cheers,
DaveK
I added a sentence "Note that calling this multiple times is
inefficient; use objfile_find_sections instead."

I added the function because it's what the gccgo frontend needs: it only
needs to find one section.

Ian
Dave Korn
2010-10-30 10:44:50 UTC
Permalink
Post by Ian Lance Taylor
This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
With this patch, libelf is no longer needed.
Almost, but not quite! The attached patch switches the lto-plugin over to
use the new objfile reader as well.

lto-plugin/ChangeLog:

* configure.ac: Don't use libelf, don't source config.gcc.
(LIBELFLIBS): Delete.
(LIBELFINC): Delete.
(LTO_FORMAT): Delete.
(SYM_STYLE): New AC_SUBST var set based on $target.
* Makefile.am (LIBELFLIBS): Delete.
(LIBELFINC): Delete.
(LTO_FORMAT): Delete.
(SYM_STYLE): Import.
(AM_CPPFLAGS): Use it. Don't use LIBELFINC.
(liblto_plugin_la_SOURCES): Don't use LTO_FORMAT, don't include
any object-format-specific source file in the link.
(liblto_plugin_la_LIBADD): Don't use LIBELFLIBS.
* configure: Regenerate.
* Makefile.in: Regenerate.
* lto-plugin-elf.c: Delete.
* lto-plugin-coff.c: Delete.
* lto-plugin.h: Delete.
* lto-plugin.c (O_BINARY): Definition moved here from lto-plugin.h.
(LTO_SEGMENT_NAME): New definition.
(LTO_SECTION_PREFIX): Definition moved here from lto-plugin.h.
(LTO_SECTION_PREFIX_LEN): New definition.
(struct sym_aux): Struct definition moved here from lto-plugin.h.
(struct plugin_symtab): Likewise.
(struct plugin_objfile): Likewise.
(struct plugin_objfile): New struct def.
(enum symbol_style): New enum type.
(add_symbols): Make static.
(claimed_files): Likewise.
(num_claimed_files): Likewise.
(sym_style): New global.
(check): Make static.
(parse_table_entry): Likewise. Respect sym_style when extracting
symbol from symtab entry.
(translate): Make static.
(resolve_conflicts): Likewise.
(process_symtab): New function, per-section callback version of
old object-format-specific handling from deleted lto-plugin-elf.c.
(claim_file_handler): Convert ELF-specific version from deleted
lto-plugin-elf.c to objfile interface and move here.
(process_options): Allow new '-sym-style=' option.
(onload): Don't call deleted onload_format_checks hook.

Bootstrapped and tested all languages except java and ada on
x86_64-unknown-linux-gnu, no regressions. There are still a couple of bugs to
iron out of the COFF-specific part of the objfile reader which I'll have fixed
later today.

OK for trunk (once the objfile changes have gone in)?

cheers,
DaveK
Dave Korn
2010-10-30 16:49:22 UTC
Permalink
Post by Dave Korn
Post by Ian Lance Taylor
This implements an object file reader/writer which does everything
required by LTO and gccgo. The ELF code works. I have not tested the
Mach-O and COFF code at all beyond compiling it; I hope that somebody
else can test those targets and fix them.
With this patch, libelf is no longer needed.
Almost, but not quite! The attached patch switches the lto-plugin over to
use the new objfile reader as well.
Bootstrapped and tested all languages except java and ada on
x86_64-unknown-linux-gnu, no regressions. There are still a couple of bugs to
iron out of the COFF-specific part of the objfile reader which I'll have fixed
later today.
OK for trunk (once the objfile changes have gone in)?
As you'll see elsethread, I fixed the COFF problems. This is the remaining
chunk of patch required to make the lto-plugin work on cygming targets.

gcc/ChangeLog:

* gcc.c (PLUGIN_PASSTHROUGH_SPEC): New macro factored out from
LINK_COMMAND_SPEC.
(LINK_COMMAND_SPEC): Use it.
(static_spec_functions[]): Add pass-through-libs entry.
(pass_through_libs_spec_func): Add related spec function.
* config/i386/cygming.h (PLUGIN_PASSTHROUGH_SPEC): Define.
* doc/tm.texi.in (PLUGIN_PASSTHROUGH_SPEC): Document.
(LINK_COMMAND_SPEC): Mention it.
* doc/tm.texi: Regenerate.
* doc/invoke.texi (pass-through-libs): Mention new spec function.

Taking this for a bootstrap and test cycle now (on top of the objfile
changes). Assuming no regressions, and one new PASS (the sole LTO test that
exercises -fuse-linker-plugin), and once all the other patches have gone in: OK?

cheers,
DaveK
Dave Korn
2010-10-30 15:28:24 UTC
Permalink
Post by Ian Lance Taylor
* objfile-coff.c: New file.
+ if (namebuf[0] == '/')
+ {
+ size_t strindex;
+ char *end;
+
+ strindex = strtol (namebuf, &end, 10);
Needs to be strtol (namebuf + 1, ....
Post by Ian Lance Taylor
+ /* We don't write out any symbols. We'll see if that causes any
+ problems. */
Not a chance of getting away with that, I'm afraid. Everything expects
there to be file and section symbols and their auxiliaries.
Post by Ian Lance Taylor
+ set_16 (hdr + offsetof (struct external_filehdr, f_magic), attrs->magic);
+ set_16 (hdr + offsetof (struct external_filehdr, f_magic), nscns);
Cut'n'pasto. Second f_magic should be f_nscns.
Post by Ian Lance Taylor
+ name_offset += namelen;
Also needs to be namelen + 1.

Attached are the revised version of the file, and a diff to show what I
changed. With this version, all the tests in gcc.dg/lto/lto.exp pass as
before (i.e. there are still a couple of pre-existing FAILs that aren't affected).

cheers,
DaveK
Ian Lance Taylor
2010-10-31 18:05:07 UTC
Permalink
Post by Dave Korn
Attached are the revised version of the file, and a diff to show what I
changed. With this version, all the tests in gcc.dg/lto/lto.exp pass as
before (i.e. there are still a couple of pre-existing FAILs that aren't affected).
Excellent, thanks. I have incorporated your objfile-coff.c in my
working directory, and will commit it when I commit the whole patch set.

I think it can be committed as soon as objfile-mach-o.c is working.

Ian
Tom Tromey
2010-11-01 16:45:02 UTC
Permalink
Ian> This patch puts the code in libiberty, but it could equally well go in
Ian> gcc. Anybody want to make an argument one way or another?

Ian> +extern const char *
Ian> +objfile_attributes_compare (objfile_attributes *attrs1,

GDB already uses the name "objfile" for one of its modules.
I don't think we have any name clashes with this patch right now, but I
would prefer to avoid the eventual confusion.
So, if this is in libiberty, could it please have a different name?

thanks,
Tom
Ian Lance Taylor
2010-11-01 18:17:58 UTC
Permalink
Post by Tom Tromey
Ian> This patch puts the code in libiberty, but it could equally well go in
Ian> gcc. Anybody want to make an argument one way or another?
Ian> +extern const char *
Ian> +objfile_attributes_compare (objfile_attributes *attrs1,
GDB already uses the name "objfile" for one of its modules.
I don't think we have any name clashes with this patch right now, but I
would prefer to avoid the eventual confusion.
So, if this is in libiberty, could it please have a different name?
Sure.

Any suggestions?

Ian
Paolo Bonzini
2010-11-02 10:50:53 UTC
Permalink
Post by Ian Lance Taylor
Ian> This patch puts the code in libiberty, but it could equally well go in
Ian> gcc. Anybody want to make an argument one way or another?
Ian> +extern const char *
Ian> +objfile_attributes_compare (objfile_attributes *attrs1,
GDB already uses the name "objfile" for one of its modules.
I don't think we have any name clashes with this patch right now, but I
would prefer to avoid the eventual confusion.
So, if this is in libiberty, could it please have a different name?
Sure.
Any suggestions?
readobj/objread?

Paolo
Paolo Bonzini
2010-11-02 10:52:41 UTC
Permalink
Post by Paolo Bonzini
Post by Ian Lance Taylor
Post by Tom Tromey
Ian> This patch puts the code in libiberty, but it could equally well go in
Ian> gcc. Anybody want to make an argument one way or another?
Ian> +extern const char *
Ian> +objfile_attributes_compare (objfile_attributes *attrs1,
GDB already uses the name "objfile" for one of its modules.
I don't think we have any name clashes with this patch right now, but I
would prefer to avoid the eventual confusion.
So, if this is in libiberty, could it please have a different name?
Sure.
Any suggestions?
readobj/objread?
This obviously meant objread/objwrite. :)

Paolo
Eric Botcazou
2010-11-06 19:57:55 UTC
Permalink
Post by Ian Lance Taylor
* lto-objfile.c: New file.
* lto-elf.c: Remove file.
This has removed the support for compatible architectures present in lto-elf.c
and hasn't added any replacement in libiberty, so a bunch of LTO tests fail
again on 32-bit SPARC/Solaris.

The old code reads:

/* Return true if ELF_MACHINE is compatible with the cached value of the
architecture and possibly update the latter. Return false otherwise.

Note: if you want to add more EM_* cases, you'll need to provide the
corresponding definitions at the beginning of the file. */

static bool
is_compatible_architecture (Elf64_Half elf_machine)
{
if (cached_file_attrs.elf_machine == elf_machine)
return true;

switch (cached_file_attrs.elf_machine)
{
case EM_SPARC:
if (elf_machine == EM_SPARC32PLUS)
{
cached_file_attrs.elf_machine = elf_machine;
return true;
}
break;

case EM_SPARC32PLUS:
if (elf_machine == EM_SPARC)
return true;
break;

default:
break;
}

return false;
}

so it was not only accepting object files with compatible architectures in the
same LTO compilation, but also updating the current architecture for later
write operations. This is modelled on the linker.
--
Eric Botcazou
H.J. Lu
2010-10-24 19:11:29 UTC
Permalink
Post by Ian Lance Taylor
The Go frontend was approved for inclusion with gcc by the steering
committee a while back: http://gcc.gnu.org/ml/gcc/2010-01/msg00500.html .
Assuming it is OK with the release managers, I would like to get it into
the gcc 4.6 release.  I have been running behind my intended schedule,
for which I apologize.  However, the code is almost ready to merge, in
the sense that the only remaining changes are Go specific and should not
affect other languages.  The only thing I hope to clean up further
before the merge is additional separation between the Go frontend proper
and the gcc-specific interface.  I'm not going to have time to do the
full planned separation, which I will continue to work on, but I hope to
have the proper framework in place for future work.
Does Go depend on split stack? There are at least 2 split stack bugs
open for x86 target.


H.J.
Ian Lance Taylor
2010-10-25 02:53:37 UTC
Permalink
Post by H.J. Lu
Does Go depend on split stack?
Yes.
Post by H.J. Lu
There are at least 2 split stack bugs
open for x86 target.
Yes, I saw those but I have not had time to look at them. I will.

Ian
Paolo Bonzini
2010-10-24 21:15:35 UTC
Permalink
Post by Ian Lance Taylor
configure.ac
Add libgo. If building Go, build C++ as a boot language.
Can you generalize this using something in gcc/go/config-lang.in?

Paolo
Ian Lance Taylor
2010-10-29 23:15:09 UTC
Permalink
Post by Paolo Bonzini
Post by Ian Lance Taylor
configure.ac
Add libgo. If building Go, build C++ as a boot language.
Can you generalize this using something in gcc/go/config-lang.in?
I have now done this on the gccgo branch. If language X's
config-lang.in sets the shell variable lang_requires_boot_language, then
if X is enabled all those languages are built during stage 1.

Ian
H.J. Lu
2010-10-29 23:20:43 UTC
Permalink
Post by Paolo Bonzini
Post by Ian Lance Taylor
configure.ac
   Add libgo.  If building Go, build C++ as a boot language.
Can you generalize this using something in gcc/go/config-lang.in?
I have now done this on the gccgo branch.  If language X's
config-lang.in sets the shell variable lang_requires_boot_language, then
if X is enabled all those languages are built during stage 1.
Don't we have boot_language for this purpose?
--
H.J.
Ian Lance Taylor
2010-10-29 23:39:31 UTC
Permalink
Post by H.J. Lu
Post by Paolo Bonzini
Post by Ian Lance Taylor
configure.ac
   Add libgo.  If building Go, build C++ as a boot language.
Can you generalize this using something in gcc/go/config-lang.in?
I have now done this on the gccgo branch.  If language X's
config-lang.in sets the shell variable lang_requires_boot_language, then
if X is enabled all those languages are built during stage 1.
Don't we have boot_language for this purpose?
No, boot_language serves a different purpose. It says that the language
itself should be built during stage 1. What I need for Go is to say
that C++ should be built during stage 1. That is, boot_language is a
yes or no value. I have added lang_requires_boot_language, which takes
a list of languages. It is similar to lang_requires, except that it
specifically builds the specified languages during stage 1.

Ian
H.J. Lu
2010-10-30 03:30:48 UTC
Permalink
Post by H.J. Lu
Post by Paolo Bonzini
Post by Ian Lance Taylor
configure.ac
   Add libgo.  If building Go, build C++ as a boot language.
Can you generalize this using something in gcc/go/config-lang.in?
I have now done this on the gccgo branch.  If language X's
config-lang.in sets the shell variable lang_requires_boot_language, then
if X is enabled all those languages are built during stage 1.
Don't we have boot_language for this purpose?
No, boot_language serves a different purpose.  It says that the language
itself should be built during stage 1.  What I need for Go is to say
that C++ should be built during stage 1.  That is, boot_language is a
yes or no value.  I have added lang_requires_boot_language, which takes
Will put

if [ Go is enabled ]; then
boot_language=yes
fi

in cp/config-lang.in work?
--
H.J.
Paolo Bonzini
2010-10-30 07:57:21 UTC
Permalink
Post by H.J. Lu
Will put
if [ Go is enabled ]; then
boot_language=yes
fi
in cp/config-lang.in work?
It's a bit backwards, no?

Paolo
Ian Lance Taylor
2010-10-31 17:53:17 UTC
Permalink
Post by H.J. Lu
Will put
if [ Go is enabled ]; then
boot_language=yes
fi
in cp/config-lang.in work?
Probably, but why should I change cp/config-lang.in if I want to support
Go? Everything required to support Go should, as much as possible, be
in the Go frontend.

Ian
Florian Weimer
2010-10-31 18:39:26 UTC
Permalink
Post by Ian Lance Taylor
The Go frontend was approved for inclusion with gcc by the steering
committee a while back: http://gcc.gnu.org/ml/gcc/2010-01/msg00500.html .
How general is the garbage collector and its support infrastructure in
the compiler? AFAICS, it's precise, unlike the Boehm/Dehmers/Weiser
collector used by the Java front end.
Ian Lance Taylor
2010-10-31 18:47:22 UTC
Permalink
Post by Florian Weimer
Post by Ian Lance Taylor
The Go frontend was approved for inclusion with gcc by the steering
committee a while back: http://gcc.gnu.org/ml/gcc/2010-01/msg00500.html .
How general is the garbage collector and its support infrastructure in
the compiler? AFAICS, it's precise, unlike the Boehm/Dehmers/Weiser
collector used by the Java front end.
The garbage collector is a work in progress. What I say here is how it
works at present, but everything will change in some as yet undetermined
manner.

At the moment the only compiler support for the garbage collector is
that some allocations are marked as not containing pointers. Those
blocks are ignored when scanning. When scanning memory blocks which may
contain pointers, which includes the stack, the scanning is
conservative: an integer value which happens to look like a pointer will
cause the block to which it points to be marked as live.

Ian
Loading...