dynamic library building and loading

[Andrei Alexandrescu]

Haven't done any dynamic linking with D and I need to. I'm using dmd
2.058/Linux at work to build and use dynamic libraries. Here's my attempt:

*** Makefile
all: main lib.so

main: main.d
dmd main

lib.so: lib.d
dmd -fPIC -shared lib.d -of./lib.so

*** lib.d
extern(C) int fun(string s)
{
return 42;
}

*** main.d
import std.stdio;
void main()
{
}

Running make prints:

dmd -fPIC -shared lib.d -of./lib.so
/usr/bin/ld:
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o):
relocation R_X86_64_32 against `a local symbol' can not be used when
making a shared object; recompile with -fPIC
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a:
could not read symbols: Bad value
collect2: ld returned 1 exit status

What steps do I need to take to get off the ground?


Thanks,

Andrei

[Denis Shelomovskij]

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm using dmd
2.058/Linux at work to build and use dynamic libraries.

Sorry for the OT (as you are a Linux user), but if you will ever do it
on Windows, be aware of this nasty trap:
http://d.puremagic.com/issues/show_bug.cgi?id=8130

[Andrei Alexandrescu]

Post by Denis Shelomovskij

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm using dmd
2.058/Linux at work to build and use dynamic libraries.

Sorry for the OT (as you are a Linux user), but if you will ever do it
http://d.puremagic.com/issues/show_bug.cgi?id=8130

Thanks!

Andrei

[Michael]

On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm
using dmd 2.058/Linux at work to build and use dynamic

Maybe it will help you
D:
https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d

Fortran:
https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95

Dmd 2.060 used.

DLL on OpenSUSE 64bit compiled with command:
gfortran -shared -fPIC -O3 -o simple.so simple.f95

[Andrei Alexandrescu]

Post by Michael
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm using dmd

Maybe it will help you
https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d
https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95
Dmd 2.060 used.
gfortran -shared -fPIC -O3 -o simple.so simple.f95

Thanks. The loading part is very useful, but I'm still lost when it
comes to build the shared library itself.

Andrei

[Michael]

Post by Andrei Alexandrescu
Thanks. The loading part is very useful, but I'm still lost
when it comes to build the shared library itself.
Andrei

Program loads dll at runtime using loader which is configured to
load concrete dll file(s). Like in gtkD
http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d

[nazriel]

Post by Michael

Post by Andrei Alexandrescu
Thanks. The loading part is very useful, but I'm still lost
when it comes to build the shared library itself.
Andrei

Program loads dll at runtime using loader which is configured
to load concrete dll file(s). Like in gtkD
http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d

Loading Shared lib isn't big issues here.

The bigger one is building Shared library (written in D) and
running it in host application without issues (EH, shared GC etc).

Andrei, if you find out how to make those things work, please
share your findings. I'm also in need of using shared libraries.

And yeah, probably Martin Nowak will be the best bet to get
information from.

[Michael]

Post by nazriel
Loading Shared lib isn't big issues here.
The bigger one is building Shared library (written in D) and
running it in host application without issues (EH, shared GC
etc).
Andrei, if you find out how to make those things work, please
share your findings. I'm also in need of using shared libraries.
And yeah, probably Martin Nowak will be the best bet to get
information from.

I understand. As I remember on forum were topic with same
approach for dll written in D. But I can't find a link.

[Paulo Pinto]

Post by nazriel

Post by Michael

Post by Andrei Alexandrescu
Thanks. The loading part is very useful, but I'm still lost
when it comes to build the shared library itself.
Andrei

Program loads dll at runtime using loader which is configured
to load concrete dll file(s). Like in gtkD
http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d

Loading Shared lib isn't big issues here.
The bigger one is building Shared library (written in D) and
running it in host application without issues (EH, shared GC
etc).
Andrei, if you find out how to make those things work, please
share your findings. I'm also in need of using shared libraries.
And yeah, probably Martin Nowak will be the best bet to get
information from.

Native Oberon/BlueBottle OS really a lot in dynamic modules, as
everything
is dynamically loaded. As far as I know both operating systems
have a global GC.

Not sure how Spin does it.

Singularity, uses local GC per process and does not support
dynamic loading.

Bringing this into D's context, I imagine a solution for the GC
would be to have it shared across all modules. The biggest
problem I see, is that all shared libraries have to use the same
D compiler, or a standard GC ABI needs to be defined.

--
Paulo

[Jacob Carlborg]

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm using dmd
dmd -fPIC -shared lib.d -of./lib.so
relocation R_X86_64_32 against `a local symbol' can not be used when
making a shared object; recompile with -fPIC
could not read symbols: Bad value
collect2: ld returned 1 exit status
What steps do I need to take to get off the ground?

You need to properly implement support for dynamic libraries in druntime
and DMD. I'm not entirely sure if there are some problems left in DMD,
but druntime is definitely not ready yet.

In general what needs to be done is to properly handle:

* Module infos
* TLS
* Exception handling tables

Martin Nowak has a branch for this:

https://github.com/dawgfoto/druntime/tree/SharedRuntime

[Andrei Alexandrescu]

Post by Jacob Carlborg
https://github.com/dawgfoto/druntime/tree/SharedRuntime

Thanks, I'll follow up with him.

Andrei

[Maxim Fomin]

You can build shared libraries on linux by manually compiling
object files and linking them. On windows last time I tries it
was not possible.

[Jens Mueller]

You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.

Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?

Jens

[Maxim Fomin]

On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller

Post by Jens Mueller

Post by Maxim Fomin
You can build shared libraries on linux by manually compiling
object
files and linking them. On windows last time I tries it was not
possible.

Can you give detailed steps for doing this on Linux? Because
nobody as
far as I know has made this work yet?
Jens

Dpaste seems not working, so, sorry for code

----lib.d---
import std.stdio;

static this()
{
writeln("module ctor");
}

static ~this()
{
writeln("module dtor");
}

class A
{
private string text;;
this(string text)
{
writeln("class ctor");
this.text = text;
}
void tell()
{
writeln(this.text);
}
~this()
{
writeln(this.text);
writeln("dtor");
}
static this()
{
writeln("static ctor");
}
static ~this()
{
writeln("static dtor");
}
}
---------------
-----main.d----
import lib;

void main()
{
auto a = new A("some text");
a.tell();
}
---------------

dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
linux-vdso.so.1 (0x00007fff703ff000)
liblib.so => ./liblib.so (0x00007f48158f1000)
librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000)
libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)

[Jacob Carlborg]

Post by Maxim Fomin
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller

Post by Jens Mueller

You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.

Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?
Jens

Dpaste seems not working, so, sorry for code
----lib.d---
import std.stdio;
static this()
{
writeln("module ctor");
}
static ~this()
{
writeln("module dtor");
}
class A
{
private string text;;
this(string text)
{
writeln("class ctor");
this.text = text;
}
void tell()
{
writeln(this.text);
}
~this()
{
writeln(this.text);
writeln("dtor");
}
static this()
{
writeln("static ctor");
}
static ~this()
{
writeln("static dtor");
}
}
---------------
-----main.d----
import lib;
void main()
{
auto a = new A("some text");
a.tell();
}
---------------
dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
linux-vdso.so.1 (0x00007fff703ff000)
liblib.so => ./liblib.so (0x00007f48158f1000)
librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000)
libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)

1. Does this actually run?

2. This is statically linked with druntime and Phobos. What happens when
you create an executable that links with the D dynamic library?

Last time I tried this (on Mac OS X) I got several symbols missing. This
was all symbols that are usually pointing to the executable, inserted by
the compiler. One of them would be "main" and symbols like these:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

[Jens Mueller]

Post by Jacob Carlborg

Post by Maxim Fomin
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller

Post by Jens Mueller

You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.

Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?
Jens

Dpaste seems not working, so, sorry for code
----lib.d---
import std.stdio;
static this()
{
writeln("module ctor");
}
static ~this()
{
writeln("module dtor");
}
class A
{
private string text;;
this(string text)
{
writeln("class ctor");
this.text = text;
}
void tell()
{
writeln(this.text);
}
~this()
{
writeln(this.text);
writeln("dtor");
}
static this()
{
writeln("static ctor");
}
static ~this()
{
writeln("static dtor");
}
}
---------------
-----main.d----
import lib;
void main()
{
auto a = new A("some text");
a.tell();
}
---------------
dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
linux-vdso.so.1 (0x00007fff703ff000)
liblib.so => ./liblib.so (0x00007f48158f1000)
librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000)
libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)

1. Does this actually run?

I just tried.
$ ./a.out
module ctor
static ctor
class ctor
some text
static dtor
module dtor
some text
dtor

Post by Jacob Carlborg
2. This is statically linked with druntime and Phobos. What happens
when you create an executable that links with the D dynamic library?

a.out is linked dynamically against liblib.so.

Post by Jacob Carlborg
Last time I tried this (on Mac OS X) I got several symbols missing.
This was all symbols that are usually pointing to the executable,
inserted by the compiler. One of them would be "main" and symbols
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

I'm running Linux. You can test on Mac OS X.
I'm also astonished that this works. This is great.

Jens

[Maxim Fomin]

On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg

Post by Jacob Carlborg
1. Does this actually run?

If it were non-runnable, I wouldn't posted it.

Post by Jacob Carlborg
2. This is statically linked with druntime and Phobos. What
happens when you create an executable that links with the D
dynamic library?

Solution depends on a problem. I understood Andrei's post that he
wanted a .so file or DLL. I told originally that it is possible
to make shared libraries on linux. Now I see there is some
misunderstanding. Is the problem in diving D application on
executables and shared libraries or making druntime+phobos a
shared library or loading a library at runtime? A was speaking
about the first.

Post by Jacob Carlborg
Last time I tried this (on Mac OS X) I got several symbols
missing. This was all symbols that are usually pointing to the
executable, inserted by the compiler. One of them would be
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

I have no idea about D support of shared libraries on Mac OS.

[Jacob Carlborg]

Post by Maxim Fomin
If it were non-runnable, I wouldn't posted it.

Ok, I see.

Post by Maxim Fomin
Solution depends on a problem. I understood Andrei's post that he wanted
a .so file or DLL. I told originally that it is possible to make shared
libraries on linux. Now I see there is some misunderstanding. Is the
problem in diving D application on executables and shared libraries or
making druntime+phobos a shared library or loading a library at runtime?
A was speaking about the first.

Obviously he wants to use the shared library, otherwise it would be
rather useless and pointless.

Last I checked there was a problem with:

* Using a shared library written in D with a D application
* Making druntime and Phobos shared libraries
* Possibly also compiling shared libraries

[Daniel Kozak]

On Thursday, 27 September 2012 at 09:16:48 UTC, Jacob Carlborg

Post by Jacob Carlborg

Post by Maxim Fomin
If it were non-runnable, I wouldn't posted it.

Ok, I see.

Post by Maxim Fomin
Solution depends on a problem. I understood Andrei's post that he wanted
a .so file or DLL. I told originally that it is possible to
make shared
libraries on linux. Now I see there is some misunderstanding.
Is the
problem in diving D application on executables and shared
libraries or
making druntime+phobos a shared library or loading a library
at runtime?
A was speaking about the first.

Obviously he wants to use the shared library, otherwise it
would be rather useless and pointless.
* Using a shared library written in D with a D application
* Making druntime and Phobos shared libraries
* Possibly also compiling shared libraries

With DMD 2.060 last two points are possible, only first point
(Using a shared ...) I dont try

[Jacob Carlborg]

With DMD 2.060 last two points are possible, only first point (Using a
shared ...) I dont try

I really need to try this when I get home.

[Jacob Carlborg]

Post by Maxim Fomin

Post by Jacob Carlborg
1. Does this actually run?

If it were non-runnable, I wouldn't posted it.

Post by Jacob Carlborg
2. This is statically linked with druntime and Phobos. What happens
when you create an executable that links with the D dynamic library?

Solution depends on a problem. I understood Andrei's post that he wanted
a .so file or DLL. I told originally that it is possible to make shared
libraries on linux. Now I see there is some misunderstanding. Is the
problem in diving D application on executables and shared libraries or
making druntime+phobos a shared library or loading a library at runtime?
A was speaking about the first.

Actually, I seriously doubt everything is working as expected. For
example, what happens when an application loads (via dlopen or similar)
a D dynamic library:

* Are exception handlers registered
* Are module infos properly registered
* What happens if I call Object.factory, will that find a class in the
dynamic library
* Are GC sections registered
* What happens when the library is unloaded, will all of the above be
unregistered and cleaned up

A quick look in rt.minfo in druntime shows the it uses a single array or
linked list for all module infos. It expect the module infos to be in
just one place, i.e. in the application. There's no handling when a
dynamic library is loaded.

Have a look at this:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184

That symbol points to the start of the linked list containing module
infos. That's only a single linked list, no handling of module infos
from multiple places.

rt.minfo needs to be changed to use an associative array with the keys
pointing to images (executables and loaded dynamic libraries) and the
values mapped to module infos of the given image. It also needs to
intercept when a library is dynamically loaded, extract the module infos
and register it with the runtime.

I would think that the same is true for exception handling tables, TLS
and GC sections.

[Maxim Fomin]

On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg

Post by Jacob Carlborg

Post by Maxim Fomin

Post by Jacob Carlborg
1. Does this actually run?

If it were non-runnable, I wouldn't posted it.

Post by Jacob Carlborg
2. This is statically linked with druntime and Phobos. What
happens
when you create an executable that links with the D dynamic
library?

Solution depends on a problem. I understood Andrei's post that he wanted
a .so file or DLL. I told originally that it is possible to
make shared
libraries on linux. Now I see there is some misunderstanding.
Is the
problem in diving D application on executables and shared
libraries or
making druntime+phobos a shared library or loading a library
at runtime?
A was speaking about the first.

Actually, I seriously doubt everything is working as expected.
For example, what happens when an application loads (via dlopen
* Are exception handlers registered
* Are module infos properly registered
* What happens if I call Object.factory, will that find a class
in the dynamic library
* Are GC sections registered
* What happens when the library is unloaded, will all of the
above be unregistered and cleaned up
A quick look in rt.minfo in druntime shows the it uses a single
array or linked list for all module infos. It expect the module
infos to be in just one place, i.e. in the application. There's
no handling when a dynamic library is loaded.
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184
That symbol points to the start of the linked list containing
module infos. That's only a single linked list, no handling of
module infos from multiple places.
rt.minfo needs to be changed to use an associative array with
the keys pointing to images (executables and loaded dynamic
libraries) and the values mapped to module infos of the given
image. It also needs to intercept when a library is dynamically
loaded, extract the module infos and register it with the
runtime.
I would think that the same is true for exception handling
tables, TLS and GC sections.

Posted code doesn't load libraries at runtime, it is just linked
to shared libraries.

[Andrei Alexandrescu]

Posted code doesn't load libraries at runtime, it is just linked to
shared libraries.

Exactly! (I can't believe I'm starting to get the hang of this...) But
what we ultimately need is true dynamic loading of never-seen modules.

After the initial test I tried to load symbols with dlopen(). Indeed
this works:

auto p = dlopen("liblib.so", RTLD_LAZY);

And it does work, all fine. To my dismay, as soon as I removed the flag
"-llib" from the linker command line in the makefile, dlopen() didn't
want to work anymore.

So I think in order to enable "true" dynamic loading, I'll need to
generate PIC for druntime and phobos, and then link liblib.so like this:

dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2

Is that correct?

At that point, of course, a variety of issues will need to be resolved
as people pointed out.


Andrei

[Rob T]

For me to get C or C++ to run a D function, I had to do the
following:

// ====================
// C/C++ library source
// ====================

// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization & shutdown
void init();
void done();

void bar()
{
foo(6, 7, 8);
}

int main(int argc, char **argv)
{
init();
bar();
done();
return 0;
}


// ================
// D library source
// ================
import std.stdio;
static import core.runtime;

// sample D function for test
extern (C++) // int foo(int i, int j, int k)
void foo(int i, int j, int k)
{
writefln("i = %s", i);
writefln("j = %s", j);
writefln("k = %s", k);
int t = i + j + k;
writefln("Total = %s", t);
}

// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after
loading shared lib
core.runtime.Runtime.initialize();
}

extern (C++) export void done() { // to be called before
unloading shared lib
core.runtime.Runtime.terminate();
}


// I had to include main even though this is a library.
int main()
{
return 0;
}

[Jacob Carlborg]

Post by Rob T
// ====================
// C/C++ library source
// ====================
// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization & shutdown
void init();
void done();
void bar()
{
foo(6, 7, 8);
}
int main(int argc, char **argv)
{
init();
bar();
done();
return 0;
}
// ================
// D library source
// ================
import std.stdio;
static import core.runtime;
// sample D function for test
extern (C++) // int foo(int i, int j, int k)
void foo(int i, int j, int k)
{
writefln("i = %s", i);
writefln("j = %s", j);
writefln("k = %s", k);
int t = i + j + k;
writefln("Total = %s", t);
}
// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after loading
shared lib
core.runtime.Runtime.initialize();
}
extern (C++) export void done() { // to be called before unloading
shared lib
core.runtime.Runtime.terminate();
}
// I had to include main even though this is a library.
int main()
{
return 0;
}

That is not sufficient to have everything work. It might be ok if you
limit yourself to a C subset of D. On some platforms, calling
Runtime.initialize, will miss initializing some parts.

The implementation of the runtime initialization contains a lot of code
duplication and some functions/branches are missing functionality. Example:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L346

The C main function handles the runtime initialization for a regular
statically linked executable. If you initialize the runtime via
"rt_init", which is called by "Runtime.initialize", it will miss to
initialize some parts on some platforms. The C main function should
really call "rt_init" to remove the code duplication.

Also see this post why everything will not properly work:

http://forum.dlang.org/thread/k3vfm9$1tq$1 at digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

[Rob T]

On Thursday, 27 September 2012 at 19:57:13 UTC, Andrei

Post by Andrei Alexandrescu
So I think in order to enable "true" dynamic loading, I'll need
to generate PIC for druntime and phobos, and then link
dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2
Is that correct?

Yes, that is correct.

The other thing missing is a phobos2.so for dynamic runtime
linking.

-rt

[Maxim Fomin]

On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg

Post by Jacob Carlborg
Actually, I seriously doubt everything is working as expected.

I tried to check how TLS, EX, etc. (mostly exposed to dll issue)
are working and here is some kind of test:
https://github.com/mxfm/sharedtest. Unfortunately scope(exit)
isn't executed when it is situated in a shared library and which
calls some throwing function from another shared library.
Unittests aren't working either. Regarding other parts - they
seem to work.

[Jacob Carlborg]

I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
executed when it is situated in a shared library and which calls some
throwing function from another shared library. Unittests aren't working
either. Regarding other parts - they seem to work.

That's a fairly uninteresting test. You are linking to the dynamic
library. What's interesting is loading a dynamic library using dlopen,
or similar. What's the point of using dynamic libraries if you're
linking with them?

[Maxim Fomin]

On Saturday, 29 September 2012 at 13:19:01 UTC, Jacob Carlborg

Post by Jacob Carlborg
That's a fairly uninteresting test.

I am not a D developer which means I have no incentive in blindly
portraying D as a language having shared libraries support when
in fact it has some issues. I am a D user which has incentive to
gladly report which part of D does work (or seems to work) and
which doesn't when making shared libraries.

Post by Jacob Carlborg
You are linking to the dynamic library. What's interesting is
loading a dynamic library using dlopen, or similar. What's the
point of using dynamic libraries if you're linking with them?

I was not taking about dynamic loading, but about dynamic
linking. If I understand topic right, the issue is the former,
not the latter.

BTW, in majority cases dynamic loading gives no advantages over
dynamic linking (just the opposite: doing unnecessary job which
can be done by linker and loader). In most cases, when an app is
being written, it is known which functional from which
third-party libraries would be used. The only exceptions I can
name are pluggin support and hacker's binary ELF/PE tools. I can
count few apps in windows and linux which are used often, which
dynamically load something, and if they do, they likely load
pluggins. This was made for cases when app developers by
definition cannot know beforehand full list of used shared
libraries. So, the question is just opposite: "What's the point
of using dynamic loading if you know beforehand which libraries
you use, which happens almost in all cases?".

But I certainly agree that dynamic loading is essential for some
applications, if not irreplaceable, and support of it also needed
to be inspected and improved.

[Jacob Carlborg]

I was not taking about dynamic loading, but about dynamic linking. If I
understand topic right, the issue is the former, not the latter.

The title of the thread says "... and loading".

BTW, in majority cases dynamic loading gives no advantages over dynamic
linking (just the opposite: doing unnecessary job which can be done by
linker and loader). In most cases, when an app is being written, it is
known which functional from which third-party libraries would be used.

I agree, in these cases their no point in using dynamic loading.

The only exceptions I can name are pluggin support and hacker's binary
ELF/PE tools. I can count few apps in windows and linux which are used
often, which dynamically load something, and if they do, they likely
load pluggins. This was made for cases when app developers by definition
cannot know beforehand full list of used shared libraries.

Exactly, pluggins are one of the main reasons for using dynamic loading.
An other reason is license issues. There's a D library called Derelict
which uses dynamic loading to load C libraries (OpenGL, SDL, OpenAL and
others). One of the many reason for Derelict using dynamic loading is
due to license issues/complications with these libraries.

http://dsource.org/projects/derelict/

So, the question is just opposite: "What's the point of using dynamic loading if
you know beforehand which libraries you use, which happens almost in all
cases?".

There's no point in doing that. But what I'm saying is if you know
beforehand the libraries you will use you can get quite far with static
libraries.

[Rob T]

On Saturday, 29 September 2012 at 16:09:14 UTC, Jacob Carlborg

Post by Jacob Carlborg
There's no point in doing that. But what I'm saying is if you
know beforehand the libraries you will use you can get quite
far with static libraries.

There are plenty of cases where you have to use a dynamically
loaded lib even if you know before hand what will be loaded. I
think you understand this if I read your posts correctly.

In my case I have a pre-existing C++ app that is designed to load
user defined C++ plugins. I wanted to interface D to one of my
own C++ plugins, but I apparently cannot do it reliably because
of initialization issues with the GC and possibly some other
obscure items.

If I can figure out what needs to be done to resolve the problem
in enough detail, then maybe I can hack the runtime source and
roll out a solution.

The GC always seems to pop up as a source of problems. For long
term solution, the GC should be made 100% optional (in practice
as opposed to in theory), the standard library ought to be made
to work with or wothout a GC (or simply without), and the GC
itself should be easily replaceable with alternate versions. I
think this idea has already been discussed elsewhere, and is on
the TODO list (I hope!).

--rt

[Jacob Carlborg]

There are plenty of cases where you have to use a dynamically loaded lib
even if you know before hand what will be loaded. I think you understand
this if I read your posts correctly.
In my case I have a pre-existing C++ app that is designed to load user
defined C++ plugins. I wanted to interface D to one of my own C++
plugins, but I apparently cannot do it reliably because of
initialization issues with the GC and possibly some other obscure items.

But now we're back at plugins. I think this part of the discussion is
starting to run in circles and become quite pointless. I think most of
us know that we need to support all types of libraries. Static and
dynamic, both for link time and runtime.

If I can figure out what needs to be done to resolve the problem in
enough detail, then maybe I can hack the runtime source and roll out a
solution.

You can have a look at the work done by Martin Nowak:

https://github.com/dawgfoto/druntime/commits/SharedRuntime

He has a couple of other useful branches as well, for example:

https://github.com/dawgfoto/druntime/commits/dmain2Refactoring

On Mac OS X there's also the problem with TLS. There is no native
support for TLS in Mac OS X prior to Lion (10.7). DMD has rolled its own
implementation that needs to be adapted to support dynamic libraries.

The GC always seems to pop up as a source of problems. For long term
solution, the GC should be made 100% optional (in practice as opposed to
in theory), the standard library ought to be made to work with or
wothout a GC (or simply without), and the GC itself should be easily
replaceable with alternate versions. I think this idea has already been
discussed elsewhere, and is on the TODO list (I hope!).

You can replace the GC at link time. Here's an example of a stubbed
implemented of the GC:

http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub

This is for Tango but the druntime is based on the Tango runtime so I
would guess most of this would be the same.

[Rob T]

On Sunday, 30 September 2012 at 10:57:24 UTC, Jacob Carlborg

I think most of us know that we need to support all types of
libraries. Static and dynamic, both for link time and runtime.

OK we're all in agreement on this point.

https://github.com/dawgfoto/druntime/commits/SharedRuntime

It seems that an attempt to make the runtime shared is well under
way. Did anything get into the main dmd branch or has the effort
been stalled or ...?

I've only recently been learning what's going on with D and why
it is what it is, so a breif history lesson may be needed once in
a while.

You can replace the GC at link time. Here's an example of a
http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub

I will look at this too. Thanks for the pointers.

--rt

[Jacob Carlborg]

It seems that an attempt to make the runtime shared is well under way.
Did anything get into the main dmd branch or has the effort been stalled
or ...?

Seems pretty stalled.

I will look at this too. Thanks for the pointers.

No problem.

[Johannes Pfau]

Am Sat, 29 Sep 2012 15:19:30 +0200

Post by Jacob Carlborg

I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
executed when it is situated in a shared library and which calls
some throwing function from another shared library. Unittests
aren't working either. Regarding other parts - they seem to work.

That's a fairly uninteresting test. You are linking to the dynamic
library. What's interesting is loading a dynamic library using
dlopen, or similar. What's the point of using dynamic libraries if
you're linking with them?

There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)

So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test

It should also be enhanced to test multiple shared libraries.

The good news:
* Exception handling is working
* ModuleInfos are working
* unit tests are working
* Static variables, gshared variables, tls variables are working
* Object.factory is working
* Calling functions, passing function pointers, passing classes between
dso/app is working

The bad news:
* The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
it only scans the main app.

[Jacob Carlborg]

Post by Johannes Pfau
There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)
So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test
It should also be enhanced to test multiple shared libraries.
* Exception handling is working
* ModuleInfos are working
* unit tests are working
* Static variables, gshared variables, tls variables are working
* Object.factory is working
* Calling functions, passing function pointers, passing classes between
dso/app is working

That's good. Have you tested this with DMD?

Post by Johannes Pfau
* The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it won't
be that easy on Linux.

BTW, is the runtime and phobos statically linked both with the dynamic
library and the executable?

[Iain Buclaw]

Post by Jacob Carlborg

Post by Johannes Pfau
There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)
So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test
It should also be enhanced to test multiple shared libraries.
* Exception handling is working
* ModuleInfos are working
* unit tests are working
* Static variables, gshared variables, tls variables are working
* Object.factory is working
* Calling functions, passing function pointers, passing classes between
dso/app is working

That's good. Have you tested this with DMD?

Post by Johannes Pfau
* The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it won't be
that easy on Linux.
BTW, is the runtime and phobos statically linked both with the dynamic
library and the executable?
--
/Jacob Carlborg

On Linux, there has already been an runtime implementation written
that scans /proc/self/maps and adds all data sections to the GC that
way. Whether or not DMD wishes to go down that route is their own
decision. I am looking into a solution that doesn't have any bearing
on what platform it's running on...


Regards

[Jacob Carlborg]

Post by Iain Buclaw
On Linux, there has already been an runtime implementation written
that scans /proc/self/maps and adds all data sections to the GC that
way. Whether or not DMD wishes to go down that route is their own
decision. I am looking into a solution that doesn't have any bearing
on what platform it's running on...

Well, /proc isn't available on Mac OS X so I think you have to continue
looking.

[Jacob Carlborg]

Post by Iain Buclaw
On Linux, there has already been an runtime implementation written
that scans /proc/self/maps and adds all data sections to the GC that
way. Whether or not DMD wishes to go down that route is their own
decision. I am looking into a solution that doesn't have any bearing
on what platform it's running on...

I think this is already working on Mac OS X. It's handled by:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L82

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L133

[Johannes Pfau]

Am Mon, 01 Oct 2012 14:34:21 +0200

Post by Jacob Carlborg

Post by Johannes Pfau
There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)
So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test
It should also be enhanced to test multiple shared libraries.
* Exception handling is working
* ModuleInfos are working
* unit tests are working
* Static variables, gshared variables, tls variables are working
* Object.factory is working
* Calling functions, passing function pointers, passing classes
between dso/app is working

That's good. Have you tested this with DMD?

Not yet.

Post by Jacob Carlborg

Post by Johannes Pfau
* The GC doesn't scan TLS/__gshared/static data in dynamic
libraries, it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it
won't be that easy on Linux.
BTW, is the runtime and phobos statically linked both with the
dynamic library and the executable?

I tested two different configurations:

druntime and phobos are shared libraries as well (this is the correct
solution, it's mostly working except for the GC issues)

druntime and phobos static linking: phobos and druntime are statically
linked into the app, libdso.so is not linked against druntime/phobos at
all. (this is a hack to get better test results: With a shared
druntime, you can't call GC.collect twice, because the first call
frees important objects in druntime and then the second call segfaults)

[Jacob Carlborg]

Post by Johannes Pfau
druntime and phobos are shared libraries as well (this is the correct
solution, it's mostly working except for the GC issues)

I agree.

Post by Johannes Pfau
druntime and phobos static linking: phobos and druntime are statically
linked into the app, libdso.so is not linked against druntime/phobos at
all. (this is a hack to get better test results: With a shared
druntime, you can't call GC.collect twice, because the first call
frees important objects in druntime and then the second call segfaults)

Ok.

[Jacob Carlborg]

Post by Johannes Pfau
There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)
So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's
where the implementation of "rt_init" is located.

[Johannes Pfau]

Am Mon, 01 Oct 2012 15:11:49 +0200

Post by Jacob Carlborg

Post by Johannes Pfau
There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)
So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's
where the implementation of "rt_init" is located.

the problem is that we don't want the C main function in a shared
libgdruntime.so, because you might want to use libgdruntime.so in a
C/C++ app which has it's own main function.

So we currently don't link in dmain2.o into the shared library and it
must be included manually when linking an application.
(But dmain2 also contains some stuff that really should be in
libdruntime.so, so this source file should probably be split up at some
time.)

[Iain Buclaw]

Post by Johannes Pfau
Am Mon, 01 Oct 2012 15:11:49 +0200

Post by Jacob Carlborg

Post by Johannes Pfau
There are some reasons for dynamic libraries linked at compile time,
one is that we have to start somewhere and they are required for
plugins / dynamically loaded libraries as well ;-)
So I started a small test suite for GDC (could be adapted to other
compilers). It currently only tests compile time linking of dynamic
libraries, but adjusting the test to use runtime loading should be
easy. But it's pointless as long as we have no runtime support.
https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's
where the implementation of "rt_init" is located.

the problem is that we don't want the C main function in a shared
libgdruntime.so, because you might want to use libgdruntime.so in a
C/C++ app which has it's own main function.

That is at least one of the theories behind it anyway. :-)

The more I think about it, the less I think I would want a C++ app to
link against a D shared library though.

With C, there will need to be defined a common interface header -
similar to what used to be mars.h in rt/ but also contain some useful
runtime functions.


Regards

[Jacob Carlborg]

Post by Johannes Pfau
the problem is that we don't want the C main function in a shared
libgdruntime.so, because you might want to use libgdruntime.so in a
C/C++ app which has it's own main function.
So we currently don't link in dmain2.o into the shared library and it
must be included manually when linking an application.
(But dmain2 also contains some stuff that really should be in
libdruntime.so, so this source file should probably be split up at some
time.)

I'm not sure if I follow this correctly or not, but why is this needed
to be handled manually? If you pass "-shared" to the compiler just skip
linking dmain2.o, otherwise link with it. Would that work?

[Johannes Pfau]

Am Mon, 01 Oct 2012 19:18:46 +0200

Post by Jacob Carlborg

Post by Johannes Pfau
the problem is that we don't want the C main function in a shared
libgdruntime.so, because you might want to use libgdruntime.so in a
C/C++ app which has it's own main function.
So we currently don't link in dmain2.o into the shared library and
it must be included manually when linking an application.
(But dmain2 also contains some stuff that really should be in
libdruntime.so, so this source file should probably be split up at
some time.)

I'm not sure if I follow this correctly or not, but why is this
needed to be handled manually? If you pass "-shared" to the compiler
just skip linking dmain2.o, otherwise link with it. Would that work?

Yes something similar would work, it's just not yet implemented.

GDC should detect if we're linking against a shared druntime and then
it should automatically add dmain2.o to the linker command.

[Bottled Gin]

Post by Jacob Carlborg
Actually, I seriously doubt everything is working as expected.
For example, what happens when an application loads (via dlopen
* Are exception handlers registered
* Are module infos properly registered
* What happens if I call Object.factory, will that find a class
in the dynamic library
* Are GC sections registered
* What happens when the library is unloaded, will all of the
above be unregistered and cleaned up

Hello D Experts

I am replying to an old thread since I want to know if and how
the situation has improved over the past few months. In
particular I want to link and call fairly complex D functions
from a C++ application.

I am exclusively using Linux. Can somebody please guide me if I
should expect things to work with current DMD master from github?

Regards
- Puneet

[1100110]

Post by Bottled Gin

Post by Jacob Carlborg
Actually, I seriously doubt everything is working as expected. For
example, what happens when an application loads (via dlopen or
* Are exception handlers registered
* Are module infos properly registered
* What happens if I call Object.factory, will that find a class in the
dynamic library
* Are GC sections registered
* What happens when the library is unloaded, will all of the above be
unregistered and cleaned up

Hello D Experts
I am replying to an old thread since I want to know if and how the
situation has improved over the past few months. In particular I want to
link and call fairly complex D functions from a C++ application.
I am exclusively using Linux. Can somebody please guide me if I should
expect things to work with current DMD master from github?
Regards
- Puneet

There's a DConf talk about shared libraries in D.

Perhaps there's useful information there?

(I'm watching it now so I don't know how detailed it is.)

[Etienne]

Post by 1100110

Post by Bottled Gin

Post by Jacob Carlborg
Actually, I seriously doubt everything is working as
expected. For
example, what happens when an application loads (via dlopen or
* Are exception handlers registered
* Are module infos properly registered
* What happens if I call Object.factory, will that find a
class in the
dynamic library
* Are GC sections registered
* What happens when the library is unloaded, will all of the
above be
unregistered and cleaned up

Hello D Experts
I am replying to an old thread since I want to know if and how the
situation has improved over the past few months. In particular I want to
link and call fairly complex D functions from a C++
application.
I am exclusively using Linux. Can somebody please guide me if
I should
expect things to work with current DMD master from github?
Regards
- Puneet

There's a DConf talk about shared libraries in D.
Perhaps there's useful information there?
http://youtu.be/i63VeudjZM4
(I'm watching it now so I don't know how detailed it is.)

Hello all,

I'm wondering if this has been resolved to date. I get this error
on master:

/usr/bin/ld: generated/linux/release/64/libphobos2.so.0.66.o:
relocation R_X86_64_32 against `.rodata' can not be used when
making a shared object; recompile with -fPIC
generated/linux/release/64/libphobos2.so.0.66.o: could not read
symbols: Bad value
collect2: error: ld returned 1 exit status

even though I've built with fPIC:

cc -c -m64 -fPIC -O3 etc/c/zlib/trees.c
-ogenerated/linux/release/64/etc/c/zlib/trees.o
cc -c -m64 -fPIC -O3 etc/c/zlib/uncompr.c
-ogenerated/linux/release/64/etc/c/zlib/uncompr.o
cc -c -m64 -fPIC -O3 etc/c/zlib/zutil.c
-ogenerated/linux/release/64/etc/c/zlib/zutil.o

etc...

[Maxim Fomin]

On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg

Post by Jacob Carlborg
Last time I tried this (on Mac OS X) I got several symbols
missing. This was all symbols that are usually pointing to the
executable, inserted by the compiler. One of them would be
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

"_deh_" problem AFAIK arise when non-D and D code is merged and
can be avoided. For example, assume C/C++ application using D
library. This can be done by writing D main function which
forwards call to C/C++ main function which uses D shared library
code. Again, sorry for code post, dpaste isn't working:
---main.cpp---
extern "C" void foo(void);

#include <stdio.h>

class CPP
{
public:
CPP() { printf("CPP ctor\n"); }
};

CPP cpp;

extern "C" int c_main_func()
{
printf("C main function reached\n");
foo();
// C/C++ application
return 0;
}
---dmain.d---
import std.stdio;

extern(C) int c_main_func();

static this()
{
writeln("main module ctor");
}

static ~this()
{
writeln("main module dtor");
}

int main()
{
return c_main_func();
}
---test.d---
import std.stdio;

static this()
{
writeln("lib module ctor");
}

static ~this()
{
writeln("lib module dtor");
}

extern(C) void foo()
{
auto a = new A("data");
writeln("in foo");

}

class A
{
string x;
static this() { writeln("static ctor"); }
static ~this() { writeln("static dtor"); }
this(string x) { writeln("ctor"); this.x = x; }
~this() { writeln("dtor"); writeln(x); }
}
------
# g++ main.cpp -c
# dmd -c dmain.d
# dmd -c test.d -fPIC
# gcc -shared test.o -o libtest.so
# gcc dmain.o main.o -ltest -lphobos2 -lpthread -lrt -L.
-Wl,-rpath=.
# ./a.out
CPP ctor
main module ctor
lib module ctor
static ctor
C main function reached
ctor
in foo
static dtor
lib module dtor
main module dtor
dtor
data
# ldd ./a.out
linux-vdso.so.1 (0x00007fff4cd55000)
libtest.so => ./libtest.so (0x00007f970520b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9704fd3000)
librt.so.1 => /lib64/librt.so.1 (0x00007f9704dcb000)
libc.so.6 => /lib64/libc.so.6 (0x00007f9704a26000)
/lib64/ld-linux-x86-64.so.2 (0x00007f970540e000)
#

[Andrei Alexandrescu]

On 9/27/12 6:26 AM, Maxim Fomin wrote:
[snip]

Thanks! I adapted your code as follows and it works with 2.058 on Centos.

*** lib.d
import std.stdio;

extern(C) int fun()
{
writeln(", world!");
return 42;
}

*** main.d
import std.stdio;

extern(C) int fun();

void main()
{
write("Hello");
fun();
}

*** Makefile

PHOBOS_PATH=/path/to/phobos/

main: main.o liblib.so
gcc main.o -llib -lphobos2 -lpthread -lrt -L. -L${PHOBOS_PATH}
-Wl,-rpath=. -o main

liblib.so: lib.d
dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so

%.o: %.d
dmd -c $<

clean:
rm -f main main.o lib.o liblib.so

Building the makefile and running ./main prints the classic message.
Thanks all for answering, this will get us going.


Andrei

[Jacob Carlborg]

Post by Maxim Fomin
[snip]
Thanks! I adapted your code as follows and it works with 2.058 on Centos.

I seriously doubt that everything is working properly, have a look at my
reply to Maxim Fomin:
http://forum.dlang.org/thread/k3vfm9$1tq$1 at digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

[nazriel]

Post by Maxim Fomin
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller

Post by Jens Mueller

Post by Maxim Fomin
You can build shared libraries on linux by manually compiling
object
files and linking them. On windows last time I tries it was
not
possible.

Can you give detailed steps for doing this on Linux? Because
nobody as
far as I know has made this work yet?
Jens

Dpaste seems not working, so, sorry for code
[snip]

Hmm, that looks nice, seems that stuff improved from last time I
checked.

Gonna test this out today. I have project that would gain from
"Hot swapping" of Shared libraries. Temporarily solution I made
was just writing plugins in C++.

Btw, sorry for OT.
What exactly doesn't work in Dpaste? It seems to work fine for
me(TM).
If those are UI glitches, try pressing F5 2-3 times ;D

[Maxim Fomin]

Post by nazriel
Btw, sorry for OT.
What exactly doesn't work in Dpaste? It seems to work fine for
me(TM).
If those are UI glitches, try pressing F5 2-3 times ;D

Dpaste was completely down when I tried to post code. It was in
the middle of today.

[Daniel Kozak]

Hi,

I have same issue, but it is possible make shared library,
first of all you have to make shared variant of druntime and
phobos library, than it should work ok.

Now I am at work, when I come back home I will post some more
details about this.

Daniel Kozak

On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm
using dmd 2.058/Linux at work to build and use dynamic
*** Makefile
all: main lib.so
main: main.d
dmd main
lib.so: lib.d
dmd -fPIC -shared lib.d -of./lib.so
*** lib.d
extern(C) int fun(string s)
{
return 42;
}
*** main.d
import std.stdio;
void main()
{
}
dmd -fPIC -shared lib.d -of./lib.so
relocation R_X86_64_32 against `a local symbol' can not be used
when making a shared object; recompile with -fPIC
could not read symbols: Bad value
collect2: ld returned 1 exit status
What steps do I need to take to get off the ground?
Thanks,
Andrei

[Daniel Kozak]

Now I try it, and it is not required to build shared variant of
druntime and phobos, only rebuild it with -fPIC



On Thursday, 27 September 2012 at 06:12:38 UTC, Daniel Kozak

Post by Daniel Kozak
Hi,
I have same issue, but it is possible make shared library,
first of all you have to make shared variant of druntime and
phobos library, than it should work ok.
Now I am at work, when I come back home I will post some more
details about this.
Daniel Kozak
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei

Post by Andrei Alexandrescu
Haven't done any dynamic linking with D and I need to. I'm
using dmd 2.058/Linux at work to build and use dynamic
*** Makefile
all: main lib.so
main: main.d
dmd main
lib.so: lib.d
dmd -fPIC -shared lib.d -of./lib.so
*** lib.d
extern(C) int fun(string s)
{
return 42;
}
*** main.d
import std.stdio;
void main()
{
}
dmd -fPIC -shared lib.d -of./lib.so
relocation R_X86_64_32 against `a local symbol' can not be
used when making a shared object; recompile with -fPIC
could not read symbols: Bad value
collect2: ld returned 1 exit status
What steps do I need to take to get off the ground?
Thanks,
Andrei

[Johannes Pfau]

Am Thu, 27 Sep 2012 08:26:36 +0200

Post by Daniel Kozak
Now I try it, and it is not required to build shared variant of
druntime and phobos, only rebuild it with -fPIC

In the end you'll probably need a shared druntime & phobos: Let's say
your main app doesn't use std.datetime and you statically link
libphobos. Then the linker might strip std.datetime from your
executable. If your shared library now needs std.datetime it won't
work.

The funny part is that on ARM+GDC we can at least build druntime &
phobos as a shared library, but on x86/64 last time I checked it didn't
work because of non-PIC assembly.

https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support

[Rob T]

On Thursday, 27 September 2012 at 07:54:29 UTC, Johannes Pfau

Post by Johannes Pfau
Am Thu, 27 Sep 2012 08:26:36 +0200

Post by Daniel Kozak
Now I try it, and it is not required to build shared variant
of druntime and phobos, only rebuild it with -fPIC

Let's say
your main app doesn't use std.datetime and you statically link
libphobos. Then the linker might strip std.datetime from your
executable. If your shared library now needs std.datetime it
won't
work.

I'm using C++ main app that dynamically loads shared libs at
runtime, but I cannot load D shared libs because they cannot be
linked to the non-fPIC built phobos2/druntime lib. It seems that
the only way to get things working is to re-build the C++ code so
that it is linked with non-fPIC phobos2, but this is not
something I want to be doing.

So if I read this right, I can build phobos/druntime with -fPIC
from the source code and it will work OK? If so, then why was
this not done with the latest distrubution package, or is this
only possible after a certain version number?

--rt

[Jacob Carlborg]

I'm using C++ main app that dynamically loads shared libs at runtime,
but I cannot load D shared libs because they cannot be linked to the
non-fPIC built phobos2/druntime lib. It seems that the only way to get
things working is to re-build the C++ code so that it is linked with
non-fPIC phobos2, but this is not something I want to be doing.
So if I read this right, I can build phobos/druntime with -fPIC from the
source code and it will work OK? If so, then why was this not done with
the latest distrubution package, or is this only possible after a
certain version number?

No, it will not work ok. Depending on what you do in the D code it might
work, but everything will not work. Perhaps if you limit yourself to a C
subset. See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1 at digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

[Johannes Pfau]

Am Fri, 28 Sep 2012 02:04:11 +0200

Post by Rob T
So if I read this right, I can build phobos/druntime with -fPIC
from the source code and it will work OK? If so, then why was
this not done with the latest distrubution package, or is this
only possible after a certain version number?
--rt

It shouldn't compile on x86 (32bit) with -fPIC because there's some
incompatible asm code in druntime and phobos. If dmd doesn't warn in
these cases, this code will fail as soon as it's called.

GDC used to warn about these errors, but as we dropped the D inline
assembler that code doesn't affect us anymore and there are no more
warnings.

The fix for this issue is simple: Do not modify the EBX register, or at
least save & restore it.

Old patch, which shows what needs to be done:
https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Example asm code which probably doesn't work:
https://github.com/D-Programming-Language/druntime/blob/master/src/core/cpuid.d#L432


I just tried building a shared druntime & phobos with gdc:
Druntime seems to be working, Phobos doesn't link here because
something's wrong with the fstat64 symbol. I haven't done any real
testing and we probably need some additional support in the runtime, but
the compiler part seems to be OK with gdc.
http://gdcproject.org/wiki/SharedRuntime
https://github.com/D-Programming-GDC/gdc/pull/32

[Iain Buclaw]

Post by Johannes Pfau
Am Fri, 28 Sep 2012 02:04:11 +0200

Post by Rob T
So if I read this right, I can build phobos/druntime with -fPIC
from the source code and it will work OK? If so, then why was
this not done with the latest distrubution package, or is this
only possible after a certain version number?
--rt

It shouldn't compile on x86 (32bit) with -fPIC because there's some
incompatible asm code in druntime and phobos. If dmd doesn't warn in
these cases, this code will fail as soon as it's called.
GDC used to warn about these errors, but as we dropped the D inline
assembler that code doesn't affect us anymore and there are no more
warnings.
The fix for this issue is simple: Do not modify the EBX register, or at
least save & restore it.

In GDC D1 phobos, one hack around it was to use db to manually write
out the bytecode instruction (thus bypassing GCC 'ebx' clobbered
checks). Which was a fiendish workaround to say the least. :~)

[Rob T]

I am trying out gdc 4.7 branch with -fPIC, but the info I'm
getting is that even with a successulf PIC build I will still not
be able to reliably execute D functions directly from C/C++ code.

Post by Johannes Pfau
https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Does anyone know why patches like this are taking so [bleeping]
long to get recognized and implemented?

-rt

[Iain Buclaw]

I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that
even with a successulf PIC build I will still not be able to reliably
execute D functions directly from C/C++ code.

Post by Johannes Pfau
https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Does anyone know why patches like this are taking so [bleeping] long to get
recognized and implemented?
-rt

Because the [bleeping] runtime does not support running from a shared library.

[Rob T]

Post by Iain Buclaw
Because the [bleeping] runtime does not support running from a
shared library.

I suppose the answer is very complicated, but why can't the
runtime execute as a shared lib? Is it a design limitation of the
runtime model? This sort of problem should have been dealt with
from the ground up, not now. What can be done to fix it, or is it
a major redesign + re-write effort?

-rt

[Iain Buclaw]

Post by Iain Buclaw
Because the [bleeping] runtime does not support running from a shared
library.

I suppose the answer is very complicated, but why can't the runtime execute
as a shared lib? Is it a design limitation of the runtime model? This sort
of problem should have been dealt with from the ground up, not now. What can
be done to fix it, or is it a major redesign + re-write effort?
-rt

The big implementation issue is that the runtime only passes two data
ranges to the GC to scan. Global data segment and TLS data segment of
the resultant binary/executable. So any static data stored in shared
libraries aren't scanned / recognised as data we want to keep. So the
use of them may cause sudden unwanted collections.

The way I intend to address it is to have each module handle it's own
gshared/thread data and pass the information to the D runtime during
the module construction stage (.ctor) - there is already something
done this way for _Dmodule_ref - so it may end up being that two new
fields will be tacked onto it; void[] tlsdata, void[] gsharedata;


Regards

[Jacob Carlborg]

I suppose the answer is very complicated, but why can't the runtime
execute as a shared lib? Is it a design limitation of the runtime model?
This sort of problem should have been dealt with from the ground up, not
now. What can be done to fix it, or is it a major redesign + re-write
effort?

See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1 at digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

[Rob T]

Post by Iain Buclaw
The way I intend to address it is to have each module handle
it's own
gshared/thread data and pass the information to the D runtime
during
the module construction stage (.ctor) - there is already
something
done this way for _Dmodule_ref - so it may end up being that
two new
fields will be tacked onto it; void[] tlsdata, void[]
gsharedata;

I re-built libgphobos and libgdruntime with -fPIC and I can now
successfully create dynamically loaded D libs. I have
successfully linked a dynamic D lib to a C++ app, executing some
test code successfully. I have not yet tried to dlopen a dynamic
D lib from C++, but I will try maybe tomorrow.

My simple dynamic lib test seems to run fine, but I understand
that there may be problems, such as the GC failing to work
properly, and perhaps more can/will go wrong as per Jacob's post
http://forum.dlang.org/post/k4219f$uft$1 at digitalmars.com

Any idea when/if you will get around to implementing a fix? Wish
I could help but I've only just started looking at the source
code, so whatever I try to fix will probably cause more harm than
good for a while (but it's a start).

--rt

[Andrei Alexandrescu]

Now I try it, and it is not required to build shared variant of druntime
and phobos, only rebuild it with -fPIC

Could you please send a troika composed of one dynlib, one loader using
it, and a makefile that puts the all together?

Thanks much!


Andrei