GC object finalization not guaranteed

[Leandro Lucarella]

I've just found out[1] this[2]:

The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.

Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

[1] http://proj.llucax.com.ar/blog/dgc/blog/post/-43101db1
[2] http://www.digitalmars.com/d/1.0/class.html#destructors

[Robert Jacques]

On Sat, 18 Apr 2009 11:24:14 -0400, Leandro Lucarella <llucax at gmail.com>

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

Well, a couple of quick tests show that under normal situations (i.e.
normal program termination and termination from an exception) the
finalizers do run (D2). However, if a finalizer throws an exception, then
the rest of the finalizers aren't called. Also, if you call
std.c.stdlib.exit, the finalizers won't run.

[Leandro Lucarella]

Post by Robert Jacques

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

Well, a couple of quick tests show that under normal situations (i.e.
normal program termination and termination from an exception) the
finalizers do run (D2). However, if a finalizer throws an exception,
then the rest of the finalizers aren't called. Also, if you call
std.c.stdlib.exit, the finalizers won't run.

Well, I'm not talking about experimentation, I'm talking about source code
;)

The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

Here is a simple example that demonstrate the case:

$ cat finalize.d

version (Tango)
import tango.stdc.stdio;
else
import std.c.stdio;

class A
{
~this() { printf("term\n"); }
}

A a;

static this() { a = new A; }

void main()
{
version (FREE)
a = null;
}

$ for compiler in dmd dmd2 gdmd; do for version in '' '-version=FREE'; do $compiler $version finalize.d; echo $compiler $version; ./finalize; echo; done; done
dmd

dmd -version=FREE
term

dmd2

dmd2 -version=FREE
term

gdmd

gdmd -version=FREE
term

(I have my ldc copy broken right now, that's why I didn't test that
compiler ;)


This can happen behind your back too if some nice non-pointer value in the
stack or static data unfortunately is a valid pointer.

This example shows that:

$ cat finalize2.d

version (Tango)
import tango.stdc.stdio;
else
import std.c.stdio;

class A
{
~this() { printf("term\n"); }
}

size_t x;

void main()
{
A a = new A;
x = cast(size_t) cast(void*) a;
version (FREE)
x = 0;
}

(the results are the same as the previous example)

[Don]

Post by Leandro Lucarella

Post by Robert Jacques

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

Well, a couple of quick tests show that under normal situations (i.e.
normal program termination and termination from an exception) the
finalizers do run (D2). However, if a finalizer throws an exception,
then the rest of the finalizers aren't called. Also, if you call
std.c.stdlib.exit, the finalizers won't run.

Well, I'm not talking about experimentation, I'm talking about source code
;)
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

I don't understand why D even has finalizers. Can they do anything useful?

[Leandro Lucarella]

Post by Don

Post by Leandro Lucarella
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

I don't understand why D even has finalizers. Can they do anything useful?

Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Of course that can be done manually if you are well organized when
programming, but then, you can do that with memory too, and the whole GC
loose its usefulness, so I guess is a valid argument for other things too
=)

[Walter Bright]

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

[Leandro Lucarella]

Post by Walter Bright

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

Then, why allowing finalizers in non-scope objects? It's a little
confusing...

[Daniel Keep]

Post by Walter Bright

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

But you can't tell in a dtor whether you're being destroyed
deterministically or not. The only safe assumption is that you aren't,
thus rendering dtors worse than useless.

All we'd need is a bool passed to a dtor to tell it whether it's being
destroyed deterministically or not and all will be rainbows and sunshine.

... except for the opDotExpr discussion. Anyone who posts to that
thread is kuh-razy!

-- Daniel

[Christopher Wright]

Post by Daniel Keep

Post by Walter Bright

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

But you can't tell in a dtor whether you're being destroyed
deterministically or not. The only safe assumption is that you aren't,
thus rendering dtors worse than useless.
All we'd need is a bool passed to a dtor to tell it whether it's being
destroyed deterministically or not and all will be rainbows and sunshine.
... except for the opDotExpr discussion. Anyone who posts to that
thread is kuh-razy!
-- Daniel

That should be a relatively easy change, actually, and it's a pretty
good idea.

[Michel Fortin]

Post by Daniel Keep

Post by Walter Bright

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

But you can't tell in a dtor whether you're being destroyed
deterministically or not. The only safe assumption is that you aren't,
thus rendering dtors worse than useless.

Actually you could declare your class as scope, that'd make sure every
instance is scope and gets destructed in a timely manner.

Hum, perhaps destructors should only be allowed on scope classes. :-)

[Daniel Keep]

Post by Michel Fortin

Post by Daniel Keep

Post by Walter Bright

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

But you can't tell in a dtor whether you're being destroyed
deterministically or not. The only safe assumption is that you aren't,
thus rendering dtors worse than useless.

Actually you could declare your class as scope, that'd make sure every
instance is scope and gets destructed in a timely manner.

Except that's a pain in the butt to actually use. You can't store them
in other classes or structs and you have to create all objects at the
highest scope they're used. I proposed changes to let scope instances
to be returned from functions and stored in other scope classes to make
them more useful, but that never went anywhere. :P

Post by Michel Fortin
Hum, perhaps destructors should only be allowed on scope classes. :-)

Probably, but only assuming scope objects are made more usable.

-- Daniel

[Fawzi Mohamed]

Post by Don

Post by Leandro Lucarella

Post by Robert Jacques

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

when the main thread ends other threads might be running, and removing
memory they are using is not necessarily a good idea.
Exceptions might delay things indefinitely and allocate more memory.

Actually with tango you can tweak what will happen with
gc_getTermCleanupLevel (in gc.d), but I don't think that there is a
satisfying solution for everybody. I think that the default in tango is
quite reasonable (normal collect).

Post by Don

Post by Leandro Lucarella

Post by Robert Jacques
Well, a couple of quick tests show that under normal situations (i.e.
normal program termination and termination from an exception) the
finalizers do run (D2). However, if a finalizer throws an exception,
then the rest of the finalizers aren't called. Also, if you call
std.c.stdlib.exit, the finalizers won't run.

Well, I'm not talking about experimentation, I'm talking about source code
;)
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

I don't understand why D even has finalizers. Can they do anything useful?

Two things that I acn think of, there might be more:
- Release external resources. Well probably it is better not to rely on
them if the resource is finite (basically always), but it can be useful
at times
- weak refs

[Leandro Lucarella]

Post by Fawzi Mohamed

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

when the main thread ends other threads might be running, and removing
memory they are using is not necessarily a good idea.

Then I guess gc_term() should be invoked when all threads have terminated.
Is there any technical difficulty to do that?

Post by Fawzi Mohamed
Exceptions might delay things indefinitely and allocate more memory.

Then I guess gc_term() should be invoked when all exceptions were
processed. Is there any technical difficulty to do that?

I really didn't take a look at the compiler runtime stuff, so I don't
really know how all that works.

Post by Fawzi Mohamed
Actually with tango you can tweak what will happen with
gc_getTermCleanupLevel (in gc.d), but I don't think that there is
a satisfying solution for everybody.

Yes, I saw that (I commented about that in the blog post), but there is
still no option to run the finalizers to all the live objects.

Anyways, I wonder why it's not part of the GC specs in druntime.

Post by Fawzi Mohamed
I think that the default in tango is quite reasonable (normal collect).

Yes... With the current specs at least, I guess.

I think that if finalization is not guarantee, at least at
program exit, there shouldn't be there. It's useless.

Finalization should be possible only for scope classes if no finalization
guarantee can be provided by the GC. It's useless and error prone
(introducing a bug very difficult to track). Who wants randomly executed
destructors?

[Leandro Lucarella]

Post by Leandro Lucarella

Post by Fawzi Mohamed

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

when the main thread ends other threads might be running, and removing
memory they are using is not necessarily a good idea.

Then I guess gc_term() should be invoked when all threads have terminated.
Is there any technical difficulty to do that?

Post by Fawzi Mohamed
Exceptions might delay things indefinitely and allocate more memory.

Then I guess gc_term() should be invoked when all exceptions were
processed. Is there any technical difficulty to do that?
I really didn't take a look at the compiler runtime stuff, so I don't
really know how all that works.

Ok, this is the code that handle the D main (Tango runtime
compiler/ldc/rt/dmain2.d, main() function, around line 260):

void runMain()
{
debug(PRINTF) printf("main runMain\n");
result = main(args);
}

void runAll()
{
debug(PRINTF) printf("main runAll\n");
gc_init();
_moduleCtor();
if (runModuleUnitTests())
tryExec(&runMain);
thread_joinAll();
_d_isHalting = true;
_moduleDtor();
gc_term();
}

tryExec(&runAll);

debug(PRINTF) printf("main dtor\n");
_STD_critical_term();
_STD_monitor_staticdtor();

return result;

If thread_joinAll() is called before gc_term(), I can't see how a thread
could be still running when gc_term() is called. So, in terms of threads,
I don't see a problem here.

About exceptions, I think it could be solved too. One option is to require
gc_term() to be "nonthrow" (I mean, for D1 and D2). I think the specs
doesn't say anything about destructors throwing exceptions. I think they
shouldn't but let's suppose they could. Since there is no way the use can
catch a destructor exception thrown in the GC (well, not in an useful way,
one could try/catch the new call or something but that doesn't work for
concurrent collectors that can sweep in a separate thread anyway, so
I think it's a bad idea), I guess it would be acceptable to:
a) Ignore the exception
b) Let the user provide a callback to handle that kind of exceptions

Then, gc_term() ca be moved outside the tryExec(&runAll) call and
gc_term() can safely call all the live objects finalizers and provide
guaranteed finalization.


PS: I have not read all the runtime, just this small part, so maybe I'm
missing the big picture =)

[Fawzi Mohamed]

Post by Leandro Lucarella

Post by Leandro Lucarella

Post by Fawzi Mohamed

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

when the main thread ends other threads might be running, and removing
memory they are using is not necessarily a good idea.

Then I guess gc_term() should be invoked when all threads have terminated.
Is there any technical difficulty to do that?

Post by Fawzi Mohamed
Exceptions might delay things indefinitely and allocate more memory.

Then I guess gc_term() should be invoked when all exceptions were
processed. Is there any technical difficulty to do that?
I really didn't take a look at the compiler runtime stuff, so I don't
really know how all that works.

Ok, this is the code that handle the D main (Tango runtime
void runMain()
{
debug(PRINTF) printf("main runMain\n");
result = main(args);
}
void runAll()
{
debug(PRINTF) printf("main runAll\n");
gc_init();
_moduleCtor();
if (runModuleUnitTests())
tryExec(&runMain);
thread_joinAll();
_d_isHalting = true;
_moduleDtor();
gc_term();
}
tryExec(&runAll);
debug(PRINTF) printf("main dtor\n");
_STD_critical_term();
_STD_monitor_staticdtor();
return result;
If thread_joinAll() is called before gc_term(), I can't see how a thread
could be still running when gc_term() is called. So, in terms of threads,
I don't see a problem here.

daemon theads are not joined

Post by Leandro Lucarella
About exceptions, I think it could be solved too. One option is to require
gc_term() to be "nonthrow" (I mean, for D1 and D2). I think the specs
doesn't say anything about destructors throwing exceptions. I think they
shouldn't but let's suppose they could. Since there is no way the use can
catch a destructor exception thrown in the GC (well, not in an useful way,
one could try/catch the new call or something but that doesn't work for
concurrent collectors that can sweep in a separate thread anyway, so
a) Ignore the exception
b) Let the user provide a callback to handle that kind of exceptions

finalizer should not throw already now, if they do an exception is raised.

Post by Leandro Lucarella
Then, gc_term() ca be moved outside the tryExec(&runAll) call and
gc_term() can safely call all the live objects finalizers and provide
guaranteed finalization.
PS: I have not read all the runtime, just this small part, so maybe I'm
missing the big picture =)

[Leandro Lucarella]

Post by Fawzi Mohamed

Post by Leandro Lucarella
If thread_joinAll() is called before gc_term(), I can't see how a thread
could be still running when gc_term() is called. So, in terms of threads,
I don't see a problem here.

daemon theads are not joined

So, you can keep threads going on even when you have terminated the GC?
That's odd! I think that's an accident waiting to happen =)

Anyways, if you do have a thread live when the GC is terminate, a lot of
care have to be taken, I don't see why "do not use GC allocated memory" in
threads that live longer than the GC should not be added to the
limitations. It seems pretty reasonable.

Post by Fawzi Mohamed

Post by Leandro Lucarella
About exceptions, I think it could be solved too. One option is to require
gc_term() to be "nonthrow" (I mean, for D1 and D2). I think the specs
doesn't say anything about destructors throwing exceptions. I think they
shouldn't but let's suppose they could. Since there is no way the use can
catch a destructor exception thrown in the GC (well, not in an useful way,
one could try/catch the new call or something but that doesn't work for
concurrent collectors that can sweep in a separate thread anyway, so
a) Ignore the exception
b) Let the user provide a callback to handle that kind of exceptions

finalizer should not throw already now, if they do an exception is raised.

Ok, so then exceptions thrown by destructors are not a problem either.

[Fawzi Mohamed]

Post by Leandro Lucarella

Post by Fawzi Mohamed

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

when the main thread ends other threads might be running, and removing
memory they are using is not necessarily a good idea.

Then I guess gc_term() should be invoked when all threads have terminated.
Is there any technical difficulty to do that?

well killing a daemon thread might not be so easy, the cleanest thing
would be to simulate an exception into it and unwind the stack.
If you are not careful you might kill a thread when it did acquire a
lock that might deadlock your own thread.

Post by Leandro Lucarella

Post by Fawzi Mohamed
Exceptions might delay things indefinitely and allocate more memory.

Then I guess gc_term() should be invoked when all exceptions were
processed. Is there any technical difficulty to do that?

this should be sort of doable (without having looked at the details)

Post by Leandro Lucarella
I really didn't take a look at the compiler runtime stuff, so I don't
really know how all that works.

Post by Fawzi Mohamed
Actually with tango you can tweak what will happen with
gc_getTermCleanupLevel (in gc.d), but I don't think that there is
a satisfying solution for everybody.

Yes, I saw that (I commented about that in the blog post), but there is
still no option to run the finalizers to all the live objects.

well you could try to ignore all roots but the gc ones (you don't want
to deallocate it before you have finished using it :).

Post by Leandro Lucarella
Anyways, I wonder why it's not part of the GC specs in druntime.

Post by Fawzi Mohamed
I think that the default in tango is quite reasonable (normal collect).

Yes... With the current specs at least, I guess.
I think that if finalization is not guarantee, at least at
program exit, there shouldn't be there. It's useless.

not true, there are still cases in which it is useful (weak, ref, and
what Robert Jaques said.
Actually one of the first comment when I added gc_setTermCleanupLevel,
was nice so now my program can finish faster, not everybody wants to
wait too long after the program has finished doing useful stuff for it
to die.

Post by Leandro Lucarella
Finalization should be possible only for scope classes if no finalization
guarantee can be provided by the GC. It's useless and error prone
(introducing a bug very difficult to track). Who wants randomly executed
destructors?

destructors are executed in a randmized order (so you might consume an
external resource if it is limited). That they might not be executed at
all, is only a small extra problem. Indeed they are less useful than
(for example) C++, but can still have some uses.

Fawzi

[Leandro Lucarella]

Post by Fawzi Mohamed

Post by Leandro Lucarella

Post by Fawzi Mohamed
when the main thread ends other threads might be running, and removing
memory they are using is not necessarily a good idea.

Then I guess gc_term() should be invoked when all threads have terminated.
Is there any technical difficulty to do that?

well killing a daemon thread might not be so easy, the cleanest thing
would be to simulate an exception into it and unwind the stack. If you
are not careful you might kill a thread when it did acquire a lock that
might deadlock your own thread.

But don't you agree that if a thread live longer than the GC, there are
a lot of problems to take care of? I see that scenario as a very rare
special case, not as a reasonable default. I guess some mechanism (like
gc_setTermCleanupLevel()) can be provided to tell the GC not to call
finalizers can be good enough for that special cases. And in that cases,
it's completely acceptable that the finalization guarantee is not provided
(but just because you asked not to be provided).

Post by Fawzi Mohamed

Post by Leandro Lucarella

Post by Fawzi Mohamed
Exceptions might delay things indefinitely and allocate more memory.

Then I guess gc_term() should be invoked when all exceptions were
processed. Is there any technical difficulty to do that?

this should be sort of doable (without having looked at the details)

I don't see a problem with this either.

Post by Fawzi Mohamed

Post by Leandro Lucarella
I really didn't take a look at the compiler runtime stuff, so I don't
really know how all that works.

Post by Fawzi Mohamed
Actually with tango you can tweak what will happen with
gc_getTermCleanupLevel (in gc.d), but I don't think that there is
a satisfying solution for everybody.

Yes, I saw that (I commented about that in the blog post), but there is
still no option to run the finalizers to all the live objects.

well you could try to ignore all roots but the gc ones (you don't want
to deallocate it before you have finished using it :).

If the program ends, if the GC ends, you shouldn't be using any GC
allocated memory in the first place, don't you?

Post by Fawzi Mohamed

Post by Leandro Lucarella
Anyways, I wonder why it's not part of the GC specs in druntime.

Post by Fawzi Mohamed
I think that the default in tango is quite reasonable (normal collect).

Yes... With the current specs at least, I guess.
I think that if finalization is not guarantee, at least at
program exit, there shouldn't be there. It's useless.

not true, there are still cases in which it is useful (weak, ref, and
what Robert Jaques said.

Ok, it's useful for memory related bookkeeping. That's true. But it's too
easy to introduce bizarre bugs for managing other resources.

And I think providing finalization guarantee is not that hard either.

Post by Fawzi Mohamed
Actually one of the first comment when I added gc_setTermCleanupLevel,
was nice so now my program can finish faster, not everybody wants to
wait too long after the program has finished doing useful stuff for it
to die.

Again, I'm not talking about collection. I really don't think the programs
should do a collection before it ends, it has no point, who am I freeing
memory to? What I think it's it necessary is to call the finalizers of the
live objects. Another way to see things is the program has not "finished
doing useful stuff" until all finalizers are called, because finalizers
are supposed to be useful =)

Post by Fawzi Mohamed

Post by Leandro Lucarella
Finalization should be possible only for scope classes if no finalization
guarantee can be provided by the GC. It's useless and error prone
(introducing a bug very difficult to track). Who wants randomly executed
destructors?

destructors are executed in a randmized order (so you might consume an
external resource if it is limited). That they might not be executed at
all, is only a small extra problem. Indeed they are less useful than
(for example) C++, but can still have some uses.

I'm not talking about order when saying "randomly", I'm talking about the
destructor being executed at all =)

[Robert Jacques]

Post by Don

Post by Leandro Lucarella

Post by Robert Jacques
On Sat, 18 Apr 2009 11:24:14 -0400, Leandro Lucarella

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

Well, a couple of quick tests show that under normal situations (i.e.
normal program termination and termination from an exception) the
finalizers do run (D2). However, if a finalizer throws an exception,
then the rest of the finalizers aren't called. Also, if you call
std.c.stdlib.exit, the finalizers won't run.

Well, I'm not talking about experimentation, I'm talking about source code
;)
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

I don't understand why D even has finalizers. Can they do anything useful?

Yes. I use them to manage GPU memory/resources. Another are wrapper
classes around manually allocated arrays (to avoid the false pointer
problem with very large arrays). Both of these examples don't require
finalization at program termination (unlike file handles/sockets, etc)

[Don]

Post by Robert Jacques

Post by Don

Post by Leandro Lucarella

Post by Robert Jacques
On Sat, 18 Apr 2009 11:24:14 -0400, Leandro Lucarella

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?

Well, a couple of quick tests show that under normal situations (i.e.
normal program termination and termination from an exception) the
finalizers do run (D2). However, if a finalizer throws an exception,
then the rest of the finalizers aren't called. Also, if you call
std.c.stdlib.exit, the finalizers won't run.

Well, I'm not talking about experimentation, I'm talking about source code
;)
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

I don't understand why D even has finalizers. Can they do anything useful?

Yes. I use them to manage GPU memory/resources. Another are wrapper
classes around manually allocated arrays (to avoid the false pointer
problem with very large arrays). Both of these examples don't require
finalization at program termination (unlike file handles/sockets, etc)

OK, those both make perfect sense -- they are both memory issues. It
still seems that gc is perfect for memory, but no good for anything else.

[Leandro Lucarella]

Post by Don
I don't understand why D even has finalizers. Can they do anything useful?

Yes. I use them to manage GPU memory/resources. Another are wrapper classes around manually allocated arrays (to avoid the false pointer problem with very
large arrays). Both of these examples don't require finalization at program termination (unlike file handles/sockets, etc)

OK, those both make perfect sense -- they are both memory issues. It still seems that gc is perfect for memory, but no good for anything else.

Well, you mean for resources the OS free for you at program end, right?
For other resources is not useful because the not guaranteed finalization
(if you don't have the OS to cover your ass, the GC won't either with this
limitation).

I think shared memory is an example of memory resource that's not freed by
the OS on program exit.
--
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
----------------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------------
Ah, se va en 1981? Pero por qu?? ... Ah, porque ya hab?a sido determinado,
entonces quiere decir que pronto vamos a elegir presidente nuevo nosot...
Ah, nosotros no? Ah, lo van a elegir en la ... Ah! Quiere que le diga? Muy
bien pensado, porque cada vez que lo elegimos nosotros no duran nada!
-- Tato vs. Tato (1980, Gobierno de Videla)

[Don]

Post by Leandro Lucarella

Post by Don
I don't understand why D even has finalizers. Can they do anything useful?

Yes. I use them to manage GPU memory/resources. Another are wrapper classes around manually allocated arrays (to avoid the false pointer problem with very
large arrays). Both of these examples don't require finalization at program termination (unlike file handles/sockets, etc)

OK, those both make perfect sense -- they are both memory issues. It still seems that gc is perfect for memory, but no good for anything else.

Well, you mean for resources the OS free for you at program end, right?
For other resources is not useful because the not guaranteed finalization
(if you don't have the OS to cover your ass, the GC won't either with this
limitation).

No, that's not what I mean at all. I actually think it's a good thing
that the GC doesn't call finalizers at the end -- making it obvious that
you shouldn't be using it for resource management.

Post by Leandro Lucarella
I think shared memory is an example of memory resource that's not freed by
the OS on program exit.

Really? That sounds like an OS memory leak. Security issue, too: run
your program, allocated shared memory, then exit. Repeat until all
memory is exhausted.

[Robin KAY]

Don wrote:
[snip]

Post by Don

Post by Leandro Lucarella
I think shared memory is an example of memory resource that's not freed by
the OS on program exit.

Really? That sounds like an OS memory leak. Security issue, too: run
your program, allocated shared memory, then exit. Repeat until all
memory is exhausted.

Run your program, create a file, then exit. Repeat until all disk space
is exhausted.

SYSV IPC objects have a persistent identifier through which they can be
accessed by multiple processes.

[Leandro Lucarella]

Post by Don

Post by Leandro Lucarella
I think shared memory is an example of memory resource that's not freed by
the OS on program exit.

Really? That sounds like an OS memory leak. Security issue, too: run
your program, allocated shared memory, then exit. Repeat until all
memory is exhausted.

man 3 shm_open

Anyways. Again, the discussion is diverging very quickly. The point is,
according to the specs, the GC is allowed to not call finalizers at all.
So any current D program relying on the GC calling a destructor *ever* is
broken according to the specs, even when destructors are used for weak
pointers, for freeing malloc'ed memory or anything.

I think finalizers should be removed or fixed (I vote for the latter,
because it's easy and efficient to fix it).

Even more, I'm considering this a bug in the specs even for D1, so I'll
file a bug for this.

[Leandro Lucarella]

Post by Leandro Lucarella

Post by Don

Post by Leandro Lucarella
I think shared memory is an example of memory resource that's not freed by
the OS on program exit.

Really? That sounds like an OS memory leak. Security issue, too: run
your program, allocated shared memory, then exit. Repeat until all
memory is exhausted.

man 3 shm_open
Anyways. Again, the discussion is diverging very quickly. The point is,
according to the specs, the GC is allowed to not call finalizers at all.
So any current D program relying on the GC calling a destructor *ever* is
broken according to the specs, even when destructors are used for weak
pointers, for freeing malloc'ed memory or anything.
I think finalizers should be removed or fixed (I vote for the latter,
because it's easy and efficient to fix it).
Even more, I'm considering this a bug in the specs even for D1, so I'll
file a bug for this.

http://d.puremagic.com/issues/show_bug.cgi?id=2858

[Walter Bright]

Post by Leandro Lucarella
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

The why is because of speed. What's the point of running a gc pause on
program exit? The OS recovers all the memory anyway.

[Leandro Lucarella]

Post by Walter Bright

Post by Leandro Lucarella
The current GC implementation don't call finalizers for data that's still
reference when the program *ended* (this is allowed by the specs, so it's
fine, the question is why it's allowed by the specs).

The why is because of speed. What's the point of running a gc pause on
program exit? The OS recovers all the memory anyway.

You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

And I'm not talking about doing a collection at program exit either. I'm
talking about just calling the finalizers.

What I say is to do this (in some kind of pseudocode, which is actually
valid code in my example naive GC implementation =):

void gc_term()
{
foreach (cell; this.live_list)
rt_finalize(cell.ptr);
}

No collection, no free.

What do you think about that? And about finalizers being completely
useless if the excecution is not guarateed? I would really like to know
(Sean's opinion would be appreciated too, being in charge of druntime).

Maybe this can be an improvement for D2 (I don't think you'll like to
change the specs for D1 =).

[Rainer Deyke]

Post by Leandro Lucarella
You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

It can't be handled by the GC either, because:
- This would require a high-level wrapper that knows about the "bye"
packet around a low-level socket that doesn't know. By the time the
high-level wrapper is finalized, the low-level socket may already have
been collected.
- It is bad form to wait for the next garbage-collection cycle before
cleanly terminating connections.

What you need is RAII, not garbage collection.

[Leandro Lucarella]

Post by Rainer Deyke

Post by Leandro Lucarella
You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

- This would require a high-level wrapper that knows about the "bye"
packet around a low-level socket that doesn't know. By the time the
high-level wrapper is finalized, the low-level socket may already have
been collected.

I don't know what you are talking about, I'm talking about this:

class X {

//...

~this()
{
socket.send(bye_packet);
socket.close();
}

Post by Rainer Deyke
- It is bad form to wait for the next garbage-collection cycle before
cleanly terminating connections.
What you need is RAII, not garbage collection.

I'm talking about long lived resources for example.

I don't understand people defending non-guaranteed finalization over
guaranteed finalization, even when it's:
a) Possible
b) Easy to do
c) Probably more efficient to terminate the program than the "safe"
fullcollect()

I really don't get it. Seriously.

[Leandro Lucarella]

Post by Leandro Lucarella

Post by Rainer Deyke

Post by Leandro Lucarella
You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

- This would require a high-level wrapper that knows about the "bye"
packet around a low-level socket that doesn't know. By the time the
high-level wrapper is finalized, the low-level socket may already have
been collected.

class X {
//...
~this()
{
socket.send(bye_packet);
socket.close();
}

Ok, socket can be collected when the destructor of X is called. Now I get
you point.

I still think having guaranteed finalization make the language much more
robust, though. And it's easily doable, I don't see why it's not done.

[Walter Bright]

Post by Leandro Lucarella
You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

That shouldn't be handled by a finalizer. A "bye" packet can be handled
by a static destructor.

Post by Leandro Lucarella
And I'm not talking about doing a collection at program exit either. I'm
talking about just calling the finalizers.

The finalizers are found by doing a collection scan to see what's left,
and then running the finalizers.

Post by Leandro Lucarella
What I say is to do this (in some kind of pseudocode, which is actually
void gc_term()
{
foreach (cell; this.live_list)
rt_finalize(cell.ptr);
}
No collection, no free.
What do you think about that? And about finalizers being completely
useless if the excecution is not guarateed? I would really like to know
(Sean's opinion would be appreciated too, being in charge of druntime).

Finalizers are fairly useless for gc. But they are a lot more useful as
RAII destructors.

Post by Leandro Lucarella
Maybe this can be an improvement for D2 (I don't think you'll like to
change the specs for D1 =).

[Christopher Wright]

Post by Walter Bright

Post by Leandro Lucarella
You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

That shouldn't be handled by a finalizer. A "bye" packet can be handled
by a static destructor.

That requires more work -- you have to keep track of a bunch of
instances that are active, and then the static destructor goes through
all of them and nukes them each in turn. It isn't a whole lot of work, I
admit. But it is -- or at least, it is perceived as -- an idiom to make
up for an inadequacy of the garbage collector.

Post by Walter Bright
Finalizers are fairly useless for gc. But they are a lot more useful as
RAII destructors.

Then maybe finalizers should be divorced entirely from garbage collection.

[Leandro Lucarella]

Post by Christopher Wright

Post by Leandro Lucarella
You missed the point. I'm not talking about freeing the memory. I'm
talking about finalizers. A finalizer could send a "bye" packet throgh the
net. That can't be handled by the OS.

That shouldn't be handled by a finalizer. A "bye" packet can be handled by a static destructor.

That requires more work -- you have to keep track of a bunch of
instances that are active, and then the static destructor goes through
all of them and nukes them each in turn. It isn't a whole lot of work,
I admit. But it is -- or at least, it is perceived as -- an idiom to
make up for an inadequacy of the garbage collector.

Finalizers are fairly useless for gc. But they are a lot more useful as RAII destructors.

Then maybe finalizers should be divorced entirely from garbage collection.

Tha's an option too, but I think it's more useful to provide guaranteed
finalization, which is very doable too.
--
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
----------------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------------
When I was a child I had a fever
My hands felt just like two balloons.
Now I've got that feeling once again
I can't explain you would not understand
This is not how I am.
I have become comfortably numb.

[Unknown W. Brackets]

The simple solution is this:

1. If your class object only involves memory, freed OS handles, etc., it
should be used as-is. This is most class objects. Destructors are
needed to clamp resource use (see File class.)

2. If your class object involves hardware handles, transactional
assurance, or data integrity, it must be scoped. This is the same as it
was in C, except D has better constucts for it (see scope.)

I don't see the problem. For the majority of objects that only involve
memory, life is easier. For the rest, life is still easier (just not as
much.)

-[Unknown]

Post by Leandro Lucarella
The garbage collector is not guaranteed to run the destructor for
all unreferenced objects.
Is there any reason why D can't guarantee that all finalizers will be
called, at least when the program ends?
[1] http://proj.llucax.com.ar/blog/dgc/blog/post/-43101db1
[2] http://www.digitalmars.com/d/1.0/class.html#destructors

[Leandro Lucarella]

Post by Unknown W. Brackets
1. If your class object only involves memory, freed OS handles, etc., it
should be used as-is. This is most class objects. Destructors are
needed to clamp resource use (see File class.)

A File class implementation, for instance, can loose data if they use
buffers. For example:

class File
{
//...

~this()
{
this.flush(); // write remaining buffered data
close(handle);
}
}

In this case, the OS will close the handle, but the data will not be
written.

So I guess this kind of things has to go in your 2nd group.

And this is a good example for a valid use that will break if guaranteed
finalization is not provided. A log file is a good example of an object
that lasts for all the program lifetime and needs guaranteed finalization.

Post by Unknown W. Brackets
2. If your class object involves hardware handles, transactional
assurance, or data integrity, it must be scoped. This is the same as it
was in C, except D has better constucts for it (see scope.)
I don't see the problem. For the majority of objects that only involve
memory, life is easier. For the rest, life is still easier (just not as
much.)

Ok, I agree. Do you see any problems with support to guaranteed
finalization?

My proposal is:
a) Add to the specs that finalization is guaranteed, in the worse case at
program termination.
b) Standarize the gc_setTermHandlerDepth() (or whatever the name is)
c) Use a "finalize all the live data" at program exit as a default for
the TermHandlerDepth (without running a collection, but this is
completely implementation dependant).

Comments?

[Unknown W. Brackets]

Well, I guess it would be doable to guarantee destruction, but *only* if
order of destruction was not guaranteed.

In other words, at the end, the GC would not scan, it would just
destroy. Scanning is a problem, due to cycles and various other things,
but destroying all roots would be potentially doable.

I disagree that a log file needs guaranteed finalization, however. It's
when you think that way that you ignore the possibility of hardware
failure. I'm a server guy - and all the daemons I use log immediately
to the filesystem, without buffering. Anything else would be unacceptable.

I may be wrong, but I'm pretty sure std.stream's files do not work as
you suggest. BufferedFile may, but I do not think File does (aside from
OS buffers and journals which are fine; an fsync is not mandatory.)

-[Unknown]

Post by Leandro Lucarella

Post by Unknown W. Brackets
1. If your class object only involves memory, freed OS handles, etc., it
should be used as-is. This is most class objects. Destructors are
needed to clamp resource use (see File class.)

A File class implementation, for instance, can loose data if they use
class File
{
//...
~this()
{
this.flush(); // write remaining buffered data
close(handle);
}
}
In this case, the OS will close the handle, but the data will not be
written.
So I guess this kind of things has to go in your 2nd group.
And this is a good example for a valid use that will break if guaranteed
finalization is not provided. A log file is a good example of an object
that lasts for all the program lifetime and needs guaranteed finalization.

Post by Unknown W. Brackets
2. If your class object involves hardware handles, transactional
assurance, or data integrity, it must be scoped. This is the same as it
was in C, except D has better constucts for it (see scope.)
I don't see the problem. For the majority of objects that only involve
memory, life is easier. For the rest, life is still easier (just not as
much.)

Ok, I agree. Do you see any problems with support to guaranteed
finalization?
a) Add to the specs that finalization is guaranteed, in the worse case at
program termination.
b) Standarize the gc_setTermHandlerDepth() (or whatever the name is)
c) Use a "finalize all the live data" at program exit as a default for
the TermHandlerDepth (without running a collection, but this is
completely implementation dependant).
Comments?

[Leandro Lucarella]

Post by Unknown W. Brackets
Well, I guess it would be doable to guarantee destruction, but *only* if
order of destruction was not guaranteed.

Yes, of course, order *can't* be guaranteed (unless you add read/write
barriers and a lot of overhead at least =)

Post by Unknown W. Brackets
In other words, at the end, the GC would not scan, it would just
destroy.

Yes, that's the proposal.

Post by Unknown W. Brackets
Scanning is a problem, due to cycles and various other things,

I don't see any problems with scanning, it's just time-consuming and
provides no benefits. Even more, the actual implementation do
a fullcollect() (which marks, scans and sweeps).

Post by Unknown W. Brackets
but destroying all roots would be potentially doable.

My proposal is to destroy everything live. There is no need to follow the
roots, all memory known as live should be finalized. Again, there is no
need on scanning the roots.

Post by Unknown W. Brackets
I disagree that a log file needs guaranteed finalization, however. It's
when you think that way that you ignore the possibility of hardware
failure. I'm a server guy - and all the daemons I use log immediately
to the filesystem, without buffering. Anything else would be
unacceptable.

Maybe you don't need that kind of security, but you would like your
program to behave as expected in normal situations (i.e. when no hardware
failure is involved).

Anyway the problem is no because of some example (I think it was a big
mistake from my part to take the discussion into the examples field). The
problem is the specs says now that "The garbage collector is not
guaranteed to run the destructor for all unreferenced objects.", it
doesn't say this is only applicable for destruction at the end of the
program. As far as the specs go, a GC that don't call finalizers during
collection *ever* is conforming.

[Unknown W. Brackets]

Personally, I usually use destructors to clean up or to decrement use
counts, bleach secure data, etc. but it should be guaranteed that if
data is no longer allocated, the destructor is called. That may be
elsewhere in the spec, though.

Also, hardware failure should always be expected:

Webservers like Apache handle it.
Browsers like Firefox handle it.
Databases like MySQL handle it.
Operating systems like Windows handle it.
Productivity apps like Word handle it.
FTP clients like SmartFTP handle it.
Communication apps like Trillian handle it.

No one should esteem to write crapware.

-[Unknown]

Post by Leandro Lucarella

Post by Unknown W. Brackets
Well, I guess it would be doable to guarantee destruction, but *only* if
order of destruction was not guaranteed.

Yes, of course, order *can't* be guaranteed (unless you add read/write
barriers and a lot of overhead at least =)

Post by Unknown W. Brackets
In other words, at the end, the GC would not scan, it would just
destroy.

Yes, that's the proposal.

Post by Unknown W. Brackets
Scanning is a problem, due to cycles and various other things,

I don't see any problems with scanning, it's just time-consuming and
provides no benefits. Even more, the actual implementation do
a fullcollect() (which marks, scans and sweeps).

Post by Unknown W. Brackets
but destroying all roots would be potentially doable.

My proposal is to destroy everything live. There is no need to follow the
roots, all memory known as live should be finalized. Again, there is no
need on scanning the roots.

Post by Unknown W. Brackets
I disagree that a log file needs guaranteed finalization, however. It's
when you think that way that you ignore the possibility of hardware
failure. I'm a server guy - and all the daemons I use log immediately
to the filesystem, without buffering. Anything else would be
unacceptable.

Maybe you don't need that kind of security, but you would like your
program to behave as expected in normal situations (i.e. when no hardware
failure is involved).
Anyway the problem is no because of some example (I think it was a big
mistake from my part to take the discussion into the examples field). The
problem is the specs says now that "The garbage collector is not
guaranteed to run the destructor for all unreferenced objects.", it
doesn't say this is only applicable for destruction at the end of the
program. As far as the specs go, a GC that don't call finalizers during
collection *ever* is conforming.

[Jason House]

Post by Walter Bright

Post by Leandro Lucarella
Close a connection gracefully for example, I guess (I mean, send a "bye"
packed, not just close the socket abruptly). Same for closing files
writing some mark or something. They can be risky when finalization is not
deterministic though.

Scoped objects should be used for that, not gc.

What about member variables of scope objects? IIRC classes can't have scope member variables which can be frustrating when creating wrappers.

Can scope global variables exist?