Discussion:
JNI - question about jmethodid values.
Thomas Stüfe
2018-11-24 15:41:18 UTC
Permalink
Hi Peter,

(adding serviceability)

A jmethodid is a pointer to malloc'ed memory.

In more detail, jmethodid points to a table slot containing pointers
to Method* objects, and that table itself lives in C heap. See class
JNIMethodBlock, JNIMethodBlockNode:
http://hg.openjdk.java.net/jdk/jdk/file/30a02b4e6c06/src/hotspot/share/oops/method.cpp#l1928

If I understand the NetBeans profiler coding right, they assume the
jmethodIDs to cluster in up to four clusters of relative vicinity,
with each cluster spanning 30 bits? That is quite an assumption to
make.

This may work more or less accidentally, since malloc'ed pointers are
also clustered naturally. A Clib implementation may allocate only in
the traditional data segment (below sbrk) - I believe AIX and Solaris
do this - which means all malloced pointers are as close to each other
as the data segment is large. But C-Libraries often allocate C-Heap
memory in separate mmape'd blocks (e.g. glibc arenas), and there is no
guarantee where those blocks are. You may still often get lucky - if
all jmethod id block allocations happen in a short timeframe, or if
you just have not many users of C-Heap in the process.

But it is not guaranteed to work. I would probably rather use a
hashmap or similar.

Side note: I believe (and someone please correct me if I am wrong)
that jmethodid used to live in the PermGen, in the Java Heap, and that
only since the PermGen removal in JDK8 they live in C-Heap. If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.

Best Regards, Thomas
I have recently been looking at some native code that's part of the
NetBeans profiler. This code was crashing for me because it was based
on incorrect assumptions about the sizes of various primitive types.
There is some code which is intended to map jmethodid values to jint
values (see below). This is trivial on 32 bit platforms because the
are the same size. However on 64-bit platforms this is not the case
because jmethodid is (according to the docs) to be treated as a
pointer to an opaque structure and is 64-bit.
So, the code splits the 64-bit jmethodid into a 30-bit 'offset' part
and a 34-bit 'base' part. It then stores the high part in one of four
slots and returns a 32 bit int which is 2 bits for the slot id and 30
bits from the offset part. For the reverse process, it looks up the
slot id, gets the base address and ORs it with the offset. This seems
to work in practice but it is based on the assumption that all
jmethodids will have one of only four base parts - any more than four
can't be specified by a 2 bit id.
So my question is: is this a sound assumption? The answer needs some
detailed knowledge of the internals of the JRE I suppose, so if this
mailing list is not the right place to ask, what is the right place?
I want to fix my crash in the netbeans profiler but I want to do the
right thing rather than just patching it up (which I have done by
doing casts more carefully)
Thanks for your help.
Peter
https://github.com/apache/incubator-netbeans/blob/5405b8c93eeafdd213cffcc59df002464af8d838/profiler/lib.profiler/native/src-jdk15/Stacks.c#L66-L92
and: https://github.com/apache/incubator-netbeans/blob/5405b8c93eeafdd213cffcc59df002464af8d838/profiler/lib.profiler/native/src-jdk15/Stacks.c#L66-L92
Thomas Stüfe
2018-11-26 13:00:13 UTC
Permalink
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
JC Beyler
2018-11-26 19:15:03 UTC
Permalink
Hi all,
Post by Thomas Stüfe
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but
it is not defined by the SPEC to be like this so, in essence, it is an
implementation detail that could be not true in future releases ;-)

Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-26 19:33:03 UTC
Permalink
Hey JC,
Post by JC Beyler
Hi all,
Post by Thomas Stüfe
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
I politely disagree :)

JNI spec says that you have a function like this:

jmethodID GetMethodID(..)

returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.

So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.

I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.

Cheers, Thomas
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
David Holmes
2018-11-26 21:35:03 UTC
Permalink
Please all, don't cc the discuss list once a topic has been directed to
the correct mailing list!

Thanks,
David
Post by Thomas Stüfe
Hey JC,
Post by JC Beyler
Hi all,
Post by Thomas Stüfe
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
JC Beyler
2018-11-26 21:39:00 UTC
Permalink
Hi Thomas,

Not sure I agree either (but this first part of the email might be you and
I rat-holing and we could/should perhaps just let it go :-)):
- The specs do not say at all what happens to the internals of the
jmethodIDs and not even that de-referencing them is safe. You are right
that the spec currently says that the jmethodID should come from a
GetMethodID. But that is all the JNI spec says when it gives a definition
of a jmethodID parameter. However, the JNI general spec about jmethodIDs
explicitly does say [1]:

"A field or method ID does not prevent the VM from unloading the class from
which the ID has been derived. After the class is unloaded, the method or
field ID becomes invalid. The native code, therefore, must make sure to:

- keep a live reference to the underlying class, or
- recompute the method or field ID"

And: "Forcing JNI functions to check for all possible error conditions
degrades the performance of normal (correct) native methods." [1]

Technically, the JNI spec might want to change the sentences "The methodID
argument must be obtained by calling GetMethodID()" to something like "The
methodID argument must be obtained by calling GetMethodID() and still be
valid, see [1]"

Turns out that JVMTI spec does provide some of the checks for you and lets
you call with invalid jmethodIds but this is not ok to assume for JNI.

Btw, when you look at the implementation of the JNI methods right now, if
you try to pass an older jmethodID, it will actually segfault. This might
be a corner case that is a bit awkward but if you do (I have a test example
if you want to see the code to force this path):
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID (which
now points to NULL), the code will segfault

So remembering jmethodIDs and then assuming that JNI calls will always
return gracefully (whether with an error or not) is actually wrong. For
JVMTI, this seems to hold in terms of spec and implementation from what I
see.

Thanks,
Jc

[1]
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
Post by Thomas Stüfe
Hey JC,
Post by JC Beyler
Hi all,
Post by Thomas Stüfe
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit
but it is not defined by the SPEC to be like this so, in essence, it is an
implementation detail that could be not true in future releases ;-)
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Please all, don't cc the discuss list once a topic has been directed to
the correct mailing list!
Thanks,
David
Post by JC Beyler
Hey JC,
Post by Thomas Stüfe
Hi all,
Hi Peter,
Post by Thomas Stüfe
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit
but it is not defined by the SPEC to be like this so, in essence, it is an
implementation detail that could be not true in future releases ;-)
Post by JC Beyler
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Post by Thomas Stüfe
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
--
Thanks,
Jc
Thomas Stüfe
2018-11-26 22:07:59 UTC
Permalink
Hi JC,
Hi Thomas,
- keep a live reference to the underlying class, or
- recompute the method or field ID"
And: "Forcing JNI functions to check for all possible error conditions degrades the performance of normal (correct) native methods." [1]
Technically, the JNI spec might want to change the sentences "The methodID argument must be obtained by calling GetMethodID()" to something like "The methodID argument must be obtained by calling GetMethodID() and still be valid, see [1]"
Turns out that JVMTI spec does provide some of the checks for you and lets you call with invalid jmethodIds but this is not ok to assume for JNI.
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID (which now points to NULL), the code will segfault
Hmm, would this not be a bug we should fix? Especially since we leak
the JNIMethodBlockNodes just to be able to keep their slots alive?
So remembering jmethodIDs and then assuming that JNI calls will always return gracefully (whether with an error or not) is actually wrong. For JVMTI, this seems to hold in terms of spec and implementation from what I see.
Thanks,
Jc
Thanks, Thomas
[1] https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
Post by Thomas Stüfe
Hey JC,
Post by JC Beyler
Hi all,
Post by Thomas Stüfe
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Please all, don't cc the discuss list once a topic has been directed to
the correct mailing list!
Thanks,
David
Post by JC Beyler
Hey JC,
Post by Thomas Stüfe
Hi all,
Hi Peter,
Post by Thomas Stüfe
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Post by Thomas Stüfe
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
--
Thanks,
Jc
d***@oracle.com
2018-11-26 21:44:37 UTC
Permalink
Doesn't the spec say it's the caller's responsibility to keep the class
from being unloaded?

"
A field or method ID does not prevent the VM from unloading the class
from which the ID has been derived. After the class is unloaded, the
method or field ID becomes invalid. The native code, therefore, must
make sure to:

* keep a live reference to the underlying class, or
* recompute the method or field ID

if it intends to use a method or field ID for an extended period of time.

The JNI does not impose any restrictions on how field and method IDs are
implemented internally.

"


however, then the JVMTI spec has errors like
|JVMTI_ERROR_INVALID_METHODID|
<https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#JVMTI_ERROR_INVALID_METHODID>
that seem to imply that the JVM can (or must?) detect an invalid methodid.

dl
Post by Thomas Stüfe
Hey JC,
Post by JC Beyler
Hi all,
Post by Thomas Stüfe
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
Post by Thomas Stüfe
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-26 22:08:12 UTC
Permalink
Hi Dean,
Doesn't the spec say it's the caller's responsibility to keep the class from being unloaded?
"
keep a live reference to the underlying class, or
recompute the method or field ID
if it intends to use a method or field ID for an extended period of time.
The JNI does not impose any restrictions on how field and method IDs are implemented internally.
"
however, then the JVMTI spec has errors like JVMTI_ERROR_INVALID_METHODID that seem to imply that the JVM can (or must?) detect an invalid methodid.
dl
Yes, I took the fact that JVMTI functions can return
JVMTI_ERROR_INVALID_METHODID to be a requirement that the VM must be
able to detect this, and therefore the jmethodid slot must not move or
be reused.

Thanks, Thomas
Hey JC,
Hi all,
Hi Peter,
Hi Thomas,
Thank you very much for the detailed explanation. For your
information, the relevant NetBeans bug is
https://issues.apache.org/jira/browse/NETBEANS-1428
A jmethodid is a pointer to malloc'ed memory.
OK. Just in case I haven't understood it - does this mean a jmethodid,
once 'created', won't change (after garbage collection or whatever)?
yes. It lives on, unchanged, forever. Even if the associated class is
unloaded. That must be since outside JNI/JVMTI code may call in with
outdated jmethodids which we must be able to handle gracefully.
This is the current design and most likely will remain for quite a bit but it is not defined by the SPEC to be like this so, in essence, it is an implementation detail that could be not true in future releases ;-)
I politely disagree :)
jmethodID GetMethodID(..)
returning a jmethodid which you can then use in subsequent JNI
functions. These functions are required to behave in a predictable way
if the jmethodid has gotten invalid. The spec has no way to inform the
caller if a jmethodid has gotten invalid by class unloading. Since the
jmethodid is handed up to the caller and stored by him, there is no
way to change its value on the fly either.
So even though the spec does not specifically say that jmethodid lives
forever, it is laid out in a way that prevents jmethodids from ever
becoming invalid, and from changing their numerical value. This can of
course change, but not without changing the JNI spec.
I think my point is that Peter can rely on this behavior without fear
that it will change in the future without him noticing. Before this
behavior changes, the JNI spec would have to change.
Cheers, Thomas
Thanks,
Jc
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-28 14:59:07 UTC
Permalink
Hi Coleen,

(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
Wait, really? So, As a JNI caller I should do this:

jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?

I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Also, the jmethodIDs were never in the PermGen but in CHeap allocated
memory blocks. The Method* that was in the jmethodID was in the
permgen. I don't think there was any guarantee of contiguity (is that a
word) but they are allocated together in a block. After permgen was
removed, the blocks were sorted by class loader, so that the class
loader data could point to them for clearing.
Ah, thank you for for clarifying. I guess I was reading your change
for "6964458: Reimplement class meta-data storage to use native
memory" wrong.

I took a second closer look at now jmethodIDs were generated before
6964458. I think see now how it worked: they were weak global
references living in a malloced JNIHandleBlock, similar to today, and
were set to NULL once the Method* is unloaded, but never removed since
NULL is != deleted_handle().

Thanks, Thomas
Thanks,
Coleen
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-28 15:00:20 UTC
Permalink
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
Post by Thomas Stüfe
Also, the jmethodIDs were never in the PermGen but in CHeap allocated
memory blocks. The Method* that was in the jmethodID was in the
permgen. I don't think there was any guarantee of contiguity (is that a
word) but they are allocated together in a block. After permgen was
removed, the blocks were sorted by class loader, so that the class
loader data could point to them for clearing.
Ah, thank you for for clarifying. I guess I was reading your change
for "6964458: Reimplement class meta-data storage to use native
memory" wrong.
I took a second closer look at now jmethodIDs were generated before
6964458. I think see now how it worked: they were weak global
references living in a malloced JNIHandleBlock, similar to today, and
were set to NULL once the Method* is unloaded, but never removed since
NULL is != deleted_handle().
Thanks, Thomas
Thanks,
Coleen
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
c***@oracle.com
2018-11-28 15:03:26 UTC
Permalink
Post by Thomas Stüfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native JNI
code is supposed to deal with nulled out jmethodIDs.

Maybe they predictably crash?

Coleen
Post by Thomas Stüfe
Post by Thomas Stüfe
Also, the jmethodIDs were never in the PermGen but in CHeap allocated
memory blocks. The Method* that was in the jmethodID was in the
permgen. I don't think there was any guarantee of contiguity (is that a
word) but they are allocated together in a block. After permgen was
removed, the blocks were sorted by class loader, so that the class
loader data could point to them for clearing.
Ah, thank you for for clarifying. I guess I was reading your change
for "6964458: Reimplement class meta-data storage to use native
memory" wrong.
I took a second closer look at now jmethodIDs were generated before
6964458. I think see now how it worked: they were weak global
references living in a malloced JNIHandleBlock, similar to today, and
were set to NULL once the Method* is unloaded, but never removed since
NULL is != deleted_handle().
Thanks, Thomas
Thanks,
Coleen
Post by JC Beyler
Thanks,
Jc
Post by Thomas Stüfe
Post by Thomas Stüfe
But it is not guaranteed to work. I would probably rather use a
hashmap or similar.
I need to look at the implications on more detail but think it would
make sense to use long/jlong instead of int/jint on all platforms; the
extra memory use shouldn't be a problem. I think the IDs are just
stored on the Java side and used to get the method name and signature
later. That should be a loss-free cast, shouldn't it?
Sure.
Post by Thomas Stüfe
If this is
true, this 4x30bit assumption may actually have worked before jdk8,
since the java heap is allocated as one continuous space, with the
PermGen clustered in one part of it.
Indeed we did only start to get crashes on JDK9 and later (only
observed on Windows, macOS seems OK and other platforms have not been
tested)
Yours sincerely,
Peter
Cheers, Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-28 15:08:49 UTC
Permalink
Post by c***@oracle.com
Post by Thomas Stüfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native JNI
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.

Save that JC wrote that there are some JNI function sequences where
the VM would still crashes:

<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>

which looks like just a bug to me.

..Thomas
c***@oracle.com
2018-11-28 15:19:55 UTC
Permalink
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native JNI
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function sequences where
<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also.  We don't guarantee a lot of things with
JNI.  Maybe instead of clearing, we could install a Method* that throws
NSME.

But I guess why leak the jmethodID memory if it's going to crash anyway
when using it?

Coleen
Post by Thomas Stüfe
..Thomas
Thomas Stüfe
2018-11-28 15:37:13 UTC
Permalink
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native JNI
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function sequences where
<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also. We don't guarantee a lot of things with
JNI. Maybe instead of clearing, we could install a Method* that throws
NSME.
But I guess why leak the jmethodID memory if it's going to crash anyway
when using it?
Precisely :) We pay for it, we may just as well use it.

..Thomas
Coleen
Post by Thomas Stüfe
..Thomas
JC Beyler
2018-11-28 16:22:32 UTC
Permalink
We pay for it to support easily the JVMTI spec. JNI does not check for NULL
and will crash if you pass a jmethod that points to NULL. I've checked that
pretty much every method will crash as they all call resolve_jmethod_id at
the start.

As I said before, it's not a bug, the spec is just ambiguous because at the
method definitions of the spec it just says the jmethodIDs have to come
from a GetMethodID call but the more general spec that both I and Dean
quoted say that it is the native agent's job to ensure the class does not
get unloaded to keep the jmethod valid [1].

Thanks,
Jc

[1]
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
Post by Thomas Stüfe
Post by c***@oracle.com
On Wed, Nov 28, 2018 at 3:59 PM Thomas StÃŒfe <
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code
should
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
first test whether it's NULL. I think that is the predictable
behavior
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be
changed
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native JNI
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function sequences where
<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also. We don't guarantee a lot of things with
JNI. Maybe instead of clearing, we could install a Method* that throws
NSME.
But I guess why leak the jmethodID memory if it's going to crash anyway
when using it?
Precisely :) We pay for it, we may just as well use it.
..Thomas
Coleen
Post by Thomas Stüfe
..Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-28 16:36:01 UTC
Permalink
What I meant was that since we already pay for the memory leak, we
could just change this behavior and handle NULL methodIDs gracefully.
We do this already for JVMTI.

Otherwise, if we do not what to do this check and consider this to be
the caller's responsibility, I do not see the point of keeping the
NULLed-out jmethodID tables around. What for, just to make the crash
to be a bit more predictable?

..Thomas
We pay for it to support easily the JVMTI spec. JNI does not check for NULL and will crash if you pass a jmethod that points to NULL. I've checked that pretty much every method will crash as they all call resolve_jmethod_id at the start.
As I said before, it's not a bug, the spec is just ambiguous because at the method definitions of the spec it just says the jmethodIDs have to come from a GetMethodID call but the more general spec that both I and Dean quoted say that it is the native agent's job to ensure the class does not get unloaded to keep the jmethod valid [1].
Thanks,
Jc
[1] https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native code should
first test whether it's NULL. I think that is the predictable behavior
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be changed
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI code".
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native JNI
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function sequences where
<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also. We don't guarantee a lot of things with
JNI. Maybe instead of clearing, we could install a Method* that throws
NSME.
But I guess why leak the jmethodID memory if it's going to crash anyway
when using it?
Precisely :) We pay for it, we may just as well use it.
..Thomas
Coleen
Post by Thomas Stüfe
..Thomas
--
Thanks,
Jc
JC Beyler
2018-11-28 16:46:17 UTC
Permalink
The biggest issue is that the JNI spec has no means of "gracefully" exiting
when a jmethodID is NULL, take the call static methods, the spec says:

"RETURNS:
Returns the result of calling the static Java method.

THROWS:
Exceptions raised during the execution of the Java method."

So now, this has to get fixed to say something like it might throw if the
jmethodID is not valid? Seems "messy" to me, I'd rather just fix the spec
where it says:

PARAMETERS:
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a static method ID.

to saying:
PARAMETERS:
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a valid static method ID.

Thanks,
Jc
Post by Thomas Stüfe
What I meant was that since we already pay for the memory leak, we
could just change this behavior and handle NULL methodIDs gracefully.
We do this already for JVMTI.
Otherwise, if we do not what to do this check and consider this to be
the caller's responsibility, I do not see the point of keeping the
NULLed-out jmethodID tables around. What for, just to make the crash
to be a bit more predictable?
..Thomas
Post by JC Beyler
We pay for it to support easily the JVMTI spec. JNI does not check for
NULL and will crash if you pass a jmethod that points to NULL. I've checked
that pretty much every method will crash as they all call
resolve_jmethod_id at the start.
Post by JC Beyler
As I said before, it's not a bug, the spec is just ambiguous because at
the method definitions of the spec it just says the jmethodIDs have to come
from a GetMethodID call but the more general spec that both I and Dean
quoted say that it is the native agent's job to ensure the class does not
get unloaded to keep the jmethod valid [1].
Post by JC Beyler
Thanks,
Jc
[1]
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
On Wed, Nov 28, 2018 at 3:59 PM Thomas StÃŒfe <
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native
code should
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
first test whether it's NULL. I think that is the predictable
behavior
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be
changed
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI
code".
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native
JNI
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function sequences where
<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also. We don't guarantee a lot of things with
JNI. Maybe instead of clearing, we could install a Method* that
throws
Post by JC Beyler
Post by Thomas Stüfe
NSME.
But I guess why leak the jmethodID memory if it's going to crash
anyway
Post by JC Beyler
Post by Thomas Stüfe
when using it?
Precisely :) We pay for it, we may just as well use it.
..Thomas
Coleen
Post by Thomas Stüfe
..Thomas
--
Thanks,
Jc
--
Thanks,
Jc
c***@oracle.com
2018-11-28 17:02:29 UTC
Permalink
Post by JC Beyler
The biggest issue is that the JNI spec has no means of "gracefully"
exiting when a jmethodID is NULL, take the call static methods, the
Returns the result of calling the static Java method.
Exceptions raised during the execution of the Java method."
So now, this has to get fixed to say something like it might throw if
the jmethodID is not valid? Seems "messy" to me, I'd rather just fix
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a static method ID.
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a valid static method ID.
I don't think the spec should be changed so that jmethodID prevents a
class from being unloaded.
Post by JC Beyler
Thanks,
Jc
What I meant was that since we already pay for the memory leak, we
could just change this behavior and handle NULL methodIDs gracefully.
We do this already for JVMTI.
Otherwise, if we do not what to do this check and consider this to be
the caller's responsibility, I do not see the point of keeping the
NULLed-out jmethodID tables around. What for, just to make the crash
to be a bit more predictable?
..Thomas
Post by JC Beyler
We pay for it to support easily the JVMTI spec. JNI does not
check for NULL and will crash if you pass a jmethod that points to
NULL. I've checked that pretty much every method will crash as
they all call resolve_jmethod_id at the start.
Post by JC Beyler
As I said before, it's not a bug, the spec is just ambiguous
because at the method definitions of the spec it just says the
jmethodIDs have to come from a GetMethodID call but the more
general spec that both I and Dean quoted say that it is the native
agent's job to ensure the class does not get unloaded to keep the
jmethod valid [1].
Post by JC Beyler
Thanks,
Jc
[1]
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
I believe this describes the JNI function table, not jmethodIDs.

No, a jmethodID doesn't keep the class from being unloaded, nor do we
want it to.  This would add undue burden to the GCs, in that they would
have to trace metadata or some side structure of oops to keep classes
alive that have jmethodIDs.

Coleen
Post by JC Beyler
Post by JC Beyler
On Wed, Nov 28, 2018 at 7:38 AM Thomas StÃŒfe
Post by Thomas Stüfe
On Wed, Nov 28, 2018 at 4:03 PM
Post by c***@oracle.com
On Wed, Nov 28, 2018 at 3:59 PM Thomas StÃŒfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss
before :-)
Post by JC Beyler
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
If the class is unloaded, the jmethodID is cleared. 
Native code should
Post by JC Beyler
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
first test whether it's NULL.  I think that is the
predictable behavior
Post by JC Beyler
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. }  ?
I thought jmethodid is opaque, and its value itself
cannot be changed
Post by JC Beyler
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
from the VM, no?
Oh you probably meant "native code in the VM", not
"native JNI code".
Post by JC Beyler
Post by Thomas Stüfe
Post by c***@oracle.com
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how
native JNI
Post by JC Beyler
Post by Thomas Stüfe
Post by c***@oracle.com
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought  it would gracefully reject, e.g. on JVMTI
with
Post by JC Beyler
Post by Thomas Stüfe
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function
sequences where
Post by JC Beyler
Post by Thomas Stüfe
<quote jc>
     - Get a jmethodID and remember it
     - Wait until the class gets unloaded
     - Get the class to get reloaded and try call the old
jmethodID
Post by JC Beyler
Post by Thomas Stüfe
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also.  We don't guarantee a lot of
things with
Post by JC Beyler
Post by Thomas Stüfe
JNI.  Maybe instead of clearing, we could install a Method*
that throws
Post by JC Beyler
Post by Thomas Stüfe
NSME.
But I guess why leak the jmethodID memory if it's going to
crash anyway
Post by JC Beyler
Post by Thomas Stüfe
when using it?
Precisely :) We pay for it, we may just as well use it.
..Thomas
Coleen
..Thomas
--
Thanks,
Jc
--
Thanks,
Jc
JC Beyler
2018-11-28 17:04:36 UTC
Permalink
Sorry I meant:
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp17074

:)
Jc
Post by JC Beyler
The biggest issue is that the JNI spec has no means of "gracefully"
exiting when a jmethodID is NULL, take the call static methods, the spec
Returns the result of calling the static Java method.
Exceptions raised during the execution of the Java method."
So now, this has to get fixed to say something like it might throw if the
jmethodID is not valid? Seems "messy" to me, I'd rather just fix the spec
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a static method ID.
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a valid static method ID.
I don't think the spec should be changed so that jmethodID prevents a
class from being unloaded.
Thanks,
Jc
Post by Thomas Stüfe
What I meant was that since we already pay for the memory leak, we
could just change this behavior and handle NULL methodIDs gracefully.
We do this already for JVMTI.
Otherwise, if we do not what to do this check and consider this to be
the caller's responsibility, I do not see the point of keeping the
NULLed-out jmethodID tables around. What for, just to make the crash
to be a bit more predictable?
..Thomas
Post by JC Beyler
We pay for it to support easily the JVMTI spec. JNI does not check for
NULL and will crash if you pass a jmethod that points to NULL. I've checked
that pretty much every method will crash as they all call
resolve_jmethod_id at the start.
Post by JC Beyler
As I said before, it's not a bug, the spec is just ambiguous because at
the method definitions of the spec it just says the jmethodIDs have to come
from a GetMethodID call but the more general spec that both I and Dean
quoted say that it is the native agent's job to ensure the class does not
get unloaded to keep the jmethod valid [1].
Post by JC Beyler
Thanks,
Jc
[1]
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
I believe this describes the JNI function table, not jmethodIDs.
No, a jmethodID doesn't keep the class from being unloaded, nor do we want
it to. This would add undue burden to the GCs, in that they would have to
trace metadata or some side structure of oops to keep classes alive that
have jmethodIDs.
Coleen
Post by Thomas Stüfe
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
On Wed, Nov 28, 2018 at 3:59 PM Thomas StÃŒfe <
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off discuss before :-)
If the class is unloaded, the jmethodID is cleared. Native
code should
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
first test whether it's NULL. I think that is the predictable
behavior
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. } ?
I thought jmethodid is opaque, and its value itself cannot be
changed
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Post by Thomas Stüfe
from the VM, no?
Oh you probably meant "native code in the VM", not "native JNI
code".
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
Sorry for the confusion.
I did mean native JNI code, but I actually don't know how native
JNI
Post by JC Beyler
Post by Thomas Stüfe
Post by Thomas Stüfe
Post by c***@oracle.com
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought it would gracefully reject, e.g. on JVMTI with
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function sequences where
<quote jc>
- Get a jmethodID and remember it
- Wait until the class gets unloaded
- Get the class to get reloaded and try call the old jmethodID
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also. We don't guarantee a lot of things
with
Post by JC Beyler
Post by Thomas Stüfe
JNI. Maybe instead of clearing, we could install a Method* that
throws
Post by JC Beyler
Post by Thomas Stüfe
NSME.
But I guess why leak the jmethodID memory if it's going to crash
anyway
Post by JC Beyler
Post by Thomas Stüfe
when using it?
Precisely :) We pay for it, we may just as well use it.
..Thomas
Coleen
Post by Thomas Stüfe
..Thomas
--
Thanks,
Jc
--
Thanks,
Jc
--
Thanks,
Jc
David Holmes
2018-11-28 21:06:15 UTC
Permalink
Not sure if this discussion is diverging but to be clear there is no way
for a JNI programmer to check whether a jmethodid is valid or not. The
onus is one the programmer to ensure classes are not unloaded while
there are pre-computed jmethodid's. It's a quality of implementation
issue, to me, how/if use of an invalid jmethodid is detected.

Any API that accepts a jmethodid implicitly assumes/expects it is a
valid one. That is not something we should have to state.

David
Post by JC Beyler
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp17074
:)
Jc
Post by JC Beyler
The biggest issue is that the JNI spec has no means of
"gracefully" exiting when a jmethodID is NULL, take the call
Returns the result of calling the static Java method.
Exceptions raised during the execution of the Java method."
So now, this has to get fixed to say something like it might throw
if the jmethodID is not valid? Seems "messy" to me, I'd rather
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a static method ID.
env: the JNI interface pointer.
clazz: a Java class object.
methodID: a valid static method ID.
I don't think the spec should be changed so that jmethodID prevents
a class from being unloaded.
Post by JC Beyler
Thanks,
Jc
On Wed, Nov 28, 2018 at 8:36 AM Thomas Stüfe
What I meant was that since we already pay for the memory leak, we
could just change this behavior and handle NULL methodIDs gracefully.
We do this already for JVMTI.
Otherwise, if we do not what to do this check and consider this to be
the caller's responsibility, I do not see the point of keeping the
NULLed-out jmethodID tables around. What for, just to make the crash
to be a bit more predictable?
..Thomas
Post by JC Beyler
We pay for it to support easily the JVMTI spec. JNI does not
check for NULL and will crash if you pass a jmethod that
points to NULL. I've checked that pretty much every method
will crash as they all call resolve_jmethod_id at the start.
Post by JC Beyler
As I said before, it's not a bug, the spec is just ambiguous
because at the method definitions of the spec it just says the
jmethodIDs have to come from a GetMethodID call but the more
general spec that both I and Dean quoted say that it is the
native agent's job to ensure the class does not get unloaded
to keep the jmethod valid [1].
Post by JC Beyler
Thanks,
Jc
[1]
https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp16696
I believe this describes the JNI function table, not jmethodIDs.
No, a jmethodID doesn't keep the class from being unloaded, nor do
we want it to.  This would add undue burden to the GCs, in that they
would have to trace metadata or some side structure of oops to keep
classes alive that have jmethodIDs.
Coleen
Post by JC Beyler
Post by JC Beyler
On Wed, Nov 28, 2018 at 7:38 AM Thomas Stüfe
On Wed, Nov 28, 2018 at 4:19 PM
Post by c***@oracle.com
On Wed, Nov 28, 2018 at 4:03 PM
Post by c***@oracle.com
On Wed, Nov 28, 2018 at 3:59 PM Thomas Stüfe
Post by Thomas Stüfe
Hi Coleen,
(moved to svc-dev since David did shoo us off
discuss before :-)
Post by JC Beyler
Post by c***@oracle.com
Post by c***@oracle.com
Post by Thomas Stüfe
If the class is unloaded, the jmethodID is
cleared.  Native code should
Post by JC Beyler
Post by c***@oracle.com
Post by c***@oracle.com
Post by Thomas Stüfe
first test whether it's NULL.  I think that is the
predictable behavior
Post by JC Beyler
Post by c***@oracle.com
Post by c***@oracle.com
Post by Thomas Stüfe
that the spec requires.
jmethodID method;
..
if (*method == NULL) { .. invalid method id .. }  ?
I thought jmethodid is opaque, and its value itself
cannot be changed
Post by JC Beyler
Post by c***@oracle.com
Post by c***@oracle.com
Post by Thomas Stüfe
from the VM, no?
Oh you probably meant "native code in the VM", not
"native JNI code".
Post by JC Beyler
Post by c***@oracle.com
Post by c***@oracle.com
Sorry for the confusion.
I did mean native JNI code, but I actually don't know
how native JNI
Post by JC Beyler
Post by c***@oracle.com
Post by c***@oracle.com
code is supposed to deal with nulled out jmethodIDs.
Maybe they predictably crash?
Coleen
I always thought  it would gracefully reject, e.g. on
JVMTI with
Post by JC Beyler
Post by c***@oracle.com
JVMTI_ERROR_INVALID_METHODID.
Save that JC wrote that there are some JNI function
sequences where
Post by JC Beyler
Post by c***@oracle.com
<quote jc>
     - Get a jmethodID and remember it
     - Wait until the class gets unloaded
     - Get the class to get reloaded and try call the
old jmethodID
Post by JC Beyler
Post by c***@oracle.com
(which now points to NULL), the code will segfault
</quote>
which looks like just a bug to me.
It may be misuse of JNI also.  We don't guarantee a lot
of things with
Post by JC Beyler
Post by c***@oracle.com
JNI.  Maybe instead of clearing, we could install a
Method* that throws
Post by JC Beyler
Post by c***@oracle.com
NSME.
But I guess why leak the jmethodID memory if it's going
to crash anyway
Post by JC Beyler
Post by c***@oracle.com
when using it?
Precisely :) We pay for it, we may just as well use it.
..Thomas
Post by c***@oracle.com
Coleen
..Thomas
--
Thanks,
Jc
--
Thanks,
Jc
--
Thanks,
Jc
Loading...