[pool] Possible deadlock or user error (me) commons-pool 2.5.0

Discussion:

Bruce Milner

2018-09-07 21:56:23 UTC

Hello,

I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.

I recently replaced a lot of hand crafted code with the commons-pool2
implementation.

The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.

The config is basically
        GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
        config.setBlockWhenExhausted(true);
        config.setMaxTotal(120);
        config.setMaxTotalPerKey(60);
        config.setTestOnBorrow(true);
        config.setTimeBetweenEvictionRunsMillis(60000);
        config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
        /**
         * For database connections, use FIFO so that we get rid of
older connections first before newer ones.
         */
        config.setLifo(false);
        return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);

There are 150 of these threads waiting on a lock to release connections
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
    at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)

and 158 of these threads waiting to open connections.
java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
    at com.ilrn.util.sql.Database.getConnection(Database.java:1273)

For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.

Has anyone run into this? It seems like some sort of deadlock.

--
Bruce Milner
Senior Software Developer (Emberex)

---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org
For additional commands, e-mail: user-***@commons.apache.org

Gary Gregory

2018-09-08 00:05:21 UTC

Permalink

Hi,

A side question: Have you tried Apache Commons DBCP (which is based on
Commons Pool)?

https://commons.apache.org/proper/commons-dbcp/

Gary

Post by Bruce Milner
Hello,
I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.
I recently replaced a lot of hand crafted code with the commons-pool2
implementation.
The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.
The config is basically
GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
config.setBlockWhenExhausted(true);
config.setMaxTotal(120);
config.setMaxTotalPerKey(60);
config.setTestOnBorrow(true);
config.setTimeBetweenEvictionRunsMillis(60000);
config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
/**
* For database connections, use FIFO so that we get rid of
older connections first before newer ones.
*/
config.setLifo(false);
return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);
There are 150 of these threads waiting on a lock to release connections
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
and 158 of these threads waiting to open connections.
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.
Has anyone run into this? It seems like some sort of deadlock.
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------

Bruce Milner

2018-09-08 18:19:38 UTC

Permalink

Hello,

I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.

With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.

The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.

I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45

--bruce

Post by Gary Gregory
Hi,
A side question: Have you tried Apache Commons DBCP (which is based on
Commons Pool)?
https://commons.apache.org/proper/commons-dbcp/
Gary

Bernd Eckenfels

2018-09-09 01:51:18 UTC

Permalink

Hello Bruce,

This sounds a bit like a discussion we had about missing wakeups. I think its was related to depleted pools. Didnt find the discussion, hopefully somebody else recalls the conditions? I think it was not fixed.

Gruss
Bernd
--
http://bernd.eckenfels.net

________________________________
Von: Bruce Milner <***@emberex.com>
Gesendet: Samstag, September 8, 2018 8:19 PM
An: ***@commons.apache.org
Betreff: Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Hello,

I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.

With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.

The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.

I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45

--bruce

Post by Gary Gregory
Hi,
A side question: Have you tried Apache Commons DBCP (which is based on
Commons Pool)?
https://commons.apache.org/proper/commons-dbcp/
Gary

Mark Thomas

2018-09-08 18:36:39 UTC

Permalink

Post by Bruce Milner
Hello,
I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.
I recently replaced a lot of hand crafted code with the commons-pool2
implementation.
The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.
The config is basically
        GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
        config.setBlockWhenExhausted(true);
        config.setMaxTotal(120);
        config.setMaxTotalPerKey(60);
        config.setTestOnBorrow(true);
        config.setTimeBetweenEvictionRunsMillis(60000);
        config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
        /**
         * For database connections, use FIFO so that we get rid of
older connections first before newer ones.
         */
        config.setLifo(false);
        return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);
There are 150 of these threads waiting on a lock to release connections
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
    at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
and 158 of these threads waiting to open connections.
java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
    at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.
Has anyone run into this? It seems like some sort of deadlock.

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org
For additional commands, e-mail: user-***@commons.apache.org

Bruce Milner

2018-09-09 03:17:17 UTC

Permalink

Hello Mark,

I sent the thread dump directly to you.

Post by Mark Thomas

Post by Bruce Milner
Hello,
I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.
I recently replaced a lot of hand crafted code with the commons-pool2
implementation.
The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.
The config is basically
        GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
        config.setBlockWhenExhausted(true);
        config.setMaxTotal(120);
        config.setMaxTotalPerKey(60);
        config.setTestOnBorrow(true);
        config.setTimeBetweenEvictionRunsMillis(60000);
        config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
        /**
         * For database connections, use FIFO so that we get rid of
older connections first before newer ones.
         */
        config.setLifo(false);
        return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);
There are 150 of these threads waiting on a lock to release connections
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
    at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
and 158 of these threads waiting to open connections.
java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
    at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
    at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
    at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.
Has anyone run into this? It seems like some sort of deadlock.

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?
Mark
---------------------------------------------------------------------

Mark Thomas

2018-09-09 07:16:03 UTC

Permalink

On 08/09/18 19:36, Mark Thomas wrote:

<snip/>

Post by Mark Thomas

Post by Bruce Milner
For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.
Has anyone run into this? It seems like some sort of deadlock.

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?

Thanks. Lots of threads waiting for the lock. None holding it. That is
very strange.

Some variation of this?

https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org
For additional commands, e-mail: user-***@commons.apache.org

Phil Steitz

2018-09-09 17:20:32 UTC

Permalink

Post by Mark Thomas
<snip/>

Post by Mark Thomas

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?

Here is one thing to look at. Note that the take request from
borrowObject comes from
p = objectDeque.getIdleObjects().pollFirst();

This LinkedBlockingDeque method does not register the thread as
waiting on the notEmpty condition in LinkedBlockingDeque the way
that say
p = objectDeque.getIdleObjects().pollFirst(
borrowMaxWaitMillis,
TimeUnit.MILLISECONDS);
does.

Phil

Post by Mark Thomas
Mark
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org
For additional commands, e-mail: user-***@commons.apache.org

Bruce Milner

2018-09-11 22:38:17 UTC

Permalink

Post by Mark Thomas
<snip/>

Post by Mark Thomas

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?

Thanks. Lots of threads waiting for the lock. None holding it. That is
very strange.
Some variation of this?
https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html
Mark
---------------------------------------------------------------------

Yeah, I looked closer at thread dump and no thread holding lock. Also
looked in past several months of logs and no StackOverflowError's .
My guess now is JVM bug.

Thanks for looking!

--bruce