ActiveMQ producer/consumer hangs when on of the consumer didn't respond

[ashoknan]

We are using ActiveMQ 5.1 & Spring 2.5.4...
We are using ActiveMQ Embeeded broker

<bean id="jmsFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<value>vm://localhost?broker.persistent=false</value>
</property>
</bean>
</property>
</bean>

<bean id="destination" class=" org.apache.activemq.command.ActiveMQQueue">
<constructor-arg ref="jmsQueueJndiName"></constructor-arg>
</bean>

<bean id="calendarEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="calendarEventProcessor" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_CALENDAR"/>
</property>
</bean>

<bean id="runtimeEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="runtimeEventProcessor" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_RUNTIME"/>
</property>
</bean>

<bean id="systemEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="systemEventProcessor" />
<property name="concurrentConsumers" value="5" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_SYSTEM"/>
</property>
</bean>

<bean id="genericCalendarEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="genericCalendarEventProcessor" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_GENERIC"/>
</property>
</bean>

We are sending the email from all these listeners after doing some
logic.when something happens in the SMTP side and if there is no response
from SMTP side and if the Listener is waiting then the entire
producer/consumer is halted.

My Question is why One Consumer blocking all other consumer /producer. (see
the stack trace which shows one of the Consumer is waiting for some response
from SMTP server).

ActiveMQ Session Task" - Thread ***@2554
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:97)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)

locked ***@13a80ea
at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
at
com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:1440)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1260)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
at javax.mail.Service.connect(Service.java:275)
at
org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:389)
at
org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:342)
at
org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:338)
at
com.jpmc.ibtech.spring.email.MimeEmailService.sendEmail(MimeEmailService.java:61)
at
com.jpmc.spex.background.events.processor.AbstractEventAction.sendEmail(AbstractEventAction.java:63)
at
com.jpmc.spex.background.events.processor.AbstractEventAction.execute(AbstractEventAction.java:79)
at
com.jpmc.spex.background.events.processor.system.SystemEventAction.execute(SystemEventAction.java:47)
at
com.jpmc.ibtech.spring.jms.EventProcessor.onMessage(EventProcessor.java:101)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:531)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:466)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:407)
at
org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:290)
at
org.springframework.jms.listener.SimpleMessageListenerContainer$2.onMessage(SimpleMessageListenerContainer.java:266)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:967)
locked ***@1e58347
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:122)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:192)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:

I was wondering why this is blocking other concurrent consumer for the same
selector.

locked java.util.concurrent.locks.ReentrantLock$***@10dbec7
See the attached ThreadDump which shows Producer is waiting..

I saw that..I can enable ProduceFlowControl...I am just worried whether it
will start spilling to the disk and create more problems.

I am not sure whether setting SMTP timeout will avoid the consumer waiting
indefinitely.


http://www.nabble.com/file/p22586789/Thread%252BDump%252Bfor%252BActiveMQ_2.tdump.txt
Thread%2BDump%2Bfor%2BActiveMQ_2.tdump.txt

[ashoknan]

Am i not supposed to use PooledConnectionFactory for consuming messages.See
the below Java Doc

http://activemq.apache.org/maven/act...onFactory.html


A JMS provider which pools Connection, Session and MessageProducer instances
so it can be used with tools like Spring's JmsTemplate. NOTE this
implementation is only intended for use when sending messages. It does not
deal with pooling of consumers; for that look at a library like Jencks such
as in this example
I am not sure whether this is outdated.

Please let me know if anyone faced this issue before

[Gary Tully]

Prefetch <http://activemq.apache.org/what-is-the-prefetch-limit-for.html> is
the problem with pooling consumers. A consumers pre-fetched messages, when
that consumer is in a pool, may stay there till the pool is closed and will
not be available to other consumers.

Post by ashoknan
Am i not supposed to use PooledConnectionFactory for consuming messages.See
the below Java Doc
http://activemq.apache.org/maven/act...onFactory.html
A JMS provider which pools Connection, Session and MessageProducer instances
so it can be used with tools like Spring's JmsTemplate. NOTE this
implementation is only intended for use when sending messages. It does not
deal with pooling of consumers; for that look at a library like Jencks such
as in this example
I am not sure whether this is outdated.
Please let me know if anyone faced this issue before
--
http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22614510.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[ashoknan]

Thanks Gary for your response..We have a prefetch of
10..jms.prefetchPolicy.all=10.

The thing which i am not sure ,why when one of the Consumer got the stuck in
the smtp response other consumer for the same selector is stuck.(I have 5
Conusmer for the same selector). Not only the consumer got stuck the entire
producer/consumer is stuck(Attachment has thread stack). See the queue stat
when the entire producer/consumer is stuck.
queueSize=20548
memoryPercentUsage=100
memoryUsagePortion=1.0
memoryLimit=67108864
enqueueCount=400720
dequeueCount=380172
inFlightCount=10
maxEnqueueTime=227302
minEnqueueTime=1
averageEnqueueTime=32840.18067399298
consumerCount=8
producerCount=0
isProducerFlowControl=true


Is it because that particular consumer is holding the shared connection?

Also i am thinking whether setting the SMTP timeout will avoid the thread
getting stuck forever and it may not block the connection.

Does it make sense?

Post by Gary Tully
Prefetch <http://activemq.apache.org/what-is-the-prefetch-limit-for.html> is
the problem with pooling consumers. A consumers pre-fetched messages, when
that consumer is in a pool, may stay there till the pool is closed and will
not be available to other consumers.

Post by ashoknan
Am i not supposed to use PooledConnectionFactory for consuming messages.See
the below Java Doc
http://activemq.apache.org/maven/act...onFactory.html
A JMS provider which pools Connection, Session and MessageProducer instances
so it can be used with tools like Spring's JmsTemplate. NOTE this
implementation is only intended for use when sending messages. It does not
deal with pooling of consumers; for that look at a library like Jencks such
as in this example
I am not sure whether this is outdated.
Please let me know if anyone faced this issue before
--
http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22614510.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com