Recently, I was performing a load test on Cometd and Jetty together. The Cometd had a version 1.0.0(too old, not even maintained now) and Jetty version 8.1.11. I found that my Jetty was going out of memory, its heap memory allocated for the process was full. So the notifications started failing.

First, I analyzed the jetty logs and found that logs were full with Timeout Exceptions in between the notifications. Timeout exceptions were like

WARN:oejut.Timeout:EXCEPTION java.lang.NoSuchMethodError: org.eclipse.jetty.util.LazyList.removeFromArray([Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; at org.cometd.server.ClientImpl.removeSubscription(ClientImpl.java:495) at org.cometd.server.ChannelImpl.unsubscribe(ChannelImpl.java:339) at org.cometd.server.ClientImpl.unsubscribeAll(ClientImpl.java:527) at org.cometd.server.AbstractBayeux.removeClient(AbstractBayeux.java:526) at org.cometd.server.ClientImpl.remove(ClientImpl.java:370) at org.cometd.server.continuation.ContinuationClient.remove(ContinuationClient.java:220) at org.cometd.server.continuation.ContinuationClient$1.expired(ContinuationClient.java:60) at org.eclipse.jetty.util.thread.Timeout.tick(Timeout.java:140) at org.eclipse.jetty.util.thread.Timeout.tick(Timeout.java:153) at org.cometd.server.continuation.ContinuationBayeux$1.run(ContinuationBayeux.java:76) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462)

Now this was the time to take the heap dump and analyze it to find who is the real culprit. The following command helped me in taking the heap dump.

jmap -F -dump:File=heapDump <jetty-pid>

You can also run the following command to check the instances of each and every object.

jmap -F -histo <jetty-pid>

Now I analyzed the heap dump using Java VisualVM(). I found following classes had the maximum number of instances(almost 2.3 million).

org.eclipse.jetty.util.ArrayQueue

org.cometd.server.ChannelImpl[]

org.cometd.server.continuation.ContinuationClient

org.cometd.server.continuation.ContinuationClient$1

org.cometd.server.continuation.ContinuationClient$2

Now it was time to check the code, but before that, I also found the number of instances of the ContinuationClient, ContinuationClient$1, ContinuationClient$2 was equal to the number of Timeout exceptions that had occurred. So from the heap dump and the logs, it was quite clear that the removeFromArray method is not functioning properly due to which the instances were not getting cleaned up.

On checking the code, I was unable to find any issue, so I thought of upgrading the cometd to a version which supports Jetty 8.1.11. So, I searched it on the web and put a question on SO. The answers to the question were quite clear and helped me to find where the issue lies.

from the answers, it seems like the best way to find cometd version compatibility with jetty version is to check the pom.xml of specified version of cometd.

For example, the cometd 2.6 pom.xml says it is built with jetty version 7.6.10.v20130312 , while the cometd 2.7 pom.xml says it is built with jetty version 7.6.13.v20130916 and 8.1.13.v20130916.

So now the solution to my problem seems to be both jetty and cometd latest upgrade.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s