OK, let me tell you a painful story about a developer an application and an application server called Glassfish 3.1.2.2 ;-) Beware, this story is still in progress so I may still be updating this post till the issue is resolved.
(Forgive me my poetic mood - this post will be less technical than the previous ones ..though you will find some technical details, solutions etc in this text .. somewhere ;-)
So what the problem was/is? At the beginning it appeared to be a total mystery. After running cheerfully on the server for days or even weeks...application hangs ...and nothing could be done to bring it back to life, except for restarting the whole server (I mean the machine itself) - and we are talking about production environment here!
Something had be done ..but what ? Fortunately, there is this monitoring tool that came to my attention - VisualVM and it has glassfish plugin. Installation of this product was definitely first step in the right direction.
The tool revealed that during the freeze or 'deadlock' all http thread-pool threads were waiting to lock Object which was held by "connector-timer-proxy". Connector-timer-proxy itself was trying to reclaim a connection that was marked as 'leaked' but for some reason couldn't reclaim it. Taking a look at database connections revealed 4 inactive connections....and one active (the one that should have been reclaimed).
Killing the database connection - brought Glassfish back to life ... everything unfreezed!
My thoughts on this one are that the connection was actually doing something (admittedly it must have been a very long operation) and thus the connection couldn't be reclaimed. Or I am wrong here? I don't know yet.
One thing I did, I set statement timeout on Glassfish. So I suppose if for some reason I have a statement running for too long ... it will be canceled first and then if need the connection could be reclaimed without problem... Let's hope it is the case, and not the fact that I am dealing with some kind of GF's bug...(Time will show).
Anyway, if you have any ideas on this matter - don't be shy - let me know...
