We are getting this intermittent error (presume under relatively higher load than normal circumstances) Server side function executions gets broken pipe while trying to write back the response. Its clear client hung up before server function is done. Trying to understand what parameters client uses to hung up / Server uses to terminate the connection.
System Alert from Distributed System
: [warning 2013/02/01 19:31:52.594 HKT Function Execution Processor22759 tid=0x111dd] Exception on server while executing function : com.************************.GetPositions@268354ce
com.gemstone.gemfire.cache.execute.FunctionException: IOException while sending the last chunk to client
at com.gemstone.gemfire.internal.cache.execute.ServerToClientFunctionResultSender65.lastResult(ServerToClientFunctionResultSender65.java:85)
at com.gemstone.gemfire.internal.cache.execute.DistributedRegionFunctionResultSender.lastResult(DistributedRegionFunctionResultSender.java:82)
at com.gemstone.gemfire.internal.cache.execute.DistributedRegionFunctionResultSender.sendException(DistributedRegionFunctionResultSender.java:218)
at com.************************..GetPositions.execute(GetPositions.java:73)
at com.gemstone.gemfire.internal.cache.execute.AbstractExecution.executeFunctionLocally(AbstractExecution.java:337)
at com.gemstone.gemfire.internal.cache.execute.AbstractExecution$2.run(AbstractExecution.java:301)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at com.gemstone.gemfire.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:639)
at com.gemstone.gemfire.distributed.internal.DistributionManager$8$1.run(DistributionManager.java:1063)
at java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at com.gemstone.gemfire.internal.cache.tier.sockets.Message.flushBuffer(Message.java:538)
at com.gemstone.gemfire.internal.cache.tier.sockets.Message.sendBytes(Message.java:514)
at com.gemstone.gemfire.internal.cache.tier.sockets.ChunkedMessage.sendChunk(ChunkedMessage.java:306)
at com.gemstone.gemfire.internal.cache.execute.ServerToClientFunctionResultSender65.lastResult(ServerToClientFunctionResultSender65.java:79)
... 10 more
From observation we see below message before the above ones. Not necesarily always its just an observation.
System Alert from Distributed System
: [warning 2013/02/01 19:31:50.906 HKT ServerConnection on port 54413 Thread 32147 tid=0x111e9] Server connection from [identity(*****************(26906:loner):55257:55d79179:ServiceClient,connection=3; port=45025]: Handshake accept failed on socket Socket[addr=/14.129.48.113,port=45025,localport=54413]: java.net.SocketException: Broken pipe
Any bells?