I/O error reading from mux connection: java.io.EOFException

Discussion:

Bryan Thompson

2008-11-18 19:54:21 UTC

Hello,

I was hoping that someone might shed some light on this exception. The exception occurs with Jini 2.1, Java 1.6.0_07, NIO enabled, and running under Windows XP Service pack 2 (my laptop). All services are running on a single host. The code is a modestly complex JOIN and the exception occurs frequently enough to be expected when the data scale is on the order of 10M rows. I've followed the dicussions concerning some possibly related errors, but, really, there does not appear to be much out there on this. Related links and stack trace are below.

Thanks in advance,

-bryan

See http://osdir.com/ml/java.sun.javaspaces/2005-09/msg00015.html

See http://osdir.com/ml/java.sun.javaspaces/2005-09/msg00029.html

See http://forums.sun.com/thread.jspa?threadID=5284995 (perhaps post there?)

See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4868432

See http://osdir.com/ml/java.sun.jini/2004-04/msg00274.html

ERROR: 116515 pool-1-thread-54 SYSTAP-BBT.systap.com 6a2d5f17-c36c-4799-ae4c-f61a2e07461a com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:839): java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: I/O error reading from mux connection: java.io.EOFException

java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: I/O error reading from mux connection: java.io.EOFException

at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)

at java.util.concurrent.FutureTask.get(FutureTask.java:83)

at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:827)

at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1173)

at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:915)

at com.bigdata.service.proxy.ClientAsynchronousIterator.hasNext(ClientAsynchronousIterator.java:514)

at com.bigdata.relation.rule.eval.pipeline.DistributedJoinTask.nextChunk(DistributedJoinTask.java:389)

at com.bigdata.relation.rule.eval.pipeline.JoinTask$BindingSetConsumerTask.call(JoinTask.java:830)

at com.bigdata.relation.rule.eval.pipeline.JoinTask.consumeSources(JoinTask.java:666)

at com.bigdata.relation.rule.eval.pipeline.JoinTask.call(JoinTask.java:449)

at com.bigdata.relation.rule.eval.pipeline.JoinTask.call(JoinTask.java:1)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.RuntimeException: java.io.IOException: I/O error reading from mux connection: java.io.EOFException

at com.bigdata.io.SerializerUtil$1.deserialize(SerializerUtil.java:69)

at com.bigdata.service.proxy.RemoteAsynchronousIteratorImpl$RemoteElementImpl.readExternal(RemoteAsynchronousIteratorImpl.java:201)

at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

at com.sun.jini.jeri.internal.runtime.Util.unmarshalValue(Util.java:221)

at net.jini.jeri.BasicInvocationHandler.unmarshalReturn(BasicInvocationHandler.java:1242)

at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:825)

at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)

at net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)

at $Proxy11.nextElement(Unknown Source)

at com.bigdata.service.proxy.ClientAsynchronousIterator$ReaderTask.call(ClientAsynchronousIterator.java:350)

at com.bigdata.service.proxy.ClientAsynchronousIterator$ReaderTask.call(ClientAsynchronousIterator.java:1)

... 5 more

Caused by: java.io.IOException: I/O error reading from mux connection: java.io.EOFException

at com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read(Session.java:943)

at net.jini.jeri.connection.ConnectionManager$Outbound$Input.read(ConnectionManager.java:521)

at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2266)

at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2674)

at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2698)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1936)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

at com.bigdata.io.SerializerUtil$1.deserialize(SerializerUtil.java:65)

... 18 more

Caused by: java.io.EOFException

at com.sun.jini.jeri.internal.mux.SocketChannelConnectionIO.handleReadReady(SocketChannelConnectionIO.java:384)

at com.sun.jini.jeri.internal.mux.SocketChannelConnectionIO.access$200(SocketChannelConnectionIO.java:41)

at com.sun.jini.jeri.internal.mux.SocketChannelConnectionIO$Handler.handleSelection(SocketChannelConnectionIO.java:464)

at com.sun.jini.jeri.internal.runtime.SelectionManager$SelectLoop.run(SelectionManager.java:288)

at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:136)

... 1 more

--------------------------------------------------------------------------
Getting Started: http://www.jini.org/wiki/Category:Getting_Started
Community Web Site: http://jini.org
jini-users Archive: http://archives.java.sun.com/archives/jini-users.html
Unsubscribing: email "signoff JINI-USERS" to ***@java.sun.com

Bryan Thompson

2008-11-19 13:47:51 UTC

Permalink

Michael,

I have disabled NIO and tried it. So far, I have not seen the mux error
replicated without NIO. However, I do get an OOM after a while. I have
done some more testing on the laptop and I believe that the root cause
is an OOM (both with and without NIO) and that the rest of the errors
are cascaded consequences. So, I think that NIO is not "at fault" but
just that the problem is revealed in a different manner when NIO is
enabled. I plan to do some multi-machine testing on server platforms
and I will see how they behave in contrast to a resource starved laptop.

Thanks,

-bryan

________________________________

From: Michael McConnell [mailto:***@gmail.com]
Sent: Tuesday, November 18, 2008 5:40 PM
To: Bryan Thompson
Subject: Re: I/O error reading from mux connection:
java.io.EOFException

Just curious - can you disable NIO and see what happens?

Also -- saw this posted in one of the archives

> It appears that the server (TransientOutriggerImpl) sees the
following:
>
> (1) Client invokes server with write()
> (2) Server processes request
> (3) Server attempts to send response to client
> (4) Server reports EOFException because client has closed it's
> connection sometime after (1) but before (3).

Yep, seems that way.

> If the client has initiated
> this, it's not a surprise it sees no exception

I wouldn't expect the client-side remote call to both close the
connection this early and still return normally from the call.

> I'd definitely look to enable logging in the JERI layers on
the client
> and see what's happening.

On Tue, Nov 18, 2008 at 1:54 PM, Bryan Thompson
<***@systap.com> wrote:

Hello,

I was hoping that someone might shed some light on this
exception. The exception occurs with Jini 2.1, Java 1.6.0_07, NIO
enabled, and running under Windows XP Service pack 2 (my laptop). All
services are running on a single host. The code is a modestly complex
JOIN and the exception occurs frequently enough to be expected when the
data scale is on the order of 10M rows. I've followed the dicussions
concerning some possibly related errors, but, really, there does not
appear to be much out there on this. Related links and stack trace are
below.

Thanks in advance,

-bryan

See
http://osdir.com/ml/java.sun.javaspaces/2005-09/msg00015.html

See
http://osdir.com/ml/java.sun.javaspaces/2005-09/msg00029.html

See http://forums.sun.com/thread.jspa?threadID=5284995
(perhaps post there?)

See
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4868432

See
http://osdir.com/ml/java.sun.jini/2004-04/msg00274.html

ERROR: 116515 pool-1-thread-54 SYSTAP-BBT.systap.com
<http://systap-bbt.systap.com/> 6a2d5f17-c36c-4799-ae4c-f61a2e07461a
com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFut
ure(BlockingBuffer.java:839): java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.io.IOException: I/O error reading from
mux connection: java.io.EOFException

java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.io.IOException: I/O error reading from
mux connection: java.io.EOFException

at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)

at
java.util.concurrent.FutureTask.get(FutureTask.java:83)

at
com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFut
ure(BlockingBuffer.java:827)

at
com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext
(BlockingBuffer.java:1173)

at
com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(
BlockingBuffer.java:915)

at
com.bigdata.service.proxy.ClientAsynchronousIterator.hasNext(ClientAsync
hronousIterator.java:514)

at
com.bigdata.relation.rule.eval.pipeline.DistributedJoinTask.nextChunk(Di
stributedJoinTask.java:389)

at
com.bigdata.relation.rule.eval.pipeline.JoinTask$BindingSetConsumerTask.
call(JoinTask.java:830)

at
com.bigdata.relation.rule.eval.pipeline.JoinTask.consumeSources(JoinTask
.java:666)

at
com.bigdata.relation.rule.eval.pipeline.JoinTask.call(JoinTask.java:449)

at
com.bigdata.relation.rule.eval.pipeline.JoinTask.call(JoinTask.java:1)

at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at
java.util.concurrent.FutureTask.run(FutureTask.java:138)

at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:885)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:907)

at java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.RuntimeException:
java.io.IOException: I/O error reading from mux connection:
java.io.EOFException

at
com.bigdata.io.SerializerUtil$1.deserialize(SerializerUtil.java:69)

at
com.bigdata.service.proxy.RemoteAsynchronousIteratorImpl$RemoteElementIm
pl.readExternal(RemoteAsynchronousIteratorImpl.java:201)

at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)

at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751
)

at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

at
com.sun.jini.jeri.internal.runtime.Util.unmarshalValue(Util.java:221)

at
net.jini.jeri.BasicInvocationHandler.unmarshalReturn(BasicInvocationHand
ler.java:1242)

at
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocat
ionHandler.java:825)

at
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationH
andler.java:659)

at
net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:
528)

at $Proxy11.nextElement(Unknown Source)

at
com.bigdata.service.proxy.ClientAsynchronousIterator$ReaderTask.call(Cli
entAsynchronousIterator.java:350)

at
com.bigdata.service.proxy.ClientAsynchronousIterator$ReaderTask.call(Cli
entAsynchronousIterator.java:1)

... 5 more

Caused by: java.io.IOException: I/O error reading from
mux connection: java.io.EOFException

at
com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read(Session.java:
943)

at
net.jini.jeri.connection.ConnectionManager$Outbound$Input.read(Connectio
nManager.java:521)

at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:22
66)

at
java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.ja
va:2674)

at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStre
am.java:2698)

at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1936)

at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753
)

at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753
)

at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at
java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)

at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)

at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753
)

at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at
java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)

at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)

at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

at
com.bigdata.io.SerializerUtil$1.deserialize(SerializerUtil.java:65)

... 18 more

Caused by: java.io.EOFException

at
com.sun.jini.jeri.internal.mux.SocketChannelConnectionIO.handleReadReady
(SocketChannelConnectionIO.java:384)

at
com.sun.jini.jeri.internal.mux.SocketChannelConnectionIO.access$200(Sock
etChannelConnectionIO.java:41)

at
com.sun.jini.jeri.internal.mux.SocketChannelConnectionIO$Handler.handleS
election(SocketChannelConnectionIO.java:464)

at
com.sun.jini.jeri.internal.runtime.SelectionManager$SelectLoop.run(Selec
tionManager.java:288)

at
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:136)

... 1 more

------------------------------------------------------------------------
-- Getting Started: http://www.jini.org/wiki/Category:Getting_Started
Community Web Site: http://jini.org <http://jini.org/> jini-users
Archive: http://archives.java.sun.com/archives/jini-users.html
Unsubscribing: email "signoff JINI-USERS" to ***@java.sun.com

--
Surely the Germans, who gave us "schadenfreude" and
"weltschmertz", have a term for "the uneasy semi-certainty that one's
assertion is soundly backed by a long-forgotten source." - unknown

A positive attitude may not solve all your problems, but it will
annoy enough people to make it worth the effort.
- Herm Albright

Slower minds keep right.
- unknown

--------------------------------------------------------------------------
Getting Started: http://www.jini.org/wiki/Category:Getting_Started
Community Web Site: http://jini.org
jini-users Archive: http://archives.java.sun.com/archives/jini-users.html
Unsubscribing: email "signoff JINI-USERS" to ***@java.sun.com

Bryan Thompson

2009-01-11 12:47:11 UTC

Permalink

Hello,

I was wondering if anyone has considered whether java spaces can be used
to build the same kinds of distributed lock, distributed queue, and
master election control structures that are the primary focus of
zookeeper[1,2,3]. Zookeeper gives a guarentee of the ordered of events
and provides some basic mechanisms for creating nodes within its "space"
that are linked to the life span of the client which created them and
which are assigned in a sequence that may be used to build up a variety
of queue like structures, including distributed locks and master
elections.

So, my question is, are there patterns for creating the same kinds of
control structures using java spaces?

Thanks,

-bryan

[1] http://hadoop.apache.org/zookeeper/docs/r3.0.1/
[2] http://hadoop.apache.org/zookeeper/docs/r3.0.1/recipes.html
[3]
http://hadoop.apache.org/zookeeper/docs/r3.0.1/zookeeperProgrammers.html

--------------------------------------------------------------------------
Getting Started: http://www.jini.org/wiki/Category:Getting_Started
Community Web Site: http://jini.org
jini-users Archive: http://archives.java.sun.com/archives/jini-users.html
Unsubscribing: email "signoff JINI-USERS" to ***@java.sun.com

Guy Korland

2009-01-11 22:18:46 UTC

Permalink

Hi Bryan,

GigaSpaces (http://www.gigaspaces.com/wiki) provides a distributed
implementation of JavaSpaces.

As so it provides means for all the facilities you mentioned including
distributed lock (based on javaspaces transaction's model), distributed
queue (based on javaspaces notify model + fifo extension) and primary
election mechanism (based on jini lookup service).

GigaSpaces provides many more facilities to allowing development and
deployment of distributed application almost as easy as development of
stand alone one.

[1]
http://www.gigaspaces.com/wiki/display/XAP66/Space+Locking+and+Blocking
http://www.gigaspaces.com/wiki/display/XAP66/Pessimistic+Locking
<http://www.gigaspaces.com/wiki/display/XAP66/Space+Locking+and+Blocking
%20http:/www.gigaspaces.com/wiki/display/XAP66/Pessimistic+Locking>

[2] http://www.gigaspaces.com/wiki/display/XAP66/Mule+Queue+Provider
http://www.gigaspaces.com/wiki/display/XAP66/Browsing+JMS+Queues

[3]
http://www.gigaspaces.com/wiki/display/XAP66/Terminology+-+Data+Grid+Top
ologies

Guy

________________________________

From: Bryan Thompson [mailto:***@systap.com]
Sent: Sunday, January 11, 2009 2:47 PM
To: JINI-***@JAVA.SUN.COM
Subject: java spaces and zookeeper

Hello,

I was wondering if anyone has considered whether java spaces can be used
to build the same kinds of distributed lock, distributed queue, and
master election control structures that are the primary focus of
zookeeper[1,2,3]. Zookeeper gives a guarentee of the ordered of events
and provides some basic mechanisms for creating nodes within its "space"
that are linked to the life span of the client which created them and
which are assigned in a sequence that may be used to build up a variety
of queue like structures, including distributed locks and master
elections.

So, my question is, are there patterns for creating the same kinds of
control structures using java spaces?

Thanks,

-bryan

[1] http://hadoop.apache.org/zookeeper/docs/r3.0.1/

[2] http://hadoop.apache.org/zookeeper/docs/r3.0.1/recipes.html

[3]
http://hadoop.apache.org/zookeeper/docs/r3.0.1/zookeeperProgrammers.html

------------------------------------------------------------------------
-- Getting Started: http://www.jini.org/wiki/Category:Getting_Started
Community Web Site: http://jini.org jini-users Archive:
http://archives.java.sun.com/archives/jini-users.html Unsubscribing:
email "signoff JINI-USERS" to ***@java.sun.com

--------------------------------------------------------------------------
Getting Started: http://www.jini.org/wiki/Category:Getting_Started
Community Web Site: http://jini.org
jini-users Archive: http://archives.java.sun.com/archives/jini-users.html
Unsubscribing: email "signoff JINI-USERS" to ***@java.sun.com