New EC2 instances not available to all users...
charliejones opened this issue · comments
EC2Box v0.30.03 on Ubuntu 16.04 LTS
We have multiple Full Access users. When we add new EC2 instances they don't always show up in their inventory. So far there doesn't appear to be any rhyme or reason as to who this effects.
Strange! It does only make the call to AWS for the server list when the page first loads. Also, it filters so only "running" servers show by default and it may take a minute to get to that state.
Just to add a little more detail to the report. It's currently effecting a single user. He cannot see new EC2 instances that are being added. This has also happened to me in the past and for the life of me I can't recall what fixed it (if anything). The user has logged out, changed networks (work and home), rebooted his machine and tried several browsers. I've also restarted the EC2box process. I've also had the user perform a view source to ensure that the EC2 instances weren't in the HTML, they are not.
I have a recurring exception in the logs...see attached.
ec2box-exception.txt
Based on the exception it looks like EC2Box failed authenticating to the instance. I would make sure the SSH key set for the new instance is working and registered correctly in EC2Box. It does look like there is an issue here... it should display an "Authentication Failed" back to the user, but it's getting a null pointer and erroring out instead. .. so I'll look into that!
Those same instances are showing up when I log into EC2Box. That's the puzzling part. Here's another exception that keeps occurring:
2017-06-14 18:04:28.539:WARN:oejwja.JsrEvents:qtp428746855-14: Unable to report throwable to websocket (no @onerror handler declared): com.ec2box.manage.socket.SecureShellWS
java.lang.IllegalStateException: Not valid for write: id=node013oqi15vfbivo1tb53vhbso95959 created=1497458616381 accessed=1497460617530 lastaccessed=1497460616792 maxInactiveMs=1800000 expiry=1497462417530
at org.eclipse.jetty.server.session.Session.checkValidForWrite(Session.java:611)
at org.eclipse.jetty.server.session.Session.setAttribute(Session.java:757)
at com.ec2box.common.util.AuthUtil.setTimeout(AuthUtil.java:220)
at com.ec2box.manage.socket.SecureShellWS.onMessage(SecureShellWS.java:112)
at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
at org.eclipse.jetty.websocket.jsr356.annotations.OnMessageTextCallable.call(OnMessageTextCallable.java:63)
at org.eclipse.jetty.websocket.jsr356.annotations.JsrEvents.callText(JsrEvents.java:197)
at org.eclipse.jetty.websocket.jsr356.endpoints.JsrAnnotatedEventDriver.onTextMessage(JsrAnnotatedEventDriver.java:386)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:66)
at org.eclipse.jetty.websocket.jsr356.endpoints.JsrAnnotatedEventDriver.onTextFrame(JsrAnnotatedEventDriver.java:368)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:162)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:375)
at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:182)
at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:142)
at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:85)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:256)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:679)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
I am having a hard time recreating error, but this looks like the session has been invalidated. Do you have users sharing accounts or anything like that?
I updated to the latest version of EC2Box in hopes that this might fix the problem. It didn't, but it did allow me to delete this particular user's account ;)
After recreating the account, the user was able to see the entire inventory of EC2 instances. So this appears to be somehow tied to the user's account.