Updating django-socketio to use the latest socketio client (0.8.5)
tabouassaleh opened this issue · comments
I am working on a web app using django and socket.io, and have been trying to use django-socketio to connect the two. I noticed that the the socketio client packaged with the app was using xhr-multipart and xhr-polling on the latest FireFox (10.0.2) and Chrome (17.0.963.56), respectively. I tried replacing the client library with the latest version (0.8.5) but now I get an error on the server side:
Traceback (most recent call last):
File "/home/taa/env/lib/python2.6/site-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/home/taa/env/lib/python2.6/site-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/home/taa/env/lib/python2.6/site-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/home/taa/env/lib/python2.6/site-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/home/taa/env/lib/python2.6/site-packages/gevent/socket.py", line 504, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/home/taa/env/lib/python2.6/site-packages/gevent/socket.py", line 490, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<SocketIOServer fileno=3 address=0.0.0.0:8182>: Failed to handle request:
request = GET /socket.io/1/?t=1330535867416 HTTP/1.1 from ('216.16.232.86', 1154)
application = <django.contrib.staticfiles.handlers.StaticFilesHandler object at 0x2d0b950>
x.x.x.x - - [2012-02-29 11:17:52] "GET /socket.io/1/?t=1330535867416 HTTP/1.1" 500 0 0.384851
Any idea what the issue may be? And any plans to upgrade the socketio client library packaged with the app or make the app compatible or socketio-client v0.8.5?
I don't know much about the WebSocket protocol, but I'm willing to help however I can, such as testing different browsers and what not.
Thanks for adding the issue.
The bundled socket.io JavaScript is tied to the server implementation in the public release of gevent-socketio: https://bitbucket.org/Jeffrey/gevent-socketio
It looks like the version in that repo is much more up to date than the public release on pypi - so you could try upgrading it to the version on bitbucket and see how you go.
I replaced the socketio client library with the one from gevent-socketio repository. Django now chocks in the client_start function. Looking into it further, it appears the socket coming to client_start has the session field set to None, making it unable to use the session_id as a key to populate the CLIENTS dict. Could this be related to gevent-socketio issu #15 Session ID is not honored for WebSocket and Flash Socket transports?
@tgcondor I wasn't able to get django-socketio working with HEAD @ gevent-socketio last time I tried. If you could figure it out that'd be great!
@philipn gevent-socketio tip has changed its API quiet a bit. I'm working now on trying to propagate the changes to django-socketio.
I got the communication working with the latest gevent-socketio, which includes socketio client v0.8.2, which is compatible with latest modern browsers that support websockets (tested in FF and Chrome). The code is not release ready, but for anyone trying to get this working in the mean time, I made a fork with my changes at https://github.com/tgcondor/django-socketio. Note that with the new code, you don't need to patch the socket.io JS anymore. I'll update the example accordingly in due time.
Tony that's awesome! I'll check it out soon and hopefully at some point I can merge it all back in.
Just an update on this - I've pulled the latest work from https://github.com/tgcondor/django-socketio and I'm currently in the process of getting everything working for a new release for the latest gevent-socketio and socket.io.js itself.
Another update. Still a lot more research needed.
- Latest socket.io now implements arbitrary "on" events, which seems like it would make the current server-side channels implementation redundant.
- Seems like it's no longer possible to send non-string data from JS to a socket, as data arrives cast to an unusable string (eg [object Object]) by the time it hits the server-side socket. Not sure if this is a gevent-socketio issue.
Hey whats the current status of this project in terms of support for socket.io v8 or v9?
No progress yet that I'm aware of.
Ahh cool thanks. I thought that the the updated build status kinda meant it was, but the docs were updated it.
Hey, is django-socketio still alive? I tried the testapp from February of last year and nothing seems to be working -- it uses xhr-multipart for current versions of both Firefox and Chrome, which then throws a long number of "[Errno 104] Connection reset by peer" on the server. That all seems related to this issue here, which from what I gather is no longer being worked on, right?
I noticed that this django app seems to have implemented socket.io 0.96, but that it otherwise doesn't seem as complete as django-socketio: https://github.com/fish2000/django-signalqueue/tree/master/signalqueue
Also IE10 will have websocket, so given that the socketio stuff only works with a considerable amount of hacking, would it be more intelligent to just go directly for websockets? What do the developers see.
@johanneswilm There was a bunch of recent work to get the underlying library, gevent-socketio, working with the latest socketio and generally refactored. These changes haven't yet been pulled into django-socketio.
@philipn: Are you referring to the work that was mentioned above, three months ago? I wondered if that was before Firefox and Chrome came out with versions that support websockets. That's why I'm asking.
@johanneswilm RE: socketio versus straight websockets, check out the socketio faq:
Why not just call it WebSocket if the actual WebSocket is not present and mimick its API?
Socket.IO does more than WebSocket, even if WebSocket is selected as the transport and the user is browsing your website with an ultra modern browser. Certain features like heartbeats, timeouts and disconnection support are vital to realtime applications but are not provided by the WebSocket API out of the box.
This is akin to jQuery's decision of creating a feature-rich and simple $.ajax API as opposed to normalizing XMLHttpRequest.
ok ok. yes I had actually read that before. I just spent the last two days or so trying to figure out what package exactly I should choose, and lots of the involved things seem to be (half) dead or to have moved their repository without announcing anything anywhere, so I wondered if the right way to go was to reduce complexity by cutting out at least one of the layers.
Just to clarify a few points.
This project won't ever deal with any specific browser implementations of WebSockets. It's purely designed to get you up and running using both Django and socket.io - socket.io being the portion that deals with all the browser intricacies.
The first paragraph on the homepage (via the README) describes the current state of django-socketio, and points to this thread as the discussion point for what needs to happen, so let's try and keep it on topic.
Just to re-iterate on the current state of the project: there hasn't been any work done on django-socketio for a few months. During that time socket.io has changed significantly both API-wise and to keep up with the newer browser WebSocket implementations. The underlying back-end libs django-socketio makes use of (gevent-socketio etc) have recently been brought up to date with socket.io as well.
Now all that needs to happen is to bring django-socketio up to date with these. The initial work has already been done (referenced earlier in this thread). More work needs to be done to look at django-socketio's events and channels features - whether they still make sense with the latest socket.io, and what new implementations of events and channels might look like if they still do make sense for the project.
Personally I've had every spare moment available to me spent on other open source projects over the last few months that unfortunately have a much larger audience and demand that django-socketio, so I simply haven't had any time for it. That aside, I've every intention of bringing it up to date once I get some time available. If anyone else decides to roll their sleeves up and get working on that before I do, then all the better. But it will happen either way. If this needs to happen earlier for you, then thanks to the nature of open source, you or anyone else has the absolute power to make it happen.
Again let's try and keep this thread on topic since the homepage now points to it. If the state of this project changes in any way, it'll be mentioned here and/or on the homepage README - no need to ask about it.
sorry, I never meant to criticize. I merely wondered whether this effort was still alive. This is not primarily due to what you have done, but because in this corner of free software there just seems to be a lot of things that just stop working.
For example, the tictactoe django example mentioned on the gevent-socketit does not currently work, the meinheld server project seems to not have been developed beyond last year, the gevent-socketio based chat application doesn't work with current browsers, and when googling for "gevent socketio" one is taken to an outdated repository, django-websocket doesn't seem to have any updates since 2010. When I then noticed that django-socketio has this issue (which likely means that it's not working in recent browsers either), I just wanted to make sure that things are still alive before looking further at the source.
No worries - no criticism taken, just wanted to be really clear about where things are at, what needs to happen, and how they might happen.
You're right it's a fast moving (and breaking) space.
A note to readers who are looking for an alternative solution in the mean time (while django-socketio is being brought up to date).
I've had good success using django behind Tornado server with TornadIO2 socket.io library. It doesn't have any deep integration with django, but as @stephenmcd pointed out, they may not be needed given the new structure of the socket.io API which supports arbitrary events.
I have written a few helper functions and classes for the using TornadIO2, but I haven't had time to open source them yet. However, I'd be happy to answer any questions on this topic. Please message me privately to avoid spamming, or publicly at the place-holder repository django-tornadio2.
Some work looks like it's been done here:
https://github.com/wuzuf/django-socketio
wuzuf essentially throws out the current decorator-based stuff and uses django-socketio as a thin wrapper over the gevent-socketio Namespace stuff.
While this works, I think we'd be well served by having something along the lines of the current decorator-based approach -- something that can be more seamlessly integrated into your existing Django views rather than something you build out separately as a Namepace. Should be easy to add a little decorator in here to mimic the old behavior.
The question is: should someone using django-socketio have to grok the whole API model of gevent-socketio? I think it's overly complicated. The existing django-socketio API is really easy to understand :)
Even a signal-handler-type approach -- registering functions to fire on events -- seems better. Having to construct a Namespace object just kinda rubs me the wrong way..
Awesome thanks @philipn I'll definitely check that out soon.
As an update, I've been working on a personal side-project that uses the latest gevent-socketio. It's an IRC web client that I have a need for, but one of the main motivators is to spend some quality time with the latest gevent-socketio which has changed quite a lot since I last worked on django-socketio, and get a good understanding of what we might do for the next version of django-socketio.
Here's the IRC project I mentioned too: https://github.com/stephenmcd/gnotty
@philipn that's the general approach I had in my head as well - trying to stay as close to the original django-socketio implementation as possible, so long as it makes sense.
Did anyone succeed in integrating with newer version of gevent-socketio?
I have run the example chat app today and it almost worked... Except that I need to apply two patches to socketio (I mean (gevent-socketio) in order to activate Websockets. Those fixes are already present in master.
@stephenmcd Any updates on this? We are preparing a new release and removing the 'beta' tag on gevent-socketio and would love for django-socketio to be supported as well. I could help getting it going if you want
Not yet - there will be commits here as soon as it happens :-)
I have been using the latest gevent-socketio in other projects to get a better feel for how the next version of django-socketio will look which has been really interesting. Check this post I did on it: http://blog.jupo.org/2012/11/18/mobile-irc-with-bonus-robots/
Is this movement still active or should we consider writting our own patch for 0.9+ ?
I was recently contacted by someone who asked the same question and expressed interest in getting the work done. I'm not sure where they're at with it, but if you'd like to collaborate on the upgrade with myself and anyone else interested, then by all means let's make it happen :-)
I am the one that contacted Stephen :)
I'm interested in updating django-socketio to the new gevent and started looking at it over the past weekend but didn't actually get anything done. I'll try to spend some more time with it in the next few days although GDC madness is upon us so it might have to wait until April.
I would love to coordinate!
My initial thoughts: expose the new functionality introduced in later socketio versions (custom events, rooms) and get rid of the obsolete channels.
I think its important to retain the django-socketio functionality of being able to register to events without implementing a class. The event decorator should accept the event type as well as an optional namespace -- including for 'connect' events (in the current version channel cannot be specified for the connect event)
Re: namespaces. It should definitely be possible to register to events without specifying a namespace, but assuming you DO want a namespace class I feel like its cleaner using a decorator to specify the event type as opposed to using the method name the way gevent does it. (def on_event_type)
Would this mean to re-write the current codebase? Because, I have nothing against the current events.py approach but looking at Tornado it seems more conventional to let the code live in a views.py as controller. Due to the behaviour of websockets the problem is the connect, receive, send/emit/broadcast, close cycle. Which are 4 operations which should be independant. I would prefer a class style approach where each room lives within a namespace class implementing those 4 operations plus additional logic.
Another very important point is being flexible, because WebSockets are a new standard rapidly evolving we see a lot of movement in SocketIO and this issue-thread is a result of being left behind by those changes.
I will try and rewrite as little as possible :) If you prefer the class based approach you can probably stick to the base gevent-socketio (https://github.com/abourget/gevent-socketio/blob/master/examples/django_chat/chat/sockets.py). I really hope to mess around with this sometime soon
Hey,
I noticed there were some updates and a new Pypi release.
I forgot about the details behind all this, so just to clarify: If one puts django-socketio into a project, will it use Websockets in Firefox and chrome, or will it use one of the fallbacks? If it uses fallbacks, are there other projects you suggest to use as alternatives?
@shlomozippel I agree that the current decorator approach should be kept nice and simple.
Can anyone give insight to the current status of this project?
@jjkester For now, I would recommend using the the gevent-socketio package directly. The django-socketio project needs someone to work to integrate the new gevent-socketio work. gevent-socketio is much simpler than it was when this project was first created, so you shouldn't have that much difficulty.
There has recently been a new release of django-socketio (0.3.7). It was my udnerstandign that it's not really usable wiht current webbrowsers. What browseres exactly are working and which ones aren't (or using a fallback)?
I've managed to get working django-socketio with newest socketio-client (0.9.16) and gevent-socketio 0.3.5-rc2
I need someone to help me write tests (and not unit tests because those are useless. We need some selenium tests that uses actually socketio and browser...) and documentation. @stephenmcd also I redesigned api for using it so I don't know if it could be merged to django-socketio...
https://github.com/Solution4Future/django-socketio/
You can see how it works here:
https://github.com/galuszkak/djangodash/
This implementation could have bugs, but it really works.
@galuszkak I will fork your project and set up a basic pytest-bdd + pytest-bdd-splinter base, so you can test with selenium. You should probably create a travis.ci project so we can use that to test. Probably will have some spare time to do that this evening (its 12:00pm here).
If you have any questions, remarks or want to discuss something beforehand, please contact me.
Awesome stuff @galuszkak - when you're ready can you create a pull request?
@stephenmcd I think in next 2 weeks. Thanks to @maikelwever I've got pretty great framework for testing so I will write test and documentation and example_project.
Cheers
Kamil
@galuszkak We will be happy to have an example project. Implementation is still confusing to me... :/
@khamaileon Yes I will be doing it next week (soory this week I'm going on PyCon PL so I will be unavailable).
Implementation maybe isn't so perfect but you create events.py in your application and write somethings like this:
from django_socketio.events import Namespace, BroadcastMixin
@Namespace('/users')
class UserNamespace(BaseNamespace, BroadcastMixin):
def on_EVENT(self, obj)
#do_whatever_you_want
# methods self.emit('event_name', data)
# methods self.broadcast_event_not_me('event_name', data)
# methods self.broadcast('event_name', data)
Client side is just:
var socket = io.connect('/users/')
socket.emit('event_name', data)
socket.on('event_name', function(data){
console.log("I received " + data)
})
So I think it's very easy. Do you have any more question? I will be happy for any feedback.
Thank you for the example @galuszkak. It shows me the way :)
I've just deleted the last few messages in this thread since they were entirely unrelated to the topic of upgrading to latest socketio.
By all means if you have any issues, open a ticket - but please don't post them in this thread, thanks.
@galuszkak what happened to that PR?
@galuszkak I'd also be interested in the PR. :)
For anyone asking what's happening with this thread, if it's still open and nothing has been mentioned, that means the work hasn't been done!
I just deleted a handful of "is anything happening here?" messages, since they're just noise.
For information.
I have pending work, the gevent-socketio package changed a lot and i think many features implemented in this package have been moved inside gevent-socketio.
I have enough work done but i mostly rely on source code and blogs to fix it. I may be able to finish it my my work so far is draft. If anyone feels like picking up from what i've left on my branch so far feel free, It might take me some time because i have something else to finish too.
https://github.com/papaloizouc/django-socketio/tree/support_for_0_3_6
http://www.pixeldonor.com/2014/jan/10/django-gevent-and-socketio/
Thanks @papaloizouc that looks really useful!