Erroneous "Consuming a byte in the end state" warnings.

[Tom Christie]

Hiya. Great stuff!

Planning to use your module for Starlette, see encode/starlette#102 (Its an async framework, so we need a streaming parser like this, and your API is the only one I could find that I can easily adapt into a nice Sans-IO

I've noticed that the module is erroneously issuing "Consuming a byte in the end state" warnings.

example.py

from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
from multipart.multipart import parse_options_header
import multipart
import uvicorn

app = Starlette()

@app.route('/', methods=['POST'])
async def homepage(request):
    content_type, params = parse_options_header(request.headers['Content-Type'])
    boundary = params.get(b'boundary')

    # No callbacks here, just exercise the parser
    parser = multipart.MultipartParser(boundary, callbacks={})

    # Feed the parser with data from the request.
    async for chunk in request.stream():
        parser.write(chunk)

    return PlainTextResponse('Uploaded OK\n')


if __name__ == '__main__':
    uvicorn.run(app, host='0.0.0.0', port=8000)

Running the app:

$ python example.py 
INFO: Started server process [10257]
INFO: Waiting for application startup.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
WARNING: Consuming a byte in the end state
WARNING: Consuming a byte in the end state
INFO: ('127.0.0.1', 56109) - "POST / HTTP/1.1" 200

curl:

$ curl -ik -F file=@/tmp/test.txt http://0.0.0.0:8000
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
server: uvicorn
date: Wed, 10 Oct 2018 15:40:42 GMT
content-length: 11
content-type: text/plain; charset=utf-8

Uploaded OK

I'll see if I can find some time later this week or next to try to resolve the issue.

[Andrew Dunham]

Hi there! That's pretty exciting news 😀 As you may have noticed, I'm not spending a ton of time on this project, so if you'd like, I'd be happy to add you as a collaborator on this repo so you can make the changes you need?

[Tom Christie]

@andrew-d Glad you're excited! Me too. 😎

Yup, very welcome to add me as a collaborator if that'd be helpful. Your call tho - either way is fine with me.

[Tom Christie]

Something worth having a look at just for context, is the implementation we use around python-multipart, here... https://github.com/encode/starlette/pull/102/files#diff-1fdcaee681d45ab76f328db5273aa60bR121

Ie. push data to write() and then deal with a list of resulting messages.

That's a really nice approach for Sans-IO as it lets us deal with integrating the I/O component, without having to structure the flow-control using callbacks. (The h11, h2, and wsproto libraries all take this "stream of events" approach too)

• https://github.com/python-hyper/h11
• https://github.com/python-hyper/hyper-h2
• https://github.com/python-hyper/wsproto

[Andrew Dunham]

I did this earlier today, but forgot to comment: made you a maintainer of the repo, and on PyPI. I don't use this project any more, so feel free to take it from here 😄

[Tom Christie]

Fab, thanks for your work on this - really great stuff.

Two options I'd like to consider, but need to know how you feel about either of them.

1. Merging your work directly into Starlette. MultiPart parsing is such an integral part that I'd seriously consider bringing it in to the core package. (Plus it'd keep my maintainership life simple.)
2. Moving the repo to the "encode" organisation.

[Andrew Dunham]

Merging your work directly into Starlette. MultiPart parsing is such an integral part that I'd seriously consider bringing it in to the core package. (Plus it'd keep my maintainership life simple.)

I think I'd prefer to keep this package distinct, for now - there's probably people that want to use this outside the context of a web framework (or, much as I hate to say it, are stuck on older versions of Python), and merging into Starlette would probably make it harder for them to use this package. A specific example I'm thinking of would be to parse emails, for example.

Moving the repo to the "encode" organisation.

No particular objection to this! 👍