Support paging

[Juarez Bochi]

See section 7 of the specification

[Marco Palladino]

+1

[Thibault Charbonnier]

I started working on that here.

I'd like you to tell me what you expect from the behaviour of pagination @jbochi ? Currently, I return the paging_state as part of the response metadata, and it is up to the user to add it as an option on the next request he'll be executing. It is just a wip/draft.

My thinking is that now the library should offer an execute method with a callback that will continue the query until the has_more_pages flag is unset. And on each page of results, give (page, rows) to the user.

[Juarez Bochi]

Great job, @thibaultcha !

I have not read your implementation in detail yet, but it looks good to me.

I just wonder if it's possible to handle the paging transparently, just like the Python driver does. We could also try to implement the callback syntax available for the nodejs driver. I like the paging_state idea, but it would be even better if we could keep the syntax as close as possible to the official drivers.

I also like the auto paging feature, but I believe we should use a limit of 5000 rows by default, like the other drivers do.

Thanks for such a significant effort to improve this driver :)

[Juarez Bochi]

Ooops. I just read how the node.js driver's callback syntax works again and it would not make sense for us. It must use a callback since the query is asynchronous.

I think the paging_state is fine as it is, but it would be nice to support automatic paging as well.

Anyway, please feel free to send a pull request. We can try to implement that later.

Thanks again for your great work.

[Thibault Charbonnier]

Ok, my fork now fully supports manual paging, but I think adding automatic paging wouldn't be too much work. There are just many ways of doing it, what I'm thinking, is that if you want to mimic the python driver:

query = "SELECT * FROM users"  # users contains 100 rows
statement = SimpleStatement(query, fetch_size=10)
for user_row in session.execute(statement): # execute() here implements the iterator interface
    process_user(user_row)

What would be the closest in Lua is returning a custom iterator if auto_paging is set to do:

local query = "SELECT * FROM users"  -- users contains 100 rows
for page, user_row in session:execute(query, nil, { auto_paging = true }) do
    -- page would be the current page number?
    process_user(user_row)
end

What do you think of that?

[Juarez Bochi]

Awesome! I think we can close this issue and open a new one if we decide to change the API later.