Scrolling search results

[tommazzo]

Hi,

Is there any way to scroll search results (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html or http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#scan)?

I was able to get a scroll_id by using the following options when performing a search:

var options = {
    _index: "myindex",
    search_type: "scan",
    scroll: "10m"
};

This gives me a scroll_id in the data returned by the search, but I have not found any way to use it to actually retrieve the results. How can I do this?

Thank you for your help in advance!

Best regards,
Thomas

[Joshua Thomas]

Thanks for this issue report! There is now a method named scroll in the core module... to use it, check out this snippet:

var
  elasticsearch = require('es'),
  config = {
    _index : 'kittehs'
  },
  es = elasticsearch(config);

// first search, specifying scan search_type
es.search({
    search_type : 'scan',
    scroll : '10m'
  }, {
    query : {
      match_all : {}
    }
  }, function (err, data) {
    // next, perform the scroll with the _scroll_id value returned
    es.scroll({ scroll : '10m' }, data['_scroll_id'], function (err, data) {
      // now we have the results of the scroll...
    });
  });

[tommazzo]

That's exactly what I was looking for. Thanks!

[Sean Fern]

@brozeph I was just playing with scroll in putting together https://github.com/gsf/esindexdump and I was thinking it would be cool to create a little wrapper method on es that returns a readable stream, i.e.:

scrollResult = es.scroll();
scrollResult.on('data', function(chunk) {
  // each chunk is a scroll result
})
scrollResult.on('end', function () {
  // all done
})

Anyway, just an idea to chew on.