Query with limits returns whole collection

[Kamil Ciemniewski]

I've got about 100000 records in db.
I'd like to select only 10 of them.

so I write:

q, _ := mongo.Marshal(map[string]string{})
ret, error := coll.Query(q, 0, 10)

count := 200000 // because query gives me more than all of them....
keywords := make([]Keyword, count)

doc, _ := ret.GetNext()
for ; ret.HasMore(); doc, _ = ret.GetNext() {
fmt.Println("", iter)
mongo.Unmarshal(doc.Bytes(), &keywords[iter])
iter++
}

... and then I see about 100000 lines

0
1
2
...

101032
101033

So Query behaves ( maybe only on my machine ) just
like FindAll, no matter what limit and skip I give it..

I don't know if I'm doing something wrong or is there really
some problem with this method.

[D S]

Hi kamilc

I don't know if I'm doing something wrong or is there really some problem with this method.

the problem really is

[D S]

The problem was in HasMore() sub
When the cursor is coming to an end, this function selects all that was in the collection.
Perhaps this is part of a larger plan, but currently commenting GetMore() in HasMore() solves the problem of limits

 --- a/mongo/cursor.go
 +++ b/mongo/cursor.go
 @@ -27,6 +27,7 @@ func (self *Cursor) GetNext() (BSON, os.Error) {
 }
 
 func (self *Cursor) HasMore() bool {
+       /*
       if self.pos < self.docs.Len() {
               return true

        }
@@ -35,6 +36,7 @@ func (self *Cursor) HasMore() bool {
        if err != nil {
                return false
        }
+       */

        return self.pos < self.docs.Len()
 }

[Kamil Ciemniewski]

deletus: That did the trick
Thank you

[hokapoka]

deletus, that indeed does fix the the Query & allows limit / skip to work, but this causes another issue.

When using FindAll(q), it limits the results to 101.

I would imagine there's a simple fix for this, currently it's not an issue for me as I'll be using limit & skip, but it appears that's why the HasMore() sub contains the if statments.

[D S]

Hi hokapoka

I would imagine there's a simple fix for this

here you are

diff --git a/mongo/collection.go b/mongo/collection.go
index 88acb43..f619f36 100644
--- a/mongo/collection.go
+++ b/mongo/collection.go
@@ -117,7 +117,7 @@ func (self *Collection) Query(query BSON, skip, limit int32) (*Cursor, os.Error)
                return nil, os.NewError("wrong responseTo code")
        }
 
-       return &Cursor{self, reply.cursorID, 0, reply.documents}, nil
+       return &Cursor{self, reply.cursorID, 0, limit, reply.documents}, nil
 }
 
 func (self *Collection) FindAll(query BSON) (*Cursor, os.Error) {
diff --git a/mongo/cursor.go b/mongo/cursor.go
index d10790d..c182eca 100644
--- a/mongo/cursor.go
+++ b/mongo/cursor.go
@@ -14,6 +14,7 @@ type Cursor struct {
        collection *Collection
        id         int64
        pos        int
+       limit      int32
        docs       *vector.Vector
 }
 
@@ -38,7 +39,18 @@ func (self *Cursor) HasMore() bool {
        }
        */
 
-       return self.pos < self.docs.Len()
+       if self.pos < self.docs.Len() {
+               return true
+       }
+
+       if self.limit == 0 {
+               if err := self.GetMore(); err != nil {
+                       return false
+               }
+               return true
+       }
+
+       return false
 }
 
 
@@ -68,6 +80,10 @@ func (self *Cursor) GetMore() os.Error {
                return os.NewError("wrong responseTo code")
        }
 
+       if reply.documents.Len() == 0 {
+               return os.NewError("no documents found")
+       }
+
        self.pos = 0
        self.docs = reply.documents
 

and test

dmitrys@laptop:~/tmp$ mongo
MongoDB shell version: 1.6.6-pre-
connecting to: test
> for ( i = 0; i < 210; i++ ) db.test.insert({id:i})
> db.test.count()
210
> exit
bye
dmitrys@laptop:~/tmp$ 
dmitrys@laptop:~/tmp$ cat main.go 
package main

import (
                "fmt"
                "github.com/mikejs/gomongo/mongo"
)

func main() {
        dbc, _ := mongo.Connect("localhost")
        collection := dbc.GetDB("test").GetCollection("test")
        query, _ := mongo.Marshal(map[string]string{})
        cursor, _ := collection.FindAll(query)

        var count int

        for {
                if !cursor.HasMore() {
                        break
                }

                cursor.GetNext()
                count ++
        }

        fmt.Printf("FindAll count: %d\n",count)
        cursor, _ = collection.Query(query,0,152)
        count = 0

        for {
                if !cursor.HasMore() {
                        break
                }

                cursor.GetNext()
                count ++
        }

        fmt.Printf("Query limit count: %d\n",count)
}
dmitrys@laptop:~/tmp$ 
dmitrys@laptop:~/tmp$ 8g main.go && 8l main.8 && ./8.out 
FindAll count: 210
Query limit count: 152
dmitrys@laptop:~/tmp$ 

[D S]

and why was returned 101 rows

If the client driver offers 'limit' functionality (like the SQL LIMIT keyword), then it is up to the client driver to ensure that no more than the specified number of document are returned to the calling application. If numberToReturn is 0, the db will used the default return size.

http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol

[Kamil Ciemniewski]

So old that I'm closing it now