PyMongo doesn't iterate over collection

Question

I have strange behaviour in Python/PyMongo.

dbh = self.__connection__['test']
first = dbh['test_1']
second = dbh['test_2']

first_collection_records=first.find()  
second_collection_records=second.find()


index_f=first_collection_records.count() //20 
index_s=second_collection_records.count() //120

i=0
for f in first_collection_records:
    for s in second_collection_records:
         i=i+1
         print i

and it prints only 120 times (1..120) and not 20x120 times. Can someone tell me why it doesn't iterate through outer collection ? I printed results, it always takes only first of outer and iterates over inner collection. ( I posted counts which I get in code 20 and 120, I tried with xrange and fetch by index but nothing)

Answer 1

If you want to iterate second_collection_records for every first_collection_records though, you can use:

i=0
for f in first_collection_records:
    second_collection_records.rewind() #Reset second_collection_records's iterator
    for s in second_collection_records:
         i=i+1
         print i

.rewind() resets cursor to a fresh state, enabling you to once again retrieve data in second_collection_records.

---

Explanation:

second.find()

returns a Cursor object, which contains an iterator.

When an iterator of Cursor reach its end, it no longer return anything.

thus:

for f in first_collection_records: #20

actually does iterated 20 times, but since the inner:

for s in second_collection_records:

already iterated all objects returned, the second time it is called, second_collection_records no longer returns anything, thus the code inside(i=i+1, print...) is not executed.

You can try it like this:

i = 0
for f in first_collection_records:
    print "in f"
    for s in second_collection_records: 
        print "inside s"

You will get a result:

inside f
inside s
inside s
...
inside s
inside f  <- since s has nothing left to be iterated, 
             (second_collection_records actually raised StopIteration such in generator),
             code inside for s in second_collection_records: is no longer executed
inside f
inside f

---

In depth explanation:

This line:

for s in second_collection_records: 

the loop here actually works by next() method of Cursor object, as in: calling second_collection_records.next() until second_collection_records raised StopIteration exception (In Python generator and for loop, StopIteration is caught and code inside for loop would not be executed). So in the second til last loop of first_collection_records, second_collection_records.next() actually raised StopIteration for the inner loop, not executing the code.

We can easily observe this behavior by doing this:

for f in first_collection_records:
    print "inside f"
    second_collection_records.next()
    for s in second_collection_records:
        print "inside s"

And the result:

inside f
inside s
...
inside s
inside f
Traceback (most recent call last):
  ... , in next
    raise StopIteration
StopIteration