David Cramer's Blog - Latest Comments in Large SQL Result Sets in Django

Re: Large SQL Result Sets in Django

Björn — Mon, 29 Mar 2010 03:40:51 -0000

Very useful recipe. There is a a "semi-bug" in IterableQuerySet that kicks in if you try to use it on an unordered resultset which i learnt the hard way. The order of select results where the order is not explicitly set using an order by clause is undefined.

To fix that, I changed the constructor to "self.queryset = queryset.order_by('id')" so that you are ensured that there are always some order in the results.

Re: Large SQL Result Sets in Django

David Cramer — Tue, 17 Feb 2009 11:40:59 -0000

It's nothing complicated, just a normal iterable wrapper.

for result in IterableQuerySet(qs): print result

Re: Large SQL Result Sets in Django

nitin — Tue, 17 Feb 2009 11:22:53 -0000

Hi,
I am facing similar problem for which i had blogged as well.
I think that you code may help me, but I am unable to use it.
It wud be helpful if you could give some sample usage also.

Re: Large SQL Result Sets in Django

pterk — Tue, 10 Feb 2009 04:38:53 -0000

Little typo in IterableQuerySet:

self.batch = batch

Re: Large SQL Result Sets in Django

megaman821 — Mon, 09 Feb 2009 20:22:17 -0000

Why not use the queryset's iterator method, e.g. model.objects.all().iterator()? It works fine for me with a table containing 1.2 million rows of which I write ~600 thousand to a csv file.

Re: Large SQL Result Sets in Django

Dilandau — Mon, 09 Feb 2009 19:49:47 -0000

Very usefull, Thanks