epineda / django-querysetsequence

Chain multiple (disparate) QuerySets in Django

Django QuerySetSequence

The QuerySetSequence wrapper helps to deal with disparate QuerySet classes, while treating them as a single QuerySet.

Supported Features

Listed below are features of Django's QuerySets_ that QuerySetSequence implements. The behavior should match that of QuerySet, but applied across multiple QuerySets:

• Methods that take a list of fields (e.g. filter(), exclude(), get(), order_by()) must use fields that are common across all sub-QuerySets.
• Relationships across related models work (e.g. 'foo__bar', 'foo', or 'foo_id'). syntax).
• The sub-QuerySets are evaluated as late as possible (e.g. during iteration, slicing, pickling, repr()/len()/list()/bool() calls).
• Public QuerySet API methods that are untested/unimplemented raise NotImplementedError.

QuerySet API implemented by QuerySetSequence

Methods that return new QuerySets

Method	Implemented?	Notes
filter()_	✓	See1 for information on the QuerySet lookup: '#'.
exclude()_	✓	See2 for information on the QuerySet lookup: '#'.
annotate()_	✓
order_by()_	✓	Does not support random ordering (e.g. order_by('?')). See3 for information on the QuerySet lookup: '#'.
reverse()_	✓
distinct()_	✗
values()_	✗
values_list()_	✗
dates()_	✗
datetimes()_	✗
none()_	✓
all()_	✓
union()_	✗
intersection()_	✗
difference()_	✗
select_related()_	✓
prefetch_related()_	✓
extra()_	✓
defer()_	✓
only()_	✓
using()_	✓
select_for_update()_	✗
raw()_	✗

Operators that return new QuerySets

Operator	Implemented?	Notes
AND (&)_	✓	A QuerySetSequence can be combined with a QuerySet. The QuerySets in the QuerySetSequence are filtered to ones matching the same Model. Each of those is ANDed with the other QuerySet.
_	✓	A QuerySetSequence can be combined with a QuerySet or QuerySetSequence. When combining with a QuerySet, it is added to the QuerySetSequence. Combiningg with another QuerySetSequence adds together the two underlying sets of QuerySets.

Methods that do not return QuerySets

Method	Implemented?	Notes
get()_	✓	See4 for information on the QuerySet lookup: '#'.
create()_	✗	Cannot be implemented in QuerySetSequence.
get_or_create()_	✗	Cannot be implemented in QuerySetSequence.
update_or_create()_	✗	Cannot be implemented in QuerySetSequence.
bulk_create()_	✗	Cannot be implemented in QuerySetSequence.
count()_	✓
in_bulk()_	✗	Cannot be implemented in QuerySetSequence.
iterator()_	✓
latest()_	✓	If no fields are given, get_latest_by on each model is required to be identical.
earliest()_	✓	See the docuemntation for latest().
first()_	✓	If no ordering is set this is essentially the same as calling first() on the first QuerySet, if there is an ordering, the result of first() for each QuerySet is compared and the "first" value is returned.
last()_	✓	See the documentation for first().
aggregate()_	✗
exists()_	✓
update()_	✓
delete()_	✓
as_manager()_	✓
explain()_	✓	Only available on Django >= 2.1.

Additional methods specific to QuerySetSequence

Method	Notes
get_querysets()	Returns the list of QuerySet objects that comprise the sequence. Note, if any methods have been called which modify the QuerySetSequence, the QuerySet objects returned by this method will be similarly modified. The order of the QuerySet objects within the list is not guaranteed.

Requirements

• Python (2.7, 3.5, 3.6, 3.7)
• Django (1.11, 2.1, 2.2)
• (Optionally) Django REST Framework (3.6.3+, 3.7, 3.8, 3.9)

Installation

Install the package using pip.

pip install --upgrade django-querysetsequence

Usage

# Import QuerySetSequence
from queryset_sequence import QuerySetSequence

# Create QuerySets you want to chain.
from .models import SomeModel, OtherModel

# Chain them together.
query = QuerySetSequence(SomeModel.objects.all(), OtherModel.objects.all())

# Use query as if it were a QuerySet! E.g. in a ListView.

Example

class Author(models.Model):
    name = models.CharField(max_length=50)

    class Meta:
        ordering = ['name']

    def __str__(self):
        return self.name


class Article(models.Model):
    title = models.CharField(max_length=100)
    author = models.ForeignKey(Author)

    def __str__(self):
        return "%s by %s" % (self.title, self.author)


class Book(models.Model):
    title = models.CharField(max_length=50)
    author = models.ForeignKey(Author)
    release = models.DateField(auto_now_add=True)

    def __str__(self):
        return "%s by %s" % (self.title, self.author)

# Create some data.
alice = Author.objects.create(name='Alice')
article = Article.objects.create(title='Dancing with Django', author=alice)

bob = Author.objects.create(name='Bob')
article = Article.objects.create(title='Django-isms', author=bob)
article = Book.objects.create(title='Biography', author=bob)

# Create some QuerySets.
books = Book.objects.all()
articles = Article.objects.all()

# Combine them into a single iterable.
published_works = QuerySetSequence(books, articles)

# Find Bob's titles.
bob_works = published_works.filter(author=bob)
# Still an iterable.
print([w.title for w in bob_works])  # prints: ['Biography', 'Django-isms']

# Alphabetize the QuerySet.
published_works = published_works.order_by('title')
print([w.title for w in published_works])  # prints ['Biography', 'Dancing with Django', 'Django-isms']

Django REST Framework integration

django-querysetsequence comes with a custom CursorPagination class that helps integration with Django REST Framework. It is optimized to iterate over a QuerySetSequence first by QuerySet and then by the normal ordering configuration. This uses the optimized code-path for iteration that avoids interleaving the individual QuerySets. For example:

from queryset_sequence.pagination import SequenceCursorPagination

class PublicationPagination(SequenceCursorPagination):
    ordering = ['author', 'title']

class PublicationViewSet(viewsets.ModelViewSet):
    pagination_class = PublicationPagination

    def get_queryset(self):
        # This will return all Books first, then all Articles. Each of those
        # is individually ordered by ``author``, then ``title``.
        return QuerySetSequence(Book.objects.all(), Article.objects.all())

Attribution

This is based on a few DjangoSnippets that had been going around:

• Originally from https://www.djangosnippets.org/snippets/1103/
• Modified version from https://djangosnippets.org/snippets/1253/
• Upgraded version from https://djangosnippets.org/snippets/1933/
• Updated version from django-ko-demo from The Atlantic

Contribute

• Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
• Fork the repository on GitHub to start making your changes.
• Write a test which shows that the bug was fixed or that the feature works as expected.
• Send a pull request and bug the maintainer until it gets merged and published.

---

1. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩

2. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩

3. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩

4. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩