Pagination may not work with joined models

[Thomas P]

Hey,

We've stumbled upon an issue with Scrivener and joined models counting, which prevents correct pagination.

Here is how it works:

1. Have a model with many relationships
2. Try to load it paginated with many joins:
   (In the following, Article is our model with many relationships)

result = Article |> Article.with_tags_and_assets |> Repo.paginate(%{"page_size"=> 3})
pry(26)> result.entries |> Enum.count                                                                                      
1

# BUT
result = Article |> Article.with_tags_and_assets |> Repo.paginate(%{"page_size"=> 300})
pry(28)> result.entries |> Enum.count                                                                                        
3

# with_tags_and_assets is:

def with_tags_and_assets(q) do
  q
  |> join(:left, [a], t in assoc(a, :tags))
  |> join(:left, [a], i in assoc(a, :images))
  |> join(:left, [a], d in assoc(a, :documents))
  |> join(:left, [a], v in assoc(a, :videos))
  |> preload([a, t, i, d, v], [tags: t, images: i, documents: d, videos: v])
end

# And the relationships definitions in Article

has_many :article_tags, MyApp.Models.ArticleTag
has_many :tags, through: [:article_tags, :tag]

has_many :article_assets, MyApp.Models.ArticleAsset
has_many :images, through: [:article_assets, :image]
has_many :documents, through: [:article_assets, :document]
has_many :videos, through: [:article_assets, :video]

This is caused by a mismatch between the number of rows returned by the DB, and the actual number of entities that you get, since with joins, 1 entity >= 1 row.

Especially:

• The counting method here: https://github.com/drewolson/scrivener/blob/master/lib/scrivener.ex#L152 can't work.
• The offset here is not computed correctly: https://github.com/drewolson/scrivener/blob/master/lib/scrivener.ex#L144

There is a SQL solution, presented here: http://stackoverflow.com/questions/15897055/proper-pagination-in-a-join-select/15897271#15897271

However, it requires sub-queries, which Ecto < 2.0 doesn't support. So we've come up with a 2 queries solution, which looks like:

defp paginate_with_joins(q, config) do
    # First, find out the ids of the things we want instead
    stripped_query = q
                     |> exclude(:select)
                     |> exclude(:preload)

    offset = config.page_size * (config.page_number-1)
    page_size = config.page_size
    ids = stripped_query
          |> exclude(:group_by)
          |> select([x], {x.id})
          |> group_by([x], x.id)
          |> offset(^offset)
          |> limit(^page_size)
          |> Repo.all

    # Then find the actual results using the ids as selectors instead of
    # offset & limit
    as_uuids = Enum.map(ids, fn {str_id} -> str_id end)
    entries = q |> where([x], x.id in ^as_uuids) |> Repo.all

    # We also need the total count, so we find it out
    count = stripped_query
            |> exclude(:order_by)
            |> select([x], count(x.id, :distinct))
            |> Repo.one!

    # Then we create the Scrivener-compatible page
    %Scrivener.Page{
      page_size: config.page_size,
      page_number: config.page_number,
      entries: entries,
      total_entries: count,
      total_pages: ceiling(count, config.page_size)
    }
  end

Would you be interested in including some form of this in Scrivener? We could submit a PR.

Thanks,
Thomas

[Drew Olson]

Hi @TisButMe -

Thanks so much for opening this issue. I would definitely be interesting in including this in Scrivener, a PR would be most welcome.

Let me describe the current state of scrivener and then we can discuss options for where the PR should be submitted.

• The master branch of scrivener tracks ecto ~> 1.0
• The v2 branch of scrivener has moved to a protocol-based API[1]. The ecto-specific logic has been extracted to scrivener_ecto. The scrivener_ecto library tracks ecto ~> 2.0. I will merge the v2 to master when ecto 2.0 is released.

In an ideal world, I'd love to add this functionality both to the master branch of scrivener for ecto ~> 1.0 support and to the master branch of scrivener_ecto (with subqueries) for ecto ~> 2.0 support. Obviously this is more work than 1 PR. If you're only inclined to submit one PR, I'd prefer the PR on scrivener_ecto as ecto ~> 2.0 support will be the future of the library.

Let me know your thoughts. Thanks again for the research you've done and the issue you've opened.

[1] - https://blog.drewolson.org/extensible-design-with-protocols/

[Thomas P]

Hey Drew,

I think I'll make time today to write a PR. I'll target master first, as it's the branch my company's code is using, but it shouldn't be complicated to write one for v2, although that might not be today.

However, when I tried yesterday, I couldn't run tests, even with a DB user set. It yelled at me about not finding a password key in a map. Could that be because the Scrivener tests expect the DB user to be password-less, while it's not?

By the way, nice refactor for v2! :D

Thanks,
Thomas

[Drew Olson]

You'll probably need to add an entry to test.exs similar to this line: https://github.com/drewolson/scrivener/blob/master/config/test.exs#L7

Perhaps something like:

password: System.get_env("SCRIVENER_DB_PASSWORD") || ""