Order by custom field with custom ordering logic

[Luca Corti]

Is your feature request related to a problem? Please describe.

We are using custom fields to support filtering on a composite ecto type. Specifically, we use ex_money which has a type which is backed by a composite PostgreSQL type which stores amounts as {string, decimal}. Filtering works well by defining the respective custom fields along with custom filtering logic:

custom_fields: [
      amount_amount: [
        filter: {Filters, :money, [source: :amount, field_name: "amount"]},
        ecto_type: :decimal
      ],
      amount_currency: [
        filter: {Filters, :money_currency, [source: :amount]},
        ecto_type: :string
      ]
    ]

However we also need to use the custom fields for sorting, but this is not supported.

Describe the solution you'd like

The optimal solution for us would be to have the same mechanism currently supported for filtering custom fields to be able to specify custom ordering for the composite fields.

Describe alternatives you've considered

The documentation suggests alias fields but apparently you can't support both filtering and sorting with an alias field. Also, these are stored values, not computed.

Additional context

This is more of a question about understanding the motivation behind disallowing ordering for custom fields. Is there any issue that prevents this or is it just missing functionality. In the latter case, we'd might give a shot at implementing support for this.

[Mathias Polligkeit]

For filtering, all you need to do to implement a custom field is to define a function that adds a where clause to the query. For sorting, you will not only need to add an order by clause. Flop would also not be able to build the where clause for you that is needed for cursor-based pagination. So you would have to handle the whole cursor-pagination logic on your own.

Then the question is what the interface for that should look like. A single custom field function for sorting wouldn't be enough, and since you could have multiple custom fields available for sorting, the function for applying the cursor where clause needs to be defined separately from the custom fields. I don't know from the top of my head how that interface should look like. It would also take one of the core features out of Flop and into your hands.

The easy way out would be to disallow cursor-based pagination as soon as a custom field is marked as sortable. The better way would be to revisit the idea for derived/dynamic fields at some point. There are other features I'd like to work on before that, though, so this has no priority for me at the moment.

So that's in a nutshell why it is not supported at the moment.

---

There is one workaround you can do right now in your situation. You can define join fields for anything that you can put a binding name on. This means you can do a lateral join on a sub query, add a named binding to it, and define a join field referencing that binding.

I didn't have an application with ex_money ready, but I did have one with a Postgres tsrange field.

So first you define a sub query that basically unpacks the composite type:

publish_period_query =
  from(
    f in fragment(
      "select lower(?) as lower, upper(?) as upper",
      parent_as(:article).publish_period,
      parent_as(:article).publish_period
    ),
    select: %{lower: f.lower, upper: f.upper}
  )

I don't know how else to select fields that don't come from a separate Ecto query. Also, Ecto doesn't allow you to select f directly, you have to explicitly select a map from the fragment. Maybe this can be written in a nicer way, but anyway, bear with me.

You can now perform a lateral join on that sub query and give it a name:

  q =
    from(a in Article,
      as: :article,
      inner_lateral_join: pp in subquery(publish_period_query),
      as: :publish_period
    )

The resulting SQL query will look something like this:

SELECT ... FROM "articles" AS a0
INNER JOIN LATERAL
  (
    SELECT sf0."lower" AS "lower", sf0."upper" AS "upper"
    FROM (select lower(a0."publish_period") as lower, upper(a0."publish_period") as upper) AS sf0
  ) AS s1 ON TRUE

If you were to write this query in raw SQL, you could of course simplify this quite a bit, but we're bound by the Ecto syntax here.

This admittedly looks quite horrible, but the good news is that the PostgreSQL optimizer is smart enough to see through this nonsense. Running that query as it is results in this query plan:

Seq Scan on articles a0  (cost=0.00..5.75 rows=50 width=2793)

And if you add the where clause where([publish_period: pp], pp.lower <= ^DateTime.utc_now()):

Seq Scan on articles a0  (cost=0.00..5.75 rows=17 width=2793)

Filter: (lower(publish_period) <= '2023-06-13 11:37:53.976703'::timestamp without time zone)

So the sub queries from the generated query are optimized away.

[Luca Corti]

Thanks for the context around this and the suggestions. Indeed, we are using cursor based pagination. Another option we are investigating is using the composite field directly for sorting, since it should be simpler and works for us.

This kinda works, but there is an issue with cursor pagination we are trying to make sense of.