Filtering compound fields with `:ilike_and` operator results in wrong WHERE condition.

[Linus De Meyere]

I'm a bit confused by the compound fields documentation. I tried to improve it a bit, but I think there is still an issue. When I follow the documentation and try to use the :ilike_and operator as suggested to create a compound field filter for :given_name and :family_name I expect a filter as documented (adapted a little bit to use the like operator):

WHERE (family_name ilike '%margo%' OR given_name ilike '%margo%')
AND (family_name ilike '%martindale%' OR given_name ilike '%martindale%')

The end goal here is that any row should be found that contains either margo or matindale in the :family_name column, and that contains either margo or matindale in the :given_name column. So entering the full name margo matindale should result in a match when the table contains %{given_name: "margo", family_name: "matindale"}, but also should show %{given_name: "matindale" family_name: "margo"}, if that exists somehow. That's how I'd like to use it and how the documentation suggests doing this.

I'm seeing something different though. When I enter a whole name (family name and given name separated by a space), I don't get the expected results when using the :ilike_and operator.

Using Flop.filter/2 directly, I get the following output:

defmodule Person do
  use Ecto.Schema

  @derive {
    Flop.Schema,
    filterable: [:full_name],
    sortable: [:full_name],
    compound_fields: [full_name: [:family_name, :given_name]]
  }
  schema "persons" do
    field(:family_name, :string)
    field(:given_name, :string)
  end
end

Flop.filter(Person, %Flop{filters: [%Flop.Filter{field: :full_name, op: :ilike_and, value: "margo matindale"}]}, for: Person)

#Ecto.Query<from p0 in Person,
   where: ^true and
  (^false or
     (^true and
        (^true and ilike(p0.family_name, ^"%margo%") and ilike(p0.family_name, ^"%matindale%"))) or
     (^true and
        (^true and ilike(p0.given_name, ^"%margo%") and ilike(p0.given_name, ^"%matindale%"))))>

which simplifies to:

#Ecto.Query<from p0 in Person,
   where:
      (ilike(p0.family_name, ^"%margo%") and ilike(p0.family_name, ^"%matindale%")) or
      (ilike(p0.given_name, ^"%margo%") and ilike(p0.given_name, ^"%matindale%"))
>

The last filter expression is not equivalent to the documented result, and does not return the expected result. I thought I had confused :ilike_and and :ilike_or, but that doesn't seem to give the expected results either. So either the documentation is confusing, or there is a bug (I'd hope the latter because then I could still use the :ilike_and operator once fixed).

While I was researching the compound fields, I also noticed that the :empty and :not_empty operators produce queries that are invalid. Using the filter %Flop.Filter{field: :full_name, op: :empty, value: "margo"} results in this invalid query:

#Ecto.Query<from p0 in Person,
   where: ^true and
  (^true and (^true and is_nil(p0.family_name) == ^"margo") and
     (^true and is_nil(p0.given_name) == ^"margo"))>

Not specifying any :value option results in no WHERE condition at all, which is also not desired. Same holds for the :not_empty operator.

Thanks for any clarifications in advance! Happy to provide any further info if needed.

Versions:

• Flop: 0.19.0
• Elixir: 1.14.3
• Erlang: 25
• Ecto: 3.9.4

[Mathias Polligkeit]

Thanks for the report!

The last filter expression is not equivalent to the documented result, and does not return the expected result. I thought I had confused :ilike_and and :ilike_or, but that doesn't seem to give the expected results either. So either the documentation is confusing, or there is a bug (I'd hope the latter because then I could still use the :ilike_and operator once fixed).

That definitely doesn't look right. I found the place where it goes wrong, but I didn't have time to work on a fix yet.

While I was researching the compound fields, I also noticed that the :empty and :not_empty operators produce queries that are invalid. Using the filter %Flop.Filter{field: :full_name, op: :empty, value: "margo"} results in this invalid query:

#Ecto.Query<from p0 in Person,
   where: ^true and
  (^true and (^true and is_nil(p0.family_name) == ^"margo") and
     (^true and is_nil(p0.given_name) == ^"margo"))>

Not specifying any :value option results in no WHERE condition at all, which is also not desired. Same holds for the :not_empty operator.

The empty and not_empty filters work with boolean values. Simplified:

%Flop.Filter{field: :full_name, op: :empty, value: true}
# => `is_nil(p0.name) == true`

%Flop.Filter{field: :full_name, op: :empty, value: false}
# => `is_nil(p0.name) == false`

%Flop.Filter{field: :full_name, op: :not_empty, value: true}
# => `not is_nil(p0.name) == true`

%Flop.Filter{field: :full_name, op: :not_empty, value: false}
# => `not is_nil(p0.name) == false`

nil values are ignored for empty/not_empty filters the same way as for all other operators. If you think about an HTML form, this allows you to:

• use a checkbox without hidden input for unchecked value to produce true when checked (only empty) or nil when unchecked (no filter)
• use a checkbox with hidden input for unchecked value to produce true when checked (only empty) or false when unchecked (only not empty)
• use 2-3 radio buttons to switch between true (value="true"), false (value="false") and nil (value=""), depending on the need

The changes for #286 will likely introduce filter value validation, so you should get a validation error in the future if you try to use an arbitrary string as a value for an empty/not_empty filter. I also think that "true" and "false" are not cast as boolean at the moment, I'll have a look at that as well.

[Linus De Meyere]

Thank you for taking the time!

I understand the workings of empty better now. In that regard, everything works as expected.

I recently had the usecase that I wanted to apply the empty operator by default (without hidden input, when mounting a LV), and then remove the filter when a checkbox is checked (as a way to include more items in the list, if explicitly requested).
I found a solution by always applying the empty condition on the base queryable, and then use the not_empty operator of Flop to basically append to the condition to also allow the same field to not be empty. This results in a query that does field IS NOT NULL AND field IS NULL. I'm hoping the postgres query analyzer does its job there. But if you have any suggestions to make this cleaner, I'm all ears.

Thanks again for your time on this!

[Mathias Polligkeit]

You could use a hidden/checkbox input combination for the empty operator, where the hidden input has the value "true" and the checkbox has the value "". In plain HTML:

<input name="..." type="hidden" value="true" />
<input name="..." type="checkbox" value="" />

This would result in %Flop.Filter{field: :some_field, op: :empty, value: "true"} when the checkbox is unchecked and %Flop.Filter{field: :some_field, op: :empty, value: nil} when the checkbox is checked. In the latter case, the filter wouldn't be applied.

If you are using an input component similar to the one in Phoenix 1.7-rc, you could do this by adding checked_value and unchecked_value assigns to the component.