Flop

Flop is an Elixir library that applies filtering, ordering and pagination parameters to your Ecto queries.

Features

offset-based pagination with offset/limit or page/page_size
cursor-based pagination (aka key set pagination), compatible with Relay pagination arguments
ordering by multiple fields in multiple directions
filtering by multiple conditions with various operators on multiple fields
parameter validation
configurable filterable and sortable fields
join fields
compound fields
query and meta data helpers
Relay connection formatter (edges, nodes and page info)
UI helpers and URL builders through Flop Phoenix.

Installation

Add flop to your list of dependencies in mix.exs:

def deps do
  [
    {:flop, "~> 0.20.1"}
  ]
end

If you want to configure a default repo, add this to your config file:

config :flop, repo: MyApp.Repo

Alternatively, you can add a configuration module. For more information, refer to the Flop module documentation.

Usage

Define sortable and filterable fields

To configure the sortable and filterable fields, derive Flop.Schema in your Ecto schema. While this step is optional, it is highly recommend, since the parameters you will pass to the Flop functions will come from the user side and should be validated. Deriving Flop.Schema will ensure that Flop only applies filtering and sorting parameters on the configured fields.

defmodule MyApp.Pet do
  use Ecto.Schema

  @derive {
    Flop.Schema,
    filterable: [:name, :species],
    sortable: [:name, :age, :species]
  }

  schema "pets" do
    field :name, :string
    field :age, :integer
    field :species, :string
    field :social_security_number, :string
  end
end

You can also define join fields, compound fields, max and default limit, and more. See the Flop.Schema documentation for all the options.

Query data

You can use Flop.validate_and_run/3 or Flop.validate_and_run!/3 to validate the Flop parameters, retrieve the data from the database and get the meta data for pagination in one go.

defmodule MyApp.Pets do
  import Ecto.Query, warn: false

  alias Ecto.Changeset
  alias MyApp.{Pet, Repo}

  @spec list_pets(map) ::
          {:ok, {[Pet.t()], Flop.Meta.t()}} | {:error, Flop.Meta.t()}
  def list_pets(params \\ %{}) do
    Flop.validate_and_run(Pet, params, for: Pet)
  end
end

The for option sets the Ecto schema for which you derived Flop.Schema. If you didn't derive Flop.Schema as described above and don't care to do so, you can omit this option (not recommended, unless you only deal with internally generated, safe parameters).

On success, Flop.validate_and_run/3 returns an :ok tuple, with the second element being a tuple with the data and the meta data.

{:ok, {[%Pet{}], %Flop.Meta{}}}

Consult the docs for more info on the Meta struct.

If you prefer to validate the parameters in your controllers, you can use Flop.validate/2 or Flop.validate!/2 and Flop.run/3 instead.

defmodule MyAppWeb.PetController do
  use MyAppWeb, :controller

  alias Flop
  alias MyApp.Pets
  alias MyApp.Pets.Pet

  action_fallback MyAppWeb.FallbackController

  def index(conn, params) do
    with {:ok, flop} <- Flop.validate(params, for: Pet) do
      pets = Pets.list_pets(flop)
      render(conn, "index.html", pets: pets)
    end
  end
end

defmodule MyApp.Pets do
  import Ecto.Query, warn: false

  alias Flop
  alias MyApp.Pets.Pet
  alias MyApp.Repo

  @spec list_pets(Flop.t()) :: {[Pet.t()], Flop.Meta.t}
  def list_pets(flop \\ %Flop{}) do
    Flop.run(Pet, flop, for: Pet)
  end
end

If you only need the data, or if you only need the meta data, you can also call Flop.all/3, Flop.meta/3 or Flop.count/3 directly. Note that these functions do not apply parameter validation. If the parameters are generated through a user action, always use Flop.validate/2 or Flop.validate!/2 first.

If you didn't configure a default repo as described above or if you want to override the default repo, you can pass it as an option to any function that uses the repo:

Flop.validate_and_run(Pet, flop, repo: MyApp.Repo)
Flop.all(Pet, flop, repo: MyApp.Repo)
Flop.meta(Pet, flop, repo: MyApp.Repo)
# etc.

See the docs for more detailed information.

Parameter format

Below are some examples for the parameter format, including the equivalent query parameter strings that could be used with Phoenix.

Pagination

Offset / limit

%{offset: 20, limit: 10}

?offset=20&limit=10

Page / page size

%{page: 2, page_size: 10}

?page=2&page_size=10

Cursor

%{first: 10, after: "g3QAAAABZAACaWRiAAACDg=="}

?first=10&after=g3QAAAABZAACaWRiAAACDg==

%{last: 10, before: "g3QAAAABZAACaWRiAAACDg=="}

?last=10&before=g3QAAAABZAACaWRiAAACDg==

Ordering

The order parameters are split into order_by and order_directions, so that they can be translated into query parameters.

%{order_by: [:name, :age], order_directions: [:asc, :desc]}

?order_by[]=name&order_by[]=age&order_directions[]=asc&order_directions[]=desc

Filters

A complete filter consists of the field, the operator, and the value. The operator is optional and defaults to ==. Filters need to be passed as a list and are combined with a logical AND. It is currently not possible to combine filters with an OR.

%{filters: [%{field: :name, op: :ilike_and, value: "Jane"}]}

?filters[0][field]=name&filters[0][op]=ilike_and&filters[0][value]=Jane

See the documentation of Flop.Filter and the type documentation of t:Flop.t/0 for more details.

Internal parameters

Flop is built to handle parameters generated by a user. While you could manipulate those parameters and add additional filters when you receive them, it is recommended to cleanly separate the parameters you get from the outside and the parameters that your application needs to add internally.

For example, if you need to scope a query depending on the current user, it is preferred to add a separate function that adds the necessary WHERE clauses:

def list_pets(%{} = params, %User{} = current_user) do
  Pet
  |> scope(current_user)
  |> Flop.validate_and_run(params, for: Pet)
end

defp scope(q, %User{role: :admin}), do: q
defp scope(q, %User{id: user_id}), do: where(q, user_id: ^user_id)

To add additional filters that can only be used internally without exposing them to the user, you can pass them as a separate argument. You can use the same argument to override certain options depending on where the function is used.

def list_pets(%{} = args, opts \\ [], %User{} = current_user) do
  flop_opts =
    opts
    |> Keyword.take([
      :default_limit,
      :default_pagination_type,
      :pagination_types
    ])
    |> Keyword.put(:for, Pet)

  Pet
  |> scope(current_user)
  |> apply_filters(opts)
  |> Flop.validate_and_run(flop, flop_opts)
end

defp scope(q, %User{role: :admin}), do: q
defp scope(q, %User{id: user_id}), do: where(q, user_id: ^user_id)

defp apply_filters(q, opts) do
  Enum.reduce(opts, q, fn
    {:last_health_check, dt}, q -> where(q, [p], p.last_health_check < ^dt)
    {:reminder_service, bool}, q -> where(q, [p], p.reminder_service == ^bool)
    _, q -> q
  end)
end

Relay and Absinthe

If you are serving a GraphQL API using absinthe and absinthe_relay (or even if you just need to support the Relay cursor specification), you can use the functions in the Flop.Relay module to turn the query responses into the format that is expected by Relay.

Let's say you defined node objects for owners and pets, and a connection field for pets on the owner node object.

node object(:owner) do
  field :name, non_null(:string)
  field :email, non_null(:string)

  connection field :pets, node_type: :pet do
    resolve &MyAppWeb.Resolvers.Pet.list_pets/2
  end
end

node object(:pet) do
  field :name, non_null(:string)
  field :age, non_null(:integer)
  field :species, non_null(:string)
end

connection(node_type: :pet)

Absinthe Relay will define the arguments after, before, first and last on the pets field. These are the same argument names that Flop uses, so it will already know how to apply them.

We're going to define a list_pets_by_owner/2 function in the Pets context.

defmodule MyApp.Pets do
  import Ecto.Query

  alias MyApp.{Owner, Pet, Repo}

  @spec list_pets_by_owner(Owner.t(), map) ::
          {:ok, {[Pet.t()], Flop.Meta.t()}} | {:error, Flop.Meta.t()}
  def list_pets_by_owner(%Owner{id: owner_id}, params \\ %{}) do
    Pet
    |> where(owner_id: ^owner_id)
    |> Flop.validate_and_run(params, for: Pet)
  end
end

Now all you need to do in your resolver is to call that function and to call Flop.Relay.connection_from_result/1, which turns the result into a tuple consisting of the edges and the page_info, as expected by absinthe_relay.

defmodule MyAppWeb.Resolvers.Pet do
  alias MyApp.{Owner, Pet}

  def list_pets(args, %{source: %Owner{} = owner} = resolution) do
    with {:ok, result} <- Pets.list_pets_by_owner(owner, args) do
      {:ok, Flop.Relay.connection_from_result(result)}
    end
  end
end

If you want to add additional filter arguments, you can use Flop.nest_filters/3 to convert simple filter arguments into Flop filters without requiring users of your API to know about the Flop filter format.

Let's add name and species filter arguments to the pets connection field.

node object(:owner) do
  field :name, non_null(:string)
  field :email, non_null(:string)

  connection field :pets, node_type: :pet do
    arg :name, :string
    arg :species, :string

    resolve &MyAppWeb.Resolvers.Pet.list_pets/2
  end
end

Assuming that these fields were already configured as filterable with Flop.Schema, we can use Flop.nest_filters/3 to take the filter arguments and convert them into a list of Flop filters.

defmodule MyAppWeb.Resolvers.Pet do
  alias MyApp.{Owner, Pet}

  def list_pets(args, %{source: %Owner{} = owner} = resolution) do
    args = nest_filters(args, [:name, :species])

    with {:ok, result} <- Pets.list_pets_by_owner(owner, args) do
      {:ok, Flop.Relay.connection_from_result(result)}
    end
  end
end

Flop.nest_filters/3 uses the the equality operator :== by default. You can override the default operator per field.

args = nest_filters(args, [:name, :species], operators: %{name: :ilike_and})

Flop Phoenix

Flop Phoenix is a companion library that defines view helpers for use in Phoenix templates.

michallepicki / flop