Language independent integration tests

[Hugo Parente Lima]

Hi,

I'm refactoring the Crystal language port of liquid to be 100% compatible with shopify liquid, to be more confident about the port I will port all integration tests in this repository to Crystal, however I was thinking about a better solution in a way that other liquid implementations could share the same integration tests.

A integration test can be reduced to calls to assert_template_result

    def assert_template_result(
      expected, template, assigns = {},
      message: nil, partials: nil, error_mode: nil, render_errors: false
    )

So my idea is, instead of having:

  def test_output_in_raw
    assert_template_result('{{ test }}', '{% raw %}{{ test }}{% endraw %}')
  end

We could have 2 files: output_in_raw.template and output_in_raw.expected with the respective contents:
{% raw %}{{ test }}{% endraw %}'
and
{{ test }}

There are other kind of integration tests, e.g.:

  def test_invalid_raw
    assert_match_syntax_error(/tag was never closed/, '{% raw %} foo')
    assert_match_syntax_error(/Valid syntax/, '{% raw } foo {% endraw %}')
    assert_match_syntax_error(/Valid syntax/, '{% raw } foo %}{% endraw %}')
  end

This could be split in 3 tests, invalid_raw_1, invalid_raw_2, invalid_raw_3 and the files be:
invalid_raw_1.template, invalid_raw_1.error, invalid_raw_2.template, invalid_raw_2.error, ... the contents of .error files could be always interpreted as a regex

Some tests need variables, some tests need Drop's, here things start to be language dependent, e.g.:

  def test_sort_works_on_enumerables
    assert_template_result("213", '{{ foo | sort: "bar" | map: "foo" }}', { "foo" => TestEnumerable.new })
  end

For this one possible solution would be to have the sort_works_on_enumerables.template, sort_works_on_enumerables.expected and sort_works_on_enumerables.rb for Ruby, .cr for Crystal, etc... each language need to have their own specific file...

For Ruby (and almost for Crystal too) the contents could be:

class TestEnumerable < Liquid::Drop
  include Enumerable

  def each(&block)
    [{ "foo" => 1, "bar" => 2 }, { "foo" => 2, "bar" => 1 }, { "foo" => 3, "bar" => 3 }].each(&block)
  end
end

SORT_WORKS_ON_ENUMARABLES={ "foo" => TestEnumerable.new }

I'm not 100% sure that this could work, this is why I'm asking you guys that have the expertise in shopify liquid template integration tests if this could work or if there's something I didn't see... and would be better to just create my own integration tests based no shopify liquid integration tests.

Other implementations of liquid:

• https://github.com/cobalt-org/liquid-rust
• https://github.com/osteele/liquid
• https://github.com/karlseguin/liquid
• https://github.com/amberframework/liquid.cr
• https://github.com/harttle/liquidjs
• https://github.com/harrydeluxe/php-liquid
• https://github.com/dotliquid/dotliquid

[Dylan Thacker-Smith]

This seems like the same issue as #1621, just with a different proposed solution.

Some tests need variables, some tests need Drop's, here things start to be language dependent, e.g.:

The concept of there being input variables doesn't need to be language dependent, but the input values (e.g. drops) can be. These should just not be covered by the language spec, since they are also specific to the application's liquid API.

I'm not 100% sure that this could work

We have experimented with this and see no reason why it wouldn't work. The approach we took was to monkey patch liquid to generate the test in YAML form, where a test with multiple assert_template_result calls could result in multiple YAML tests. However, we should really separate the liquid tests themselves as I propose in #1621, then replace the ruby tests with the YAML tests, so the YAML tests become the authoritative specification of the liquid behaviour.

[Hugo Parente Lima]

So I see my search for a similar issue before file this one failed 😅 .

I thought about use YAML files too, I just think separate files is better for editing, however YAML files works too... looking at the issue I found https://github.com/jg-rp/golden-liquid, that's exactly what I was looking for.

I'll close this since it's not only duplicated but https://github.com/jg-rp/golden-liquid is also probably exactly what I want.