Integration tests (proof-of-concept)

[Michael Reinstein]

I'm working on issue #950 and the only way to test this is to interact with an <input type="checkbox"> element on the page.

Our unit tests don't provide a way to script this because they don't run in an actual browser.

Integration tests would enable a larger set of testable assertions because they would run in a browser. I've protoyped this out on my local machine, current proof of concept living at tests/integration/tests.js:

import http              from 'http';
import handler           from 'serve-handler';
import puppeteer         from 'puppeteer';
import { dirname, sep }  from 'path';
import { fileURLToPath } from 'url';
import { assert }        from 'chai';


const __dirname = dirname(fileURLToPath(import.meta.url));
const LOCAL_PORT = 8081;


async function main () {
	// start the local test server
	const server = await serverListen();

	const browser = await puppeteer.launch({ headless: 'new' });

	await Promise.all([
		testIssue950(browser),
	]);

	server.close();
	browser.close();
}


async function serverListen () {
	const root = `${__dirname}${sep}..${sep}..${sep}`;
	
	const server = http.createServer((request, response) => {
		// You pass two more arguments for config and middleware
		// More details here: https://github.com/vercel/serve-handler#options
		return handler(request, response, { public: root, cleanUrls: false });
	});

	return new Promise(function (resolve) {
		server.listen(LOCAL_PORT, () => {
	  		resolve(server);
		});
	});
	
}


// https://github.com/snabbdom/snabbdom/issues/950
async function testIssue950 (browser) {
	const page = await browser.newPage();
	
	await page.goto(`http://127.0.0.1:${LOCAL_PORT}/test/integration/issue-950/index.html`);

	const c = await page.evaluate(() => {
		const checkboxes = [ ...document.querySelectorAll('input[type=checkbox]') ];
		return checkboxes.map((c) => c.checked);
    });

    assert.deepEqual(c, [ false, false ]);

    // click the first checkbox
	await page.evaluate(() => {
		const checkbox = document.querySelector('input[type=checkbox]');
		checkbox.click();
    });

    const c2 = await page.evaluate(() => {
		const checkboxes = [ ...document.querySelectorAll('input[type=checkbox]') ];
		return checkboxes.map((c) => c.checked);
    });

    assert.deepEqual(c2, [ false ]);
}


main();

Thoughts?

[Jan van Brügge]

I am a bit confused, all of our unit tests are running in the browser, even multiple different ones with browserstack

[Michael Reinstein]

Where in our unit tests do we interact with browsers? (clicking on things, entering text, etc.) ? As far as I can tell our unit tests are simple scripts that just run in the context of a browser with no interactivity. That is what is required to be able to test some of these issues.

[Alexander Kurakin]

How did I do the integration testing?
The scripts creates a bundle .m?js file from the main-test-file. (Then you run it as-you-want 😄) The output goes to the console. The failures become exceptions.

[Michael Reinstein]

Your link points to a completely different repo that doesn't seem to have anything to do with snabbdom, unless I'm missing something.

[Michael Reinstein]

@jvanbruegge after looking at browserstack a little, it seems they do offer pupeteer integration: https://www.browserstack.com/docs/automate/puppeteer/migrate-existing-test-suites

Though this doesn't answer the question of how one loads/hosts an external test page to run said tests on.

[Alexander Kurakin]

Your link points to a completely different repo that doesn't seem to have anything to do with snabbdom, unless I'm missing something.

Well, the idea is: a lightweight solution instead of servers, etc.

1. Create bundle with all tests.
2. Create HTML page with <script src="bundle.js"></script>.
3. Open in browser (maybe headless one).
4. Catch output and exceptions.

P.S. These tests are not integrational (sorry) just unit tests in the browser.

[Michael Reinstein]

Well, the idea is: a lightweight solution instead of servers, etc.

Ahh I see. Yeah lightweight/not needing a server is a great quality. On several open source repos that I've maintained in the past, the cross browser testing service is the first thing that breaks. Not relying on some 3rd party (especially a for profit one) is a strong maintenance plus.

[Alexander Kurakin]

But yeah, I missed my solution != integration tests (at all 😄)

[Simon Friis Vindum]

Our unit tests don't provide a way to script this because they don't run in an actual browser.

As @jvanbruegge stated, the current unit tests already run in a browser.

Are you sure that you can not simulate the interaction you need with JavaScript? For instance by using click()? That approach is currently being used to test the eventlisteners module here.

[Michael Reinstein]

closing since we don't like this, and since the PR was rejected that this test was intended for.

[Simon Friis Vindum]

Just to clarify, I don't consider the other PR "rejected". Rather, I'd say we should consider other options in the design space before we make a decision.

Additionally, we certainly want to close #950 and at that point, it would be nice to have a test. I think it would be ideal if that could be accomplished with the current test setup, but if not Puppeteer might fit the bill.

[Michael Reinstein]

I don't consider the other PR "rejected"

Given your latest comment in 950, I don't know how one could come to any other conclusion:

• special casing on value and checked is worrying (basically the entire strategy of the PR)
• this may not be a bug (my original thought too, but it's unclear)
• we may want to handle the input props more like react does (major departure from the PR)

That said, there is nothing wrong with it being rejected. I would rather a bad PR be rejected than ram it through to make me feel good about merging it in. :-)

It sounds like 950 is going to require a different approach, and not knowing anything about what that will be, I can't pretend to understand what's needed to test that future solution.