Investigate a way to manually define test specs (specs.json)

[rklfss]

I'm wondering what the tests comparing the specs.json are good for and if there is any case of failing?

Reason: the specs.json files are generated by the same well-functioning or malfunctioning program that generates the output to compare with (=my locally generated typedoc build).

gruntfile.js

var out = Path.join(base, directory, 'specs.json');
var result = app.convert(src);
var data = JSON.stringify(result.toObject(), null, '  ');
data = data.split(TypeDoc.normalizePath(base)).join('%BASE%');
FS.writeFileSync(out, data);

test/converter.ts

const specs = JSON.parse(FS.readFileSync(Path.join(path, 'specs.json')).toString());
let data = JSON.stringify(result.toObject(), null, '  ');
data = data.split(normalizePath(base)).join('%BASE%');

compareReflections(JSON.parse(data), specs);

The tests always succeed since the specs.json is just the written-to-a-file output that is generated a second time for comparison although the output could be wrong regarding its content.

Example:

module TestModule
{
    /**
     * TestClass class comment.
     */
    export class TestClass
    {
        /**
         * This is a property.
         */
        public property = 'property';
        
        /**
         * This is a method.
         */
        public method(): void
        {

        }
    }

    /**
     * TestClass module comment.
     */
    export module TestClass
    {
        /**
         * This is an exported variable.
         */
        export const exportedVar = 'exported';

        /**
         * This is an internal variable.
         */
        const internalVar = 'internal';

        /**
         * This is an internal function.
         */
        function internalFunction(): void
        {

        }
    }
}

...
                {
                  "id": 8,
                  "name": "internalVar",
                  "kind": 1024,
                  "kindString": "Property",
                  "flags": {
                    "isStatic": true,
                    "isExported": true
                  },
                  "comment": {
                    "shortText": "This is an internal variable."
                  },
                  "sources": [
                    {
                      "fileName": "nested-modules.ts",
                      "line": 35,
                      "character": 25
                    }
                  ],
                  "type": {
                    "type": "stringLiteral",
                    "value": "internal"
                  },
                  "defaultValue": "\"internal\""
                },

...

The isExported flag of "internalVar" is true although it should not.

I would like us to investigate a way to manuelly define specs saying "isExported flag of internalVar must not be true" or things like that.

[Anthony Ciccarello]

That sounds like a great idea! The tests could use a lot of work. I also open #542 to allow more flexibility when testing. Since I didn't write the lib, the lack of effective tests makes it especially hard to review PRs. I'd love to see this implemented.

[Anthony Ciccarello]

I just want to note that I'm also seeing that tests pass when they shouldn't. You can tell that the test shouldn't pass because the specs.json file is changed.

[Klaus Reimer]

What about merging the small PR #762 as a quick fix for this issue? This PR disables the re-building of the specs.json files during the build so the tests actually do something useful (Comparing the new results with the previously committed results). In case something changes and these changes are correct then you can run the grunt task update-specs to update the files and commit them.

I just tested this PR and it works fine. I sabotaged typedoc by changing Type.isTypeListEqual to always return false and then did a full rebuild and the unit tests successfully found the differences caused by this change. This was not the case before because the build silently replaced the JSON files and then ran the tests against the already updated files.

And don't be alarmed by the unit test failures reported for this PR. They are there because the unit tests now actually do something. So these errors are normal and can be fixed by committing the updated spec files once.

[Anthony Ciccarello]

Yep, I'm working on updating the specs now. I do think that there are some bugs to fix though so I'm held up on that. If I merged #762 now, I'm concerned any PRs would try to change unrelated specs.

Thanks for testing it out!

[Anthony Ciccarello]

While investigating this I've noticed that the tests don't run consistently if you run a converter test individually. More investigation is needed to understand that issue.

[Gerrit Birkeland]

I see #762 has been merged. Is this issue now resolved or does more work need to be done before tests can be relied on?

[Anthony Ciccarello]

Yeah, I think it can be. 👍