cjoudrey / graphql-schema-linter

This is more of a question than an issue, but curious if it's possible to do warns instead of errors, similar to how eslint structures their config.

Hey @andrewmartin, just to clarify, you're asking if it is possible to have some rules warn instead of error?

If so, at the moment that isn't possible, but I think it'd be an interesting feature to add.

If you have ideas on how you'd implement this, feel free to share. 😄

@cjoudrey sure! Just food for thought. Here's a sample eslint file:
https://github.com/standard/eslint-config-standard/blob/master/eslintrc.json#L73-L105

Simply put, we'd need to restructure the config to allow objects in addition to simple strings:

{
  "rules": [
    "input-object-values-are-camel-cased",
    "relay-connection-types-spec",
    "relay-connection-arguments-spec",
    {
       "types-are-capitalized": "warn"
    }
  ]
}

I haven't dug too deep into how the errors are thrown from the rulesets, but you'd want to (likely) just send a different exit code based on the config setting.

we'd need to restructure the config to allow objects in addition to simple strings

Your proposal makes sense to me. 😄

By default the level would be error, but can be overrided to be warn.

you'd want to (likely) just send a different exit code based on the config setting.

Yeah, upon exiting I think the exit code should be based on the level of the rules.

Something along the lines of:

If there are no errors, exit code is 0.
If there are errors and at least one of them is for a rule of level error, exit code is 1.
If there are errors and all of them are for rules of level warn, exit code is 0.

The relevant code that would need to be changed is here:

graphql-schema-linter/src/runner.js

Line 108 in c107659

return errors.length > 0 ? 1 : 0;

Feel free to open a pull request to add this feature. 😄 I think it would be a nice addition.

This would be extremely useful for me, especially in adding custom rules where I want to emit a warning instead of an error, so I can raise potential problems with schema additions/changes without breaking our builds.

Hi @cjoudrey I'm considering taking a crack at this and I'm getting acquainted with how everything is run before I make any changes.

This could potentially be implemented using the existing rulesOptions key that is now possible in the configuration file, with every rule (built-in and custom) having a simple API for options:

type RuleOption = {
  reportingLevel?: "off" | "warn" | "error"
  [key: String]: String
}

In the configuration file, this would look like this:

{
  "rulesOptions": {
    "enum-values-sorted-alphabetically": {
      "reportingLevel": "warn",
      "sortOrder": "lexicographical"
    },
    "types-have-descriptions": {
      "reportingLevel": "error"
    }
  }
}

If the reportingLevel isn't specified in a rule option, the default should be error. This is probably the easiest way to ask for warnings instead of errors while respecting the current design, which splits listing which rules to run (in rules) from any configuration options for those rules (in rulesOptions). The downside to this approach is that it's not consistent with the general syntax for configuring rules in ESLint.

For warnings, we want to provide each rule with a function to call instead of context.reportError() if the reportingLevel is set to warn. This function should push a warning to a collection defined somewhere in validator.js. Ideally we could extend the ValidationContext that is passed as context to each ValidationRule in the array passed to validate(), but failing that I think it could be done with an additional function or "custom context" passed to each rule. Each rule will need to be modified to respect the reporting level for that rule, calling context.reportError() only if the reportingLevel is error.

In order to write grouped warnings to stdout, validateSchemaDefinition() should probably be modified to return something like [filteredWarnings, filteredErrors] instead of just filteredErrors, and runner.js:111, where it's called, would be updated to const [warnings, errors] = validateSchemaDefinition(...).

The exit code in runner.js:121 probably doesn't need to change, unless some config option is desired that allows for a "max warnings" option like ESLint has, which could probably just check warnings.length if that option is specified.

Thoughts?

Warning instead of error?