Updating Reference Solution for SPE1CASE1
bska opened this issue · comments
Commit dfb0647 (PR #461) introduced a new regression test based on the SPE1CASE1 model from opm-tests. As far as I can tell, there is no automatic way of updating the reference solution—e.g., in the context of adding new data to the IWEL vector—since this regression test does not plug into the update_data request of the Jenkins system.
As of right now that means that said regression test is blocking PR OPM/opm-common#610. I'd prefer not to disable the test, but I also don't want to take on the responsibility of manually keeping the reference solutions in sync with developments elsewhere. Can we reach a reasonable compromise here?
I don't know the effort required to enable automatic updates, but unless that is simple - or at least forthcoming in readonable time I think the test should be disabled.
We already have a framework for updating reference solutions. My preference is disabling the test until it integrates with the existing framework.
We already have a framework for updating reference solutions. My preference is disabling the test until it integrates with the existing framework.
The problem is that the existing framework cannot be used here, because it can only update stuff from opm-simulators. (right, @akva2?)
I would strongly prefer to keep the test and update the reference results manually in the meantime to prevent plain-ebos from being as badly broken as it was before #461. (IIRC @hnil had some serious problems because of this.) I agree that it would be good if the jenkins infrastructure was brought up to speed, though.
BTW: the effort to update the reference data is quite small and it is not required often:
cd $EWOMS_BUILD_DIR
make ebos && ctest -R SPE1 -V
cp SPE1* $EWOMS_SRC_DIR/tests/referencesolutions
# make PRI am afraid the test framework you are trying to use does not cater to your use case. Please switch the test off and initiate a dialogue to get a set-up that fits with your goal. Right now you are blocking critical PRs from being merged.
I agree, the test should be temporarily disabled. I'll make a micro-PR to do that. The only short-term alternative is to disable ewoms tests on Jenkins, which would be a large step backwards.
The test should be done in the same way as the other ones in the first place... When that is done, we'll re-enable the test.
The way to go is to simply disable the validation of the results for all eWoms tests: all arguments against the SPE1 test raised here equally apply to tests which produce VTK output and I bet the willingness to manually update the VTK reference data is even lower than to update the ECL data. (note that this was never a problem so far, because the current test driver is older than the Jenkins infrastructure.)
Since I'm busy with more important stuff this week, I'll open a PR to that end at the beginning of next week.