OPERA-SDS-VnV-8: Verify automated product generation
LucaCinquini opened this issue · comments
Requirement: "For each PGE (from current and previous releases), verify that uploading the proper inputs to the designated Input Storage Location will trigger a succesfull PGE execution, followed by PCM moving all generated output to the designated Output Storage location. Includes inspecting the job logs to verify successful job execution.
Inspect the output location of a PGE execution to verify that it contains files with quality metrics information, with fields specific to each PGE
Inspect the latency metrics (one for production time, and one for retrieval time for FWD processing) reports generated by the PCM system, and ensure that they contain all relevant information to asses whether generation of each OPERA products meets the product latency requirement for that product.
Inspect the accountability metrics generated by the PCM system, and verify they contain all neecssary information to asses that all expected output products are generated, for each PGE."
To do (for each PGE/release):
o Enable triggers to automatically query and download data from the DAAC
- explain how you enable the trigger
o Verify that the jobs are started and finish successfully - explain how to check that jobs run and finish successfully
o When jobs complete, verify everything that the requirement asks for, for example outputs are generated, metrics are generated, etc. - explain how you check every single thing
For each release, must check all the PGEs that are enabled by then.
Luca, I have completed this: Please review and close it:
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3833507&group_by=cases:title&group_order=asc&group_id=-1
o Change title to "...Verify.... for the L3_DSWx_HLS PGE"
o Step 4: remove the download trigger and specify in parentheses (for the L3_DSWx_PGE)
o Step 9: remove (there is no need to create the directories)
o Step 11: change the trigger interval to 5 minutes and just wait 5 minutes to check the results
o Step 15: verify where those files should be located - for each job
o Step 16: instead, use the Figaro UI to verify that the jobs are running or complete
o Step 17: use the correct location where the PCM is writing the output files
o Step 19: cannot compare to a standard sample.out file, rather can only verify that the jobs produce output
o Step 20: for a few jobs, verify that...
o Step 25: verify that all fields are populated
o Step 27: "...and click the Search button"
o Must also add steps to check the Data Accountability page to check that no products are missing (ask Chris)
- Step 10: must change the trigger rule to be every 5 minutes
- Step 16: clarify that the folder is inside the LTS bucket
- Add step to verify that no jobs are missing i.e. all files were processed
- Please clean up the syntax
@LucaCinquini
Step 4 is updated to handle (1) time to trigger (2) enabling the trigger rule
Step 16 updated, as suggested.
Step 30 and 34 added to compare files processed vs reporting matrix for each tab.
Cleaned up, as suggested.
About the data accountability: currently step 30 and 34 compare the results in the metrics page with the number of jobs that were submitted. But should we compare to the number of jobs that should have been submitted, as evaluated from an independent query to CMR? What if a job does not get submitted because for example some ancillary data is missing, shouldn't this test be able to identify a problem in that case?
I guess it is better to say to check that no products are missing, or to check that the expected products are present instead jobs. We can compare the successfully ingested products.
Ready to be transferred to TestRail.
Closing.