OPERA-SDSVnV-7: Verify that generation of data products meets projects requirements
LucaCinquini opened this issue · comments
L3 requirement:
Inspect the data products generated by each PGE to verify that they cover the spatial and temporal extent specified in the project data generation table. Also, capture PCM execution metrics to verify product accountability (i.e. that all expected products are actually generated) and latency (i.e. that products are generated within the required time).
Let's split this into 2 tests:
7_1) "Inspect the data products generated by each PGE to verify that they cover the spatial and temporal extent specified in the project data generation table."
Perhaps use some tools like ncdump to inspect the products and retrieve the geographic extent and temporal interval
7_2) "Also, capture PCM execution metrics to verify product accountability (i.e. that all expected products are actually generated) and latency (i.e. that products are generated within the required time)."
Perhaps ask Chris how to generate the metrics reports and verify they conform to the standard proposed by Steven.
This requirement can probably not be closed during I&T since it requires looking at several output files during the course of several days. But rather, we could demonstrate that we are ready to do so by running several jobs distributed across the globe.
Suggestion: submit N jobs using input files available at the LP.DAAC and inspect the geographic location of the output files.
Let's only have one ticket for each PGE.
I don't think the current plan covers the procedure we discussed - there is no mention of submitting jobs and looking at the spatial and temporal coverage to at least partially close this ticket.
Also there is no procedure for verifying the second part of the requirement - the metrics report.
I wonder if this ticket can be executed after another ticket which involves running the jobs...
Step 6: can you please add the data accountability URL?
Step 4: use the HySDS UI to look where those output products fall on the Earth of the surface. To confirm the spatial resolution we must inspect the product metadata (use NC to look at the header of the files, or some similar command).
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/cases/view/1089834
Hey @oddes - this link is not accessible in public GitHub. It will be a dead-link for anyone without access. Please keep URLs like this out of the open source tickets, and instead summarize in words what you need to share. Thanks!
Luca, I have updated please review and close this ticket.
I think this one needs some more work. The requirement that it covers talks about what we will do during operations (produce the data products, keep up with the inputs, etc.). Since the requirement isn't closable during I&T, I was hoping for this test case to be verifying our ability to generate the reports that show compliance with the requirement.
Show that we can produce a report with the following info:
- the spatial extent of data products (from a single PGE) over a specified time period
- data accountability (show that we've processed all of the input files over the time period)
- retrieval time metrics (for FWD production)
- production time metrics
We will be delivering these reports to the Project once per quarter. Hopefully these reports will be enough to close out the requirement at the end in a fairly straight-forward manner.
Updated steps 10 to 15.
Further changes:
o Replace the pre-condition with a ticket that runs one or more PCM jobs
o Do not query the DAACs, rather use the local data to execute the same checks for spatial and temporal extent
Luca, updated as suggested, please review and close.
o Step 1: change title to "Produce our L3_DSWX_HLS data products"
o Get rid of step 8
o Step 6: add the exact command to run
o After step 6 add a new step to verify that the jobs are running
o Step 10: replace "out.out" with "" place holder. Also, states that the purpose is to verify that the product metadata agrees with what expected.
o Step 20: remove since it is a duplicate
o Step 17 and 19: include a step to export a file with the information
After this please email us and we will close.
Luca, Updated all the suggestions.
Thank you - I reviewed and it looks ok to me. If Steven agrees we can close this ticket.
Steven concurs - closing.