No error warning when upload to gens fails
beatrizsavinhas opened this issue · comments
Description
When running cg upload gens <case_id>, if the upload fails, there is no indication or error message to the user.
Suggested solution
Raise an error when the gens upload fails.
This can be closed when
An error is raised when gens upload fails.
Blocked by
Nothing to my knowledge.
If I'm not mistaken, gens load sample will simply print warnings for each sample that failed loading. Maybe it would be easier to catch with like an error code return e.g. around here: https://github.com/Clinical-Genomics/gens/blob/145aed2792b674fac16e5c5790882f871dbb99e3/gens/db/samples.py#L40? 😊
Indeed, logging instead of crashing there was a band-aid fix before we added case_id to our index.
How should the error message be propagated up to cg? Should I translate the exit status within cg, or is there some way to use stderr or something similar?
On a related note, how should we deal with re-runs where the case_id/individual_id-combo is already in use? We could either overwrite the old gens sample or add another key to our index.
Indeed, logging instead of crashing there was a band-aid fix before we added
case_idto our index.How should the error message be propagated up to cg? Should I translate the exit status within cg, or is there some way to use stderr or something similar?
I suggest as @dnil - let Gens thrown an exception and let cg catch it ia a try-except and log a warning. If we are uploading multiple cases from the same process we do not want exit if one fails.
On a related note, how should we deal with re-runs where the
case_id/individual_id-combo is already in use? We could either overwrite the old gens sample or add another key to our index.
I think we want to overwrite, but @dnil probably knows this better.
I dunno, it's more policy thing than anything. For some other systems we rather flag the the duplicate, protest, and let the user either delete the old case or pass a force flag to overwrite. As long as it is documented and clear I dont think it matters a whole bunch - the combos are rather unique after all. But since you ask, my two cents are that having this kind of clash should be an exception, and not just be brushed under the mat without some form of manual input. I would suggest catching the error in gens, failing with it from cg and adding a gens delete command to make it straightforward for the operator/cg to remove a case from gens and re upload if all is well.
Right, that makes a lot of sense. I've made a PR in Gens Clinical-Genomics/gens#70 that should be enough to close this issue as originally specified, but I've created a couple of issues for the other things:
To recap the approach we decided on in #3307, two other stages in mip dna upload (upload_coverage & upload_genotypes) both log a warning when a duplicate is found. Crucially however, they also take a flag that tells them to overwrite the previous entry during a rerun.
A --force flag has been added to gens load sample and it's been been hooked up to the restart parameter of MipDNAUploadAPI's upload method.
Once #3322 has been merged and deployed, this issue can be closed.
@raysloks we just removed the --restart flag from the coverage upload. The reasoning being that the automation will not update a sample coverage that has gotten a top up which would require manual intervention. Although the sample update is what should happen in this case regardless.
I am unsure what gens is exactly. However, should this not also only contain up-to-date info and therefore be overwritten by default if an already existing sample is to be uploaded?
Right, do I understand correctly that there are times when the restart-parameter here is false but we actually should overwrite old entries?
In that case I completely agree that we should always tell gens to update old entries.
Indeed, I was thinking the default state should always be overwrite. But then logging a warning if that happens.
@dnil can you confirm? Should existing samples in gens be overwritten by default with a warning added to the log by default? Or should this be a manual action?
TL; DR:Yes, it is ok to force gens upload. Unforced upload fails would be expected on each rerun. But read the logs! 😁
Sorry to be late to the backlog, busy day! 😅 To reiterate, as long as you document it carefully, I don't think it matters a whole deal.For context, @ChrOertlin the coverage report is not exactly the same as gens. Chanjo has some more connotations as it is what ppl use to do their customer side QC verification, on top of using it a bit interactively. Gens is for cytogenetic analysis, period.
The sample db of gens which is what we are talking about uploading to is a really simple thing. It will keep track of where a samples authoritative depth- and BAF files are located for when someone views them in gens, typically via scout. Most of the time there is a collision, the reason will be that there is a new reanalysis, better top-up or whatnot, in which case overwrite is exactly what we want. Before @raysloks added the --force flag, manual fixing was the only option, so cheers for that!
As for the use in cg I can see how when it makes sense to do it the same way. E.g. reruns are rather common, and should not need special hands on treatment. I would have been tempted to reuse that restart logic, or invent a new rerun flag to note this as an expected procedure. But likely adding complicated logic for a common case is not worth it!
More complicated conditions may ensue in the future when multiple pipelines use gens, and could possibly have the same sample. But we are not there yet.
Just make sure someone reads those warnings somewhat regularly to catch drift of anomalies, ideally ahead of the customers. The busy day so far was mostly because we had not for the warnings in the scout upload logs.