ecmwf / cdsapi

For the evaluation of CAMS2-40 model results, withing the CAMS2-83 project we download 22 different model results every day.

When requesting "cams-europe-air-quality-forecasts" data that is missing, or has not yet been delivered cdsapi raises an Exception with the following message:

the request you have submitted is not valid. There is no data matching your request. Check that you have specified the correct fields and values..

Currently, our application that handles the downloading needs to check the exception message
in order to find out if the data is not yet available or if there is a different kind of error.

    try:
        client.retrieve("cams-europe-air-quality-forecasts", request, tmp_path)
    except Exception as e:
        if "There is no data matching your request" in str(e):
            logger.info(f"No {model} data to be found")
            return
        logger.error(e)
        raise FailedToDownload(f"failed to download {model}") from e

As far as I can tell, the Exception we're trying to handle comes from lines 506 to 509 of apy.py

cdsapi/cdsapi/api.py

Lines 494 to 509 in f3b94a9

    
           if reply["state"] in ("failed",): 
        
               self.error("Message: %s", reply["error"].get("message")) 
        
               self.error("Reason:  %s", reply["error"].get("reason")) 
        
               for n in ( 
        
                   reply.get("error", {}) 
        
                   .get("context", {}) 
        
                   .get("traceback", "") 
        
                   .split("\n") 
        
               ): 
        
                   if n.strip() == "" and not self.full_stack: 
        
                       break 
        
                   self.error("  %s", n) 
        
               raise Exception( 
        
                   "%s. %s." 
        
                   % (reply["error"].get("message"), reply["error"].get("reason")) 
        
               )

It would be of great help if instead of raising an plain Exception cdsapi would rise a custom exception, say FailedRequest. That way we could limit the scope of the except block to handle only FailedRequest exceptions.

Even better would be if cdsapi would raise custom NoDataMatchingRequest exception when reply["error"].get("reason", "").startswith("There is no data matching your request") or similar.

Hi @avaldebe ,

Unfortunately this is not possible with the current CDS WebAPI which is the backend of the cdsapi. We will note your issue and consider it as a user requirement for the long-term plans to upgrade our system.

Many thanks,
Eddy

	if reply["state"] in ("failed",):
	self.error("Message: %s", reply["error"].get("message"))
	self.error("Reason: %s", reply["error"].get("reason"))
	for n in (
	reply.get("error", {})
	.get("context", {})
	.get("traceback", "")
	.split("\n")
	):
	if n.strip() == "" and not self.full_stack:
	break
	self.error(" %s", n)
	raise Exception(
	"%s. %s."
	% (reply["error"].get("message"), reply["error"].get("reason"))
	)

Custom FailedRequest and NoDataMatchingRequest exceptions