HTTP Exceptions not being raised/captured from Search API

[John K Kim]

Describe the bug
I am utilizing censys_hosts.search API. When an HTTP error occurs, for example 400 or 408, the API is not raising an exception as expected. Instead, I have added a manual workaround to make a second request to the same domain with HTTP requests library in order to properly capture the HTTP error.

To Reproduce
here is my main code which I am referring to:

def search_domain(query_string, all_results):
    retry_count = 0
    all_ips = []
    while True:
        try:
            search_results = censys_hosts.search(query=query_string, per_page=100, pages=50) # Page limit 50
            logging.info(f"Init Request: Searching for '{query_string}'")
            for result in search_results:
                if result == []:
                    check_http_error(query_string)
                logging.info(f"Continuing to fetch data for... '{query_string}'")
                all_ips.append(result)  # Process and store each result
            all_results[query_string] = all_ips
            break
        except CensysRateLimitExceededException as e:
            write_data_to_s3(bucket_name, output_file_key, all_results)
            logging.info(
                f"Progress saved to S3 at {output_file_key} in case of potential failure. Applying exponential backoff...")
            exponential_backoff(retry_count)
            retry_count += 1
            continue # Ensure retrying the loop from where it was interrupted
        # HTTP
        except HTTPErrorException as e:
            handle_exception(e, query_string, all_results)
            break
        except CensysSearchException as e:
            handle_exception(e, query_string, all_results)
            break
        except Exception as e:
            handle_exception(e, query_string, all_results)
            break

Expected behavior
if result == []: check_http_error(query_string) is the line I have added which utilizes a different function I made in order to make an HTTP request and capture the error properly.

Without this, CensysSearchException and Exception clauses never capture any error -- instead an empty response is returned.

For example, AndyOBrien.com should return either 500 or 408 but instead it returns an empty value with no HTTP exception raised (without my manual HTTP error check included).

I am not sure if I am not capturing errors correctly or confused about something, but would appreciate some clarity.