Giters

outflanknl / RedELK

Red Team's SIEM - tool for Red Teams used for tracking and alarming about Blue Team activities as well as better usability in long term operations.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Hybrid Analysis - error handling around max API hits

MarcOverIP opened this issue · comments

commented

Error handling with Hybrid Analysis fails in some cases:

  1. HTTP error code 525 as reported by daemon.log. This means a TLS error. I believe this is reported by Cloudflare. See also the full HTTP webpage that is reported in the daemon.log as stated below
  2. HTTP error code 429 as reported by daemon.log. This means Too many requests, rate limited. Im not sure if the quota checking actually understands and adjusts accordingly. Given the amount of errors on this in my test setup, I tend to believe that ioc_ybridanalysis.py just yolo checks any amount of hashes and hopes for the best.

1. Output from daemon.log on SSL issue:

<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title>www.hybrid-analysis.com | 525: SSL handshake failed</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="https://github.com/cdn-cgi/styles/main.css" target="_blank" rel="nofollow" />


</head>
<body>
<div id="cf-wrapper">



    <div id="cf-error-details" class="p-0">
        <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-8">
            <h1 class="inline-block sm:block sm:mb-2 font-light text-60 lg:text-4xl text-black-dark leading-tight mr-2">

              <span class="inline-block">SSL handshake failed</span>
              <span class="code-label">Error code 525</span>
            </h1>
            <div>
               Visit <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" target="_blank" rel="noopener noreferrer">cloudflare.com</a> for more information.
            </div>
            <div class="mt-3">2022-12-05 12:00:16 UTC</div>
        </header>

        <div class="my-8 bg-gradient-gray">
            <div class="w-240 lg:w-full mx-auto">
                <div class="clearfix md:px-8">

<div id="cf-browser-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-browser block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">You</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Browser

  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-cloudflare-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">
    <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" target="_blank" rel="noopener noreferrer">
    <span class="cf-icon-cloud block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
    </a>
  </div>
  <span class="md:block w-full truncate">Amsterdam</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
    <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" target="_blank" rel="noopener noreferrer">
    Cloudflare
    </a>
  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-host-status" class="cf-error-source relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-server block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-error w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">www.hybrid-analysis.com</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Host

  </h3>
  <span class="leading-1.3 text-2xl text-red-error">Error</span>
</div>

                </div>

            </div>
        </div>

        <div class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
            <div class="clearfix">
                <div class="w-1/2 md:w-full float-left pr-6 md:pb-10 md:pr-0 leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What happened?</h2>
                    <p>Cloudflare is unable to establish an SSL connection to the origin server.</p>
                </div>

                <div class="w-1/2 md:w-full float-left leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What can I do?</h2>
                          <h3 class="text-15 font-semibold mb-2">If you're a visitor of this website:</h3>
      <p class="mb-6">Please try again in a few minutes.</p>

      <h3 class="text-15 font-semibold mb-2">If you're the owner of this website:</h3>
      <p><span>It appears that the SSL configuration used is not compatible with Cloudflare. This could happen for a several reasons, including no shared cipher suites.</span> <a rel="noopener noreferrer" href="https://support.cloudflare.com/hc/en-us/articles/200278659">Additional troubleshooting information here.</a></p>
                </div>
            </div>

        </div>

        <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">774c9dcd9b650ae0</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
      Your IP:
      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
      <span class="hidden" id="cf-footer-ip">188.166.47.245</span>
      <span class="cf-footer-separator sm:hidden">&bull;</span>
    </span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->


    </div>
</div>
</body>
</html>```
commented

Normally this is tackled by first calling HA to request the remaining quota amount:

RedELK/elkserver/docker/redelk-base/redelkinstalldata/scripts/modules/alarm_filehash/ioc_hybridanalysis.py

Lines 33 to 73 in 5f50fc6

def get_remaining_quota(self):
"""Returns the number of hashes that could be queried within this run"""
url = "https://www.hybrid-analysis.com/api/v2/key/current"
headers = {
"Accept": "application/json",
"User-Agent": "RedELK",
"api-key": self.api_key,
}
# Get the quotas, if response code != 200, return 0 so we don't query further
response = requests.get(url, headers=headers)
if response.status_code != 200:
self.logger.warning(
"Error retrieving Hybrid Analysis Quota (HTTP Status code: %d)",
response.status_code,
)
return 0
api_limits_json = response.headers.get("api-limits")
api_limits = json.loads(api_limits_json)
# First check if the limit has been reached
limit_reached = get_value("limit_reached", api_limits, False)
if limit_reached:
return 0
# Extract the limits and usage
limits_minute = get_value("limits.minute", api_limits, 0)
limits_hour = get_value("limits.hour", api_limits, 0)
used_minute = get_value("used.minute", api_limits, 0)
used_hour = get_value("used.hour", api_limits, 0)
remaining_minute = limits_minute - used_minute
remaining_hour = limits_hour - used_hour
self.logger.debug(
"Remaining quotas: hour(%d) / minute(%d)", remaining_hour, remaining_minute
)
# Return the remaining quota per minute
return remaining_minute

It should still check some of the hashes and stop when the limit is reached:

RedELK/elkserver/docker/redelk-base/redelkinstalldata/scripts/modules/alarm_filehash/ioc_hybridanalysis.py

Lines 104 to 113 in 5f50fc6

# Get the remaining quota for this run
remaining_quota = self.get_remaining_quota()
ha_results = {}
# Query HA API for file hashes
count = 0
for md5 in hash_list:
if count < remaining_quota:
# Within quota, let's check the file hash with HA
ha_result = self.get_ha_file_results(md5)