[BUG] Shodan`block in request` error
ssnkhan opened this issue · comments
Describe the bug
I am beginning to see several of these errors on each run. I have checked my log file, but the error does not seem to related to a specific rule.
*A* `Shodan::Error` *occured in background*: The search query was invalid.
```/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:41:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:45:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:60:in `block in search'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'```
*A* `Shodan::Error` *occured in background*: The search query was invalid.
```/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:41:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:45:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:60:in `block in search'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'```
*A* `Shodan::Error` *occured in background*: The search query was invalid.
```/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:41:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:45:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:60:in `block in search'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'```
*A* `Shodan::Error` *occured in background*: The search query was invalid.
```/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:41:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:45:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:60:in `block in search'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'```
*A* `Shodan::Error` *occured in background*: The search query was invalid.
```/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:41:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:45:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-3.10.1/lib/mihari/analyzers/shodan.rb:60:in `block in search'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'
/var/lib/gems/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/core_ext/range/each.rb:9:in `each'```
Steps to reproduce
Unable to isolate to specific rule, as my log files suggest rules work sometimes, but not other times. I can see the The search query was invalid. but can't seem to correlate this back to a rule.
Expected behavior
No errors.
Actual behavior
Generating errors.
Screenshots
N/A.
System Information:
- OS: Ubuntu
- Ruby version: 2.7.0
- Mihari version: 3.10.1
Additional context
@ssnkhan The search query was invalid. is an error message from the Shodan API.
Probably your query in a rule is not well escaped.
Not sure -- when I check the logs, I can see the errors are generated in entirely different sections of my rule files. Sometimes I see the error once, in one part of my rule file. And sometimes I see like 5x of them in a different part of the rule file. Could something in the response be causing the issue?
Again, the error is caused by the Shodan API. Probably the Shodan API returns API response inconsistently.
So I guess you will get the same error in the Shodan web UI. (I think you need some tries to confirm that)
Thank you -- adding some UTF declarations/exports to my bash file solved it:
#encoding: utf-8
export LANG="C.UTF-8"
export LC_ALL="C.UTF-8"