GeoIP crashes when looking up for IP

Question

GeoIP crashes when looking up for IP

srilumpa opened this issue 11 years ago · comments

Hi,

I have a weird behaviour when using the GeoIP::country method: when trying to geolocate some address, through a script, a resque job or the IRB console, a crash occurs kicking me out the IRB console.

I wasn't able to find out exactly why. All I know is that the crash occurs while calling the IPSocket::getaddress method in geoip.rb on line 508. I'm not sure why as I logged requests to this method and retried the requests with the failed addresses but I wasn't able to reproduce the crash.

All I got is the core dump, but I'm not sure I can easily send you about 2500 lines of debug messages. Here is at least the backtrace of the crash

-- Ruby level backtrace information ----------------------------------------
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/bin/rake:23:in `<main>'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/bin/rake:23:in `load'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/bin/rake:33:in `<top (required)>'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:75:in `run'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:165:in `standard_exception_handling'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:78:in `block in run'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:100:in `top_level'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:115:in `run_with_threads'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in `block in top_level'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in `each'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:149:in `invoke_task'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:161:in `invoke'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:168:in `invoke_with_call_chain'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:175:in `block in invoke_with_call_chain'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:231:in `execute'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:231:in `each'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:236:in `block in execute'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:236:in `call'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:134:in `work'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:134:in `loop'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:143:in `block in work'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:266:in `fork'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:266:in `fork'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:146:in `block (2 levels) in work'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/worker.rb:195:in `perform'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:145:in `perform'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:145:in `call'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:133:in `block (2 levels) in perform'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-job-stats-0.3.0/lib/resque/plugins/job_stats/duration.rb:24:in `around_perform_job_stats_duration'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:133:in `block (3 levels) in perform'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:133:in `call'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:137:in `block (2 levels) in perform'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-lock-timeout-0.4.1/lib/resque/plugins/lock_timeout.rb:263:in `around_perform_lock'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/resque-1.24.1/lib/resque/job.rb:138:in `block (3 levels) in perform'
/var/www/my_app/releases/20130809163951/app/workers/fetch_alerts.rb:33:in `perform'
/var/www/my_app/releases/20130809163951/app/models/xxx/source.rb:93:in `import'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence.rb:241:in `create'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/threaded/lifecycle.rb:173:in `_creating'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence.rb:246:in `block in create'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence.rb:80:in `save'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence.rb:51:in `insert'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence/operations/insert.rb:26:in `persist'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence/insertion.rb:23:in `prepare'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/callbacks.rb:130:in `run_callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:414:in `_run__246524166145139314__save__4173549135879904193__callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/persistence/insertion.rb:24:in `block in prepare'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/callbacks.rb:130:in `run_callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:429:in `_run__246524166145139314__create__4173549135879904193__callbacks'
/var/www/my_app/releases/20130809163951/app/models/xxx/alert.rb:71:in `block in <class:Alert>'
/var/www/my_app/releases/20130809163951/app/models/xxx/alert.rb:192:in `geolocate'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/geoip-1.2.1/lib/geoip.rb:183:in `country'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/geoip-1.2.1/lib/geoip.rb:233:in `city'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/geoip-1.2.1/lib/geoip.rb:508:in `lookup_ip'
/var/www/my_app/releases/20130809163951/vendor/bundle/ruby/1.9.1/gems/geoip-1.2.1/lib/geoip.rb:508:in `getaddress'

The geolocate method in xxx/alert.rb looks like this:

  def geolocate
    begin
      source_loc = Dash::Application.config.geoip_db.country(self.sourceip)
      if !source_loc.nil?
        self.source_country_code = source_loc[:country_code2]
        self.source_country_name = source_loc[:country_name]
        self.source_city_name = source_loc[:city_name]
        self.source_latitude = source_loc[:latitude]
        self.source_longitude = source_loc[:longitude]
      end
    rescue
    end

    begin
      destination_loc = Dash::Application.config.geoip_db.country(self.destinationip) # this is line 192
      if !destination_loc.nil?
        self.destination_country_code = destination_loc[:country_code2]
        self.destination_country_name = destination_loc[:country_name]
        self.destination_city_name = destination_loc[:city_name]
        self.destination_latitude = destination_loc[:latitude]
        self.destination_longitude = destination_loc[:longitude]
      end
    rescue
    end
    return nil
  end

Does some of you have an idea about what this could come from? What could I give you to help you finding out what is happening?

Clifford Heath · Answer 1 · Tue Aug 27 2013 06:32:39 GMT+0800 (China Standard Time)

Looks like a problem inside IPSocket.getaddress. This is C code that calls into your operating system's DNS resolver, so there's potentially a lot of complexity hidden down there. Some earlier implementations might not have been thread-safe, but I don't know what Ruby 1.9.1 does. If you know how to use GDB you can investigate it using the core dump, but apart from that there's not a lot I can do,

srilumpa · Answer 2 · Tue Sep 03 2013 16:44:29 GMT+0800 (China Standard Time)

Ok. So, after a little digging I've found out that it comes from our (slightly) overloaded network which generate packet loss that makes DNS requests closing incorrectly.

Sorry for the inconvenience :(

Clifford Heath · Answer 3 · Wed Sep 04 2013 16:24:50 GMT+0800 (China Standard Time)

If bad DNS packets cause your DNS client library to crash, you have a bug in the client library, not in the packets. But I already suspected that, thanks for notifying me.