execute leaks memory

[Pistos]

See minimal code example here. As the code loops, the process increases in memory usage.

[James Tucker]

prep_finalizer is subject to the usual failure of defining finalizers. it encloses the statement object that is supposed to finalize, and as such, prevents it from ever being collected.

[Pistos]

Thanks for the insight, @raggi. I locally commented out the prep_finalizer line in rdbi-driver-postgresql, but it seems not to have helped in the case of the sample code.

[James Tucker]

Well, now you're not clearing the results at all, right?

[Pistos]

Makes sense. So will it leak one way or another?

[Pistos]

i.e. is this problem unsolveable within RDBI?

[James Tucker]

@Pistos it requires at minimum the implementat of pre_finalizer in rdbi to be fixed, but also the api to be changed, as the block isn't really all that suitable. The api should probably be something like:

   add_finalizer @pg_result, :clear

[James Tucker]

Without changing the API (but fixing the obvious bug in prep_finalizer) you would need to make a lambda factory in say a class method, something like:

  def self.finalize_result_proc(result)
    proc { result.clear }
  end

and then consume that like:

  prep_finalizer &self.class.finalize_result_proc(@pg_result)

[Mike Pomraning]

add_finalizer @pg_result, :clear

@raggi, yes an approach like this caps memory in @Pistos' example, if we also drop or WeakRef the open_statements tracking in RDBI::Database.

[Mike Pomraning]

FWIW I'd be comfortable with an upon_finalization(obj, method, *other_args_to__send__) for anything that reasonably needs it: Database, Statement, and Cursor. API change, yes, but only for driver authors, and they have to touch their code anyway to avoid this leak.

[Mike Pomraning]

Please see branch sth-leak, which addresses @Pistos' mem leak by:

1. WeakRef'ing certain child handles here and there
2. finalizer'ing Result's @data and secretly orphaned Statement objects from dbh.execute()

It requires the postgresql driver not to call prep_finalizer, for @raggi's cited reasons. This branch also addresses Issue #34 FWIW.

(dbh.open_statements should really be dropped; it's just not useful. We should think hard about whether we really want a last_statement/last_result member, and, if we do, whether we want to keep track of all child handles for all major objects.)

[Pistos]

@pilcrow: So this requires a complementary change in rdbi-driver-postgresql? Thanks for fixing this, BTW!

[Mike Pomraning]

So this requires a complementary change in rdbi-driver-postgresql?

@Pistos, they're a bit orthogonal, but both good ideas. Use of prep_finalizer() ironically prevents pg sth finalization, so commenting it out is necessary to prevent a leak of (libpq?) resources. (I'd guess the ruby libpq-bindings do finalization cleanup correctly.) Use of the current master branch implementation of dbh.execute() easily leaks all sth, so changing its behavior is necessary, too.

I want to make sure I'm right and then mail the list and other driver authors. (Sorry this is taking so long — I'm timesliced rather thinly lately.)

[Pistos]

If there's anything I can help with, let me know. I'm willing to try experimimental branches on a system I have [which is using RDBI].

[Pistos]

FWIW, this has been running in production for 10 days or so now, with no problems, and with nicely reduced memory leakage.

[Mike Pomraning]

Excellent

On Tue, Mar 12, 2013 at 11:05 AM, Pistos [e-mail] wrote:

FWIW, this has been running in production for 10 days or so now, with no
problems, and with nicely reduced memory leakage.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/33#issuecomment-14784484
.

[Mike Pomraning]

@semmons99 will try to smoke rdbi-[oj]dbc in the short term...

[Shane Emmons]

@pilcrow: no problems with odbc/jdbc

[Mike Pomraning]

Fixed in master / 1.1.0

[Pistos]

Thanks!

[Pistos]

So what about rdbi-driver-postgresql? Would something like this fix that side? It's what I've been using in production.

[Mike Pomraning]

So what about rdbi-driver-postgresql? Would something like this fix that side? It's what I've been using in production.

I expect you can omit that entirely as long as your #finish() method clears the pg_result.