Idea: Allow distinguishing between private and public state

[Isak Sky]

Sometimes some state is needed in components for efficiency, being able to react to external events, or for being able to properly shut down later. But even though the state is needed, it doesn't necessarily make sense to expose this state to other components. Exposing the additional state would only open opportunities for mistakes, and/or make the component clunkier to use.

Example

Let's say I create a component that on start, creates a db object, and and some object that will periodically mutate it. For the private state, I would then hold on to a mutable db object, and a timer-type object (in order to be able to use it in ":halt").

But it would 0 sense to expose these things to other components. The only thing they should care about is an immutable view of the current db.

I talked to a few people in Clojurians, and they pointed out that one can solve the problem by convention or other tricks, and that makes sense. In some cases, it can also be solved by using "higher-order-components" , e.g., duct.scheduler.simple. Thought I'd share this idea anyway, in case, in case you think it makes sense:

;; In integrant, new record
(defrecord State [private-state public-state-fn])

;; Implementation changes...

In client code:

(defmethod ig/init-key ::service [_ _]
  (ig/map->State
    :private-state {:exec-service  (create-exec-service) :datascript-conn (create-conn) } ;; Constructors omitted
    :public-state-fn
    (fn [private-state]
      ;; Returns an immutable view in this case. 
      ;; A function that returns this (requiring 0 arguments) is what would get exposed to other components.
      @(:datascript-conn private-state)))

  (defmethod ig/halt-key! ::service [_ private-state]
    (.stop (:exec-service private-state))
    private-state)

In the init-key, if people don't return this new State record, it would just work like it does now. (That is, private state is always the same as public state, and doesn't require an invocation to access).

[James Reeves]

An alternative implementation is to have some sort of post-ref transformation that's applied after keys have been referenced.

(defmethod ig/init-key ::service [_ _]
  {:exec-service (create-exec-service) :datascript-conn (create-conn)})

(defmethod ig/deref-key ::service [_ {:keys [datascript-conn]}]
  @datascript-conn)

(defmethod ig/halt-key! ::service [_ {:keys [exec-service]}]
  (.stop exec-service))

Let me think about this.

[James Reeves]

I've added a resolve-key method to Integrant that works in the same way as deref-key in my previous command. It's currently released as 0.8.0-alpha1.

As an alpha release, this functionality may change if I discover there's a problem with it, but if you want to try it out, please let me know how you get on with it.

[Isak Sky]

Nice! I gave it a shot, and I found one bug so far:

It works when initially loading state, but after calling (reset), the private state is passed to the consumer components, instead of the thing returned from resolve-key.

One other thing is that with the current approach, it isn't possible to update the object client components see without restarting the system. So the current approach would work if one for example returned an object that dereferences private state, e.g.,:

(reify SomeImmutableProtocol
   (nth [_ n] (aget (:my-array private-state) n)
   (count [_] (alength (:my-array private-state)))

But if you wanted to return a snapshot (i.e., something that cannot change, like how the above object could return different lengths at different points in time) instead, you wouldn't be able to (*). (An example would be returning @my-datascript-conn.)

Technically it is not true that you wouldn't be able to pass different objects at different points in time to client components, because you could just return (reify IDeref [_] ...) from resolve-key, though here it seems like you are encouraging approach 1 over approach 2, though I think approach 2 is the more robust and idiomatic in Clojure.

What do you think about requiring a deref (or 0 arity function invocation) on the value returned from resolve-key in client components before use?

[James Reeves]

It works when initially loading state, but after calling (reset), the private state is passed to the consumer components, instead of the thing returned from resolve-key.

Ah, right. The resume function is missing the extra resolve argument. That should be an easy fix.

One other thing is that with the current approach, it isn't possible to update the object client components see without restarting the system.

Sorry, I don't understand what you mean by this. Could you explain your use-case?

[Isak Sky]

My use case is to load a ton of data into an in-memory db, and then periodically update it incrementally from a standard relational database. I then want to expose a read-only view of the current value of this in-memory db to other components.

With the current approach, I can only do one of these things:

1. Expose a read-only view of the initial value.
2. Return an object that proxies read-type calls to an object that may change from under me. (Not idiomatic, hard to reason about)
3. Return something like (reify IDeref (deref [_] (get-snapshot private-state)) from resolve-key, then deref the input to client components from this component before use

I think 3 is probably about as good as it gets from the client component point of view, though it could be made better from the source component point of view.

So far you've solved the private-state vs public state problem, though not the snapshot-can-change-over-time problem. Not sure if you agree with me that the latter is an important use case.

[James Reeves]

Thanks for explaining. I don't think this is a use-case that's tied to Integrant in particular. Integrant handles initiating a dependency graph of components, but once the system is started it has little to no involvement. Even halting is something of a concession to REPL-based development, since in production you can't guarantee halt! will ever be called on the system.

If you want something that looks like state, either wrap it in a IDeref, as you pointed out, or supply an atom that's updated with a new database snapshot whenever the data is refreshed.

[Isak Sky]

Ok, I understand. Sounds good.

[James Reeves]

The resume issue should be fixed in 0.8.0-alpha2.

[Isak Sky]

Great, works now 👍 .

I see that if I fetch an item from the system manually when playing at the REPL, I get the private state instead of the resolved value. Is there a way to get the resolved value when working at the REPL (without creating a new component)?

[James Reeves]

No, and I think the system should contain the "private" state. I think it's one thing to change what is passed via a reference, but quite another to hide the state entirely.

If you want to make it clear, you could put all the public state under a :public key, and all the private state under :private, then you could write (-> system :foo :public) to get the public reference.

[Isak Sky]

@weavejester - Ok, thanks.