Provide pre-configured facades for common objects, such as hash tables

[Pierre Neidhardt]

Agents and facades are a great abstraction for concurrent access to any kind of data.
It'd be great to provide an abstraction to commonly used data, such as hash tables and maybe lists, plists, vectors, etc.

Example:

(setf *facade* (make-facade-hash-table))
  
; then

(facade-get *facade* INDEX)
(facade-set *facade* INDEX VALUE)

This could be done pretty cleanly and safely by declaring the facades in separate packages.

Thoughts?

[Manfred Bergmann]

Yeah, good idea.
If we can carve out a nice and simple interface for this.
But I think it should be in a companion system/asd that can be added as separate dependency, which also has it's own tests, etc.

[Pierre Neidhardt]

Some random thoughts:

1. All structures can be accessed by index (keyword for plists, numeric index for lists, anything for hash tables, slot name for classes, etc.).

2. Most structures are ordered. For these, we can define a settable first method.
   Classes, structs and hash-tables are not ordered.

3. Instead of facade-set as above, we could make facade-get setf-able.

4. Instead of make-facade-hash-table, we could have a make-facade function that takes a type.

5. Can we leverage cl-containers or generic-cl?

[Manfred Bergmann]

Hard for me to imagine how this could look like. I usually try it in form of some code.
Maybe as a start just an INDEX might work. But down the road I'd imagine that array/vector and key based collections probably need different behavior.
A 'setf' in combination with a 'get' (as a place) I think is one of the more awkward things in Common Lisp. I'd keep it simple for now. The setf could be added later as another abstraction, or convenience. (but see below for some other thought).
I think 'make-facade' with a type is OK, since there may be different types. But I'm wondering if 'facade' is too generic. Shouldn't it be visible here that this is an agent?
We can of course use third-party libraries. However, due to dependencies each supported third-party container/collection should have it's own system so that the user can choose one or the other. Maybe support for CL internal collections can be integrated in main system.
I'm wondering (and actually I'd find that cool) if the agent could mimick the API of the individual collection. I.e. for CL hash-table use 'setf' together with 'gethash'. For vector, use 'elt', 'vector-push', etc. For FSet use 'lookup', 'with', etc. so that the user doesn't need to remember a different set of API. Maybe he chose the collection because he already knows it.
Considering this, the constructor could already reflect the type of collection to create, like 'make-hash-table-agent', or 'make-vector-agent'. 'map-agent' for FSet, or so.
What do you think?

[Pierre Neidhardt]

But I'm wondering if 'facade' is too generic. Shouldn't it be visible here that this is an agent?

OK.

for CL hash-table use 'setf' together with 'gethash'

You mean cl-gserver's own gethash? Do you have some pseudo code in mind?

The drawback of the standard lookup functions is that they are not method and thus not extensible. However, we could rely on generic-cl instead and extend its methods.

Finally, no big difference for me between make-hash-table-agent and make-agent 'hash-table. Maybe having a generic make-agent has the benefit of making it extensible, thus allowing the user to wrap their own data structure.

We could always automatically generate make-hash-table-agent in make-agent if it's a macro.

[Manfred Bergmann]

If you need something quick, just implement something around an agent like you think best suites your needs which you can use to try if it actually solves the problem. It should be relatively straight forward.
See in the readme: https://github.com/mdbergmann/cl-gserver#wrapping-an-agent
Or open a discussion in the 'Discussions' section where we can look at code.

While we can continue here to discuss how this could work in a more generic way.

[Manfred Bergmann]

You mean cl-gserver's own gethash? Do you have some pseudo code in mind?

Yes. A gethash function as part of the hash-table agent facade that works the same way as cl:gethash.
gethash should be easy. But not sure how to implement the setf together with gethash in the agent. Because what gethash would return is not necessarily a 'place' as needed by setf. But I'm not sure how to implement setf, never done it.

A gethash could simply be:

(defun gethash (key hash-agent)
  (agt:agent-get hash-agent (lambda (hash-table) (cl:gethash key hash-table))))

Finally, no big difference for me between make-hash-table-agent and make-agent 'hash-table.

OK, that's fine.

[Manfred Bergmann]

I've played a bit with this.
See: https://github.com/mdbergmann/cl-gserver/blob/master/src/hash-agent.lisp

[Manfred Bergmann]

And the tests:
https://github.com/mdbergmann/cl-gserver/blob/master/tests/hash-agent-test.lisp

[Pierre Neidhardt]

Thanks for this prototype! Some feedback: - The `make-hash-agent` `hash-table-keys` argument name is confusing, since hash tables have "keys" as part of their structure. Maybe `hash-table-arguments`? - Should `make-hash-agent` really have a `context` and `dispatcher-id` argument? Alternatively, global parameters could be let-bound, so we could write something like (let ((*context* ...) (dispatcher-id ...)) (make-hash-agent :test 'foo)) Question: Why did you name the test agent `cut`?

[Manfred Bergmann]

• The make-hash-agent hash-table-keys argument name is confusing,
  since hash tables have "keys" as part of their structure. Maybe
  hash-table-arguments?

Yeah, ok. We can name it 'hash-table-arguments'. I though it wouldn't be needed at all passing the hash-table-arguments as &rest but I couldn't make that work.

• Should make-hash-agent really have a context and dispatcher-id
  argument?
  Alternatively, global parameters could be let-bound, so we could write
  something like

(let ((context ...)
(dispatcher-id ...))
(make-hash-agent :test 'foo))

We can also make this using dynamic bindings. The Tasks api also does this. But I believe those 'hash-agents' are not created randomly on the fly to the masses but only in selected places.
The benefit of using dynamic bindings would be that &rest could be used for the hash-table arguments.
However, this assumes that no other configuration options will ever be needed, like agent name, or queue size.
So the more flexibility offers the design as is.

Question: Why did you name the test agent cut?

'cut' is short for 'class under test'. Hmm, somehow I've gotten used to this.

[Pierre Neidhardt]

You could also mix together the &key arguments of `make-hash-table` with :context and :dispatcher-id, since they are unlikely to conflict with any implementation.

[Manfred Bergmann]

You could also mix together the &key arguments of make-hash-table with :context and :dispatcher-id, since they are unlikely to conflict with any implementation.

You mean just have &rest and take from it 'context' and 'dispatcher-id' if it exists but pass rest along to make-hash-table unfiltered?

[Pierre Neidhardt]

Yes. But unfortunately the function arguments would not be explicit (and thus not show up in eldoc). Would be nice if their would be a portable way to capture the lambda list of `make-hash-table` since it varies across implementation. SBCL has: ```

(sb-introspect:function-lambda-list 'make-hash-table)

(&KEY (SB-IMPL::TEST 'EQL) (SB-IMPL::SIZE 7) (SB-IMPL::REHASH-SIZE 1.5) (SB-IMPL::REHASH-THRESHOLD 1) (SB-IMPL::HASH-FUNCTION NIL SB-IMPL::USER-HASHFUN-P) (SB-IMPL::WEAKNESS NIL) (SB-IMPL::SYNCHRONIZED)) NIL ```

[Manfred Bergmann]

Yeah, the explicitness of the parameters meant for the agent (or the lack of that) is an issue.
I'm inclined to keep the API for make-hash-agent as is with renaming 'hash-table-keys' to 'hash-table-args'.
I'm also thinking about explicitly making 'context' required. Because not providing 'context' means creating an agent outside of any actor-system, which means the user must take care of cleaning it up and it will also not be able to use any shared dispatcher. So maybe enforcing a 'nil' here (or a valid context) emphasizes this.

[Pierre Neidhardt]

Sounds good!

[Manfred Bergmann]

Refactored the hash-agent.
I'd create new tickets for implementing this for vector/array, list, alist/plist as separate items.

[Pierre Neidhardt]

Cool, thanks! Should I open the tickets?

[Pierre Neidhardt]

1.7.6 is missing it seems.

[Manfred Bergmann]

1.7.6 is missing it seems.

From readme? Yes. I'll still have to update documentation.

Should I open the tickets?

Please do.

[Pierre Neidhardt]

Typo, I meant "1.7.6 git tag".

[Manfred Bergmann]

Uhm, yes. No tag yet.
Let me finish the documentation today then I'll tag.
I was still a bit unsure if the package is OK where this is in. But I think it is OK.

[Manfred Bergmann]

OK, done.
Tagged and documentation added.