feature proposed to control prdcr overhead

[Benjamin Allan]

At present we cannot limit the result of

prdcr_add name=X host=X type=active xprt=sock auth=ovis port=411 interval=20000000
prdcr_start name=X

to transport only sets or only streams; currently we always get a download of the set directory and a stream list.

proposal:
For separating stream processing from set processing (which at L2 and higher may be a scalability desire), it would be very nice if we could add another config property on the ldmsd_prdcr controlled by data=<streams,sets,all> (which defaults to 'all' if not given).

It looks like this might be reasonably implemented in prdcr_connect_cb, probably at no additional cost in memory if we add 'unsigned short data_moved' after port_no in struct ldmsd_prdcr.

prdcr_add name=X.streams host=X type=active xprt=sock auth=ovis port=411 interval=20000000 data=streams
prdcr_start name=X.streams

[Benjamin Allan]

@tom95858 @narategithub is there something I'm missing in the code that makes this a completely bonkers idea?

I went to create a streams-only L2 today and found the memory use rather substantial in spite of doing nothing to use sets in the L2 config.

[Tom Tucker]

@baallan, I don't think your goal is bonkers at all. I do think we need to think through exactly what we are trying to accomplish. For example, how do the changes you propose work in the presence of stream load balancing, i.e. which mid-tier ldmsd are responsible for forwarding streams data to the next level. Which clients at the top tier end up processing the streams data.

I think what we may need is a configuration object for streams that lives apart from the producer, but again I think this requires some requirements and some thought.