Giters

microsoft / Trill

Trill is a single-node query processor for temporal or streaming data.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Is Trill alive?

djl-85 opened this issue · comments

commented

Hi,
I am considering using Trill for stream analytics. But there is something that keeps me unsure if this is the correct library.

I cannot see too much references over the Internet about this library. Also, I read some issues in this repo and nobody answered the developers questions.
Is anyone out there supporting this? The documentation is good enough for starting developing with Trill, but lacks lots of information. For example, I could see some methods, that are not mentioned in the documentation at all. For example, I could not read anything about Pivot. What is that for? (It is just an example).

I hope someone answer.
Thank you.

commented

Please, can someone answer?

commented

I'm guessing that the project isn't going to receive any updates for the foreseeable future. Which is a shame, but the library is pretty mature and is still usable

commented

@peterfreiling or @arunkm can you please sed some light on this?

commented

Hi @wassim-k and @AlgorithmsAreCool,
We want to keep the project active. While we do not have any major features currently planned for Trill, we will fix bugs and minor changes required.
Let me know, if you have any specific questions.

commented

Thanks for that @arunkm :)

commented

Trill has stalled because nobody knows how to use it? I am a seasoned developer and I can't make any sense of this library. Plodding on trying to make head or tails of what seems to be a VERY not-easy library to use.

commented

@johnds1974 No pain, no gain? 😉

It is very tricky to write complex queries but the performance was worth it for our use case

commented

I agree with both of you @johnds1974 abd @AlgorithmsAreCool, while Trill is the most complex library I've had the pleasure of working with, it is complex because it expresses very complex concepts in very few lines of code.

We are using Trill successfully in production in multiple services running complex queries and there have been times where it took me a full day to only write less than 10 lines of code of Trill but I believe it would have taken me much longer with potentially 100s of lines of code do it otherwise.

The reality is, there are no other alternatives in the .NET space (that I'm aware of and I've looked) and once you've invested the time into the library, you'll get it back manifold.

@johnds1974 if you have any particular questions, I'd be happy to help where I can.

commented

@AlgorithmsAreCool @wassim-k Thanks for the replies. I guess I was just getting frustrated by my need to find a good data-analysis capable library without too much of a learning curve.

So the reason why I wanted to try Trill, or MoreLinq (Trill alternative), was my need to analyse/aggregate etc both existing and live time-series data in a way that wouldn't require having to re-run queries over an accumulating dataset.

My requirements:

  1. Queries need to be fast
  2. Aggregations need to be fast
  3. Aggregations/Groupings etc need to update in real-time as more data flows in (no re-querying)
  4. My code needs to react to/be notified that whichever aggregates I have defined over the data have changed.

Where I'm getting stuck is ingress into Trill (a bizare workflow of: source => rx-observable => stream-events => streamable => stream-observable), several jumps/transforms to get to a point (I have not gotten that far) where maybe, just maybe, I can satisfy my requirements above.

Not sure if either of your projects have similar requirements and if so, I'm 100% sure both of you, when first reading the Trill documents must have thought..."whaaaaaaatttt the fffffffffffff.....!!!!?????" :-D

commented

Haha, yep, the PDFs are overwhelming the first few reads.

If all you need to GroupBy a certain key and apply a Scan function over time-series data ordered by time then, for simplicity, LINQ or MoreLinq might be enough.

If you need to apply some windowing functions, Tumbling, Hopping, etc... or execute temporal joins between multiple time-series data sources then Trill is the right tool, especially when dealing with historical data.

For low latency applications, refer to this comment I wrote while trying to figure that out myself.

To avoid re-querying, then you can push existing + live data to a Kafa/Event Hub topic and consume that into Trill while checkpointing the topic periodically along with Trill state checkpoins.
Beware of this gotcha that I had to go through when checkpointing.

And good luck!