Vision for Nevermore... fork it?

[Paul Stovell]

I'm working on a PR (#93) and have working async support in Nevermore. As I'm doing it, there's a few things on my mind.

My original goals for Nevermore had been:

• Make something that's general purpose (useful outside of our company)
• Make something that embraces SQL (it's a perfectly good query language for SQL Server)
• Make something that's not too clever or too complex (e.g., not like Entity Framework)

In essence, I wanted Dapper, but with better JSON support. But that was 2016, and things have changed a lot since, and I don't know if any of these hold true anymore.

As I revisit Nevermore this week and hit a bunch of areas that have been built our since I feel like Nevermore has drifted from that. I also find myself wanting to refactor and clean up things, but really don't want to add more of a burden to Octopus developers (nor do I feel confident editing the code in Octopus anymore).

Some examples of what I mean.

Nevermore has its own Query language

Originally Nevermore just provided Query, Where, and OrderBy - and the Where only took a string. This was enough for 90% of the stuff in Octopus, and the rest could be done with a custom SQL query + ExecuteReader, or a SQL view.

Over time this query language got built out. Nevermore now has its own AST, which is quite powerful! It can build some pretty complex queries, with joins and subqueries and so on.

I know SQL pretty well, and I know joins, and I can write joins in SQL from scratch - as could most developers we hire. But I don't think even many existing developers at Octopus today could write one of these queries by hand, or even imagine what the resulting SQL looks like:

var orders = CreateQueryBuilder<IDocument>("Orders");
var customers = CreateQueryBuilder<IDocument>("Customers")
    .Where("IsActive = 1")
    .OrderBy("Created");

var accounts = CreateQueryBuilder<IDocument>("Accounts").Hint("WITH (UPDLOCK)");

var actual = orders.InnerJoin(customers.Subquery())
    .On("CustomerId", JoinOperand.Equal, "Id")
    .On("Owner", JoinOperand.Equal, "Owner")
    .InnerJoin(accounts.Subquery()).On("AccountId", JoinOperand.Equal, "Id")
    .DebugViewRawQuery();

My initial reaction was "why are we going so far to avoid people just writing SQL"? But I think I get it - with spaces, and with permissions in Octopus - there's probably a high need for some kind of composition for queries. And I could imagine code in Octopus that creates a simple query, then passes it to some other part of Octopus that adds conditions etc. to it - and that composing this in the object model is much easier than trying to compose arbitrary strings.

But, there's an awful lot of code behind this - and there's complexity to it. It's definitely more like Entity Framework than like Dapper. I have seen a few smart engineers at Octopus feel like they are struggling when trying to build queries like this.

Related documents

There's a lot of code in Nevermore devoted to maintaining the related documents table in Octopus.

https://github.com/OctopusDeploy/Nevermore/blob/master/source/Nevermore/Util/DataModificationQueryBuilder.cs

This is obviously very Octopus specific - Octofront doesn't need it for example.

In Octofront we added the concept of "hooks" to Nevermore transactions - the ability to add a hook that runs on insert, or update or delete. We do this to update statistics on companies for example. I thought of adding this to Nevermore directly, then changing the code of the related document to use that abstraction (maybe moving that code to Octopus) but it's a lot of work.

Future changes: tiny strings, custom types

I think #94 and #87 add more complexity to Nevermore (particularly around custom types).

Desire to change public API's

I'd really like to clean up a few things in the public API and make Nevermore more approachable for new developers. One example is removing the 85 parameters needed to new up a RelationalStore (I'm glad #94 removes one!)

There are many overloads for methods like Insert:

void Insert<TDocument>(TDocument instance, TimeSpan? commandTimeout = null) where TDocument : class, IId;
void Insert<TDocument>(string tableName, TDocument instance, TimeSpan? commandTimeout = null) where TDocument : class, IId;
void Insert<TDocument>(TDocument instance, string customAssignedId, TimeSpan? commandTimeout = null) where TDocument : class, IId;
void Insert<TDocument>(string tableName, TDocument instance, string customAssignedId, string tableHint = null, TimeSpan? commandTimeout = null) where TDocument : class, IId;

Some of these are also inconsistent - e.g., you can Insert with a specific table name, but you cannot Update to a specific table name.

Adding "async" versions of each will double the number.

I'd like to collapse them into:

void Insert<TDocument>(TDocument instance, InsertOptions options = null) where TDocument : class, IId;

Where InsertOptions contains those options which are less frequently used. Then adding async support is just adding one extra method.

Changes like this to the API seem necessary to try to simplify the code, but introduces more burden on Octopus developers to migrate to them.

Summary

I think where I'm getting to with all of this is that:

1. I think Nevermore has become more complex
2. I think it has become more specific to Octopus
3. It's likely to continue to become more complex and more Octopus specific
4. I feel a bit stuck on whether to (a) live with it, (b) send PR's to refactor some things around or (c) fork it

I suppose the bigger question is: what's our vision behind Nevermore now? Are we OK with it having a higher learning curve and complexity for Octopus developers? Do we want it to be specific to Octopus or keep it general-purpose? Do we want it to have a life outside of Octopus?

[Robert Wagner]

The InnerJoin/On syntax isn't new, it's been there as long as I can remember and as long as Nevermore has been open source. I don't like it either and we've been working on removing them where we can. There are only 20 occurrences of InnerJoin in Octopus Server. I'd be fine with removing it.

There's a lot of code in Nevermore devoted to maintaining the related documents table in Octopus. Octofront doesn't need it for example.

The implementation is generic through and can be used by other systems for the same reason Octopus uses it, like %|projects-1|% queries are super slow. Octofront might need it if it had to do lookups like that in big tables.

Do we want it to have a life outside of Octopus?

Based on past evidence with OctoDiff, Octostache, etc there is little appetite internally to pro-actively maintain it unless it's directly related to Octopus needs. If it's to have a life of it's own, we need to find an owner, internally or externally who has the time to invest in maintaining it.