Giters

jbogard / bulk-writer

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Evaluate support for IAsyncEnumerable

goodmanmd opened this issue · comments

commented

BulkWriter and EnumerableDataReader could potentially benefit from accepting IAsyncEnumerable instead of IEnumerable. Evaluate its use in the project.

commented

Will this be additive? or a breaking change?

commented

At present this does not look possible. The current non-async code relies on implementing EnumerableDataReader to convert the input IEnumerable into something the SqlBulkCopy class can work with -- an IDataReader (other possibilities are DataRow[] and DataTable). Unfortunately, none of these options expose methods that are async aware (neither in System.Data.SqlClient nor Microsoft.Data.SqlClient).

We might be able to introduce an adapter to convert IAsyncEnumerable -> IEnumerable and thus trivially expand the interface to support IAsyncEnumerable, but I suspect this will add complexity for no gain.

Side note: a feature request has been made for Microsoft.Data.SqlClient to expand the SqlBulkCopy interface to accept an IEnumerable. If that were to be implemented, that would seem to open the door for IAsyncEnumerable for sake of parity. dotnet/SqlClient#142

commented

@goodmanmd and @ChrisMissal I spent a little time looking at this, and I do think it's possible without much work at all. I forked the repo and put together a little POC for it.

The main changes are:

https://github.com/michaelnero/bulk-writer/blob/master/src/BulkWriter/Internal/AsyncEnumerableDataReader.cs

https://github.com/michaelnero/bulk-writer/blob/master/src/BulkWriter/BulkWriter.cs#L75

https://github.com/michaelnero/bulk-writer/blob/master/Demo/Program.cs#L61

I didn't put the compiler directives in to make sure the package can target anything other than netstandard21, but let me know if you think this will work, and I'll put together a proper PR for it.