Nuget

Storks	Storks Azure Storage

Storks

A .Net library for loading of data and separating properties into storage.
Useful for large messages in queues with limited message sizes

Find the reasoning for this project on my blog:
http://blog.liammorrow.xyz/2017/04/overcoming-azure-storage-queue-message.html

Installation

Get via nuget at:

Core

Contains base interfaces and default implementations using file and in memory data stores
Install via the Package Manager Console
Install-Package Storks

Azure Storage Implementation

Contains an implementation using Azure Blob storage as a data store
Install via the Package Manager Console
Install-Package Storks.AzureStorage

Usage

A common use case of storks is passing large messages or files in an environment which limits message size.
Imagine you have a web service that allows you to convert HTML to PDF using a tool like wkhtmltopdf. Unfortunately, this tool requires the GDI library which many service providers limit access to on server environments. So you need to create a worker that reads from a queue to run the HTML->PDF conversion then reports back.

You write it up but hit a snag. Your queue messages are limited to 64kB! That will work for the majority of your html, but all it takes is one file that is too large to queue. This is where storks comes in.

By using a StoreBackedProperty, you can now effortlessly pass objects as large as you like into your queue!
Your queue message object now becomes this:

class HtmlToPdfMessage{
  public StoreBackedProperty<string> Html { get; set; } 
}

And your push to queue method becomes this:

public async Task PushHtmlToPdfMessage(string html){
   
   // Create a new data communicator.  Here we use a LocalFileDataCommunicator which stores message data on the local HDD
   // The communicator simply gets and retrieves byte data when given a unique ID.
   // The Storks.AzureStorage package implements a data communicator with Azure Blob storage being the backing store
   IStoreBackedPropertyDataCommunicator dataStore = new LocalFileDataCommunicator("path/to/directorystorage");
   
   // Create a new StoreBackedPropertyController.  Alternatively, we could use a DI framework to get it
    var controller = new StoreBackedPropertyController(dataStore);
    var storedData = await controller.StoreValueAsync(html);
    var message = new HtmlToPdfMessage{
      Html = storedData
    };
    
    // Then you'd do your usual message queue operations here with message
}

Then to dequeue simply do the reverse:

public async Task DequeueHtmlToPdf(HtmlToPdfMessage message){
   
   // Create a new data communicator.  Here we use a LocalFileDataCommunicator which stores message data on the local HDD
   // The communicator simply gets and retrieves byte data when given a unique ID.
   // The Storks.AzureStorage package implements a data communicator with Azure Blob storage being the backing store
   IStoreBackedPropertyDataCommunicator dataStore = new LocalFileDataCommunicator("path/to/directorystorage");
   
   // Create a new StoreBackedPropertyController.  Alternatively, we could use a DI framework to get it
    var controller = new StoreBackedPropertyController(dataStore);
    var retrievedData = await controller.GetValueAsync(message.Html);
    
    // Then you'd do your usual message queue operations here with retrievedData
}

Automatic Data Retrieval

Storks integrates with Json.Net to allow for automatic data retrieval when a message is deserialized with JsonConvert.Deserialize.
This is the method which Azure Webjobs use to deserialize queued messages into POCOs.
Simply call the method AutoBindLoadedJson on any IStoreBackedPropertyController to use that controller to deserialize messages. An example can be found in the Storks.Examples.AzureWebJobs project

Custom Data Types

By default, Storks can automatically store string and byte[] StoreBackedProperty types.
When using the default StoreBackedPropertyController implementation, Storks can also handle most POCOs using BSON serialization provided by Json.Net.

If you need to serialize another type, simply implement the IStoreBackedPropertyEncoder interface and register it to your controller with the RegisterEncoder method on your controller. The encoder provides methods to serialize the type into a byte[] and back to your type. Have a look at the StringStoreBackedPropertyEncoder for reference.

Contributing

Fork it!
Create your feature branch: git checkout -b my-new-feature
Commit your changes: git commit -am 'Add some feature'
Push to the branch: git push origin my-new-feature
Submit a pull request

License

Licensed under the MIT license, can be found in LICENSE file

Special thanks:

Icons made by Roundicons from www.flaticon.com is licensed by CC 3.0 BY

Thanks to Appveyor for providing free builds for Open source projects!

LiamMorrow / Storks