UserStorage Project

The goal of this project is to create an easily configured distributed application that has open WCF API and communicates its state through the network.

UserStorage service is a simple service that stores user records and provides an API for managing user records and searching. It is possible to run several instances of this service and share user records between them. The only instance that allows READ and WRITE operations is called MASTER NODE. The other instances allow only READ operations. Those are known as SLAVE NODES. The only one READ operation in API is SEARCH, and there are two WRITE operations - ADD and REMOVE. That means UserStorage service can operate in two modes - MASTER and SLAVE. Responsibilities of the service in MASTER mode includes spreading the changes to all services that operate in SLAVE mode.

In other words, MASTER NODE accepts READ (SEARCH) and WRITE (ADD/REMOVE) operations, changes its state, and sends an update to all SLAVES NODE that accepts only READ (SEARCH) operations. If a client sends WRITE request to a SLAVE NODE, the node replies with an error.

Described approach when MASTER NODE owns original data and other SLAVE NODES have only the copy is known as MASTER-SLAVE data replication. Possible solutions here are:

MASTER NODE sends updates to all SLAVE NODES by himself.
SLAVE NODES sends a request to MASTER NODE and MASTER replies with a bunch of updates.
Other...

We recommend using the first approach, because we think that this solution is simpler that others.

Also, a MASTER NODE has a persistent storage for user record information when the application is not working. SLAVE NODES have only in-memory storage, and they do not save its state when they are not running. A persistent storage uses the file system to save user records when an application is shutting down and load them when it starts. A good question here is how to initialize the internal state of a SLAVE NODE when an application starts. The answer to this question is a part of the architectural design of this project.

A MASTER NODE sends updates to SLAVE NODES using TCP as a transport channel and internet sockets as endpoints.

The one main thing about this project is that the final application should be configurable, and all application settings should be placed in App.config file. SLAVE NODE is the same application as a MASTER NODE except differences in application configuration file.

Template

In the UserStorage folder you can find a solution template that you can use for building your own application. Let's take a look at the C# projects in the folder:

UserStorageApp - a console application project with predefined App.config. This project should not contain any service related code, only initialization and configuration logic. The configuration file has a custom section that is named serviceConfiguration. This section is for defining services configuration and settings. Visual Studio also provides IntelliSense support for this section because the section schema is defined in ServiceConfiguration.xsd file.
UserStorageServices - a class library project for all service related code.
UserStorageServices.Tests - a class library project with all unit tests for service related behavior.
ServiceConfigurationSection - a class library project that stores classes for handling serviceConfiguration custom section in App.config.
UserStorage.Diagnostics - a class library project that stores WCF contracts for the special DiagnosticsService that monitors current state of other services.
UserStorageMonitor - a console application project that access DiagnosticsService and prints service state.

UserStorage service operates over an entity that describes a user and has relevant name - User class in UserStorageServices project. This class is pretty simple, and has FirstName, LastName and Age fields only.

class User
{
	public string FirstName { get; set; }

	public string LastName { get; set; }

	public int Age { get; set; }
}

UserStorageServices project also has UserStorageService class that is a template for UserStorage service you will be working with.

It is worth mentioning that this code is only the initial template - you are allowed not only to add new code, but also the code refactor it in a way you like.

We encourage you to practice TDD and actively use Git during this exersise. Here are some principles that might be useful for you:

Prepare

Create a new repository on github. Move all content of the master branch in this repository to your new repository.
Install StyleCop or Visual StyleCop. Open UserStorage solution and run StyleCop to check your code and to make sure that there are no code issues.
Check unstaged files in your repository.

$ git status
On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)
...

Add files to the staging area. Check status of staged files.

$ git add *
$ git status
...

Review changes using git diff. (Notice that git diff doesn't return any changes anymore.)

$ git diff
(no output)
$ git diff --staged
(changes output)

Commit and publish all changes. Check status.

$ git commit -m "Add UserStorage template."
...
$ git status
On branch master
nothing to commit, working directory clean

Edit README.md and mark all checkboxes in this section. Check status and review changes. Commit changes.

$ git status
$ git diff --staged
(no output)
$ git diff
(changes output)
$ git add *.md
$ git status
$ git diff
(no output)
$ git diff --staged
(changes output)
$ git commit -m "Mark completed items."
[master ...] Mark completed items.
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git status
On branch master
nothing to commit, working directory clean

Publish changes to github.

$ git push

Now you have the initial version of your repository uploaded to the github.

Step 1 - Service

The class diagram below shows the current relationship between Client and UserStorageService classes.

Create a new branch with name "step1", and switch to this branch. Make sure that you are on "step1" branch before continue.

$ git checkout -b step1
$ git branch
  master
* step1

Add a new Id field to the User class. Use System.Guid as a field type. The field value should uniquely identify a user in the storage. Review changes. Commit changes.
Add an internal storage to UserStorageService class. Consider collections from System.Collections.Generic namespace. A new identifier should be populated and assigned to each new entity before adding it to a collection. Implement Count property getter to return the amount of users in the storage. Review and commit.
UserStorageService class contains Add() method that adds a new user to the storage. The method has one guard clause and one validation statement. Tests for the methods of the class are located in UserStorageServiceTests class. Think what more validation rules you can add here. Add tests for those rules, and then write code to implement them.

Test-First: add use cases in form of tests to UserStorageServiceTests class (red tests), and only then add implementation to the Add method (make your tests green).

Review and commit.

Test-First: add use cases (red) and then add an implementation for Remove method (green). Review. Commit.
Test-First: add use cases (red) and then add an implementation for Search method (green). Use cases:
- Search by FirstName.
- Search by LastName.
- Search by Age.

Review and commit.

Add a new bool field IsLoggingEnabled to UserStorageService class, and add logging functionality to Add method:

if (IsLoggingEnabled)
{
    Console.WriteLine("Add() method is called.");
}

Add logging to Remove and Search methods too. Review and commit.

Run StyleCop to make sure the code you have added fits defined code standards. Fix all code issues StyleCop identified. Review and commit.

$ git status
$ git diff
$ git commit -m "Fix StyleCop issues."

Mark all completed items in README.md. Review and commit.
Publish "step1" branch to remote branch on github.

$ git push -u origin step2

Switch to master branch. Merge "step1" branch into master. Publish changes to master branch on github.

$ git checkout master
$ git branch
* master
  step1
$ git merge --squash step1
$ git status
$ git diff --staged
$ git commit -m "Add implementation for Add, Remove and Search methods. Add logging."
$ git log --oneline
$ git status
On branch master
nothing to commit, working directory clean

Step 2 - Extract

The class diagram below shows the application state after all refactorings in the current step.

Create a new branch with name "step2", and switch to this branch.

UserStorageService is responsible not only for storing user records, but also for generating new identifier and data validation. You will apply Single Responsibility Principle (SRP) to the service in the next two refactorings.

Extract Class refactoring: extract strategy of generating new user identifier into a new class.
- Create a new interface in UserStorageServices project, give it a meaningful name.
- Test-First: create a new class in UserStorageServices project that implements the interface, and move your code (generation of a new identifier) from UserStorageService class to your new class.
- Modify UserStorageService to create a new instance of your new class, and use it to generate an identifier when adding a new user.

Run all tests to make sure that UserStorageService works as expected.

Review and commit.

Extract Class: extract strategy of validating user data when adding a new user to the storage.
- Create a new interface in UserStorageServices project, give it a meaningful name.
- Test-First: create a new class in UserStorageServices project that implements the interface, and move your code (validation of the user data) from UserStorageService class to your new class.
- Modify UserStorageService to create a new instance of your new class, and use it to validate a user data when adding a new user.

Run all tests to make sure that UserStorageService works as expected.

Review and commit.

Extract Interface: extract an interface for the UserStorageService class.
- Create a new interface IUserStorageService in UserStorageServices project, give it a meaningful name.
- Add all public methods and properties from UserStorageService class to your new interface.
- Refactor _userStorageService field in Client class: change the field type to your new interface.
- Refactor constructor in Client class to use Constructor Injection to set userStorageService field.

Run tests, review and commit.

Configure logging using App.config.
- Refactor your UserStorageService class to use boolean switch instead of IsLoggingEnabled property.
- Use enableLogging boolean switch that is already added to your App.config.
- Remove unnecessary IsLoggingEnabled property.
- Run application with enableLogging switch enabled and disabled to make sure that logging works properly.

Run tests, review and commit.

Run StyleCop. Fix issues. Commit.
Mark. Commit.
Publish "step2" branch to github.
Switch to master branch. Merge "step2" branch into master. Publish changes to master branch on github.

Step 3 - Compose and decorate

New branch "step3".
Composite validator.
- Refactor your class that validates user data to extract validation logic for each validation rule to a separate class.
- Use Composite design pattern to create a composite validator.

Run tests, review and commit.

Validation exceptions. Create a custom exception for each validation case. Examples: FirstNameIsNullOrEmptyException, LastNameExceedsLimitsException, AgeExceedsLimitsException. Each validator rule class should throw its own exception. Modify tests.

Run tests, review and commit.

Extended search functionality. Add new functionality to your Search method for supporting these use cases:
- Search by FirstName and LastName.
- Search by FirstName and Age.
- Search by LastName and Age.
- Search by FirstName, LastName and Age.

Add new tests. Run tests, review and commit.

Extract logging functionality.
- Extract Class: extract logging functionality to a separate class that inherits IUserStorageService interface.
- Use Decorator design pattern to create a log decorator.
- Make UserStorageServiceDecorator class abstract.
- Modify your application code to create a new log decorator and pass it to the Client class instead of UserStorageService class.

Run tests, review and commit.

Refactor UserStorageServiceLog to use Trace Listeners to log all UserStorageService method calls.
- Configure TextWriterTraceListener by using a configuration file.
- Replace Console.WriteLine method calls with appropriate Debug or Trace methods.
- Add more listeners to the App.config to support console, XML and CSV output.
- Comment CSV and XML listeners before commit.

Run tests, review and commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step3". Merge "step3" into master. Publish.

Step 4 - Master-Slave

New branch "step4".
Test-First: make UserStorageService work in two modes - MASTER AND SLAVE NODE.
- Add a new UserStorageServiceMode enum with two values - MasterNode and SlaveNode.
- Extend UserStorageService class constructor with new parameters - UserStorageServiceMode and IEnumerable<IUserStorageService>.
- If the service works as the MasterNode it should allow Add, Remove and Search method calls.
- If the service works as the SlaveNode it should allow only Search method call. When Add or Remove is called the service should throw NotSupportedException.
- If the service works as the MasterNode it should make a call for Add and Remove method to all dependent SLAVE NODES with the same parameters.

The sequence diagram below shows how MASTER NODE communicates with SLAVE NODES when the Add method is called:

Add new tests first, then add implementation to UserStorageService. Run tests, review and commit.

Change the code of your application to have the MASTER NODE that is connected with two SLAVE NODES.
Add a new interface INotificationSubscriber and implement Observer design pattern as shown on the class diagram below:

Run tests, review and commit.

Replace Conditional with Polymorphism for UserStorageService class. Use generalization-related refactorings to do that.
- Rename UserStorageService class to UserStorageServiceBase class, and make it abstract.
- Create two derived classes - UserStorageServiceMaster and UserStorageServiceSlave.
- Move code behavior that is specific to MASTER NODE to UserStorageServiceMaster class.
- Move code behavior that is specific to SLAVE NODE to UserStorageServiceSlave class.
- Add a new property ServiceMode to IUserStorageService interface with UserStorageServiceMode type, and implement it in both derived classes.
- Refactor your code according to the class diagram below:

Run tests, review and commit.

Replace Observer design pattern with C# events. Use C# Event Implementation Fundamentals, Best Practices and Conventions article to avoid the common mistakes when implementing that task.

Refactor tests, run tests, review and commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step4". Merge "step4" into master. Publish.

Step 5 - Persistence

New branch "step5".
Test-First: create a new interface IUserRepository and two new classes:

There's an inconsistency in the diagram above - the IUserRepository interface has three methods Get, Set, and Query. Add also Delete method in this interface.

Run tests, review and commit.

Refactor your code.
- Extract Method: extract all code in UserStorageService class that access an internal user collection to private method with Get, Set and Query method names.
- Move Field: move a user collection from UserStorageServiceBase to UserMemoryCache class.
- Move Method: move your new private Get, Set and Query methods to UserMemoryCache class, and make them public.

Add new tests, run all tests, review and commit.

Modify UserMemoryCacheWithState:
- Stop() should save repository state to the disk file.
- Start() should load respository state from the disk file.
- Client should call Start() method before making any calls to IUserStorageService to load the repository state.
- Client should call Stop() method after all calls to IUserStorageService to save the repository state.
- Use "repository.bin" as a file name.
- Use BinaryFormatter to save the repository user data in binary format (binary serialization).

Add new integration tests (save to a file, load from a file).

Run all tests, review and commit.

Use ConfigurationManager.AppSettings to store a name of repository user data file. Use the setting value to configure UserMemoryCacheWithState class from the outside.

Run tests, review and commit.

Strategy: extract serialization functionality as a strategy, and a new strategy to support XML serialization using XmlSerializer.

Add new tests, run all tests, review and commit.

Give your own meaningful names to UserMemoryCache and UserMemoryCacheWithState classes.

Refactor, run tests, review and commit.

Add implementation that will allow UserStorage services save the last generated identifier to continue generating after shutdown.

Refactor, run tests, review and commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step5". Merge "step5" into master. Publish.

Step 6 - Refactor

New branch "step6".
Refactor files in UserStorageServices project: move interfaces and classes related to repository functionality to a separate folder.

Run tests, review and commit.

Refactor files in UserStorageServices project: move interfaces and classes related to user storage functionality to a separate folder.

Run tests, review and commit.

Apply "interface segregation principle" (ISP):

Modify tests, run tests, review and commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step6". Merge "step6" into master. Publish.

Step 7 - Notify

New branch "step7".
Add code from this gist to your project.

Run tests, review and commit.

Add new implementation and refactor existed functionality:

Add new tests, run tests, review and commit.

Refactor you code: serialize NotificationContainer to XML and pass it to receiver as a string.

Modify tests, run tests, review and commit.

Composite design pattern: create a composite notification sender to allow MASTER NODE to send notifications to a variable amount of receivers.

Add new tests, run tests, review and commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step7". Merge "step7" into master. Publish.

Step 8 - Application Domains

New branch "step8".
Refactor infrastructure code: each instance of the user storage service class should be activated in a separate AppDomain. Each MASTER and SLAVE NODE should be created in separate application domain.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step8". Merge "step8" into master. Publish.

Step 9 - Configure

New branch "step9".
Use an application configuration file to setup services configuration.
- There is a custom configuration sections in App.config that has "serviceConfiguration" name. Use ConfigurationManager.GetSection method to get configuration as an object.
- There is no support for network communication. Ignore host and port settings.
- Hardcode any custom behavior (if name=="master-us" or if type="UserStorageMaster") if necessary.

var serviceConfiguration = (ServiceConfiguration)System.Configuration.ConfigurationManager.GetSection("serviceConfiguration");

Run StyleCop, fix issues, commit. Mark, commit. Publish "step9". Merge "step9" into master. Publish.

Step 10 - Reflect

New branch "step10".
Refactor validation: create new attribute classes and use reflection to validate input from the client like in this example:

class User
{
    [ValidateMaxLength(20)]
    [ValidateNotNullOrEmpty]
    [ValidateRegex("([A-Za-z])\w+")]
    public string FirstName { get; set; }

    [ValidateMaxLength(25)]
    [ValidateNotNullOrEmpty]
    [ValidateRegex("([A-Za-z])\w+")]
    public string LastName { get; set; }

    [ValidateMinMax(18, 110)]
    public int Age { get; set; }
}

Refactor existed tests and add new test, run tests, commit.

Create a new attribute and apply it to MASTER NODE and SLAVE NODE classes.

[MyApplicationService("UserStorageMaster")]
class UserStorageServiceMaster
{}

[MyApplicationService("UserStorageSlave")]
class UserStorageServiceSlave
{}

Run tests, review, commit.

When creating a new service instances use this attributes to find a service type in UserStorageServices assembly that matches a value of the type attribute in serviceInstance node in App.config. Use Activator class to create a new service instance. Hardcode is allowed here.
Run StyleCop, fix issues, commit. Mark, commit. Publish "step10". Merge "step10" into master. Publish.

Step 11 - Synchronize

New branch "step11".
The collection in your repository is going to be used in multi-threading environment, and this may lead to concurrency issues. Make your repository thread-safe using lock statement.

Run tests, review and commit.

Make your repository thread-safe using ReaderWriterLockSlim class.

Run tests, review and commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step11". Merge "step11" into master. Publish.

Step 12 - WCF

The diagram below shows the expected application architecture:

New branch "step12".
Design and implement a new WCF service to allow other applications access service endpoints to work UserStorage services. Each UserStorage service should have its own WCF service for handling incoming requests. Use apiPort service parameter in App.config and "http://localhost:apiPort/userStorage" template for WCF service endpoint. You can use DiagnosticsService as an example, just notice that DiagnosticsService works as a singleton and configured using App.config file.

Run tests, review, commit.

Create a new console application and connect to WCF services for MASTER and SLAVE NODES. You can use UserStorageMonitor as an example.

Run tests, review, commit.

Implement MyDiagnostics service to show actual status of all loaded services online.

Run tests, review, commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step12". Merge "step12" into master. Publish.

Step 13 - Network

New branch "step13".
Replace functionality of notification receivers and senders to allow them to communicate over the network using TCP protocol:
- For MASTER NODE - send update notifications to all registered SLAVE NODE endpoints.
- For SLAVE NODE - listen to endpoint and receive update notifications from MASTER NODE.
- Note: Use NetworkStream, TcpClient and TcpListener or Socket to establish communication channel between nodes.
Use host and port settings from an application config to setup notification senders and receivers for all services.
Run StyleCop, fix issues, commit. Mark, commit. Publish "step13". Merge "step13" into master. Publish.

Step 14 - Checkpoint

New branch "step14".
Remove any hardcode that was added on the previous steps.
Review the project codebase with criteria that are mentioned in presentation "Writing High Quality Code in C#". Fix and refactor if necessary.
Run StyleCop, fix issues, commit. Mark, commit. Publish "step14". Merge "step14" into master. Publish.

Step 15 - Cache

New branch "step15".
Apply Decorator design pattern to repository:

Run tests, review, commit.

Implement UserRepositoryDelayer - this class should wait for a timeout before calling the next repository in the decorator chain. The goal of this class is to emulate slow storage.

Run tests, review, commit.

Implement UserRepositoryCache using MemoryCache class. This class should store a user object in a cache before it will expire (use expirationInterval).

Run tests, review, commit.

Create a chain of decorators: UserRepositoryCache->UserRepositoryDelayer->UserRepositoryCache(WithState). Test the cache behavior.

Run tests, review, commit.

Run StyleCop, fix issues, commit. Mark, commit. Publish "step15". Merge "step15" into master. Publish.

YauheniZmitrovich / user-storage-project

UserStorage Project

Template

Prepare

Step 1 - Service

Step 2 - Extract

Step 3 - Compose and decorate

Step 4 - Master-Slave

Step 5 - Persistence

Step 6 - Refactor

Step 7 - Notify

Step 8 - Application Domains

Step 9 - Configure

Step 10 - Reflect

Step 11 - Synchronize

Step 12 - WCF

Step 13 - Network

Step 14 - Checkpoint

Step 15 - Cache

About

Languages