Need help with finding aggregates and bounded contexts in our domain
enumag opened this issue · comments
Hi everyone, here is the issue I promised to create in prooph/event-sourcing#79.
Note that I'm going on vacation this saturday and will only have a limited ability to answer questions. Feel free to postpone reading this until August 18th or so when I'll be back to answer anything you might wish to know.
draw.io diagram: https://drive.google.com/file/d/19DbQy1YJuU2klJF9nBa3bGHYI8oVA6UU/view?usp=sharing
EDIT: This might be better quality: https://user-images.githubusercontent.com/539462/43907077-04feb6fc-9bf5-11e8-85f5-f15c5ae20f5f.png
In the draw.io representation I went for something close to an UML diagram, but focusing just on the relations. The diagram represents what a relational schema would look like - in fact this is pretty much what our PostgreSQL projection looks like. I did cut a few less important things here and there such as that each ContactCard hold some addresses, phones, emails etc - these details are not important for this issue in my opinion.
The purpose of this issue is to help my team determine what the aggregates here are and where are the bounded contexts boundaries because based on previous discussions with prooph comminity I don't think our current model is correct. (Which is to be expected since our experience with DDD is limited.) Ideally I'd like to create a case-study based on this to provide an example to others who are new to DDD.
Now to explain the business, what the entities and relation mean and what processes are there.
The business is about providing accountancy services to the owners of large buildings or organizations that take care of the building when each apartment is owned by a different person.
First let's say the database is empty so we need to add some data. The first thing to add is a building, it's entrances (each entrance has a different address) and the apartments in the building. Then we need to add the contracts - who owns and who lives in each apartment. These people need to pay some fees every months which is where our company comes in - to help determine what fees they should pay and observe if they are indeed paying them. When adding a new building, most of the data about apartments, contracts and people are loaded from an external database or an import file.
Next we need to tell the system what fees should the people pay and how to calculate the exact amounts (since they can differ each month but can be calculated). This is represented by the FeeRecipe entity. Each contract usually has around 5-10 FeeRecipes. As you can see in the diagram there is a FinancialAccount entity between Contract and FeeRecipe. Most often each contract only has one FinancialAccount but in some edge cases there can be more. This is required because the Fees on different FinancialAccounts are treated a bit differently in some cases based on the attributes of the FinancialAccount but the details are unimportant.
At the end of each month an automated process takes each (still active) FeeRecipe and generates a new Fee for it for that month. The calculation requires the formula from the FeeRecipe and the current ApartmentAttributeValues on the related Appartment. ApartmentAttributes are properties of the apartment such as bodycount (how many people live in the apartment at the time) which can change every now and then. Each change is represented by a new ApartmentAttributeValue.
Next looking from the other side of the diagram there is an AccountingOrganization. In most cases an AccountingOrganizations is 1:1 with Building but there are exceptions with one AccountingOrganization handling multiple buildings and also exceptions with one building being split to multiple AccountingOrganizations. Therefore we made them completely independent.
FinancialAccountGroup is actually a brand new entity we came up with just this morning after discovering some serious problems in our PaymentAllocations process. Basically it's a group of FinancialAccounts that holds some options how should the Payments be allocated to Fees. FinancialAccountGroup is actually the scope for the PaymentAllocations process - take all not-fully-allocated Payments and FeeItems related to the FinancialAccountGroup and do the process described in the previous issue.
PaymentPreference holds a reference number that we can use to match a given payment from the import to a specific person and FinancialAccountGroup. The process of matching imported payments to people converts UnassignedPayment to Payment.
Finally the ContactCard represents a Person or Company and can be referenced pretty much anywhere in the system. It can be an employee of our company, an owner of an apartment, an organization managing a building, a contractor our company cooperates with etc.
Our struggle mostly comes down to the uncertainty of "how big should an aggregate be" - where the answer is of course "it depends" but I'm unsure "what exactly it depends on". Should each of the entities in the diagram be an aggregate? If not then how far should one aggregate go? Is there some limit to how much data should an aggregate hold or how many different methods, related commands and events it should have? For now most of the things on the diagram are actually aggregates (with a few more aggregates planned to oversee the automated processes) with the structure being split to around 5 bounded contexts. Is that too many?
Another problem is that some parts of the structure are pretty much pure CRUD - Building, Entrance, Apartment, ApartmentAttributeValue, Contract, FinancialAccount, FeeRecipe, ContactCard - all of that is pretty pure CRUD, albait some parts are automated with imports. The reason why we chose to use DDD are the processes around generating fees, finding out which payment was payed by who and which payment pays which fees (the last process is explained a bit more in #6). Also having the entire event sourced history will be beneficial in some of the processes and also other parts that are not yet represented on the diagram (such as calculating penalties for late payments). This puts us in a not very good position where we need to deal with CRUD in ES/CQRS while knowing that ES/CQRS is not really the best thing for CRUD.
To give some numbers a building typically has 1-5 entrances, an entrance can have up to a 100 or so apartments, each apartment has a about 1-3 contracts active at any given time (+ many historical contracts that already ended), and each contract has about 10 fees generated each month.
I didn't write down our current aggregates and bounded contexts here but as said above, most entities on the diagram are in fact aggregates now. Bounded contexts are RealtyRegistration, FeeCalculating, PaymentCollecting, ContactAdministration, Accounting - I feel it's unnecessary to tell you what's where since it's most likely wrong and you can probably imagine what's where anyway.
I have some ideas on my own about where would I split the aggregates now after getting some experience, but I'm not confident enough about it to start refactoring the application that way. And I'm still very unsure about where to split the bounded contexts. Here are some of my current thoughts:
I sometimes feel like a Building should be an aggregate (containing Entrances, Apartments, possibly even Contracts?) but then I remember that one Building can have several hundreds Apartments and it starts to feel too large. Then maybe a Contract aggregate (containing FinancialAccounts, FeeRecipes, Fees and PaymentAllocations) - but some people can own an apartment for decades, sending hundreds of Payments and paying (or maybe not paying) thousands of Fees. And even such giant Contract aggregate still can't generate new Fees on it's own because it doesn't have ApartmentAttributeValues. Should the aggregate also contain Apartments as well then to be able to calculate Fees? But that would make it even bigger.
Now regarding what you told me in prooph/event-sourcing#79 that I should check for existence of related entity IDs not in the command handler but only after storing what happened into EventStore. As you can see, nearly every entity in the system holds an ID of one or more other entities. For example a Contract holds an ID of the Apartment and then up to 4 IDs of ContactCard (SuperiorParty, SubordinateParty + each of them possibly represented by another ContactCard, such as a company owning a building being represented by one of their employees).
It just doesn't feel right to me to deal with non-existent entity as explicit part of the business process as @fritz-gerneth suggested. If used that approach to every such problem it would require me to write a lot of additional code to handle such events and negate anything that was based on that invalid data. There isn't any business to do about it either - the system got an invalid ID either due to some mistake of the client working with it or on purpose from a hacker. It doesn't make sense to ask them for anything, we just need to negate everything that happened since then and I can't help but feel we should have prevented it from happening instead.
Let's say someone added a new Contract with nonexistent ID of the SuperiorParty ContactCard. Then before the error could be discovered they sent commands to create FinancialAccount, FeeRecipe, generate a Fee from the FeeRecipe and to allocate an existing Payment to it. Then the system finds out the Contract should not exist and needs to revert not only all of that but also any PaymentAllocations that happened since then because they should have been allocated differently. Dealing with all that feels like a total overkill for a simple missing ID case. But of course feel free to convince me now that you know the context, I most likely missed a lesson again.
That should be about everything important I can think of right now. Feel free to ask any questions and thanks in advance for any help you can give me. Also, where can I find a donate button?
Also, where can I find a donate button?
There is no donate button ;). The price you have to pay is that you need to share all information publicly, which is often not possible because exposing domain logic does mean that competitors can learn from your domain. Most businesses don't allow that.
Therefor it is hard to find real world domain examples except the ones used in DDD books. But each domain is different so the examples are still only examples.
The second disadvantage is that there is no SLA. We can try to help, if we have enough spare time. The more complex the problem is, the longer it will take to understand the problem and work out a solution. This process might not match with your project deadlines. For example I'm not available the next weeks and when I'm back online I'll have a lot of paid work on my desk which definitely has higher priority.
Having said this, if finding a solution is time critical you might be better of with a paid DDD consultant.
If time is not a problem, we can try to help. For now I'd ping a few people and see if one can take over during my offline time.
The thing is doing all this remote is hard. Let's see if we can still come up with a good solution ;)
Ok made some noise: https://twitter.com/prooph_software/status/1027672367902941184
What we need:
The price you have to pay is that you need to share all information publicly, which is often not possible because exposing domain logic does mean that competitors can learn from your domain. Most businesses don't allow that.
Yeah, I got a permission from my boss. It's fine as long as we don't share source code and not delve too deep in our algorithms but those details are not important here anyway.
It's not really time critical to solve this issue. The time critical thing was to get some understanding how to implement a complicated process which you already helped us with in the previous issue. We'll have to implement those processes first anyway and finish some more features before we can get to any refactoring based on this issue.
This issue is mainly meant for me and my team to learn more about DDD. It's not only about getting some advice for our domain but also about knowing the reasons.
Thanks for any help.
cool, perfect base for a case study
Can you describe for each "CRUD" entity (Building, Entrance, Apartment, ApartmentAttributeValue, Contract, FinancialAccount, FeeRecipe, ContactCard):
- Where do you get the data from?
- Who is responsible for the data?
- How often does the data change?
- What data changes influence the fee generation process and how?
If you can provide some organisational information about your company that would be nice, too (if not possible, because you are not allowed to expose those information, then skip the question):
- Which departments of your company work with the system and what specific part do they use or are responsible for?
- Is only one developer team working on the software or do you have multiple teams? If so, who works on which part?
- Do you integrate with third-party services or provide an API for foreign systems?
Where do you get the data from?
When possible the data are imported from a file (let's say XML) or from an external database. When not possible they are added in manually by our employees.
Later modifications are all done manually as far as I know.
Who is responsible for the data?
Our emloyees, although I'm not sure if that's what you're asking.
How often does the data change?
Not too often. Usually just when an apartment is sold. Other than that people can report changes in their contacts, addresses etc.
What data changes influence the fee generation process and how?
Any change on FeeRecipe and any change on ApartmentAttributeValue. Possibly something on Apartment but that should be all.
EDIT: Forgot to answer the how part. Change on FeeRecipe usually means change to the formula to calculate the fee. Change on ApartmentAttributeValue means a change to the variables used in that formula. We're using Symfony/ExpressionLanguage to calculate the actual fee amounts from this data.
The company is not too large so 1 dev team, couple of employees who work with the system, each taking care of a few buildings. We will surely integrate with 3rd party APIs to import some data and the plan is to also provide our own API in the future - that's why the backend is GraphQL application with frontend being a JS client running in the browser.
Ok, I've put together a summary in a gist.
You can find it here: https://gist.github.com/codeliner/0204894c29173e8d0aa24bff0d4e88a3
@enumag You can read through it and if you think I've missed some important information then please add a comment to the gist.
I tried to group information from your explanation, sort them and highlight some key phrases, concepts, attention points, questions ...
Don't consider this as complete, correct or whatever. It IS ONLY my current understanding based on the information you gave me.
Our first step now is to work out a context map. The good news is, that we can keep the context map simple because only one team is working on the system and as you told us the company structure is also relatively flat with a few people working with the system.
Also the context map we are drawing is not a final thing. Whenever new concepts arise for example integration of external systems, growing company, new software modules ....
You have to revisit the context map and check if it needs to be aligned.
One of your questions:
And I'm still very unsure about where to split the bounded contexts. Here are some of my current thoughts ....
Please read this article about DDD context maps: https://www.infoq.com/articles/ddd-contextmapping
and based on the explanation in the article try to draw such a map for your domain.
You've named 6 bounded contexts. I tried to match your explanation and the contexts in my summary (see headlines). I've also marked each context as either supporting or core.
Here is a short blog post about the classification: http://blog.jonathanoliver.com/ddd-strategic-design-core-supporting-and-generic-subdomains/
Attention My assumptions are maybe wrong. Treat them more like questions and try to focus on the Ubiquitous Language to identify context boundaries (the linked article provides a good manual).
Maybe start by answering this question:
What are the reasons for the current 6 Bounded Contexts?
Answer for each:
- RealtyRegistration
- ...
- FeeReciepe
- ....
- Accounting
- ...
- ContactAdministration
- ...
- PaymentCollecting
- ...
- FeeCalculating
- ....
Hint
ContactAdministration is a good example for a dedicated bounded context. You said:
Finally the ContactCard represents a Person or Company and can be referenced pretty much anywhere in the system. It can be an employee of our company, an owner of an apartment, an organization managing a building, a contractor our company cooperates with etc.
So one part of the system deals with managing those contacts. They are referenced in other parts of the system but have different meanings depending on the context where they are used.
@codeliner I'm about to leave and I most likely won't have internet connection for a week so I won't be able to read your comment and answer the questions. Maybe my colleague @tomcizek could? Really sorry about this, I did warn about my absence in the first post though so you're hopefully not too surprised. Thank you very much for your effort!
@enumag no problem. Answer when you're back. As I said, I'll be offline the next weeks, too. Just wanted to trigger context mapping. You'll need some time for it anyway ;)
@enumag did you find the time for the context map?
@codeliner I had some other priorities after the vacation but I'll try to make it next week.
So first to answer the question about the reasons for the current bounded contexts:
- RealtyRegistration
- Adding a realty and all the details about the apartments and people is the first thing you need to do in the system. It can be done manually or by some import. It made sense to separate this from the rest of the logic in the system.
- With that said the border between RealtyRegistration and FeeCalculating is really unclear for now.
- FeeCalculating
- This context is supposed take care of the second part where we need to generate the fees every month.
- The main struggle here is that to generate those fees we need some data from RealtyRegistration so we're unsure if those data should be in RealtyRegistration or in FeeCalculating or somehow in both. I'm mainly talking about the ApatrmentAttributeValues (such as body count and area). This is the weakest point in the current structure.
- PaymentCollecting
- We need to collect payments from multiple sources (files, external APIs, manual registration) so again we wanted to separate this from the rest.
- But then, as you know, we need to allocate the payments to generated fees so it's unclear what's correct here.
- Accounting
- At the moment this BC only contains AccountingOrganization. It feels empty to be honest but putting this aggregate inside PaymentCollecting doesn't feel right either because it will also be needed for other things than payments. Again this is one of the weak points in the current structure and most likely it's wrong.
- ContactAdministration
- This one I'm quite satisfied with for now. It separates the contact information from everything else since each contact can be used for several different purposes.
- One thing I'm note quite sure about here is how to handle deletion because you should not be able to delete a contact that is used somewhere else. But this BC doesn't know if a contact is used and where. Should it be aware of it or not?
- FeeRecipe
- Actually there must have been some misunderstanding between us. FeeRecipe is not a bounded context. It's an aggregate inside FeeCalculating bounded context.
Next I'll try to draw a context map with these bounded contexts.
There is one thing I forgot to mention about the domain graph I posted earlier.
In the graph some of the entites are red, some are blue and some are purple. I forgot to mention what those colors were about.
These colors are related to the security - defining privileges who can manipulate what entities and how. We need the privileges to not be system-wide but rather building-wide. But things like payments are not related to one building - when they are imported it's unclear to which building they belong and when they are allocated then one payment can be allocated to fees in multiple buildings. So instead these things have accountingOrganization-wide privileges.
- red = this entity belongs to exactly one building and is a subject of building-wide privileges
- blue = this entity belongs to exactly one accounting organization and is a subject of accounting-organization-wide privileges
- violet = this entity belongs to exactly one building AND exactly one accounting organization, some users will have a building-wide privilege to access it, others will have accounting-organization-wide privilege - having either is enough to access / manipulate the entity
As for current context map, that's actually pretty simple:
RealtyRegistration ---- FeeCalculating ---- PaymentCollecting ---- Accounting
Plus ContactAdministration which is connected to all four of the above RealtyRegistration, PaymentCollecting and Accounting.
I'm not really sure about the relationships between them. The article provides some examples but no clear definition what possible relationships there can be between bounded contexts and how to recognize them. The bounded contexts are not microservices communicating via API so I don't think they are upstream-downstream, right? So is everything just partnership or what? I'm afraid I'm missing something here - I don't know how to qualify the relationships. Really sorry that I can't provide anything better. Maybe you can push me in the right direction?
Maybe I should try to write down the relationships somehow. You could try to advise me with the qualification.
- RealtyRegistration - FeeCalculating
- FeeCalculating needs some data from RealtyRegistration to perform its tasks.
- FeeCalculating - PaymentCollecting
- Fees and Payments need to be allocated to each other.
- PaymentCollecting - Accounting
- Each entity in PaymentCollecting belongs to one AccountingOrganization. I've explained the reason why AccountingOrganization is separate above.
- ContactAdministration - RealtyRegistration
- Each building has several employees of our company assigned to it. Those employees are taking care of whatever is needed to be done regarding that building.
- ContactAdministration - PaymentCollecting
- Each payment needs to be assigned to the person who paid it. Or more exactly to a FinancialAccountGroup that belongs to that person. Each FinancialAccountGroup holds an id of a ContactCard.
- ContactAdministration - Accounting
- AccountingOrganization is a corporate body and as such it holds id of a ContactCard that holds the information about it.
thx @enumag
I try to answer until end of week
@enumag I think we are on a good track.
Regarding the Context Map: we can keep it simple.
To quote from the first article:
So far, we considered a simple scenario with only one development team. This allowed us to ignore the cost of communication, assuming (maybe optimistically) that every developer in the team is aware of "what's going on with the model". A more complex scenario might include some of the following influencing factors:
- domain complexity (requiring many different domain experts)
- organization complexity
- longer projects (time)
- very big project (person days)
- many external, separate or legacy systems involved
- large team size or multiple development teams
- distributed or offshore teams
- human factor
That said, upstream/downstream etc. has less to do with microservices but more with team organisation and integration of external systems. A context map helps to identify these, but because a single team is working on the system and data finds its way into it through imports or manual editing, we don't need to classify the relationships.
So can you draw a simple context map based on the relations you've written in your last comment?
FeeRecipe
Actually there must have been some misunderstanding between us. FeeRecipe is not a bounded context. It's an aggregate inside FeeCalculating bounded context.
Ok, let's see if this is still the case after Event Storming. I've put it in its own bounded context based on your initial explanation of the domain, but as I said. This can be totally wrong.
In general: Do you know the statement:
All models are wrong but some are useful.
Don't worry too much about the current design. It worked so fare, so it is useful. What we do here will also produce a wrong model. New requirements and new knowledge about the domain will change the shape again.
https://twitter.com/jessitron/status/1040284446824509440
Before I can answer your questions regarding the correctness of the bounded contexts we need to do a second activity. We need to event storm the high level business processes.
In the previous issue we did that for payment allocation, so we zoomed into one process and discovered it: https://gist.github.com/codeliner/97ef981147a95a3096c7c89d6dd3d32c
But we need to zoom out again and start on a high level without the details. Start with imports of building, contacts, etc. and draw a timeline of important events along the way.
Ok, let's see if this is still the case after Event Storming. I've put it in its own bounded context based on your initial explanation of the domain, but as I said. This can be totally wrong.
It might be correct but I don't yet understand your reasons for splitting this one aggregate into separate BC. I was just stating how it is currently implemented.
In general: Do you know the statement:
All models are wrong but some are useful.
Don't worry too much about the current design. It worked so fare, so it is useful. What we do here will also produce a wrong model. New requirements and new knowledge about the domain will change the shape again.
Yeah, I know about that. I'm well aware that the goal here is simply to maybe produce something a little less wrong and slightly more useful. It's mostly about learning how to analyze domains so that we can continue the tornado on our own and hopefully teach others how to do it.
Ok, I'll try to draw the context map and a high level overview of the timeline.
There is one more thing. Consulting some other people elsewhere, they said a few things that seem to suggest something along these lines: "Command + Aggregate data should always be enough to perform an action. If the action needs other data from elsewhere then it's not really a command and some refactoring is needed."
They didn't yet answer to confirm if it really is their opinion though. What do you think about this statement? If it is true it kinda contradicts what I'm doing so far and could be the core of what I'm doing wrong.
Current context map: https://drive.google.com/file/d/17gcgNg5-tHvqGaZB7BIks6UkrUHM011P/view?usp=sharing
Here is the timeline. The blue bubbles are the three processes we created separate aggregates for based on our previous discussion. Note that import of a new building is not implemented yet - we have enough data from the old system so it's not needed yet. Notifications are not implemented yet either.
Let me know what should I be more specific about.
Command + Aggregate data should always be enough to perform an action.
Any aggregate root you have foremost is a consistency context. If any command requires outside data to be viable this suggests that your command (and the action it takes on the AR) is directly depenedant on the other data source. Its state (and invariants) become dependent on something else and the aggregate root cannot protect all invariants anymore. That's why a command in its pure form should not do this.
So far the theory. I have seen a few places where this still might be acceptable:
- Cross cutting concerns like logging / authorization (depending it is part of your domain or not)
- Is this merely a shortcut from the API (e.g. it puts something else first, to make the API for the command easier, e.g. file uploads)? Then this should be resolved on the API level already and not on the command handler.
- Commands / Events that are not domain commands/events but application commands/events. Of course those might trigger a domain command on their down with the data loaded :)
To conclude: all domain commands should be self-contained. All data required for the operation on the aggregate should be part of the command (or the aggregate already). If not this often suggests your application layers are leaking into your domain.
I think we need a tool that tracks discussions and documents along the way like we do here within a github issue, but this tool would allow branching of discussions.
The "commands should be self-contained" thingy is an interesting question but it is not the right time yet to discuss it. We should focus on the model and domain behavior from a business POV. Once we're satisfied we can turn that into software and look for design patterns that help us.
thx @enumag for the timeline. I'll do an Event Storming based on it. Maybe I can set up realtimeboard for collaboration because Event Storming without interaction does not produce the same results. But I hope that it is still good enough to share a vision of the model, even if it is simplified and somewhat blurry.
As you said, it's more about teaching the practice than developing the best possible model.
ok, I played a bit with this board here: https://awwapp.com/b/u9blyfjog/#
Not really happy with the usability. realtimeboard is much better, but it does not have free boards where you can collaborate without an account.
I just started to add a few sticky notes. Feel free to continue or edit. If you want to add a note it is best to copy an existing one then change text and color.
Commands / Events that are not domain commands/events but application commands/events. Of course those might trigger a domain command on their down with the data loaded :)
@fritz-gerneth Could you tell me more about these "application commands/events"? In what ways are they different from "domain commands/events". It's not like I never heard of them but the meaning is vague for me.
For example let's take the payment allocations process from my previous issue. When a user clicks a button to start the process the first thing that happens on the server is the command handler searching the projections to see which payments and fees will be a part of that process. Since the command results in new PaymentAllocationsProcess aggregate being created, I guess it counts as a domain command? How would you refactor it to get rid of checking the projections in command handler? I can't let the user send me the data - I couldn't trust them.
@codeliner Yesterday I saw something on the board you linked but it seems empty now? Did you delete it? Anyway maybe it would be better to use realtimeboard. Free version allows three collaborators which should be enough in this case.
@enumag oh ok, the service is really not the best. Didn't touch the board since yesterday. Empty for me as well :(
Ok, can you create a board on realtimeboard and invite me? I already have three collaborators connected with my account. use: kontakt at codeliner dot ws
Ok I created a new board and invited you.
Here is a public link to the board for anyone who wants to see the results (no login required but right now the board is empty): https://realtimeboard.com/app/board/o9J_kzYG5GQ=/
thx @enumag , unfortunately this week is a little bit crazy so I can't fill the board until next week.
@codeliner Of course, no rush. Should I try to fill it with something myself? What kind of sticky notes should I focus on? I'm currently reading some articles such as this one to help me better understand the process.
Basic colors/types
start with events and add other types if needed for understanding.
I'm guessing that for the purposes of this event storming I should ignore how the current system works, right? So instead I added in the high-level events based on the timeline I posted above. Is that ok? What should I try to do as the next step?
Adding commands aggregates etc. doesn't feel right since most of these events are high-level and won't actually exist in the system...
Normally your team would be in a room with one or more domain experts and would use the high level events to start discussions around them by zooming into the processes to add more detailed events + other types based on the explanations of the domain experts.
Unfortunately, we can't do that remotely. You could try it in your company and update the board with the results.
If that's not possible we use the high level events + our previous discussions as a basis to rethink the bounded contexts, identify the core domain and look at the aggregates and processes of the core domain.
While doing this we can revisit the board and add stuff to keep discussions focused on the business model.
The one who knows the most about the domain is actually one of the dev team because he worked on the original system and knows all its problems. He just left for a vacation though.
Another issue with this is that we have already sort of done it when we started working on the system so we would most likely just repeat our previous mistakes.
I can try to zoom in on the events myself next week but... our previous attempt ended up with mostly CRUD events which doesn't seem to be the right way. I'm not quite sure how to avoid doing that again - most of the things happening there indeed are CRUD. Or maybe it's correct to focus on the CRUD things since that's what's causing most of the issues in the current system? What do you think?
That's the problem @enumag If your domain expert is a developer you end up with a technical view instead of the business view. Business people (let's say a sales person) is not interested or doesn't even know about CRUD, Aggregates, Microservices, etc. Event Storming is perfect tool to keep the focus on business processes.
I still don't have much time, but I try to give a short example based on the current board:
A dialog between two actors: the developer (Dev) and the domain expert (Exp)
Dev: "What about this Fee recipe added. Who adds those fee recipes?"
Exp: "Well, our realty administration team is responsible for adding fee recipes."
Exp puts a new yellow sticky note with realty administration team in front of fee recipe added note.
Dev: "And does it happen that they need to change existing fee recipes?"
Exp: "As long as no fees were generated for a fee recipe it can be changed, yes. People can make mistakes and correct them, but if fees were generated we need to know which recipe was used."
Dev: "Ok, but what happens if a mistake is first noticed after fees were generated?"
Exp: "For the next month a new fee recipe needs to be added. We keep the old one as reference, but the new one becomes active. Also the customer service team will contact the person and apologize for the wrong invoice. We'll pay back the difference."
Dev: "I see. Let's add the process on the wall."
Dev and Exp add more sticky notes
....
No word about transactions, entities, consistency and all the technical stuff. Basically, we try to identify what happens over time. Which business processes need to be supported by the system. Who uses the system and in which situations. If we have a good understanding of it, then we can start talking about implementation details like CRUD etc.
So yes, trying the same modeling process again without a non technical (real) domain expert will likely produce the same unsatisfying result.
What can you do?
Ask the people who work with the existing system about the processes. What do they expect from the system. What features would help or support them in their daily work? Do an Event Storming session with them. Just try it!
Who is responsible for payment allocations? Ask them the same questions. And so on.
The explanation above is my advice for the real world. What we can do now is that you play the role of the domain expert and I play the role of the developer. I don't know much about the system yet. I talked to a developer already and they told me about the current modules of the system and gave me an overview of the domain. But I still try to understand how the pieces work together and I hope that you can shed some light (and update the board accordingly ;))
My first questions:
Who adds those fee recipes?
What happens if a fee recipe has a mistake?
@codeliner Now this is for sure some import notes to remember.
The non-programmer domain experts are a busy with other work most of the time so we can't have them brainstorming all the time. But we're certainly trying to think about the system from their point of view. Also whenever we're unsure what they would do in certain situation we consult with them.
For fee recipes you actually got it pretty close. There are some nuances about what to do with already generated incorrect fees. I tried to added some notes to the board regarding that. Hopefully that should answer your questions. I tried to use the color scheme from your comment above (+ I added light green for conditions).
There is one new thing for you here - "accounted for fees". Usually in april we need to wrap up the previous year. When that happens some documents are created and most actions are no longer available with the payments and fees for that year (and they become "accounted for"). We didn't delve too deep into that just yet - we have other things to solve before that.
There is one thing though about the notes I just added to the board. While these actions are needed to fix incorrect fee recipes, they are not specific to that process. It's just some tools the building referent (employee of our company who works with the system) needs to have at their disposal. For example unallocating and reallocating payments is also done when we discover some old payment that was not added into the system because of a mistake. So while I added the notes close to fee recipe addition for now, it doesn't really feel correct to have it that way. How should things like that be represented on the board?
thx @enumag I'll check the board and your comment at the weekend.
It's a good time to link one of the best DDD talks (I'm aware of) in case you don't know the talk:
Cyrille Martraire — Interviewing Domain Experts: heuristics from the trenches
@enumag really good progress on the board 👍
So while I added the notes close to fee recipe addition for now, it doesn't really feel correct to have it that way. How should things like that be represented on the board?
There are no strict rules. We only work with sticky notes so we can move them around, remove some or duplicate others.
Just add the other scenarios like this one:
For example unallocating and reallocating payments is also done when we discover some old payment that was not added into the system because of a mistake.
and copy & paste unallocating and reallocating payments notes.
I guess unallocating and reallocating payments is also done when an ApartementAttributeValue is wrong? Can you add that scenario to the board as well?
Oki, I'll try to add some more scenarios next week. I'll try to find some time for that talk you linked, didn't see that one yet.
Side note: Event Storming is like brain storming. You add every piece of information to the board and rearrange as needed. Then we can use the notes to identify contexts by grouping them. But first we need them on the board. Still looking for the exact border of the core domain. One thing is for sure. Event Sourcing is a really good choice here, at least for the core domain. But maybe your team can use CRUD-ish approach in the supporting sub domains. I don't want to start a technical discussion yet, but let you know what I think will be the end result of this thread.
Let's add the notes from my previous comment to the board and then look at the tactical side of DDD.
I'll try to find some time for that talk you linked, didn't see that one yet.
👍
@cyriux is a really good entertainer, too. So the talk has both: fun and value! Absolutely worth watching.
There is one more thing on my mind. In our previous eventstorming we mostly focused on adding the data we need into the system and the common tasks performed with those data. We didn't talk too much about these scenarios when something went wrong and needs to be fixed because they are not too common and the goal of the system is to help the system admins make them even less common. From this point of view, I have to ask - are you sure that there problematic cases are what we should focus on? Or is it exactly the point of doing event storming to focus on such scenarios despite their rarity in production?
Combining CQRS with CRUD-ish subdomains was one of my early ideas. I scrapped it because I didn't have nearly enough knowledge to decide what should go through CQRS and what shouldn't - and I still don't, never mind how actually combine them in the code. I'm very interested in learning more about that. Especially how to separate an event-sourced and non-event-sourced parts of a system. It seems quite impossible at first glance since the event-sourced part will need some data from the rest.
are you sure that there problematic cases are what we should focus on?
That's a very good question! Let me try to explain:
By asking questions like "What happens if ...?" "How do you handle that scenario?" "Why are you doing it?" ... you trigger discussions with the domain expert and learn more about the domain.
Look at my two questions and your answers:
- Who adds those fee recipes?
It's just some tools the building referent (employee of our company who works with the system) needs to have at their disposal.
Great! Now I have a name for the actors working with the system. I can add it to my vocabulary and use it in further discussions. It's now part of the ubiquitous language.
- What happens if a fee recipe has a mistake?
There is one new thing for you here - "accounted for fees". Usually in april we need to wrap up the previous year. When that happens some documents are created and most actions are no longer available with the payments and fees for that year (and they become "accounted for"). We didn't delve too deep into that just yet - we have other things to solve before that.
One simple question and I learned a completely new detail about the domain. That's great. I can take it into account when thinking about the different parts of system.
Keep in mind that our async, remote communication here is not comparable to a real event storming session. In the real world we would only need a couple of hours. If all ppl are in the same room and you have unlimited modeling space something magical can happen ;). The talk I've linked covers some more tips and tricks to interview domain experts. Event Storming is not the only way, but a very efficient one. See the notes I've used at the beginning of the thread: https://gist.github.com/codeliner/0204894c29173e8d0aa24bff0d4e88a3
*In the gist I took your explanations and highlighted important words (UL). Grouped the information etc. *
I've done it because I need to get an overview of the domain. We need the same understanding of the business. You are my domain expert at the moment. I learn from your knowledge, then I can build a system that supports your work.
Ok, let's move on:
I already know that ApartmentAttributes are very important for generating fees but I don't see them on the board. Hence, I asked the next question to trigger a discussion about this concept ;)
But you are absolutely right that we should not discuss every little detail. During an event storming session you can add a "hot spot" note and move a discussion about the details to a later time. I've updated the board with such a hot spot marker. You're right. We are all very busy and we need to get as much information as possible in the shortest possible time.
Having said this, I'd like to ask another question:
You told me about ApartmentAttributes. They are assgined to a Building by the responsible building referent when a new building is added. You also said that those ApartmentAttributes can change, right? There is this monthly automated process that generates fees using the FeeRecipes and ApartmentAttributes assigned to a building.
Is there some kind of "monthly deadline" for building referents until ApartmentAttributes changes should be made, so that the automated fee generation process can work with a fixed set of information? Can you add the processes regarding ApartmentAttributes to the board?
Sorry, I'm a bit busy this week rewriting our GraphQL layer to something easier to maintain. If it goes well I might even write an article how to use Prooph with GraphQL.
I'll answer when I can but it will most likely be next week.
Sorry, I'm a bit busy this week rewriting our GraphQL layer so something easier to maintain. If it goes well I might even write an article how to use Prooph with GraphQL.
That would be interesting to read 👍
I guess unallocating and reallocating payments is also done when an ApartementAttributeValue is wrong?
I see I didn't exactly answer this question.
If we find out that a unit attribute changed several months ago and wasn't reported we need to unallocate the payments since then, delete the generated fees for that period, generate new fees using the correct values and then reallocate the payments. So it's actually even more complicated than the previous case because we need to delete the wrong fees too.
EDIT: The answer above was actually incorrect. After checking with my teammates I found out it works differently. It's not necessary to change the old fees in this case because the difference will be calculated with when we wrap up the year the next april. Basically it doesn't matter if the people paid something wrong, it will all be fixed in this yearly audit but we don't need to actually change the old fees in most cases.
Is there some kind of "monthly deadline" for building referents until ApartmentAttributes changes should be made, so that the automated fee generation process can work with a fixed set of information?
No because there is no way to tell if we received all the changes or if there even were any changes. I'll double check with my colleagues though.
EDIT: Yeah, no deadline here.
Can you add the processes regarding ApartmentAttributes to the board?
Sure, I'll try to find some time today - have been neglecting this for too long.
EDIT: Considering that the history doesn't change in this case there is no process. Just change the ApartmentAttributeValue with the date when it did change and you're done.
@enumag Do you have some example data? A building + entrance + apartment + apartment attribute values + fee recipe
And a simple/basic example of a fee calculation?
I've created an example project: https://github.com/proophsoftware/fee-office
Next steps can better be discussed with code ;)
I'd add a prototype using Event Machine in some contexts and Doctrine a document store in others. This will show you how you can split the system into contexts and use CRUD-ish entities in supporting subdomains and event sourcing in the core.
If I cut out the details that are irrelevant to this (such as some fields that are only needed to print them on documents) then I could generate some random data with Faker. I'm not familiar with Event Machine yet though. Where should I put the data and in what format?
@enumag you can use a gist. Simplified sample data would be great. Just enough to build a prototype.
Oh and what type of contacts can exist? Some sample contacts would be great, too.
@enumag regarding the format itself: JSON would be nice
FeeCalculating: The main struggle here is that to generate those fees we need some data from RealtyRegistration so we're unsure if those data should be in RealtyRegistration or in FeeCalculating or somehow in both. I'm mainly talking about the ApatrmentAttributeValues (such as body count and area). This is the weakest point in the current structure.
I can give you the answer now:
In both ;)
First of all I'd also add FeeRecipe as an entity to RealtyRegistration. RealtyRegistration can contain a structural model. You can design it using Doctrine or another ORM. However, in the demo I'll use Event Machine for that context, too. Just to get a feeling if event sourcing is beneficial for that context as well. The change history could be interesting. Also versioning FeeRecipe and ApartmentAttributeValue sounds like a good idea.
In the FeeCalculating context FeeRecipe and ApartementAttributeValue are value objects. Remember each context has a boundary. An entity or aggregate in one context can be a value object in another context. The former context is responsible for managing the lifecycle. The latter context gets read access (through interfaces) and can use the data for its own processing logic.
As soon as we have some example data I can show how that works in code ;)
ping @enumag How do we want to proceed? Do you have time to provide the data?
I didn't forget, I'm just really busy... I'll try to give it priority on thursday.
thx @enumag can't wait to continue ;) Looking forward to our next iteration.
@codeliner Here. This json contains some testing buildings, entrances, apartments, contracts, accountingOrganizations and related contacts. It doesn't have things like unitAttributes, feeRecipes, feeItems, payments etc. yet. Is it enough for now or should I continue?
Here is the script I used to generate it: https://github.com/enumag/fee-office-example-data
In the FeeCalculating context FeeRecipe and ApartementAttributeValue are value objects. Remember each context has a boundary. An entity or aggregate in one context can be a value object in another context. The former context is responsible for managing the lifecycle. The latter context gets read access (through interfaces) and can use the data for its own processing logic.
Yeah I know an aggregate from one context can be a value object in another. But I'm not really sure how you want to use interfaces there. I'm very curious to find out.
I need to take good look at the demo you made. Didn't have time to check it out yet or just briefly. Anything I should pay special attention to?
It doesn't have things like unitAttributes, feeRecipes, feeItems, payments etc. yet. Is it enough for now or should I continue?
We definitely need that as well ;)
But I'm not really sure how you want to use interfaces there. I'm very curious to find out.
Therefor, I need at least one example of FeeRecipe + ApartementAttributeValue so that I can show you how this can be managed in one context and used for processing in another context.
Anything I should pay special attention to?
For now take a look at the module system: https://proophsoftware.github.io/fee-office/intro/module_system.html#1-3
Each module is a bounded context. That's the basis for the implementation. Now that I have some examples I'll start with RealtyRegistration . I'll link to important pieces of code and get back to you with questions.
Ok, added more data here:
Note: generated fees don't match the feeRecipes - it's not really possible to do that with random data without the real logic that generates fees.
EDIT: Of course in practice most of the entities hold some additional data - for example fees have a dueDate which is important later to generating more fees as penalties for late payments. That's of course completely out of scope here.
Oh so you're using Zend framework for this one. This is quite interesting because I always wanted to see an app written using middlewares. I have some questions about the architecture:
- Can the modules have dependencies on each other? If so, is there any danger of running into cyclic dependencies?
- Where would ProcessManagers be located? (note that some ProcessManagers are cross-module while others are not)
- If they should be in the application layers (you wrote the application layer manages communication between modules) then in which application layer in the module as microservice scenario?
- How would async service buses work?
- Where would projections be located?
hehe, your questions are ahead of the example ;) I try to address them while implementing the modules. These are important questions and my goal is not only answer your questions with the demo but share the concept with as many people as possible.
That said, I'll only give short answers now and get back to each in detail when I have some reference implementation.
Can the modules have dependencies on each other? If so, is there any danger of running into cyclic dependencies?
No. This would violate rule 4 "No module is allowed to use a class or function defined in another module."
Modules communicate with other modules using messages. Not more, not less.
Where would ProcessManagers be located? (note that some ProcessManagers are cross-module while others are not)
A cross-module process manager always belongs to one module. A process manager reacts on an event and dispatches a new command. As long as all modules are maintained by the same team you can choose:
- publish the event in module A and react on it with a process manager in module B
- React on the event with a process manager in module A and dispatch a command handled by module B
If modules are turned into microservices later and different teams work on module/microservice A and B the question gets more interesting. Now you need adavanced context mapping and be aware of the relations between A and B as well as the API contract. But that's a longer story to tell.
If they should be in the application layers (you wrote the application layer manages communication between modules) then in which application layer in the module as microservice scenario?
This gets more clear with some working code. I'll answer later.
How would async service buses work?
You mean message broker like rabbitMQ?
- Set it up as an infrastructure service
- Require a lib in each module to connect to the broker
- Produce messages in one module and consume them in another
I'm pretty sure we can add a rabbitMQ example to the demo.
Where would projections be located?
A projection should always belong to one module. However, you can define a stream of events as a public source of information and let projections from other modules process that information. But that should be an explicit contract. If events are recorded in module A but projected in module B then both modules have a contract. A cannot simply change the structure of events. They need to be versioned like you version a public http API.
If you have a projection that generates reports and this projection does not belong to any module because it aggregates information then you should put it into a dedicated reporting module.
Hope this gives you some answers for now.
hehe, your questions are ahead of the example ;)
Yeah, I tend to do that, sorry. :-D Thanks for the answers you provided, it really made things much clearer for me!
I'm pretty sure we can add a rabbitMQ example to the demo.
Yeah that would be helpful. I'm now slightly confused as I'm trying to use psb-enqueue-producer in my stack but the way it seems work is that each message only has one consumer. This is fine for prooph commands but not for prooph events, especially in microservice setup because you might want to react to one event in several microservices.
A projection should always belong to one module. However, you can define a stream of events as a public source of information and let projections from other modules process that information.
So if I wanted to implement a GraphQL API I would need a separate module with a projection aggregating data from all other modules?
@enumag We need to discuss the design. Maybe we need to get back to event storming.
I'm thinking loud. Let's see what you've to say about it:
I'd not put FeeRecipe into FeeCalculating context.
You describe FeeCalculating context like this:
This context is supposed take care of the second part where we need to generate the fees every month.
From what we discussed so far my understanding is that the monthly fee calculation and payment allocation process should be one independent context. All data needed for this monthly process should come from one or more supporting contexts - including FeeRecipes.
This way updating/managing the data is separated from the core functionality of calculating fees and allocating payments. The team can spend most effort on getting this automated process right and use cheaper development tools/practices for the other contexts.
What do you think?
I'm thinking about two different ways:
-
Contract,FinancialAccountGroup,FinancialAccountandFeeRecipebelong toRealtyRegistrationcontext -
Contract,FinancialAccountGroup,FinancialAccountandFeeRecipecan be put in another context, maybe something likeContractAdministration
I still don't think that the monthly fee calculating process should be in the same context as payment import and payment allocation. Most fees are generated automatically in monthly audit, yes, but they can also be added manually or can be a result of a different process (PenaltyCalculating, YearlyAudit). Putting all that into the same context would make it too large I think. So instead there should be several contexts that can generate fees and the context that takes care of allocations would collect them from all sources.
EDIT: Maybe in the calculating context it would not be named "Fee" but something like "MonthlyAuditResultItem" which would only become a Fee in different context?
About where to put Contract and the related objects - I think it should be separate from RealtyRegistration. The reason is that Contract is the first thing in the hierarchy that belongs to an AccountingOrganization as well as Realty (=building). FinancialAccountGroup does not even belong to a Realty at all.
Regarding contract my thoughts go in the same direction. We did not talk much about contracts, financial account and finanzial account group. I cannot really classify them or place them in the right context.
What bothers me is the fact that FeeRecipe and ApartmentAttribute have a strong relationship. The formula defined in FeeRecipe needs to reference the right ApartmentAttribute by name AND such an attribute needs to be set on the linked apartment (and that's not a direct link but requires two hops: financial account and contract).
How is the user journey defined for a building referent who enters the data in the system? How do they ensure that the formula willl work when it comes to fee calculation?
So instead there should be several contexts that can generate fees and the context that takes care of allocations would collect them from all sources.
EDIT: Maybe in the calculating context it would not be named "Fee" but something like "MonthlyAuditResultItem" which would only become a Fee in different context?
I see, so we don't have enough stickies on the board, yet. Those processes are not visible on the board. We should add them.
oh, overlooked your question yesterday, sorry.
So if I wanted to implement a GraphQL API I would need a separate module with a projection aggregating data from all other modules?
The GraphQL API can act as an API Gateway in front of the microservices. You could write resolvers that perform remote queries to the different microservices responsible for parts of the data.
If that's not possible - due to missing experience with microservices etc - I'd go for a dedicated GraphQL module with one or more projections processing different source streams. But again, this variant couples the event streams of your microservices with this GraphQL module. So you need to be very careful with event schema changes and the like.
How is the user journey defined for a building referent who enters the data in the system? How do they ensure that the formula willl work when it comes to fee calculation?
When a formula is added or changed we parse it to see what attributes it uses and check if the apartmant has all those attributes. Though this is currently not done in a good way.
I see, so we don't have enough stickies on the board, yet. Those processes are not visible on the board. We should add them.
Ok, I'll try on thursday. Can you elaborate bit more which processes you want and maybe ask some questions I should answer with the sticks?
The GraphQL API can act as an API Gateway in front of the microservices. You could write resolvers that perform remote queries to the different microservices responsible for parts of the data.
Wouldn't that make the API too slow?
Wouldn't that make the API too slow?
Define too slow ;) Also that highly depends on a) how fast the involved microservices can respond to those requests and b) how the resolvers are implemented (caching, parallel requests, query complexity limits, ...) GraphQL is not another way of querying a database ;) It's an API layer and should be treated as such. The other question is, if it is a good idea to expose all data with one GraphQL endpoint? You create very very high coupling between your read models and the centralized GraphQL endpoint making it really hard to evolve one bounded context independent of the others.
Ok, I'll try on thursday. Can you elaborate bit more which processes you want and maybe ask some questions I should answer with the sticks?
Let's see ...
yes, but they can also be added manually or can be a result of a different process (PenaltyCalculating, YearlyAudit). Putting all that into the same context would make it too large I think. So instead there should be several contexts that can generate fees and the context that takes care of allocations would collect them from all sources.
So far we've identified 5 bounded contexts, see https://proophsoftware.github.io/fee-office/ddd/context_mapping.html#2-2
So what other contexts do you have in mind? Can you add them on the board (as frames) and put important events into them like PenaltyFeeCalculated, FeeManuallyAdded, ....
Then duplicate those sticky notes and put them all into PaymentCollecting. This indicates that the events get published to PaymentCollecting for further processing.
Define too slow ;) Also that highly depends on a) how fast the involved microservices can respond to those requests and b) how the resolvers are implemented (caching, parallel requests, query complexity limits, ...) GraphQL is not another way of querying a database ;) It's an API layer and should be treated as such. The other question is, if it is a good idea to expose all data with one GraphQL endpoint? You create very very high coupling between your read models and the centralized GraphQL endpoint making it really hard to evolve one bounded context independent of the others.
Currently the API based on the Overblog/GraphQLBundle can respond in around 100ms which is not slow but I would not say it's fast either - I don't want it to get any slower for sure. And that's already after making some optimizations. I'm trying to find ways how to optimize it further by async and parallel requests and possibly even swoole. With microservices the speed would also depend on the communication protocol between the microservices - some people simply use HTTP/REST with JSON, others say it's too slow. Maybe you could share some tips? Anyway we're getting off-topic here I think so for now I'll just go with the centralized endpoint as I did until now due to lack of experience with microservices. I should learn more about DDD first before attempting to use microservices I think.
Keep in mind that I don't want to add penalty calculating and all the other things into the example - it would grow very large that way I think. We just should remember that there are other ways to create fees and therefore should not restrict it to just the monthly fee calculating. Sure, I'll add them to the board (in some simplified form).
Sure, I'll add them to the board (in some simplified form).
👍 We try to identify boundaries. As you already know this is the hardest part. It depends on so many factors. You're working for a relatively small company with one development team. It's not uncommon in that scenario that the team creates one big application where everything seems to be coupled with each other. But you also know that this will get in your way later and probably becomes an unmaintainable mess. Our only chance here is, that we find some good context boundaries by grouping concepts that seem to belong together. Therefor, we need to follow and discuss hints like "yes, but they can also be added manually or can be a result of a different process (PenaltyCalculating, YearlyAudit)."
If we exclude those information from the design, we end up with yet another wrong model, because I don't know anything about the hidden concepts and cannot respect them. You know them, so my model would not be useful for you.
That's the whole point of DDD. A model can only be useful for the business if the developers and domain experts share the same idea of the model If there is a mismatch, the model is wrong and therefor less useful, if not useless at-all.
Ok, check the board now.
@enumag Did some updates on the board as well. What do you think?
Updates look good. I'm currently trying to split the contexts and aggregates better based on the board and other things I realized recently. I'll post it here when I'm done.
Quick question about the ContactAdministraction supporting context. Since contacts are used in most of the other contexts, does it mean I need to duplicate the ContactCard aggregate into all of them to be able to verify that a contact with given ID exists and that the contact meets all the requirements? (For example for some uses the contact might be required to have a BankAccount.)
Don't duplicate the aggregate, but use value objects in other contexts.
Take SuperiorParty and SubordniateParty of a contract as an example. Both are connected with a ContactCard. Turn them into value objects that can be constructed from contact information.
You'll see this later in the example project. But here is a short code snippet to illustrate the general idea:
$superiorParty = SuperiorParty::fromContact($contactAdministrationBus->performSyncQuery(
Query::CONTACT_ADMINISTRATION_GET_CONTACT_CARD,
Query::payload(['contactId' => $superiorPartyContactId->toString()])
));Specific value objects can contain validation rules of contact information.
Let's say a subordinate party is required to have a bank account, then a SubordinateParty value object can enforce this rule by checking information given from ContactAdministration context.
Ok you lost me here. First where would the snippet you posted be? As in which class? A CommandHandler that creates new Contract? If so, isn't it a violation of the module independence? I mean the $contactAdministrationBus needs to load and return the aggregate from the ContactAdministration context which means there is now a dependency on that module...
A CommandHandler
yes, for example. Depends a bit on your project structure but command handler or application service are good candidates.
$contactAdministrationBus don't return the aggregate. It returns information of a contact (pure data, maybe as a simple array or DTO, depends on the defined API contract)
which means there is now a dependency on that module...
Yes there is a dependency, but $contactAdministrationBus->performSyncQuery can also be a http request. We don't know what happens behind that interface ;)
Reading information from other contexts is not a bad thing, but we have to keep in mind that ContractManagment service now depends on the availability of ContactAdministration service, because we do a synchronous request. I'll detail those dependencies in the Fee Office book. Hope this short explanation gives you a first answer.
Yes there is a dependency, but
$contactAdministrationBus->performSyncQuerycan also be a http request.
Ok, now it's starting to make more sense. I'm guessing this could be implemented for example with RabbitMQ's Direct reply-to feature? It seems you guys are using that here.
Also on the side of ContactAdministration the data should be read directly from the aggregate, not from projection, right?
Assuming I understood it correctly, let me hit you with another question. Previously you said here that ApartmentAttributeValues should be in both RealtyRegistration and FeeCalculating. But why do you actually want them in FeeCalculating rather than just reading them from RealtyRegistration via a synchronous query when needed? I'm trying to understand the case when the same data are needed in multiple contexts. Where is the line between:
a. Synchronize them from the source context to the other context via some policies and duplicate the data to both contexts that way. (Which is what I assume you meant for ApartmentAttributes.)
b. Read them from the source context in a CommandHandler via a synchronous query. (As you're suggesting now for ContactCards.)
The answer is obviously "It depends". I'm interested in your thought process in these cases. What are the reasons to choose between a and b here?
Yes, rabbitMQ is one possible way. A http request hitting an API is also possible. That's implementation detail of the infrastructure ;) Remember, you're one team and can decide what you want to do. In the demo I'll directly invoke a PHP method but this is again an implementation detail. If I later switch to a remote query it's still hidden behind the $contactAdministrationBus->performSyncQuery() interface.
Also on the side of ContactAdministration the data should be read directly from the aggregate, not from projection, right?
ContractManagement is consumer of data/information owned by ContactAdministration. Now those bounded context relations kick in.
ContractManagement is the downstream context - it consumes contact information
ContactAdministration is upstream - it owns, produces and manages contact information
Let's say both contexts are developed and maintained by two different teams. Both teams have different scrum boards with different priorities and deadlines. However, they have a partnership relation and negotiate an API contract.
Part of this contract would be if contact information needs to be always consistent or if a short period of time of out-of-date information is acceptable. They could discuss the use case:
ContractMgmt-Team: A building referent needs to add a contract. They need to assign subordinate party that is a contact of your service. This contact needs to have a bank account.
ContactAdmin-Team: Well, ok no problem. We provide a "search contact by name" API that returns basic contact information and a "get ContactCard" endpoint to fetch contact details which includes the bank account.
How those APIs are designed internally is totally up to the ContactAdmin-Team. But maybe both teams decide on an SLA and define that both "search" and "contact details" should respond in not more than 20 milliseconds because ContractMgmt performs synchronous requests within their own execution time and can't be blocked for too long. So performance matters and the chance of an out-of-date contact is relatively slow. Hence, they agree that for the sake of maximum performance they take the risk of reading a contact that is eventual consistent (read information from async projection).
ContractMgmt-Team just checks contact card details and if a bank account is not included, they add a warning in the UI to ask the building referent to check contact information manually and refresh subordinate party information once bank account is available.
As you can see, you have to look at the process and consider different factors.
Another very valid design would be:
Contact cards are relatively trivial and event sourcing does not add much benefits to this particular context. ContactAdmin-Team decides to just use a CRUD-ish API in front of MongoDB and stores each contact card as a document. A thin PHP layer + MongoDB performance with correct indexes in place should meet the SLA. So the team can provide both: fast response times and consistent data.
Which is what I assume you meant for ApartmentAttributes.
It's not yet decided which integration pattern is the best one for ApartmentAttributes. I only meant the PHP classes ;). RealtyRegistration contains an ApartmentAttribute class and FeeCalculating as well. How the data gets from one context to the other is a different story. What we know for sure is that RealtyRegistration is upstream (owns ApartmentAttribute information) and FeeCalculating is downstream (consumes ApartmentAttribute information).
I'm interested in your thought process in these cases.
Coupling ;) In what cases is it ok to couple two services and in what cases is it better to keep them completely independent.
a.) = cons: redundant data, eventual consistency, more complex inter-process communication, pros: maximum service autonomy -> freedom
b.) = cons: high coupling, if upstream service is slow or even unavailable, downstream has at least a problem, pros: easier integration
Please note: the comparison is not complete. We talked about different teams and the relation between them and their contexts. This can influence the integration mechanism as well. Another factor is again performance, SLAs, hosting, ...
Given my explanation above the question is what a single team should do that works on all contexts?
Each team member can still put on either the ContactAdministration hat or the ContractManagement hat.
You know what I mean? Just imagine the contexts are developed by different teams. What would you do as a member of one of the teams. In this imaginary scenario both teams are always in a partnership relation.
In a large company with many different teams the situation is often much more complex. For example our team is currently in a Conformist relation which is the worst possible type :( Anyway, we know that and also know what to do: Put an anti-corruption-layer between us and the other team ;)
I can only stress the point that no matter if you're a single team in a small company or one of many teams in a large company, IF YOU DON'T divide the application into bounded contexts (can be modules like in the demo or real microservices), you'll create a big ball of mud sooner or later. It will bite you, so better you do the thought experiment form the beginning.
Whoa this is a long answer. But very informative!
Yes, rabbitMQ is one possible way. A http request hitting an API is also possible. That's implementation detail of the infrastructure ;)
The reason for using RabbitMQ here is that it will be the same regardless of if microservices are used or not. Maybe just some spliting to multiple queues. It seems more future-proof to me.
Coupling ;) In what cases is it ok to couple two services and in what cases is it better to keep them completely independent.
So in this case it most likely makes sense to use higher coupling since the modules are not really meant to work with complete independence - it's one system for a few employees.
I can only stress the point that no matter if you're a single team in a small company or one of many teams in a large company, IF YOU DON'T divide the application into bounded contexts (can be modules like in the demo or real microservices), you'll create a big ball of mud sooner or later. It will bite you, so better you do the thought experiment form the beginning.
Yep, exactly the reason why I started this emergency call. It is starting to make more sense now. I'll try to finish my new analysis of the domain this weekend and post it here for your review.
It is starting to make more sense now.
That's great. I'm really happy about the emergency call, too. It's the most detailed one so fare with lot's of stuff to create learning material for more developers. I't's a lot of work and needs time, but that's ok.
I'll try to finish my new analysis of the domain this weekend and post it here for your review.
looking forward to it 👍
Ok, here it is. Take your time with reading this, I know you're busy and I'm not in a hurry to use this asap. Ask questions if needed of course and tell me what you think. I'm honestly quite satisfied with this one and feel like it's finally a decent starting point.
Previously we have designed the aggregates to be the entities that appear in the domain. While I still think this makes some sense for the CRUD parts, as soon as there is some more complicated process, the aggregates should be designed around those processes. And that's exactly what I'm trying to do here.
ContactAdministration
Aggregates: ContactCard
This context is pretty obvious to be separate since contacts are used pretty much everywhere else for different purposes.
RealtyRegistration
Aggregates: ApartmentAttribute, Realty, Apartment
Just a place to register all the data about buildings, pretty much CRUD (this might change later when we explore the possibilities of import from external source). Entrance should probably be put into Realty aggregate since it's not very important. Apartment on the other hand is quite important because it will contain the ApartmentAttributeValues which will be needed in other contexts.
Accounting
Aggregates: AccountingOrganization, FinancialAccountGroup
Again some CRUD, this time oriented around accounting. The primary thing is AccountingOrganization which will hold some settings useful in other contexts. FinancialAccountGroup can hold any client-specific settings.
FeeCalculating
Aggregates: Contract, MonthlyAuditProcess
Now this is where things get more complicated and where we went really wrong with our previous design. Firstly this context should just be able to generate the calculate monthly fees, but fees from other sources are not important for that and should never appear in this context. Secondly Fee should not be a separate aggregate like we did it previously. Here is how I imagine it to work:
- Contract aggregate will hold the history of FeeRecipes and will be able to return those that are active in a given time period.
- When we need to calculate the fees a new MonthlyAuditProcess will be created. The command handler will read the active FeeRecipes from the Contract and the ApartmentAttributeValues from Apartment.
- The important thing is that there are the Contract and Apartment aggregates capable of answering such queries – that was not the case in the previous design.
- MonthlyAuditProcess will generate the fees but doesn't care about them beyond that.
- This context never receives anything about fees generated by other means or added manually nor anything about payments or payment allocations.
PaymentImport
Aggregates: PaymentImportProcess
Taking inspiration from how FeeCalculating should never handle manually added fees I decided to split the PaymentCollecting context using similar criteria. This context should take care of importing payments from files, APIs etc. and assigning them to specific FinancialAccountGroups.
PaymentAllocating
Aggregates: FinancialAccountGroup, PaymentAllocationProcess
Now this is the core business. This context should be the only one aggregating fees and payments from all other sources and should perform the pairing to find out if and when was each fee paid. To do that the FinancialAccountGroup aggregate needs to be duplicated here to hold the data of all payments and fees. However it doesn't need details like where was a payment imported from, how was a fee generated, which FinancialAccount or Contract they belong to, etc. For allocating payments we only need the dates and amounts so that's the only things about fees and payments that will be propagated into this context.
One important note is that while the events of FinancialAccountGroup will contain the entire history of payments, fees and allocations, we're not actually interested in the history here. For allocating payments we only need the current state – payments and fees that are not yet fully allocated. Therefore the FinancialAccountGroup clone here seems like a good aggregate to use snapshots on – the snapshot would only hold the current state of unallocated and partially allocated payments and fees but completely ignore all the fully allocated payments and fees since those are no longer relevant.
Another important note here is how to handle cases where we need to revert allocations for certain period of time. One could argue that to do that we're actually interested in the history here. I'm going to disagree with that because I'd completely separate the reversion from this context. Then when such a revert occurs it's not really relevant for this context – the payments and fees that had allocations reverted can be inserted here the same way as when they were originally added. It's irrelevant if they became unallocated by being new or by having allocations reverted.
PenaltyCalculating
Aggregates: FinancialAccountGroup, PaymentCalculatingProcess
Yes, another duplicate of the FinancialAccountGroup. This time however we're not interested in the same data as in PaymentCollecting which is why I separated this. The difference is that in contrast with the previous context PenaltyCalculating is only interested in the fee history including unallocated fees and payment allocations. However payments themselves are completely irrelevant here. It's also irrelevant which payment covered which fee – we're only interested in if and when was the fee paid.
Next I'd like to talk about deletion and how would you recommend dealing with it. For example someone might want to delete an old ContactCard. But then ContactCard doesn't know if it can be deleted since it doesn't know if it is linked to any realty/contract/organization/whatever. Of course there is no true deletion in EventSourcing - I can only mark the ContactCard as deleted. But I might want to delete it from projections. Here is what I can think of:
-
Don't check the usage at all, simply mark the aggregate as deleted and throw exception on any further manipulation (except undeletion if needed). This means the deleted aggregate can still be elsewhere and you can't delete it in projections either since the projection would become inconsistent (foreign key violations). But it could be hidden on any ContactCard listing.
-
Don't allow the removal. This however requires the ContactCard to track the usages. Which I'm not sure how to do while preserving full consistency.
How do you deal with it in your projects? Is there any other option I don't see? Which option do you use under what circumstances and what would you recommend here? (I'm leaning towards the soft-deletion flag in projections here.)
I'm also a bit confused about the recommended approach to write aggregates, commands, events and the domain logic. There is the now deprecated prooph/event-sourcing. Then there was the announcement about Prooph v8 stating that domain should not depend on any library. And now in proophsoftware/fee-office you're using proophsoftware/event-machine. Is EventMachine meant to be the recommended successor to prooph/event-sourcing? Isn't that contradicting the Prooph v8 announcement about domain being independent? Wasn't EventMachine only meant just for prototyping? Should I use it in my application?
Another thing on my list of things to solve is error handling. The API application is async so it just receives a command to execute and sends it to RabbitMQ. Then a consumer takes it, locates a command handler and executes it. So far so good, I have this implemented with async switch routing on prooph/service-bus.
Now I want to improve this by adding a websocket endpoint where the client could listen to some events what happened to his commands - successful exection or some error. What I'm unsure about is how to send this information from the consumer thread that executed the command to the thread with the websocket server. Any ideas? Did you try to implement something along those lines before?
EDIT: Never mind, I found some info about that here and a new enqueue component with nice API here. Of course if you have something to say about this, please do.
@enumag New context design sounds really good 👍 Need to read it again with a bit more time, though.
Regarding contact card deletion. I think it's also a GDPR issue so yeah you need a way to delete contact cards and the system should also be able to give information about the contact and what you do with the data of that person.
In the Fee Office ContactAdministration uses CRUD + immutable data types (no event sourcing). Deletion would be possible. The ContactCardCollection could return a "DeletedContactCard ({"id": "<old_uuid>", "firstName": "Deleted", "lastName": "Contact"})" when it's been ask for a deleted contact card. However, there are legal requirements in some cases which are stronger than GDPR. I'm not an expert here. Your company needs to talk to a lawyer for details. But let's say you should not delete the contact card if it is referenced in an active contract. If such rule exists then the Contract aggregate needs to inform ContactAdministration about the reference (for example with an Event: ContactAssignedToContract) and the information needs to be stored in the ContactCard, so that deletion is blocked in that case.
Regarding usage of event machine:
First take a look at the Fee Office notes:
It is a prototype implementation showing the result of a model exploration and knowledge crunching process.
I don't want to develop your next version of the product but provide some example code as fast as possible. Event Machine is the best tool available for the job. Really you can try other solutions and I bet that you'll be slower with them.
Then there was the announcement about Prooph v8 stating that domain should not depend on any library.
Look at the Model code: https://github.com/proophsoftware/fee-office/blob/master/src/RealtyRegistration/src/Model/Building.php#L10
No interface, no abstract class, nothing
Some data types use an "ImmutableRecord" implementation, which is provided by Event Machine. It's planed that this part gets moved to an independent and small library. But you could also copy and paste the "ImmutableRecordLogic" trait and remove this dependency on the data types completely.
Is EventMachine meant to be the recommended successor to prooph/event-sourcing?
If you ask me personally: yes.
From the prooph point of view: no.
;)
Event Machine is a prooph software project. It's developed by my company which is also the sponsor of the emergency-calls and therefor the Fee Office demo.
The prooph community project is independent.
There is a very important difference between prooph components and Event Machine:
the components were never meant to be used as a framework. there is no prooph way of doing things. We don't want that, because use cases are very different.
Event Machine on the other hand is a FRAMEWORK. If you're happy with the style and its limitations don't block you, you can enjoy the fastest way of developing event sourced domains. That is what we do at prooph software. No boilerplate, straight-forward event sourcing, out-of-the-box public API, solid projections .... All I learned over the years is put into Event Machine: the importance of a functional core for example.
Working with Event Machine means you build a decoupled system from the beginning. Even if it looks like everything is bound to Event Machine (which is true to some extent), it is still very easy to remove Event Machine. Split the system into microservices, refactor parts of it .... This is because of the CQRS / ES architecture and the stateless implementation of aggregates (or event sourced processes) and immutable data types. The surrounding Event Machine layer is just glue code to put everything to work.
Should I use it in my application?
For prototyping definitely yes! If you should use it in the long run is something you can only answer yourself. If you like it and it works: Why not? If it gets in your way, remove it! The Fee Office shows you a way to split an application into modules. So you can decide for each module if you want to keep Event Machine another half year or if it is time to remove it for that module. That's the great advantage of this module system. Those decisions are cheap. In an early project phase you don't need a class for each command, event etc. If you recognize that your model is wrong, you want to throw it away and start from scratch. That's easy with Event Machine because developing a model with it takes only a few days. You have an idea. Just put it together with event machine and see if it works the way you thought about it. That's a really powerful approach and we use it with success ;)
Edit:
Event Machine docs contain an article about the topic, too. You can find it here: https://proophsoftware.github.io/event-machine/news/2018-09-08-future.html#4-1
To be honest because of my previous bad experience with payloads I don't like your way of describing the schema of commands and events and then using array payloads in the aggregate. I very much prefer simple value objects like
// Using PHP 7.4 syntax (immutability can be achieved using a trait)
class RegisterBuildingCommand {
public Uuid $buildingId;
public string $name;
}On the other hand I do like some other aspects such as stateless aggregates. I was actually thinking about something like that myself previously and except for the payload usage your Building implementation + the aggregate state do appeal to me.
I think I'll take some inspiration but ultimately won't use EventMachine directly. Are you interested in whatever I come up with?
I think I'll take some inspiration but ultimately won't use EventMachine directly. Are you interested in whatever I come up with?
Yes, absolutely!
PHP 7.4 syntax would be my preferred choice, too. I'm very interested in the trait that achieves immutability! So fare, I did not find a way that works. Without read-only access to public properties I'd still stick to array payloads.
There is another advantage of using a common message class:
Let me explain with a comparison:
Do you create an extra class for each http request or do you use PSR-7 interface instead?
It's the same with Event Machine's Message class. It's an incoming message carrying information from the outside world. We just skip some layers and pass it directly to an immutable handler (a stateless function belonging to a group of functions centered around a concept). Those stateless functions take information from the message and put it into domain specific data types: https://github.com/proophsoftware/fee-office/blob/master/src/RealtyRegistration/src/Model/Entrance.php#L17
This happens explicitly, without any hidden serializer. A general firewall makes sure that no message is passed into the domain that does not match with he defined schema of that message. Anyway, each data type can and should validate the information again. There are some short cuts like shown here:
https://github.com/proophsoftware/fee-office/blob/master/src/RealtyRegistration/src/Model/Building.php#L14
these short cuts are meant to be used during prototype phase only. In fact, if you want to use Event Machine in the long-run it is recommended to put another function between Event Machine and the aggregate function. Such a function would turn message payload into data types and call the aggregate function with those data types. So you remove also the message dependency.
Another problem to solve is type safety in projections. Without a common message interface I cannot type hint for anything in projectors. We don't have Generics.
Using $message->get(Payload::A_PROPERY) or $message->getOrDefault(Payload::A_PROPERY, 'alternative'), gives you the safety that either an exception is thrown (call 1) or a default is returned (call 2) if payload property does not exist. This and the fact that each Message MUST have a json schema definition makes an Event Machine implementation extremely robust, without loosing flexibility and dynamic language features.
Event Machine and all concepts used are designed to avoid developer bugs like typos, wrong configuration, etc. Everything that blocks you from actually reasoning about the domain is reduced as much as possible. We are super fast with Event Machine not because it looks nice but because it is optimized for RAD event sourcing with high quality results (very few bugs).
I'm very interested in the trait that achieves immutability! So far, I did not find a way that works.
That's actually rather easy if you know the right tricks. See my implementation (test included!). This implementation might have some issues with extends but I believe those can be solved too. I don't extend value objects anyway since I always declare them as final.
It is hacky of course and needs some polishing. I'd also recommend turning it off on production environment by ENV var based autoloading of the trait because of speed concerns.
How do you connect the EventMachine with an event-store implementation? I noticed that the fee-office project doesn't require prooph/event-store nor prooph/event-store-client so I'm not sure where to look or even if there is such example.
@enumag Each module defines its own dependencies: RealtyRegistration composer.json
That mechanism is explained here
I've planed to integrate composer-merge-plugin, which improves the current module merge logic a lot.
I have another question. Both the old prooph/event-store and the new prooph/event-store-client have transaction support. In my application I somehow never really needed transactions and I'm starting to think that might have been a mistake. So I'd like to ask about some use-cases when a transaction is needed. I'm particularly interested if a transaction can append events to multiple aggregates (which I assume is the point of a transaction) and what the domain structure would look like in such cases. I also assume a transaction should only affect aggregates in the same context, right? So far all of my commands only worked with one aggregate. Well some command did read things from other aggregates but they always only wrote into one. But if a command can use a transaction to write into multiple aggregates then I'm not sure where that command belongs. Overall I feel like this is beginner's confusion and everything will be clear if you tell me about some case where you use transaction and how is it connected to the domain.
Also in the fee-office I notices that there aren't many tests in RealtyRegistration, particularly no tests for the Building aggregate. Are tests something you don't need to bother with too much in this example, are they on the to-do list for later or do you somehow not need them thanks to the way you're describing the domain with event-machine?
@enumag one command => one aggregate is definitely the rule to follow. That's also a DDD suggestion if you don't use CQRS / ES. An aggregate is a transaction boundary.
Why is transaction support needed then? Well, one aggregate can record multiple events during one command processing. Those events needs to be stored all together or none of them. A single call to $eventStore->appendTo() would be enough, if transaction is handled internally. However, controlling the exception from the outside gives you more control over it and maybe you want to use the event store outside of a domain model (in some script or so) and the transaction feature could be useful there.
Regarding tests:
In prototyping phase tests don't make much sense, except you want to work test-driven from the beginning. But you are right, the way event machine works allows you to work with less tests and still get a very robust implementation.
As long as you're not sure that you got the model right, tests will only be in your way when you need to refactor. That's my personal opinion. Do whatever you prefer. I'd normally add tests later. Event Machine has a nice test mode and testing pure functions is super easy.
I don't test each immutable data type because I use PHPStorm templates to generate them, so they can't contain typos.
one command => one aggregate is definitely the rule to follow. That's also a DDD suggestion if you don't use CQRS / ES. An aggregate is a transaction boundary.
Alright. Then that brings me to another question. We'll actually need something like this later down the road too.
Let's say we have a project that sells something. So they also need to generate invoices for the orders. And (at least in EU) those invoices need to be numbered, sequentially. So I thought I'd have an aggregate like InvoiceNumberSequence and then when a new invoice is needed I would get the next number from that aggregate, record an event there so that the number is not repeated, and add this number to the invoice creation payload. If both the invoice and the sequence increment are saved in one transaction then everything is ok.
But you just said not to do that so I'm trying to think about that problem from a different perspective. The best I managed to come up with so far is this: 1. Create new invoice but have it without a number, just with a randomly generated uuid for now. 2. Notify the sequence aggregate that an invoice with this ID needs a number. The sequence will record a new event that the invoice was assigned a number. 3. Notify the invoice about the number assignment and store the number to the invoice.
I think you'll agree with that approach. The problem I see with it that maybe I wouldn't want an invoice without a number to appear in my projections because it's incomplete, you can't generate a PDF for it etc. That means that projections would not have an updater for the invoice creation event but would instead wait until the invoice gets a number. But then there is a problem because projection generally only receives the one event (or did in prooph 7) but in this case I also need the data from the previous event when the invoice was created. Which means I need to either duplicate the data into the number assigned event or to pull the aggregate from event store in the updater. Both solutions feel quite bad to me to be honest.
three solutions, all have pros and cons:
1.) Duplicate information in events
- bad for the event schema, but otherwise not much of a problem
2.) Pull aggregate from store
- Quick solution with consistent result. We do that in some cases where consistency is super important in the projection, but it couples aggregate state with read concerns. That's the disadvantage
3.) Make the query aware of incomplete invoices and exclude them from result set.
- That's the cleanest solution. There is an invoice without a number, skip it. However, when refactoring queries or adding new ones you always have to keep in mind that you need to exclude incomplete invoices. So that's the option with most effort.
Yeah these are the three solutions I was thinking about and none of them feels very good to me.
-
I don't like duplicating the information in events because I think the events should serve the needs of the domain and should not be changed according to the needs to their consumers. When that happens it will soon be forgotten which information in an event is important and which was just added for other reasons.
-
When pulling aggregate from the event store in such case we must also be careful to only load events up until the currently processed one but not any events after that. Otherwise results are inconsistent. And still it is unwanted coupling as you said.
-
This adds some unwanted complexity for whoever is reading the projection.
... Overall I'm not satisfied with any of that. Let me describe another solution I just came up with:
- When receiving event with data that are not needed yet (in this case the invoice creation event) but will be needed later, save it to some separate place in the projection. For example if projection is an SQL database, create a table with simple key-value storage and save json of that event there for later. Then when some other event triggers the need for those data, extract them, move them to their final location and delete them in this key-value temporary storage. The downside here is a bit more complexity in the updaters but the main advantage is that it doesn't have a negative impact on any other part of the application and doesn't couple the updaters with aggregate state.
What do you think?
Sounds very good. You're right that's another solution. Depending on the amount of events that you need to store temporary, you could also use projection state of prooph's projections: https://github.com/prooph/event-store/blob/master/tests/Projection/AbstractEventStoreProjectorTest.php#L1005