EventCatalog v2 - What's new, thoughts and plan
boyney123 opened this issue · comments
Hello folks,
EventCatalog started about 2 years ago as a random idea focused on one thing, helping people document their event-driven architectures.
Over the past couple of years, the project has naturally grown with 13k catalogs created, 44 contributors, 600 discord members and 1.3k stars. More and more folks are looking to EventCatalog to help them document their EDA, great!
This community has grown and it's been great to watch, and just want to thank you all for using EventCatalog
As I continue to evangelize EDA, governance is becoming more and more important and area of concern for many folks. The ability to discover events, document them as well as services, domains, commands and much more!
I have had time to think about EventCatalog, what we have learnt over the past couple of years, and how some refactoring and new features can help move this project further!
Here are some initial thoughts, your feedback is welcome!
Moving to a container based application
Many folks are using EventCatalog as a static site generator, and many folks have asked in the past about key features that would require some kind of server to be there. Also many folks are trying to host EventCatalog (statically) on their own infustructure or in the cloud, for many this seems to be a challange based on the routing cofiguration in the app etc.
I want to make it as easy as possible for people to host EventCatalog whilst also providing some new feature support (integration with their brokers, API support, single sign on etc), looking into the future of features for EventCatalog, I believe it makes sense to focus on container support, allowing anyone to host EventCatalog anywhere.
EventCatalog will remain to be focused on markdown generation and ease of use, it just means many more options are open to for us as a community to support.
Keeping things simple
EventCatalog has evolved over the past 2 years, with support for domains and services, 3D node graphs and much more. Adding support for things like 3D node graphs is cool, but requires third party dependencies that may not be required. So I want to bring EventCatalog back to basics this means:
- Removing 3D node graphs (https://app.eventcatalog.dev/overview/)
- Simplifying the folder structure for events, services, domains, keeping things flat and linking them with ids
- Moving away from Lerna (monorepo) into Turbo for faster project and development
- Moving to glob based reading on the file system, meaning faster reads and bigger project support (1000s of events etc)
- Plugin documentation, focusing on the community to add plugins into the architecture
Extra features
There are many more features that EventCatalog can support, and with the move to a server based application (using containers), this opens the door to some more interesting features and things to help you document your EDA. But some initial features will be:
- Ability to document commands
- Moving to dagre for layout support with the visualiser (better visuals and simplifying the code here)
- Ability to use any component (MDX) on events, commands, domains and services
- Ability to version events, commands, domains and services with diff support
- Support for teams and users, ability to add teams or users to events, commands, domains and services
Long term feature ideas
- Single sign on support
- Ability to be notified of schema changes
- Hosting options
- Support
What this means for you?
The project will still be focused on markdown files that power your catalog. Moving to V2 will require some structure changes to your catalog, but hopefullly migration docs and tools will be there to support you. Nothing too bad (hopefully!).
EventCatalog has proven to add so much value to people around the world, and in ways I feel it's just getting started. Hopefully you are on board with this vision and changes, and I would love your feedback below.
Governance is a big problem, but I believe we can fix this, work together and build something cool along the way!
static site generation will still be possible, correct? Container deployment is an optional feature?
Hey @boyney123
I think in cases, the domains folder scaffolding was useful, when a domain has many services representing too many events, this will be hard with a flat way , but having markdown side by side for each service seems ok
static site generation will still be possible, correct? Container deployment is an optional feature?
Hey @donaldpipowitch need to think about that, I think with some server support much more could be done to help govern things. How much would this effect you?
Hey @boyney123 I think in cases, the domains folder scaffolding was useful, when a domain has many services representing too many events, this will be hard with a flat way , but having markdown side by side for each service seems ok
Hey @XaaXaaX yeh should be able to do whatever you want to do. So if you want all events in /events thats fine, if you want events in domains like /domains/mydomain/events/* this should fine. The glob system will fetch all types regardless where they are as long as they would be in an events folder. That's my thinking anyway. So quite flexible!
Would be great to have support for multiple environments, I guess most people or companies are having for example DEV / TEST / ... / PROD as environments.
Would be great if you could see which versions of events / schema's are active on which environment.
Backstage plugin/easy way to integrate with IDPs.
A REST API to manage events, services, ... would open-up extra integration options.
Would be great to have support for multiple environments, I guess most people or companies are having for example DEV / TEST / ... / PROD as environments. Would be great if you could see which versions of events / schema's are active on which environment.
Yeah need to figure this out, but 100%.
Backstage plugin/easy way to integrate with IDPs.
Yeah be interested to connect with people that want to integrate with Backstage, EC plugin here, not done much with backstage, do people see EC working along side it or backstage replacing EC?
A REST API to manage events, services, ... would open-up extra integration options.
Yes! Any ideas on the kind of things you would want to do? Things that come to my mind are download schemas, upload new versions of schemas etc,
For the REST API, it would be useful to provide CRUD operations for the users, domains, services, and all the aspects of events. For events specifically:
- bump versions
- download schemas
- edit event properties
People could then build their integration/service/pipeline (next to the plugins), in case they have special transformation needs or certain schema registry tool that they want to combine and centralize in EventCatalog.
Right now EventCatalog is an easy sell to clients and development teams. They see the value, the learning curve is low, and tossing it up in an S3 bucket and CloudFront is simple and generally costs nothing. (Pennies at most.)
I don't want to lose that. There are hosted platform that cost $10/user/month; I don't even try to justify those to clients.
Hey @donaldpipowitch need to think about that, I think with some server support much more could be done to help govern things. How much would this effect you?
It would be awesome, if static export would still be possible. Similar reasons as @kernwig said.
Would be great to have support for multiple environments.
We create an event catalog instance per stage. This seems to be a good approach - especially if you would work in a company where new stages can be dynamically created and removed.
A REST API to manage events, services, ... would open-up extra integration options.
If event catalog wants to support cases like this an approach similar to next-auth might make sense. You create your own Next application and then you just mount any page/api routes offered by event catalog that you need/want.
I personally would like to see better support of existing features in v2. Like getting mermaid support in domains (#407) or custom React component usage (similar to how it's done in Storybook).
@boyney123 thanks so much for all your work on EventCatalog.
One of the design aspects that complicates things for us in the current version is the fact that if you want to add consumers you must do this in the Event markdown files. Personally I like the idea that an Event is owned by a producing service, but as in most event systems, it doesn't really care who consumes it. The consumer itself needs to be aware of the event though. In light of that, would it be possible to define the events a consumer is subscribed to in the consuming service itself?
We were looking into the idea of allowing each event producing service to define their own EventCatalog service and the events they produce in their git repo, and similarly allow each consuming service to define their service definition in their own repo (which would ideally include references to the events it consumes). Then in our CI/CD we could pick these configurations up as a artefacts and push it to our central EventCatalog.
If for example a consumer is updated to consume another event, we only need to change the consumer configuration in its repo, this will get pushed to the central EventBridge repo, which can then add this service as a new consumer of the event, without needing to update the event configuration itself - which should really be owned by the event producer.
I hope that makes sense.