This is my self-study notes for the talk Microservices + Events + Docker = A Perfect Trio  by Chris Richardson

Success of a complex project relies on

  • Architecture
  • Process
  • Organization

Processes

  • consensus is in favour of Agile and continuous delivery/deployment

Organization

  • Consensus is that small, autonomous teams are better

Architecture choices

  • Microservices VS monolithic

Monolithic

  • Layered and modular implementations of different business functionality
  • All modules however gets packaged up as one application
  • Usually relies on one database too

Pros for monolithic

  • Easy to develop, test, deploy and scale in the initial phases of a project

Cons for monolithic

  • From a Process POV, with large, complex codebases, Agile and Continuous Delivery becomes hard to maintain.
  • From an Organizational POV, teams can no longer act with autonom```

Microservices architecture

3 different dimension to scale an application, as per The Art of Scalabilty book by ML Abbott & MT Fischer

X-axis scaling

  • Run multiple instances of your application behind a load balancer, ie like running multiple monoliths

Z-axis scaling

  • Run multiple instances of your application, but not behind a simple load balancer, it has a router that routes requests by inspecting requests for certain parameters.(Like sharding in DB world)

Y-axis scaling

  • Functional decomposition / breaking up application functions into separate applications/services.
  • Modules as per monolithic architectures are handles as separate, independent, scalable applications. Each one of which has its own DB
  • API gateway routes traffic to different microservices

Pros for microservices

  • Allows for Agile and Continuous Delivery → Process
  • Allows for small, autonomous teams → Organization

Cons for microservices

  • Complexity of
  • Developing a distributed system
  • Implementing inter-process communication
  • Handling partial failures
  • Handling individual DBs state
  • Maintaining consistency across multiple DBs can be a pain
  • Cannot use traditional 2-phase-commit. Distributed transactions are not viable in this instance.
  • Testing is harder, because of a whole tree of transitive dependencies
  • Deploying and operating
  • Implementing features that span multiple instances, services, and dependencies can be a pain
  • Docker and similar tech solves a lot of the operational and deployment complexity
  • Event management can ensure data consistency across distributed databases
  • For large, complex projects the benefits of microservices outweigh the drawbacks

Event-driven microservices

  • Case of using a shared DB between services

Event-driven Pros

  • Simple,easy to understand, and ACID(Atomicity, Consistency, Isolation, Durability)

Event-driven Cons

  • Tight coupling - Changing DB schemas would affect and hinder other services and the teams working on them

Case of each service having own DB

Pros

  • Loose coupling - DB schemas and teams can be run independently

Cons

  • Increased complexity for OPs people to manage
  • Distributed transactions(2-phase-commits) becomes unviable. How would consistency be maintained across different dbs?
  • Solution to consistency problem is to use event-driven architecture
  • All services publish to and can listen to a common event stream
  • By chaining together different events/transactions you can get consistency in your system

Chaining events implies that two things have to happen atomically(duel write problem)

  • Update DB
  • Emit and event

Possible solutions to the duel write problem

  • Application uses DB table as a message que
  • Event sourcing

Event sourcing

  • Event-centric approach to ensuring consistency
  • Using an event table, you can save a sequence of events that together changes the state
  • Updating the Event tables is one, singular action, thus solving the atomicity problem
  • Constructing the current state would simply mean replaying the events

Event sourcing Pros

  • Solves data consistency issue
  • Reliable event publishing to be sued in predictive analytics, notifications ect
  • Eliminates Object/Relational mapping problems - you’re not storing complex domain objects, you’re only storing relatively simple events
  • Keeps state reliable
  • Creates a reliable audit log
  • Allows for temporal queries
  • Keeps a complete record of preserved history, allowing for feeding past events in new features for more consistent testing

Event sourcing Cons

  • Requires application rewrites - core business logic needs to be restructured
  • Event-sourcing would require a new style of programming that you may not be used to
  • Events is a historical record of your bad design decisions
  • You need to have some way of dealing with duplicate events(idempotent handlers or duplicate detection)
  • Querying the event store can be challenging - might need to have combined or nested SQL queries to find specific entries

CQRS(Command Query Responsibility Segregation)

Possible solution is the architectural approach. You split your system in 2 sides;

  • Command
  • Query

Command contains the domain objects and handles the commands like POST,PUT,DELETE)

Query handles GET, that queries Materialized Views of data

The 2 sides are kept in sync by consuming the events from the event store(command sends events to event store, where queries pick them up and update MViews)

This is more complex than traditional monolithic queries, but is high performance and scalable

Deployments with Docker

Microservices requirements

  • Handel a large variety of languages, frameworks and versions of each
  • Handle multiple service instances throughput and availability
  • Build and deployments must be fast
  • Deployments and scaling must happen independently
  • Services must run in isolation in order to contain errors and malfunctions to only one service, not cascading to the whole system.
  • Deployments must be reliable and cost-effective
  • Service Container Pattern
  • Package a service as a container image
  • Deploy containers in different virtual machines

Benefits of containers

  • Great isolation of services
  • Easier manageability
  • Encapsulation of implementation technology
  • Efficient resource utilization
  • Fast deployments
  • Reduces developer onboarding pain wrt infrastructure versioning issues
  • Simplifies end-to-end testing

Summary

  • Use microservices to accelerate development
  • Use event-driven architecture to maintain data consistency across services
  • Use docker to simplify deployments