Event-Driven Workflow: Monitoring and Orchestrating Your Microservices Landscape with Kafka and Zeebe
A company’s core business processes nearly always span more than one microservice. In an e-commerce company, for example, a “customer order” might involve different services for payments, inventory, shipping, and more. Implementing long-running, asynchronous, and complex collaboration of distributed microservices is challenging. How can we ensure visibility of cross-microservice flows and provide status and error monitoring? How do we guarantee that overall flows always complete, even if single services fail? Or how do we at least recognize stuck flows so that we can fix them?
In this talk, we’ll demonstrate an approach based on real-life projects using the open-source workflow engine Zeebe (zeebe.io) to orchestrate microservices. Zeebe can connect to Kafka to coordinate workflows that span many microservices, providing end-to-end process visibility without violating the principles of loose coupling and service independence. Once an orchestration flow starts, Zeebe ensures that it is eventually carried out, retrying steps upon failure. In a Kafka architecture, Zeebe can easily produce events (or commands) and subscribe to events that will be correlated to workflows. Along the way, Zeebe provides monitoring and visibility into the progress and status of orchestration flows. Internally, Zeebe works as a distributed, event-driven, and event-sourced system, making it not only very fast but horizontally scalable and fault tolerant--and able to handle the throughput required to operate alongside Kafka in a microservices architecture. And because Zeebe uses gRPC for client-server communication, it’s possible to generate clients in one of ten gRPC-supported programming languages --making Zeebe accessible to a wide range of language communities.
Expect not only slides but also live hacking sessions and real-life stories!