In the Lambda Architecture website we have a brief history and description of the architecture. “Nathan Marz came up with the term Lambda Architecture (LA) for generic, scalable and fault-tolerant data processing architecture, based on his experience working on distributed data processing systems at Backtype and Twitter.
The LA aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being able to serve a wide range of workloads and use cases, and in which low-latency reads and updates are required. The resulting system should be linearly scalable, and it should scale out rather than up” [1].
Fig.1. Lambda Architecture (from http://lambda-architecture.net/)
In the site is also described how it works:
- All data entering the system is dispatched to both the batch layer and the speed layer for processing.
- The batch layer has two functions: (i) managing the master data set (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views.
- The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way.
- The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only.
- Any incoming query can be answered by merging results from batch views and real-time views. [1]
The Lambda Architecture is for applications that need low latency (from few seconds to a few hours). A news recommendation system that uses various sources, processes and normalises the input, and then indexes, ranks and stores it, is a good example of an application of this architecture [2].
References
- Lambda Architecture, http://lambda-architecture.net/
- Questioning the Lambda Architecture , https://www.oreilly.com/ideas/questioning-the-lambda-architecture
