Lambda Architecture

In the Lambda Architecture website we have a brief history and description of the architecture. “Nathan Marz came up with the term Lambda Architecture (LA) for generic, scalable and fault-tolerant data processing architecture, based on his experience working on distributed data processing systems at Backtype and Twitter.
The LA aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being able to serve a wide range of workloads and use cases, and in which low-latency reads and updates are required. The resulting system should be linearly scalable, and it should scale out rather than up” [1].
 

Fig.1. Lambda Architecture (from http://lambda-architecture.net/)

 
In the site is also described how it works:

  1.  All data entering the system is dispatched to both the batch layer and the speed layer for processing.
  2. The batch layer has two functions: (i) managing the master data set (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views.
  3. The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way.
  4. The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only.
  5. Any incoming query can be answered by merging results from batch views and real-time views. [1]

The Lambda Architecture is for applications that need low latency (from few seconds to a few hours). A news recommendation system that uses various sources, processes and normalises the input, and then indexes, ranks and stores it, is a good example of an application of this architecture [2].
References

  1. Lambda Architecture, http://lambda-architecture.net/
  2. Questioning the Lambda Architecture , https://www.oreilly.com/ideas/questioning-the-lambda-architecture

Leave a Comment

Your email address will not be published. Required fields are marked *