Architecting India: Scale during the COVID-19 epidemic

Aerospike Marketing Blog, Technology, Business

Introduction

As the world spends more time online, some companies are able to scale up and provide an excellent experience for their customers and potential customers.    

We recently convened an architectural round table panel on BrightTALK to share successes from leaders in Online Gaming, eCommerce and Recruiting.  In this post, we’ll explain why being able to scale is important; explain how each of these companies approached it; and how you might be able to emulate their triumphs.

This blog post will provide the highlights of their challenge and architectural challenges – to get the full picture, please take the time to view the presentation in its entirety.

How Online Cricket is Scaling at Dream11

The first speaker on the panel is  Srijan Gupta Director, Backend for Dream11 –  India’s biggest fantasy sports platform;  playing cricket, football and many more.  With the lack of real life events; they have had many challenges to address and opportunities to be prepared for.

With a lack of sports in real life matches – the appetite to play virtual contests is exploding.  In a popular match such as Australia vs India in the field of Cricket, here are the typical scale challenges:

  • 2.7M concurrent users
  • 40M RPM read requests at edge (match metadata)
  • 20K/sec team updates per sec
  • Leaderboard (rank calculation of 50M user for 1 contest for every match event)

Key factors as we put together our platform included;  Availability,  Cost, Latency &  Elasticity.

Dream11 diagram - Serve Changing Contests Data to 2.5M Users

Figure 1:  To be able to address providing an excellent customer experience for serving the Contests, you can see the example of how Dream11 provided these using the Aerospike platform with Spark and Kafka streaming support.

How Online Recruitment is Scaling at Naukri

The next speaker on the webinar was Jai Prakash Sharma, EVP, Technology Operations, Naukri.  For those in India, Naukri needs no introduction, but for those in other parts of the world, they are the leading No 1 recruitment and job Portal in India.  A few statistics:

  • 100 mill. daily page hits powered by 
  • 1Billion+ http calls
  • 2 million daily job applications

To face the incredible challenges of the new normal such as being able to handle the influx of new job seekers, as well as those coming from other services that were no longer functioning, they had to overcome cost and complexity challenges with a cached architecture.   

In this roundtable, they share use cases around User Authentication and User Profile Lookup.  

You can see in Figure 2 the seamless streamlining their new platform provides them, and if you listen to the recording, you will hear his description of a better solution, but we will share here the bullet points of the advantages they currently enjoy with their Aerospike implementation:

  • Central Persistent Caching with HA Clustering
  • Secondary Indices for quick data retrieval with variety of search patterns
  • Computation Capability in caching layer
  • Support for huge data volume and queries
  • Persistent Cache Across Multiple Sites in form of XDR
  • Ease of scale (SSD))
  • Resource segregation in single cluster
Diagram: A high level overview of Naukri simplified, seamless scalability

Figure 2:  A high level overview of Naukri simplified, seamless scalability.

How eCommerce innovator SnapDeal is scaling.

If you’ve been following SnapDeal,  you’ll know that they are one of India’s largest e-commerce platforms, with over 100M app downloads,  60M+ products, 300,000 sellers.  Having an advanced data pipeline allows them to be agile when the Covid crisis called on them to scale. 

Their Director, Technology, Ashish Saxena went into details about their activities that required scale.  These include being able to identify users by platform and channel in real time so that they do personalization, vernacularization, gamification, search notifications,  multi-channel attribution and retargeting.

And to provide a bit more details on just what scale; Snap processes about 4 billion events a day which effectively translates to about 15 billion Aerospike hits with about  300,000 TPS [Reads:Write = 4:1] ; with read latency at 95P < 1ms | 99P < 8ms and write latency at 60P < 1ms |96P < 8ms.

How is this accomplished?    Here is the architecture diagram shared during the webinar:

Diagram: Aerospike architecture - real-time inputs realtime database updates

Figure 3:  The ability to combine real time user inputs with realtime database updates allows cross platform user identification and sessionization.

With this amount of real time insight, they were able to track changes in interest of what kind of products are people interested in; and have been able to leverage that with different experiments for different kinds of engagements.  

Also important, we have been able to keep up with changing delivery statuses —  because the sellers containment, those keep on changing based on the available delivery manpower. There has been a tremendous amount we have been able to observe during these times, and we are able to put them into practice to grow our business.

Closing

After each of our presenters had shared their experiences, they took a number of questions from the audience.  We’ll leave you with a few of the questions and we urge you, if you have time to schedule yourself time to listen to the entire webinar at your convenience.

  • Why would a gaming platform choose the Aerospike version of NoSQL?  
  • What data platform factors should an eCommerce startup focus on?  
  • As a bank,  what should I know about moving from a cache system to Aerospike? 
  • What were the business benefits for Snapdeal resulting from the low latency?

View the webinar here.

Architecting India: Scale during the COVID-19 epidemic

View Webinar

Share:

About Author

mm

    Aerospike Marketing

    All posts by this author