Research analysis on Spotify

Wed Sep 04 2024

Disclaimer


Given the size and scale of the application Spotify, some specifics are speculation as it is not open-source software. Where specifics cannot be found through Spotifys engineering blog or third party references, it is assumed to follow standard industry practices.

Software Used by the App


Front End:

  • HTML, CSS, and JavaScript
    • HTML is used to structure the web application's content.
    • CSS is employed to style these elements responsively.
    • JavaScript, integrated with frameworks like React, brings the application to life, handling user interactions, dynamic content loading, and communicating with Spotify's backend services.
  • React
    • React is a library for building user interfaces, known for its component-based architecture. This makes it well-suited for building complex web applications like Spotify, which consist of numerous dynamic elements like playlists, music player controls, search bars, etc.
    • Redux
      • Used with React to manage the application's state in a centralised store. The Redux store holds the application state (like the current user, playlist data, and playback state).
  • Native development kits
    • Swift (iOS)
    • Kotlin/Java (Android)
    • Native development kits enable apps to run more efficiently and faster compared to cross-platform solutions.
      • Despite the business tradeoff of developing separate applications (different engineering teams, resources, etc) the increased performance and security is worthwhile.

Back End:

  • Java
    • Java's robust ecosystem, performance, and reliability make it an ideal choice for implementing the logic that interacts with both Cassandra and Kafka.
    • Java's concurrency model and efficient JVM performance characteristics are well-suited for handling the high-throughput, low-latency demands of Spotify's backend services.
  • Spring
    • Spring is a back-end framework for Java similar to Flask for Python. The Spring Framework is a powerful, feature-rich framework for building Java applications. Favoured for its support for building web services, microservices, and handling data access, security, etc
  • Apache Cassandra
    • Cassandra is a highly scalable, distributed NoSQL database designed to handle large amounts of data. It's well-suited for applications that require scalability and high availability without compromising performance. Prior to using Apache Spotify used PostgreSQL.
  • Google Cloud Pub/Sub
    • Prior to Spotifys migration to Google Cloud Platform, Spotify used Apache Kafka for their event delivery platform. Cloud Pub/Sub was chosen for its low latency, and lack of capacity limitations with scale. Event delivery is used to transport requests from clients to Spotifys processing system. Pub/Sub handles 1 trillion requests per day.

Other:

  • Caching
    • Memcached
      • Used for caching sessions, db query results, APIs, objects, etc
  • Search Engine
    • Mainly propritary, third party options would include tools like Elastisearch.
      • Searching involves the following steps:
        1. Query Processing
        2. Candidate Retrieval
        3. Ranking
    • Two recent proprietary algorithms Spotify use for search are:
      • Crowdsurf
        • Crowdsurf is aimed to reduce search friction, it works by making the assumption the user has the same intent throughout the entire sequence.
        • Eg,
          • if a user searches 'RiRi', and then listens to 'Rhianna' it will store that correlation and weight it for future users in a bipartite graph.
      • Natural Search
        • Leverages word embeddings, or sentence embeddings to increase natural language search.
        • Eg,
          • if a user searches "eat better during xmas holidays"
          • if an episode description was "Alex Straney chats to Dr. Preeya Alexander about how to stay healthy over Christmas and about her letter to patients." it would be returned despite no exact word links.

Devops:

  • Docker
    • Docker allows software to be kept as a 'container', reducing overheads typically associated with VMs (such as running a full OS).
  • Kubernetes (Via Google Kubernetes Engine (GKE))
    • Used to manage and scale services independently. K8ns is used as orchestration software in conjunction with docker to provide a 'microservice' architecture.

Hardware Used to Host the App


Spotify migrated from on-premise hardware, to Google Cloud Platform between 2016-2017. (Google Cloud, no date) Prior to their cloud migration Spotify had some 2,000 services; 20,000 daily data pipeline runs; and more than 100 Petabytes of data, across multiple data centres in four different regions. Since then Spotify has moved almost exclusively to the Google Cloud Platform.

Cloud computing provides the benefit of easier horizontal scaling, however the underlying hardware itself is similar to on-prem. The larger benefits of migrating to GCP are:

  • Maintenance and Space
  • Existing Global Infrastructure
  • Security


Effectively Spotify can focus on development/maintenance of software, while GCP handles the physical infrastructure, from server maintenance, to power management, scaling, etc.

Interaction of Technologies Within the App


Front-End to Back-End Communication:
The client applications (web, mobile) interact with the back-end services via RESTful APIs or GraphQL, requesting data (e.g., user playlists, song metadata) and performing actions (e.g., play song, follow artist). These requests are distributed via a load balancer to the microservices.


Microservices and Cloud Infrastructure:
Each microservice handles a specific piece of functionality (e.g., account management, playlist curation), running in containers for isolation and scalability, managed by orchestration tools like Kubernetes. Cloud services provide the computational resources, storage, and networking capabilities to support these services at scale.


Data Storage and Retrieval:
The application queries databases for data and NoSQL databases for unstructured or semi-structured data, leveraging indexing and caching for performance. For network traffic (and therefore cost) reduction it will prioritise data from local storage at the client, or from the CDN prior to uploading from the database.

This system design provides a simplified view.

Embedded Image


Looking into Spotifys system design, and technologies provided some great insights into data at scale. I hope you learnt something too!