Craig A Holliday

Web and Mobile Automated OTT Media Playback Testing System

Only general information is provided for this project because this project is under an NDA

TL:DR

Challenge: Develop a system capable of conducting reliable and repeatable media playback tests.

Solution: Constructed an integrated testing system with components for:

  • A local OTT media service, offering origin and CDN simulations.
  • Essential services to support the application under test.
  • Advanced network manipulation tools to emulate various conditions.
  • A dynamic test runner catering to different platforms.
  • A comprehensive test orchestrator with a user-friendly CLI interface.

Tech Stack: Docker, Linux, Toxiproxy, Appium, oclif, Node.js, Typescript, Mocha, and Chai.

Project Goals

The main goal of this project was to create a system that could provide reliable and repeatable testing for media playback tests.

These are the services and functions of the system:

  • Local OTT media service (media, origin, CDNs, etc)
  • Mock APIs for applications the application under test required. (Configuration service, metrics server, CMS server)
  • Emulated network scenarios using forms of network manipulation
  • An application that can run and configure the system.
  • An application to manage, orchestrate, and execute media playback tests.

The desire for this system was to provide the system for developers to use daily and provide a long-running test runner to continuously run tests automatically.

Functions

  • Project owner
    • Defined all the project requirements.
  • Lead developer
    • Built most of the features
    • Managed a contracting team for some features
    • Managed a few team members from our core SDK team who helped with other features

Features

The system can be broken down into these parts:

  • The local OTT media service
  • The services for the application under test
  • Network manipulation
  • The test runner
    • Written in Typescript
  • The test orchestrator
    • Written in Typescript

The Local OTT Media Service

This set of services consisted of the following:

  • An origin server
    • A simple file server
    • Custom media content for CDNs
  • Emulated CDNs
    • HTTPS Reverse proxies
  • Proprietary applications for network coding

All of these services were deployed locally with Docker.

The Services for the Application Under Test

The application under test required these services:

  • Configuration Service
    • A custom service that managed the configuration of the SDK. For example, the URL to send metrics to and certain feature flags.
  • Metrics server
    • A service that could ingest the metrics output from the SDK
    • Metrics for a session would be pulled from this service and added to the test results
  • CMS Server
    • A server that would provide all the selectable assets available for the system. This allows the application to select any of the available content for testing.

All of these services were deployed locally with Docker.

Network Manipulation

We used a custom setup using Linux, namespaces, IP routing, and Netem for network manipulation in the system. For a few less intensive tests, we used Toxiproxy.

For network manipulation, the system could:

  • Manipulate throughput
  • Manipulate latency
  • Manipulate packet loss
  • Provide repeatable test scenarios that would run in a set timeframe.

The Test Runner

The test runner was a custom set of services in the core application that would take test inputs and drive the system and the application under test.

Notable features for the test runner include:

  • Drive the entire system from media available in the origin to clicking UI elements on the application under test.
  • Utilize Appium for driving Apple and Android devices
  • Could run thousands of tests and output test results locally and to remote locations
  • Support for custom test applications for Web platforms like Browsers, WebOS, and Tizen

The Test Orchestrator

The Test Orchestrator was the main entry point for users of the system.

Notable features of the Test Orchestrator:

  • CLI built using oclif
    • Node.js and Typescript CLI framework
  • Could take multiple inputs from users to define hundreds of different types of tests
  • Would deliver test results back to users in a human-readable format
  • Tests with Mocha and Chai

Project Outcomes

This system was used over the last year of the project to provide valuable data about performance for Apple, Android, and Web platforms.

Each complete test run for the system could run 1,600 playback sessions per platform to gather performance results.

Personal Takeaways

Out of all of my projects, this was my favorite to work on. I could work through the problem statement, architect the system, and execute the development.

While this was an extensive system, by the end of the project, I could start with a simple working prototype, provide value, and then build more features. This is the preferred way to develop a project because, with such complex problems to solve, it can take time to provide value quickly.

I had never done or seen the network manipulation for this project. Working with our research team to determine the best way to perform these network manipulations was enlightening. It is a complex and niche problem to solve but a fascinating problem.

There is a lot more this project could do in the future, and I’ll be looking at different ideas for projects that do similar work to this one.