Craig A Holliday

Data Validation Tool

Only general information is provided for this project because this project is under an NDA

TL:DR

Challenge: Lengthy validation process for metrics output from SDKs.

Solution: Designed a compact application for swift record ingestion and immediate metrics validation with a user-friendly Web UI.

Tech Stack: Fastify, MongoDB with Mongoose, Svelte, deployed via Terraform and AWS.

Project Goals

This project came about because there was no easy way to test the metrics output from the SDKs the team was building.

These SDKs would collect thousands of records during a session and send the records to an application for large-scale data ingest and processing.

The process of data ingesting and processing could take hours.

This was an issue when the SDK developers needed to quickly validate that the appropriate metrics were collected during a session.

Therefore, we required a small application to ingest the records without the overhead of storing and processing necessary for production usage.

This application needed to:

  • Emulate the production application so the SDK can communicate the same way during development
  • Validate metrics and provide Pass/Fail information
  • The developers needed to run a session, request the Data Validation Tool, and quickly see if the metrics from the session were valid.
  • Deliberately not scale so that the records could be cleared frequently.
  • The overhead of scaling the application was optional because the data was validated immediately and then could be deleted.
  • Eventually, provide a Web User Interface to make searching for sessions and data faster.

Functions

  • Project Owner
    • Defined all the project requirements.
  • Lead developer
    • Developed the main features for this project.
    • Other team members were able to contribute to fixing specific issues found during the development of their respective platforms.

Features

The application used:

  • Fastify for the server framework - A simple framework I am familiar with.
  • MongoDB for the database - The database I am most familiar with. - I also used Mongoose for the object modeling.
  • Svelte for the Web User Interface - I experimented with Svelte on this project. It was perfect for this application.
  • Terraform and AWS for deployment/hosting - Though the DevOps team managed this work

Some of the features included:

  • Endpoints for the SDKs to post the collected records
  • Saving the data to MongoDB using Mongoose
  • Endpoints for retrieving the raw records. - This could be useful if the individual records for a session needed to be reviewed.
  • Endpoints to return information about the validity of a session - The validation included variables such as specific amounts of record types, record values in certain ranges, the absence of particular record types, etc.
  • A User Interface that developers could use to review all available information.

Project Outcomes

This tool was used by all developers on the team to validate records collected for sessions and to validate expected values for specific sessions.

This tool was handy when the Web team implemented the record schemas from scratch where the Mobile teams already had implementations.

Finally, this tool provided a way to validate that changes to the SDKs did not cause regressions to the record collection functionality.

Personal Takeaways

The Data Validation Tool was more valuable as we went through development.

If anything, I would have built this tool sooner and been more strict about the team utilizing the tool.

This type of tool is required when building metrics collection into SDKs or applications.

It is the best way to test the output of an application quickly, but more importantly, the output can validation for the output can be shared with other team members.