Experiment Tracking: What it is, Best Practices & Tools in 2024

Machine learning is changing every industry and business function with hundreds of applications and use cases. However, AI and ML projects have a high rate of failure. Inspired by DevOps practices for software development, MLOps brings efficiency to machine learning model development processes and reduce the chance of failure for ML projects.

Developing ML models involves lots of trial and error, or experiments. Tracking these experiments in a structured way is an integral part of every successful ML project. In this article, we’ll explore one of the components of MLOps practices: experiment tracking.

What is experiment tracking?

Experiment tracking is the practice of keeping track of important information (metadata) about different experiments when developing a machine learning model. Experiments and metadata about them can mean:

• Different ML models
• Model hyperparameters such as the size of a neural network
• Different versions of training data
• Codes used in model development

This is a non-exhaustive list and the important metadata about experiments depends on the project.

Why is experiment tracking important?

Tracking ML model experiments in a structured manner enables data scientists to identify the factors that affect model performance, compare the results, and select the optimal version.

A typical process of developing an ML model involves collecting and preparing training data, selecting a model, and training the model with prepared data. A small change in the training data, model hyperparameters, model type, or code that is written to run the experiment can drastically change model performance. 

Data scientists usually run different versions of the model by changing the components of the model. Hence, achieving the best-performing model according to one or more performance evaluation metrics is an iterative process. Without tracking the experiments conducted during the ML model development process, it is not possible to compare and reproduce the results of different iterations.

How can experiment tracking be implemented?

Manually recording all the information about different experiments to spreadsheets is an option for tracking experiments especially if you don’t run too many experiments. However, machine learning projects typically involve numerous variables to track, and these variables have complex relationships with each other. Therefore, manually tracking experiments can be time-consuming and hard to scale.

Fortunately, there are tools that are designed to track machine learning experiments. These tools:

• Provide a hub to store different ML projects and their experiments
• Can be integrated with various model training frameworks
• Automatically register all the information you want about experiments
• Have user-friendly UI to search and compare experiments
• Leverage visualizations to represent experiments which helps users interpret the results quickly. Visualizations also help in communicating the results to others with no technical background.
• Can let you track hardware consumption of different experiments

What are the best practices for tracking ML experiments?

To achieve the most of your ML tracking, you need to define:

• the objective of the experiment
• evaluation metrics (accuracy, explainability, etc.)
• experiment variables (different models, hyperparameters, datasets, etc.)

For example, if you are trying to increase model accuracy, determine the accuracy metrics and hypothesize “If we use the model X, it will deliver more accuracy compared to model Y”. Trying several things without a framework and picking the best one is counterproductive if you haven’t decided what makes an experiment successful.

What are the tools for experiment tracking?

There are both open source and commercial tools for tracking experiments. Some popular tools include:

Experiment tracking is a part of the MLOps practices to streamline machine learning model development processes. MLOps platforms that provide end-to-end machine learning lifecycle management also include tools for experiment tracking. Some MLOps platforms are:

• Amazon SageMaker
• H2O.ai
• Iguazio
• MLFlow

You can also check our MLOps tools article for a comprehensive account.

If you have more questions about experiment tracking and about vendors, we would like to help: