Two characteristics of Exp system:

1. Trustworthiness: the results can be trusted $\rightarrow$ solid statistical foundation, monitoring system
2. Scalability: adding new users/experiments easily with low cost

Four core components of an Exp system:

1. Experimentation portal:
   • Experiment management:
     • Audience: targeting & traffic filters
     • Overall Evaluation Criteria (OCE): key metric to determine the success of experiment
     • Size & duration: Power analysis
     • Experiment template: random imbalance
     • Experiment interactions: Isolation groups
     • Variant behavior: experiment owner defines behavior for each variant.
   • Metric definition & validation:
     • Metric Definition Language (MDL):
       • MDL is language-independent.
       • The user-provided metric definition gets translated into an internal representation, which in turn can be compiled into a number of target languages such as SQL.
       • MDL ensures that the definition of a metric is consistent across all such pipelines.

• Result visualization & exploration
  • Quick (near real time) checks are performed
  • The goal is to serve as a single place where experiment results & analysis is collected.
  • Simplest form of experiment result is experiment scorecard:
    • Collecting observed metrics for control and treatment
    • Calculate the delta (absolute and relative)
    • Statistical test
  • In case of many experiments, it’s important to have a tool that automatically identify and highlight interesting insights.

2. Experiment execution service
   • Goal: generate experiment variants and delivering them to the product.
3. Log processing service
4. Analysis service