Telemetry/Experiments

	THIS PAGE MAY BE OUTDATED
	This article is in parts, or in its entirety, outdated. Hence, the information presented on this page may be incorrect, and should be treated with due caution until this flag has been lifted. Help by editing the article, or discuss its contents on the talk page.

Telemetry Experiments is a feature that allows Firefox to automatically download and run specially-designed restartless addons based on certain conditions.

Experiments are available to anyone who has Telemetry enabled, which is the default on pre-release versions of Firefox. This feature is currently only available in desktop versions of Firefox.

Experiments can be targeted at various populations by specifying conditions in an experiment manifest. http://gecko.readthedocs.org/en/latest/browser/experiments/experiments/manifest.html has details on the possible conditions, and there it is also possible to write complex conditions in JavaScript which can filter on any data in the telemetry or Firefox Health Report payloads.

This page provides a high-level overview of the process to build and deploy an experiment.

Product Approval

Product approval is required to run an experiment. User-facing experiments should be approved by a Firefox product manager (Jeff Griffiths). For experiments which have no user-visible effects, this approval is trivial.

Data Collection

For experiments that intend to collect new data, a data collection review is required. Before building such an experiment, contact bsmedberg or a data-collection peer to talk about experiment design and how the results of the experiment will be measured and reported. Depending on the nature of the experiment and its data-collection system, he may require additional privacy or security review.

Experiment designer should have clear questions that are being answered and should have a written plan for analyzing data to provide answers. If you need help to come up with a plan, please talk to the analyst or data engineer that works with your team. If you don't know who that is, please talk to Rebecca Weiss or Ryan VanderMeulen.

Once collected, you'll analyze the data using Map/Reduce tooling: Refer to the wiki page Custom Telemetry Analysis with Spark.

Engineering

Experiments are a special type of restartless extension. They can do anything a regular restartless extension can do. Detailed instructions on the mechanics of developing and testing the experiment can be found at Developing a Telemetry Experiment.

Note that Telemetry Experiments need to be signed to run in FF 42+, just like all other add-ons. There are more details on the signing process on the page linked above.

Extension Lifecycle

Most experiments live for a fixed period and are uninstalled by the experiment system at the end date or after the specified elapsed time.

A few experiments are intended to collect data once and then uninstall themself. These experiments should uninstall themselves when done by calling Experiments.disableExperiment("FROM_API").

Staging

To deploy an experiment, write a patch to http://hg.mozilla.org/webtools/telemetry-experiment-server/ for experiment deployment, as described at Developing a Telemetry Experiment, and bsmedberg or a peer can review. Pushes there auto-deploy to the staging server at https://telemetry-experiment-dev.allizom.org/.

QA Testing

All experiments must receive a QA sign-off before being deployed. For experiments that involve a nontrivial amount of new code or changes, the QA team is available to verify the experience on the staging server, including data collection. Teams with dedicated QA resources can also validate their experiments on their own if they so choose. To request QA sign-off, refer to the QA Guide For Telemetry Experiments - Benjamin Smedberg is the QA lead for experiments.

Final Deployment

Final deployment of an experiment must be approved by Lawrence Mandel or a release-management peer. Send an email to release-drivers@mozilla.org and cc legal-bd-telemetry@mozilla.com with the following information:

• A brief summary of the experiment, the questions that will be answered, and how the experiment will answer the questions
• bug(s) where the code was developed and reviewed
• details about experiment deployment: what channel(s) will the experiment be deployed to, how many users will be participating, and how long will the experiment run? This data can be copied from the staging server.

Here is an example of an approval email:

To: release-drivers@mozilla.org
Cc: legal-bd-telemetry@mozilla.com
Subject: Intent to ship: Telemetry experiment for verification of telemetry experiments bug

This is a request for approval to ship the following Telemetry experiment: 
Telemetry experiment for verification of telemetry experiments bug.

Summary
In bug 1052714, I plan to ship a telemetry experiment to 50% of nightly channel users to verify that a fix
for a race condition setting telemetry experiment branches is actually fixed. This experiment will have no
user effect, and only exists to test the system. The results will be collected and analyzed from FHR
payloads.

Related Bugs:
bug 1052714

Start Date
    14-Aug-2014
End Date
    10-Sep-2014
Maximum runtime
    7.0 days
Update Channels
    nightly
Build ID
    20140820030202- 
Instance population
    All nightly instances
Sample Rate
    50%

After final approval, file a bug to deploy to production at https://telemetry-experiment.cdn.mozilla.net/ . Use bug 1057464 or bug 1131864 as a model for deployment.

Monitoring

Use http://bsmedberg.github.io/telemetry-experiments-dashboard/ to monitor experiment adoption.