Hi, this is Alexey Belozerov, author of PerfectPixel, a pixel-perfect design comparison tool used by 350k people. At some point of product growth an owner needs to launch experiments and measure impact against a reference group - i.e. A/B testing. Browser extensions are no different. Unfortunately there isn't any lib or tool tailored for Chrome/web extension development specifically, so I've decided to write my own and open-source it.

Browser extensions specifics

Compared to websites, extensions have a long update cycle - each new version should pass moderation and the process of rollout into the userbase, which can take up to several weeks. So it's critical to have a way to manage experiments asynchronously.

Requirements for extensions A/B testing library

1. Multiple experiments at a time
2. Extension user should be assigned to an experiment branch and persist its state
3. Remote kill switch and pin of the winning variant
4. Work in service worker, content script, sidebar, settings page etc.

The design

Config

I've started with designing the remote config as an orchestration point. Config is a static JSON file located on a remote server. Either a product owner or an AI agent can update the config based on the testing strategy and data collected and apply the changes to users' extension instances.

{
 "experiments": {
   "BUTTON_COLOR_1": { "active": true },
   "UPGRADE_LINK_1": { "active": true, "pin": "VARIANT_2" }
 }
}

We have a list of experiments that can be activated independently. When enough data is gathered and the winning variant is known, it can be pinned via config with the pin field.

To avoid any limitations with website CORS/CSP, the config is fetched from the service worker:

chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
  if (message?.type === 'GET_AB_CONFIG') {
    fetchConfig().then(sendResponse);
    return true; // async response
  }
});

Framework

The framework itself consists of entities:

1. A/B testing client - encapsulates storage and configuration including remote management.

   abTestingClient = createAbTestingClient(…)

2. Experiments - defined variants with assigned payload.

   experiment = defineExperiment(…)

3. Resolver - a function of the Client that uses Experiment data to attach the user to an experiment branch and persist state.

   abTestingClient.resolve(experiment)

Data storage

For Chrome Extensions I simply use chrome.storage.local with the following characteristics: survives updates, data kept until extension is deleted; website code or other extensions don't have access to this storage.

The data storage interface should allow using other sources in the future, like our API server, if we decide to assign experiment branches to a user account, not just an extension instance.

Usage

Content script:

import { defineExperiment, createAbTestingClient } from 'ext-ab-testing';
import { chromeLocalStorage } from 'ext-ab-testing/chrome';

const BUTTON_COLOR = defineExperiment({
  key: 'BUTTON_COLOR_1',
  variants: [
    { id: 'GREEN', color: '#27ae60' }, // 50% of users will be assigned to this variant
    { id: 'BLUE', color: '#2980b9' } // 50% of users will be assigned to this variant
  ]
});

const ab = createAbTestingClient({
  storage: chromeLocalStorage,
  config: () => chrome.runtime.sendMessage({ type: 'GET_AB_CONFIG' }) // Remote config fetch
});

const { active, variant } = await ab.resolve(BUTTON_COLOR); // variant assigned and persisted, next calls return persisted value

button.style.background = active && variant ? variant.color : some_default_color;

In the above example BUTTON_COLOR is resolved for each user uniformly at random; once resolved, the variant is persisted, and subsequent calls return the persisted value.

When “pin” is set in config, every resolve for that experiment will always return the variant that is pinned.

When an experiment is inactive resolve always returns { active: false, variant: null } and never touches storage.

If the config param Promise is rejected (remote config is unavailable), all experiments automatically resolve as inactive.

React

Experiments can be used on the frontend with the useExperiment hook:

import { useExperiment } from 'ext-ab-testing/react';

function UpgradeButton() {
  const { loading, active, variant } = useExperiment(ab, BUTTON_COLOR);
  if (loading) return null; // resolve in flight, no flash of a wrong arm
  if (!active || !variant) return <DefaultButton />; // kill switch off: default behaviour
  return <Button color={variant.color} />;
}

Measuring impact

Assigned experiment values should be tracked with analytics systems. Example of how it can work with Google Analytics:

const { active, variant, pinned } = await ab.resolve(BUTTON_COLOR);

if (active && variant && !pinned) {
  // resolve() returns the persisted variant on every call, so guard the
  // event with a flag - assignment is sent to GA only once per user
  const trackedKey = `AB_TRACKED.${BUTTON_COLOR.key}`;
  const alreadyTracked = await chromeLocalStorage.get(trackedKey);
  if (!alreadyTracked) {
    await fetch( `https://www.google-analytics.com/mp/collect?measurement_id=${GA_MEASUREMENT_ID}&api_secret=${GA_API_SECRET}`, {
      method: 'POST',
      body: JSON.stringify({
        client_id: clientId, // your persisted GA client id
        events: [{
          name: 'experiment_assignment',
          params: {
            experiment: BUTTON_COLOR.key, // 'BUTTON_COLOR_1'
            variant: variant.id // 'GREEN' or 'BLUE'
          }
        }]
      })
    });
    await chromeLocalStorage.set(trackedKey, variant.id);
  }
}

Register experiment and variant as event-scoped custom dimensions; they can be used to segment the audience and compare the conversion rate of target events.

Wrap up

An A/B testing library with management via remote config, easy to automate by an AI agent, allows you to conduct a series of predefined experiments on an already deployed extension.

The library is open-sourced under the MIT license and available on npm: ext-ab-testing