A/B tests

#Before you start

• Two versions. One side of a test is always the version that's live when you start. The other is any saved version you pick. So make the change you want to test first, then decide which version should be live.
• Real visitors. Each version needs about 100 visitors before the result means much. A preview address is kept out of search engines and rarely gets that many, so most tests belong on a site in production.
• Edit rights. Editors, Admins and Owners can start and end tests.
• A quiet moment. A test can't start while something is building, and a site runs one test at a time.

#Start a test

#How visitors are split

When you press Test against live, choose how many new visitors see the version under test: Half (50%), A quarter (25%), A tenth (10%), or Custom for any share from 5 to 95 percent. Then press Start test. The strip says how the split stands, for example “90% see v11, 10% see v12”. A smaller share is gentler on a bold change, but takes longer to give a clear answer.

Each visitor keeps the version they first saw, through a cookie that lasts 30 days. Every page they open comes from their version, so someone who reads the home page, leaves and comes back next week still sees the same one.

#What counts as a conversion

Each visitor counts once as a visitor and at most once as a conversion. Counts from one internet address are also capped at 20 an hour, so nobody can tilt a test by reloading.

#Read the results

The strip refreshes every 15 seconds while the studio is open. Each version has its own box: the share of its visitors who converted, then a line such as “18 of 410 visitors converted”. The version that was live when you started is marked “live before the test”. Under the boxes, one line sums it up:

The same line gives the Change in conversion (how much better or worse the version you picked does, in percent) and the Chance that it converts better.

#End the test and keep the winner

Keeping a version is the only way to end a test. Starting and ending one hold no credits, because both publish saved versions without building.

#What's locked while a test runs

Both versions are published together, so anything that would publish something new waits until you keep one. While a test runs you can't:

• ask for a change (“An A/B test is running on this site. End it by keeping one version, then ask for your change.”);
• undo, redo or press Make live;
• start another test;
• save code from the File Explorer, publish from the Visual Editor, or Replace a picture on the Media tab;
• use Apply to the live site, Publish or Back to preview;
• save the project as a template, or change its address.

Questions still work: you can ask Briks about the site at any time. See Asking questions.

#Worked example: a plumber tests two headlines

Harbour Plumbing is a sample business: a two-person firm in Bristol whose site is in production. Its hero headline reads “Plumbing and heating repairs across Bristol”. The owner suspects a promise about speed would bring more calls.