Split tests

A before-and-after comparison cannot separate your change from seasonality or a search update. A split test can. Relay changes a random half of similar pages, keeps the other half as they are, and compares the two groups in Search Console.

You find split tests under Experiments & Results.

Which pages can run a testCopy Anchor Link

A test needs one group of pages that share a template, such as every page under /integrations/.

1. Click New split test. Enter a name, your hypothesis, the field you change and a page pattern, such as ^/integrations/[^/]+$.
2. Relay lists the matching pages from the last 56 days of Search Console data and checks:
   • at least 30 pages with a click, and at most 200;
   • at least 1,000 clicks in total;
   • that every page comes from the same route file in your repository;
   • that the group can detect a change of 15% or less.
3. If the group fails a requirement, Relay says which one and by how much, and creates nothing.

The minimum detectable effect (MDE) is the smallest change the group can reliably show. "This group can detect a change of 11% or more" means that a smaller real effect will probably show as inconclusive.

How the groups are madeCopy Anchor Link

Relay sorts the pages by clicks, pairs neighbours, and puts one page of each pair in the variant group at random. It draws again until the two groups have similar clicks and moved alike before the test. The seed is stored, so anyone can repeat the assignment. A page never changes group during a test.

The pull requestCopy Anchor Link

1. Click Review pull request. Relay shows the exact files it adds: a list of the variant pages and a helper, inSplitVariant. The helper reads only that list, so search engines always see the same version of a page.
2. Choose who writes the template edit. A developer can make it, inside generateMetadata or the metadata export, as the pull request description shows. Or Relay can draft it: text after or before the current value, or another field such as integration.name. Relay checks each variant title or description against the length and wording rules before you approve.
3. Click Approve and open.
4. Click Check pull request. Relay checks that only the three files changed, that the list matches the test, and that the helper is used only inside the metadata.
5. Merge the pull request and deploy it. Click Confirm deployment. Relay uses GitHub's production deployment of the merge commit, or your confirmation.

The checker supports App Router templates.

While the test runsCopy Anchor Link

• Relay checks the live site after the deployment. The test starts when at least 95% of pages show the expected title or description. It checks again on days 7 and 21. If more than 5% of pages drift, the result is marked contaminated.
• Relay excludes the deployment day and the next 6 days, then observes 28 days.
• The daily job refreshes the data. You can also click Refresh now.
• If the estimate is 10% or worse after 14 days, with strong evidence of harm, Relay tells you, and you can remove the test early.

Reading the resultCopy Anchor Link

Relay reports an estimated effect on clicks, a 90% interval and a p-value, from an exact randomization test on the pairs.

Before the window ends, the result is marked Interim, and it is never labeled positive. If the groups moved differently before the test, the result is marked imbalanced and gets no positive or negative label.

If Google Analytics is connected, the result also shows the estimated effect on organic sessions and key events that start on the test pages. Relay estimates them the same way, and they never change the label.

Limits. The label comes from Search Console clicks only. Internal links and competition between pages of the two groups, and any change that reaches only one group, can bias the estimate.

Check it yourself. Click Download CSV, download the reference script, and run python3 split-test-reference.py export.csv. It needs only Python 3, and it recomputes the estimate, the interval and the p-value with the same method and seed.

Backtest on your dataCopy Anchor Link

Click Run backtest to see how the test would have behaved on your own history. Relay replays it 200 times on the 91 days before its baseline, with new random groups each time:

• With no change, the 90% interval should exclude 0 in about 10% of runs. More than 12% means the group gives too many false results.
• With a synthetic +15% lift on the variant pages, the share labeled positive shows the test's power.

The backtest makes one Search Console request and changes nothing. A group that cannot run a test offers Run a backtest anyway, so you can see how it would behave.

Ending the testCopy Anchor Link

• Apply to all pages opens a pull request that keeps the variant for every page and removes the split code. It is available when the window ends without a negative result.
• Remove test opens a pull request that keeps the current values and removes the split code. It is available at any time.

You review the exact change before Relay opens either pull request.