Implementing A/B Test for Search

SearchUnify admins can run A/B Testing on Search Clients to see which search variant leads to a higher click-through rate (CTR), average click position (ACP) and case deflection (Deflection). Three search variants are available in SearchUnify:

  • Keyword Search

  • Neural Search

  • Hybrid Search

Once an A/B Test is set up, search traffic is split between the selected variants based on the proportions and time period defined by the SearchUnify admin. After the test ends, detailed reports are generated for each variant. These reports help admins compare key search performance metrics and make data-driven decisions to enhance the user experience.

Prerequisites

Start an A/B Test for Search

  1. Log into SearchUnify.

  2. Click A/B Test.

    NOTE. The A/B Test feature has been introduced in the Q2 '25 release. Customers on an older version will not see the tab..

  3. Click New A/B Test or Configure.

    NOTE. If Neural or Hybrid search isn't active for your instance, you will see this error message: Neural/Hybrid search is not provisioned for your account. Please contact the support team for assistance.

  4. Enter the following details:

    • Name - Provide a name for your A/B Test.

    • Search Client - Select a Search Client where the A/B Test will run.

    • Select Search Methods - Select the search variant. Three options are available:

      • Neural Search

      • Keyword Search

      • Hybrid Search

      You can select either two variants or all three.

    • User Traffic Distribution - This is a toggle where you can select how the search traffic is distributed among each method:

      • When two variants are selected, the default values displayed are 50% and 50%.

      • When three variants are selected, the default values displayed are 10% (Neural), 80% (Keyword), and 10% (Hybrid).

    • Select test duration - Slide the toggle to run the test for as short as a day and as long as 90 days. The default value is 7 days.

  5. Review the details.

    NOTE. You cannot edit A/B Test, once it's up and running.

  6. Once you're sure of all the parameters, click Create.

  7. A dialog box is displayed, showing the creation process of up to three sub-search clients, each corresponding to a selected variant. Once the search clients are created, click Okay.

  8. You'll see the A/B Test screen. Click Start A/B Test to kick off your controlled experiment.

Delete or Stop an A/B Test

SearchUnify admins can stop or delete an A/B Test report. All active reports have the tab Running or Running (<n> days left) label on them.

  • To stop a test, click Stop A/B Test.

  • To delete a test, click Discard A/B Test.

Fig. A snapshot of A/B Testing report.Fig. A snapshot of A/B Testing report.

The difference between stopping and deleting is that the stopped test remains visible on the A/B Test tab.

Key Metrics

This section defines the three search metrics in the context of A/B Test for Search.

  • CTR: It is the Click-Through Rate. All searches with clicks divided by the total number of searches. Example: If there are 70 clicks for 100 searches, then the CTR is 70%.

  • ACP: The average position where the users click the search results.

    Example: If there are three clicks on positions 2, 5, and 7 during a test period, then the average click position for that test period will be (3 + 5 + 7) / 3 = 5

  • Deflection: This is the Stage One Deflection.

Instance-Level Changes by A/B Testing Activation

When A/B Test for Search is turned on, then the following changes take place in your SearchUnify instance.

Search Clients

A/B Test runs on Search Clients. You can see up to three sub-search clients on the Search Client screen.

Fig. A snapshot of the sub-search clients on the Search Clients screen.

Each sub-client is a test variant. When you open a sub-search client for editing, only the only Relevancy tab is open for configuration.

Fig. A snapshot of the editable tabs in a sub-search client.

When you open a main search client for editing, you can make changes on the following tabs:

  • Configurations

  • Analytics Settings

  • Advertisements

  • End-User Feedback

These tabs cannot be altered:

  • Content Sources

  • Relevancy

  • Language Manager

Fig. A snapshot of the editable tabs in a search client where A/B testing is on.

The only way to get complete testing capability back is to stop or delete the A/B Test running on the Search Client.

You can run only one A/B Test on a search client at a time.

Search Analytics

The sub-search clients are displayed in the Select Search Client/Ecosystem dropdown. You can select a sub-search client to view its analytics data.

Fig. A snapshot of the sub-search clients on Search Analytics.

Search Tuning

You can run Auto Boosting for each sub-search client individually.

In Manual boosting, only Content Boosting is available for the sub-search clients.

Fig. A snapshot of the editable tuning tabs for a sub-search client.