Implementing A/B Test for Search

SearchUnify admins can run A/B Testing on Search Clients to see which search variant leads to a higher click-through rate (CTR), average click position (ACP) and case deflection (Deflection). Three search variants are available in SearchUnify:

• Keyword Search

• Neural Search

• Hybrid Search

Once an A/B Test is set up, search traffic is split between the selected variants based on the proportions and time period defined by the SearchUnify admin. After the test ends, detailed reports are generated for each variant. These reports help admins compare key search performance metrics and make data-driven decisions to enhance the user experience.

Prerequisites

• Provisioned Neural or Hybrid search

Start an A/B Test for Search

1. Log into SearchUnify.

2. Click A/B Test.

3. Click New A/B Test or Configure.

   Note. If Neural or Hybrid search isn't active for your instance, you will see this error message: Neural/Hybrid search is not provisioned for your account. Please contact the support team for assistance.

   

4. Enter the following details:

   • Name - Provide a name for your A/B Test in this field.

   • Search Client - From the dropdown menu, select a Search Client where the A/B Test will run.

   • Select Search Methods - Select two or more search variants from the following options:

     • Neural Search

     • Keyword Search

     • Hybrid Search

   • User Traffic Distribution - Configure traffic distribution between the search variants.

     • When two variants are selected, the default values are 50% (Variant One) and 50% (Variant Two). This means that, out of 100 searches, 50 searches are handled by Variant One and 50 by Variant Two.

     • When three variants are selected, the default values displayed are 10% (Neural), 80% (Keyword), and 10% (Hybrid). This means that, out of 100 searches, 10 searches are handled by Neural Search, 80 by Keyword Search, and 10 by Hybrid Search.

   • Select test duration - Slide the toggle to run the test for as short as a day and as long as 90 days. The default value is 7 days.

5. Review the details because you cannot edit an A/B test once it is up and running.

6. Once you're sure of all the parameters, click Create.

   

7. A dialogue box is displayed, showing the creation process of up to three sub-search clients, each corresponding to the selected variants. Once the search clients are created, click OK.

   

8. You will see the A/B Test screen. Click Start A/B Test to initiate your controlled experiment.

   

Delete or Stop an A/B Test

SearchUnify admins can stop or delete an A/B Test report. All active reports have the tab labeled Running or Running ( <n> days left) on them.

• To stop a test, click Stop A/B Test.

• To delete a test, click Discard A/B Test.

The difference between stopping and deleting is that the stopped test remains visible on the A/B Test tab

Key Metrics

This section defines the three search metrics in the context of the A/B Test for Search

• CTR: This is the Click-Through Rate, calculated by dividing the number of searches with clicks by the total number of searches. Example: If there are 70 clicks for 100 searches, then the CTR is 70%.

• ACP: This refers to the average position where users click on the search results.

  Example: If there are three clicks at positions 2, 5, and 7 during a test period, the average click position for that period will be .

• Deflection: This refers to the Stage One Deflection, which has been documented in Stage 1 Case Deflection from Search.

Instance-Level Changes by A/B Testing Activation

When the A/B Test for Search is turned on, the following changes occur in your SearchUnify instance.

Search Clients

Up to three sub-search clients are visible on the Search Client screen where the A/B Test is running.

Each sub-client is a test variant. When you open a sub-search client for editing, only the Relevancy tab is available for configuration.

When you open a main search client for editing, you can make changes on the following tabs:

• Configurations

• Analytics Settings

• Advertisements

• End-User Feedback

following tabs cannot be altered:

• Content Sources

• Relevancy

• Language Manager

The only way to regain complete testing capability is to stop or delete the A/B Test running on the Search Client. You can run only one A/B Test on a search client at a time.

Search Analytics

The sub-search clients are displayed in the Select Search Client/Ecosystem dropdown. You can select a sub-search client to view its analytics data.

Search Tuning

You can run Auto Boosting for each sub-search client individually.

In Manual boosting, only Content Boosting is available for the sub-search clients.