A/B Test

Improving the effectiveness of the search function and user experience is a critical goal for any platform. As an admin, improving search functionality and delivering a better user experience is essential.

With A/B Test, you can easily monitor performance and receive detailed reports for each search variant (see details below). These insights empower you to make informed, data-driven decisions that optimize search results and enhance overall user satisfaction.

What is A/B Test

A/B Test offers a systematic way for administrators to evaluate and enhance search functionality. It empowers administrators to optimize search functionality by conducting controlled experiments. Through structured experimentation, A/B Test helps administrators improve and fine-tune search performance.

A/B Test is an arbitrary experiment where generally two variants (A and B) are shown to different segments of users at the same time, and statistical analysis is used to determine which variant yields a better outcome based on a defined metric (e.g., click-through rate, conversion rate, etc.).

  • "A" represents the current version (the control).

  • "B" represents a modified version (the variation).

With this new SearchUnify feature, Admins can control actions at the configurable level and adjust search based on different variants. With A/B Test, Admins can create control groups, execute form performance over set intervals, and gather valuable insights through observational studies during the testing phase.

Objectives of A/B Test

A/B Test is designed to achieve the following outcomes:

  • Enhance search effectiveness: By evaluating different approaches such as Neural, Keyword-based, or Hybrid Search.

  • Enable controlled testing: Without disrupting the live production environment.

  • Facilitate evidence-based decision making: Through detailed analytics and performance tracking.

Benefits of A/B Test

SearchUnify highly recommends the implementation of an A/B Test framework for advanced user search functionalities tailored for actual user behavior and preferences. Key benefits of A/B Test are mentioned below:

  • Data-Driven Decision Making: Helps use real user data to determine which version performs better.

  • Enhanced Search Experience: Helps deliver more accurate and engaging results, improving how users interact with search.

  • Continuous Performance Tuning: Enables continuous refinement of search settings by tracking measurable outcomes like clicks, engagement, and relevance.

  • Risk Mitigation: Tests updates in a controlled environment to minimize potential negative effects before rolling them out widely.

Implementation Suggestions

  • Run tests for 7–14 days to ensure enough traffic for reliable results.

  • Assign clear, descriptive names to your tests for easy report management.

  • Monitor the early metrics, but wait until the test fully completes.

How A/B Tests Are Set Up and Run

Creating a new A/B Test

  1. Accessing the Module: Admins can find the new A/B Test option in the left-hand menu bar, displaying past test reports or a prompt to create the first test if none exist.

  2. Test Setup Options: Admins can select among several test types—starting with 'Search' and expanding soon to include LLMs, Prompts, Embeddings, Search Configurations, and Tuning.

  3. Configuring the Test for Search: Follow the steps listed below

    • Assign a unique, descriptive test name for easy identification.

    • Select the Search Client to test.

    • Select at least two search measures (Neural, Keyword, Hybrid). Variants are automatically labeled A, B, and optionally C.

    • Set traffic distribution percentages using an intuitive slider (default ratios apply depending on selection).

    • Define the test duration (1 to 90 days), with 7 days as the default recommendation.

  4. Launching the Test: Upon confirmation, the platform clones the selected Search Client into sub-clients based on the configurations of each variant. The admin can then begin the test and track its progress through the Reports dashboard.

For detailed information on implementing the A/B Test, refer Implementing A/B Test for Search

Managing and Modifying Tests

How to Edit the Test

  • Search relevancy can be adjusted during the test, but other settings within sub-clients remain locked.

  • When an A/B Test is in Draft or Running mode for a search client, no new test can be created on that client until the current one is either completed or discarded.

How to Stop or Delete the Tests

  • Tests can be stopped manually at any point. While all progress and data are retained, the test cannot be resumed once stopped.

  • Deleting a completed or stopped test will permanently erase all associated data. A confirmation is required to proceed with deletion.

  • Removing a search client will also permanently remove all linked A/B Tests.

Understanding A/B Test Reports

  • Test Statuses:

    • Start A/B Test: Test ready but not started.

    • Running – Early: Test underway but data is insufficient for conclusions.

    • Running (X Days Left): Ongoing with metrics being collected.

    • Stopped: Test halted manually, cannot resume.

    • Completed: Test finished, full results available.

  • Performance Metrics:

    Reports include key indicators such as:

    • Average Click Position (where users click in results)

    • Click-Through Rate (frequency of clicks)

    • Deflection Rate (Stage 1 Deflection calculation)

  • Report Access:

    Admins can view and compare variant performance, enabling data-driven decisions to optimize search settings.

Once a test is complete, admins can review results in the Reports tab and consult with their Customer Success Manager (CSM) for strategic guidance based on the outcomes.

Next Steps and Support

Once a test is complete, admins can review results in the Reports tab and can leverage the insights to refine search strategies. For further recommendations or guidance, admins are encouraged to reach out to their dedicated Customer Success Manager (CSM) for personalized consultation.

Conclusion

A/B Test optimizes the search functionality based on real user behavior i.e. just data. Teams can roll out changes in a controlled environment, measure what’s actually happening, and base decisions on facts. Adopting A/B Test into the admin workflow enables organizations to test configurations without impacting the actual configurations users work with. By testing in controlled environments and analyzing actual user behavior, admins can confidently implement the changes that offer the most value to their users.