About Neural, Hybrid and Keyword Search

This document offers comprehensive details on Neural Search and Hybrid Search, explaining how both can be utilized within SearchUnify.

Overview

Let’s first understand why neural search is needed or required? A regular keyword search consists of searching for keywords inside a document; for this reason, the algorithm cannot take into account the real meaning of the query and documents.

NOTE

Many documents that might be of interest to the user are not found in keyword search because they use different wording.

Neural search tries to solve exactly this problem – it attempts to enable searches not by keywords but by meaning.

Neural Search

Neural Search is a step forward in finding information by leveraging artificial intelligence (AI) to create artificial neural networks. This type of Machine Language (ML) model learns through training. Neural-search is based on natural language understanding (NLU), a subset of natural language processing (NLP).

Neural Search can handle complex queries, understand context, and adapt to user preferences over time, leading to a more personalized search experience. It has the capability to analyze large datasets and identify patterns that traditional search engines might miss. Neural Search scores are always in the range 0-1.

For example: A human might write rules to match documents containing the word “cat” with queries about pets, but a neural search engine could learn to match those documents without using the word “cat”, instead using words like "kitty" or "feline" or “catty”.

Fig: Figure depicting Neural search execution.

Neural Search offers significant improvements in search quality, user satisfaction, and overall efficiency. Its adoption signifies staying at the forefront of technological advancements and providing users with the best possible search experience.

NOTE.

In SearchUnify, when you will edit a Search Client in the Relevancy tab; you will see two new toggles: Neural Search and Hybrid Search. You can enable only one of them at a time. More details are available in the sections given below.

Vectorization

Vectorization is the process of converting textual data into numerical vectors that can be processed by neural networks which are mathematical representations that can be processed by machine learning algorithms. This transformation allows the neural network to understand and analyze the semantics of the text.

In SearchUnify, when you enable a Neural search or a Hybrid search, all the documents in Content Source that you have made searchable in the Search Client; , SearchUnify will create a vector for all of them. This is called the vectorization process.

In Neural search, the data whether it is text or any docs; they are transformed into numerical vector representations known as embeddings. These embeddings are generated using neural network models trained on large datasets. The process of converting data into vector embeddings is often referred to as encoding.

To understand the concept of vectorization, one must be familiar with Cosine similarity. Cosine similarity is a metric used to measure the similarity between two vectors in a multi-dimensional space.

NOTE.

In the case of Neural search, you can see the results from vectorization in real time. In the case of Hybrid search, the vectorization will happen only at the specified time; it will not be in real time.

Once the vectorization process is complete, neural search will be up and running and it will be available to be used in the Search Client. The difference between neural and hybrid search is purely based on vectors.

Vectorization process

An overview of the vectorization process in Neural Search is mentioned below:

Tokenization: The text is split into individual tokens, which could be words, phrases, or characters, depending on the chosen tokenization strategy.
Word Embedding: Each token is represented as a dense numerical vector, known as a word embedding. Word embeddings capture semantic relationships between words based on their context in a large corpus of text.
Sequence Representation: If the input text consists of multiple tokens (e.g., sentences or documents), the individual word embeddings are combined to create a sequence representation.
Normalization: The numerical vectors are often normalized to have a consistent scale or range, which can improve the performance of the neural network during training.
Input to Neural Network: The vectorized representation of the text is fed into the neural network as input. The network then learns to extract relevant features and patterns from the vectorized data through training on labeled examples.

By vectorizing text data, Neural Search systems can effectively process and analyze textual information, enabling tasks such as document retrieval, question answering, and semantic similarity calculation.

In SearchUnify, once vectors are created and neural search is live, any changes to the searchable fields will automatically re-trigger the vectorization process. This occurs because the modifications affect the Content Source, requiring the vectors to be updated accordingly.

Vector Creation of Documents and Queries

Pre-trained Embeddings:
- Use models trained on large datasets to generate embeddings for words, sentences, or documents.
Document Vectorization:
- Each document is processed to generate a vector. This can be done by averaging word vectors (for word embeddings) The resulting vectors represent the semantic content of the documents.
Query Vectorization:
- Similar to document vectorization, the query is processed to generate a vector. This vector captures the semantic meaning of the query, allowing for a comparison with document vectors.

Example:

Tokenization:
- Document: "Machine learning is fascinating."
- Query: "What is machine learning?"
Word Embeddings :
- Convert each word into a dense vector.
- Document: ["Machine", "learning", "is", "fascinating"]
  - Vectors: [v1, v2, v3, v4]
- Query: ["What", "is", "machine", "learning"]
  - Vectors: [q1, q2, q3, q4]
Average Word Vectors (or use Sentence Embeddings):
- Document Vector: Average(v1, v2, v3, v4)
- Query Vector: Average(q1, q2, q3, q4)
Similarity Calculation:
- Use cosine similarity to compare the query vector with document vectors.
- Rank documents based on their similarity to the query vector.

Cosine Similarity

Cosine Similarity is a measure used to determine how similar two vectors are in a multi-dimensional space. It calculates the cosine of the angle between two vectors, which ranges from -1 to 1. In the context of neural search, it is frequently used to quantify the similarity between two documents or text passages.

If the Cosine Similarity is “1”, - it means the vectors are pointing in the same direction (i.e., they are identical).
If the Cosine Similarity is “0”, it means that the vectors are orthogonal (i.e., they have no correlation).

Note: Orthogonal vectors have no directional similarity. They do not share any common direction or component.
If the Cosine Similarity is “-1", it means the vectors are pointing in opposite directions (i.e., they are diametrically opposed or completely dissimilar).

Note: Cosine Similarity uses only the direction of the vectors to decide which vectors are alike.

The cosine similarity between the query vector and each document vector is calculated. This measures the cosine of the angle between the two vectors and indicates their similarity.

Scoring and Ranking Neural search results

Understanding the range and interpretation of neural search scores is crucial for effectively using these scores to rank and retrieve the most relevant documents in response to a query.

Scoring

In neural search, documents, queries, or other text elements are typically represented as vectors in a high-dimensional space. Each dimension of the vector represents a different feature or aspect of the text. Cosine similarity measures the cosine of the angle between two vectors.

Neural search scores typically fall within a specific range that indicates the degree of relevance or similarity between search queries and documents. The exact range depends on the specific neural model and its configuration. Generally, the scores are normalized to a range between 0 and 1, where:

0 indicates no similarity or relevance.
1 indicates maximum similarity or relevance.

Ranking

In neural search, cosine similarity is often used to compare the similarity between a query vector and document vectors. Documents with higher cosine similarity scores to the query are considered more relevant and are typically ranked higher in search results.

Using Neural Search

Neural Search is an on-demand feature. You can contact your Customer Success Manager (CSM) to get Neural search and the CSM will enable it in your instance.

To use Neural Search, simply, turn on the Neural Search toggle.

Fig: Snapshot of Neural search in Relevancy tab.

Vectorization starts as soon as the Neural Search toggle is turned on. Users can see the results powered by the traditional keyword search while the vectorization is in progress.

While the vectorization is in progress, the Neural Search toggle will display any one of these five states:

Waiting. This is the state at the time of the first vectorization or after changes in the configuration.
In-progress.
- Search Client is linked with its first content source
- Search Client is linked with its second or a later content source
- Search Client fields in the searchable column are changed
- An object in the content source is removed
Paused. Manual or frequency crawling is in progress.
Temporary Disabled. Vectorized content source is removed from the search client and another content source is added.
Error Encountered During Vectorization. Vectorization is unsuccessful for any reason.

For more information on how to use Neural Search, please see Neural Search.

Hybrid Search

Hybrid search is a search approach that combines multiple search technologies or methods to deliver comprehensive and effective search results.

Hybrid search combines Keyword and Neural search to improve search relevance. The primary goal to use Hybrid Search is to use Keyword + Neural Search to overcome challenges such as exact match and boostings.

Hybrid Search is Keyword + Neural Search

Hybrid search is a combination of full text and vector queries that execute against a search index that contains both searchable plain text content and generated embeddings. For query purposes, Hybrid search is a single query request that includes both keyword search and vector query parameters.

Fig: Figure depicting Hybrid Search execution.

Hybrid search has a vector component and a keyword component. It has a score based on keyword and it has a score based on the cosine similarity as well. If you enable any type of search algorithm, be it neural or hybrid, you should always do some relevancy testing and then only decide which search algorithm works best for your business.

NOTE.

In the case of Hybrid search, set frequency is the regular interval at which vectorization takes place i.e. when data is crawled and vectorization takes place. There are two cases for this:

Case 1: In Hybrid search, the vectorization results are only displayed as per the SearchUnify’s set weekly frequency. So, your Hybrid search query results will be vectorized as per the set frequency and then the results will be displayed.

Please note this is only in the case when the customer has purchased only the (Hybrid search) feature and not the full (Neural + Hybrid search) feature.

Case 2: In Full (Neural + Hybrid Search) feature, in Hybrid search you can see real time results just like Neural search.

Using Hybrid Search

Hybrid Search is an on-demand function. You can contact your Customer Success Manager (CSM) to use Hybrid Search and the CSM will activate Hybrid Search in your instance.

To use Hybrid Search, simply, turn on the Hybrid Search toggle.

Fig: Snapshot of Hybrid search in Relevancy tab.

Vectorization starts as soon as the Hybrid Search toggle is turned on for the first time. Users can see the results powered by the traditional keyword search while the vectorization is in progress.

While the vectorization is in progress, the Hybrid Search toggle will display any one of these five states:

Waiting. First vectorization or after changes in the configuration
Set Frequency. Changes have been made into a content source or search client.
In Progress. Vectorization is underway. Unlike Neural Search, where vectorization takes place immediately after an admin edits a content source or a search client, vectorization in Hybrid Search takes place only at regular intervals.
Temporary Disabled. Vectorized content source is removed from the search client and another content source is added.
Error Encountered During Vectorization. Vectorization fails for any reason.

For more information on how to use our Hybrid Search, please see Hybrid Search.

Key differences between Keyword, Neural and Hybrid search

Some major comparisons between Keyword search, Neural search, and Hybrid search are listed down so that you can make informed decisions for your business requirements.

Table: Comparison between Keyword search, Neural search, and Hybrid search

Feature	Keyword Search	Neural Search	Hybrid Search
Search Method	Matches exact keywords or phrases provided	Utilizes deep learning models for contextual search	Combines multiple search methods (e.g., keyword, neural)
Understanding	Lacks understanding of context or semantics	Understands context and semantic meaning of queries	Combines semantic understanding and keyword-based matching
Flexibility	Limited flexibility, relies on predefined keywords	More flexible, can interpret natural language queries	Offers flexibility by combining different search approaches
Query Expansion	Requires manual query refinement	Automatically expands queries based on context	May utilize both manual and automatic query expansion techniques
Accuracy	Dependent on the specificity of keywords and relevance ranking algorithms	Accuracy is dependent on the customer data and the type of queries searched by end users.	Accuracy is dependent on the combination of search methods used
Relevancy Ranking	Typically based on keyword frequency or relevance	Uses complex algorithms for relevance ranking	Combines relevance ranking strategies from multiple approaches
Training Data	Relies on structured data and predefined rules	Trained on large datasets to understand natural language	Training may involve various data sources to cover different search aspects

Choosing Keyword, Neural or Hybrid search

We recommend conducting relevancy testing on your data to determine the best search algorithm for your business. It's free to try on your sandbox, where you can enable neural or hybrid search. If you decide to use it in production, you can pay to enable it on-demand.

Now, let's explain how to choose between keyword, neural, or hybrid search with an example.

Example: If you search for "how do I add a new user in the Admin Panel?" in the Search Client, you may get a lot of results.

But, if you use neural search with a similar query like "how do you add a new user?", you'll get more relevant results like adding a user, changing user roles, or adding users on mobile.

This is because neural search understands your intent better, even when the exact keywords aren’t in the documentation. Both neural and hybrid search work well for these types of queries.

If you search for something very specific, like "does documentation version 23.07 exist?", keyword search works perfectly. It finds the exact match because version numbers or specific error codes are keywords with no semantic meaning. However, neural search may not perform as well, since it looks at semantic relevance and might bring up related documents, like release notes. This shows that while keyword search works best for exact matches, neural search isn’t always better for every query. Understanding this difference is key.

Sometimes, keyword search works better. It's important to know that the best search method—keyword, neural, or hybrid—depends on your data and the types of queries users are making. Understanding this helps you choose the most relevant search approach.