KB User's Guide - Understanding the New Semantic Search Engine
What is Semantic Search?
A semantic search engine understands the meaning and intent behind a user's query rather than simply matching exact words. It does this by analyzing how words relate to one another and how they are commonly used in natural language. This capability is commonly known as Natural Language Processing (NLP).
Traditional keyword- and title-based search relies primarily on finding exact matches between a user's search terms and the words stored in document titles, keywords, or metadata. Semantic search takes a different approach. It first attempts to understand the overall context and intent of a query, then identifies content that is conceptually related—even when it does not contain the exact words the user entered.
In addition, semantic search evaluates the full content of documents rather than focusing only on titles and keywords. This allows it to surface relevant information based on the meaning of the document as a whole.
This is especially valuable for users who are unfamiliar with the content available on a site or the terminology used by document authors. Users often describe a topic differently than the person who created the content. In these situations, a traditional search may fail because there are no exact keyword matches. Semantic search, however, can identify documents that discuss the same concepts using different language, making it easier for users to find the information they need.
How the KB's traditional keyword/title search engine works
To understand the differences between the two search engines, it helps to first understand how the traditional search engine works. The traditional KB search engine can be considered a "strict" or "literal" search. This means that every word entered in the search needed to have a match in the fields being searched. As a result, searching a greater number of terms would result in few results (i.e. more documents would be filtered out).
Traditional search process
When a user performs a search in the keyword/title search engine, the following happens:
-
Certain common "noise" words (like "how", "the", "is", etc.) are filtered out and ignored.
-
Synonyms for your search terms are brought in from in our a manually-maintained synonyms list, if applicable.
- E.g. entering word "delete" in your search results in the words "deleting", "deletes", "remove", "removing", etc. also being searched.
-
The resulting set of search terms (i.e. your original search with noise words removed and synonyms added as alternatives) is run against the title and keywords fields of the KB site's documents.
-
If at least one result is found, only the documents that match ALL of the entered search terms or their synonyms are returned as results.
-
If no results are found, the same set of terms is searched against the body of the KB site's documents (i.e. a "fulltext" search).
-
If no results are found again, the same set of terms is searched against the contents of any text-indexed attachments in the site's KB documents (i.e. a "fulltext + attachments" search).
-
If there are still no results, the user sees a message stating that no matching documents could be found. At this point, they would need to refine their search themselves to try to find results.
Put another way, traditional keyword- and title-based search depends heavily on documents being tagged with relevant keywords. When most documents contain comprehensive keywords but a few do not, those documents can become difficult to discover. This is because traditional search engines typically place significant emphasis on titles, keywords, and metadata rather than the full content of a document. As a result, documents with limited keyword coverage may not appear in search results, especially when users enter queries containing multiple search terms.
Another characteristic of keyword-based search is its reliance on partial word matching. In some cases, this improves the search experience. For example, a search for "email" may successfully return documents containing "emails" without requiring an exact match.
However, partial matching can also produce less relevant results, particularly for shorter search terms. For instance, a search for "bus" may return documents containing words such as "busy" or "business", even when those documents are unrelated to transportation. This can make it more difficult for users to identify the results that are most relevant to their search.
How the semantic search engine works
The new semantic search engine uses a large language model to understand the relationships between words. Where the traditional keyword/title search bluntly removes noise words and incorporates pre-defined synonyms, the semantic search engine uses all of the searched words to interpret the overall meaning of the search. It also bases results off of all text fields, rather than just the title and keyword fields. This has the following advantages:
-
It encourages searching with "natural language", i.e., phrasing searches as questions or statements. This is more in line with how we tend to use larger search engines, like Google.
-
Synonyms are handled on a much broader scale than the original search engine, as they are based on general word usage in the English language rather than manually populated sets of synonyms.
-
Typos and common misspellings will usually be understood and processed as the intended word, so they will still produce search results.
-
When a word with multiple meanings is searched, other words included in the search will influence the results towards one of those meanings.
- e.g. searching "How do I submit an application" will yield more results referencing job or scholarship applications, whereas searching "How do I install an application" will yield more results pertaining to software applications.
-
You are less likely to accidentally filter out a relevant result when you use more search terms.
-
Documents that are lacking a robust set of keywords will be returned as results more often than the traditional keyword/title search.
Additionally, the KB's new search engine is technically a hybrid between semantic search and the traditional search. In effect, there is a limit on how many documents will be returned by the semantic search engine (with the default limit being 20). Any documents with literal keyword/title matches that were not returned by semantic search will then be added to the results.
Overall, the semantic search engine will return, on average, a larger number of results. Compared to the keyword/title search, where it is not uncommon to get fewer than ten results, semantic search will more often return 15+ results.