How to fall in love with your on-site search (again)

Google has set a standard for search. We expect to find anything we need on the first page of search results. Finding things on the net is easy, fast, and often target-oriented.

If you then look at your website, you often feel little of this magic. Your website delivers search results that are rarely what you expect. Here are some problems we’ve seen with on-site search results.

• Content isn’t found at all.
• Old content appears higher in results than new content.
• Less important articles appear before important cornerstone landing pages.
• Metadata is not indexed and therefore search results are inaccurate.
• Content related by synonym or alternate spellings is not appearing.
• Certain words are not taken into account while searching.
• You get support requests that users “searched but couldn’t find it.”

Results like this lead users to a frustrating user experience, which makes visitors just leave. Users expectations are high, and as website content grows, reaching those expectations becomes more important. It may be shocking, then, that “84% of companies don’t actively optimize or measure their on-site search.”

But… What if on-site search could be as good as Google’s?

That is not completely impossible. For that, you have to spend some time with your website search. A search is only as good as the index on which it is built.

Practically, your search index is your knowledge library. It takes curation and maintenance.

Let’s consider how Solr search works. When someone conducts an on-site search with Solr, it’s not like a “find in page” direct search of text. Solr has already indexed words and where they are found into a kind of table. It even makes sure that words can be found with the correct accents and without. It uses word stems so that “runs” and “running” also means “run.”

That table includes ratings of how important the terms are. This is where you have control. For example, pages that include terms that show up in a H1 or H2 heading will likely be more relevant than a page which mentions the term in passing. You can adjust the scores of these, but you have to be very careful because small changes can have a massive impact.

The search on your website can have different foundations. These range from very simple to very complex search engines. In this blog post, we want to focus on the search engine “Solr”, an open-source search engine which we like to use in TYPO3 projects because it is fast, efficient, and very adaptable. If you install this search without a big concept, you will get solid results—but solid is not enough for us in this blog post.

So how do you get better search results and thus a higher satisfaction with your search? Actually, for better search results you need to start with some basic questions.

• Who uses the search?
• How is the search used?
• What are users searching for?

Then you need to understand these concepts:

• How the index is created.
• How to rate search results in Solr to improve relevance.

We really love digging into these issues with our customers because the outcomes have such a big impact right away. A good on-site search is hard work but makes things so much easier for your customers. And while we are all customers elsewhere we deeply know that a good on-site search is sometimes the key factor if we stay or leave a page. I myself find it really frustrating to don’t find things on a webpage. So let’s look at those questions and concepts and get your on-site search onto the next level.

Start your investigation by looking at what different target groups are using your website and what do they expect in terms of search results?

You know the content of your site well, but what seems obvious to you might not be for your users. Bias can get introduced. Depending on what team developed the site, the content might be prioritized differently. The IT, Marketing, or PR teams might each have a different understanding of which content is more important. But internal ideas don’t often align with external users‘ search behaviour.

Example: on a magazine site, internal editors often searched for their names and were irritated that not all their articles appeared at the top of the search results. After having a look at the search requests of external readers, it became clear that they practically never searched for individual authors, but only for article names.

The case shows: it is important:

• to become aware of your internal view
• to know who is using your site
• to balance the search results between internal and external search expectations

The example just mentioned shows: The who and the how of a search request are closely interwoven. The view, external or internal, on a search always controls the approach to a search.

Another example: You have an internal product catalog that you search by only using internal product numbers (because you know them well). You find it frustrating if the search on your webpage returns no or incorrect hits. The fact that data may be missing from the search index also raises another question: Do your website customers use the search in the same way and search for internal product numbers? Isn’t it more likely that the external user searches for keywords or titles of products instead?

• Your view of the search not only determines what we expect in terms of search results, but it also determines what data you are looking for in a search result.
• For us, as an agency, it is important to get an idea of what you are expecting from a search result. While we can give you an external view onto the project it is always good to know what you are expecting.

Also, the Top5 search results without hits are displayed. This helps you discover when users are looking for content but missing it. If you regularly keep an eye on this tool, you can make specific changes.

Example: If you discover that certain search terms are not returned with results, there are several options:

1. You assign these search terms to existing products/pages… in metadata such as description text, keywords, etc.
2. In another module of the Solr search, you can assign synonyms to the search index and by doing this, you can get more results for a search term (example: the searched word is “iPhone” you assigned “smartphone”, “cellphone”… to it).

It is important to regularly monitor what users are looking for on your site. In some cases, this will result in manual actions such as adding keywords or synonyms to your products/pages.

That sounds like work­­—it is. But the search index of your site search is only as intelligent as the information available.

Now that we’ve reviewed the search experience and how to improve it, we can look at how search criteria and ratings affect which search results.

Google continuously adjusts their ranking of what signals improve or harm search engine ranking. For example, their search results favour sites with SSL certificates, because they want to promote secure sites and sites with better mobile-first experience rank higher because they offer a better user experience.

You can apply these same concepts to your Solr on-site search.

In the case of Solr, the default rating algorithm looks like this:

queryFields = content^40.0, title^5.0, keywords^2.0, tagsH1^5.0, tagsH2H3^3.0, tagsH4H5H6^2.0, tagsInline^1.0, description^4.0, abstract^1.0, subtitle^1.0, navtitle^1.0, author^1.0

This tells your index how important a keyword is depending on where it appears. Compare how the “content” of the page is rated with 40 points and the H1 page title rated at 5 points.

This means keywords appearing in the content of a page are always rated higher than the actual H1 page title. But will that make sense for product pages?

By adjusting the points you can upgrade or downgrade corresponding content, and this can be different for individual page types. Other meta-information such as “publication date” or custom fields like “revenue figures” could be included in these adjustments.

Google’s search may seem like magic, but it’s not. Google throws a huge amount of money and effort at maintaining its index. It’s constantly changing its index and the basis calculated on the results with several teams working around the clock.

So, if you want to improve your search on your website, it is important to deal with your search and its result list. And, if you are dissatisfied with the results, change them.

You may have noticed: It is not magic at all. It is simple mathematics and concepts that make your search results better. We can help you with both: the concept and the maths. So give us a shoutout if you need some help. So let’s fall in love with our on-site search again.