How to Choose an Ecommerce Site Search Engine: 10 Criteria

How to Choose an Ecommerce Site Search Engine: 10 Criteria
This is a special guest post by Daniel Davidson, the creative director for SearchSpring

Picking the right ecommerce site search engine can be a confusing and tricky process.

It’s a complicated field mired in nuances. If you’re not on guard, it’s easy to fall victim to seductive features that won’t drive conversions. Worse, you can easily end up getting sold ecommerce site search features you’ll never use.

Without being an ecommerce site search expert yourself, how do you tell all the available providers apart? How do you decide what you actually need and what the best solution for your store is?

This article is not a technical side-by-side comparison. Instead …

This is a guide to help choose the right ecommerce site search engine through 10 criteria:
  1. Relevant Results
  2. Natural Language Processing
  3. Machine Learning
  4. Personalization
  5. Implementation Process
  6. Ongoing Support
  7. Pricing Structure
  8. Comparisons
  9. Control vs Automation
  10. Questions to Ask

Keep reading to unearth the criteria.


But, to go behind the scenes and find out how onsite search is just one part of what helps the average Shopify Plus merchant grow between 126% and 274% YoY …

Then download the full recordings and slide decks from our two-part webinar event.

1. Relevant Results

“Can your site search return relevant results for ‘pink and brown shoes with a gold clasp for prom’?”

It’s a common challenge posed by providers. Why?

First, because conversational search is gaining steam — particularly with the rise of voice-enabled devices. Second, because — in some cases — it’s an opportunity for the search provider to show off their tool’s advanced abilities. The question is: with text still driving the majority of onsite searches, how advanced does your site’s search engine need to be? Even more to the point …

Why Is This a Problem?

The goal of any solution is to help your shoppers find what they’re looking for. That’s such an obvious statement, it may feel unnecessary.

But in the case of relevant results, relevant is relative.

If you sell products with detailed or even duplicate descriptions, niche B2B products (e.g., parts), or have a heavy SKU-based catalog, then you may need exactly this type of site search engine.

Still, even then, most conversational searches won’t be filled with “stop words.” Instead, they’ll likely follow a variation of this pattern: “attribute” (e.g., pink) + “product type” (e.g., shoes) + “category” (e.g., prom).

How To Choose An Ecommerce Site Search Engine: 10 Criteria

What to Do Instead

Start by examining how your customers are already using your onsite search engine. If you look at your internal site search queries report from GA, odds are you won’t see many queries longer than 1-3 words:

Google Analytics Search Term Report Sample

Bring those search terms with you when evaluating providers to ensure that — no matter how awe-inspiring their demo is — the tool delivers relevant results relative to your business.

2. Natural Language Processing

Anytime a buzzword like Natural Language Processing comes up, you should be on guard.

NLP is often used as an umbrella term. Depending on who you’re talking to, this could be a robust solution or a simplistic set of basic functionality.

The most common example you’ll see for NLP will be something along the lines of a shopper searching for “red jumper.” The example will have a query like “red jumper” returning results for “red sweaters / pullovers” even though “jumper” isn’t found in the product data. Pretty neat.

However, something like this is achievable through the use of a simple synonym library, which nearly every third-party solution offers.

Natural language processing within an ecommerce site search engine

Synonyms aren’t sexy. They also have limitations and scalability issues.

Another example that has the NLP label might actually be contextual or product awareness. Being able to tell the difference between “dress shirt vs. shirt dress” or “oven rack vs. rack oven” on the fly is a desirable feature, but not necessary for every retailer out there.

Why Is This a Problem?

NLP is a legitimate feature, but it can and should be broken down into more specific definitions.

When you don’t dig into the specifics, you might only be getting a demonstration of basic functionality like synonyms with a fancier label. That’s a problem if you’re expecting an automated system with sophisticated language parsing.

What to Do Instead

The fundamental question here is: Are synonyms good enough or do you need more? If your product catalog is small to medium, then basic functionality may be enough for you.

On the other hand, if you have a product catalog thousands deep and a high-volume traffic, then managing a synonym library may be more trouble than it’s worth.

If you can’t understand the solution in very clear terms, it’s time to ask for a better explanation.

Facing this situation, something like product awareness and product typing could be what you’re looking for.

We could go on and on about all the things under the umbrella of NLP, but that’s not the point. The point is, you need to avoid the buzzword trap. Insist on clarity. NLP is a real term, but it is a term suffering abuse. Speaking of buzzwords …

3. Machine Learning

Often you’ll hear “machine learning” or “AI” thrown into the conversation without much context or explanation. Anytime “AI / machine learning” is used as an adjective and not as a noun, you should stop, dig into examples and real-world applications, or ask what they’re really talking about.

Machine learning: Onsite search position over time

Why Is This a Problem?

For the same reason as the NLP point above, it could be smoke and mirrors.

Depending on who you’re talking to, “machine learning” could mean anything from basic monitoring and automation of tasks to an actual autonomous learning system that verifiably gets smarter on its own.

For example, you might hear something like this, “We use our machine learning algorithm to boost your most relevant products to the top of your search results…”

Is “machine learning” intended to be an adjective or descriptor, or is it a key component of the solution’s technology?

It’s hard to know with the phrasing.

It could be something as simplistic as leveraging basic user behavior. Or, it could be something incredibly technical and robust that monitors multiple facets of data across the entire shopper’s journey, autonomously enhancing product results on the fly.

What to Do Instead

First, ask what they mean by machine learning and ask for the proof. Does their solution actually learn, or is it just the automation of tasks?

Even when complex, it shouldn’t be impossible to show what they mean by, “machine learning.” Ask for examples that are directly related to your search query history and product catalog.

Second, be honest with yourself, be clear about what issues you’re trying to solve, and know the outcomes that are most important to your business.

Most of the time a “machine learning” need is, in reality, a product recommendation concern, a relevancy issue, or a simple merchandising task.

At the end of the day, “machine learning” is going to mean something different depending on who you’re talking to.

It’s okay to ask for clarity. If they can’t give it, proceed with caution.

4. Personalization

It’s rare you’ll hear or see “personalization” on its own. It’ll generally be mentioned with “NLP” or “machine learning.”

Here’s the pitch: “...machine learning algorithms optimized for 1:1 personalization combined with each shopper’s browsing patterns with natural language processing…”

Or to simplify: “...promote products based on 1:1 user behavior…”

Why Is This a Problem?

Personalization is a bug zapper in ecommerce. We can’t help but fly towards the bright blue light with abandoned caution.

As with the other buzzwords, personalization is a phrase that is suffering massive abuse. But it means something different to everyone – ironic, no?

Personalization and recommendations through an ecommerce onsite search engine

Will the search solution’s personalization feature influence the entire shopping experience, providing a unique set of results for each shopper? Or, is it a repackaged pitch for product recommendations, a long-standing upsell and cross-sell feature?

While it sounds promising …

There are a lot of gotchas that need to be addressed to discover personalization’s viability for your store.
  • How much data is needed before the personalization kicks in?
  • How will products be shown with the personalized boost?
  • What happens if the relevancy is off?
  • How do you ensure those past metrics are positive and not negative?
  • How do you test it? How do you measure it? Etc.

In addition to the technical aspects of personalization, you should question the claims that it will supposedly deliver.

Given that nearly everyone is selling personalization differently, one benefit can’t be accredited to another’s solution simply because they share the same buzzword. Where’s the data, where’s the proof of the claims?

What to Do Instead

While there are 1:1 solutions that can deliver, it’s not for everyone, and it’s not something you can take lightly.

Answer this question: “What does personalization mean to me and how do I see it benefiting my shoppers?”

At surface level, this word sounds smart. Slow down. Take the time to dig into these pitches.

  • How would some of these features on a practical level work for your store?
  • Do you have the catalog size to necessitate the need?
  • Do you have the traffic and necessary data to provide a meaningful use case?

5. Implementation Process

There are two sides to this coin. A quick and easy one-click install or a lengthy hands-on integration and review process.

Both have their strengths and weaknesses.

Why Is This a Problem?

Common sense goes out the door and the bias is thick depending on who you’re talking to.

If you’re talking to a one-click provider, they’ll knock the lengthy integrators all day long. Vice versa for the other folks.

The thing is … they’re both right. Its relative and depends on who you are and what you need with your store.

A one-click installation sounds painless, but if your store has complex logic and custom functionality, something quick could cause a lot of pain if it breaks. On the other hand, if you have a standard build without custom development, then a lengthy hands-on integration process could be overkill.

Another aspect is considering the levels of control built into the solution. If the controls are limited, then there’s not much to manage. So, a one-click install might present a quick solution.

Yet, if the solution provides robust tools that let you alter how the solution performs and returns results, then that DIY model leaves the burden on you, the shop owner, to optimize the tool’s settings.

What to Do Instead

You can make a solid argument for either solution no matter the circumstances. It’s a trade-off. You could have an identical setup to another store, but your internal needs are different. Maybe they weren’t technically proficient so a hands-on solution was better for them. Meanwhile, a DIY one-click installation might be more optimal for you.

Decide what trade-offs you’re willing to make. Are you okay with sifting through docs, quality checking, monitoring, tweaking and optimizing settings yourself? Are there even any settings to optimize?

Both integration models have their benefits. You need to define which benefits are useful and which are a burden.

6. Support

It’s easy to get lost in the feature sets and forget about what you’re going to do when problems come up in the future. Unfortunately, there will always be problems.

Does your solution provide the level of support you’re going to need?

Do you want to get on the phone and talk to a human right away and have them fix it? Do you avoid the phone like the plague and prefer to pour through docs to manage the fix yourself?

You can pick the best-fit software solution out there, but if you find yourself in a position where you need support and it’s not there … well, let’s not even go there.

You know how ugly that can get. Don’t get stuck 6–12 months down the road with the wrong type of support.

7. Pricing Structure

There is no one size fits all.

If the solution you’re looking at has a pricing structure from the bottom of the barrel to the top of the ivory tower, you should question its true cost.

A cheaper solution could cost you more in the long run by being too limited. A solution like this could force you to quickly re-evaluate after you’ve quickly maxed out the functionality.

On the flip-side, you could get it all plus the kitchen sink, and never use the majority of the features a tool has to offer.

How To Choose An Ecommerce Site Search Engine: 10 Criteria

What Pricing Really Means

If it’s truly valuable and manageable for the mom-and-pops, then it might lack the heftiness a team of merchandisers and ecommerce managers are going to need.

Equally true, a tool ideally built for a team of ecommerce professionals will more than likely be overwhelming for the smaller store operators.

Another aspect to consider is the level of human involvement. Do you need it? If so, ensure there’s the TLC of a professional team behind the solution for support. If not, make sure you’re comfortable with the self-help docs and online training the tool provides.

While pricing might not be indicative of quality on its own, it is a piece of the pie and should play a part in your overall evaluation.

8. Comparisons

This isn’t really a tip, but we can’t leave it out. Unless your store is a giant marketplace, then any reference to Amazon is meaningless. It’s apples and oranges.

Sales reps and marketers have to stop using Amazon examples for comparison. Store owners should stop using Amazon as the model for how to do things.


Unless you’re operating with a comparable bankroll and have a similarly sized user base, there are infinitely better comparisons, examples, and inspirational resources to use. Comparing yourself against Amazon is lazy at best, and dangerous at worst.

Question anyone who does.

9. Control vs Automation

We’ve touched on it in varying degrees throughout this article, but it’s important enough to call out. What is automated and what control do you have to influence that automation?

Some stores might want a fully automated solution. They have no desire to and never intend on logging into the tool to manage the settings.

Others will want to use the solution as a tool. They’ll want to add campaigns, optimize settings, audit reports, etc …

Automation vs control settings in an ecommerce site search solution

What Kind of Controls Do They Offer?

If you want to work with the tool yourself, you’ve got to consider the kind of controls available.

Some tools offer dials, knobs, and sliders to control their settings. For the less technically inclined, this is ideal. These controls let you optimize by a range. For others, this can feel like it’s a little too much like guesswork lacking fidelity.

How To Choose An Ecommerce Site Search Engine: 10 Criteria

Other tools will give and require more precision through input boxes and percentage based UI elements. Neither is better than the other. One is built for ease of use; the other, for precision.

10. Questions:

How Am I Going To Use This Feature?

Most site search engines have features that do remarkable things. They have incredibly valuable and useful features that will make your jaw drop.

The thing is, not everyone can take advantage of every feature.

To single out one example of many, several providers offer geo-targeting. It’s a cool feature with a ton of possibility. However, except for very rare circumstances, it’s not the most useful feature for a lot of stores.

Will it be useful for you?

Is your product catalog that large, and are your shopper demographics that diverse? Will you see a benefit from the reordering of your product relevancy based on geolocation?

For some, that’s a resounding yes. For others, it sounds neat, but it’s not relevant to their needs. If it’s a must-have, then you’ve shortened your list of providers. Inversely true, if a provider has a block of features that you don’t need, then you shouldn’t be paying for it — scratch them off your list.

Will The Solution Work With My Data?

Any provider you talk to or look at will be able to use your product data from Shopify. That’s a given.

What they do with that data should be a topic of discussion.

Data in and data out for an ecommerce site search engine

How clean is your data? Will your data be optimized or altered before being used? Will your data be used regardless of what irregularities and issues you might have inadvertently introduced over the years?

It’s hard to ensure every product is added to your Shopify store the same way, over long periods of time, with multiple hands in the cookie jar.

For example, a common data variance issue is with colors. If your products have the color set to “Blue”, “blue”, “blu”, “BLUE” what happens? Will those be grouped into one facet of “Blue” or will you have four variations of that word displayed?

Common data variance issues within an onsite search engine

Another common issue is with facet grouping. Using the color example, what happens if you have 3 teal products, 5 light blue products, 2 baby blue products, 1 sky blue product, and so on?

Will the shopper see an over-granular amount of facet options? Can the solution provide facet grouping allowing all related products to live under a parent color option like blue?

Obviously, the results will vary depending on your data and how the site search solution accounts for these and other data issues.

Will I Lose Any Custom Work?

Even if you’re using a popular Shopify theme, you’ve probably put at least a small level of customization to your store. Make sure you don’t lose that work.

How To Choose An Ecommerce Site Search Engine: 10 Criteria

Will the solution integrate flawlessly or will it require additional work? If it needs more work, who does it? You or them? What’s the cost?

The most common things that should be discussed and evaluated will be:

  • Custom design work
  • Custom functionality
  • Custom business logic
  • Unique pricing rules and logic
  • Current merchandising efforts

How Will the Solution Handle Errors?

No matter how smart a solution is, your shoppers will eventually find a way to trip it up. What happens then? Does the solution plan for these unfortunate eventualities and provide safeguards that will allow you to minimize shopper errors?

Displaying a visual autocomplete that updates with search and product suggestions as the shopper types is a great way to minimize common typos and odd spelling errors. 

How To Choose An Ecommerce Site Search Engine: 10 Criteria

With strong error tolerance, Red Dress Boutique’s onsite search still delivers fast and accurate results for common misspellings like “dres yelo,” “bots,” and “snak prin”

Deliver a “Did You Mean” product results page for the search errors that slip through the autocomplete safeguards. When a misspelling or typo slips through and is close enough to your actual product data, the solution can provide the most probable results the shopper intended to find.

For the queries that slip through even the “Did You Mean” safeguards, a Zero Results report can be helpful. Being able to identify search queries that are not generating any product results is helpful. The solution that allows you to easily create synonyms and redirects for these frequent queries is even more helpful.

Ensure the solution you’re looking at provides multiple levels of controls for identifying and fixing those pesky shopper errors.

What Is the Ecommerce Site Search Engine’s Value Proposition?

I hear and see this one asked all the time.

First, consider what you’re asking and how you’re conditioning the sales representative to respond. It is a valid question and one that should be asked, but question your own motives …

A seasoned rep will respond appropriately. They’ll question you on what problems you’re trying to solve. They’ll ask to find out what’s important to you, and provide their value prop in light of that context.

Without that discovery, then you’re most likely going to get the company’s mission statement and boilerplate value props. It might be tailored to outshine the other providers you’re looking at, but without the proper context, you’ll just get another sales pitch.

It’s inevitable that this question will be asked by many of you. So, when you do, both you and the sales rep should relate this question to the specific challenges you’re facing.

Choosing an Ecommerce Site Search Engine: Takeaways

All these tips can be boiled down to a few simple items:

  • Question your own motives. Are you buying into a fad or buzzword? Ensure you are properly evaluating your real-world needs and those of your shoppers.
  • Not every tool is the same. Some offer more functionality than others. Some offer more human involvement while others leave you to your own devices. Define what level of involvement you’re willing to spend and how much you’d like to receive.
  • Make certain your current store front’s functionality will not be lost, altered, or changed in undesirable ways.
  • Understand how the solution will influence your way of doing business. Will it relieve your current workload or will it add additional burden?

It’s a lot of information to consider, but take it seriously. A smooth, robust, site search can make a drastic impact on your business. Happy hunting!

About the Author

How To Choose An Ecommerce Site Search Engine: 10 Criteria

Daniel Davidson is the creative director for SearchSpring, a white-glove ecommerce site search and merchandising solution for etailers. He’s been with SearchSpring and in the industry for over 6 years, and he recently took all that experience and wrote this best practices article about site search.