Introduction

If you are responsible for an intranet or customer facing search application you probably do your best to keep up with industry best practices. You track your top queries and confirm that they return good results; you stay on top of your search trends and vocabulary changes over time; and you track how well you're doing with respect to "successful searches". But what makes up a successful search? And what are the clues that a search was a "failed search".

Let's set some definitions - and feel free to let us know if you differ with our choices.

A successful search is one where the user finds the information he or she was looking for - it answers the question that prompted the search. On your intranet, it means an employee has found the right content and can go on with his/her day. In customer facing environments, it means a sale; either immediate and direct, or deferred until your customer buys product at a local store. A successful search is a win.

A failed search is one where the user does not find the expected information and goes away unhappy - perhaps to yell at your Help Desk, to call your boss's boss to complain, or just to go away muttering about how awful search is. In the customer facing side, a failed search leads to call to Customer Support; or in the worst case, it maps into a lost sale. No one wants a failed search.

But how do we know when a search is successful? How will you know a failed search when you examine your logs? Ah it's never easy with search -those database guys have it so easy!

Two Scenarios

Scenario A: Some of the markers for a successful search might include a user session where the visitor performs a search, and opens a document that was either in the best bets area or near the top of the organic result list. The visitor spends some time on the document, then leaves. A success?

Scenario B: A visitor comes to your site, does a search, and leaves without opening any documents. A failure?

OK, those were trick scenarios.

In Scenario A, consider the visitor does a search and opens the first document. She stays there, reading for a while, but can't find the right information. In fact, the document is so unhelpful the visitor closes the browser and leaves your site. That can't be good.

In Scenario B, the user comes looking for specific information and does a search. The first document summary shows up and has just the right information - a name, a phone number, or a product number. (Try this at home: My caller ID tells me I missed a call from an area code I don't recognize. On Google, search for "area code 773" and Google tells me it's Chicago, Illinois. No more clicks needed.)

Why should you care?

Are you interested in making your search better? You may track your top queries, top "no hits" results, and search trends, but if you really want to improve search, you need to find a way to know when a search failed. And for that, you need to reach some consensus among your search team.

Defining "Failure" (or Success!)

As an example, you might have a failed search when a user:

• Performs a series of related searches, but leaves without opening any documents. Be careful here, you'll also want to track if they clicked on any navigators.
• Executes a single search, opens no documents, and leaves your site, and no "Best Bet" was presented. In this case you might want to check if there is in fact a good document that would have answered that question, and how far down on the results list was this? (AKA it's Ordinal Rank) Did it wind up on page 2 or 3? Many users won't check multiple pages, some won't even scroll the first page if they don't see the doc they want near the top.
• Performs many searches and opens many documents. This can be tricky. Were they a bunch of similar searches looking for the same thing, with the user trying repeatedly to find things by changing their search? Or was the user happy with your site and searching for a lot of different things?

You can see that almost any definition of a user behavior indicating a failed search can be very similar to a successful search. You need to decide within your organization what definition you will use - and measure it over time. You should periodically use spot checks, where you take 10 incidents meeting your failure or success criteria and run them manually, then look at the results. If more than 2 or 3 searches results don't seem to match the criteria, then your criteria should be reevaluated.

You can also use focus groups, user feedback, and help desk feedback. Or find something else to measure, but at least measure "something" over time. Based on your definition, you can observe the results as you tune your engine.

Steps to Follow:

Define what a failed search is for your site

Measure your results over time

Tune your results and relevance to reduce failed search instances

Validate your definitions with users; update as needed

Now, for those of you who believe that it's best to focus on the positive - great! Define what a successful search is for you; then follow the same sort of process to increase successful searches. Just remember people are much less likely to tell you when a search was great then when it as bad, so you may get significantly less user input.

Times and Trends

We always remind our clients that search is not "set it and forget it" technology; it needs to be tracked over time and periodically adjusted. Over time clear trends will emerge, and all your small improvements will start to add up. Have faith that users will appreciate the continuous improvements on some level.

When somebody mentions Trends you might think "Oh no! Time series analysis... where's my old stats textbook!?" Relax! Sure, if you've got the math smarts, bring it, but most folks it's not about Star Trek's Mr. Spok, sometimes it's s more about Star Wars' Yoda and ... sensing a disturbance in The Force. Trends either look smooth over time, or they don't. When the lines look "different" to you, there's probably a reason.

There's also a warning we'd like to give you about math and trends. When you notice that some particular statistic has changed, and also and also that some other event happened, be a bit hesitant to declare that "A caused B". It could be that they are just random coincidences; checking things over time will help ferret those out. Or there could be some actual correlation, but it could be an indirect link, perhaps something there's something in the real world that is causing both. Distinguishing correlation from true causality is something even mathematitions don't totally agree on, but in this case you'd at least expect to see some type of characteristic and reasonably consistent time lag between A and B before you declare that A causes B.

And don't get too hung up on absolute values; it's also about the changes over time. One site might initially measure a 20% failure rate for search; another similar site might take an initial reading of 40%. Depending on the definition of failure each site uses, along with many other factors, it's not necessarily the case the site B is "twice as bad" as site A, perhaps they are actually very similar. However, if the trend over time for site B, after making steady improvements over time, drops to 20% (from an initial 40%), then the site has likely dramatically improved search. Whereas the site that's initial definition of 20% failure has only dropped to 19% might not have made much progress.

We believe this so strongly in this idea, of tracking trends over time, that we even suggest that web sites keep a global timeline of changes, sort of a web site and search engine journal. Later, when a non-trivial change in performance or user behavior is detected, you can go back and see what was going on at that time. As a non-obvious example, suppose the number of searches shows a pronounced and sustained drop at some point in time; this might trigger some amount of concern. Upon consulting your journal, you realize that this drop was right after you updated the site's navigation structure by adding direct links to things are frequently searched for. Now when visitors first get to the home page, a larger percentage immediately see what they are looking for and click directly to it, thus bypassing search. This is success, not failure, the site is more efficient for visitors.

Computers are pretty good at spotting statistically important changes, but it often takes humans to interpret whether the changes are good or bad. Don't fight it let your tools flag things that are "interesting", in that they deviate from the norm, and then take the time passes those changes, or delegate to somebody who really seems interested.

Public Facing vs. Intranet Search Assessments

Measuring search engine performance in the enterprise is a bit different than tracking the performance of a public site. Generally you'll have less search activity, and no direct revenue numbers to measure success against. But the good news it's often easier to do something about it.

Enterprise Search 2.0 is also about collaboration: provide your user a way to give you feedback when the search has not worked. Trust me, unlike the "thumbs up" feedback some sites seek, people will tell you when they have failed to find what they wanted. Smart companies not only let their users provide feedback, but give them an option to seek immediate help from a support desk or call center when they have a failed search. A link to a help desk or call center is a great way to learn how well your web content and search links are working in the real world.

And unlike public facing search, you can follow up with specific employees if you don't understand exactly what problem they were having, or want their opinions on changes. This can be an invaluable way for you to gather context from real users. Knowing who in your company is adamant about search, and proactively including them, will provide immeasurable value as you continue to tune your search relevance over time.

Conclusion

It might be tempting to worry about satisfaction with enterprise/internal search systems and content just as you do for public facing/eCommerce content: after all, employees are not likely to leave just because the company portal stinks. But younger employees might simply abandon the internal search engine all together! We've actually seen this happen. On top of that, poor information access makes all employees less productive.

Also, bad habits on the intranet have a tendency to make their way onto customer facing sites; conversely, good content and search practices that become entrenched on the public side of the IT house can have a positive influence on the Enterprise systems as well.

Remember, good search is not a one shot deal. You need to systematically review it over time despite vendor claims, there are no Magic Beans!