Document Search Is Artificially Terrible

The basic thesis of document search is that you know what you’re looking for. The system should help you find it. Instead it makes you guess which exact words appear in the filename.

You need a document. You know it exists. You might even remember who created it. Or when it was created. Or what it was about. None of this helps. Because your enterprise document search system only finds exact keyword matches. “Marketing budget Q3” returns nothing. “Third Quarter Marketing Expenses” returns nothing. “Q3 Mktg Budget Final v2” finally works. Because that’s what someone named the file.

Probably when you first started using your company’s document management system, someone showed you search tips. Use quotes for exact phrases. Add filters for file type and date. Try wildcards if you’re not sure of spelling. These helped a bit. But after you’ve been there for a while, you realize the truth. Document search doesn’t work. So you just ask colleagues where things are. Or give up and recreate the document.

Another thing that surprisingly many companies do is accept that document search is broken? You can see the logic. We’ve tried multiple search systems. None work well. Maybe document search is just hard. This creates an obvious problem. Your documents are still unfindable. There is no actual solution in “accepting” that search doesn’t work. So probably you’ll keep wasting time searching. And searching. And eventually giving up. It compounds.

The Keyword Problem

One amusing irony? Document search should be easier than web search. Your company has maybe 100,000 documents. Google indexes billions of web pages. But Google Search works better than your internal SharePoint search. On some plausible assumptions this seems backwards.

Documents get found based on keyword matches. According to whatever words appear in filenames. Or in content if you’re lucky. With no understanding of synonyms. Or context. Or what you actually meant.

In a world of keyword-only document search, search times go up. The average knowledge worker spends 20-30% of their time searching for information. That’s probably not everyone’s experience. But still. A meaningful chunk of work time gets spent trying different search terms. Hoping something works.

If you’re a product manager needing last quarter’s roadmap, is this optimal? Like, “I will spend 30 minutes trying different keyword combinations, and when I finally find something there’s a good chance it’s the wrong version”? I feel like when I put it like that it sounds terrible?

The basic problem is that traditional document search engines do text matching. Not understanding. When you search for “Q3 marketing budget,” the system looks for documents containing those exact words. If the document says “third quarter” instead of “Q3,” no match. Says “expenses” instead of “budget,” no match. Says “mktg” instead of “marketing,” no match. You have to guess the exact terminology someone used. Months or years ago. When naming or writing the document.

The Folder Structure Delusion

At a 2022 information management conference, someone from Microsoft said something interesting. Organizations spend enormous effort creating folder structures. Believing that good organization solves findability. It doesn’t. People can’t remember where anything is. According to a source who attended.

If you’re trying to organize documents, are folders optimal? The answer is no! Obviously not! Folders assume documents belong in one place. They don’t. A sales proposal is both a sales document and a client document. Goes in “Sales” or “Clients”? Different people choose differently. So you end up with duplicates. Or people can’t find anything. Because they look in the wrong folder.

Much like the Dewey Decimal System, folder hierarchies made sense in physical libraries. Where books needed physical locations. Digital documents don’t need this limitation. They can exist in multiple conceptual places simultaneously. Through metadata and tags. But most organizations still use folders. Because that’s what we’ve always done.

I have sometimes mused about why folder structures persist. Folders, I sometimes write, are like organizing your email by creating hundreds of folders and manually filing each message. It sounds organized. Until you need to find something. Was that email in “Projects” or “Clients” or “Important” or “2024”? Nobody knows. Things like “just search my email” or “let Gmail organize it automatically” are best understood not as “ooh giving up on organization.” But rather “rigid hierarchical organization doesn’t match how humans actually retrieve information.”

Companies create folders like “Client Projects,” “Customer Work,” and “Customer Projects.” All contain similar files. Nobody remembers the difference. Or why three folders exist. Documents get saved to whichever folder someone happens to click first. Document search across all three returns duplicates. Different versions. No way to tell which is current. This is not a system. It’s chaos with neat labels.

How Semantic Search Actually Works

Twenty years ago, companies cared about document organization. Much better to have good folder structures than bad structures. But they also just accepted that document search was hard. Hoped employees would remember where things were.

They thought about “the value of good search.” If you found documents quickly, how much was attributable to good folder design? How much was attributable to just knowing where your team kept things?

Shouldn’t AI help here? If document repositories have reached a critical point where folder structures don’t scale? Finding documents is so time-consuming that productivity suffers? The market should create better document search tools. Part of the thesis is that search can understand meaning. Not just match keywords.

Organizations implementing semantic search powered by natural language processing report 60-80% time savings. In document discovery. Document search queries that used to require trying 5-7 keyword variations now work on first try. Documents that were effectively unfindable now surface immediately. Employee frustration with document search has decreased dramatically.

The technology works like this: AI reads and understands document content using transformer models and large language models. Not just keywords. When you search for “contracts expiring soon,” the system understands you want contracts. With end dates. In the near future. It finds contracts with renewal dates in the next 90 days. Even if the word “expiring” never appears in any document. The AI understands temporal concepts. Document types. Intent.

Search for “complaints about product quality” and the system finds customer emails, support tickets, and feedback forms. Even if they don’t contain the word “complaint.” The AI recognizes negative sentiment. Quality-related topics. Organizes results by severity and date. Without you specifying file types or locations or exact phrases.

Another benefit is contextual ranking. Traditional keyword document search returns results sorted by relevance scoring based on term frequency. Documents with more keyword matches rank higher. Even if they’re outdated or irrelevant. Semantic search using BERT and similar models understands context. Recent documents about current products rank higher than old documents about discontinued products. Documents from authoritative sources rank higher than drafts.

Looking Ahead

I’m generally skeptical of claims that AI will make document search perfect. That companies cannot find documents without neural networks. That only enterprises with massive search budgets can implement semantic document search.

Plenty of small organizations exist. Whose basic needs can be met with better keyword search and metadata tagging. Most organizations don’t need Google-level search sophistication. They just need document search that understands synonyms. And document types.

Technology has gotten quite accessible. Document search platforms from Elastic, Coveo, and Sinequa offer semantic capabilities. Implementation that used to require search experts now uses pre-trained models. Pricing has evolved from enterprise-only. To options that work for mid-market companies.

AI-powered document search is prioritizing practical improvements. Immediate findability gains. Over long-term fundamental research into information retrieval theory. Companies are “under growing pressure to show results.” Their investment in search technology needs to pay off. Reduce time wasted. Tinkering with theoretical search algorithms for years won’t help employees find documents today.

Perhaps a little odd that document search has been broken for so long? Document search has a long history of disappointing users. That everyone complains about but few fix. Still, when you get big enough, you care about efficiency. When employees waste 10 hours per week searching? That’s expensive.

The Bottom Line

If you gave me a document search system to build? I promised it would help you find documents? In the meantime I made it only search for exact keyword matches in filenames? You’d be like “that’s not search, that’s string matching!”

Using traditional keyword document search with Boolean operators? That was acceptable in 1995. An enterprise document search system with filters and facets? Better. Though there is still too much guessing required.

A semantic document search system powered by natural language understanding? Significantly better. AI can understand intent. Recognize synonyms. Interpret context. High probability you’ll find what you need on the first try.

Maybe it’s expensive. These advanced document search platforms cost money to implement and maintain. Seems like a necessary investment though. Better than employees wasting 10 hours weekly searching.

What if you didn’t upgrade document search? Just expected employees to get better at guessing keywords? Well, same problem. People shouldn’t need to guess. Document search should understand what they mean.

But you might have an answer. “Advanced document search systems themselves aren’t very reliable.” Complexity and false positives constantly eat away at their value. The only way to find documents is knowing where they’re stored.

You might reasonably reply: “That doesn’t scale.” When new employees join they don’t know where anything is. When documents move folders break.

This objection strikes me as correct. Though some IT departments will disagree.

Historically most organizations have keyword document search. With various levels of sophistication. Hoping employees learn to search better through practice.

The question is no longer whether AI will improve document search. But how quickly your organization will implement it. The gap between companies with semantic document search will widen. Those still using keyword matching will fall behind. Organizations embracing AI-powered document search today? Will be best positioned to actually find their documents tomorrow.