Intel and the Big Search Box

I was recently researching new ultrabooks and visited the Intel site for some insight on current processors.  When I arrived, I was  a bit shocked at what I saw…

Source: Intel.COM

Waiting for me  was not the typical corporate web site, with menu-based navigation (at least not immediately apparent).  Rather, Intel was presenting, dare I say, a very "Google" like approach – enter a keyword-based query to find what you want. 

I found the whole experience a bit off-putting.  I don't generally recommend our clients lead with a search-based content findability approach. In fact, I've argued against even making search a primary content findability technique.

If anyone has insight on this approach (e.g. real analytics supporting this experience strategy over more traditional IA), I'd love to hear about it.

Improve Code Reuse with Search

In past, I have been critical of simply crawling content with a search engine and using the basic keyword query approach to find things.  However, there are some situations that do lend themselves to just this approach (though with a tad bit of “intelligence”).  One of these situations is code reuse. 

At Consejo, we use Subversion (coupled with VisualSVN Server) for source control.  Subversion (or SVN) is an open-source source control system. VisualSVN Server provides a terrific web interface to the repository.  Combined with the VisualSVN plugin for Visual Studio, we are able to very effectively manage our code production across clients with little or no cost.  Unfortunately, with our team dispersed, it’s sometimes difficult to make everyone aware of what’s already been built for one client or another.

SIDE NOTE: I have personally been critical of using open source solutions and even wrote an article arguing against the open source model.  However, it’s clear there are cases where open source makes sense (yup I was wrong).  Further, we’ve been a financial supporter of SVN (through donations) since we started using the tool regularly.

Much of what consulting companies do is based on past work.  It’s critical to how we work that every consultant is kept abreast of what the firm is doing, across industries, disciplines and clients.  For example, we recently built two different applications, for two very different clients.  Both applications, however, used the same licensed interface controls.  In fact, many of the interactions each application’s user base had with their respective application were very similar.  As a result, techniques we used, problems we solved and utility code (code not specific to a client or application) we constructed could be leveraged across both projects with minor updates.  This saved both our consultants and our clients time and, more importantly, money.   Unfortunately, since both projects overlapped, there was no a good way, besides knowing team members on both projects, to capture and surface code reuse opportunities in an automated way.  Each team has to basically know what the other team was doing and ask specific questions (or discover reuse opportunities through happenstance).  Not a fabulous nor scalable model. 

While we solved this problem the “old fashioned” way, it’s not a good long-term solution.  As projects get more complicated, team members more geographically or temporally dispersed, the “old fashioned” way becomes very burdensome and super inefficient.   So what’s the solution?

Historically, we’d been looking for ways to effectively expose our SVN repository to our consultants, outside of Visual Studio and a web browser.  The thought was that if developers were able to search for specific code constructs or even development pattern names, there was a reasonable chance of finding reusable code snippets or whole libraries (if they existed).   And, while there are a few tools that help in this regard like FishEye and SvnQuery, neither was a perfect fit for our needs.  Since VisualSVN Server presents the repository as a web site, why couldn’t we just use a standard search engine like MS Search Express, SharePoint or GSA (Google Search Appliance) to crawl the repository and allow consultants to query it like a web site? 

The answer, as Kenneth Scott points out in his blog post on crawling VisualSVN with a search engine, is “not exactly.”  The problem stems from the way VisualSVN renders the repository web interface (you should read his post for the details).  However, Kenneth solved this problem through the use of an HTTP handler and gave us exactly what we needed.  Using his utility, we can indeed use a standard search engine (take your pick) and then allow our developers to search for an example of an excel-like editing experience using a Telerik GridControl or discover if we’ve built a custom authentication provider for SharePoint; both of which we’ve done, but only one I knew about until recently.

You may still be questioning why this is a good idea?  I’ve been critical of this kind of shot gun search approach in the past.  Why should this work now?  The reason is the very narrow information domain and the very specific terms developers use.  Both work together in a way that makes finding content, using straight keyword queries, more reasonable.  For example, if I want to find example of our use of the Telerik GridControl, I can simply search for the RadGrid class name.  If I want to figure out if we’d built a .NET membership provider, I can search for the inheritance statement.  In the first case, I may get an overwhelming number of results.  However, they’ll all be examples of use, since the only time that class would appear is when I’m using it in code (in other words, relevant).  The second example would produce far fewer results and likely give me exactly what I need immediately (and relevant and, probably, highly precise).

In the end, I’m only really disappointed that I didn’t discover Kenneth’s blog post sooner, but my past search queries were about SVN or Subversion, not VisualSVN (poor search strategy on my part).  However, as we develop this code search feature inside of our intranet, I’m excited by the prospect of finding internal examples of code we can reuse.

If you have a similar feature inside your firm, I’d love to hear how it works and if it’s yield higher code reuse.

Why “Search” instead of “Find”

In July 2010, I posted “Can search really solve information ‘finability’.”  Since that post, I’ve run into a number of clients who continue to insist that they need to search (even if “search stinks” in their organization).  Yet, almost universally, they report the search experience misses their expectations.  This should be no surprise, as lots of firms struggle with this very problem.  However, I’d like to suggest an alternate hypothesis: they actually want “find,” not “search.”

For better or worse (mostly worse), loads of folks closely associate the act of search with the expectation of locating desired content.  Typically, at least outside of an organization, this means assuming they should navigate to Google or Bing (mostly Google). When they arrive, they enter a few keywords and press “GO.”  At this point, they’re presented with a set of results from which they choose one that looks promising.  In part, any perceived success is just that; they don’t necessarily expect to find what they want.  If, however, they happen to find the object of their desire, they are pleased. 

While both “search” and “find” are verbs, search does not imply a goal, simply an action.  Search describes everything in the scenario I just relayed except the part where you’ve found the appropriate destination.  Find, by contrast, is explicitly the goal and characterized by viewing your content.  If you need a very concrete example, if child goes missing, the goal is not to search for the child.  The goal is to find the child.

Therefore, in regards to content, consider changing the conversation.  Use the word FIND instead of SEARCH.  In doing so, you begin to think of the goal and not of one particular approach.  Further, if we focus on finding content, we can also measure a success rate.

This orientation change opens up a whole world of opportunities.  For example, start with the simplest model: place relevant content on the first page they see on an intranet.  While this may seem wholly impractical (too much content, no context to judge relevance), this kind of solution is possible.  Use what you know about your employees/users.  If you know within what department they work, you can begin surfacing content from their department. Not specific enough? What about adding in the role they serve and surface content targeted to that role (NOTE: a good driver for metadata).

Beyond actually locating the content in plain site, try surfacing tasks associated with the desired content. For example, display tasks like “Submit an Expense Report,” instead of requiring users to search for the expense report form or “Fill out a Timesheet,” which links to the time reporting system (or simply an interface to immediately report time).  In this way we’re presenting navigable elements that are easily understood and focus on actual tasks employees/users need to accomplish without the need to search.

However, if you really must provide search, give them help.  Provide them with a targeted search facility that enables them to narrow the scope of the search (i.e. don’t return results from the entire enterprise if they’re looking for a project related document).  Give them metadata to enable precise queries like “Author = Jane Smith.”  Finally, give them “canned” or pre-developed queries created by “experts” who can construct search queries that ensure result precision.  In this way, we’ve moved away from the simply keyword-driven approach to a more intelligent model that reduces “noise” and improves precision through carefully use of the technology. As an interesting alternative, execute these queries in the background and simply display the result like other navigation on the page; in this way, the user doesn’t actually have to execute the query and you’ve just saved them a step in finding content.

In short, search (as a tool) must necessarily become one of an array of techniques used to find content.  However, it should absolutely not be the first or only approach.  We must think in terms of FIND, not SEARCH.  Find is the concrete goal and has a measurable success rate; search is simply an action that, while measurable, does not necessarily lead to your user’s goal.

Education and Training are Required

The title of this blog is a bit of a mix of themes from Mr. Nielsen’s latest post to his Alert Box.  However, the underlying message was that users don’t actually know how to find content (broadly) and that as search improves, their abilities actually deteriorate.    The implication was that insufficient training and education (both) are provided to users related to information technology.

This scary conclusion was based on research he and his firm have been doing in Asia-Pacific.  He stated that they watched “more than 100 searches for a broad range of tasks.  Only once did I see a user change strategy.”  If you immediately thought, after reading this quote, that changing keyword terms is a change of strategy, Mr. Nielsen offers this retort: “… simple query reformulations don’t count as a strategy change because they were essentially variants on a single approach.”  In Mr. Nielsen’s example, his search research subject was trying to find the differences between a cold or influenza using the symptoms as search terms.  The actual change in strategy is when the research participant started searching for the disease rather than the symptoms.

In our work, many clients try to rely too heavily on search.  They believe good content organization (taxonomy) and tagging (metadata) are unnecessary.  With many of them implementing SharePoint, they often immediately want to install FAST because of a misplaced perception that the standard search technology is inadequate.  In fact, the real challenge is that most users don’t understand how to use search in the first place and most firms don’t invest sufficiently in good content organization.  Further, clients also haven’t provided any education or training as to the use of search, taught their users how to leverage a taxonomy to find content and/or don’t leverage features of the search engine that could improve results or reinforce the right sort of search behavior. 

As Mr. Nielsen points out, search does not solve the findability problem.  And worse yet, as engines get better at figuring out what users want, those same uses continue to degrade their ability to truly engage in research on the web.  

Firms need to invest in education for their workforce if they hope to successfully use technology (any technology).  They also need to invest in training.  The difference?  Training imparts a skill within the context of a specific situation, where education teaches a concept that can be applied in many contexts.   To be truly successful, organizations must provide both.  Then and only then will firms begin to really extract the value they’ve created with the tools they’ve implemented.

Can Search really solve Information “Findability?”

We’ve all heard the lament from end users and IT professionals alike: “I can’t find what I need in our … [fill in the blank here – network share, intranet, portal, etc].”  The reason, virtually universally, is that: 1) there’s an unintelligible organization to the content and 2) using the internal search tool never seems to return the results people need/want.  As a result, there’s a growing sentiment that “the solution” is to just get a Google appliance (or other inappropriately expensive search tool) and everything will be fine.  It’s worthy of note that this sentiment comes from all levels in the organization – from executive business management to senior IT management and all the way down to the average joe/jane employee (IT or not) - making the statement carry that much more weight.

Interestingly, there’s a certain logic to this statement (perverse though it may be to people who spend their professional careers mired in trying to solve the problem of findability).  When most people search for content on the web, they overwhelmingly use Google’s search technology.  In my own experience, if the content exists, it’s likely I can find it with Google search.  How Google makes is happen is a mix of complex mathematics, massive computing power and (seemingly to me) black magic, since even companies with vast resources invested in bettering search (like Microsoft) have yet to equal Google’s search success.  However, isn’t Google’s success also partly due to perception and, frankly, low expectations?

By and large, research suggests that most people “hope” to find content when searching the web.  If I execute a query using Google’s search site, I don’t actually know if the content I want exists.  I hope that it does and, when I find it,  I’m pleasantly surprised.  By contrast, when I search for content internally, it’s not a question of “if” the content exist, it’s a question of where.  I know that the content lives somewhere inside my … [again fill in the blank – file share, intranet, portal, etc], I just don’t know where. 

When I use my portal’s search engine to execute a query, I expect to find the content I want.  However, when the content I know exists doesn’t appear in the search results (or doesn’t seem to appear), I get very frustrated (can I get a Marine hoo-rah if you empathize??).  If I’m feeling particularly persistent, I may execute another query, perhaps using different keywords, to see if I can find my content in a different way.  This second attempt may or may not yield better results.  In the end though, my perception is I’ve just wasted a bunch of time and I’m no better off.  Worse yet, I have now come to two very harsh, though potentially inaccurate, conclusions: 1) Google rocks and I don’t understand why we don’t use it internally and 2) whatever search technology we are using internally stinks (I’ve heard clients use more colorful language, which I’ll spare you here).  Also, and not any less important, I’m very deeply disappointed that I will be spending more wasted energy duplicating an effort to either recreate the content I can’t find or, at least, doing duplicative research to support the work I actually have to complete.

If you’re still reading, I suspect you not only empathize, but are hoping I’ll give you “the solution.”  My honest response is that if I had a silver bullet solution, I would probably be very wealthy.  I would be cruising the Atlantic on a large yacht, with a drink in my hand and not searching for anything in particular except perhaps the elusive lime to add a little something extra to my libation.  Since my reality is that there is no silver bullet solution (and I don’t have a boat), I offer some truisms and suggestions about finding content in your organization and how to make that process better:

• Taxonomy is required (really) 
  If you want people to find content, in any repository, you must (I don’t use this work lightly) apply some sort of organizing  principle to that content (“organizing principle” is synonymous with taxonomy).  You don’t have to look much further than your physical file rooms (organized alphabetically), biology (order, genus, species, etc – you remember high school biology no?) or virtually any other repository of stuff to figure out that if it’s not organized you’ll have a slim chance of finding what you need.  To provide a very concrete example, look at Best Buy’s navigation for a technology category approach that helps you find the product you want; if you can’t find what you want, let it stand as an example of how poor taxonomy can confound someone.  That said, I recently had a CIO tell me that he didn’t “get” the value of taxonomy and that we all live in a “search-based world.”  Again, though I respect this particular CIO and I get the sentiment, it’s just not reality.  You must develop a way to organize your content and make that organization (taxonomy) infinitely clear to the people for whom it will be the path to finding content nirvana.   Frankly, not everyone will use search to find content and a poor taxonomy will impact your search tool’s ability to return relevant results.
• Google is good, but it’s not likely to work internally
  It’s true search technology has improved tremendously.  Google is clearly the leader.  However, their success is not actually magic and probably won’t translate to internal search success; public search and “enterprise” search are different beasts; Google is very good at public search, but haven’t fared any better than any other search company within the enterprise.  Also keep in mind that, Google’s dominance in general public web search stems from a number of factors, most of which do not exist inside of any singular company (a few of those key dimensions are listed here):
  • An overwhelming volume of content that can imply topic relevance through cross-linking (keywords in one document and/or site that are linked to other content and/or sites that imply authority for that keyword)
  • Massive computing power that can be used to execute complex algorithms (their specific “secret sauce”)
  • Search analytics that help create patterns of relevancy based on actual user click behavior (result select or re-query)
  • Legions of Google staffers that constantly improve Google’s algorithms over time (based on content data and analytics).  Despite what you may have heard about what Google does for a living, it’s all about search (not phones or collaboration software).  No other part of their business comes close to generating the amount of revenue (obviously through ads shown with search results) or has the same amount of resources devoted to it.
  • Like Microsoft had “help” from IBM becoming a dominant player in the PC operating system business, Google had a similar giant help them early on – Yahoo.  When Yahoo was the most popular site on the web, guess what search technology they used?  That deal made it much easier for Google to supplant their patron, in later years, as the leader in search.
• Search is not the exclusive answer to findability
  Even if someone could create the perfect search engine that returned very precise and relevant results, why do you want to force your user community to execute a search just to find content?  Doesn’t the mere act of search introduce yet another set of clicks and keystrokes?  Isn’t the goal to “get there in less than two clicks” and “provide answers quickly?”  Think about your own experiences with search.  How long does it take to formulate the appropriate query?  To review the results?  To find the right result, click on it and validate it’s actually what you want?  Wouldn’t it be easier to create a very clear path to the right content?  Better yet, don’t you already know a good deal about your employees and couldn’t you predict at least some of their content needs?  If so, why not just surface that content on the home page of your portal (or other appropriate location)?  Better yet, wrap those items in an RSS feed and deliver them through a desktop gadget (works on both MACs and PCs).  Do you think I’m crazy?  This technology surely couldn’t exist you say (or it’s too complicated to implement).  Have you ever visited your iGoogle page?  Have you ever ordered anything from Amazon?  These sites aren’t doing anything that revolutionary in the year 2010.  They’re simply using data they already know about you to automatically find what might be relevant (or asking you to choose topics for the taxonomy that you are interested in viewing).  Is it perfect?  No.  Does it save time in the long run?  Absolutely.  Does it provide you a somewhat “searchless” content findability experience?  You bet.  Could you still supplement this approach with a search tool?  Of course.
• Successful search experiences require constant work
  Just because you spun up a Google appliance, SharePoint search (or FAST) or implemented Autonomy, you’ve only just begun.  Like anything else in this world, success comes from discipline, dedication and an ongoing effort to ensure success.  Search is no different.  More than one of our clients has called and asked for our help in improving their search experience.  Unfortunately, virtually all of them have done next to nothing, beyond installing the software, to make search work properly.  Whether it’s not excluding common navigation items on a web site (the same navigation shown on every page messes with relevancy for those keywords), avoiding the application of metadata on documents (like no title or descriptions – not even counting the “fancy” stuff like author, department, main topic area) or leaving patently irrelevant content in the repository past it’s expiration (like the snow day policy from 2001 that for some crazy reason is the most relevant document when using “snow policy” as a query), clients frequently thwart their own best intentions.  To make search work, you must do a bare minimum, like excluding content that should never be a result (like a home page), apply metadata with appropriate values, perform periodic “gut check” searches that should yield consistently relevant results and monitor the behavior of your users by actually looking at analytic data (you’d be surprised how many queries return zero results or where no one clicks on a result).  If you just do this minimum, your search experience will dramatically improve.

If I’ve prattled on too long in this post and you’ve missed my point about findability, here’s the short version: search is not the exclusive answer to finding content; you must start with a taxonomy that everyone understands (and fits your content), implement search properly and monitor progress (making changes when necessary).  Anyone who says differently is either selling a search tool or has been sold a bridge in Brooklyn.

Content Types in SharePoint

Taxonomy expert Stephanie Lemieux (Earley & Associates) and I had a great Skype discussion about creating and managing content types in SharePoint.  The conversation centered around whether you should create lots of content types, supporting a very granular taxonomy of documents or creating fewer content types, supporting a larger variety of documents.  Stephanie published a great blog article about it on her blog.  

Stephanie and I both concluded that fewer content types is probably a far better bet.  In fact, there are both taxonomic and technical reasons supporting our position.  From a content management perspective, content management guru Bob Boiko had a great quote in the article:

"How many content types is the right number? The fewest possible to squeeze the most value out of the info you possess.”

Free Webinar Series: Improving SharePoint Search

I will be joining a list of fantastic speakers on a free four-part webinar series on improving SharePoint search.  The series was developed by Earley & Associates. 

From the web site:

“From basic search scopes to custom properties, unstructured content to structured data, we’ll cover a variety of methods you can use to improve your search experience and results.”

Please join us for one or all of the calls during the month of June.  You can register here.

Using SharePoint Search as a Research Content Source

Back in Office 2003, Microsoft introduced the concept of the "task pane."  The task pane was a panel that appeared on the right side of the Office interface (at least for the "first class" Office products).  The task pane represented a few different functions -- from showing the status of a document in SharePoint, to allowing you to manage permissions on a document.   Included with these functions was something called the "Research" task pane. 

The Research task pane allows you to search various information sources using basic keyword-type searches.  For example, if you highlight a word in a document and right click, you see a function called "Look Up."  When you select that option, the feature searches through the dictionary (for example) for the definition of the term you highlighted.  You can see this function in figure 1-1.

Figure 1-1 The context menu allows you to lookup a term in Office 2007

When you execute this function, the Research task pane dutifully appears on your right.  By default, the task pane appears to the right and displays the search result. In addition, it exposes a number of other sources that the search can be executed against (shown in figure 1-2).

Figure 1-2 The task pane and the content sources.

When I first saw this feature, I thought it was a way to extract additional dollars out of buyers of Office, since a good number of the research sources were premium sites that wouldn't allow you access to the content without a subscription.  However, I then noticed a link at the bottom of the task pane labeled "Research Options."  Clicking on this link allows you to control what sources are included in your task pane, as well as allowing you to add addition sources as shown in figure 1-3.

Figure 1-3 The Research Options dialog

As it turns out, it's easy to add your own sources, in addition to enabling ones already defined within Office.  In fact, one of the best sources of "research" is SharePoint.  SharePoint's own search web service is already setup to be included as a search source (MOSS, not WSS). 

To add your SharePoint site as a search source, follow the instructions below:

1. Click on Add Services button
2. Type in the URL of the SharePoint search service.  The default URL for the search web service is: http://[server_name]/_vti_bin/search.asmx.  Simply replace [server_name] with the host of your SharePoint site.
3. Click on OK and Office will validate the selection to determine whether there's a compatible search service.  
4. Once Office validates the search service, it will show you the name of the search service available to be included.  The service name will be the name of the primary MOSS site where search is hosted.
   
   
   
5. Simply click on Install and your MOSS search service will be added to your research source.

Once you've added your own internal search as a source, any lookup done within first class citizens of Office (Word, PowerPoint, Excel and Access), can use that source to look up terms and return search results, just as any other source.   When users click on a search result, Internet Explorer automatically opens, preserves the search results in pane on the left side of the interface and show the specific, chosen result in the main window (as shown in figure 1-4).  Now, all users have to do is navigate the various results until they find the one they feel meets their requirement.

Figure 1-4 Search results preserved in IE

This feature is particularly valuable for organizations that need to connect employees with content throughout the enterprise.  In addition, it adheres to a good rule of thumb about portals: don't make your employees leave the applications they're comfortable using -- surface the functionality in the applications where they do their work

Access Denied when trying to Schedule a Crawl in SharePoint

While I'm not a huge fan of simply re-posting content originating on other individual's blog, this particular re-post is warranted, if for no other reason that it was a bear to find the solution. My hope is that more references to the posting will increase the likelyhood that someone else will spend less time finding the answer in the future.

Now, effectively the problem I encountered was an "Access is Denied" message when trying to schedule a search crawl in Office SharePoint Server 2007 (RTM). While, I could manually start a crawl and it would successfully complete, I was not able to create a schedule for either an incremental or a full crawl. Apparently, the trouble was missing permissions on the Tasks folder in the Windows directory; the "WSS_WPG" group (created by SharePoint during installation) needed read and write access to the folder.

It totally makes sense that SharePoint would leverage scheduled tasks to kick off scheduled crawls, but I would have hoped that the installation process set this for you. In my case, the server I dedicated to the index function was a "recycle" and was never reloaded with the OS -- the prior implementation of SharePoint was uninstalled and I re-installed the new implementation over top. In hindsight, this may not have been such a good idea, but we live and learn.

Here's the article that saved me the trouble of rebuilding my machines (yes I was that frustrated): http://www.folin.se/index.php/category/microsoft-office-sharepoint-server-2007/

Many thanks to Michael Folin for posting such a great tip!