Hosted Search Engines for Nonprofit websites
Alder Consulting Back to Resources page
Site Search Engines
(On a Shoestring)

A website search feature doesn't have to be expensive or time consuming to create – there are a number of free or cheap services that allow you to create a site search with just a basic knowledge of HTML. This article provides an overview of the top services and helps you think through which service is right for you. 

This article was researched in December 2003. Product information changes rapidly - please verify specific product facts before relying on the information below.


Introduction

Let’s say you have a website with lot of detailed and valuable information spread across many pages.   Your users want to find things on the site based on such a plethora of terms that it’s hard to keep track.  Your budget doesn’t quite allow the latest million-dollar content management system, but you’d like to allow your users to search the information on your site.   Is there hope?

There is. There are a number of hosted search tools that allow users to search your information on your site, with minimal effort from you.  This article will provide an overview of the top free services and those under $250 a year, and help you think through which service is right for you.


Do You Really Need A Search?

Before you start looking for the right engine, it’s important to consider whether a search will actually help your users find things on your site.  If your site is quite small, or has a smattering of information on various topics, a search may actually distract them from finding the information they need – or that you want them to find.

Many users will go directly to the search function, if it exists, without much consideration of the site navigation.  If their search doesn’t return any results for their topic, they are likely to leave the site with the assumption that it doesn’t have what they need – without noticing a link for it that may be in the navigation or on the homepage.  That being said, a search function can be very useful for sites with very task oriented users (continually coming to the site to find, say, Form 10C45Z, or guidelines for the Cuiverlicz protocol), comprehensive information on one or many topics, or listings of jobs, services, events, or the like.


About Searching with Hosted Tools

For an example of they type of site search we’re talking about, take a look at www.compumentor.org.  Enter a search term (say, “networking”) in the search box on the upper left and click “Search”.  You’re taken to a page that shows all the pages on the site that match your search.  While the look of this new page is different from the rest of the site in this example, that’s doesn’t have to be true – many hosted tools will allow you to customize the search results page so that it fits right in.  Compumentor is using Atomz, a free hosted search tool.  While we have no special knowledge, this search feature is likely free to them and took no more than an hour or two to set up.

How do these tools work?  The key factor that makes it an hosted tool is that the search engine itself doesn’t reside on your site or your server.  Instead, the search engine might be in a different state (or country!).  In order to be able to respond to search requests, the search engine relies on an “index” that it has created of your site in advance.   This index contains, in essence, a record of every word on every page.  The index is created with a program called a “spider”, which “crawls” your site over the internet – it starts at a URL you provide and clicks on every link it can find. This index is stored on the search service’s computers.

To allow your users to search, you add a small piece of HTML code to your website that produces a text box for the search terms and search button.  It also contains a link back to the search engine.  When your users enter a search term, it’s sent to the remote search engine.  The engine searches the index and then shows the user a list of pages that match their terms. This list of results, called a “results page” is displayed from the service’s computers, but when the user clicks to view a page from the results, they are returned to your site - hopefully without realizing they left!

The hosted search tools don’t require any installation other than the addition of the piece of HTML code to your web page or pages (this code is generally provided by the service and can simply be copied and pasted).  For this reason, they can be quickly setup by anyone with a basic knowledge of HTML.


Are These Search Tools Right for You?

Hosted search tools have a lot of advantages – primarily, they provide a reliable search for minimal effort and cost.  But they’re not the best solution for every situation.  There are two other types of search implementations that you may want to consider:
  • Installed search engines.  If you have access to install software on your server (i.e. your website is running on a physical computer in your building, or you’re paying a hosting company hundreds of dollars a month), you may want to consider installed search engine software.   There are a number of open source engines that are free to those who know what to do with them.  In general, they are considerably more complicated to set up than an HOSTED tool, but offer more flexibility in how the search functions and appears to the user.
  • Database searches.  If some of the information that appears on your website is stored in a database, a custom-built search of your database might be appropriate.  This is useful when you want to search within a very limited and specific set of information – like last names or book titles.  As opposed to finding every occurrence of these words in all pages, as an indexed search would, a database search can be designed to provide very specific results – like a member page or a book description.  A database search will need to be coded specifically for your database by an experienced programmer.

Considering The Search Tools

There are a lot of hosted search tools out there.  A number of considerations that can help you figure out which one is the right one for you.
  • Page Limits.  Most search tools price based on how many web pages are in your site.  The more pages, the more it costs per year.  This article focuses on tools for sites of about 1000 pages or less.
  • Indexing Frequency.  As discussed above, the search tools “index” your site at periodic intervals.  The index that is created is matched up against search terms to determine what results to provide.  If your site is rarely updated, then it doesn’t matter how frequently your site is indexed.  If you have a lot of new information each week, however, your users won’t be able to find the new information unless the site is indexed weekly.  Most tools allow anywhere from monthly to daily indexing.  Some allow you to schedule periodic indexing, while others require you to manually kickoff the index process each time (by going to the search tool’s website).
  • Reliable Company.  As with anything else, it’s important to look at how well established the search tool is to ensure that it won’t be out of business tomorrow.  Look at how long the company behind it has been in business, who else is using the tool, and how well known it is.
  • Template Customization.  As mentioned above, items that match a user’s search are displayed on a “search results” page.  This page is physically located on the search tool’s server.  However, many tools allow you to customize this page – some so extensively, using full HTML, that the results page can precisely match the look and the navigation of the rest of the site.  Others are quite inflexible and require you to use particular formats or colors. 
  • Document Indexing.  If your site includes a lot of resources that are posted as documents (such as PDF or Word documents), make sure your search tool can index these documents.  
  • Login.  If your site is secured by a password, make sure the search tool can handle a username and password.
  • Quality of Search.  There are a number of different algorithms used by the tools to match a user’s search phrase to the pages that are returned.  It’s difficult to talk about “quality” of these algorithms in the abstract, as it depends a lot on the site content and structure.  The best way to judge is to set up the free trials on your site and see what you think of the results.  The search tools vary considerably in their flexibility, however, with some providing notable benefits over others.  Consider factors such as:
    • Exclusion of Repeated text:  If your site has a lot of text that is repeated on every page, such as extensive navigation or footers, this text can throw off the search results substantially.  When someone searches on a word included in, say, a navigational drop-down, every page with this drop-down on it will be returned.  Some tools have the ability to exclude this repeated text. 
    • Sophisticated Matching:  There are a number of ways that search engines make it easier for users to find results.  Some have spell-checks.  Others support “stemming” (finding grammatical derivations of the search term – for instance, finding “walk” and “walking” if you enter “walked”), “stopwords” (removing common words like “and” or “the” from search terms – generally accepted to improve the quality of most searches), or even “synonyms” (finding conceptually related terms – for instance, finding “nonprofit” if you enter “not-for-profit” or “Africa” if you enter “Nigeria”).
    • Ability to Affect Weighting:  Some tools will allow you to give particular emphasis to particular page text like metadata descriptions (very useful if you want really high quality results and are willing to put a lot of time into descriptions) or to define “best bets” – pages that should always come up first for particular search terms.
  • Quality of Result Display.  The display of search results should be understandable and succinct, and, ideally match the look and feel of the rest of your site.  This consists of three main factors:
    • Results Listing.  The listing for each page should allow the user to determine if it will meet their needs.  Some engines will show the metadata description for each page (which is created in the HTML itself), which is a big advantage if you have created descriptions.  Some show an excerpt, which show how the search terms appear on the page.  Some engines show only the first hundred or so words on the page – which can be useless in determining if the page has the information for which you’re looking.
    • Template Customization.  As per the above.
    • Tool Branding and Ads.  Many tools require you to include a link to them, their logo, or even ads.  While the branding like links and logos may detract a bit from a highly polished website, it likely won’t affect your user’s experience too much.  Think carefully, however, whether subjecting your users to ads is worth any minimal advantages.
  • Support.  These search tools are generally easy to setup and maintain, and shouldn’t need a lot of support from the company.  However, if you feel like you will need some help, make sure support is offered.  The tools offer different levels of support ranging from no support at all to 24x7 phone and email support.
  • Usage Reports.  It’s good practice to periodically check to see what your users are searching for.  Even the most minimal reporting tools offered show this information; more complex reporting tools also display how many results were returned for each search, top searches for which no results were found, and the like.

Comparing the Search Tools

Finally, to the part you have likely been waiting for:  a comparison of the tools themselves.  The descriptions below give an overview of the key strengths and features of each major competitive hosted (a list of those we didn’t find competitive follow).  The stars on the descriptions below mark those we consider the best values.

For more information, see the detailed Search Tool Comparison spreadsheet. This spreadsheet rates each tool by the factors above, to allow easy comparison and sorting.  We have also set up a Search Comparison Page, which allows you to run searches of the Alder Consulting site using the free trials for all these tools to allow you to compare.
  • **Master.com ( www.master.com/texis/master/app ) Despite the alarmingly unfriendly interface on their website, Master.com provides a solid free search tool contender which can handle very large sites (up to 5000 pages).  Indexing can be scheduled weekly, and the template can be fully customized.  The result descriptions show only the first text of the page (rather than excerpts or descriptions).  No ads, but a large logo and byline (which can be removed for $40/ year).
  • **SpiderLine ( www.spiderline.com ).  SpiderLine has a number of packages, starting with a Bronze package at $50/ year for 500 pages and 6 indexes a month, which offer great value for the money.  Their Silver package is $150/ year for 1000 pages and 33 indexes a month.  Both packages are very flexible:  full HTML customization, support for many document types, can login to a secure site, excludes repeated words, has many options to control search weightings, and includes extensive reports.  They also include a powerful feature:  the ability to define custom synonyms (to show results for “nonprofit” if youenter “not-for-profit”) for instance. 
  • **Google ( www.google.com/services/websearch.html ).  The basic free Google search is just not very competitive with the other free services.  Google will only index you when it indexes all the rest of the web – which results in 30- 45 days between indexes.  If you have a small, low traffic site, it may never index all your pages at all.  In addition, there is very little customization for the search results page, which includes logo and advertisements, and there is no reporting at all.  If you have a large site that is extensively linked to by other sites, the Google engine may provide better quality results than other tools, however.  Google has a separate University Search plan, which allows customization and reporting, which is available to “any accredited university or educational institution, or any organization at an accredited educational institution worldwide.” Update: As of September 2004, Google has opened up the University Search plan to all 501C3 nonprofits. While it could still be 30-45 days between indexes for a low traffic site, this new plan offers substantially more customization and reporting.
  • FreeFind ( www.freefind.com ). Freefind provides both a free service, which supports about 2000 pages, and a paid service (1000 pages for $228/ year).  Both plans allow daily indexing which supports PDF documents, ability to login to secure sites, fully HTML customizable search results template, and a nice description of result pages.  The major downsides of the free service are the substantial ads and FreeFind branding on the search page, and back-of-pack (read: slow) service for support and indexing. 
  • PicoSearch ( www.picosearch.com ).  Picosearch's free plan doesn't compare well to Atomz or Master.com (can't schedule indexing, minimal customization, and lots of ads).  Their professional plan could be useful for the right site, however.  It supports 3000 pages for $249/ year, daily scheduled indexing, full HTML customization, support for many document types, can login to a secure site, excludes repeated words, and has a number of options to control search weightings.  For some reason, however, the search assumes an “OR” " between multiple search terms instead of the standard "AND" (i.e. searching on “control panel” will find all pages with either the work “control” or the word “panel”)
  • WhatUSeek ( http://sitelevel.whatuseek.com ).  WhatUSeek has a free plan (1000 pages, manual indexing only, logo and ads).  They also have plans that allow 250 pages for $54/ year, 500 pages for $108/ year, and 1000 pages for $216/year.  All plans allow full HTML customization, document indexing, login, and control of search weightings including “best bets”.  The paid plans allow scheduled indexing 3 times a week.  None of the plans, however, support Boolean searches (OR, NOT, phrases, etc.).
  • Atomz ( www.atomz.com/search ).  Atomz provides the best of the free search tools for most sites.   They provide weekly scheduled indexing, a high quality search, and a fully HTML customizable search results template for sites that are 500 pages or less.   Your search result page must display a relatively sizable Atomz logo.  It doesn't index documents (PDFs, for instance), and cannot login to a secure site.  This may not be the best choice if your site may exceed 500 pages at some point:  as opposed to all the other free services, which provide a feasible upgrade to a paid service, Atomz's (enterprise quality) paid services start at about $5000/ year. Update: As of September 2004, Atomz is no longer providing an ad-free search tool. Bummer. You either need to show Atomz's ads, or purchase their paid service at about $5000/year.

Need More Options?

There are actually a lot more hosted search tools out there.  Some were eliminated because they were quite uncompetitive in terms of features for the price; one (SpaceSurfer) because it has ads which actually float over your search results.  If you’re curious, here’s some more:


For More Information

Search Tool Comparison Spreadsheet (View now)
A more detailed spreadsheet, also put together by Alder Consulting, which allows you to compare search tools by specific features.

Search Comparison Page (View now)
An Alder Consulting webpage which allows you to run searches against the Alder Consulting site using the free trial version of all of these search tools.

SearchTools.com (www.searchtools.com)
A great site with extensive lists of search tools (both hosted and others), reviews, and articles about search engine tools and concepts.


About Alder Consulting: Alder creates powerful websites and databases for prices that nonprofits can afford. For more information, see www.alderconsulting.com, email us at laura@alderconsulting.com, or contact us at 718-208-8172.