Limitations of Search Engines

Search engines are far from perfect:

• They rank websites in part according to concepts such as intrinsic authority, which in many cases are flawed, and which allow rankings to be manipulated.
• They filter results according to the information that a particular user has given them, which rarely provides an accurate reflection of a user’s real interests.
• Although it is less of a problem than it used to be, search engine rankings have always been manipulated by keyword–stuffing: the inclusion of unnaturally large numbers of search terms into web pages.
• Rankings are almost certainly affected in some way by the search engine companies’ commercial interests: their dependance on paid advertising, and their promotion of their own products.

Part of the blame falls on the search engines’ users:

• People who use search engines, and people who own websites, underestimate the vast numbers of individual web pages that often touch on a particular topic.
• Search engine rankings depend fundamentally upon the search terms that are used. Most users of search engines do not know how to create the most appropriate search terms.

Inaccurate Search Engine Rankings

Search engine rankings are frequently inaccurate. In order to understand why they are less than perfect, it is necessary to understand how the rankings are calculated. Search engines attempt to rank websites mainly according to two factors:

• significance, which can be increased by skilled search engine optimisation,
• and popularity, which is largely out of the hands of the website’s owner and its designer.

Significant Links to Your Website

Most search engines place great emphasis on the number of significant links to websites, and are able to detect the approximate number and quality of these links. The greater the number of significant links, the more reputable the website will appear to be.

The important criterion is quality, not quantity. Your website’s rankings will gain little or no benefit by attracting links from websites that are unrelated to the material on your own website.

Significance is defined in two ways:

• Relevance to the topic under discussion;
• The authority or importance of the website that provides the link.

Relevant Links

Relevance is usually easy to identify, and can often be reduced to the specificity of the words that are used on a particular web page. So, for example, a website for a shop in Crawley, Sussex, will normally gain more benefit by acquiring a link from a website that deals only with the town of Crawley rather than one that deals with the entire county of Sussex.

Authoritative Links

Measuring the authority of a website is much more difficult. Certain websites, such as those of the BBC and the New York Times, are considered by most search engine companies to be authoritative for a large number of topics. A link from one of these websites is both valuable and difficult to acquire.

Many instances of irrelevant search results are due to the imperfect application of the concept of authority. A common complaint is that searches for information about commercial products are rendered useless because of the number of worthless comparison websites that often occupy the first few pages of search results. Although keyword–stuffing is frequently a factor, product comparison websites also seem to be ranked highly because they are considered authoritative.

At the time of writing, Google considers Wikipedia to be authoritative, which is why it often appears at the top of search rankings, even when the Wikipedia page in question is not particularly relevant to the search query. Wikipedia’s bizarre bureaucratic system allows the editing of individual pages to be controlled by cliques or interested parties. Consequently, search results can be deliberately manipulated.

Filter Bubbles: Personalised Search Results

Most web users make little effort to secure their privacy online. Search engines take advantage of this laxity to assemble profiles of each user’s interests. These profiles influence the search results that are displayed to the users.

If you neglect basic online privacy precautions, you are likely to receive search results that match the search engine company’s view of your preferences and interests. Two web users of the same age, in the same city, with the same circle of friends, typing the same search terms at the same time into the same search engine, may well receive two substantially different sets of search results. The difference will be due to the information that the users have given to the search engine company through their past online behaviour.

For an amusing illustration of how filter bubbles work, see http://dontbubble.us/. To ensure that the search results you receive are as objective as possible, see the advice on our Search Engines and Online Privacy page.

The Effect of Paid Advertising on Search Engines

Once a search engine becomes established and popular, its primary purpose will normally change. From providing a source of relatively objective information for its users, it now provides a source of advertising revenue for its owners. The normal, organic search results are used to attract readers, whose attention can then be sold to advertisers.

There is always a conflict between a search engine company’s ability to provide useful information for free and its need to keep its paying customers, the advertisers, happy. If the organic search results are too accurate, few people will bother to look at the advertisements. If the advertisements are given too much prominence on the page of search results, the page will become less trusted as a source of objective information. As well as the inevitable technical limitations, there is institutional pressure to make search engine results less informative than they could be.

It is commonly understood that the commercial print and broadcast media’s dependence on advertising revenue affects the way in which many news topics are selected and reported. Little research has been done on how the objectivity of search results is affected by the same dependence on advertising revenue, but it is reasonable to suppose that it is affected in some way, considering that advertising revenue is an even greater proportion of income for companies such as Google than for traditional media companies.

Related to this is the search engine companies’ ownership of other web–based businesses. For obvious institutional reasons, a search engine company will be encouraged to promote its own products over the products of rival companies. Any favouritism may not be easily detected. It is common knowledge that Google owns YouTube, and that Bing is owned by Microsoft, but each conglomerate also contains a host of other companies whose ownership is less well known.

Great Expectations

Good search engine optimisation techniques will help to get your website noticed, but they can’t work miracles.

There Are Lots of Other Websites Out There

People with a website to advertise tend to expect too much of search engines, either through underestimating the sheer number of websites that touch on a particular topic, or through overestimating the abilities of the search engines.

Most Web Users Don’t Know What They Are Doing

They also overestimate the ability of internet users to make the most of what the search engines offer. Few users delve beyond the first couple of pages of search results, and fewer still read the search engines’ guidelines to efficient searching (see our How to Use Search Engines page for a quick lesson).

Speedy Results

You should be aware that merely submitting a website to a search engine does not guarantee that the search engine will include that website in its search results. Different search engines work in different ways, with varying levels of efficiency. They also work at different speeds: some become aware of new websites almost instantly, while others may take weeks. One website created by Lab 99 Web Design, to do with the JFK assassination, had its first visitor via a search engine only a few hours after we submitted it, but that is very unusual.

Starting at the Bottom

The sad truth is that most new websites start near the bottom of most search engines’ rankings and work their way up over time. You should be very wary of organisations claiming to guarantee that your website will instantly appear near the top of the rankings. There are many underhand ways of achieving this, and the search engines are wise to most of them. It is quite possible that your website will indeed appear near the top of the rankings, but it won’t stay there for long if the wrong methods are used. Once the search engines identify fraud, they will penalise your website, and perhaps even blacklist it.