Web development and Internet services

Understanding Search Engines and Directories

A TAPACOM resource document
by Eugene Villaluz
Revised January 16, 2008

CONTENTS


INTRODUCTION

  • Search engines and directories – what’s the difference?
  • Why is it so difficult to get good search engine results?
  • Which are the top search engines/directories?

OPTIMIZING YOUR WEB SITE FOR THE SEARCH ENGINES

  • Include search terms in your content
  • Search terms – single key words vs. keyword phrases
  • Consider including misspelled words
  • Avoid ‘stop words’
  • What are meta tags?
  • Another tag – the alt tag
  • Yet another good spot for key words – text links

YOUR SITE ARCHITECTURE – IS IT WORKING FOR YOU OR AGAINST YOU?

  • Frames
  • Dynamic Pages
  • Flash

SUBMITTING YOUR SITE

  • Should I use one of those automatic submission programs?
  • Aren’t search engine listings free?
  • Yahoo and LookSmart
  • How does my site’s popularity affect my listings?

HELP! HOW CAN I TRACK ALL MY LISTINGS FOR ALL MY SEARCH TERMS ON ALL THE TOP SEARCH ENGINES?!


HOW CAN I FIND OUT WHICH SEARCH ENGINES ARE DRIVING TRAFFIC TO MY SITE?


USEFUL LINKS


NEED MORE HELP?

INTRODUCTION

Search engines are an important element in any web site promotion program. One study found that 86 percent of people learn about sites through search engines (source: Georgia Institute of Technology). However, it’s also important to note that search engines are just one way to generate traffic to your web site. When preparing your Web site Promotion Plan consider including additional tactics that may be more effective for your market, your products/services and your target audience.

Search engines and directories – what’s the difference?

For the people using them, search engines and directories are basically the same. They are a means by which to find web sites. In both cases the visitor enters a search term and receives results related to that search term. However, search engines and directories are quite different in terms of how they generate those results.

Search engines send a ‘robot’ to ‘crawl’ through a web site and index its pages in the search engine database. Directories use human editors to review a site and to determine its appropriateness to be listed at all and under which categories. Therefore it’s important to consider the needs of both ‘robots’ and human editors when submitting your site to search engines and directories. (For the sake of simplicity, the term ‘search engines’ is used in this document to refer to both search engines and directories, unless otherwise noted.)

Why is it so difficult to get good search engine results?

The four main reasons why it is difficult to place well with the search engines are:

  1. Virtually every search engine uses a different set of criteria by which to index sites.
  2. To protect the integrity of their search engine, they tend to not divulge this information to the public.
  3. They change their criteria frequently.
  4. There are thousands of sites, if not more, competing for the same search engine results.

For these reasons – before you embark on submitting your site to the search engines – a few words of caution:

  1. Don’t expect a miracle, such as being in the top 10 for the keyword “software”.
  2. Be patient. Search engines can take days, weeks or months to include your site. Submit and resubmit on at least a monthly basis.
  3. Ensure your site is ready for the search engines before you submit it. Is your web site ‘optimized’ for the search engines?
  4. Don’t try to outsmart the search engines. Work within their rules. To do otherwise risks being blacklisted and not indexed at all.

Which are the top search engines/directories?

Presently Google is still the top dog. But the landscape changes rapidly. Take a look at the top leaders here:

<http://www.seoconsultants.com/search-engines/>

OPTIMIZING YOUR WEB SITE FOR THE SEARCH ENGINES

To optimize your web site for the search engines simply means developing your site with the search engines’ criteria in mind. When you’ve incorporated into your site the elements that the search engines’ robots are looking for, then you are increasing the odds that you will be well-listed. Conversely, optimizing your site also means not doing the things that can have a negative effect on your site’s search engine listings.

Include search terms in your content

The number one way to achieve good search engine listings for a particular search term is to include that search term in the text of one or more of your web pages. Not too many times (that would be considered spamming the search engine), but just enough so that the search engine robot can tell that that page is really about that search term. It helps if the search term appears a few times and if it appears at the beginning and the end of the text.

Focus your efforts on five to ten of your most important search terms and if possible, focus a separate page on each term (or at most two to three terms). Otherwise you’ll risk diluting the weight of each search terms, and the ‘robots’ won’t think the page or your site is really about that subject.

After you’ve achieved good search listings for the most important search terms, create an additional page on your site to focus on a secondary search term. And submit that page to the search engines.

An important note – search engine ‘robots’ can only read text when it is contained in HTML code. That is, if your web designer uses a graphic to display the text block (often to better control the font being displayed or to present the text within an image), the ‘robots’ aren’t going to be able to read that text and the benefit of incorporating key words is completely lost.

Search terms – single key words vs. keyword phrases

Of course, some key words are going to be much more competitive than others. Ranking well on just about any single word will be extremely difficult. For example ranking well for the keyword “travel” would be virtually impossible compared to ranking in the top 10 for “3 day Caribbean cruises.”

Also, single key words usually return the least targeted leads. When you target longer keyword phrases there is a much higher likelihood that you have focused in on exactly the right prospects. Someone entering “travel” as their search term could be searching for any type of travel to any worldwide destination. The person who enters “3 day Caribbean cruises” is definitely interested in a 3 day Caribbean cruise, and therefore a strong prospect for such services.

The best thing to do is to target search terms that are multi-word keyword phrases that your target will actually search for and that your competitors may not be using, thereby giving you a better shot at achieving good listings.

Consider including misspelled words

One area that tends to be overlooked when optimizing your site for the search engines is the use of misspelled key words. Including common misspellings, in areas that are not visible to the site visitor (eg. meta tags), of some of your key words may increase the likelihood someone who misspells the keyword will find your site. For example, if ‘accommodation’ is one of your key words you would also want people who enter ‘accommodate’, ‘accomadate’, ‘accommodate’, or ‘accommodate’ to still find your site.

Avoid ‘stop words’

Most search engines conserve resources by disregarding and eliminating common words that are deemed to have little value. These are referred to as ‘stop words’ and include ‘a’, ‘the’, ‘and’, ‘or’, ‘between’, etc. With the elimination of these words, search engines are able to speed up their searches and save disk space. If your site contains ‘stop words’ in areas where a robot looks for copy to index (eg. meta tags, site content), they are ignored, possibly affecting how you’re site will be indexed. Of course it’s impossible to eliminate these words from the majority of the text on your web site. But make every effort to avoid or minimize their use in your title tag, description tag, keyword tag, alt tags, text links and in the first sentence or two of the body copy.

What are meta tags?

Meta tags appear in the code of a web page. They cannot be seen by the site visitor unless they use the “View – Source” command on their browser. (The typical visitor to your site is not likely to be interested in viewing your source code.) However, many search engine robots will view them and take them into consideration when crawling through your site. It’s important to note that creating good meta tags does not automatically equate good search engine listings. They are just one factor. And some search engines don’t read them at all.

There are three basic meta tags to be included on at least every main page.

  1. title tag
  2. description tag
  3. keyword tag

While not technically a ‘meta’ tag, the title tag is included here to ensure that it is never forgotten and to ensure that it is treated with utmost importance.

To be read properly, the title tag must be the first tag. Microsoft FrontPage will place it last. But it’s important to manually cut and paste it so that it appears as the first tag in the head tag of the code.

The title tag is almost always picked up as the hypertext linked title in your listing (the first line, blue and underlined, linked to your web site). Be sure to use search terms in your title that will be searched, will be motivating to be clicked on and, most importantly, that appear in the text of your web page. The title tag should be five to nine words long and should not include any ‘stop words’.

The description tag is also frequently picked up as the description in your listing. It should complement the title and the text found on the page. Remember that your audience for this description is both the ‘robot’ visiting your site and the people conducting the searches who you want to click on your listing to get to your site. This means compromising between a well-written description and one loaded with search terms and avoiding ‘stop words’. The description tag should be between twelve to fifteen words long.

The keyword tag adds search terms for which the search engines may index your site. Keep this list to 20-30 words, avoid ‘stop words’ and avoid single key words. It helps if these search terms are reinforcing the same search terms found in the text of the page.

As there is a limit to the number of words the search engine robots will read within the meta tags, it is generally recommended each section or page of your site focus on a separate set of search terms. In this way, you are not diluting the weight of each search term and are therefore more likely to have your pages well-indexed. Although web sites are designed with the home page as the front door, so to speak, in truth, when another page achieves a good listing on the search engines that is also a good thing. Given that it is so difficult to achieve good listings, any traffic that arrives to the site from a search engine is welcome, regardless of the page through which they entered.

Below is a template for creating meta tags for your site. Just replace the text in blue with your actual title, description or key words.

<TITLE>Title tag should be between 5-9 words, most
important keyword phrases and company name should be
included, though if you want to put greater emphasis
on the keyword phrase omit company name in the title
tag for some pages</TITLE>


<meta name="Description" content=" Description of page,
keyword rich, few if any ‘stop words’.; word count
between 12–15, keyword laden to help get indexed by
search engine robots, but also motivating for people
to click on" />




<meta name="Keywords" content="popular keyword
phrases searchers would use to find information
presented on this page; phrases are separated by
commas, include plural and singular forms, if there’s
a common misspelling to one of your key words
include that too to reach those who misspell!
– Use WORDTRACKER.COM to evaluate your keyword
options; repeat keyword phrases found within page
text, word count between 20–30” />


Another tag – the alt tag


Search engine robots are able to read alt tags behind the graphic images on your web pages. When you mouse over an image you may see a small text box appear. This is the text that appears within the alt tag. The alt tag was designed so that when a site visitor had their browser set to not view graphics, they would still see something of value in the text that appeared instead. With faster modems and high speed Internet access fewer people are turning off the graphics. However, search engine robots are still able to read the text in an alt tag. Each graphic on a page can focus on a few different keyword phrases, as long as they are reinforcing keyword phrases found within the text of the page.


Below is an image code example that includes the alt tag.


<img src="name_of_image.gif" alt="keyword phrase 1,
keyword phrase 2" width="14" height="105" alt="" border="0">


Some organizations choose not to use alt tags, believing that the appearance of the text boxes when visitors mouse over the images would detract from the site’s design. Of course this is fine, as long as the decision is made fully recognizing that this opportunity to further influence the search engine robots is not being used.


Yet another good spot for key words – text links


Keep in mind that the search engine robots read all text contained within the HTML code on your site. This includes the text links you have to other pages. Your navigation system may use graphic images (which the search engine robots can’t read) or it may use text links. If you use text links, consider the words within those text links carefully. Can they be written to include your most important key words? If so, then you’ve got more ammunition in your battle for good search engine listings.


YOUR SITE ARCHITECTURE – IS IT WORKING FOR YOU OR AGAINST YOU?


How your site is built can have a detrimental effect on your search engine listings. If the search engine ‘robots’ aren’t able to navigate through your site or read the code properly, you’re not going to get good listings. Specifically frames, dynamic pages and flash cause the greatest dilemma . That’s not to say your web designer shouldn’t use these technologies. But rather, recognizing their effect on search engine listings, you’re now in a position to take appropriate action with supplemental pages.


Frames


When you view a web page made up of frames, each frame actually constitutes a separate HTML page. The search engine robot can only view one of these HTML pages at a time. If it views the navigation bar, it’s not going to view the page content and will have very little text (or search terms/key words) with which to index the page.


Once in a frames site, the ‘robot’ has tremendous difficulty navigating through the site. If it does manage to index a content page, visitors are directed to that page without the benefit of the navigation system, because the navigation buttons/links appear in a separate frame or HTML page. In any event, neither search engine ‘robots’ nor visitors who find the site on a search engine are able to view the site as the web designer intended.


Using the No Frames Tag


Because the search engine robot is looking for some text, any text, if it indexes the navigation page it will often grab the no frames tag (following the frame set tag) to use as the description in your listing. Because the no frames tag was originally intended to be viewed by those whose browser cannot read frames, web designers would use this tag to merely state “This is a frames enabled site. Your browser does not support frames.”  However, this is a missed opportunity.


Use the no frames tag to include a few or several paragraphs of keyword-laden text. The search engines will have text to read and key words to identify. Visitors whose browser doesn’t support frames will also have information about your business. Include your contact information and these people will be able to contact you for further information.


Below is an example of the no frames tag. Note it appears after the frame set tag which appears after the closing of the head tag.


</head>
 <frameset cols="20%,*" border="10">
 <frame src="contents.html" name="contents">
 <frameset rows="*,4*">
 <frame src="header.html" scrolling="no" name="header">
 <frame src="intro.html" name="body">
 </frameset>


<noframes>


<body>


<h2>
 A headline would go here
 </h2>


An introductory paragraph in which the first sentence or two
 may appear as the description for your search engine listing.


<p>A second paragraph focusing on the main topic of
 your web site.
 <p>
 A third paragraph focusing on a secondary topic of
 your web site.
 <p>
 And so on…
 </body>
 </noframes>
 </frameset>


Dynamic Pages


Dynamic pages are those whose content is stored in a database. The page does not actually exist until the page is called upon and the content brought forward from the database. By way of comparison, static pages are stored on your server waiting to be viewed. The search engine robots typically cannot view pages that aren’t there, so to speak. (Alta Vista and Google can now index dynamically generated pages, but ONLY when those pages are submitted. That is, their ‘robots’ will not find the pages on their own.)


The URL for dynamically generated pages, typically contains a question marks and/or percent signs. The problem is that many robots cannot read beyond these characters. So a URL ending in .cfm or .asp is fine. The problem occurs when the url contains the .asp? or .cfm? extension.


Dynamic pages are an excellent means for managing your site content and could reduce the amount of storage space you need. Do not avoid dynamic pages, just recognize that they require some extra consideration where search engines are concerned.


“Plan B”


A simple fix is to take your most important pages and create duplicate, static pages to submit to the search engines. Just be sure that when the dynamic page is updated, so is the static page. To the visitor, the static page should look no different than the dynamic page. The only difference is that the static page was able to be indexed by the search engines.


Software is available that will eliminate the question mark from the URL. For example XQASP (www.xde.net) is especially developed for web sites seeking better search engine results for their dynamically generated content. Apache web server software includes a rewrite module that will convert URLs with question marks to URLs without question marks. Cold Fusion pages can also be configured to provide search engine friendly URLs. Impress upon your web designer the need to do so. Have your marketing team or those responsible for your search engine listings and your technology team work together. They each possess distinct knowledge and only when they work together will you get the best search engine results.


Flash


Information contained within the use of Flash technology cannot be read by a search engine ‘robot’. This includes text as well as links. Unable to read the text, the ‘robot’ can’t identify the key words for which to index your site. Unable to follow the links, the ‘robot’ can’t navigate through your site to find additional pages.


“Plan B”


If you must use flash, then apply one of these alternatives.


Create an alternative entrance page, one that uses keyword-laden text focusing on your most important search terms for which you want to be indexed. Or create several pages, each focusing on different sets of search terms.


Create your Flash page within a two frame frame set. The first frame can be just one pixel in height – not even noticeable to your site visitors. However, it contains a no frames tag just as mentioned in the Frames section above; thereby giving the search engine ‘robots’ text to read and key words with which to index your site.


SUBMITTING YOUR SITE


We would love to provide you with a list of the URLs for submission pages for each of the search engines. However, one or more of them are likely to be out of date in a few days or weeks! They change that fast!


The simplest thing to do is to visit the home pages of each of the top 10 or top 20 search engines and look for a small text link (often at the bottom of the page) which says something like ‘submit a site’ or ‘add url’. (Note on google.com you have to first click on ‘about google’.


Create a folder in your bookmarks labeled ‘search engine submission pages’ and bookmark all the submission pages for easy future reference.


Should I use one of those automatic submission programs?


They’re everywhere – software programs you can download, services you can pay for – programs that will automatically submit your web site to hundreds, thousands, even hundreds of thousands of search engines and directories!


Beware! First of all, it’s been said that 95 percent of all searches are conducted on the top search engines. Second, many of the sites that make up the thousands of sites being automatically submitted to are Free For All (FFA) pages or other sites designed only to capture your email address and send you Spam! Third, and most important, the major search engines may penalize sites for using automatic submission tools; often by not indexing them at all. Since the top search engines are far too important to risk being penalized, it’s better to take the hour or two to manually submit your site to each search engine or directory manually.


If your site is relevant to a particular region or niche target, then you will want to supplement your top search engine list with regional and niche search engines. You can find lists of these at


<http://www.completeplanet.com/>.


Aren’t search engine listings free?


When search engines began, particularly Yahoo which led the pack, they needed content, they needed listings. So it was free to submit your site to be indexed. Then the number of sites being submitted grew out of control, particularly for the directories whose human resources were being taxed with the number of submissions and the public were upset when their sites weren’t reviewed, let alone indexed. The result was that the directory sites moved to a submission fee model – guaranteeing a site review within a matter of days. Though that didn’t guarantee a listing, as the directory needed to maintain the integrity of its listings. Eventually, on some directory sites, all submissions (LookSmart) or all commercial submissions (Yahoo) required a submission fee.


In answer to the web site owner’s frustration with the difficulty in getting well-listed on search engines, the “pay for placement” model appeared. Advertisers typically bid to pay an amount per impression or per click through. The higher the amount they pay, the higher their listing will appear. Generally, the pay per click model is preferable as you are only paying for actual traffic to your site. There are many such “pay for placement” search engines listed on


<http://www.payperclicksearchengines.com/>


Of course, there are still many search engines to which you can submit your site for free. But more and more search engines are placing paid listings ahead of the free listings. Paying for search engine listings is at least worth testing for your market.


Yahoo and LookSmart


Yahoo has maintained its position as the most popular search engine / directory. For this reason alone, it is recommended that you pay to submit your site. If you meet their criteria, it is likely (though not guaranteed) that your site will be listed. Be sure to include your strongest keyword phrases in your description so that you will be found for those phrases as well as within the category. The Yahoo editor may edit your description, and may change the category in which you are listed, but you will have the opportunity to appeal their decision.


LookSmart’s search results appear on a number of search engines including Microsoft’s MSN, AltaVista, Excite/Webcrawler, NetZero, Netscape,and iWon. If you have the funds to do so, consider taking advantage of LookSmart’s Express Submission service.


How does my site’s popularity affect my listings?


There is an assumption on the part of the search engines that web sites with more inbound links to them are of greater quality. Those with more links are deemed to be more popular. Rightly or wrongly, when a search engine ‘robot’ finds a link to your site when it’s crawling through another site, it adds to your site’s popularity quotient. Since search engines want to serve quality results, web sites that for which the search engine finds more inbound links may be given a higher ranking than those with lower link popularity.  For this reason, it is important to request links from sites with related or complementary content. This can include trade directories, buyers guides, hobby sites, regional directories and much more. Avoid wherever possible trading reciprocal links (ie. a link to their site from your site in exchange for a link on their site) as this requires time to maintain and leads your visitors away from your site. Focus on sites for whom adding a link to your site would be beneficial to their visitors.


HELP! HOW CAN I TRACK ALL MY LISTINGS FOR ALL MY SEARCH TERMS ON ALL THE TOP SEARCH ENGINES?!


Well, you could visit all the search engines and enter your search terms one by one and view the top 100 or top 200 results. That is, if you have days to spend at this task!


Or you could use a fabulous program called WebPosition (www.webposition.com) to do this work for you. This program has many features, but it’s worth the price just for the ‘Reporter’ function. It works in the background, while you’re working on other tasks. Depending on your connection speed it can take just minutes to know if you’re listed in the top 10, 100 or 200 for as many search terms as you request on all the major search engines!


Analyze the results and you’ll know


    

  • which search engines require submitting to again,
    • 

  • which listings you could potentially jeopardize if you resubmit,
    • 

  • which search terms are working and not working,
    • 

  • which pages have been listed for which search terms,
    • 

  • how your listings appear (ie, title and description),
    • 

  • which competitor listings appear higher than yours, and
    • 

  • a whole lot more!
    • 



HOW CAN I FIND OUT WHICH SEARCH ENGINES ARE DRIVING TRAFFIC TO MY SITE?


Talk to your web hosting provider to inquire about a web log analyzer that may already be available or can easily be added to your service.


Two excellent programs are Analog and Summary.


In addition to knowing how many visitors your site receives every day, you’ll also know which search engines they came from and which search terms they entered on those search engines to ultimately get to your site. This is valuable information as you strive to improve your search engine listings over time.




USEFUL LINKS


Alta Vista’s help page

<http://www.altavista.com/help/search/default>


Search Engine Watch

<http://searchenginewatch.com>


Optimizing for international search engines – Part I

<http://www.clickz.com/showPage.html?page=843331>


Optimizing for international search engines – Part II

<http://clickz.com/showPage.html?page=843931>


Over 200 articles sourced by Dr. Wilson of WilsonWeb.com

<http://www.wilsonweb.com/cat/cat.cfm?page=1&subcat=mp_Search>


NEED MORE HELP?


If you would like more help with optimizing your web site for the search engines, submitting to the search engines, increasing your link popularity or further promoting your web site, contact TAPA Communications.


Whether you just want a little guidance and training for your staff to get started, or you want a fully-managed, comprehensive web site promotion plan, we’re here to help you.


Call us at:


Toll free (888) 388-8128

(808) 550-8554


Web: <http://www.tapacom.net>

																	

					
					
					
				


							

	
Comments are closed.