Skip navigation

One Year Anniversary Review: Searching and Search Engines

Over the past year, I’ve written a lot about search engines. In many ways, they are the holy grail, the impossible dream, and the unsolved mystery.

We are totally dependent upon them. In order to find anything on the web, we must go through a portal of a search engine. It is the epitome of the phone book for the Internet.

You can buy your way into a listing, or you can push and shove your way in using what are called SEO (Search Engine Optimization) techniques, or you can sit and wait and let search engine nature take its course and crawl up the page ranking naturally. Which one is faster? Paid listings. The other two make no difference in speed today.

Getting in the door of search engines used to take months. First you had to get their attention by submitting your site to them. Then you waited. There are services available which can monitor your site and send you an email automatically to let you know when a search engine web crawler visits. I remember making a chart of when Google, Yahoo, MSN, and others would visit, then searching their search engines to determine how long it took from their visit for my pages to be actually listed. Sometimes it was hours, other times it was months.

Second, you had to wait. You waited for a search engine to visit. Once a search engine visited, you had to wait for their information to be processed into their database, and made available for searching. I remember the early years of brags that so and so search engine had search X million web pages, when in reality, they might have but they were 3-6 months behind processing all the data found. It was a patience game.

Third, you had to do everything you could to your content and code to make sure it was search engine friendly. Every code had to be validated, web designs checked repeatedly for errors that might stop a search engine in its tracks, ruining the very chance you’d been begging for. Every word was evaluated as a potential keyword, and content was keyword packed. You wait a month or two for a visit from a search engine, you learn to do everything you can to make the house ready for their visit because you might not get another chance for a good first impression.

Ah, the “good” old days. Today, pinging services send a knock on the door to search engines and tag services the second your post is published on your blog. The ping is an electronic signal that says “Hey, we got something new here. Come visit!” Search engines arrive within seconds, minutes, or at most hours, and your post is added to their database in seconds. Instantly searchable and findable.

While developing a new blog and web page design recently, I found my test site in Google within 2 hours of installation, saying “hello world” in traditional WordPress fashion before I was ready to say hello to anyone. So much for working on search engine optimization before the search engine finds you today.

In a society that believes in instant gratification, this is exciting stuff. However, inclusion doesn’t get you found. It just gets you included.

Exploring the Mysteries of the Search Engine

As hard as we work to get into a search engine, it’s understanding how a search engine searches, what they find, and what they do with the information they find that is still a mystery to many.

With Google’s search engine algorithm’s patent released to the public last year, Google’s techniques were revealed – if you could understand all the technical babble. I spent some time translating the information and publishing it in “Secret Out – How Google Ranks Websites”. Here are a few highlights of what it takes to pass Google’s test for inclusion and page ranking.

In the past, the number of incoming links scored high, but a judgment on quality of the incoming link source was added to the mix. If the linking page and site had a high page rank value itself, then clearly, it knew a good thing when it linked to it. Still, this link quality aspect became harder to define as so many sites were joining the web and the quality became diluted.

…Google also states very clearly that exchanging and buying links won’t work. They know the tricks and techniques. Also, getting links from documents that have no content, just links, also won’t work. Links without content won’t score high.

…One of the other criteria is the age of the domain. Again, driven by spam sites which pop up and die off quickly, the age of the domain is usually a clue they tend to be in for the long haul…Built into the Google page ranking technique is the ability to track current and historical trends, fads and seasons…This is an interesting aspect of page ranking. If your site continues to push keywords long past the fad’s life span, then this could be seen as keyword spamming. Yet, using trends and fads keywords as they come and go could attract attention. Luckily, the rest of the criteria in the page ranking evaluation can help to clear out abusers of keywords related to the current fad or current event.

…How often you update your pages and add content is monitored over time. It isn’t just how much but when. If you update or add hundreds of articles within a very short time, this is suspicious, but if you rarely update your site or add content over time, then your ranking will probably drop. Finding a happy medium is still a hit and miss angle, but the information seems to point to consistency not just random spurts of energy.

…Keywords and keyword density still play an important roll in evaluating the content and content history. Putting keywords in titles, links, headings, tags, and throughout the page is still critical to the success of your site’s page ranking and keyword ranking results.

…Like other comparative search engines, Google’s patent also tells of how page rankings are compared across the board and monitored over time. The traffic is recorded and monitored. How much traffic each page gets as well as the overall site. User behavior is checked. Google keeps track of how long visitors stay on your site and from what pages they exit your site. You also get points for bookmarking or adding to favorites. Keyword search results are constantly monitored. What keywords brought the visitor to your site and what keywords they used to search once on your site.

…The validity of the site’s code and structure plays a small part, but is still part of the criteria. Make sure your site’s code is validated, checked for errors, and friendly to search engines. Any errors in your page structure or code can easily thwart a search engine’s process through your site. Table designed sites rank low while CSS based designs are much more search engine friendly.

Spelling is still important. Not that Google’s patented page ranking process includes a spell checker – words that are not recognized get dumped. If misspelled keywords are among your missed spellings, then your site will be hurt in the rankings.

A couple things about the patent’s information was really surprising. The fact that link exchanges, especially purchased links, don’t work. In fact, it can be held against you as a penalty.

The other surprising bit of information was that age mattered. I get a lot of email and comments on my blog articles about SEO techniques from people complaining that they get no traffic and they aren’t included in a search engine. I check out their site or blog and they’ve 5 posts written within the past 5 days. Already they’re anxious? Gees, we used to sit back and wait for months. Five days is nothing. No one, in the history of the planet as I know it, had an instant audience for their published material except for Oprah. But she had years of building an audience before the Oprah “O” Magazine was released. Her audience was built-in. Yours is earned.

The older your site is, as well as how active it has been over that time period, the older your outgoing and incoming links, all play a role in helping Google determine, on a comparative basis, how your page should rank in the search engine results. Even if you have more traffic than your neighboring blog, seniority counts.

I’ve spent a lot of time writing about how search engines explore, visit, and see your site. Wanting to explore search engines fully to help you understand, I wrote with “How People Search the Web and How They Can Find Your Blog”, because, no matter what search engines do, if you want to be found, you have to help people find you:

Think about how you search the web. Do you start out vague, guessing at possible keyword combinations? Or do you think it through to come up with the specific phrase or term to narrow your search immediately? Do you often search within those results (refining your search) or start over? How many pages of the results do you go through before you start over or just give up? Do you start with Google or Yahoo, or another search engine? Which search engines are people using the most?

Answering these questions tells you a lot about how people search and about how people would search for you.

In “How Search Engines See, Search, and Visit Your Website”, I wrote:

…a search engine doesn’t care about how pretty your website is and how many pictures or dazzling graphics you have. All it wants is information and that comes from content, titles, keywords, and meta tags. Style sheets are completely ignored as a search engine crawler digs through your site.

Web page designs now encompass more than just some words and a few pictures. A lot of designers are opting for Macromedia’s Flash, vblogging, podcasts, and other visual expressions to showcase their business and products with dramatic visual displays, video, sound, and slide shows. Photographers are definitely taking advantage of all the bells and whistles they can for visual impact.

Unfortunately, when it comes to getting picked up by search engines, if you don’t have text or some basic textual data on your page, it’s difficult for the search engines to gather information about your site. They don’t “look” at the site, they just crawl through it gathering data. No data, no collection, no listing. Even if you use splashy graphics, make sure there is some underlying code that tells the search engines what you do and why they should bother with you.

I wanted you to actually “see” what a search engine sees when they visit your site, which helps you understand what you need to do to clean house in preparation for a search engine visit, so I wrote “See What Search Engines See When They Visit Your Website” and included tips and tools to help you see what search engines see when they visit:

As part of your SEO (Search Engine Optimization) efforts, take time to run your website or blog through a few tests to see how a search engine sees your website or blog when they visit. It’s a good learning experience.

…If you find an element that is stopping the search engine as it moves through your web page or website, then fix it. Check your page for validation errors to ensure you’ve found the possible errors that may distract or halt a search engine’s process through your site.

Closely examine the list of links the search engine crawler finds. Is it finding enough? Is it finding other pages within your site to continue on its path? Or is it finding more links to external sites, encouraging the crawler to leave your site rather than spend a little more time within it, gathering information? Is it finding the “right” links to your categories, Pages, and site search tags? Make the appropriate changes to your navigation links to increase the search engine’s opportunities to flow through your site to see all the web pages.

The value of links within your website or blog, called intrasite links, is more important than you can imagine. Links that provide navigation for the user, and links that connect one post to another, also provide a pathway for search engines to follow so they find all of the pages on your site. In “Exploring How Search Engines Explore”, I wrote:

The core purpose of search engine optimization is:

1. Help search engines search your entire website or blog.
2. Provide key words and phrases to help a search engine determine how to categorize and prioritize your content in their search results.
3. Help the searcher use the search engines to find information on your website or blog.

The latter two points are aspects of search engine optimization you control through the content you provide on your site. It’s what you write, how you write, and the words you write with which get picked up by the search engines’ web crawlers or spiders.

…There is an often overlooked sub-point in the first point. The phrase “search your ENTIRE website”.

Having a well-designed, coded, and keyword packed front page is fine, but what if the search engine crawler can’t get past the first page of your blog? Is the web crawler really finding every page on your site? Or only a few of them? The key to the first point is helping the search engine gather information about every page on your website.

To a web crawler, a link is an open door to pass through and investigate. If it finds content on the other side, it stays, gathering information and looking for more doors to pass through.

With pings, submitting your site to search engines is almost obsolete. You can do it, but it’s redundant if the site is on the list of pinged sites that your blog pings. Older search engine sites and directories, such as international or non-web crawling directories, still require submissions, though that is changing today, too.

I’ve included articles in the lists below to help you submit your site to a wide range of search engines and services, but for the most part, do nothing if you are using a WordPress blog and your blog will be found by search engines. Instant gratification.

Search Engine Optimization – SEO Techniques

Inclusion in a search engine is so important, now that search engine site submissions are practically obsolete, the focus needs to be on how to make your site as search engine friendly as possible.

In a massive “how to” article, I summed up vast years of experience in search engine optimization in “Do-It-Yourself Search Engine Optimization”, listing every technique I know on how to make your site as search engine friendly as possible, and get success with search engine page ranking:

Search Engine Optimization (SEO) is based upon making your blog accessible to search engines to gather information to store in their database, and to help users search for information related to your blog. Remember, SEO means helping search engines and helping users find the information they need when using a search engine, thus encouraging them to visit your website when it appears in the search results.

You can put your do-it-yourself search engine optimization techniques to work on your entire website, but it helps to start with one web page to clean up the core optimization problems, so we’ll start there. When you find consistent and redundant errors, then you can fix them throughout your website or blog.

So, the goal is to help search engine crawlers move through your website collecting information to be stored in the search engine’s database. The key to your blog’s data making it successfully into a search engine’s database is to:

1) Make sure there are no road blocks in the path of a search engine crawler.

2) Make sure the crawler can move through your blog, examining all your web pages.

3) Provide adequate keywords and key phrases which clearly help categorize your content.

4) Provide clearly labeled tags and categories recognized by tagging service crawlers and many search engines today.

5) Take advantage of pinging services.

Many people think that SEO equals income. SEO is not about making money. It is about making your site search engine friendly. I went on to explain:

What is not in this list is anything that has to do with advertising. I want to be clear on this. Search engine optimization is about making your website easily accessible by search engines, it is not about how to make money with your website. Yes, SEO practices can help you make money on your blog, but only because it is lovely underneath the hood to both search engines and users, not because your ads work better. That’s a different subject known as e-commerce, search marketing or website analytics. A lot of people think SEO is e-commerce but it isn’t. Related but not the same. We’ll be focusing on your blog’s relationship with search engines and how this helps you to have a better blog.

Search Engine News and Changes

The search engine has evolved and changed, especially over the past year or so. There are now specialty search engines and directories dedicated strictly to blogs. I wrote about them in “New Search Engines Help Users Find Blogs” and “Upping the Blog Search Ante: Google Blogsearch”.

With the development of website feeds, there are now sites and services which accept feed submissions, which I wrote about in RSSTop55 – Best Blog Directory And RSS Submission Sites.

The public release of the Google search algorithm patents helped reveal even more about the mysteries of how Google ranks web pages and I wrote about those in “Google Page Rank Uses Domain Age to Score”, “Next Generation Search Engine Results May Include Profiling”, “Exploding Blog Page Rank Misconceptions by Recommending a New Page Ranking System”, and “Google Patent News You Need To Know”. The issue of profiling by search engines is one that will rage on for a long time, but it’s already here. Search engines are great record keepers, evaluating how you search, what you search, and when you search, and they are already pushing content towards you, based on your personal history of searching. There will be more on this, so stay tuned.

I also wrote a bit about the business of search engines, especially about Google as they are moving into many fields outside of searching, including “A World Domination Project Coming Your Way – Google World”, “Google Tackles Travel Search”, “Google Page Ranks, Google News, Google Gossip, Google Blues”, “New Google Base is Launched: Good for Selling Stuff?”, and “Google Buys Some of AOL”.

As the mysteries of search engine technology grows, hopefully the responsibilities of a website administrator or blog owner will decrease. It’s happening now as pinging has replaced site submissions in many ways, but the task of making sure the code within the site and the content language is not only search engine friendly, but search friendly, is a never ending task.

Articles about Search Engines, Search Engine Optimization, and Searching


Site Search Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Copyright Lorelle VanFossen, member of the 9Rules Network

Member of the 9Rules Blogging Network

One Comment

  1. Posted September 5, 2006 at 6:38 pm | Permalink

    Hi Lorelle,
    that’s a really nice and extensive recap of your thinking, linking and writing. Will be put to good use.

    Speaking of search engines I hope you’ll bear with me for a slant apporach to your topic while pointing your attention towards a newly started search engine dedicated to helping refugees around the world. Refugees United is a NGO – their website can be found at http://www.refunite.org and their mission in short is to help refugees lost to each other to regain contact and hopefully reunite. I am helping the people behind Refugees United create some interest and publicity towards the project. My simple hope is to get altruistic bloggers and website owners all over the internet to donate at least a link, helping them to gain visibility in the conventional search engines.
    Thanks. Paul.


9 Trackbacks/Pingbacks

  1. [...] Writing up the one year anniversary review of articles I’ve written about searching and search engines, I ran across this interesting bit I wrote in “How Google Ranks Websites”: Spelling is still important. Not that Google’s patented page ranking process includes a spell checker – words that are not recognized get dumped. If misspelled keywords are among your missed spellings, then your site will be hurt in the rankings. [...]

  2. [...] One Year Anniversary Review: Searching and Search Engines [...]

  3. [...] One Year Anniversary Review: Searching and Search Engines [...]

  4. [...] One Year Anniversary Review: Searching and Search Engines [...]

  5. [...] One Year Anniversary Review: Searching and Search Engines [...]

  6. [...] One Year Anniversary Review: Searching and Search Engines [...]

  7. [...] One Year Anniversary Review: Searching and Search Engines [...]

  8. [...] One Year Anniversary Review: Searching and Search Engines [...]

  9. […] One Year Anniversary Review: Searching and Search Engines […]

Post a Comment

Follow

Get every new post delivered to your Inbox.

Join 20,280 other followers

%d bloggers like this: