One of the unpleasant aspects of this business is delivering bad news to a prospective client. As in, "I know you just spent six figures on an allegedly search-engine-friendly website, and we're sorry to report that your core navigation system is invisible to Google."

Nobody likes hearing that.

Here at Profit Rank we make a big deal out of the distinction between Search Engine Optimization and Organic Search Marketing. Fundamentally, SEO is a set of web design techniques, whereas Organic Search Marketing is a marketing strategy that often incorporates SEO. Unfortunately, to many clients, SEO is also a magical black box that invites many vendors to play just a little too fast and loose with their assertions. After all, if the client has no way of checking up on you until long after you've cashed his check, why not... er... gild the lilly a bit?

Before SEO

We can probably all agree that a major objective of SEO is to make a given web page as attractive as possible to a search spider. It follows, then, that before one even considers SEO, one should ensure that the search spider is able to access the content.

A search spider is nothing more than a computer program that browses the Web and makes notes about what it finds. There are untold billions of Web pages out there, so it stands to reason that a search spider should be able to do its work as quickly and efficiently as possible. Search spiders accomplish this by ignoring more or less all of the images, layout, and other cosmetic stuff that defines the modern web browsing experience. As a result, a web page appears very different to a search spider than it does to you or I.

A critical point here is that spiders ignore client-side scripting (like JavaScript), embedded objects (like Flash), and Cascading Style Sheets. CSS is primarily used for page layout, so ignoring it has little or no effect on the spider's access to content. JavaScript, on the other hand, is essential technology for building the drop-down menus, sliders, and other controls that comprise the main navigation components on many more sophisticated websites. Depending on how it's built, a JavaScript-based navigation system may or may not be visible to a search spider.

Flash-based navigation systems are always invisible to search.

Note: There have been some rumblings on the web that Google is teaching its spider to run JavaScript. As research performed just this month reveals, though, that hasn't happened yet. Don't hold your breath.  

The hard truth here is that your expensive new drop-down menu system may actually render your new website less visible to search than the clunky old site it replaced. The reason is that, while your human users can comfortably access every page on your site within one or two clicks of your menu system, the search spiders may be forced to take the long way around, via your site map and any links they can find scattered about your pages.

But short of waiting to see what pages actually get indexed, how can you tell whether and to what extent your new site is accessible to search?

Lynx to the Rescue

Lynx was one of the first web browsers ever written. Released into the public domain by the University of Kansas in 1993, it remains by far the most widely-used text-based browser.

Lynx is important because it gives the user a view of the Internet that is very similar to what a search engine spider sees. Critically, Lynx does not run client-side scripts, so the user will experience similar constraints caused by scripted navigation elements. The contrast between a website's appearance in Internex Explorer or Firefox and its presentation in Lynx can be striking, to say the least.

Note: Lynx is a console application, and can be somewhat of a challenge for a novice to configure and run. To make things as simple as possible, we've compiled a special installation with instructions for Windows users. Click here for more information.

Have a look at your website in Lynx, and many search indexing questions answer themselves. For clients of SEO agencies and web development firms, Lynx in an invaluable tool for evaluating whether the product really lives up to the vendor's assertions.

The bottom line: if you can't see it in Lynx, then Google can't see it, either!

You Start From Where You Are

So let's say the worst has happened: your company has spent hundreds of thousands of dollars on a fancy new website, and Profit Rank's free Lynx download reveals that most of the site is buried beneath a scripted menu system that Google can't penetrate. What do do?

First, don't panic! The good news is that modern navigation systems are highly modular, and with a little pressure—doesn't your contract specify a search-engine-friendly site?—you may be able to convince your vendor to replace yours.

Of course, your vendor may not know how to do that. Many older web development firms use a templated navigation system that they developed in-house and have been tweaking ever since. If they went the wrong direction with it in 2001, fixing it might cost them more than your contract is worth.

In that case, you may have to bring in a third party. Again, don't panic: if your site is data-driven, as most significant sites are, there are a number of options out there that can salvage the majority of your old vendor's code while swapping out the navigation system.

Whatever you do, look before you leap. Just like lawyers and accountants, vendors in the Web Development and Internet Marketing space have a vested interest in making their craft seem as technical and labyrinthine as possible, so as to keep you dependent and themselves employed. Sometimes things really are that complicated... but mostly they aren't.

An hour or two of early consultation with a knowlegable and impartial third party can save you boatloads of cash and grief down the road.