Optimizing Dynamic Content
Although the recent move by some engines to begin indexing
dynamic content seems to indicate that most engines will soon
accept such content, the current reality is that only a select
few search engines currently index dynamic content.
The most prominent of these are Google, FAST and Inktomi.
Of these, only Google will actually crawl through a dynamic
site, indexing content along the way. In order to avoid getting
caught in a loop, FAST's spider will follow links from a static
page to a dynamic page, but not from one dynamic page to another.
Inktomi will only index dynamic pages submitted through their
paid inclusion program. At $40/yr per URL, this option can
quickly become cost-prohibitive to any site with a large number
of such pages.
For smaller sites with a managable number of dynamic pages,
a simply solution is to create a site map that features links
to each of the dynamically generated pages. This way, a content
management system can still be used to update the dynamic
pages, but search engines have a static page that serves as
a doorway to them so that they can be properly spidered.
For large sites, the best way to assure the spidering and
indexing of dynamic content is to employ one of the many methods
used to change the syntax of a dynamic URL so that it appears
static.
For example:
http://www.company.com/index.cfm?category=hats&product_id=41
would be re-written as:
http://www.company.com/category/hats/product_id/41/index.cfm
There are several ways to change the syntax of dynamic URLs,
including Apache rewrite scripts, Cold Fusion functions, mod
rewrites of URLs on a Unix server and a method that employs
custom 404s. I've outlined a couple options below
Set up your web site so that the dynamic content is contained
in a server side include file. The include file can then by
called by a non-dynamic page. This alleviates the problem
of dynamic content not being spidered because it displays
that same content on a spiderable HTML page where it can be
read and indexed without any problems.
Web sites that use ASP code often use an id as a parameter.
(i.e. widget.asp?id=5) A script can be written that will generate
.asp files for each parameter value on the web site. This
would allow the "widget.asp?id=5" url to be displayed
as "widget5.asp."
|
