There was a Google Webmaster event today at the Dubai Press Club in the Dubai Media City. Matthew Chewy Trewhella, a developer advocate from Google, talked about the working of common search engines.
When you specify META keywords on your page, modern search engines compare the keywords with the content of the page to identify web spam.
When Google displays the description for a search result, it usually picks it up from the META description. If this is not specified, Google picks it up from the Open Directory Project (ODP). If there isn't an ODP entry either, Google brings something up on it's own.
Some firms have a corporate website and a micro-website for a product or a project. Google can treat the two sites as one if they have enough cross-links, thus giving the micro-website the PageRank advantage that the parent website has.
The frequency with which Google crawls the website depends on how frequently the website is updated. The PageRank takes about 1-3 months to stabilize so when you launch a new website or change your domain, you would have to wait and watch for a while.
When you're implementing paid links or forums on your site, make sure you add a REL="nofollow" to prevent the target websites from negatively affecting your site's PageRank.
For more control over how search engines visit your website, use a robots.txt file on your website. You can specify the User-Agent, Allow and Disallow to indicate which paths should be avoided and which paths should be indexed by the search engine. A sitemap also helps Google with crawling the website.
The folks at Google are constantly making changes to the search. There are over 200 quality attributes that are used in computing the PageRank and each attribute is managed by a team of about 5 developers. When a tweak is made to a quality attribute, the results are measured and that change is either kept or rolled back.
Trawhella presented the Google Webmaster tools and explained the various options provided by it. There's a lot there that tweaks how Google . He also suggests using the Lynx/Links text browser to see what search engines see on your website.
I didn't stay for the second half in which Trawhella was to speak about Google Analytics and site optimization.