Google Sitemaps

bjsteiger · 04.06.2005, 22:40

Google have started a new service called Google Sitemaps - a collaborative crawling system for improving the coverage and freshness of Google index. Participation is free. Inclusion isn’t guaranteed https://www.google.com/webmasters/sitemaps

Google Sitemaps is intended for all web site owners, from those with a single web page to companies with millions of ever-changing pages. If you want Google to crawl more of your web pages or want to be able to tell Google when content on your site changes, this is the service for you. Using Sitemaps, Google hopes to expand the coverage of the web and improve the time to inclusion of website content in their index. They however don’t guarantee that they will l crawl or index all of your URLs.

However, the problem is that it doesn't pick up the dynamic pages of Mambo or any other dynamic application. I am not a php programmer, but I noticed that osCommerce modified the application so that it could read their dynamic pages. I thought this would be an excellent addition to Mambo. I wish I knew how to program to do this -- is there anyone out there that could figure this out?

denisbaldwin · 06.06.2005, 07:58

Is there any reason not to do it? Not that I don't believe Google has the best intentions.. but is there something about this that might hurt your rankings on other engines? or do some other damage?

gin · 06.06.2005, 11:10

It's only my IMHO but I also have some doubt about the recent real intentions of Google.

Bye,
Giorgio

KriKit · 06.06.2005, 11:15

I don't think Mambo's xml conforms to googles specifications for sitemap...
https://www.google.com/webmasters/si.../protocol.html

Perhaps a new Mambot is in order?

I don't think this is a bad thing. It allows the site to request when google should update its index of a website. However one major problem... How do they know your the owner of the site? I guess thats what the XML specifications are for, to insure this doesn't happen.

bjsteiger · 07.06.2005, 17:10

Quote:

Originally Posted by denisbaldwin

Is there any reason not to do it? Not that I don't believe Google has the best intentions.. but is there something about this that might hurt your rankings on other engines? or do some other damage?

From Google's site:

Quote:

The Sitemap Protocol allows you to inform search engine crawlers about URLs on your Web sites that are available for crawling. A Sitemap consists of a list of URLs and may also contain additional information about those URLs, such as when they were last modified, how frequently they change, etc.

Sitemaps are particularly beneficial when users can not reach all areas of a Web site through a browseable interface — i.e. users are unable to reach certain pages or regions of a site by following links. For example, any site where certain pages are only accessible via a search form would benefit from creating a Sitemap and submitting it to search engines.

The resulting output from the xml file is simply a url and the frequency of the content change. From what I have learned from other sites, sitemaps are a very positive thing to most search engines because it assists the spiders crawl your site more efficiently.

Here is an small sample of the output XML:

Code:

<url>
  <loc>http://www.echodeep.com/about.php</loc> 
  <lastmod>2005-04-23T22:15:35Z</lastmod> 
  <priority>0.5</priority> 
</url>
<url>
  <loc>http://www.echodeep.com/archives</loc> 
  <lastmod>2005-06-04T18:20:26Z</lastmod> 
  <priority>0.5</priority> 
</url>
<url>
  <loc>http://www.echodeep.com/archives.php</loc> 
  <lastmod>2005-06-01T18:36:02Z</lastmod> 
  <priority>0.5</priority> 
</url>
<url>
  <loc>http://www.echodeep.com/archives/2005</loc> 
  <lastmod>2005-06-01T18:36:01Z</lastmod> 
  <priority>1.0</priority> 
</url>

While, I understand the concern I do not think that will happen in this instance.

bjsteiger · 07.06.2005, 17:14

Quote:

Originally Posted by KriKit

I don't think Mambo's xml conforms to googles specifications for sitemap...
https://www.google.com/webmasters/si.../protocol.html

Perhaps a new Mambot is in order?

I don't think this is a bad thing. It allows the site to request when google should update its index of a website. However one major problem... How do they know your the owner of the site? I guess thats what the XML specifications are for, to insure this doesn't happen.

So what do you think would take to make a mambot?

ApSpecialists · 07.06.2005, 23:56

This may only be an issue for those that can't use SEF for some reason - or - where they do, but don't update their 'robots.txt' file so that Google (or any other crawler) knows where to go. For instance... If you use a subdirectory for Mambo, the crawler checks the root directory's file by default - and never compares it to any other robot file(s) elsewhere on the site. If it doesn't find one at all, it will only follow the links embedded within the default 'home page' in the root directory.

Google is better than most when it comes to interpreting dynamic tags (? or &, etc.) but if the string is too long it will truncate it, and not crawl a page as intended. I suspect that if there's too many parameters, it just doesn't cache the too-long string while it does the crawl.

I use SEF on every site I've done, and there's never a problem with having Google properly index the sites - except - where session ID's or some other 'hashed' strings are included with the URL (mambo-phpShop, for instance). Session numbers change, and hashed strings may contain characters that 'break' the page crawl.

Also... it would help even more to use the per-page meta tags that Mambo provides. Better that you choose the keywords and key-phrase when the specific page's content differs from the overall meta tags for the entire site.

gobayview · 08.06.2005, 01:06

Quote:

Originally Posted by KriKit

I don't think Mambo's xml conforms to googles specifications for sitemap...

This is very easy to do if you have the Xaneon Extensions compoent installed. Everything you need is right there in the mos_sef table. I've done my own, but the code is quite sloppy so I wouldn't presume to offer it up. I'll check with Xaneon to see if it's something they might consider.

Spice

KriKit · 08.06.2005, 12:43

Quote:

Originally Posted by gobayview

This is very easy to do if you have the Xaneon Extensions compoent installed. Everything you need is right there in the mos_sef table. I've done my own, but the code is quite sloppy so I wouldn't presume to offer it up. I'll check with Xaneon to see if it's something they might consider.

Spice

I'm not sure I know what you mean, I've installed this component but am unable to see how I can use this to change/parse rss feeds from my site.

gilahacker · 15.06.2005, 18:48

Quote:

Originally Posted by KriKit

I'm not sure I know what you mean, I've installed this component but am unable to see how I can use this to change/parse rss feeds from my site.

@KriKit: From what I've read... it's not going to read your rss feeds. It just uses an xml file which is basically a list of your site's page url's with information on how important they are and how often they are likely to change.

Now for my question:

Wouldn't it be best for mambo users to have it check their Apache access log instead of trying to have a script generate a big ol' xml file of all your url's?

That way google gets a list of the pages people are actually viewing on your site and they might be ranking them (in importance) by the amount of hits they're getting. (Kinda scary giving google that information isn't it?)

This of course wouldn't work for newer sites that don't have any real traffic yet.

One of my sites is a news site and new content is added on an almost daily basis, there are already several hundred pages on the site, having a script run at specified intervals to keep an updated list of urls seems a little crazy when the access log practically provides that information already.

Now if I could just get to my access log! I'm beginning to think that my webhost doesn't allow direct access to it cause I can't find it!

My main site is getting a few hundred visitors a day from google and google's got almost all my pages indexed already. I'm really wondering if google is going to give priority in rankings to sites that are using the sitemaps goober as opposed to sites that aren't. That's the only reason I'm even checking into this...