dynamic sitemap

[learning_brain]

I have an image search engine which is a dynamic site leading to lots of variations.

http://www.qualityimagesearch.com/vi...?img_id=******

My issue is the number of images. I now have a dynamically generated sitemap at http://www.qualityimagesearch.com/sitemap.php (which takes a while to load), but google (although registering the total number) is only indexing about 5,000 of them.

Is there a size limit or timeout with sitemaps?

[lemon-tree]

There are indeed limitations to a sitemap:

'Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap'

To get around these size limitations, you can use a sitemap index (e.g. http://flight-plan-database.tk/sitemap-index.xml ) files, which is essentially a sitemap of sitemaps. Each sitemap index can hold up to 50,000 sitemap urls (or 10MB), of which each sitemap can hold 50,000 page urls, so in total you could in theory submit 2,500,000,000 urls using 1 level of sitemap index. If you wanted to try to submit more than that, you'd need more than one sitemap index.
Additionally, Google will not necessarily index every file in the sitemap (e.g. Google indexes about 10% of my pages).

Also, Google will be looking at your sitemap quite often which might be a bit of a drag on your site if the generation is slow; instead you could cache it and regenerate once a day with a cron job of that script.

[robstime]

Nvm answered above.

[learning_brain]

There are indeed limitations to a sitemap:

'Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap'

Yah - this is a problem then....

To get around these size limitations, you can use a sitemap index (e.g. http://flight-plan-database.tk/sitemap-index.xml ) files, which is essentially a sitemap of sitemaps. Each sitemap index can hold up to 50,000 sitemap urls (or 10MB), of which each sitemap can hold 50,000 page urls, so in total you could in theory submit 2,500,000,000 urls using 1 level of sitemap index. If you wanted to try to submit more than that, you'd need more than one sitemap index.
Additionally, Google will not necessarily index every file in the sitemap (e.g. Google indexes about 10% of my pages).

Also, Google will be looking at your sitemap quite often which might be a bit of a drag on your site if the generation is slow; instead you could cache it and regenerate once a day with a cron job of that script.

Hmm - interesting - I noticed in yours, you have each on in a directory, whereas mine will all be at one level. Can I have a sitemap directory of different sitemas all in the same directory and, if so, do all SE's recognise sitemap-indexes?

One last thing on this, I'm guessing your sitemaps are dynamic but are your indexes too?

[lemon-tree]

It doesn't make ay difference whether they're in a folder or not, as long as the URL to the sitemap is correct. I only set it up that way so that I can determine which is which at a glance.

My sitemaps are dynamic and therefore I should really be putting caching on them with regeneration, just I've never got around to it. The sitemap index is just a static file though at the moment, as running a script for two items would be a waste.

[learning_brain]

That makes sense - I couldn't figure how you'd make an effective dynamic index anyway.

I'm guessing you set up each index with a mysql limit 0-1000, 1001-2000, 2001-3000 etc?

[lemon-tree]

Each sitemap is just a direct query with no limit, as I know there are less than 50,000 items that could ever return. It's pretty inefficient, but I have higher priorities.