Robot Meta Tag Question

[moniquedbabin79]

Hello,

I am VERY new to HTML/XHTML and web design. I've dabbled a bit, but I'm finally taking a class and learning how build a page the right way.

I'm just learning about meta tags and how I can use them to make my site more visible on search engines. That all seems pretty clear.

What I don't yet understand is the robot tag. I understand that this either instructs robots to crawl and index (or not to crawl and index) my page and to follow (or not to follow) the links to crawl and index those pages.

What is unclear to me is whether or not this is a good thing. It seems like it would be a good thing in order to make my site more visible/searchable on the search engines. What are the reasons that I might want to include the "index" and "follow" content in the robot meta tag? What are the reasons that I might want to include "noindex" and "nofollow?"

What happens if I don't include a robot tag at all?

Thanks to anyone who can help me understand this tag!

Cheers,

M. B.

[techairlines]

If you don't include the robot tag at all, it assumes the default index, follow, so if this is what you want, the robot tag is not needed.

You want index, follow if you want search engines to index that page as well as click through all the links on that page and possibly add those as well. You'll want noindex and nofollow for private pages that you don't want to be made public on search engines.

[moniquedbabin79]

Thank you! That's what I thought, but it was hard to tell from reading the literature.

I definitely want my site indexed. If any pages are private, I plan to password protect them.

Thanks for your time!

[michman]

welcome to the forums!

[leafypiggy]

Robot files are very useful, but bear in mind:

NOT ALL CRAWLERS (ROBOTS) HAVE TO LISTEN TO ROBOT META TAGS, OR ROBOT.TXT FILES!

Many, like Google, Yahoo, Bing, etc, listen to your instructions, but many more, including spam bots, don't pay attention to these files and do whatever you want.

Even if you password protect your files, you want to tell google, yahoo, etc, not to index them! Hackers can see these files, and they will be able to find them, and usually break encryption very easily, especially if you're new to the game!

Signing up for tools like http://google.com/webmastertools , https://siteexplorer.search.yahoo.com/ , and http://bing.com/webmaster/ can help you increase your traffic, site index info, keywords, and a lot of other things. They're very useful, even to us veterans!

[bhupendra2895]

You can put robots.txt at the root of your server to tell the search engines what not to index and xml sitemap file to tell search engines what to index and how to index.

[moniquedbabin79]

Thanks, everyone! These tips are *really* helpful! For now, I'll definitely stick with putting stuff on the web that I don't care if it gets crawled. In fact, it sounds like if I want to publicize my business that I will want to be sure that I get indexed.

Great advice so far! Thanks! (That's also a good point about how the robots won't necessarily obey your noindex/nofollow command. Creepy! Of course, we all know that there's no real privacy on the web . . . .). Sigh.

If anyone else has additional advice, beyond meta tags, about how to get noticed online, I would love to hear it. I'm thinking about using a web site to begin contracting my services as a professional writer and instructional designer. There are many out there, so I'm sure it will be a challenge to make myself stand out among the others.

I can already see that these forums will be a useful tool!