LinkBack URL
About LinkBackswhy?
That's one part of my robots.txt that got indexed. I belive it is strongly against the rules that search engines index sites in the robots.txt.Code:User-agent: * Disallow: /bigjoe/
Edit:
see:
Tillåten=allowed
Community Advocate
why?
That's one part of my robots.txt that got indexed. I belive it is strongly against the rules that search engines index sites in the robots.txt.Code:User-agent: * Disallow: /bigjoe/
Edit:
see:
Tillåten=allowed
Last edited by galaxyAbstractor; 05-18-2008 at 11:46 AM. Reason: Automerged Doublepost
Community Support Team
Sometimes, it takes time to have Google process things.
That was probably from the last time GoogleBot went to your forum.
I suggest waiting a few days and then see what happens.
Community Advocate
Originally Posted by aopsftw
Sometimes, it takes time to have Google process things.
That was probably from the last time GoogleBot went to your forum.
I suggest waiting a few days and then see what happens.
I made that robots.txt at january and google bot can access all. The Bigjoe thing was added 2 weeks ago and google dled it 3 hours ago.
x10 Spammer
How you save your file is it robots.txt or robot.txt ???
please double check that 's' .Many people often make this mistake.
Community Advocate
it is robots.txt but I noticed there is difference between /bigjoe/ and /bigjoe
x10 Spammer
But i think Disallow: /bigjoe/ is correct itself. Am i right?
where is your robots.txt located. Is it right inside your public_html folder itself?
Please give your site URL. Is it an Addon-Domain or Parked one?
x10 Sophmore
you need to disallow /bigjoe/* and /bigjoe/sub_dir/* and all the other sub-folders
don't forget the *
to be on the safe side, just put in every single filename!
If my post helped you, you know what to do... hints: reputation, blue, checkbox
Official Member of the Anti-Apple Club
http://www.robotstxt.org/robotstxt.html
About /robots.txt
In a nutshell
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
* robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
So don't try to use /robots.txt to hide information.
x10Hosting Member
thx
Posting Permissions
Reply With Quote