The robots.txt file!

[god]

I thought i'd give you guys an idea about this file!
As we all know, the google crawler engine (the one that gets files into the google database, so you can search them) has access to pages in websites, that you don't have access to ! like a /admin/ folder that's .htaccess'd (password protected)
now website owners dont want people to know the content of these directories, so they .htaccess them (password protect them), but with a google search + google cache, u can still see the contents! so they should stop google from accessing these directories.. how ? in the site's main, they make a file called robots.txt
for example http://www.********.com/robots.txt
the robots.txt file looks like this:

# everything after a "#" is not taken into consideration
# you can write anything here ! 
# ammouna
User-agent: * <<< specifies which user agent should be allowed
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/

so these directories wont be shown by the google search, and are secure!
Now how you can actually use this info ? it depends it can be quite useful, and sometimes meaningless!

the most famous robots.txt file on the net is ..... uh ... nsita lol ill update it later..

ya i found it here it is www.whitehouse.gov/robots.txt
check it out its ok to open it :P

[OS7]

MAn what do u meen by this ma fhemet chi
Can you please tell us what do u meen directly? and thank you

[SysTaMatIcS]

so wats the use of the robot.txt , we wont have access to the files , i dont get it

[Krazy]

As I understood...
If you add it in your website, then google search won't access the files that are in the folders mentioned.

But I think we can access the pages if we directly try to access the folder.
So you'll know what they don't want google search access those folders.

[Justin]

systa... its not .exe wtf?!

it's a .txt file... a notepad file.. let's call it.. gaining useful infos... & a big thk u for God for these amazing posts & ideas that without him we wouldn't learn them...

[god]

Originally Posted by systamatics

so wats the use of the robot.exe , we wont have access to the files , i dont get it

in reply to that:

Originally Posted by god

Now how you can actually use this info ? it depends it can be quite useful, and sometimes meaningless!

Figure that out on yourself :P as i said it depends, if you're trying to find something in a website, this file might give u a better understanding of the structure of the site, and maybe tell you what CMS the website is using... look i cant tell u what to do letter by letter that would be illegal :P + i wont do it hehe