Block unwanted searches
About robots.txt and search engines
All websites are searched. When and where your site is searched is controlled by a simple text file called the robots.txt. The name is a web-standard and you can make a simple file in any text editor. A robot.txt file is not a security file and does not block folders or files from access – you can simply enter any address from your site and go there. Robot text does block about 99% of unwanted detection and automation from rogue robots and many spammers.
To exclude all robots from the entire server (place file with your index.html file)
User-agent: *
Disallow: /
To allow all robots complete access
don’t use a robot.txt file at all
To exclude all robots from any part of the server
User-agent: *
Disallow: /client/
Disallow: /personal-folder/
Disallow: /guest-ftp/
To allow a single robot
User-agent: Google
Disallow:
To exclude a single robot
User-agent: BadBot
Disallow: /
And you can exclude any single html or document page
User-agent: *
Disallow: /~client/abc/website1.html
Disallow: /~client/abc/website2.html
Disallow: /~client/abc/website3.html
