A user agent is a software agent that acts on behalf of the user. The user agent acts as a client in a network protocol.
A site owner wishes to give instructions to a web robot will place a text file, robot.txt in root of the website. The file functions as a request that robots do not access certain pages, for example out of preference for privacy from search engine results, or avoidance of misleading categorization of a site. Search engines follow this standard.
robot.txt is not a useful means of security for restricting access from bots. Security by obscurity is not encouraged and not secure.