robotparser
¶
robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt
Classes¶
Entry () |
An entry has one or more user-agents and zero or more rulelines |
RobotFileParser ([url]) |
This class provides a set of methods to read, parse and answer questions about a single robots.txt file. |
RuleLine (path, allowance) |
A rule line is a single “Allow:” (allowance==True) or “Disallow:” (allowance==False) followed by a path. |
URLopener (*args) |