robotparser.RobotFileParser.parse

RobotFileParser.parse(lines)

parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines.