Kaydet (Commit) 2d95f1ad authored tarafından Raymond Hettinger's avatar Raymond Hettinger

SF patch #911431: robot.txt must be robots.txt

(Contributed by George Yoshida.)
üst 3aa82c07
...@@ -83,7 +83,7 @@ class RobotFileParser: ...@@ -83,7 +83,7 @@ class RobotFileParser:
self.entries.append(entry) self.entries.append(entry)
def parse(self, lines): def parse(self, lines):
"""parse the input lines from a robot.txt file. """parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by We allow that a user-agent: line is not preceded by
one or more blank lines.""" one or more blank lines."""
state = 0 state = 0
...@@ -148,7 +148,7 @@ class RobotFileParser: ...@@ -148,7 +148,7 @@ class RobotFileParser:
def can_fetch(self, useragent, url): def can_fetch(self, useragent, url):
"""using the parsed robots.txt decide if useragent can fetch url""" """using the parsed robots.txt decide if useragent can fetch url"""
_debug("Checking robot.txt allowance for:\n user agent: %s\n url: %s" % _debug("Checking robots.txt allowance for:\n user agent: %s\n url: %s" %
(useragent, url)) (useragent, url))
if self.disallow_all: if self.disallow_all:
return False return False
......
...@@ -1962,7 +1962,7 @@ repr Redo repr() but with limits on most sizes. ...@@ -1962,7 +1962,7 @@ repr Redo repr() but with limits on most sizes.
rexec Restricted execution facilities ("safe" exec, eval, etc). rexec Restricted execution facilities ("safe" exec, eval, etc).
rfc822 RFC-822 message manipulation class. rfc822 RFC-822 message manipulation class.
rlcompleter Word completion for GNU readline 2.0. rlcompleter Word completion for GNU readline 2.0.
robotparser Parse robot.txt files, useful for web spiders. robotparser Parse robots.txt files, useful for web spiders.
sched A generally useful event scheduler class. sched A generally useful event scheduler class.
sets Module for a set datatype. sets Module for a set datatype.
sgmllib A parser for SGML. sgmllib A parser for SGML.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment