It was thus said that the Great Natalie Pendragon once stated:
> For the GUS crawl at least, the crawler doesn't identify itself _to_
> crawled sites, but it does obey blocks of rules in robots.txt files
> according to user-agent. So it works without needing a user-agent
> header.
>
> It obeys user-agent of `*`, `indexer`, and `gus` in order of
> increasing importance.
>
> There's been some talk of the generic sorts of user-agents in the
> past, which I think is a really nice idea. If `indexer` is a
> user-agent that both sites and crawlers had some sort of informal
> consensus on, then sites wouldn't need to worry about keeping up with
> any new indexers popping up.
>
> Some other generic user-agent ideas, iirc, were `archiver` and
> `proxy`.
That's a decent idea, but that still doesn't help when I want to block a
particular bot for "misbehaving" (in some nebulous way). For example,
there's this one bot, "The Knowledge AI" which sends requests like
/%22http:/wesiseli.com/magician/%22 [1]
(and yes, that's an actual example, pulled straight off the log file). It's
not quite yet annoying enough to block [2] but at least I have some chance
of blocking it via robots.txt (which it does request).
-spc
[1] I can't quite figure out why it includes the quotes as part of the
link. *All* the links on my websites look like:
<a href="
http://example.com/">
and for the most part, it can parse those links correctly. And
that's not limited to just *one* bot, but several of them have that
behavior.
[2] Although it's nearly impossible to find anything out about it as the
user-agent string is literally "The Knowledge AI", so it might be
worth blocking it just out of spite.
![Atom Feed](data:image/svg+xml,<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg"
id="RSSicon"
viewBox="0 0 8 8" width="16" height="16">
<title>RSS feed icon</title>
<style type="text/css">
.button {stroke: none; fill: orange;}
.symbol {stroke: none; fill: white;}
</style>
<rect class="button" width="8" height="8" rx="1.5" />
<circle class="symbol" cx="2" cy="6" r="1" />
<path class="symbol" d="m 1,4 a 3,3 0 0 1 3,3 h 1 a 4,4 0 0 0 -4,-4 z" />
<path class="symbol" d="m 1,2 a 5,5 0 0 1 5,5 h 1 a 6,6 0 0 0 -6,-6 z" />
</svg>)