Revise common web user agents, add command-line and web crawlers

[csarven]

---

Preview | Diff

[jyasskin]

Thanks!

[jyasskin]

I think these are only user agents when they're helping end-users interact with those sites. So the system they're embedded in could be a user agent, but the crawlers and bots themselves aren't.

[martinthomson]

Yeah, I don't agree that web crawlers are user agents, except in the narrow conditions Jeffrey points to. I would not support the notion that they are assumed to be user agents for that reason. Instead, I would say that many tools can sometimes act in the role of user agent, when directed toward achieving goals set by or for a person. Then use crawlers as one of a set of examples, maybe. The example of curl or wget also fits that description, though not when used as part of a scripted CI system, for example.

In all these cases, being a user agent is not the natural state of the thing.

[jyasskin]

This is also related to #12, where we've been asked to clarify how libraries that help implement UAs should interact with this document. I suggest we drop these lines from this change, and address the idea more thoroughly while fixing #12.

[csarven]

We need more discussion/clarification on this so I've created #14 and removed the text

[jyasskin]

I think this is true, at least when an end user is driving curl, as opposed to it being part of a larger system, but I'm not sure that it's worth the reader's time to call it out. Are there any requirements set by the rest of the document that you think command-line tools are at risk of violating? Or other concrete implications of making the reader's mental model include these tools?

[csarven]

We need more discussion/clarification on this so I've created #14 and removed the text