Document in contributing guide what the process is for removing docs

[willingc]

There seems to be some confusion about whether it is best to remove a doc completely or to make it an orphan doc. This is related to redirects. It's beyond my expertise or historical knowledge of the project so will defer to opinions of others.

Once we have consensus or a decision, let's document it. Thanks.

See #1377, #1397

[jeanas]

@pradyunsg made a valid point on #1345: if we redirect to an external domain, Sphinx's linkcheck will not check the redirect link, at least until documatt/sphinx-reredirects#6 is merged, whereas it will check it if we make an orphan page. So, I think it is reasonable to ask for orphan pages in that case.

On the other hand, for an internal redirect, this is not a concern. We know that we don't break links ourselves.

[pradyunsg]

So... The obvious question here is what options do we have and what behaviours do the various options bring?

I'm on a bus ride to get my glasses fixed, so let's see how much I can figure out on this ride.

• Orphan documents
  • HTTP status: 200 OK
  • user-facing behaviour: Clearly indicates that the document no longer exists to users.
  • SEO: Does NOT explicitly direct search engines to the new locations (although they do seem to understand these pages).
  • link fragments: Does NOT preserve link fragments when the page is merely relocated, rather than replaced.
  • mode of operation: Involves use of meta tags for a redirect, or a reference that the user needs to click.
  • version control: as a bunch of extra stub files.
• sphinx-reredirects
  • HTTP Status: 200 OK
  • user-facing behaviour: Clearly indicates that the document no longer exists.
  • SEO: (I dunno)
  • link fragments: Does NOT preserve link fragments when the page is merely relocated, rather than replaced.
  • mode of operation: configurable, but defaults to the as meta tag as orphan files.
  • version control: as a constant in the conf.py file.
• Read the Docs UI redirects
  • HTTP status: 30X Moved (this should play nicer with search engines).
  • user-facing behaviour: somewhat transparent forwarding, since this is an HTTP level forwarding.
  • link fragments: does preserve link fragments.
  • mode of operation: server-side integration, with configuration set up on Read the Docs' project page.
  • version control: as a separate file with supporting automation, if we do Use a version-controlled file as source of truth for Read the Docs redirects #1408.

And, even though the bus had to wait due to a protest calling for a ceasefire in Gaza, I didn't manage to finish this on the bus ride. And, yes, my glasses as fixed.

[pradyunsg]

Having done this review, I think the option I prefer out of all of these is using Read the Docs' redirects.

They are the only option that preserves fragments and transparently forwards readers with an HTTP-level redirect.

I guess past me did something similar along these lines, because I remember now that I wanted to move pip's documentation over to using these redirects that are in that YAML file in the root of that repo. Sorry folks for derping and originally proposing the orphan files approach here. 😅

[sinoroc]

I think I prefer orphan documents. We add a prominent header at the top of the document stating that the document is outdated (and maybe also the reason why) and of course a link to a better resource (or multiple links). Otherwise we do not change (or very minimal change) the content of the document itself. I would also be fine with removing the content and leaving only the deprecation notice. I do not really like redirects on the web.

[jeanas]

I've been reading about how HTML (meta refresh) redirects, as made by sphinx-reredirects, affect SEO. Although there is mixed advice in the wild, the overall theme seems to be "it's probably okay-ish but not recommended, use HTTP redirects instead".

[hugovk]

In general, I think the most important thing is to structure and name things so that readers can get the relevant packaging info they need.

And you've all been doing an excellent job, it's great to see the improvements in recent weeks! 👍

Second, anyone making a website should put effort into avoiding linkrot when possible. It can be surprising how long links can live on elsewhere, and preserving links is part of serving our readers.

• https://www.nngroup.com/articles/fighting-linkrot/
• https://www.theverge.com/2021/5/21/22447690/link-rot-research-new-york-times-domain-hijacking

[jeanas]

#1434 highlights an aspect of the problem which has not been mentioned so far, namely how our restructurings affect projects linking to us with Intersphinx. Specifically, we can always keep labels so that :ref: still works, but there is also :doc:. I have no data on how widely used this is.

Here is how each of the options behaves:

• Orphan documents

  The references keep working. However, site authors probably want to update them with the new location when they notice.

• sphinx-reredirects

  Existing hyperlinks with :doc: on built Sphinx sites will keep working, but builds will start to fail (yes, I've tested it, the redirects are not part of the Intersphinx inventory).

• RTD Web UI redirects

  Same as sphinx-reredirects.

Do we want to reconsider orphan documents in light of this? (My opinion is that the advantages of HTTP redirects are still attractive.)