[reST] refactor

[jrappen]

Note

This PR is mainly meant to make future work easier, not necessarily
fix stuff now. If you have suggestions for stuff that should be fixed
now, leave a comment below.

---

current TODOs

• compare comments in syntax file
• add tests to prove linked issues can be closed

---

Additions:

• symbol index lists implicit hyperlink targets (sections,
  footnotes, citations)
• highlight embedded python and raw-code directives (css, js, json,
  jsonc, html, python, toml, yaml), compare [reStructuredText] Code-blocks are not handled #3158
• explicit hyperlink targets and anonymous explicit hyperlink
  targets
• correctly highlight grid tables with optional inline markup
• correctly highlight inline interpreted text with optional
  interpreted text roles
• completions with strict scope limits
• indentation settings based upon official docs
• neg. tests for comments
• highlight empty comments and section separators between empty
  lines

Fixes:

• line blocks (with optional inline markup)
• footnotes and citations
• pop explicit markup blocks before the next explicit markup block
• correctly highlight block comments
• include inline markup within block quotes
• allowed symbols for heading markers, compare [reStructuredText] Headers using some of the supported characters aren't highlighted #2894
• inline internal targets may contain symbol characters, compare
  [RestructuredText] single quotes break link targets #1204
• inline hyperlink references
• section headings with or without adornments
• literal blocks meta scope
• enumerated lists
• field lists
• punctuation of inline substitution references
• tests for embedded python

Changes:

• the syntax file now uses branching to correctly highlight
  section headings and their punctuation
• scope names have been adjusted with regard to the syntax
  scope naming guide
• match punctuation.definition.end rules for inline items
  before invalid.illegal.newline
• pop line blocks on empty line
• pop explicit markup blocks before next explicit markup block
• moved some rules from block-quote-block context to literal-block
• removed inline markup from literal blocks
• updated list of allowed characters for internal link labels,
  compare [reST] Internal link labels should allow more character types #793
• changed auto_complete_selector default
• re-ordered test file to match order of syntax file
• include linebreak in match for first line of directives
• differentiate known (constant.language) and unknown (constant.other)
  directive names

References:

• fixes [reST] Internal link labels should allow more character types #793
• fixes [RestructuredText] single quotes break link targets #1204
• fixes [reStructuredText] Headers using some of the supported characters aren't highlighted #2894
• fixes [reStructuredText] Code-blocks are not handled #3158
• duplicate the work @deathaxe had done in [Common] Add git conflict marker highlighting #4047, so his changes to the
  syntax don't get lost

Thanks to these contributors:

• deathaxe [email protected]
• Giampaolo Rodola [email protected]
• Victor Wheeler [email protected]

[giampaolo]

Thanks a lot for taking care of this.

• If possible, could you also add syntax highlighting for these kind of links?

  This is a paragraph `Process.exe()`_.
  
  .. _`Process.exe()`: https://psutil.readthedocs.io/en/latest/#psutil.Process.exe

Example file having lots of these: https://raw.githubusercontent.com/giampaolo/psutil/refs/heads/master/HISTORY.rst.

[vwheeler63]

Hello, @jrappen !

Good morning! And thank you for the invitation in sublimetext-io/docs.sublimetext.io#131.

My name is Vic.

I have been implementing these reST commands as I have found the need for them in my work and writing.

import sublime
import sublime_plugin


class VicsRstEncapsulateFieldCommand(sublime_plugin.TextCommand):
    """
    If (conditions applied in keymap):
    - we are in a reStructuredText source file, and
    - all selections have some text selected

    then wrap the selected text like this:

        :selected text:
    """
    def run(self, edit):
        sel_list = self.view.sel()

        # Replace each selection with edited string.
        for rgn in reversed(sel_list):
            selected_text = self.view.substr(rgn)
            new_text = ':' + selected_text + ':'
            self.view.replace(edit, rgn, new_text)

        # De-select each selection, leaving cursor on right edge of its
        # previous selection.  This avoids the danger of a stray keystroke
        # wiping out the new text.
        for i, rgn in enumerate(sel_list):
            del sel_list[i]
            sel_list.add(sublime.Region(rgn.end()))


class VicsRstEncapsulateLiteralCommand(sublime_plugin.TextCommand):
    """
    If (conditions applied in keymap):
    - we are in a reStructuredText source file, and
    - all selections have some text selected

    then wrap the selected text like this:

        ``selected text``
    """
    def run(self, edit):
        sel_list = self.view.sel()

        # Replace each selection with edited string.
        for rgn in reversed(sel_list):
            selected_text = self.view.substr(rgn)
            new_text = '``' + selected_text + '``'
            self.view.replace(edit, rgn, new_text)

        # De-select each selection, leaving cursor on right edge of its
        # previous selection.  This avoids the danger of a stray keystroke
        # wiping out the new text.
        for i, rgn in enumerate(sel_list):
            del sel_list[i]
            sel_list.add(sublime.Region(rgn.end()))


class VicsRstEncapsulateInterpretedTextRoleCommand(sublime_plugin.TextCommand):
    """
    If (conditions applied in keymap):
    - we are in a reStructuredText source file, and
    - all selections have some text selected

    then wrap the selected text like this:

        :role_name:`selected text`
    """
    def run(self, edit, role_name):
        sel_list = self.view.sel()

        # Replace each selection with edited string.
        for rgn in reversed(sel_list):
            selected_text = self.view.substr(rgn)
            new_text = f':{role_name}:`{selected_text}`'
            self.view.replace(edit, rgn, new_text)

        # De-select each selection, leaving cursor on right edge of its
        # previous selection.  This avoids the danger of a stray keystroke
        # wiping out the new text.
        for i, rgn in enumerate(sel_list):
            del sel_list[i]
            sel_list.add(sublime.Region(rgn.end()))

    // --------------------------------------------------------------------
    // reStructuredText Tools
    // --------------------------------------------------------------------
    {
        "keys": ["ctrl+alt+f"],
        "command": "vics_rst_encapsulate_field",
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },
    {
        "keys": ["ctrl+alt+l"],  // Lower-case 'L'
        "command": "vics_rst_encapsulate_literal",
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },
    {
        "keys": ["ctrl+alt+t"],
        "command": "vics_rst_encapsulate_interpreted_text_role",
        "args": {
            "role_name": "term"
        },
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },

Use Case for vics_rst_encapsulate_field

This happens a lot working on other people's writings when they use (or over-use) unordered lists, or even worse, definition lists, that make what SHOULD be a nice, compact 8-item list of things into a pages-long thing that is more difficult to read. Then I convert these to field lists, which, in the RTD theme, makes a nice, compact, very readable list.

Use Case for vics_rst_encapsulate_literal

Occasionally I see something like FILE PATHS in a document are inconsistently marked—some as literals, some not—and I go through, say, 100 pages and make them all literals for consistency.

Use Case for vics_rst_encapsulate_interpreted_text_role

While building a glossary of a large work recently (over 100 pages), I lost count of how many times vics_rst_encapsulate_interpreted_text_role got used to create

:term:`word_or_phrase_in_glossary`

And that's just in 1 "document"!. It saved TONS of otherwise-error-prone keystrokes!

I expect more of these interpreted-text roles to come in the near future. I know vics_rst_encapsulate_interpreted_text_role is long, but I used it specifically so that people new to reST will be learning the correct terminology and so will know what to look up when they are learning reST

---

Helping Others Learn reST

I noticed you listed 3 useful reST learning and reference links in the Preferences > Package Settings > reStructuredText (reST) > Documentation portion of your reST menu.

I have accumulated a few more Docutils reference that I find I use regularly.

Additional Thoughts

reST alone, for any serious work, is kind of "bare-bones" without Sphinx. Sphinx should be very interesting to Sublime Text users because it is Python based, and like Sublime Text, is very customizable, and outputs in [html, dirhtml, singlehtml, json, htmlhelp, qthelp, devhelp, epub, latex, text, man, texinfo, xml, pseudoxml], and can even do external hyperlink checks, among other things.

- Documentation
    - Docutils (Basic reST)
        - [reST Introduction](https://docutils.sourceforge.io/docs/ref/rst/introduction.html)
        - [reST Primer](https://docutils.sourceforge.io/docs/user/rst/quickstart.html)
        - Reference
            - [Markup Specification](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html)
            - [Tables](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#tables)
            - [Substitution](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#substitution-references)
        - [Directives](https://docutils.sourceforge.io/docs/ref/rst/directives.html)
        - [Interpreted Text Roles](https://docutils.sourceforge.io/docs/ref/rst/roles.html)
        - [Examples](https://docutils.sourceforge.io/docs/user/rst/demo.html)
        - [Quick Ref](https://docutils.sourceforge.io/docs/user/rst/quickref.html)
    - Sphinx (Sphinx-Extended reST; Extends Docutils Basics)
        - [Command Line Tools](https://www.sphinx-doc.org/en/master/man/index.html)
        - [sphinx-build Command Line](https://www.sphinx-doc.org/en/master/man/sphinx-build.html)
        - [Main Documentation Hub](https://www.sphinx-doc.org/en/master/)
        - [Configuration](https://www.sphinx-doc.org/en/master/usage/configuration.html)
        - [Directives](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html)
        - [Directives—Admonitions](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#admonitions-messages-and-warnings)
        - [Hyperlinks](https://www.sphinx-doc.org/en/master/usage/referencing.html)
        - [Interpreted Text Roles](https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html)
        - [Glossary](https://www.sphinx-doc.org/en/master/glossary.html#term-environment)
        - [Supported Programming Languages](https://pygments.org/docs/lexers/)
        - Themes
            - [Sphinx Themes Gallery](https://sphinx-themes.readthedocs.io/en/latest/)
            - [Customizing Output Using HTML Templates](https://www.sphinx-doc.org/en/master/development/html_themes/templating.html)

And between the Docutils and Sphinx documentation, the subject of hyperlinks is SO POORLY DOCUMENTED that I wrote my own reST + Sphinx Hyperlink Documentation here—which, IMO, is FAR easier to understand. (There are about 12 ways to create hyperlinks, and only about 8 of them are useful.)

I hope this helps.

Kind regards,
Vic

[jrappen]

@vwheeler63 Thanks Vic!

I do have a question, though. When replacing multiple selections, why go forward through the list instead of backwards?

You say this is well tested, I ... doubt it?! Because if you go forward you'd have to re-adjust the current selection and all following ones every step, if you go backward you can just replace. Your way doesn't make sense in my mind, without testing. I'm fairly certain, I've seen other plugins use the reverse order for the same reason, too.

[vwheeler63]

I do have a question, though. When replacing multiple selections, why go forward through the list instead of backwards?

You say this is well tested, I ... doubt it?! Because if you go forward you'd have to re-adjust the current selection and all following ones every step, if you go backward you can just replace. Your way doesn't make sense in my mind, without testing. I'm fairly certain, I've seen other plugins use the reverse order for the same reason, too.

That's a great question! Interestingly, I initially made it backwards using (reversed()) but after closely examining the py38/sublime.py file to see what was "facade" vs what was "real" (e.g. View objects are just plain Python objects with a view_id, selection and settings_object properties -- all the actual ACTIVITY is carried on behind the borders of the sublime_api object), I found out that as the loop is iterating, it actually does a fresh fetch of the 2nd, 3rd and subsequent regions. So when the first region does the self.view.replace(...), it actually pushes the REAL regions (still inside Sublime Text) forward, so all the replaces happen as expected! I didn't know that until I studied the class Selection's __iter__(self, id) function closely to see that fresh fetch happening with each loop iteration.

sublime.Selection objects are 100% facade (just a Python object with a view_id integer), so everything that happens with them is as passed in on the __getitem__(), __delitem__(), etc. functions, and the ID is passed into the sublime_api functions.

There are definitely some things (when additional editing is being done) that require keeping track of the displaced text. I am developing another complex packages that does so in several places.

I am aware that some kind of self.view.replace() calls destroy the Point indices of the incoming regions, and force you to have to do replace operations in reverse order, or else keep track of the displacement. But for whatever reason, this replace loop preserves the selections and displaces them automatically. Using reversed() in the loop has the same result if you are more comfortable with that.

[vwheeler63]

@jrappen with regard to forward-loops working:

Case 1:

If you iterate through a list of regions that are anything other than a sublime.Selection object, then they are just a Python list of regions, which are themselves merely Python objects, and are NOT connected with Sublime Text. The result of this is that any insertions and deletions in the buffer (including self.view.replace(...)) cause the buffer context to shift, but because the list of regions is DISCONNECTED from the actual buffer, the indices contained therein get invalidated each time the buffer contents shift. The result is that you have to either do them in reverse (e.g. using reversed()), so the unprocessed regions are not invalidated before they are processed, or keep track of the number of characters of "shift" you have introduced into the buffer.

Example:

    pat = pc_setting.placeholder_line_regex.pattern
    flags = sublime.FindFlags.NONE  # NONE = regex
    fmt = r'\1,\2,\3'
    extractions = []
    placeholder_rgns = cmd.view.find_all(pat, flags, fmt, extractions)

    # Here `placeholder_rgns` is NOT a `sublime.Selection` object!

    adjust = 0
    for i, rgn in enumerate(placeholder_rgns):
        adjusted_rgn= sublime.Region(rgn.a - adjust, rgn.b - adjust)
        # Here, any self.view.replace() operations have to keep track of the
        # shift in the buffer contents going forward.  This is because
        # `placeholder_rgns` IS NOT a `sublime.Selections` object.
        ...compute `new_text` here...
        self.view.replace(edit, adjusted_rgn, new_text)
        adjust += (adjusted_rgn.size() - len(new_text))

Case 2:

However, when you are iterating on an actual sublime.Selection object, every loop iteration (apparently since build 4023) makes a fresh call into the sublime_api to fetch the indexed "selection" (which Sublime Text has been shifting with the buffer contents on the fly, as self.view.replace() operations happen), and so, like when you have multiple carets present in a View, keystrokes of printable characters introduce changes into the buffer, but Sublime Text is also shifting the "live" carets forward in the buffer on the fly.

The same thing happens if you iterate on a sublime.Selection object like this:

        sel_list = self.view.sel()   # Unlike the above, this is a `sublime.Selection` object.

        # Replace each selection with edited string.
        for rgn in sel_list:
            selected_text = self.view.substr(rgn)
            new_text = '``' + selected_text + '``'
            self.view.replace(edit, rgn, new_text)

works as intended because of the __iter__(self) method in the sublime.Selection class fetches the CURRENT version of the indexed selection, so any previous self.view.replace() operations will have shifted the selections inside Sublime Text (i.e. on the other side of the sublime_api object).

You can see how the "current version" of each indexed selection is returned for each iteration of the for rgn in sel_list loop here (from the class Selection class in python38/sublime.py file):

    def __iter__(self) -> Iterator[Region]:
        """
        Iterate through all the regions in the selection.

        .. since:: 4023 3.8
        """
        i = 0
        n = len(self)
        while i < n:
            yield sublime_api.view_selection_get(self.view_id, i)
            i += 1

It is better to have "reversed()" in the iterator.

[jrappen]

@vwheeler63 I cleaned up our discussion a bit, as it makes it easier for me to read while going through TODOs here. I hope you don't mind.

[jrappen]

@vwheeler63 Do you know of a page that lists Sphinx's additions to reST?

[vwheeler63]

@vwheeler63 I cleaned up our discussion a bit, as it makes it easier for me to read while going through TODOs here. I hope you don't mind.

Not at all.

@vwheeler63 Do you know of a page that lists Sphinx's additions to reST?

It's not in one place, but what the Sphinx doc guys did is they let the Docutils documentation handle the basics, and the Sphinx docs (e.g. Directives, Interpreted Text Roles, etc.) only documents things that Sphinx added (or significantly changed), which IMO, VERY much helps the reader understand what Sphinx contributed to the reST world, as well as where to look for the documentation, because in literally 99% of the cases, Sphinx didn't change any of the Docutils basics, but only ADDED to the functionality with additional directives. (Sphinx harnesses the Docutils engine internally, which was brilliantly designed for extension.)