[RFC] Scopes for module/namespace access

[deathaxe]

Intro

In general a trend/principle in syntax definitions can be found which ends up with scoping the declaration/definition of constructs with entity.name.<construct>. When calling/using such constructs, something like variable.<construct> or variable.other.<construct> is used.

The most popular example is entity.name.function vs. variable.function.

The goal is clear - distinguish definition and usage of the same object.

Question

How are namespaces or modules to handle in that manner?

A couple of syntaxes including C, C++, Python, PHP, Java, JavaScript, Erlang, ... support such concepts. Most of them use entity.name.namespace to scope the identifier in the definition/declaration statement as the ST3 documentation at https://www.sublimetext.com/docs/3/scope_naming.html says:

Namespaces, packages and modules use the following scope. There are usually not multiple types of such constructs in a language, so this scope should suffice.

entity.name.namespace

But I can't find a common solution how to scope a namespace/module upon usage.

    module_name:function_name()

• C# distinguishes between support.namespace.cs and variable.other.namespace.cs
• CSS uses entity.other.namespace-prefix.css or entity.name.namespace.wildcard.css
• Erlang uses entity.name.type.class.module.erlang
• Python just scopes them as meta.generic-name upon usage or meta.import-name in import statements.
• ...

Can we find a common scope for that usage?

1. From my point of view anything starting with entity. is a no-go when we talk about usage.

2. The most pleasant approach with respect of existing scoping guidelines and implementations seems to be variable.other.namespace. So we'd end up in

   entity.name.namespace vs. variable.other.namespace

3. To keep up with the concept of function declaration and usage, I also could imagine to use variable.namespace. So we'd end up in

   entity.name.namespace vs. variable.namespace

Thoughts?

[keith-hall]

I completely agree with point 1.
for namespace usage, I think its worth having the distinction between namespaces provided in the standard library and those that are user defined (i.e. like the C# syntax definition does atm), as I like to color them differently.

[FichteFoll]

For method or static function calls on objects/classes, do we treat those as namespaces as well? There is no way to differentiate those from modules in most languages.

[deathaxe]

I did not directly think of static class member access, but this is a very good question though.

Basically a class is an extended concept of a namespace, which allows to create copies of the internal variables by instantiating. Because of that access semantics are equal in most languages and can't indeed be differentiated.

Furthermore everything most languages call package, module or class is most likely nothing else than a namespace.

I personally like the idea of using namespace to generally identify all kinds of such constructs in an abstract way, which can be applied to several syntaxes.

In general it might be a question of the languages concept of whether we use namespace or class.

Examples:

1. C uses namespaces only, C++ knows both.
2. A Perl source file can form a package, which can be instantiated like a C class.
3. A Python source file (module) is nothing else than a namespace, while it can contain classes.
4. Erlang uses modules which are basically namespaces only.

I just tend to suggest to limit scope names to namespace and class instead of using unique ones for each language like package or module, etc.

The question keeps open what to use, if no decision can be made.

Scope names would look like:

- meta.namespace / entity.name.namespace / variable.other.namespace
- meta.class / entity.name.class / variable.other.class

[FichteFoll]

Another question: when importing a module that is sourced from a file with the same name (and not explicitly specified), would the import statement be a usage or a definition of the namespace?

[deathaxe]

I think this depends on the perspective.

Importing a module means to declare it in order to be able to use it and its members. It can be compared to declare/define a variable or a function.

From the global point of view, an import is a usage of an existing module.

I tend to prefer the local perspective.

See python:

import os    # os <- entity.name.namespace

# os <- variable.other.namespace
os.getcwd()

The usage of os in os.getcwd() does not work without declaration of import os.

The interesting question here is - how about

from os import getcwd  # <- os = usage or definition?

getcwd()

or

from os import path   # <- os = usage, path = definition?

path.join()

[wbond]

The topic of scoping qualifiers is not well addressed currently, so I'd like to come up with something to move forward. Currently run into this issue in lots of places (like #737, where I was just working).

I think part of the issue is that sometimes we know looking at the code that a qualifier is a class name, or a namespace, etc. However, it isn't always clear when highlighting which we are currently dealing with, and for some it is impossible.

Because of this, and the fact that we need have many different syntaxes with different nuances, I think we need to come up with a somewhat generic scope that can be applied to the identifier qualifiers in a "path". This could be for a function call, a type name, an inherited class name, an XML namespace.

Previously I think we had thrown around the idea of a new top-level scope, such as identifier (does that ring a bell @FichteFoll?). Either way, I'm not sold on that idea, but I was poking around at existing syntaxes and was thinking about using the following:

entity.other.qualifier

Currently entity.other is used for inherited-class and attribute-name primarily. This would somewhat play off of the entity.other.inherited-class scope, since this is a place that qualifiers are sometimes seen.

I would imagine that most users wouldn't want these too heavily colored. Additionally, a series of entity.other.qualifier and variable.* should be scoped with meta.path.

Thoughts?

[wbond]

I think the most obvious alternative to entity.other.qualifier, or (entity.qualifier as @FichteFoll suggested on Discord) is:

variable.qualifier

This keeps most such identifiers in syntaxes under variable, for better or worse, and leaves entity.other as sort of a historical relic for inherited class names and HTML tags and attributes.

[Thom1729]

It sounds like we're talking about the following four types of constructs:

1. Definitions of namespaces (e.g. namespace foo {} in C++).

Here, the whole construct should get a meta scope, namespace should be storage, and foo should get entity.name.

2. Explicit imports (e.g. import os in Python).

This deserves more examples, because import syntax varies greatly between languages. I think that we can come up with an answer generic enough for general use.

Taking the broad view, import os is a declaration that declares the name os. It would be reasonable to scope os with entity.name. The statement from os import getcwd declares the name getcwd, which we could also scope with entity.name, but it does not declare os, so we should not scope it with entity.name. In both cases, os serves a special syntactic purpose: it's the name of a module, and it doesn't behave like an ordinary identifier. I'm not sure what the right scope would be for that -- let's call it FOO for now. Then, the scopes should be as follows:

import os
       ^^ FOO entity.name.something

from os import getcwd
     ^^ FOO
               ^^^^^^ entity.name.something

3. References to a namespace (e.g. os.getcwd() in Python).

My biggest concern is with (3). In many languages, namespaces or modules are first-class values and a name representing a namespace is used in the same way as any other name. For instance, in Python, os.getcwd() is an ordinary expression where os is an ordinary variable no different from the foo in foo.bar().

In such languages, the only way to try to highlight references to a namespace is to guess based on the name of the variable. This is a bad idea, because the guess would often be wrong, and users get annoyed when some identifiers are colored differently seemingly at random.

In some languages, namespace references are used with special syntax. For instance, in C++, a namespace reference may be followed by the scope resolution operator ::. We could scope foo specially in foo ::, just like in foo() we may scope foo as variable.function instead of variable.other. The key here is that the difference is syntactic. In Python, we can't say that foo in foo() is actually a function, or callable (it may be a runtime error), but there is nevertheless a purely syntactic justification for highlighting it as a function.

We should keep in mind that we can't scope these things reliably. In foo\n(), we cannot recognize foo as a function.

4. General dotted paths.

Most of the time in most languages, a dotted path like foo.bar.baz represents a value foo and a sequence of property accesses. The semantics will vary. We don't have a standard scope for foo and bar, and we should (maybe something with "property" or "attribute" in it). On the other hand, in most common languages foo isn't really distinguishable from a baseline variable. In some languages, we can say that it's an "object" of some kind, but in many languages everything's an object anyway.

We can try to scope the entire path, but to be honest I've never liked doing this. It's fundamentally unreliable and annoying to implement, and even in real-world use a typical file would likely see a lot of misses. Plus, I don't really see the motivation: why have a special meta scope for foo.bar but not foo + bar? We're not going to highlight it differently, and we can't guess reliably enough for automated tools to use it, so why do it at all?

---

Other thoughts:

• Regexps with a lot of captures or lookaheads are bad for performance. The fastest way for a sublime-syntax to work is one token at a time. Highlighting a token based on what comes after it is often impossible and always annoying.
• We could do meta.path reliably with nondeterministic parsing. However, leaning on nondeterminism for such a ubiquitous construct might be bad for performance, because the performance penalty for nondeterminism is proportional to the cube of the depth of nondeterminism.
• I don't think that a class is really a kind of namespace, at least not to the degree that they should use the same scopes.

[deathaxe]

I don't really see the motivation: why have a special meta scope for foo.bar but not foo + bar?

The initial post doesn't say something about metas.

That said, the question just is: Which scope to use for the declaration of a namespace vs. the usage of it?

The scoping guideline says: meta.namespace entity.name.namespace to be used for all kinds of namespace, module, package definitions, but there is no general rule about how to scope the usage of a namespace (assuming it can be identified). The initial post lists some examples being used in different syntaxes. Comparing to function definition and calling, scoping namespace access identifiers with entity. feels odd.

With regards to variable.function in function-calls, the variable.namespace sounds reasonable. If a more general approach is desired as we may not be able to distinguish namespaces from classes or anything like that, variable.qualifier sounds good.

We should keep in mind that we can't scope these things reliably.

This issue was not raised to propose adding much guess work to syntaxes. It just tries to find answers for situations / languages, which allow to identifiy namespaces as different scopes for same things were found in already existing syntax definitions.

If something can't be identified reliably, an as general as possible scope should be applied.

[FichteFoll]

This topic is huge. I apologize for a wall of text, but this touches a very fundamental concept of how definitions and references are applied (so it very much relates to #1861).

It seems the question is three-way.

1. How to scope namespace definitions (namespace foo in C++)?
2. How to scope an import/global usage of a namespace (import foo, from foo import bar, from foo import bar as baz)?
3. How to scope a local usage of a namespace, which quickly becomes equivalent to asking how to scope a qualifier?

Usually it cannot be decided for 3. whether a namespace, a class, some random object or whatever is being accessed, so I'll hold off on that for now.

For 1., I believe entity.name.namespace is the correct scope for the final identifier, assuming a construct like namespace abc::def is allowed. meta.namespace should span the entire definition of the namespace including its body.

For 2., I second @deathaxe's opinion in that an import that assigns something to an identifier becomes a declaration and should then be scoped as entity.name.import (suggestion). I believe that to be a general enough scope to be usable for all various types you can import.

However, not all imports also assign to a specific identifier. Some imports work on a literal basis and behave as if the referenced file was imported verbatim into the current file (barring preprocessor checks to prevent re-importing), e.g. #include <string>. The other form are wildcard imports like from asyncio import *. Those imports are usages of or references to namespaces (or files, which I guess can behave like namespaces for our purpose) and should thus follow general qualifier scoping rules, which don't exist yet (see 3. below). However, seeing as we can be quite sure to have at least one identifier than can definitely assumed to be a namespace, we might want to use variable.namespace for that usage. Once we decided on scope names for qualifiers (see below).

---

Now for the important part: How do we scope qualifiers and usages of variables or identifiers in general?

First, I believe we should clarify on terminology.

• An identifier is a unique name made of words or characters that are atomic, i.e. changing any one character of it is impossible, and do not represent a keyword or a (language) constant, excluding case-insensitive languages. They can be used to reference a (virtual) storage of data or a (virtual) collection of structured data for storage.
• A qualifier is a path built from one or multiple identifiers that can traverse a hierarchy of (virtually) structured data. Every identifier is also a qualifier on its own. Multiple identifiers that describe the path to a certain (virtual) data are joined using an accessor that separates identifiers syntactically. This may even be a whitespace character.

Unless I am mistaken, we can use these concepts to represent most, if not all, currently used patterns in programming languages.

---

Going forward, I conclude that we have to answer the following questions:

1. Should we scope every identifier?
2. Should we scope every qualifier? Note that every identifier is a qualifier.
3. Which kind of scope should we use for identifiers/qualifiers?
   We can choose between a colored (variable), uncolored (meta) or new (identifier/qualifier) scope (on the first level).
   Should we introduce a new main
4. Is "item access" in the form of indexing of a sequence or through key access of a mapping also part of the qualifier (and should thus be scoped as such)?
5. What about function calls?
6. (How much effort will this all be and can we ever hope to get all default syntaxes to follow it?)

---

I'll proceed with answering these question by myself, but I'm interested in your opinions.

1. Yes. This question has two parts.

Currently, the variable scope has mostly been used for identifiers at various locations in a qualifier or depending on its semantics, like whether it represents a type/class or a function. However, its primary use is for the final identifier. Interestingly, support is equivalent in this regard except that identifiers scoped as support aren't user-defined and exist by default on behalf of the language's environment.
I believe it is in our best interest to continue this tradition and not break backwards compatibility basically everywhere if we were to apply a different top-level scope to identifier references.

Talking about declarations or definitions of identifiers, i.e. where they are usually used without the presence of qualifiers, is handled by entity.name and we should keep that.
Exceptions: Almost all entity.other scopes like entity.other.inherited-class (used as a reference to a class to be inherited). Imo we should adjust these to new guidelines eventually.

The second part of the question considers identifiers that are not the last segment of a path and where we usually don't know much. Encountering them at any place in a language like Python could mean anything from a namespace (module/class?), a type (class), a function or any other object data reference. However, we may be able to guess at what the identifier references based on naming conventions for constants or (built-in) types.

To conclude, yes, we should scope each identifier to the best of our abilities. If we cannot tell what an identifier represents (at its position in a qualifier), choose a generic scope. To make things easier, all identifiers should get this generic scope and only for those whose meaning we can decipher feasibly we add another scope.

2. Depends. I don't think this hurts us in any way and we may be able to add sub-level scopes when we know what kind of qualifier we expect at this place. I don't really see a particular use case for color schemes currently, but the Expand Selection to Scope command could easily make use of this.

3. Qualifiers should definitely not get a colored scope. meta.qualifier seems fine to me. We don't need a new top-level scope for non-atomic syntax elements.

For identifiers we currently have variable and support for those we can interpret. I don't think adding another top-level scope for basically the same purpose is worth it, so we should keep backwards compatibility on this. I don't think joining these two would be worth it either.

However, the scope for where we "don't know" is crutial. JavaScript currently scopes these as variable.other.readwrite; Python has meta.generic-name. For our candidates, I conclude meta.identifier and variable.other. The latter clearly indicates that the semantics are unknown and no third level is needed for this assessment, while the former is more subtle and could be stacked with variable and support.

I think choosing variable.other is optimal because strictly speaking meta.identifier would need to be added to every entity.name as well and we don't really gain anything from that. Instead, all references of any type (function, class, namespace) may be found simply through a selector on variable, support (in case the user overrides built-ins).
Where known, the type of the referenced data should be added as the second level, i.e. variable.function, variable.namespace or variable.constant. Whether or not types should be scoped as variable.type or instead remain as storage I'll leave to another time and RFC.

4. No. As mentioned in 2., we barely benefit from scoping the qualifier and trying to include item-access or more is a lot of effort for seemingly no gain. Besides, at least according to my initial definition, they aren't even a part of a quantifier.

5. same as 4.

6. I believe the work required is manageable.

For Python, most of this is already finished and I just need to exchange scope names. For other languages, you'll most likely end up with a similar context layout where you need to scope path segments and accessors individually anyway.

The largest effort, as always, falls down to establishing the updated scoping guidelines and color schemes together with user perception, although we hardly have any backwards-incompatible changes in here.
The biggest pain points will probably come from adding variable to identifiers that didn't have them before and changing variable.parameter to entity.name.parameter (not talked about in here).

---

I hope you've been following me until this point, although I hope I structured it decently enough despite writing on it for an hour or so.

Did I miss a certain aspect? Does a language you know not fall into the raster/structure I imagined? Do you think non-first-level identifiers in a qualifier should get a variable.member instead of variable.other?

[Thom1729]

Your thoughts parallel my own. In particular, I agree that we have to address the general problem in a general way.

I do think that variable.other is the correct scope for a “generic” identifier. However, I think that a property name (bar in foo.bar) is a different kind of identifier deserving a different scope: variable.member, or variable.property, or something. In my opinion, it's the first identifier that's different from the others, not the last.

In some cases, it may make sense to further specialize these scopes in a purely additive fashion and on a best-effort basis. For example:

foo
# <- variable.other
foo()
# <- variable.other.function
foo.bar
#   ^ variable.member
foo.bar()
#   ^ variable.member.function

All that said, I'm not convinced that it's worthwhile to scope “paths” in the general case. An example in Python:

class Foo(collections.abc.Sequence):
    ...

There's an argument for scoping the path collections.abc.Sequence (say, meta.superclass). But I think that this argument is slightly misplaced — collections.abc.Sequence is just an expression. We would want to scope it the same way even if it were some other kind of expression — not because of its “internal” nature as a qualified path, but because of its “external” nature as the superclass part of a class declaration. (Granted, in a lot of languages, the only expression allowed there would be a qualified path, but we have to design the scoping guidelines for the general case.)

In this example, it might make sense to use a special additional scope for Sequence. (Currently we use entity.name.inherited-class.) But we should still have the regular variable.member scope.