Add ISBL lexer

[MedvedTMN]

Add new lexer:

• ISBL built-in language DIRECTUM
  Add New theme:
• ISBL editor light

[MedvedTMN]

I did not have errors for Ruby 2.5.1. What to do?

[dblessing]

It may have been a transient failure. I restarted the job to see what happens.

[MedvedTMN]

Problem still exists

[dblessing]

This may have been related to #892. Rebase against master and see if that fixes things.

[pyrmont]

@MedvedTMN Thanks for submitting the PR and I'm sorry it's taken so long to get this addressed. I joined the maintainers group a couple of months ago and have been working backwards chronologically through the outstanding PRs which is why I hadn't got to yours yet :(

I realise that @vidarh in his review had identified this as having only minor issues but I'm afraid that's not my conclusion. There are a couple of problems:

1. The architecture of this lexer results in a large number of String objects being added to Rouge. At this stage Rouge works by loading all lexers into memory. Running rake check:memory on master and on this branch shows that this lexer would lead to an increase in allocated memory of 1MB and to retained memory of 400KB. That's a significant amount compared to other lexers. Looking at the lexer, the vast majority of strings are constants. Does ISBL differentiate between constants and non-constants by capital letters? If it did, you could get rid of all these constants by merely highlighting any capitalised word as a constant.

2. If including all of these strings is inescapable, we would be looking to do two things. First, the strings should be put behind memoised class methods. You currently store these all as local variables which require allocation at the time the class is read. Using class methods can delay that allocation. A memoised class method looks like the following:

   def self.some_identifier
     @some_identifier ||= %w(...)
   end

   Second, these strings can be stored in a separate file and then brought in by the memoised method when needed. You can see how this would work by looking at the Gherkin lexer:

   rouge/lib/rouge/lexers/gherkin.rb

   Lines 20 to 24 in 7836be9

   	# self-modifying method that loads the keywords file
   	def self.keywords
   	load File.join(__dir__, 'gherkin/keywords.rb')
   	keywords
   	end

   The file that is loaded by the Gherkin lexer is generated by a Rake task. You would need to create a similar task as part of the PR submission for ISBL.

3. The lexer is missing the frozen_string_literal pragma we use for all files in Rouge.

4. The lexer is called ISBL but the description suggests that the name of the language is DIRECTUM. Which name is the name of the language? If it is ISBL, the name of the lexer should be ISBL rather than Isbl.

5. The lexer contains inconsistent usage of whitespace for indenting. Rouge follows common Ruby practices and indents with two spaces. Please update the lexer accordingly.

6. Since the lexer was originally submitted, we have made RuboCop checks part of the test suite (you can run this at the command line by typing rake). To avoid RuboCop's ambiguous regular expression warnings, the parameter passed to rule should be prefaced with %r and a delimiter. The simplest delimiter is / such that rule /.../ becomes rule %r/.../.

7. We generally prefer using \b rather than lookaheads in patterns. I realise that ISBL, due to its provenance, may be used by users who will write identifiers with Cyrillic characters. Does this mean you can't use \b?

8. The PR includes two themes. I would suggest these be removed. I apologise for sounding a bit harsh but these appear to be primarily (if not only) of interest to the small subset of users that would want to highlight ISBL. Rouge's suggested approach for those situations is for users to include their own theming files separately in their projects.

Sorry you've had to wait so long only to then be hit with such a long list of suggested changes. I'll try to be as responsive as I can be to any follow-up questions you have.

[pyrmont]

@MedvedTMN Do you want it close this PR? It looks like perhaps you're planning to create your own fork?

[MedvedTMN]

No, I do not want to close this PR. I just wanted to update the local project files to correct your comments. Am I doing something wrong?

[ashmaroli]

Looks like @MedvedTMN changed the line-endings on all files and committed them..

[ashmaroli]

@MedvedTMN May I know what OS you're working on..? It looks like you altered the line endings of numerous files.

[MedvedTMN]

@MedvedTMN May I know what OS you're working on..? It looks like you altered the line endings of numerous files.

Windows 10

[MedvedTMN]

How can I fix it?

[ashmaroli]

@MedvedTMN Fixing this is perhaps too much for you (I'm assuming that you're not an advanced Git user). But you may want to read through the following documentation for future contributions:
https://help.github.com/en/articles/configuring-git-to-handle-line-endings

I'll try to fix this for you via a PR to your branch some time later..

[MedvedTMN]

I'll try to fix this for you via a PR to your branch some time later..

Thank you very much for your help

[pyrmont]

@MedvedTMN Thanks for fixing the trailing whitespace issue; however, you've added back in all the extraneous stuff in the visual sample you removed earlier. Can you delete that stuff again?

[MedvedTMN]

The current example most fully demonstrates the capabilities of the language. If the example is too large, I can shorten it. Or do you need to remove any specific extraneous stuff?

[pyrmont]

@MedvedTMN The code added back by this commit is the extraneous stuff I'm referring to.

[MedvedTMN]

What is wrong with this code?

[pyrmont]

@MedvedTMN Did you look at the link? All the code you added back in isn't necessary, is it? The point of the visual sample is to test the rules in the lexer. I don't see why a lexer that's 97 lines of code total needs a visual sample that's almost 300 lines.

[pyrmont]

@MedvedTMN To pick one thing as illustrative, you have multiple instances of multiline comments. One or two would suffice. I'm not familiar with this language so it's hard for me to identify definitively what's duplicative but it looks to me like there are many instances like this. The visual sample appears to be a copy and paste from existing code. A more minimal sample that better targeted the rules of the lexer would make this easier to maintain.

[MedvedTMN]

it's all right?

[pyrmont]

@MedvedTMN Is this correct syntax for ISBL? Looking quickly at the rules, it looked to me like .custom_type should be lexed as a type but that wasn't what happened. Could you have a look?

[MedvedTMN]

No, this is the wrong type definition for the ISBL variable. The following definition will be considered correct:
VariableName: ObjectType|IReference[.ReferenceType]|IEDocument[.CardType][ = Expression]
This is how this code looks in the built-in code editor:

[pyrmont]

Cool. Can you update the visual sample so that it's correct?

[MedvedTMN]

edit this part:
// Types
type : IApplication
type : .custom_type

or this:
Employees : IReference.РАБ = CreateReference(EMPLOYEES_REFERENCE; ArrayOf("Пользователь"; SYSREQ_STATE); FALSE)

[MedvedTMN]

Done

[pyrmont]

Thanks!

[pyrmont]

@MedvedTMN I realise this took a long time but we finally got there in the end :) This will be part of the next release of Rouge. The gem is scheduled to be pushed on Tuesday 14 April 🎉

[jneen]

@MedvedTMN Apologies for the late post, but where did you get the lists of ISBL keywords/builtins you're using here? I'd like to transition to auto-generating them from docs.

[MedvedTMN]

@MedvedTMN Apologies for the late post, but where did you get the lists of ISBL keywords/builtins you're using here? I'd like to transition to auto-generating them from docs.

Documentation is available in the built-in help system for Directum 5. I extracted it from the system database tables at one point.

[jneen]

Is there a publicly available and parseable download for this? Say, from a github repository? I'm afraid I don't read Russian.

[MedvedTMN]

Is there a publicly available and parseable download for this? Say, from a github repository? I'm afraid I don't read Russian.

Unfortunately, there is no information in the public domain, as this is a commercial product.

[jneen]

I wonder then whether we should maintain this list of keywords at all, if there is no publicly available source for it.

[MedvedTMN]

I wonder then whether we should maintain this list of keywords at all, if there is no publicly available source for it.

Projects from this system are managed in GitLab, which uses this library for syntax highlighting. This highlighting is very helpful during code reviews, so I think it's best to keep it, at least in its current form. But the decision is ultimately yours.

[jneen]

I am happy to keep it, if we can find a source for a publicly-accessible list of builtins that doesn't have to be maintained manually. For now though I think it is best to remove, as in #2236.

If you have a self-hosted GitLab, you are welcome to write your own plug-in and maintain the builtin lists yourself.