Meeting

[m1kit]

I'll take the minutes here.

[m1kit]

July 7th Meeting

Agenda

• Part Ⅰ: Progress Report
  • JSON template format
  • Pre-generated resources
  • Structured Normalizer
  • Draft API
• Part Ⅱ: Discussion
  • Review of previous changes
  • Should we transform regex?
  • API design

Discussion

Should we transform regex?

I want to assume input templates are already "normalized".
This is because of difficulties of regex transformation.

• whilst should be normalized to while, but \whilst should be kept as is
• S should be normalized to s while \S should be kept
• a-z should be normalized to a- z while [a-z] should be kept
• {2} should be normalized to

[goneall]

@m1kit A couple of follow-ups on the meeting.

I did some research on the Python regex and found a couple open source implementations using sre_parse that may help solve the regex transformation issue discussed.

sre_dump.py (Released into the public domain) "pretty-prints" the regular expression. You may be able to take a similar approach, even borrow some of the code with attribution, and transform the regular expression. Some of this may be easy (changing literal spaces in to back ticks) while other transformations may be more challenging (handling character classes that may contain spaces or the \s.

I also found a project sre_yield which may not be as applicable but has a lot of sre parsing code which can provide some examples.

[m1kit]

Thank you, @goneall.
I agree, regex normalization is possible with sre_parse and sre_dump.

One concern is, sre_parse is an undocumented internal module. Maybe the implementation is not provided in some environments.
Also, sre_parse does not provide a way to construct regex from a parsed tree. I found sre_dump can do that, but it is quite old (not updated since 2003, still written in Python2). Using these dependencies can impact portability or maintainability of our library, so we should carefully examine.

[goneall]

@m1kit I agree with both of your concerns. If the normalization is done while generating the pre-compiled regular expressions, it would be done in a more controlled environment. That being said, using undocumented internal modules is generally considered a risky approach. It would be nice to find a more supportable alternative. As far as using sre_dump, we would probably have to update and maintain a version of this library. It is nice, however, that we have a starting point.

In the Java implementation, the regex is applied un-normalized against the original text starting at a position matched by the previous matching text. It is very complicated and somewhat incompatible with the current regex approach, however. Perhaps breaking up the regex into multiple regexes - normalized and un-normalized and being able to map the start and end for the different matches. I'll think about this some more and see if I can come up with any simpler suggestions.

[m1kit]

To conclude, we have three approaches in mind:

• Replacing patterns in regex, without considering its semantics
  • This can break some regexes, but we may assume there is no such corner case
• sre_parse
  • I think this is an ideal approach, but we need to do some additional work to maintain sre_dump
• breaking up the regex into multiple regexes
  • It's possible, but the algorithm would be complicated...

I want to try sre_parse over the next few days, and if it fail, I want to try another approach.

[goneall]

I want to try sre_parse over the next few days, and if it fail, I want to try another approach.

Sounds like a good plan. Let me know how it goes.

[goneall]

@m1kit @anshuldutt21 it turns out 28th for our next meeting probably won't work for me - I'll likely still be in an area with limited phone reception.

I created a doodle survey for some alternative times after I'm back from my travels:

https://doodle.com/meeting/participate/id/mepRZ2bM