GSoC 2021 Generic Discussion

[m1kit]

Welcome to a discussion page for my GSoC 2021 project

• GSoC Official Site
• Original Proposal (it is possible that the plan has changed since the original proposal)
• Discussion Portal (here🙌)
• Task Kanban

To the mentors: I would appreciate it if you could comment on issues with the awaiting review tag 🙏

[goneall]

Proposed change to Original Proposal: Add a step in the plan between 3 and 4 of using this new library in the SPDX online tools replacing the current Java call to license matching. This could be an optional goal.

[anshuldutt21]

Hey @m1kit. It would also be better to schedule some time in your proposal for the documentation and code refactoring. For this project,, a lot of testing work is required which you will need to do occassionally to ensure all functionalities work.

[m1kit]

I plan to do such work once around the First Evaluation👍

[m1kit]

I think the repository name may contain the language name, but I don't think the package name should contain the language name. This is because this project is intended to be ported to other languages. For example, if a python version is released to PyPI as python-license-matcher, and a java version is completed, what name should it be released to maven? I think it would be better for branding to have the same name for all languages.

[goneall]

Agree with repository name containing the language and the package name not containing the language.

Seems like this solves both issues.

Of the list above, I like yalm - googling it there aren't many conflicts. lycense may conflict with a trademarked term used for a pest control service (?). I also like having short names that are unique in google - easier to find and remember.

Perhaps yalm as the package name and yalm-python as the repo name?

Just my thoughts - I'm flexible :)

[m1kit]

I'll start coding with the name yalm. If you come up with a better idea, please contact me and I can change it.

[anshuldutt21]

We already have repositories by the name of license-matcher in spdx. Since there are also many other license matchers implemented in java and other languages, omiting python wouldnt be such a good idea imo. Maybe something like python-license-matcher or PyLicenseMatcher would work. Gary what do you think about it.

…

On Thu, 3 Jun 2021 at 11:09 PM, mikit ***@***.***> wrote: These are just a few ideas I came up with: - spdx_python_licensematching - spdxlassifier - analicense - licensifier - pyslim (stands for PYthon Spdx LIcense Matcher) already taken - yalm (stands for Yet Another License Matcher) - lipymatch - lycense Do you find anything looks good to you? We can change the name later if before uploading to PyPI, so I think it's okay to start coding before we decide the name. Key nouns - spdx - license - python - py - pattern - template Key verbs - match - identify - detect - classfy - analyze - parse - recognize - see — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKZ4QLUPOB6GESLRPVS3U5TTQ6455ANCNFSM45K4S3MA> .

[m1kit]

Coding period begins!

Hi @goneall , @anshuldutt21 !
The coding period started today. I've been making some preparations for the project so far, but today I'm going to start working on it in earnest.

Our current goal is to release the alpha version of the library to PyPI(!) by the mid-term evaluation on mid-July.
Based on my proposal, I set four milestones, one for each two-week period.
I have also added some more issues that should be completed by the mid-term evaluation.
If you have any comments about the plan or the work, please feel free to let me know.

[m1kit]

I had a lot of exhausting things going on in my real life until last week, so I took some time off. Now I have some time to work, so I'll report back by the time you get back.

[goneall]

For some reason, I'm not getting notified on the updates to the yalm-resources project. Could you try doing pull requests then merging the changes when you make updates? That will give me a change to review.

[m1kit]

@goneall I created a draft PR for pre-gerenated template feature(this includes a big refacotring and API changes), can you subscribe it?

Also, if you have some free time next week, can you set-up an online meeting for about 1hr? I think we have some discussion topics, and it should have done before alpha library release.

[goneall]

@m1kit I was notified of the draft PR - so the subscription is working :)

It would be great to meet next week.

I setup up a Doodle poll with some time suggestions: https://doodle.com/poll/n5qzfmt32xx8tk8p?utm_source=poll&utm_medium=link

I'll setup a zoom call once we decide a time.

@anshuldutt21 would you be able to join us as well?

[anshuldutt21]

Sure @goneall.

[m1kit]

Hi @goneall, @anshuldutt21.

First evaluation has been completed. I am happy to be able to continue this project. And thank you for your feedback.
Since the summer vacation hasn't started yet in Japan, I'm not working every day due to classes, etc., so it's difficult to report every day, but I'll try to report every two days.

As for my recent progress, I took a break from yalm for a few days to work on a library called terregex. This is a library for transforming regular expressions based on their structure, using sre_parse and sre_dump internally. This library currently has only the minimum functionality that yalm would need, but it is already available on PyPI.
https://pypi.org/project/terregex/

And the regular expression in the yalm-resources repository has now been changed to normalize the contents of var tags as well. There is still a lot of room for improvement at this part, but I believe that the introduction of structure-based normalization is a big step forward.
https://github.com/m1kit/yalm-resources/blob/main/src/spdx_xml/normalizer.py#L52-L54

Originally, we had planned to have an alpha release by this time, but due to the regular expression issue, I'd like to spend this week on testing and writing the documentation.

[m1kit]

As a sidenote, I'd like to introduce a strange issue I'm facing.

The MIT license template contains a regex like below:

this software and associated documentation files|this source file

By using terregex, we get this:

(?:this\`software\`and\`associated\`documentation\`files|urce\`file)

The last part of regex is bogus(urce file).
This is caused by a combination of an unintentional optimization by sre_parse and a bug in sre_dump.

Given the regex, sre_parse "optimizes" the pattern like this:

this so(ftware and associated documentation files|urce file)

And sre_dump does not emit parenthesis around branching | by a bug.
This is how the output is produced.

I did convert sre_dump with 2to3 to update it to Python3 syntax, but apart from that, I need to do some maintainance work on the library🙃

[goneall]

@m1kit Thanks for the update. I really like the separate Terregex module approach - very clean and can be re-used by others needing similar functionality.

I agree with your change in priorities focusing on the Regex issues. It looks like sre_dump does need a bit of maintenance. Too bad there isn't an upstream github project we could contribute to.

One suggestion - since you are making modifications to sre_dump.py, you could add a comment at the beginning pointing to the original url and adding a notice summarizing the changes that your changes are release under the same license as sre_dump or under Apache-2.0 - whichever you prefer.

[goneall]

BTW - updates every 2 days is fine - main thing is to know what to expect in terms of updates.

[m1kit]

@goneall
Hi Gary, this is a progress report for the past 3 days.

I released terregex 0.0.5 - includes a bugfix for the issue above.

Also, I'm testing the functionality of yalm-python's detect_license feature with two major OSS licenses - MIT and Apache 2.0.
Unfortunately, both of them are unsucessful for different reasons.

For MIT, there is a issue on the space handling again on an optional tag. Here we can see there are spaces before/after the tag. I normalized spaces at the beginning and the end of the tag inside, but not outside. I need to modify the normalizer to normalize these "outside" spaces.

For Apache, there is a issue related to my resource loading system. Currently, I'm loading resources via importlib's resource feature. This treats resources as Python modules. Since Apache license's resources are located under a directory Apache-2.0, the library complains the name is invalid (recall the . has a special meaning in python package path). I'm gonna relocate license-specific resources under directories with no special characters (e.g. Apache_2_0).

I except these two fixes won't take much time.

[goneall]

@m1kit Thanks for the update. Good job on the terregex - normalizing the regexes is a rather complex problem to solve.

I've found Apache to use may of the XML tags, so once you get that working I would expected many of the licenses to pass. However, I would expect some further iterations with other licenses since there is a wide variety of alt and optional usage.

[m1kit]

Now I fixed the folder name issue (all names are escaped like this).

Next I try to deal with spaces issue by modifying the normalizer.

[m1kit]

Sorry for the late update, due to an unexpected research task, I was not able to work on this for the last two days. Now I have some time, working on the spacing issues above.

[goneall]

@m1kit - Can you provide an update on testing? I would replace the Java license compare with yalm in the SPDX online tools. I would like to successfully test yalm against the current set of SPDX licenses and their test files first. Let me know if you recommend any further testing before making that change.

[m1kit]

I shared an update in a PR #22 , please refer to it

[m1kit]

Final Evaluation!

Hi @goneall , @anshuldutt21,

The final evaluation period of the GSoC program has begun and we have one week to submit our report.
Note that the evaluation is just a checkpoint and it does not affect much on development of yalm - as I said my goal is to deliver this library to the world.

However of course, I do not want to fail the evaluation.
This week I'm working on the documentation with sphinx, and submit PyPI project page as work product for the evaluation.
(Recently I'm struggling at reStructuredText syntax, maybe I'll prepare GitHub Wiki pages as a backup option)

If there is anything else I should do to prepare for the final evaluation, please let me know.

Also, if you have any doubts prior to the final evaluation, we can also set up a meeting.

[goneall]

@m1kit I was also thinking it was time to switch focus to documenting the work and preparing the final submission. The main criteria is that it meets Google's standards.

In addition, I would like to include an update to this project on the SPDX github site. We could merge it back to the original project and rename it to YALM, create a new project, or clone a copy from your site. Let me know your preference and thoughts.

When documenting the results, be sure to include an update on the remaining issues and current results of tests that pass and don't pass.

Let me know when you have a draft you would like reviewed.

[m1kit]

Hi @goneall , @anshuldutt21,

I wrote the report here:
https://github.com/m1kit/yalm-python/wiki/gsoc-final-report

Please take a look if you have time.

Thanks

[m1kit]

In addition, I would like to include an update to this project on the SPDX github site. We could merge it back to the original project and rename it to YALM, create a new project, or clone a copy from your site. Let me know your preference and thoughts.

My preference is...

1. rename spdx/spdx_python_licensematching to spdx/yalm-python and merge everything

• it enables me to continue working on this repo and send PRs

2. create a new repo named spdx/yalm-python
3. fork this repo as spdx/yalm-python

• I think GitHub does not allow this - SPDX already owns spdx_python_licensematching

[goneall]

Overall - the final report looks good and it looks like it meets the requirements for GSoC and SPDX.

A few suggested improvements:

• Make it clear in the work product that you added the added the Regex transpiler (e.g. - to improve performance, I added ...)
• Similar comment for terregex - terregex is a library to transform a regex pattern -> terregex is a library I developed to transform a regex pattern.
• To make it clearer where to find the code for terregex, add a "link to code" at the bottom (similar to your PR's). Even though you included the link at the beginning, add the link to code makes it a bit clearer
• You mentioned an issue in Accuracy Improvements "Better regex normalization" - can you add an issue(s) describing this and link to the new issue?

[m1kit]

Make it clear in the work product that you added the added the Regex transpiler (e.g. - to improve performance, I added ...)
Could you elaborate on this? The concept of transpiling existed in anshul-san's implementation.
The thing I changed is, the order of processing - normalize first, transpile next.

I updated the report from the other comments - thanks.

[goneall]

Could you elaborate on this? The concept of transpiling existed in anshul-san's implementation.

I was actually thinking of something different - the pre-processing of the regexes into the stored repo which you do make clear later in the write-up. You can ignore this comment.

[goneall]

1. rename spdx/spdx_python_licensematching to spdx/yalm-python and merge everything

@anshuldutt21 any concerns with renaming the repo? If not, I'll go ahead and rename. After renaming, @m1kit can create a PR so we can review/merge.

[anshuldutt21]

@goneall Please go ahead with renaming the repo.

[goneall]

I just renamed the repo in SPDX - @m1kit you can go ahead and put in a PR for the project.

[m1kit]

Made a PR - spdx#17