How to differentiate between license and notice for matches?

[ghsa-retrieval]

I would like to find both licenses and notices with scancode-tookit but need to differentiate between them. The scan result seems to only generate opaque categories is_legal or is_manifest. The former is set for both licenses and notices, so it is not helpful. Exact matches to license text can be told apart from notices by checking the rule url. The ones in "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses" or "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/licenses" are definitely true ones. However there are also other rules which identify licenses, such as https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/rules/apache-2.0_1059.RULE

Now all these rules do indicate in their definition whether they are for a license through is_license_text: yes or a notice is_license_notice: yes. How do I get access to these attributes in the scan results? Is there any particular flag that needs to be set?

Example command that I have used:
scancode --copyright --email --info --license --license-text --url --classify --json scan.json .

[AyanSinhaMahapatra]

I'm not sure if you're trying to identify license vs notice files or you're trying to identify license texts/notice texts detected, so I'll specify both:

1. Eventually we would have a is_notice flag similarly like the is_legal flag, and you can use this to differentiate. See Add is_notice flag to the --classify option  #3822 and Add is_notice flag to the --classify option #3822 #4142 which is related.
2. You can also get from a detected license match, what is the type of the rule. For example, every license match has a rule_identifier . You can use the --license-references option to get the complete data on the detected rules and licenses added to your scan results, and get the is_license_text and is_license_notice flags from there.

[ghsa-retrieval]

Thank you so much for your reply! The --license-references option is what I need for my use case then. I'll have to check the rule used in the match against the ones present in license_rule_references but this would then get the desired result.

[AyanSinhaMahapatra]

Yes, this information used to be inlined and available with all the matches earlier, but in the new LicenseDetection output data format, we decided to only have this in the results optionally based on the CLI option, and in one place to avoid redundant data across the results.

[ghsa-retrieval]

I just somehow missed that while searching in the docs. Perhaps I stumbled upon some examples that still showed the old output format, while not looking at the one option that actually does produce the desired output.

[pombredanne]

ah, so we may not have fully up-to-date docs?
@ghsa-retrieval also as you are driving through your use case, bring it up if you think we can implement this in the code for everyone!

[ghsa-retrieval]

@pombredanne I just double checked, no I think the latest docs are correct. I think I might have landed on an older version coming from Google search, that's on me. is_license_text is mentioned on the following pages:

• https://scancode-toolkit.readthedocs.io/en/latest/how-to-guides/add_new_license_detection_rule.html?highlight=is_license_text
  • Describes what rules should contain in their metadata block
• https://scancode-toolkit.readthedocs.io/en/latest/reference/license-detection-reference.html?highlight=is_license_text
  • Shows the old vs. the new format
• https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/scan-options-post.html?highlight=is_license_text
  • Explains the --license-references option
• https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/synopsis.html?highlight=is_license_text
  • Show JSON output containing licenses where is_license_text is also listed under

All of these seem correct to me. Apologies for the confusion.

As for my use case, I'm trying to identify license texts and notices in cloned repositories so that I can determine the URL to the file for each, which I think is out of scope for scancode-toolkit. This is essentially a part of a tool that we intend to use in combination with DejaCode, ScanCode.io, and PurlDB. The intention is to achieve the following:

1. Have the actual original license text for redistribution available as files
2. Being able to perform sanity checks for projects in DejaCode where the package may have been scanned. If there are any information missing or inconsistent
3. Sometimes the packages might not have the full license text included, so checking the VCS/Git might be needed. ScanCode.io can check codebases, but we need this to integrate with existing workflows and documentation pipeline. We also need the precise version-tagged link to the file. This is where scancode-toolkit is used to determine the files.
4. Generate output that can be processed for the internal documentation and user documentation. This includes links to either the original license text or links to the license declaration/notice if there are multiple licenses that apply.

[pombredanne]

@ghsa-retrieval re:

All of these seem correct to me. Apologies for the confusion.

Please do not apologize, your contributions are super valuable!

re: your use case

As for my use case, I'm trying to identify license texts and notices in cloned repositories so that I can determine the URL to the file for each, which I think is out of scope for scancode-toolkit. This is essentially a part of a tool that we intend to use in combination with DejaCode, ScanCode.io, and PurlDB.

This may not be a use case for scancode-toolkit, but that's a use case for ScanCode.io with the output available in DejaCode and PurlDB alright. I think there is some basics in DejaCode where we are trying to identify key files.