Fixed #519 - Remove the ASCII Limitation

chinyeungli · chinyeungli · commit b6e8b3ff4e58 · 2023-03-08T18:21:28.000+08:00
* Note that I still tend to keep ASCII-only for field name
 * This commit also reove the test pipeline for python3.6

Signed-off-by: Chin Yeung Li &lt;tli@nexb.com&gt;
diff --git a/.travis.yml b/.travis.yml
@@ -10,7 +10,6 @@ dist: xenial
 
 language: python
 python:
-  - "3.6"
   - "3.7"
   - "3.8"
   - "3.9"
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -1,11 +1,15 @@
 ==============================
 Changelog
 2023-xx-xx
-    Release 7.3.0
+    Release 8.0.0
 
     * Fixed the transform code for xlsx and json
     * Remove irrelevant error for attrib
     * Add support to identify worksheet name for XLSX input
+    * The severity error level for "contains illegal name characters (or empty spaces) and is ignored" is changed from ERROR to WARNING
+    * Remove the limitation to ASCII only
+    * Drop support for python3.6
+    * Update valid chatacters for file/path name
 
 
 2022-10-24
diff --git a/Dockerfile b/Dockerfile
@@ -6,7 +6,7 @@
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
 
-FROM python:3.6-slim-buster 
+FROM python:3.7-slim-buster
 
 RUN apt-get update \
  && apt-get install -y bash bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libgomp1 libpopt0\
@@ -27,7 +27,7 @@ RUN bash -c "source ./configure"
 # Add aboutcode to path
 #ENV PATH=$HOME/aboutcode-toolkit:$PATH
 
-# Set entrypoint to be the aboutcode command, allows to run the generated docker image directly with the aboutcode arguments: 
+# Set entrypoint to be the aboutcode command, allows to run the generated docker image directly with the aboutcode arguments:
 # `docker run (...) <containername> <about arguments>`
 # Example: docker run --rm --name "aboutcode" -v ${PWD}:/project -v /tmp/result:/result aboutcode-toolkit attrib /project /result/c.html
 ENTRYPOINT ["./bin/about"]
diff --git a/README.rst b/README.rst
@@ -21,7 +21,7 @@ In addition, this tool is able to generate attribution notices and
 identify redistributable source code used in your project to help you comply
 with open source licenses conditions.
 
-This version of the AboutCode Toolkit follows the ABOUT specification version 3.2.3 at:
+This version of the AboutCode Toolkit follows the ABOUT specification version 3.3.0 at:
 https://aboutcode-toolkit.readthedocs.io/en/latest/specification.html
 
 
@@ -39,7 +39,7 @@ Build and tests status
 
 REQUIREMENTS
 ------------
-The AboutCode Toolkit is tested with Python 3.6.2 or above only on Linux, Mac and Windows.
+The AboutCode Toolkit is tested with Python 3.7 or above only on Linux, Mac and Windows.
 You will need to install a Python interpreter if you do not have one already
 installed.
 
@@ -69,11 +69,6 @@ To install all the needed dependencies in a virtualenv, run (on posix):
 or on windows:
     configure
 
-Note
-~~~~
-   For MacOS users, it's a known issue the Python36 may case SSL Certificates error if the Certificates is not up to date.
-   A solution is to run: `sudo /Applications/Python\\ 3.6/Install\\ Certificates.command` to upgrade the outdated certificates.
-
 ACTIVATE the VIRTUALENV
 -----------------------
 To activate the virtualenv, run (on posix):
diff --git a/about.ABOUT b/about.ABOUT
@@ -1,6 +1,6 @@
 about_resource: .
 name: AboutCode-toolkit
-version: 7.3.0
+version: 8.0.0
 author: Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez
 copyright: Copyright (c) nexB Inc.
 description: |
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
@@ -11,15 +11,15 @@ jobs:
       parameters:
           job_name: ubuntu18_cpython
           image_name: ubuntu-18.04
-          python_versions: ['3.6', '3.7', '3.8', '3.9', '3.10']
+          python_versions: ['3.7', '3.8', '3.9', '3.10']
           test_suites:
               all: venv/bin/pytest -n 2 -vvs
 
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu20_cpython
           image_name: ubuntu-20.04
-          python_versions: ['3.6', '3.7', '3.8', '3.9', '3.10']
+          python_versions: ['3.7', '3.8', '3.9', '3.10']
           test_suites:
               all: venv/bin/pytest -n 2 -vvs
 
@@ -53,7 +53,7 @@ jobs:
       parameters:
           job_name: win2019_cpython
           image_name: windows-2019
-          python_versions: ['3.6', '3.7', '3.8', '3.9', '3.10']
+          python_versions: ['3.7', '3.8', '3.9', '3.10']
           test_suites:
               all: venv\Scripts\pytest -n 2 -vvs
 
diff --git a/docs/source/home.rst b/docs/source/home.rst
@@ -20,7 +20,7 @@ In addition, this tool is able to generate attribution notices and
 identify redistributable source code used in your project to help you comply
 with open source licenses conditions.
 
-This version of the AboutCode Toolkit follows the ABOUT specification version 3.2.3 at:
+This version of the AboutCode Toolkit follows the ABOUT specification version 3.3.0 at:
 https://aboutcode-toolkit.readthedocs.io/en/latest/specification.html
 
 
@@ -38,7 +38,7 @@ Build and tests status
 
 REQUIREMENTS
 ------------
-The AboutCode Toolkit is tested with Python 3.6.2 or above only on Linux, Mac and Windows.
+The AboutCode Toolkit is tested with Python 3.7 or above only on Linux, Mac and Windows.
 You will need to install a Python interpreter if you do not have one already
 installed.
 
@@ -69,13 +69,6 @@ To install all the needed dependencies in a virtualenv, run (on posix):
 or on windows:
     configure
 
-.. note::
-   For MacOS users, it's a known issue the Python36 may case SSL Certificates error
-   if the Certificates is not up to date.
-
-   A solution is to run: `sudo /Applications/Python\\ 3.6/Install\\ Certificates.command`
-   to upgrade the outdated certificates.
-
 ACTIVATE the VIRTUALENV
 -----------------------
 To activate the virtualenv, run (on posix):
diff --git a/docs/source/specification.rst b/docs/source/specification.rst
@@ -1,7 +1,7 @@
 .. _specification:
 
 ===============================
-ABOUT File Specification v3.2.3
+ABOUT File Specification v3.3.0
 ===============================
 
 Purpose
@@ -57,8 +57,7 @@ The meaning of this ABOUT file is:
 Specification
 =============
 
-An ABOUT file is an ASCII YAML formatted text file. Note that while Unicode characters
-are not supported in an ABOUT file proper, external files can contain UTF-8 Unicode.
+An ABOUT file is an YAML formatted text file.
 The key for the licenses field and the license_expression are dejacode license key.
 
 ABOUT file name
@@ -67,11 +66,9 @@ ABOUT file name
 An ABOUT file name can use a limited set of characters and is suffixed with a
 ".ABOUT" extension using any combination of uppercase and lowercase characters.
 
-A file name can contain only these US-ASCII characters:
+A file name can contain any characters and digits with the following exception and condition:
 
--   digits from 0 to 9
--   uppercase and lowercase letters from A to Z
--   the following symbols: ``"_", "-", "+", ".", "(", ")", "~", "[", "]", "{", "}", "@", "%"``
+-   the following symbols are not supported: ``", #, &, ', *, \, :, ;, <, >, =, ?, /, ^, `, |``
 -   The case of a file name is not significant. On case-sensitive file systems
     (such as on Linux), a tool must report an error if two ABOUT files stored in the same
     directory have the same lowercase file name. This is to ensure that ABOUT files can be
@@ -81,7 +78,7 @@ A file name can contain only these US-ASCII characters:
 Lines of text
 -------------
 
-An ABOUT file contains lines of US-ASCII text. Lines contain field names/values pairs.
+An ABOUT file contains lines of text. Lines contain field names/values pairs.
 The standard line ending is the LF character. The line ending characters can be any LF,
 CR or CR/LF and tools must normalize line endings to LF when processing an ABOUT file.
 Empty lines and lines containing only white spaces that are not part of a field value
@@ -109,7 +106,7 @@ The field value is separated from the field name by a ":" colon. The ":" colon
 can be followed by one or more spaces that must be ignored. This also applies
 to trailing white spaces: they must be ignored.
 
-The field value is composed of one or more lines of plain US-ASCII printable text.
+The field value is composed of one or more lines of plain printable text.
 
 When a field value is a long string, additional continuation lines must start with
 at least one space. In this case, the first space of an additional continuation
@@ -169,9 +166,7 @@ for long texts or to reference a common text in multiple ABOUT files such as a
 common license text. In this case the field name is suffixed with "_file" and the
 field value must be a path pointing to the file that contains the actual value of the
 field. This path must be a POSIX path relative to the path of the ABOUT file. The file
-content must be UTF-8-encoded text. This is in contrast with field values contained
-directly in an ABOUT file that must be US-ASCII- encoded text and allows to support
-non-ASCII text content.
+content must be UTF-8-encoded text.
 
 For example, the full license text for a component is often stored in a separate file named COPYING:
 
diff --git a/etc/scripts/gen_requirements.py b/etc/scripts/gen_requirements.py
@@ -34,7 +34,7 @@ def gen_requirements():
         type=pathlib.Path,
         required=True,
         metavar="DIR",
-        help="Path to the 'site-packages' directory where wheels are installed such as lib/python3.6/site-packages",
+        help="Path to the 'site-packages' directory where wheels are installed such as lib/python3.7/site-packages",
     )
     parser.add_argument(
         "-r",
diff --git a/etc/scripts/gen_requirements_dev.py b/etc/scripts/gen_requirements_dev.py
@@ -36,7 +36,7 @@ def gen_dev_requirements():
         type=pathlib.Path,
         required=True,
         metavar="DIR",
-        help='Path to the "site-packages" directory where wheels are installed such as lib/python3.6/site-packages',
+        help='Path to the "site-packages" directory where wheels are installed such as lib/python3.7/site-packages',
     )
     parser.add_argument(
         "-d",
diff --git a/setup.cfg b/setup.cfg
@@ -15,7 +15,6 @@ classifiers =
     Development Status :: 5 - Production/Stable
     Intended Audience :: Developers
     Programming Language :: Python :: 3 :: Only
-    Programming Language :: Python :: 3.6
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
@@ -55,7 +54,7 @@ zip_safe = false
 
 setup_requires = setuptools_scm[toml] >= 4
 
-python_requires = >=3.6
+python_requires = >=3.7
 
 install_requires =
     attrs
diff --git a/src/attributecode/__init__.py b/src/attributecode/__init__.py
@@ -20,9 +20,9 @@
 
 import saneyaml
 
-__version__ = '7.2.1'
+__version__ = '8.0.0'
 
-__about_spec_version__ = '3.2.3'
+__about_spec_version__ = '3.3.0'
 
 __copyright__ = """
 Copyright (c) nexB Inc. All rights reserved. http://dejacode.org
diff --git a/src/attributecode/model.py b/src/attributecode/model.py
@@ -720,7 +720,7 @@ def validate_field_name(name):
     if not is_valid_name(name):
         msg = ('Field name: %(name)r contains illegal name characters '
                '(or empty spaces) and is ignored.')
-        return Error(ERROR, msg % locals())
+        return Error(WARNING, msg % locals())
 
 
 class License:
@@ -948,7 +948,7 @@ def hydrate(self, fields):
         if illegal_name_list:
             msg = ('Field name: %(illegal_name_list)r contains illegal name characters '
                '(or empty spaces) and is ignored.')
-            errors.append(Error(ERROR, msg % locals()))
+            errors.append(Error(WARNING, msg % locals()))
         return errors
 
     def process(self, fields, about_file_path, running_inventory=False,
diff --git a/src/attributecode/util.py b/src/attributecode/util.py
@@ -55,7 +55,8 @@ def to_posix(path):
 UNC_PREFIX_POSIX = to_posix(UNC_PREFIX)
 UNC_PREFIXES = (UNC_PREFIX_POSIX, UNC_PREFIX,)
 
-valid_file_chars = string.digits + string.ascii_letters + '_-.+()~[]{}|@%' + ' '
+valid_file_chars = '_-.+()~[]{}@%!$,'
+invalid_file_chars = string.punctuation.translate(str.maketrans("", "", valid_file_chars))
 
 
 def invalid_chars(path):
@@ -65,7 +66,8 @@ def invalid_chars(path):
     path = to_posix(path)
     rname = resource_name(path)
     name = rname.lower()
-    return [c for c in name if c not in valid_file_chars]
+
+    return [c for c in name if c in invalid_file_chars]
 
 
 def check_file_names(paths):
@@ -74,11 +76,6 @@ def check_file_names(paths):
     there are no case-insensitive duplicates in any given directories.
     Return a list of errors.
 
-    From spec :
-        A file name can contain only these US-ASCII characters:
-        - digits from 0 to 9
-        - uppercase and lowercase letters from A to Z
-        - the _ underscore, - dash and . period signs.
     From spec:
      The case of a file name is not significant. On case-sensitive file
      systems (such as Linux), a tool must raise an error if two ABOUT files
diff --git a/tests/test_gen.py b/tests/test_gen.py
@@ -19,9 +19,10 @@
 from testing_utils import get_temp_dir
 from testing_utils import get_test_loc
 
+from attributecode import CRITICAL
 from attributecode import ERROR
 from attributecode import INFO
-from attributecode import CRITICAL
+from attributecode import WARNING
 from attributecode import Error
 from attributecode import gen
 from unittest.case import skip
@@ -66,7 +67,7 @@ def test_check_newline_in_file_field(self):
 
     def test_check_about_resource_filename(self):
         arp1 = '/test/t@est.c'
-        arp2 = '/test/t!est.c'
+        arp2 = '/test/t|est.c'
         msg = ("Invalid characters present in 'about_resource' "
                    "field: " + arp2)
         expected2 = Error(ERROR, msg)
@@ -81,7 +82,7 @@ def test_load_inventory(self):
         errors, abouts = gen.load_inventory(location, base_dir=base_dir)
 
         expected_num_errors = 29
-        assert len(errors) == expected_num_errors 
+        assert len(errors) == expected_num_errors
 
         expected = (
 '''about_resource: .
@@ -106,7 +107,7 @@ def test_load_inventory_without_about_resource(self):
         expected_error = [Error(CRITICAL,  "The essential field 'about_resource' is not found in the <input>")]
 
         assert errors == expected_error
-        assert abouts == [] 
+        assert abouts == []
 
     def test_load_inventory_without_about_resource_from_attrib(self):
         location = get_test_loc('test_gen/inv_no_about_resource.csv')
@@ -115,7 +116,7 @@ def test_load_inventory_without_about_resource_from_attrib(self):
         errors, abouts = gen.load_inventory(location, base_dir=base_dir, from_attrib=from_attrib)
 
         expected_num_errors = 0
-        assert len(errors) == expected_num_errors 
+        assert len(errors) == expected_num_errors
 
         expected = (
 '''about_resource: .
@@ -132,7 +133,7 @@ def test_load_inventory_with_errors(self):
         base_dir = get_temp_dir()
         errors, abouts = gen.load_inventory(location, base_dir=base_dir)
         expected_errors = [
-            Error(ERROR, "Field name: ['confirmed copyright'] contains illegal name characters (or empty spaces) and is ignored."),
+            Error(WARNING, "Field name: ['confirmed copyright'] contains illegal name characters (or empty spaces) and is ignored."),
             Error(INFO, 'Field about_resource: Path'),
             Error(INFO, "Field ['resource', 'test'] is a custom field.")
         ]
@@ -165,7 +166,7 @@ def test_load_inventory_simple_xlsx(self):
 
         assert abouts[0].name.value == 'cryptohash-sha256'
         assert abouts[1].name.value == 'some_component'
-        
+
         assert abouts[0].version.value == 'v 0.11.100.1'
         assert abouts[1].version.value == 'v 0.0.1'
 
@@ -188,7 +189,7 @@ def test_load_scancode_json(self):
                     'authors': [], 'packages': [], 'emails': [], 'urls': [], 'files_count': 9,
                     'dirs_count': 1, 'size_count': 32826, 'scan_errors': []}
 
-        # We will only check the first element in the inventory list 
+        # We will only check the first element in the inventory list
         assert inventory[0] == expected
 
 
diff --git a/tests/test_model.py b/tests/test_model.py
@@ -582,7 +582,7 @@ def test_About_rejects_non_ascii_names_and_accepts_unicode_values(self):
         test_file = get_test_loc('test_model/parse/non_ascii_field_name_value.about')
         a = model.About(test_file)
         expected = [
-            Error(ERROR, "Field name: ['mat\xedas'] contains illegal name characters (or empty spaces) and is ignored.")
+            Error(WARNING, "Field name: ['mat\xedas'] contains illegal name characters (or empty spaces) and is ignored.")
         ]
         assert expected == a.errors
 
diff --git a/tests/test_util.py b/tests/test_util.py

-Original file line number
+Diff line change
 language: python
 python:
 -  - "3.6"
   - "3.7"
   - "3.8"
   - "3.9"
Original file line number	Diff line number	Diff line change
`@@ -34,7 +34,7 @@ def gen_requirements():`
`34`	`34`	`type=pathlib.Path,`
`35`	`35`	`required=True,`
`36`	`36`	`metavar="DIR",`
`37`		`- help="Path to the 'site-packages' directory where wheels are installed such as lib/python3.6/site-packages",`
	`37`	`+ help="Path to the 'site-packages' directory where wheels are installed such as lib/python3.7/site-packages",`
`38`	`38`	`)`
`39`	`39`	`parser.add_argument(`
`40`	`40`	`"-r",`
Original file line number	Diff line number	Diff line change
`@@ -36,7 +36,7 @@ def gen_dev_requirements():`
`36`	`36`	`type=pathlib.Path,`
`37`	`37`	`required=True,`
`38`	`38`	`metavar="DIR",`
`39`		`- help='Path to the "site-packages" directory where wheels are installed such as lib/python3.6/site-packages',`
	`39`	`+ help='Path to the "site-packages" directory where wheels are installed such as lib/python3.7/site-packages',`
`40`	`40`	`)`
`41`	`41`	`parser.add_argument(`
`42`	`42`	`"-d",`