Merge pull request #513 from github/lcartey/update-supported-docs

Update Coding Standards coverage documentation

Author: rvermeulen

README.md

@@ -6,15 +6,12 @@ This repository contains CodeQL queries and libraries which support various Codi
 
 _Carnegie Mellon and CERT are registered trademarks of Carnegie Mellon University._
 
-This repository contains CodeQL queries and libraries which support various Coding Standards for the [C++14](https://www.iso.org/standard/64029.html) programming language.
+This repository contains CodeQL queries and libraries which support various Coding Standards for the [C++14](https://www.iso.org/standard/64029.html), [C99](https://www.iso.org/standard/29237.html) and [C11](https://www.iso.org/standard/57853.html) programming languages.
 
 The following coding standards are supported:
 - [AUTOSAR - Guidelines for the use of C++14 language in critical and safety-related systems (Releases R22-11, R20-11, R19-11 and R19-03)](https://www.autosar.org/fileadmin/standards/R22-11/AP/AUTOSAR_RS_CPP14Guidelines.pdf). 
 - [MISRA C++:2008](https://www.misra.org.uk) (support limited to the rules specified in AUTOSAR).
 - [SEI CERT C++ Coding Standard: Rules for Developing Safe, Reliable, and Secure Systems (2016 Edition)](https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=494932)
-
-In addition, the following Coding Standards for the C programming language are under development:
-
 - [SEI CERT C Coding Standard: Rules for Developing Safe, Reliable, and Secure Systems (2016 Edition)](https://resources.sei.cmu.edu/downloads/secure-coding/assets/sei-cert-c-coding-standard-2016-v01.pdf)
 - [MISRA C 2012](https://www.misra.org.uk/product/misra-c2012-third-edition-first-revision/).
 

docs/user_manual.md

@@ -23,6 +23,7 @@
 | 0.15.0  | 2023-05-24 | Mauro Baluda    | Clarify AUTOSAR C++ supported versions.                                                                                 |
 | 0.16.0  | 2023-07-03 | Luke Cartey     | Remove reference to LGTM, update the name of the query pack                                                             |
 | 0.17.0  | 2023-08-16 | Luke Cartey     | Update list of supported compiler configurations.                                                             |
+| 0.18.0  | 2024-01-30 | Luke Cartey     | Update product description and coverage table.                                                             |
 
 ## Release information
 
@@ -48,15 +49,16 @@ A _coding standard_ is a set of rules or guidelines which restrict or prohibit t
 
 The _CodeQL Coding Standards_ product is a set of CodeQL queries for identifying contraventions of rules in the following coding standards:
 
-| Standard                                                                                                             | Version | Total rules | Total supportable rules | Status            |
-| -------------------------------------------------------------------------------------------------------------------- | ------- | ----------- | ----------------------- | ----------------- |
-| [AUTOSAR C++](https://www.autosar.org/fileadmin/standards/R22-11/AP/AUTOSAR_RS_CPP14Guidelines.pdf) | [^1] R22-11, R21-11, R20-11, R19-11, R19-03 | 397         | 375                     | Implemented       |
-| [CERT-C++](https://resources.sei.cmu.edu/downloads/secure-coding/assets/sei-cert-cpp-coding-standard-2016-v01.pdf)   | 2016    | 83          | 83                      | Implemented       |
-| [CERT C](https://resources.sei.cmu.edu/downloads/secure-coding/assets/sei-cert-c-coding-standard-2016-v01.pdf)       | 2016    | 99          | 99                      | Under development |
-| [MISRA C](https://www.misra.org.uk/product/misra-c2012-third-edition-first-revision/)                                | 2012    | 172         | 169                     | Under development |
+| Standard                                                                                                             | Version | Rules | Supportable rules | Implemented rules | Status            |
+| -------------------------------------------------------------------------------------------------------------------- | ------- | ----------- | ----------------------- | ----------------- | ------- |
+| [AUTOSAR C++](https://www.autosar.org/fileadmin/standards/R22-11/AP/AUTOSAR_RS_CPP14Guidelines.pdf) | [^1] R22-11, R21-11, R20-11, R19-11, R19-03 | 397         | 372                     | 370[^2] | Implemented       |
+| [CERT-C++](https://resources.sei.cmu.edu/downloads/secure-coding/assets/sei-cert-cpp-coding-standard-2016-v01.pdf)   | 2016    | 83          | 82                      | 82 | Implemented       |
+| [CERT C](https://resources.sei.cmu.edu/downloads/secure-coding/assets/sei-cert-c-coding-standard-2016-v01.pdf)       | 2016    | 99          | 97                      | 97 | Implemented |
+| [MISRA C](https://www.misra.org.uk/product/misra-c2012-third-edition-first-revision/)                                | 2012    | 175         | 164                     | 162[^3] | Implemented |
 
+Not all rules in these standards are amenable to static analysis by CodeQL - some rules require external or domain specific knowledge to validate, or refer to properties which are not present in our representation of the codebase under analysis. In addition, some rules are natively enforced by the supported compilers. As CodeQL requires that the program under analysis compiles, we are unable to implement queries for these rules, and doing so would be redundant.
 
-Not all rules in these standards are amenable to static analysis by CodeQL - some rules require external or domain specific knowledge to validate, or refer to properties which are not present in the our representation of the codebase under analysis. For each rule we therefore identify whether it is supportable or not. Furthermore, a rule can be supported in two ways:
+For each rule we therefore identify whether it is supportable or not. Furthermore, a rule can be supported in two ways:
 
    - **Automated** - the queries for the rule find contraventions directly.
    - **Audit only** - the queries for the rule does not find contraventions directly, but instead report a list of _candidates_ that can be used as input into a manual audit. For example, `A10-0-1` (_Public inheritance shall be used to implement 'is-a' relationship_) is not directly amenable to static analysis, but CodeQL can be used to produce a list of all the locations that use public inheritance so they can be manually reviewed.
@@ -66,6 +68,8 @@ Each supported rule is implemented as one or more CodeQL queries, with each quer
 The datasheet _"CodeQL Coding Standards: supported rules"_, provided with each release, lists which rules are supported for that particular release, and the _scope of analysis_ for that rule.
 
 [^1]: AUTOSAR C++ versions R22-11, R21-11, R20-11, R19-11 and R19-03 are all identical as indicated in the document change history.
+[^2]: The unimplemented supportable AUTOSAR rules are `A7-1-8` and `A8-2-1`. These rules require additional support in the CodeQL CLI to ensure the required information is available in the CodeQL database to identify violations of these rules.
+[^3]: The unimplemented supportable MISRA C 2012 rules are `Rule 9.5` and `Dir 4.14`. `Rule 9.5` requires additional support in the CodeQL CLI to ensure the required information is available in the CodeQL database to identify violations of these rules. `Dir 4.14` is covered by the default CodeQL queries, which identify potential security vulnerabilities caused by not validating external input.
 
 ## Supported environment
 
@@ -110,9 +114,9 @@ For C the codebase under analysis must comply with C99 or C11 and use one of the
 
 | Compiler | Version | Standard library    | Target architecture   | Required Flags             |
 | -------- | ------- | ------------------- | --------------------- | -------------------------- |
-| clang    | 10.0.0  | glibc (default)     | x86_64-linux-gnu      | -std=c11                   |
-| gcc      | 8.4.0   | glibc (default)     | x86_64-linux-gnu      | -std=c11                   |
-| qcc      | 8.3.0   | glibc (default)     | gcc_ntoaarch64le      | -std=c11 -nopipe           |
+| clang    | 10.0.0  | glibc (default)     | x86_64-linux-gnu      | `-std=c11` or `-std=c99`   |
+| gcc      | 8.4.0   | glibc (default)     | x86_64-linux-gnu      | `-std=c11` or `-std=c99`   |
+| qcc      | 8.3.0   | glibc (default)     | gcc_ntoaarch64le      | `-std=c11 -nopipe` or `-std=c99 -nopipe` |
 
 Use of the queries outside these scenarios is possible, but not validated for functional safety. In particular:
  - Use of the queries against codebases written with more recent versions of C (as supported by CodeQL) are not validated in the following circumstances:

rules.csv

@@ -1,4 +1,4 @@
-Language,Standard,ID,Queryable?,Obligation level,Enforcement level,Allocated target,Description,Similar,Package,Difficulty,Justification for querability level
+Language,Standard,ID,Supportable,Obligation level,Enforcement level,Allocated target,Description,Similar,Package,Difficulty,Justification for querability level
 cpp,AUTOSAR,A0-1-1,Yes,Required,Automated,Implementation,A project shall not contain instances of non-volatile variables being given values that are not subsequently used.,M0-1-6,DeadCode,Medium,
 cpp,AUTOSAR,A0-1-2,Yes,Required,Automated,Implementation,The value returned by a function having a non-void return type that is not an overloaded operator shall be used.,M0-1-7,DeadCode,Easy,
 cpp,AUTOSAR,A0-1-3,Yes,Required,Automated,Implementation,"Every function defined in an anonymous namespace, or static function with internal linkage, or private member function shall be used.",M0-1-10,DeadCode,Easy,
@@ -500,7 +500,7 @@ c,CERT-C,CON41-C,Yes,Rule,,,Wrap functions that can fail spuriously in a loop,CO
 c,CERT-C,CON43-C,OutOfScope,Rule,,,Do not allow data races in multithreaded code,,,,
 c,CERT-C,DCL30-C,Yes,Rule,,,Declare objects with appropriate storage durations,,Declarations8,Hard,
 c,CERT-C,DCL31-C,Yes,Rule,,,Declare identifiers before using them,,Declarations1,Medium,
-c,CERT-C,DCL36-C,No,Rule,,,Do not declare an identifier with conflicting linkage classifications,,,,
+c,CERT-C,DCL36-C,No,Rule,,,Do not declare an identifier with conflicting linkage classifications,,,,Compiler enforced in all supported compilers
 c,CERT-C,DCL37-C,Yes,Rule,,,Do not declare or define a reserved identifier,,Declarations1,Easy,
 c,CERT-C,DCL38-C,Yes,Rule,,,Use the correct syntax when declaring a flexible array member,,Declarations2,Easy,
 c,CERT-C,DCL39-C,Yes,Rule,,,Avoid information leakage when passing a structure across a trust boundary,,Declarations7,Hard,
@@ -585,7 +585,7 @@ c,CERT-C,POS51-C,OutOfScope,Rule,,,Avoid deadlock with POSIX threads by locking
 c,CERT-C,POS52-C,OutOfScope,Rule,,,Do not perform operations that can block while holding a POSIX lock,,,,
 c,CERT-C,POS53-C,OutOfScope,Rule,,,Do not use more than one mutex for concurrent waiting operations on a condition variable,,,,
 c,CERT-C,POS54-C,OutOfScope,Rule,,,Detect and handle POSIX library errors,,,,
-c,CERT-C,PRE30-C,No,Rule,,,Do not create a universal character name through concatenation,,,Medium,
+c,CERT-C,PRE30-C,No,Rule,,,Do not create a universal character name through concatenation,,,Medium,Compiler enforced in all supported compilers
 c,CERT-C,PRE31-C,Yes,Rule,,,Avoid side effects in arguments to unsafe macros,RULE-13-2,SideEffects4,Medium,
 c,CERT-C,PRE32-C,Yes,Rule,,,Do not use preprocessor directives in invocations of function-like macros,,Preprocessor5,Hard,
 c,CERT-C,SIG30-C,Yes,Rule,,,Call only asynchronous-safe functions within signal handlers,,SignalHandlers,Medium,
@@ -614,8 +614,8 @@ c,MISRA-C-2012,DIR-4-9,Yes,Advisory,,,A function should be used in preference to
 c,MISRA-C-2012,DIR-4-10,Yes,Required,,,Precautions shall be taken in order to prevent the contents of a header file being included more than once,M16-2-3,Preprocessor2,Medium,
 c,MISRA-C-2012,DIR-4-11,Yes,Required,,,The validity of values passed to library functions shall be checked,,Contracts,Hard,
 c,MISRA-C-2012,DIR-4-12,Yes,Required,,,Dynamic memory allocation shall not be used,,Banned,Medium,
-c,MISRA-C-2012,DIR-4-13,Yes,Advisory,,,Functions which are designed to provide operations on a resource should be called in an appropriate sequence,,Contracts,Hard,
-c,MISRA-C-2012,DIR-4-14,Yes,Required,,,The validity of values received from external sources shall be checked,,Contracts,Hard,
+c,MISRA-C-2012,DIR-4-13,No,Advisory,,,Functions which are designed to provide operations on a resource should be called in an appropriate sequence,,,,Rule 22.1, 22.2 and 22.6 cover aspects of this rule. In other cases this is a design issue and needs to be checked manually.
+c,MISRA-C-2012,DIR-4-14,Yes,Required,,,The validity of values received from external sources shall be checked,,Contracts,Hard,This is supported by CodeQLs default C security queries.
 c,MISRA-C-2012,RULE-1-1,No,Required,,,"The program shall contain no violations of the standard C syntax and constraints, and shall not exceed the implementation's translation limits",,,Easy,"This should be checked via the compiler output, rather than CodeQL, which adds unnecessary steps."
 c,MISRA-C-2012,RULE-1-2,Yes,Advisory,,,Language extensions should not be used,,Language3,Hard,
 c,MISRA-C-2012,RULE-1-3,Yes,Required,,,There shall be no occurrence of undefined or critical unspecified behaviour,,Language3,Hard,
@@ -773,5 +773,4 @@ c,MISRA-C-2012,RULE-22-6,Yes,Mandatory,,,The value of a pointer to a FILE shall
 c,MISRA-C-2012,RULE-22-7,Yes,Required,,,The macro EOF shall only be compared with the unmodified return value from any Standard Library function capable of returning EOF,,IO3,Hard,
 c,MISRA-C-2012,RULE-22-8,Yes,Required,,,The value of errno shall be set to zero prior to a call to an errno-setting-function,ERR30-C,Contracts3,Medium,
 c,MISRA-C-2012,RULE-22-9,Yes,Required,,,The value of errno shall be tested against zero after calling an errno-setting-function,,Contracts3,Medium,
-c,MISRA-C-2012,RULE-22-10,Yes,Required,,,The value of errno shall only be tested when the last function to be called was an errno-setting-function,,Contracts3,Medium,
-,,,,,,,,,,0,
+c,MISRA-C-2012,RULE-22-10,Yes,Required,,,The value of errno shall only be tested when the last function to be called was an errno-setting-function,,Contracts3,Medium,

scripts/generate_rules/generate_package_description.py

@@ -96,7 +96,7 @@ def generate_short_name(title):
       language = rule[0]
       standard = rule[1]
       rule_id = rule[2]
-      queryable = rule[3]
+      supportable = rule[3]
       obligation_level = rule[4]
       enforcement_level = rule[5]
       allocated_targets = rule[6]
@@ -106,8 +106,8 @@ def generate_short_name(title):
       difficulty = rule[10]
       # Find all rules in the given language and package
       if language == language_name and package == package_name:
-        if not queryable == "Yes":
-          print("Error: " + standard + " " + rule_id + " is marked as part of package " + package_name + " but is not marked as queryable.")
+        if not supportable == "Yes":
+          print("Error: " + standard + " " + rule_id + " is marked as part of package " + package_name + " but is not marked as supportable.")
           sys.exit(1)
 
         # Add the AUTOSAR obligation, enforcement and allocated target as query properties.

scripts/verify_rule_package_consistency.py

@@ -47,7 +47,7 @@
 
             standard = rule[1]
             rule_id = rule[2]
-            queryable = rule[3]
+            supportable = rule[3]
             obligation_level = rule[4]
             enforcement_level = rule[5]
             allocated_targets = rule[6]
@@ -57,9 +57,9 @@
             difficulty = rule[10]
             # If the rule is associated with a package
             if package:
-                if not queryable == "Yes":
+                if not supportable == "Yes":
                     print(
-                        f"ERROR: {standard} {rule_id} is included as part of package {package} but is not marked as queryable.")
+                        f"ERROR: {standard} {rule_id} is included as part of package {package} but is not marked as supportable.")
                     failed = True
                 else:
                     package_rules_from_csv[package].add(rule_id)
@@ -98,7 +98,7 @@
                         failed = True
                     if not rule_id in package_rules_from_csv[package_name]:
                         print(
-                            f" - ERROR: Rule {rule_id} included in {package_name}.json but not marked as queryable in rules.csv.")
+                            f" - ERROR: Rule {rule_id} included in {package_name}.json but not marked as supportable in rules.csv.")
                         failed = True
             rules_csv_rule_ids = package_rules_from_csv[package_name]