4f536a2 may make it harder for some consumers to use version ranges

[ElectricNroff]

As mentioned in the #426 issue, 4f536a2 allows one artifactID per element of the affected array. However, each element of the affected array presumably needs to fully describe affected software with other information (e.g., vendor, product, and versions); otherwise, the element could easily be misinterpreted by consumers that do not process artifactID properties. In other words, what would previously be expressed with a wide version range, e.g.,

{"dataType":"CVE_RECORD","dataVersion":"5.1","cveMetadata":{"cveId":"CVE-1900-0001",
"assignerOrgId":"b3476cb9-2e3d-41a6-98d0-0f47421a65b6","state":"PUBLISHED"},
"containers":{"cna":{"providerMetadata":{"orgId":"b3476cb9-2e3d-41a6-98d0-0f47421a65b6"},
"affected":[

    {
      "vendor":"v","product":"p",
      "versions":[{"versionType":"semver","version":"1.0.0","lessThan":"1.0.3","status":"affected"}],
      "defaultStatus":"unaffected"
    }

],
"descriptions":[{"lang":"en","value":"d"}],"references":[{"url":"https://a.ai"}]}}}

might now be expressed with several narrow version ranges, e.g.,

{"dataType":"CVE_RECORD","dataVersion":"5.1","cveMetadata":{"cveId":"CVE-1900-0002",
"assignerOrgId":"b3476cb9-2e3d-41a6-98d0-0f47421a65b6","state":"PUBLISHED"},
"containers":{"cna":{"providerMetadata":{"orgId":"b3476cb9-2e3d-41a6-98d0-0f47421a65b6"},
"affected":[

    {
      "artifactID": "gitoid:blob:sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "artifactType": "artifact",
      "vendor":"v","product":"p",
      "versions":[{"versionType":"semver","version":"1.0.0","lessThan":"1.0.1","status":"affected"}],
      "defaultStatus":"unknown"
    },
    {
      "artifactID": "gitoid:blob:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "artifactType": "artifact",
      "vendor":"v","product":"p",
      "versions":[{"versionType":"semver","version":"1.0.1","lessThan":"1.0.2","status":"affected"}],
      "defaultStatus":"unknown"
    },
    {
      "artifactID": "gitoid:blob:sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
      "artifactType": "artifact",
      "vendor":"v","product":"p",
      "versions":[{"versionType":"semver","version":"1.0.2","lessThan":"1.0.3","status":"affected"}],
      "defaultStatus":"unknown"
    }

],
"descriptions":[{"lang":"en","value":"d"}],"references":[{"url":"https://a.ai"}]}}}

Some of the common consequences may include:

1. It is unclear what is meant by mentioning the same vendor and product in multiple array elements, each with different version ranges but no other directly distinguishing details. For example, there is no documentation stating that the conclusion of unknown for 1.0.1 in array element 0 is superseded by the conclusion of affected for 1.0.1 in array element 1. Also, there is no way to express a higher-level defaultStatus that applies across multiple array elements, and thus part of the intended meaning (i.e., there is a fix in version 1.0.3 and later) is lost.
2. If humans had been visiting the cve.org website to view summary information such as affected from 1.0.0 before 1.0.3, they would now need to piece together information such as affected from 1.0.0 before 1.0.1 and affected from 1.0.1 before 1.0.2
3. If a producer later needs to add information that applies to every affected version, such as "platforms": ["Windows"], they now need to add that in multiple places
4. If there is a large amount of shared information across all affected versions (e.g., many programRoutines) and many affected versions, the document could become much larger because of the new type of duplication of information
5. Consumers need to understand that "defaultStatus":"unknown" or "defaultStatus":"unaffected" does not mean that the gitoid refers to an unknown/unaffected artifact (the schema file at 4f536a2 does not explicitly say that other properties in the same affected array element are irrelevant to the meaning of artifactID)
6. Consumers might incorrectly assume that an artifactID must be related to product and version information in the same array element, e.g., a file matching bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb may normally be found only if version 1.0.1 is installed. However, the above organization of artifactID data might instead mean that the three gitoids correspond to three different files (e.g., large.js, medium.js, and small.js) and those files are exactly the same regardless of whether 1.0.0, 1.0.1, or 1.0.2 is installed. In other words, the schema does not inform producers about whether the data organization is expected to be intuitive to human readers.
7. Admittedly, a producer could write "version":"1.0.0","lessThan":"1.0.3" in three different array elements (that have different artifactID values), but this does not inform consumers about whether the producer is mentioning a set of artifacts that cover all three cases (e.g., whether 1.0.0, 1.0.1, or 1.0.2 is installed) or has instead chosen to mention three artifacts that can only occur if 1.0.2 is installed (e.g., large.js, medium.js, and small.js if those were always different for different versions of the product).

There could, of course, be a later migration to a different data format in which a single array element can mention multiple artifacts, perhaps along with information about what file is associated with each artifact.

[alilleybrinker]

"affected":[
    {
      "vendor":"v",
      "product":"p",
      "versions":[{"versionType":"semver","version":"1.0.0","lessThan":"1.0.3","status":"affected"}],
      "defaultStatus":"unaffected"
    },
    {
      "artifactID": "gitoid:blob:sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "artifactType": "artifact",
      "vendor":"v",
      "product":"p",
      "defaultStatus":"affected"
    },
    {
      "artifactID": "gitoid:blob:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "artifactType": "artifact",
      "vendor":"v",
      "product":"p",
      "defaultStatus":"affected"
    },
    {
      "artifactID": "gitoid:blob:sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
      "artifactType": "artifact",
      "vendor":"v",
      "product":"p",
      "defaultStatus":"affected"
    }

],

[alilleybrinker]

@ElectricNroff or @ccoffin this can be closed, as the discussions in the last two QWG meetings have resolved the conversation in favor of not including the versions field when fine-grained identifiers are used, per the example I gave in my last comment. You can see the updates to the PR as well #410.

[ElectricNroff]

the discussions in the last two QWG meetings have resolved the conversation in favor of not including the versions field when fine-grained identifiers are used, per the example I gave in my last comment.

This was not resolved in any QWG meeting. Near the end of the discussion in the July 17 QWG meeting was a statement of "in theory you could use the versions field" followed by mentioning that there would be documentation drafts to share. People can then look at these documentation drafts to assess whether they are adequate to replace the long-standing documentation of:

cve-schema/schema/CVE_Record_Format.json

Lines 280 to 281 in a29f28e

	"defaultStatus": {
	"description": "The default status for versions that are not otherwise listed in the versions list. If not specified, defaultStatus defaults to 'unknown'. Versions or defaultStatus may be omitted, but not both.",

[alilleybrinker]

Hm, okay, that's a different takeaway than I had from the conversion. That said, you can see the updates to the descriptions in #410: https://github.com/CVEProject/cve-schema/pull/410/files#diff-3432912d973dd8498a2a94ac46e81947595936c8952a58901c6e9907b7d5cedb

[ElectricNroff]

This is a very different proposal that forbids a number of important use cases.

cve-schema/schema/CVE_Record_Format.json

Line 148 in b532f24

{"required": ["platforms"]},

It's realistic that the best available information about the presence of vulnerable code is a file hash combined with a platform, e.g., even though the file itself is identical on all systems, its code is only vulnerable on 32-bit systems or only vulnerable on Windows. Now, however, I am not allowed to express that.

cve-schema/schema/CVE_Record_Format.json

Line 146 in b532f24

{"required": ["programFiles"]},

It may be true that any occurrence of a hash with any file name is a strong indication of a vulnerable system, but it may simultaneously be true that computing hashes of every file is prohibitively expensive (e.g., if I am looking for that hash on the desktop systems of all end users, who may have that file in any directory among terabytes of other files). I may realistically know that the hash normally occurs for usualName.exe and that there isn't any common situation in which the file is renamed. If I were allowed to write "programFiles": ["usualName.exe"] then it's possible to do vulnerability assessment over a feasible search space. However, I am not allowed to express that.

cve-schema/schema/CVE_Record_Format.json

Line 147 in b532f24

{"required": ["programRoutines"]},

I may realistically know that any occurrence of the hash corresponds to a file with vulnerable code, but I may simultaneously know that the system is vulnerable only if a specific function in that file is called. For example, there are many cases in PHP where a vulnerable file is present in many custom projects, but there is no single convention for what that file should a named. If It were allowed to write "programRoutines": ["theOneBadFunction"] then that could suggest the next step in vulnerability confirmation after locating a file with that hash. However, I am not allowed to express that.

cve-schema/schema/CVE_Record_Format.json

Line 150 in b532f24

{"required": ["versions"]}

Providing hashes of vulnerable files is often much better than providing no hashes at all. However, consumers often benefit from explicit information about coverage. For example, my approach to vulnerability assessment may be very different if I know that the hashes were intentionally selected to correspond to the most commonly installed vulnerable version (i.e., the latest version at the time the vulnerability was discovered), and that the producer was intentionally limiting their level of effort by not providing hashes for any other vulnerable version. The data format, in effect, only allows providing a bag of hashes and does not allow one to express any correlations between those hashes and the producer's coverage effort.

Sometimes package names might be considered fine-grained. Based on the https://packages.debian.org/bookworm/linux-headers-6.1.0-32-4kc-malta page, I might choose to write:

    {
      "omniborArtifactID": "gitoid:blob:sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "omniborArtifactType": "artifact",
      "collectionURL": "https://packages.debian.org",
      "packageName": "linux-headers-6.1.0-32-4kc-malta",
      "defaultStatus": "affected"
    }

Here, the reader needs to know that "defaultStatus": "affected" applies to the fine-grained information in omniborArtifactID and does not apply to the fine-grained information in packageName.

Sometimes file names are fine-grained. For example, when software is built with the https://www.npmjs.com/package/rollup approach, there can be a file name such as index-EH2A_Wr9.js where (very roughly) EH2A_Wr9 is a fine-grained inherent identifier for the collection of JavaScript code that was used to build this index-EH2A_Wr9.js file (EH2A_Wr9 is a substring of a hash). I can write:

    {
      "vendor":"v",
      "product":"p",
      "programFiles": ["index-EH2A_Wr9.js"],
      "defaultStatus":"affected"
    }

Here, the reader might not know that whether "defaultStatus": "affected" applies to product p in general, or only applies to the fine-grained object with the name index-EH2A_Wr9.js.

[alilleybrinker]

@ElectricNroff can you please comment on the PR itself in the future? Making separate issues that reference commit hashes, and then continuing to have conversations there instead of on the main PR, makes it more difficult to track the status of conversions about that PR.

---

Now regarding the use cases you spell out, none of these make sense in the context of OmniBOR Artifact IDs, which is why they are excluded.

• Use of platform alongside omniborArtifactID: OmniBOR Artifact IDs in a CVE Record can be used to either identify binaries or other "final artifacts" shipped to users, or to identify build inputs which CVE consumers could search Input Manifests for. You either have the binary or input files or you do not. The platform does not provide relevant additional information.
• Use of programFiles alongside omniborArtifactID: While you're right that programFiles may be helpful to reduce the search space, I do not think we should recommend this, as it could only be used to increase the speed of identifying true positives, but should never be used to determine a system is not affected. File names change all the time; the name a file has when it is distributed from the upstream provider may frequently not match the name of the file after it is installed on the system. Encouraging users to search by file name and then only check the hash when file names match would severely handicap the value of the hash-based matching.
• Use of programRoutines alongside omniborArtifactID: Given the two use cases for OmniBOR in CVE (identifying final artifacts like binaries or identifying build inputs), programRoutines isn't a particularly useful fit for either. In the final artifacts case, there's little utility in identifying a routine, given that there's no consistent mechanism to identify if the specific routine is even present in the final binary, let alone that it is actually called. While some CVE consumers may go so far as to try reverse engineering binaries for this information, there is no guarantee any compiler retains that function, rather than optimizing it away via inlining. Depending on the language names may also be mangled due to generics and monomorphization, as happens in C++ and Rust for example. In the build input case, the only relevant check is whether the artifact identified is in a user's local Input Manifests, so the program routine is not relevant.
• Use of versions alongside omniborArtifactID: We've discussed this repeatedly in the QWG. The versions structure is not built to accompany fine-grained identifiers, and is more likely to confuse rather than clarify.

Regarding the question of how defaultStatus should be interpreted in an object with an omniborArtifactID, it should be interpreted as applying to that artifact. Yes, the object is required to also contain a vendor/product pair and/or a collectionURL/packageName pair, but the documentation can be made clear that those ought to be interpreted as being narrowed by the omniborArtifactID.

Relatedly, this is why I'd originally advocated for permitting the omniborArtifactID to fulfill the "identifier-like" requirement inside of product objects, to avoid any ambiguity. I believe we can fix the ambiguity with documentation, but I also still believe it's the inferior solution.

[ElectricNroff]

Here's a full realistic example in which file hashes are important because the vulnerable software component is not mentioned in an SBOM (that is fed into an asset database used for vulnerability assessment). In this example, the vulnerability assessment is intended to report specific production servers (of one enterprise) that are vulnerable. It is not a vulnerability assessment that occurs during a development stage, build stage, or other pre-deployment stage.

adm-zip is a popular JavaScript package that is sometimes installed by copying .js files into other projects, without declaring a dependency. Also, adm-zip is often used by web applications that execute JavaScript on the server side. These may be web applications deployed only on an enterprise's own public-facing servers, not applications that are ever shipped to any user. It is realistic to have an adm-zip vulnerability that only exists when adm-zip is running on a Windows machine (e.g., because it processes file pathnames differently on Windows than on other operating systems). To be clear, the adm-zip.js file is identical when found on either a Windows server or a Linux server, but there is no risk on Linux. This is a common situation.

In this example, each instance of an adm-zip.js file on a production server is only used on that production server. A client cannot download it (e.g., from a Linux server to a Windows client machine). System administrators do not copy the adm-zip.js file from a production Linux server to a production Windows server. In other words, to plan vulnerability remediation, it is of paramount importance to know exactly what file content is present and the operating system on which it is found.

In this scenario, a file such as adm-zip.js may be in production use on a server even though adm-zip is not mentioned in the package-lock.json (or similar) file for any web application on that server. (It's, of course, possible that an SBOM lists adm-zip for some other reason - for example, it may have been detected by an SCA tool. However, many organizations do not have any such SCA tool, or the source-sweeping feature - to add an SBOM entry based on detecting the adm-zip.js file - is turned off.)

Given this environment, one can rate various types of CVE Record content:

    "affected": [
      {
        "vendor": "adm-zip Project",
        "product": "adm-zip",
        "platforms": ["Windows"],
        "versions": [
          {
            "version": "0.5.15",
            "status": "affected",
            "lessThanOrEqual": "0.5.16",
            "versionType": "semver"
          }
        ],
        "defaultStatus": "unknown"
      }
    ],
    "cpeApplicability": [
       {
            "operator": "AND",
            "nodes": [
              {
                "operator": "OR",
                "negate": false,
                "cpeMatch": [
                  {
                     "vulnerable": true,
                     "criteria": "cpe:2.3:a:adm-zip_project:adm-zip:*:*:*:*:*:*:*:*",
                     "versionStartIncluding": "0.15.5",
                     "versionEndIncluding": "0.15.6"
                  }
                ]
              },
              {
                "operator": "OR",
                "negate": false,
                "cpeMatch": [
                  {
                    "vulnerable": false,
                    "criteria": "cpe:2.3:o:microsoft:windows:-:*:*:*:*:*:*:*"
                  }
                ]
              }
            ]
          }
        ]

This is ineffective because the vulnerability assessment is using an asset database that has no information that adm-zip is used anywhere in the enterprise. Even though cpe:2.3:a:adm-zip_project:adm-zip:*:*:*:*:*:*:*:* would often be considered correct for CPE (it's the same as in CVE-2018-1002204 in NVD), the vulnerability-assessment report will not identify any servers as affected by this adm-zip vulnerability.

Suppose that gitoid:blob:sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa is correct for the adm-zip.js file from 0.15.5. and gitoid:blob:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb is correct for the adm-zip.js file from 0.15.6. In the proposed schema, one can write:

  "affected": [
    {
      "vendor": "adm-zip Project",
      "product": "adm-zip",
      "platforms": ["Windows"],
      "versions": [
        {
          "version": "0.5.15",
          "status": "affected",
          "lessThanOrEqual": "0.5.16",
          "versionType": "semver"
        }
      ],
      "defaultStatus": "unknown"
    },
    {
      "omniborArtifactID": "gitoid:blob:sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "omniborArtifactType": "artifact",
      "vendor": "adm-zip Project",
      "product": "adm-zip",
      "defaultStatus": "affected"
    },
    {
      "omniborArtifactID": "gitoid:blob:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "omniborArtifactType": "artifact",
      "vendor": "adm-zip Project",
      "product": "adm-zip",
      "defaultStatus": "affected"
    }
  ]

Also, the vulnerability-assessment tool has a a list of all Artifact IDs on each production server. In the current example, elements 1 and 2 of the affected array are now potentially effective, whereas the element 0 is still ineffective.

Here, the main questions are how is this data useful to consumers, and how will they use the data operationally. Over the short term (which could be months or years), vulnerability-assessment products may likely continue to use hash information in the same way that they use hash information today. In other words, they might not use any novel tools or approaches developed by the OmniBOR project, and instead would ingest the omniborArtifactID values from CVE Records to augment the much larger collection of hash values maintained elsewhere (extractors.hashes at https://github.com/RetireJS/retire.js/blob/b1708e30af9f9c199ce9e2e398bb4ba4f6213bf6/repository/README.md?plain=1#L36 for example). They would need to address the existence of two hashes for each file: one with the blob_<length> used in OmniBOR and one without that.

The added value of the omniborArtifactID in the CVE Record is that it is, at least potentially, provided by the software Supplier at the moment that the vulnerability is first publicly announced. If a consumer is using a commercial product from a well-funded company, it is perhaps not a huge added value, because that company might update its proprietary database of hash values (for adm-zip.js in 0.5.15 and 0.5.16) very soon after a CVE Record is published, regardless of whether the CVE Record contains any type of hash information. By contrast, if the consumer is using a free product, then omniborArtifactID in the CVE Record may be the only useful data that ever exists, or may come days or weeks ahead of other useful data.

The main operational concern is whether the CVE Record data allows any vulnerability-assessment product (commercial or free) to correctly identify the subset of production servers that need to be fixed, and present that information in a way that a user can understand.

Suppose that gitoid:blob:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb is present on 300 Linux servers and 3 Windows servers. The consumer wants a vulnerability-assessment outcome that lists only the 3 Windows servers. (Even if the Linux servers might be updated for other reasons, it is critical to update the Windows servers now.) This outcome may be difficult to achieve because there is no direct binding between "platforms": ["Windows"] and elements 1 and 2 of the affected array. A vulnerability-assessment tool might be designed to guess that any constraint (such as Windows) for that vendor/product directly applies to each omniborArtifactID for that same vendor/product, but this won't always be valid. For example, the same affected data above can exist for a separate multi-platform vulnerability where the two omniborArtifactID values instead correspond to other versions that, unlike 0.5.15 and 0.5.16, are also affected on Linux. In other words, for that separate vulnerability, the producer has not yet investigated the version range for which Linux is affected, but has established that two adm-zip.js artifacts are affected on Linux. In general, a consumer cannot assume that array elements 1 and 2 are implicit children of array element 0 just because there is a vendor/product match. Instead, each element of the affected array must be considered independent.

For this reason, I believe it is imperative that the CVE Record format allow a direct binding between an omniborArtifactID and a platforms value, e.g.,

    {
      "omniborArtifactID": "gitoid:blob:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "omniborArtifactType": "artifact",
      "platforms": ["Windows"],
      "vendor": "adm-zip Project",
      "product": "adm-zip",
      "defaultStatus": "affected"
    }

The vulnerability-assessment report would convey information such as "win-server-4, win-server-7, and win-server-22 are vulnerable to CVE-2016-987654 because they have a hash value of vulnerable adm-zip code, and will execute that code on Windows, which is an affected platform."

Of course, one should go further and allow producers to directly bind other information, such as versions, to an omniborArtifactID. This would allow the vulnerability-assessment report to be more informative, e.g., "win-server-4, win-server-7, and win-server-22 are vulnerable to CVE-2016-987654 because they have a hash value of vulnerable adm-zip code (specifically, adm-zip 0.5.16), and will execute that code on Windows, which is an affected platform." This design is analogous to the above RetireJS example, where their data format binds hashes to version numbers: "hashes": { "07f8b94c8d601a24a1914a1a92bec0e4fafda964" : "0.0.1" } (although, of course, a single Artifact ID may exist in more than one version). Mentioning a version in the vulnerability-assessment report is important because it may provide some insight into why the server is vulnerable (e.g., 0.5.16 is expected for not-yet-patched servers, but 0.5.15 indicates that something is seriously wrong).

The above concerns are also relevant to other types of platforms information, such as CPU architecture. Vulnerability assessment based on SBOMs often misses SQLite because sqlite3.c from the https://sqlite.org/download.html amalgamation is often directly copied into other projects without declaring a dependency. Similarly, files from https://github.com/nothings/stb are often directly copied into other projects without declaring a dependency. However, a file such as sqlite3.c or stb_image.h is often used in an unmodified form, allowing an omniborArtifactID match. Both sqlite3.c and stb_image.h have had vulnerabilities that only exist on 32-bit platforms or only on 64-bit platforms. Thus, for example, if these are used for an in-house C project and one knows that the enterprise only has 64-bit servers, then the combination of an omniborArtifactID and "platforms": ["32-bit"] is extremely valuable in avoiding the cost of an unnecessary urgent update to hundreds of 64-bit servers.

[alilleybrinker]

Since the Software ID proposal will be split between Package URL and OmniBOR parts, with the OmniBOR part receiving further work to improve the handling of fine-grained identifiers, I expect we'll be making improvements here that can incorporate the feedback. My immediate focus will just be on doing the splitting and getting the Package URL part over the finish line.