Possible bug with @json being treated as invalid type in frame

[dlongley]

This issue stems from: digitalbazaar/jsonld.js#506

It relates to step 1.3 in the Framing Algorithm where a frame's validity is being checked.

It seems that either that step doesn't appropriately allow for @json (which is a valid JSON-LD type) or that the algorithm should have stopped recursing when @json was detected and just accepted whatever the value was (since it cannot recurse into @json datatypes, presumably). I didn't look more closely to determine which way to go here with the algorithm (whether it's the former simple fix or not), but I expect that some test cases could help shake it out.

[pasquale95]

Hi @dlongley, have you news on this issue?
Have contributors had a look inside of the PR and gave some feedback? If there are issues in the PR let me know and I will fix it, otherwise I would advice to merge it since this feature is quite requested for correctly frame JSON-LD credentials.

[davidlehn]

@pasquale95 I'm not sure about any required spec changes. However, it would be good to propose some simplified tests in this repo to check the jsonld.js code change works. I assume tests can use inline contexts. If there are variations of the issue, it would be good to add them in case there are other code paths that need a fix. We need the tests so all implementations will have the same behavior.

[pasquale95]

@dlongley @davidlehn I don't exactly understand how testing works in this repository, but I've defined a small example in my PR #144.
Could you please have a look and tell me if it's fine?
For better understanding I would still suggest to have a look at the original issue description.

[pasquale95]

@dlongley @davidlehn any update on this PR?

[gkellogg]

There's some odd special-case stuff going on here. Because "ex:info" is defined with @type: @json, the content of the frame gets expanded as a value object. Normally, the value either needs to match exactly, or include a wildcard, but you can't wildcard the value, as it is treated as raw JSON.

I first tried the following:

{
  "@context": {
    "ex": "http://example.org/vocab#",
    "ex:info": {"@type": "@json"}
  },
  "ex:info": {
    "@value": {},
    "@type": "@json"
  }
}

but that fails, as I described, as it is expanded to { "@value": { "@value": {}, "@type": "@json" }, "@type": "@json" }

However, I got it to work as follows:

{
  "@context": {
    "ex": "http://example.org/vocab#",
    "ex:info": {"@type": "@json"}
  },
  "http://example.org/vocab#info": {
    "@value": {},
    "@type": "@json"
  }
}

(it would also work with just "http://example.org/vocab#info": {}, although it would match values which were't JSON value objects.

No spec changes are necessary, but an informative note to describe how to match JSON literals would be useful.

[pasquale95]

@gkellogg if this is not a bug, can you advice me on how should I define the correct frame for this json-ld?

{
  "@context": [
    "https://www.w3.org/2018/credentials/v1",
    "https://w3id.org/vdl/v1"
  ],
  "type": [
    "VerifiableCredential",
    "Iso18013DriversLicenseCredential"
  ],
  "credentialSubject": {
    "id": "did:key:z6MkiiViqftXJKZNnWwpS7aKM7jiJBbGiFEZzPSKYB8p8oyy",
    "license": {
      "type": "Iso18013DriversLicense",
      "document_number": "542426814",
      "family_name": "TURNER",
      "given_name": "SUSAN",
      "portrait": "/9j/4AAQSkZJRgABAQEAkACQA...gcdgck5HtRRSClooooP/2Q==",
      "birth_date": "1998-08-28",
      "issue_date": "2018-01-15T10:00:00Z",
      "expiry_date": "2022-08-27T12:00:00Z",
      "issuing_country": "US",
      "issuing_authority": "AL",
      "driving_privileges": [
        {
          "codes": [
            {
              "code": "D"
            }
          ],
          "vehicle_category_code": "D",
          "issue_date": "2019-01-01",
          "expiry_date": "2027-01-01"
        },
        {
          "codes": [
            {
              "code": "C"
            }
          ],
          "vehicle_category_code": "C",
          "issue_date": "2019-01-01",
          "expiry_date": "2017-01-01"
        }
      ],
      "un_distinguishing_sign": "USA"
    }
  }
}

The context https://w3id.org/vdl/v1 is available here (the URL itself is not working anymore). Using your testing nomenclature, the document above is the in.jsonld and my goal is to frame the driving_privileges field which is of "@type": "@json".

I tried with the frame below but it doesn't work:

{
  "@context": [
    "https://www.w3.org/2018/credentials/v1",
    "https://w3id.org/vdl/v1"
  ],
  "type": ["VerifiableCredential", "Iso18013DriversLicenseCredential"],
  "credentialSubject": {
    "@explicit": true,
    "license": {
      "type": "Iso18013DriversLicense",
      "@explicit": true,
      "driving_privileges": {}
    }
  }
}

Any help would be really appreciate.

[gkellogg]

In the frame, you’ll need to use some other term expanding to the same IRI for “driving_privilages”; other the full IRI it expands to, itself, or a compact IRI form. The issue is that the frame gets expanded when framing, then it is no longer an empty value, but an expanded JSON value, which prevents it from matching anything.

[pasquale95]

I don't understand. The issue is in the fact that "@type": "@json" is considered as an invalid type inside the frame, but this is incorrect since such type is a valid JSON-LD type.
The frame gets expanded and the json attribute gets expanded as well, but the check of the existence of such attribute declared in the frame (e.g. driving_privileges from the frame.jsonld) inside the document to frame (e.g. in the in.jsonld) is done by looking if the attribute key in expanded form (namely https://w3id.org/vdl/v1/#driving_privileges) is amongst the keys inside the document to frame in expanded form (see here). Hence, when the subframe {"@type": "@json", "@value": {}} is checked by _validateFrame, the @json type should be among the ones accepted. Therefore this PR.

Additionally, I don't understand why the @json attributes should be handled differently from the other attributes inside the frame. This would make framing more cumbersome without any specific reason.

[gkellogg]

The key is that the Framing algorithm works by first expanding the frame, which in the original case, results in the following:

[
  {
    "http://example.org/vocab#info": [
      {
        "@type": "@json",
        "@value": {
          "@value": {},
          "@type": "@json"
        }
      }
    ]
  }
]

To match a value, it would need to exactly match that value object (note the value object, itself, containing @value, rather than being a template to match anything with @type: @json. This is described in Matching on Values and the Value Pattern Matching Algorithm, and this is not a kind of wildcard.

[pasquale95]

How did you get that expansion? Unfortunately I don't really understand how to use this example, I prefer using the VDL frame as example since I am able to run and debug it (but I think this doesn't change the situation).
If I expand the following frame in JSON-LD Playground I don't get that kind of expansion for the driving_privileges attribute (which is of type @json):

{
   "@context":[
      "https://www.w3.org/2018/credentials/v1",
      "https://raw.githubusercontent.com/w3c-ccg/vdl-vocab/main/context/v1.jsonld"
   ],
   "type":[
      "VerifiableCredential",
      "Iso18013DriversLicense"
   ],
   "credentialSubject":{
      "license":{
         "type":["Iso18013DriversLicense"],
         "driving_privileges":{}
      }
   }
}

The expansion I get is the following:

[
  {
    "https://www.w3.org/2018/credentials#credentialSubject": [
      {
        "https://w3id.org/vdl#license": [
          {
            "https://w3id.org/vdl#driving_privileges": [
              {
                "@type": "@json",
                "@value": {}
              }
            ],
            "@type": [
              "https://w3id.org/vdl#Iso18013DriversLicense"
            ]
          }
        ]
      }
    ],
    "@type": [
      "https://www.w3.org/2018/credentials#VerifiableCredential",
      "https://w3id.org/vdl#Iso18013DriversLicense"
    ]
  }
]

Note that driving_privileges gets expanded and its value object is the wildcard {}.

[pasquale95]

I found why you got that value object, the frame is incorrect. We should change it to be:

{
  "@context": {
    "ex": "http://example.org/vocab#",
    "ex:info": {"@type": "@json"}
  },
  "ex:info": {}
}

This frame gets expanded as:

[
  {
    "http://example.org/vocab#info": [
      {
        "@type": "@json",
        "@value": {}
      }
    ]
  }
]

At this point, I still get the issue: jsonld.SyntaxError: Invalid JSON-LD syntax; invalid @type in frame. even if the @value is a wildcard.

[gkellogg]

Sorry, comments messed up. If you put the full example in the playground, you can use the Permalink button to capture the state and add to this comment.

[pasquale95]

Alright, here the permalink.
This frame expands as expected (without the nested type @json inside the value field), but the framing still doesn't work. Hence, the statement claiming that the issue was related to the nested value in the value problem doesn't hold anymore.

There's a bug in the framing operation itself and the bug is potentially solved with this PR (unless side-effects which I honestly don't see).

[gkellogg]

I see there's the problem you describe with the playground, but I'm not sure it's a spec bug. If you try it in my Ruby distiller, you'll get the following output:

{
  "@type": [
    "https://www.w3.org/2018/credentials#VerifiableCredential",
    "https://w3id.org/vdl#Iso18013DriversLicenseCredential"
  ],
  "https://www.w3.org/2018/credentials#credentialSubject": {
    "@id": "did:key:z6MkiiViqftXJKZNnWwpS7aKM7jiJBbGiFEZzPSKYB8p8oyy",
    "https://w3id.org/vdl#license": {
      "@type": "https://w3id.org/vdl#Iso18013DriversLicense",
      "https://w3id.org/vdl#document_number": "542426814",
      "https://w3id.org/vdl#family_name": "TURNER",
      "https://w3id.org/vdl#given_name": "SUSAN",
      "https://w3id.org/vdl#portrait": "/9j/4AAQSkZJRgABAQEAkACQA...gcdgck5HtRRSClooooP/2Q==",
      "https://w3id.org/vdl#birth_date": "1998-08-28",
      "https://w3id.org/vdl#issue_date": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2018-01-15T10:00:00Z"
      },
      "https://w3id.org/vdl#expiry_date": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2022-08-27T12:00:00Z"
      },
      "https://w3id.org/vdl#issuing_country": "US",
      "https://w3id.org/vdl#issuing_authority": "AL",
      "https://w3id.org/vdl#driving_privileges": {
        "@value": [
          {
            "codes": [
              {
                "code": "D"
              }
            ],
            "vehicle_category_code": "D",
            "issue_date": "2019-01-01",
            "expiry_date": "2027-01-01"
          },
          {
            "codes": [
              {
                "code": "C"
              }
            ],
            "vehicle_category_code": "C",
            "issue_date": "2019-01-01",
            "expiry_date": "2017-01-01"
          }
        ],
        "@type": "@json"
      },
      "https://w3id.org/vdl#un_distinguishing_sign": "USA"
    }
  },
  "https://w3id.org/security#proof": {
    "@graph": {
      "@type": "https://w3id.org/security#BbsBlsSignature2020",
      "http://purl.org/dc/terms/created": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2022-12-01T12:37:18Z"
      },
      "https://w3id.org/security#proofPurpose": {
        "@id": "https://w3id.org/security#assertionMethod"
      },
      "https://w3id.org/security#proofValue": "qYbNq0bTdOa+XeHJ8+vQsdFFUOxF2crZMqJsMWvLyy+wLRMm5sNelzoqgJDrFMnfXjZlTy4XBKlOh0rdQtxKRE9VHeH50eYYrXIrcrbsOhYNEyp45kkFpaFgLT5diA71qYzVYhVrzt86NCr5oWHvkg==",
      "https://w3id.org/security#verificationMethod": {
        "@id": "did:example:489398593#test"
      }
    }
  }
}

Perhaps it should be using a term for comparing https://w3id.org/vdl#driving_privileges, but that would potentially be an issue with compaction, or more likely something missing in the framing context used to do the compaction.

See digitalbazaar/jsonld.js#506.

[pasquale95]

The framing doesn't work since, for how the library is defined, the "@type" field inside the expanded json-ld document is expected to be either another object, a url or a string starting with ":_" (code here). During the expansion the type @json does not become any of these 3 options, thus the issue.
We need to handle this case by adding @json as another possible type. It will not be expanded to something like "http://www.w3.org/2001/XMLSchema#json" since it doesn't exist.
If you frame locally with the code of my PR you'll see that everything goes fine and the output is the correct framed document.

[gkellogg]

The history of @type is long and complicated. In JSON-LD, the value of @type within the body of a document is typically used for holding an IRI, either of the datatype for a value object, of the rdf:type of a node object. @json was added in JSON-LD 1.1 to be able to hold JSON literals; when serialized to RDF, it turns into the IRI http://www.w3.org/1999/02/22-rdf-syntax-ns#JSON with a canonicalized form of the JSON content as a string. The @json allows the value to be represented natively. This is discussed in the Object to RDF Conversion algorithm in the JSON-LD API spec.

An object (or array) form for @type is only allowed as part of Framing, as a frame document is intended to do some pattern matching.

(The use of blank nodes _: for values of @type is discouraged, although still legal.)

Your PR is for the jsonld.js library, and as I mentioned, I think it's an issue with that particular implementation. My implementation in Ruby seems to produce the proper output, and I believe the Framing Algorithm handles this case correctly. Given a difference in the implementations, it may be a good idea to add a minimal test to the Framing test suite, but I'll leave consideration of that to @davidlehn or someone else responsible for jsonld.js.

[pasquale95]

I tried your Ruby distiller, but when I frame I don't get the expected result. Maybe I'm using it wrong. Could you help me in achieving the proper framed result there?
I put the input JSON-LD document inside the input text field and the frame inside the frame one, then I click Submit but what I get is the same output you pasted in your previous comment (which is not the framed result).
Do I need to activate some of these flags?

[pasquale95]

Hi, any update since my last comment?

[pasquale95]

Hello again, sorry to bother you @gkellogg @davidlehn but I didn't get any news on this front since 2 weeks.
To me the Ruby distiller doesn't work as well, the frame is not created correctly even in this case. If there's some special flag I have to enable please let me know, otherwise I would conclude that the issue also propagates to this tool.

[gkellogg]

@pasquale95 Sorry, we're both quite caught up in work on other specifications at the moment, haven't been able to devote more time to your issue. When I checked, it seemed like the distiller was generating output that I would expect, but you may be trying to do something more specific. The distiller link I gave above should have the flags an options set properly to do this. The issue you're having is because using a property in the body of the frame that has "@type": "@json" defined in the frame context leads to a kind of double expansion and creates a pattern that can't be matched. The suggestion I made was to use the expanded form of the property, that has no expansion implied from a term definition and be explicit about the form in the frame.

So, you're example cited from above would be more like the following:

{
  "@context": {
    "ex": "http://example.org/vocab#",
    "ex:info": {"@type": "@json"}
  },
  "http://example.org/vocab#info": {}
}

That would match any value for "http://example.org/vocab#info" and then compact it using "ex:info", if it ends up being a JSON Literal.

If you think there's a bug in the distiller, we can take this up at https://github.com/gkellogg/rdf-distiller. So far, it seems that the spec is doing the right thing, although it may not be intuitive, but consider that a limitation of framing with JSON Literals.

[pasquale95]

I tried again and to me it looks that the Ruby distiller doesn't handle the JSON literals differently from what JSON-LD Playground does. Even with your suggestion I don't get the correct output.
If you try this input:

{
   "@context":{
      "ex":"http://example.org/vocab#",
      "ex:info":{
         "@type":"@json"
      },
      "ex:other":{
         "@type":"@json"
      }
   },
   "@id":"http://example.org/test/#library",
   "@type":"ex:Library",
   "ex:info":{
      "author":"JOHN",
      "pages":200
   },
   "ex:other":{
      "publisher":"JANE"
   }
}

with either this frame:

{
   "@context":{
      "ex":"http://example.org/vocab#",
      "ex:info":{"@type":"@json"},
      "ex:other":{"@type":"@json"}
   },
   "http://example.org/vocab#info":{}
}

or this frame:

{
  "@context": {
    "ex": "http://example.org/vocab#",
    "ex:info": {"@type": "@json"},
    "ex:other": {"@type": "@json"}
  },
  "ex:info": {}
}

The result is always the following:

{
  "@id": "http://example.org/test/#library",
  "@type": "http://example.org/vocab#Library",
  "http://example.org/vocab#info": {
    "@value": {
      "author": "JOHN",
      "pages": 200
    },
    "@type": "@json"
  },
  "http://example.org/vocab#other": {
    "@value": {
      "publisher": "JANE"
    },
    "@type": "@json"
  }
}

Thus, JSON literals are not handled correctly in both cases. I don't know about Ruby since I'm interested in the JS library, but I would like to know who I should reach out to propose my solution for the JS library. I'm writing on this thread since I thought you were the maintainers of the framing logic inside the jsonld.js lib, but probably I was wrong.
May I ask you who could I reach out?

[gkellogg]

Using the first frame, it the expanded "info" and "other" IRIs, the playground seems to get the right output:

{
  "@context": {
    "ex": "http://example.org/vocab#",
    "ex:info": {
      "@type": "@json"
    },
    "ex:other": {
      "@type": "@json"
    }
  },
  "@id": "http://example.org/test/#library",
  "@type": "ex:Library",
  "ex:info": {
    "author": "JOHN",
    "pages": 200
  },
  "ex:other": {
    "publisher": "JANE"
  }
}

The distiller doesn't properly compact to use the compact IRI representations, which I'll need to look at. This would be a compaction issue, not a framing issue. Term selection when compacting is notoriously complicated, but it should work consistently.

[pasquale95]

Why do you say that JSON-LD Playground seems to get the right output? By clicking on the link you provided I get the following result:

This result is indeed wrong, since the output is supposed to show only the ex:info field, not the entire input credential. In other words, no frame at all seems to be performed.

[gkellogg]

If the spec should be changed as suggested in digitalbazaar/jsonld.js#506, that would be non-editorial. Probably the case for matching JSON Literals is more involved and requires something more than can be described here, also considering issues with expansion and compaction of JSON literals.

The main issue is having enough bandwidth in this group to look into things deeply enough. More use cases would be helpful for JSON literals in general when it comes to both framing, expansion and compaction.