Skip to content

Add support for schema_uri validation #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions codecov.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ignore:
- docs/examples/**
113 changes: 97 additions & 16 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

This folder contains example of how to use this SDK.

> **Note** For simplicity, the code below does not include error handling. The go files in example folders includes it.

## Create a Custom CDEvent

If a tool wants to emit events that are not supported by the CDEvents specification,
Expand All @@ -17,21 +19,24 @@ happens, but CDEvents does not define any quota related subject.

```golang
type Quota struct {
User string `json:"user,omitempty"` // The use the quota applies ot
Limit string `json:"limit,omitempty"` // The limit enforced by the quota e.g. 100Gb
Current int `json:"current,omitempty"` // The current % of the quota used e.g. 90%
Threshold int `json:"threshold,omitempty"` // The threshold for warning event e.g. 85%
Level string `json:"level,omitempty"` // INFO: <threshold, WARNING: >threshold, <quota, CRITICAL: >quota
User string `json:"user,omitempty"` // The use the quota applies ot
Limit string `json:"limit,omitempty"` // The limit enforced by the quota e.g. 100Gb
Current int `json:"current,omitempty"` // The current % of the quota used e.g. 90%
Threshold int `json:"threshold,omitempty"` // The threshold for warning event e.g. 85%
Level string `json:"level,omitempty"` // INFO: <threshold, WARNING: >threshold, <quota, CRITICAL: >quota
}
```

For this scenario we will need a few imports:

```golang
import (
"context"
"fmt"
"fmt"
"log"
"os"

examples "github.com/cdevents/sdk-go/docs/examples"
cdevents "github.com/cdevents/sdk-go/pkg/api"
cdeventsv04 "github.com/cdevents/sdk-go/pkg/api/v04"
cloudevents "github.com/cloudevents/sdk-go/v2"
Expand Down Expand Up @@ -63,9 +68,7 @@ quotaRule123 := Quota{

// Create the base event
event, err := cdeventsv04.NewCustomTypeEvent()
if err != nil {
log.Fatalf("could not create a cdevent, %v", err)
}
examples.PanicOnError(err, "could not create a cdevent")
event.SetEventType(eventType)

// Set the required context fields
Expand All @@ -79,17 +82,23 @@ event.SetSubjectContent(quotaRule123)
// to the event so that the receiver may validate custom fields like
// the event type and subject content
event.SetSchemaUri("https://myregistry.dev/schemas/cdevents/quota-exceeded/0_1_0")

// The event schema needs to be loaded, so the SDK may validate
// In this example, the schema is located in the same folder as
// the go code
customSchema, err := os.ReadFile("myregistry-quotaexceeded_schema.json")
examples.PanicOnError(err, "cannot load schema file")

err = cdevents.LoadJsonSchema(customSchemaUri, customSchema)
examples.PanicOnError(err, "cannot load the custom schema file")
```

To see the event, let's render it as JSON and log it:

```golang
// Render the event as JSON
eventJson, err := cdevents.AsJsonString(event)
if err != nil {
log.Fatalf("failed to marshal the CDEvent, %v", err)
}
// Print the event
examples.PanicOnError(err, "failed to marshal the CDEvent")
fmt.Printf("%s", eventJson)
```

Expand Down Expand Up @@ -123,10 +132,11 @@ if result := c.Send(ctx, *ce); cloudevents.IsUndelivered(result) {
}
```

The whole code of is available under [`examples/custom.go`](./examples/custom.go):
The whole code of is available under [`examples/custom`](./examples/custom/main.go):

```shell
➜ go run custom.go | jq .
➜ cd examples/custom
➜ go run main.go | jq .
{
"context": {
"version": "0.4.1",
Expand All @@ -149,4 +159,75 @@ The whole code of is available under [`examples/custom.go`](./examples/custom.go
}
}
}
```
```

## Consume a CDEvent with a Custom Schema

CDEvents producers may include a `schemaUri` in their events. The extra schema **must** comply with the CDEvents schema and may add additional rules on top.
The `schemaUri` field includes the `$id` field of the custom schema and can be used for different purposes:
* specify the format of the data included in the `customData` field
* specify the format of the subject content of custom events
* refine the format of one or more fields of a specific CDEvent

In this examples, the custom schema is used to define the format of the `customData` for a `change.created` events, which corresponds to the following golang `struct`:

```golang
type ChangeData struct {
User string `json:"user"` // The user that created the PR
Assignee string `json:"assignee,omitempty"` // The user assigned to the PR (optional)
Head string `json:"head"` // The head commit (sha) of the PR
Base string `json:"base"` // The base commit (sha) for the PR
}
```

The goal of this example is to consume (parse) an event with a custom schema and validate it. In the example we load the event from disk. In real life the event will be typically received over the network or extracted from a database.

For this scenario we will need a few imports:

```golang
import (
"context"
"encoding/json"
"fmt"
"log"
"os"

examples "github.com/cdevents/sdk-go/docs/examples"
cdevents "github.com/cdevents/sdk-go/pkg/api"
cdevents04 "github.com/cdevents/sdk-go/pkg/api/v04"
cloudevents "github.com/cloudevents/sdk-go/v2"
)
```

Before parsing an event with a custom schema, it's required to load the schema into the SDK. This avoids having to download and compile the schema every time a message is parsed.

```golang
// Load and register the custom schema
customSchema, err := os.ReadFile("changecreated_schema.json")

// Unmarshal the schema to extract the $id. The $id can also be hardcoded as a const
eventAux := &struct {
Id string `json:"$id"`
}{}
err = json.Unmarshal(customSchema, eventAux)
err = cdevents.LoadJsonSchema(eventAux.Id, customSchema)
```

Once the schema is loaded, it's possible to parse the event itself.
In this case we know that the event is in the v0.4 version format, so we use the corresponding API.

```golang
// Load, unmarshal and validate the event
eventBytes, err := os.ReadFile("changecreated.json")
event, err := cdevents04.NewFromJsonBytes(eventBytes)

err = cdevent.Validate(event)
if err != nil {
log.Fatalf("cannot validate event %v: %v", event, err)
}

// Print the event
eventJson, err := cdevents.AsJsonString(event)
examples.PanicOnError(err, "failed to marshal the CDEvent")
fmt.Printf("%s\n\n", eventJson)
```
105 changes: 0 additions & 105 deletions docs/examples/custom.go

This file was deleted.

88 changes: 88 additions & 0 deletions docs/examples/custom/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
package main

import (
"context"
"fmt"
"log"
"os"

examples "github.com/cdevents/sdk-go/docs/examples"
cdevents "github.com/cdevents/sdk-go/pkg/api"
cdeventsv04 "github.com/cdevents/sdk-go/pkg/api/v04"
cloudevents "github.com/cloudevents/sdk-go/v2"
)

const customSchemaUri = "https://myregistry.dev/schemas/cdevents/quota-exceeded/0_1_0"

type Quota struct {
User string `json:"user"` // The use the quota applies ot
Limit string `json:"limit"` // The limit enforced by the quota e.g. 100Gb
Current int `json:"current"` // The current % of the quota used e.g. 90%
Threshold int `json:"threshold"` // The threshold for warning event e.g. 85%
Level string `json:"level"` // INFO: <threshold, WARNING: >threshold, <quota, CRITICAL: >quota
}

func main() {
var ce *cloudevents.Event
var c cloudevents.Client

// Define the event type
eventType := cdevents.CDEventType{
Subject: "quota",
Predicate: "exceeded",
Version: "0.1.0",
Custom: "myregistry",
}

// Define the content
quotaRule123 := Quota{
User: "heavy_user",
Limit: "50Tb",
Current: 90,
Threshold: 85,
Level: "WARNING",
}

// Create the base event
event, err := cdeventsv04.NewCustomTypeEvent()
examples.PanicOnError(err, "could not create a cdevent")
event.SetEventType(eventType)

// Set the required context fields
event.SetSubjectId("quotaRule123")
event.SetSource("my/first/cdevent/program")

// Set the required subject fields
event.SetSubjectContent(quotaRule123)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really should look into optional parameters at some point. Makes using things a little easier.

event, err := cdeventsv04.NewCustomTypeEvent(cdevents.CustomTypeEventOptions.WithSchemaURI("my-schema"))

This adds some flexibility on where we can add validation as well, as in we could choose to do it upon creating the struct with a cdevents.WithQuickValidate() or something as an option

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was debating between explicit setters/getters and providing some kind of context object where users could stash fields to be set:
event.WithSubjectId("foo").WithSource("bar")

But that leads to as many WithX methods as SetX methods to be defined, and I didn't really see an advantage. For the subject content, I introduced SetSubjectContent which allows setting the whole content from a Struct directly rather than field by field.

I'd like to better understand what you are proposing here, maybe something for the next WG.

event.SetSchemaUri(customSchemaUri)

// Print the event
eventJson, err := cdevents.AsJsonString(event)
examples.PanicOnError(err, "failed to marshal the CDEvent")
fmt.Printf("%s", eventJson)

// To validate the event, we need to load its custom schema
customSchema, err := os.ReadFile("myregistry-quotaexceeded_schema.json")
examples.PanicOnError(err, "cannot load schema file")

err = cdevents.LoadJsonSchema(customSchemaUri, customSchema)
examples.PanicOnError(err, "cannot load the custom schema file")

ce, err = cdevents.AsCloudEvent(event)
examples.PanicOnError(err, "failed to create cloudevent")

// Set send options
source, err := examples.CreateSmeeChannel()
examples.PanicOnError(err, "failed to create a smee channel")
ctx := cloudevents.ContextWithTarget(context.Background(), *source)
ctx = cloudevents.WithEncodingBinary(ctx)

c, err = cloudevents.NewClientHTTP()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we wrap the cloudevent library to hide all this from the user? This example would then just be something like

client, err := cdevents.NewHTTPClient()
if err != nil {
    panic(err)
}

result, err := client.Send(event)

This becomes much simpler I feel like

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is on purpose - the CloudEvents API is rich, as it supports multiple transport options each with its own configuration capabilities. I don't think we should replicate that on CDEvents side, transport is not CDEvent's concern.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrapping it does not mean a user cannot pass their own cloudevent client in. That can be done via optional params. This is standard in a lot of go sdks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that:

client, err := cdevents.NewClientCloudEventsHTTP()

is going to provide a better user experience than:

client, err := cloudevents.NewClientHTTP()

The only difference is the import of cloudevents, which is required anyway unless we wrap the IsUndelivered method and others that may be required.

Copy link

@xibz xibz Jul 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where my experience in SDKs come in. If you wrap it, you have full control on what it looks like to the user. You can add hooks, etc. With your current implementation you cannot without fully understanding two SDKs. Two! That's a bad experience

examples.PanicOnError(err, "failed to create the CloudEvents client")

// Send the CloudEvent
// c is a CloudEvent client
if result := c.Send(ctx, *ce); cloudevents.IsUndelivered(result) {
log.Fatalf("failed to send, %v", result)
}
}
Loading