Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sumologic sample using JSON requests #63

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Sumologic sample using JSON requests #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83505b8
SumoLogic contribution
agregory999 Mar 30, 2023
ca4b0a5
Update main README
agregory999 Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ This repository provides examples demonstrating how to use Oracle Functions.
| _**Logging >> Service Connector Hub >> Functions**_ | |
| Move logs from OCI to Datadog using Service Connector Hub, Logging (Source), Functions (Target) and Datadog | [sample](./samples/oci-logs-datadog) |
| Send SMS messages for logs using Service Connector Hub, Logging (Source), Functions (Target) and Syniverse SMS | [sample](./samples/oci-notification-syniverse) |
| Move Audit, Logs, and Events from OCI to SumoLogic using Service Connector Hub, Logging (Source), Functions (Target) and HTTPS | [sample](./samples/oci-serviceconnector-sumologic-python/) |
| | |
| _**Streaming >> Service Connector Hub >> Functions**_ | |
| Convert JSON to CSV format using Service Connector Hub, Streams (Source and Target) and Functions (Task) | [sample](./samples/oci-serviceconnector-streaming-json-to-csv-python) |
Expand Down
116 changes: 116 additions & 0 deletions samples/oci-serviceconnector-sumologic-python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Send OCI Events and Audit to SumoLogic via HTTPS

This is a sample to use a Service Connector Hub with Function as a target. This function will take a single JSON or list of JSON objects, and write them to a configured SumoLogic endpoint.

This sample is based on the [documented SumoLogic Connector](https://help.sumologic.com/docs/send-data/collect-from-other-data-sources/collect-logs-oracle-cloud-infrastructure/) in OCI Docs, but with a couple of minor differences:
* It sends JSON documents instead of text
* Debugging is an option

As you make your way through this tutorial, look out for this icon ![user input icon](./images/userinput.png).
Whenever you see it, it's time for you to perform an action.


## Prerequisites

Before you deploy this sample function, make sure you have run steps A, B
and C of the [Oracle Functions Quick Start Guide for Cloud Shell](https://www.oracle.com/webfolder/technetwork/tutorials/infographics/oci_functions_cloudshell_quickview/functions_quickview_top/functions_quickview/index.html)
* A - Set up your tenancy
* B - Create application
* C - Set up your Cloud Shell dev environment


## List Applications

Assuming you have successfully completed the prerequisites, you should see your
application in the list of applications.

```
fn ls apps
```


## Review and customize the function

Review the following files in the current folder:
* the code of the function, [func.py](./func.py)
* its dependencies, [requirements.txt](./requirements.txt)
* the function metadata, [func.yaml](./func.yaml)


### Configure SumoLogic Endpoint

Within the `func.yaml` file, there is a configuration parameter called `SUMOLOGIC_ENDPOINT`. This must be set either prior to deployment, or after deployment using the UI/CLI.

See [this link](https://help.sumologic.com/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-logs/) how to generate this URL, and for additional headers and changes that could be made to this function.

The function here also allows a configuration called `DEBUG` - set this to true if additional output is desired.


## Deploy the function

In Cloud Shell, run the `fn deploy` command to build *this* function and its dependencies as a Docker image,
push the image to the specified Docker registry, and deploy *this* function to Oracle Functions
in the application created earlier:

![user input icon](./images/userinput.png)
```
fn -v deploy --app <app-name>
```
e.g.,
```
fn -v deploy --app myapp
```

### Test

In Cloud Shell, run the `fn invoke` ccommand to unit test this function:

![user input icon](./images/userinput.png)
```
fn invoke <app-name> <function-name> < test.json
```
e.g.,
```
fn invoke myapp oci-serviceconnector-sumologic-python < examples-json/audit.json
```

You should see no error or output returned by the function. Function Logs will show a record of the call, and if `DEBUG` was enabled, then more will be shown.


## Create a Service Connector for OCI Audit Logging

1. From the navigation menu, select **Logging**, and then select **Service Connectors**.

2. Click Create Connector, add a Name, Description, select the compartment, select the Source as **Logging** and Target as **Functions** with no Task.

3. On Configure Source connection, select the compartment, select the _Audit log with sub-compartments enabled. (Doing so at the tenancy root will send ALL audit to SumoLogic)

4. On Configure Target connection, select the Functions compartment, the Functions App, and the `sumologicfn` Function. If prompted to create a policy, click Create.

### Screeen Shots

Service Connector Source
![Audit Log SCH](./images/ServiceConnectorAuditLog.png)

Service Connector Target
![Functions Target SCH](./images/ServiceConnectorFunctionTarget.png)

### OCI Events

This function can support OCI Events or multiple logs that are not audit. The following screen shot shows a couple OCI Events for Cloud Guard going to the same function:

![Events to Function](./images/EventsToFunction.png)

### Logging Filter

Service Connectors support Log filters as well. In order to limit the logs sent to SumoLogic, the following example filter could be applied, which will only look for POST or PUT events, and prevent the listed message types from being sent at all.

```
search "ocid1.compartment.oc1..xxxxx/_Audit_Include_Subcompartment" | (data.request.action='POST' or data.request.action='PUT') and (type!='io.k8s.coordination.v1.leases.update') and (type!='io.k8s.authentication.v1.tokenreviews.create') and (type!='com.oraclecloud.management-agent.ExchangeWorkSubmission')
```

## Monitoring Functions and Service Connector

Make sure you configure basic observability for your function and connector using metrics, alarms and email alerts:
* [Basic Guidance for Monitoring your Functions](../basic-observability/functions.md)
* [Basic Guidance for Monitoring your Service Connector](../basic-observability/service-connector-hub.md)
225 changes: 225 additions & 0 deletions samples/oci-serviceconnector-sumologic-python/examples-json/audit.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
[
{
"data": {
"additionalDetails": {
"X-Real-Port": 19991
},
"availabilityDomain": "AD3",
"compartmentId": "ocid1.compartment.oc1..xxx",
"compartmentName": "Network",
"definedTags": null,
"eventGroupingId": "/xxx",
"eventName": "ListPrivateIps",
"freeformTags": null,
"identity": {
"authType": "instance",
"callerId": null,
"callerName": null,
"consoleSessionId": null,
"credentials": null,
"ipAddress": "xx.xx.xx.xx",
"principalId": "ocid1.instance.oc1.iad.xxx",
"principalName": null,
"tenantId": "ocid1.tenancy.oc1..xxx",
"userAgent": null
},
"message": "ListPrivateIps succeeded",
"request": {
"action": "GET",
"headers": {
"Accept": [
"*/*"
],
"Authorization": [
"Bearer yyy.zzz.rpVTVki1Fo-aaa"
],
"Date": [
"Fri, 17 Mar 2023 12:57:44 GMT"
]
},
"id": "/xxx/3ABE1F95B35432E3683550035916DA4D",
"parameters": {
"vnicId": [
"ocid1.vnic.oc1.iad.xxx"
]
},
"path": "/20160918/privateIps"
},
"resourceId": null,
"response": {
"headers": {
"Connection": [
"close"
],
"Content-Length": [
"1446"
],
"Content-Type": [
"application/json"
],
"Date": [
"Fri, 17 Mar 2023 12:57:44 GMT"
],
"X-Content-Type-Options": [
"nosniff"
],
"opc-request-id": [
"/xx/yy"
]
},
"message": null,
"payload": {},
"responseTime": "2023-03-17T12:57:44.914Z",
"status": "200"
},
"stateChange": {
"current": null,
"previous": null
}
},
"dataschema": "2.0",
"id": "45f1403b-8909-40df-aa69-769f867a30df",
"oracle": {
"compartmentid": "ocid1.compartment.oc1..xx",
"ingestedtime": "2023-03-17T12:57:45.842Z",
"loggroupid": "_Audit",
"tenantid": "ocid1.tenancy.oc1..xx"
},
"source": "",
"specversion": "1.0",
"time": "2023-03-17T12:57:44.914Z",
"type": "com.oraclecloud.virtualNetwork.ListPrivateIps"
},
{
"data": {
"additionalDetails": {
"X-Real-Port": 52346
},
"availabilityDomain": "AD3",
"compartmentId": "ocid1.compartment.oc1..xx",
"compartmentName": "Network",
"definedTags": null,
"eventGroupingId": "sch-7v3vazqt2ohujqmxuizhbg7dfcq/xx",
"eventName": "UploadOCSFile",
"freeformTags": null,
"identity": {
"authType": "resource",
"callerId": null,
"callerName": null,
"consoleSessionId": null,
"credentials": "***",
"ipAddress": "10.1.8.240",
"principalId": "ocid1.serviceconnector.oc1.iad.xx",
"principalName": null,
"tenantId": "ocid1.tenancy.oc1..xx",
"userAgent": "Oracle-JavaSDK/2.46.0 (Linux/4.14.35-2047.521.4.el7uek.x86_64; Java/1.8.0_311; Java HotSpot(TM) 64-Bit Server VM GraalVM EE 20.3.4/25.311-b11-jvmci-20.3-b24)"
},
"message": "UploadOCSFile succeeded",
"request": {
"action": "POST",
"headers": {
"Accept": [
"application/json"
],
"Authorization": [
"Signature ***"
],
"Connection": [
"keep-alive"
],
"Content-Length": [
"10452526"
],
"Content-Type": [
"application/json"
],
"Date": [
"Fri, 17 Mar 2023 12:57:42 GMT"
],
"User-Agent": [
"Oracle-JavaSDK/2.46.0 (Linux/4.14.35-2047.521.4.el7uek.x86_64; Java/1.8.0_311; Java HotSpot(TM) 64-Bit Server VM GraalVM EE 20.3.4/25.311-b11-jvmci-20.3-b24)"
],
"opc-client-info": [
"Oracle-JavaSDK/2.46.0"
],
"opc-request-id": [
"sch-7v3vazqt2oujqmxuiwzhbg7dfcq"
],
"opc-retry-token": [
"69ab78a0-82a6-4ab-8f63-6c83c775e194"
],
"x-content-sha256": [
"NwJ8bFtpyzoEg3lKe8yGsRHkMEkrV7P2UFqowRds8U="
]
},
"id": "sch-7v3vazqt2ohujqmxuiwzhbg7dfcq/yy/zz",
"parameters": {
"logGroupId": [
"ocid1.loganalyticsloggroup.oc1.iad.xx"
],
"serviceConnectorCompartmentId": [
"ocid1.compartment.oc1..xx"
],
"serviceConnectorId": [
"ocid1.serviceconnector.oc1.iad.xx"
],
"serviceConnectorSource": [
"LOG"
],
"serviceConnectorSourceId": [
"null"
]
},
"path": "/20200601/actions/uploadOCSFile"
},
"resourceId": "uploadOCSFile",
"response": {
"headers": {
"Content-Length": [
"48"
],
"Content-Type": [
"application/json"
],
"Date": [
"Fri, 17 Mar 2023 12:57:42 GMT"
],
"X-Content-Type-Options": [
"nosniff"
],
"opc-content-md5": [
"b8ymx7ezMarR/7wFJq/rNg=="
],
"opc-object-id": [
"8a04e7ec-95b1-4a30-8e4f-76506f1ce3cc"
],
"opc-request-id": [
"sch-xx/xx/yy"
]
},
"message": null,
"payload": {},
"responseTime": "2023-03-17T12:57:43.674Z",
"status": "200"
},
"stateChange": {
"current": {
"timeCreated": "2023-03-17T12:57:43.000Z"
},
"previous": {}
}
},
"dataschema": "2.0",
"id": "82bc9a7c-876c-424b-b732-a6c19ca73dc9",
"oracle": {
"compartmentid": "ocid1.compartment.oc1..xx",
"ingestedtime": "2023-03-17T12:57:53.467Z",
"loggroupid": "_Audit",
"tenantid": "ocid1.tenancy.oc1..xx"
},
"source": "",
"specversion": "1.0",
"time": "2023-03-17T12:57:43.674Z",
"type": "com.oraclecloud.LoggingAnalytics.UploadOCSFile"
}
]
44 changes: 44 additions & 0 deletions samples/oci-serviceconnector-sumologic-python/examples-json/cloudguard-example.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
{
"eventType": "com.oraclecloud.cloudguard.problemdetected",
"cloudEventsVersion": "0.1",
"eventTypeVersion": "2.0",
"source": "CloudGuardResponderEngine",
"eventTime": "2023-03-21T22:23:22Z",
"contentType": "application/json",
"data": {
"compartmentId": "ocid1.compartment.oc1..xx",
"compartmentName": "CIS235-appdev-cmp",
"resourceName": "Instance terminated",
"resourceId": "ocid1.cloudguardproblem.oc1.iad.xx",
"additionalDetails": {
"tenantId": "ocid1.tenancy.oc1..xx",
"status": "OPEN",
"reason": "New Problem detected by CloudGuard",
"problemName": "INSTANCE_TERMINATED",
"riskLevel": "HIGH",
"problemType": "ACTIVITY",
"resourceName": "[email protected]",
"resourceId": "ocid1.saml2idp.oc1..xx/[email protected]",
"resourceType": "User",
"targetId": "ocid1.cloudguardtarget.oc1.iad.xx",
"labels": "Compute",
"firstDetected": "2023-03-21T22:22:41.812Z",
"lastDetected": "2023-03-21T22:22:41.812Z",
"region": "us-ashburn-1",
"impactedResourceName": "wls-ucm8",
"impactedResourceId": "ocid1.instance.oc1.iad.xx",
"impactedResourceType": "Instance",
"problemAdditionalDetails": {
"imageId": "ocid1.image.oc1..xx",
"shape": "VM.Standard.E4.Flex",
"type": "CustomerVmi"
},
"problemDescription": "Compute instances may deliver critical functions. ",
"problemRecommendation": "Ensure that the termination of the compute instance is sanctioned and performed by a permitted administrator."
}
},
"eventID": "7c9615da-209d-4cd5-96b1-c11354709cc3",
"extensions": {
"compartmentId": "ocid1.compartment.oc1..xx"
}
}
Loading