Skip to content

Add new Datasets tab details, create map fields #264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions apl/data-types/map-fields.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,14 @@ Support for creating your own map fields is coming in early 2025. To express int

## Benefits and drawbacks of map fields

The benefit of map fields is that you can store additional attributes without adding more fields. This is particularly useful when the shape of your data is unpredictable (for example, additional attributes added by OpenTelemetry instrumentation). Using map fields means that you can avoid reaching the field limit of a dataset.
Map fields help you manage high-cardinality data by storing multiple key-value pairs within a single field. One of the benefits of map fields is that you can store additional attributes without adding more fields. This is particularly useful when the shape of your data is unpredictable (for example, additional attributes added by OpenTelemetry instrumentation). Using map fields means that you can avoid reaching the field limit of a dataset.

The drawbacks of map fields are the following:
Use map fields in the following cases:
- You approach the dataset field limit.
- The shape of your data is unpredictable. For example, an OpenTelemetry instrumentation or another SDK creates objects with many keys.
- You work with feature flags or custom attributes that generate many fields.

Map fields reduce impact on field limits, but involve trade-offs in query efficiency and compression. The drawbacks of map fields are the following:
- Querying map fields uses more query-hours than querying conventional fields.
- Map fields don’t compress as well as conventional fields. This means datasets with map fields use more storage.
- You don’t have visibility into map fields from the schema. For example, autocomplete doesn’t know the properties inside the map field.
Expand Down Expand Up @@ -47,4 +52,25 @@ If an entity name has spaces (` `), dots (`.`), or dashes (`-`), you can only us
- `where ['map.field']['property.name1']['property.name2'] == 14`
- `where ['map.field'].property1.property2 == 14`

For more information, see [Entity names](/apl/entities/entity-names#quote-identifiers).
For more information, see [Entity names](/apl/entities/entity-names#quote-identifiers).

## Create map fields

To create a map field:

1. Go to the Datasets tab.
1. Select the dataset where you want to create the map field.
1. In the top right of the fields list, click <img src="/doc-assets/icons/ellipsis-vertical.svg" className="inline-icon" alt="Vertical ellipsis icon" /> **More > Create map field**.
1. In **Field name**, enter the full name of the field, including parent fields, if any. For example, `map_field_name`. For more information on syntax, see [Access properties of nested maps](#access-properties-of-nested-maps)
1. Click **Create map field**.

## View map fields

To view map fields:
1. Go to the Datasets tab.
1. Select a dataset where you want to view map fields.
1. Map fields are labelled in the following way:

- **MAPPED** means that the field was previously an ordinary field but at some point its parent was changed to a map field. Axiom adds new events to the field as an attribute of the parent map field. Events you ingested before the change retain the ordinary structure.
- **UNUSED** means that the field is configured as a map field but you haven’t yet ingested data into it. Once ingested, data within this field won’t count toward your field limit.
- **REMOVED** means that the field was configured as a map field but at some point it was changed to an ordinary field. Axiom adds new events to the field as usual. Events you ingested before the change retain the map structure. To fully remove this field, first [trim your dataset](/reference/datasets#trim-dataset) to remove the time period when map data was ingested, and then [vacuum the fields](/reference/datasets#vacuum-fields).
Binary file removed doc-assets/shots/datasets-cribl.png
Binary file not shown.
63 changes: 31 additions & 32 deletions query-data/datasets.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,9 @@ In Axiom, an individual piece of data is an event, and a dataset is a collection

## Datasets overview

When you open the Datasets tab, you see the list of datasets on the left. To explore the fields in a dataset, select the dataset from the list on the left.
When you open the Datasets tab, you see the list of datasets. To explore the fields in a dataset, select the dataset from the list on the left.

On the right, you see the following:
- The list of integration dashboards appears on the Datasets overview page. These are prebuilt dashboards automatically generated by Axiom to enhance your experience. For more information, see [Apps](/apps).
- The list of [starred queries](#starred-queries)
- The [query history](#query-history)

## Fields list

When you select a dataset, Axiom displays the list of fields within the dataset.

The field types are the following:
When you select a dataset, Axiom displays the list of fields within the dataset on the left. The field types are the following:
- String
- Number
- Boolean
Expand All @@ -31,12 +22,19 @@ The field types are the following:

This view flattens field names with dot notation. This means that the event `{"foo": { "bar": "baz" }}` appears as `foo.bar`. Field names containing periods (`.`) are folded.

On the right, you see the following:
- [Starred queries](#starred-queries)
- [Query history](#query-history)

### Edit field

Click the field name to change the following:
- Change the field description.
- Change the field unit. This is only available for number field types.
- Hide the field. This means that the field is still present in the underlying Axiom database, but it doesn’t appear in the Axiom UI. Use this option if you sent the field to Axiom by mistake or you don’t want to use it anymore in Axiom.
To edit a field:

1. To the right of the field, click <img src="/doc-assets/icons/ellipsis-vertical.svg" className="inline-icon" alt="Vertical ellipsis icon" /> **More > Manage field**.
1. Edit the following:
- Field description.
- Field unit. This is only available for number field types.
- Hide the field. This means that the field is still present in the underlying Axiom database, but it doesn’t appear in the Axiom UI. Use this option if you sent the field to Axiom by mistake or you don’t want to use it anymore in Axiom.

## Quick charts

Expand All @@ -47,33 +45,34 @@ Quick charts allow fast charting of fields depending on their field type. For ex

## Virtual fields

Virtual fields are powerful expressions that run on every event during a query to create new fields. The virtual fields are calculated from the events in the query using an APL expression. They’re similar to tools like derived columns in other products but super-charged with an expressive interpreter and with the flexibility to add, edit, or remove them any time.
Virtual fields are powerful expressions that run on every event during a query to create new fields. The virtual fields are calculated from the events in the query using an APL expression. They’re similar to tools like derived columns in other products but super-charged with an expressive interpreter and with the flexibility to add, edit, or remove them at any time.

To manage a dataset’s virtual fields, click <img src="/doc-assets/icons/virtual-fields.svg" className="inline-icon" alt="Virtual fields icon" /> in the toolbar.

For more information, see [Virtual fields](/query-data/virtual-fields).

## Map fields

Map fields are a special type of field that can hold a collection of nested key-value pairs within a single field. You can think of the content of a map field as a JSON object. The Dataset tab enables you to create map fields, and view unused and removed map fields. For more information, see [Map fields](/apl/data-types/map-fields#create-map-fields).

## Queries

Every query has a unique ID that you can save and share with your team members. The Datasets tab allows you to do the following:
- Find a past query.
- Run previously saved queries.
- Star a query so that you and your team members can easily find it in the future.
- Browse previous queries and find a past query.

### Recent queries

To find and run recent queries:
1. Click **Query library** in the toolbar.
1. Click the **Recent** tab.
1. Optional: In the top right, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
1. Find the query in the list, and then click it to run the query.
### Starred queries

### Saved queries
To find and run previously starred queries:

To find and run previously saved queries:
1. Click **Query library** in the toolbar.
1. Click the **Saved** tab.
1. Optional: In the top right, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
1. Select a dataset.
1. Optional: In the top right of the **Starred queries** section, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
1. Find the query in the list, and then click it to run the query.

### Starred queries
### Query history

To find and run recent queries:

In the **Starred queries** section on the right, you see queries saved for future use. They’re great for keeping a list of useful queries for a dataset. All starred queries are shared with your team.
1. Select a dataset.
1. Optional: In the top right of the **Query history** section, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
1. Find the query in the list, and then click it to run the query.
2 changes: 1 addition & 1 deletion send-data/aws-iot-rules.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,4 +80,4 @@ In **Rule actions**, select the action to send a message to a Lambda function, a

Use the AWS IoT Console, AWS CLI, or an MQTT client to publish messages to the topic that matches your rule. For example, `iot/topic`.

In Axiom, go to the Datasets tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
In Axiom, go to the Stream tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
2 changes: 1 addition & 1 deletion send-data/aws-s3.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -128,4 +128,4 @@ Ensure the log file you upload to the S3 bucket is in the correct format, such a
]
```

After uploading a test log file to your S3 bucket, the Lambda function automatically processes the log data and sends it to Axiom. In Axiom, go to the Datasets tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
After uploading a test log file to your S3 bucket, the Lambda function automatically processes the log data and sends it to Axiom. In Axiom, go to the Stream tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
6 changes: 1 addition & 5 deletions send-data/cribl.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,7 @@ Open Cribl’s UI and navigate to **Destinations > HTTP**. Click on `+` Add New

- **Name:** Choose a name for the destination.

- In the Axiom UI, click the Datasets tab and create your dataset by entering its name and description.

<Frame caption="Auth overview">
<img src="/doc-assets/shots/datasets-cribl.png" alt="Auth overview" />
</Frame>
- In the Axiom UI, [create a dataset](/reference/datasets#create-dataset).

- **Endpoint URL:** Input the URL of your Axiom log ingest endpoint. This should be something like `https://api.axiom.co/v1/datasets/DATASET_NAME/ingest`. Replace `DATASET_NAME` with the name of your dataset.

Expand Down