Add new Datasets tab details, create map fields (#264)

Co-authored-by: Christopher Ehrlich <[email protected]>

Author: manototh

apl/data-types/map-fields.mdx

@@ -14,9 +14,14 @@ Support for creating your own map fields is coming in early 2025. To express int
 
 ## Benefits and drawbacks of map fields
 
-The benefit of map fields is that you can store additional attributes without adding more fields. This is particularly useful when the shape of your data is unpredictable (for example, additional attributes added by OpenTelemetry instrumentation). Using map fields means that you can avoid reaching the field limit of a dataset.
+Map fields help you manage high-cardinality data by storing multiple key-value pairs within a single field. One of the benefits of map fields is that you can store additional attributes without adding more fields. This is particularly useful when the shape of your data is unpredictable (for example, additional attributes added by OpenTelemetry instrumentation). Using map fields means that you can avoid reaching the field limit of a dataset.
 
-The drawbacks of map fields are the following:
+Use map fields in the following cases:
+- You approach the dataset field limit.
+- The shape of your data is unpredictable. For example, an OpenTelemetry instrumentation or another SDK creates objects with many keys.
+- You work with feature flags or custom attributes that generate many fields.
+
+Map fields reduce impact on field limits, but involve trade-offs in query efficiency and compression. The drawbacks of map fields are the following:
 - Querying map fields uses more query-hours than querying conventional fields.
 - Map fields don’t compress as well as conventional fields. This means datasets with map fields use more storage.
 - You don’t have visibility into map fields from the schema. For example, autocomplete doesn’t know the properties inside the map field.
@@ -47,4 +52,25 @@ If an entity name has spaces (` `), dots (`.`), or dashes (`-`), you can only us
 - `where ['map.field']['property.name1']['property.name2'] == 14`
 - `where ['map.field'].property1.property2 == 14`
 
-For more information, see [Entity names](/apl/entities/entity-names#quote-identifiers).
+For more information, see [Entity names](/apl/entities/entity-names#quote-identifiers).
+
+## Create map fields
+
+To create a map field:
+
+1. Go to the Datasets tab.
+1. Select the dataset where you want to create the map field.
+1. In the top right of the fields list, click <img src="/doc-assets/icons/ellipsis-vertical.svg" className="inline-icon" alt="Vertical ellipsis icon" /> **More > Create map field**.
+1. In **Field name**, enter the full name of the field, including parent fields, if any. For example, `map_field_name`. For more information on syntax, see [Access properties of nested maps](#access-properties-of-nested-maps)
+1. Click **Create map field**.
+
+## View map fields
+
+To view map fields:
+1. Go to the Datasets tab.
+1. Select a dataset where you want to view map fields.
+1. Map fields are labelled in the following way:
+
+- **MAPPED** means that the field was previously an ordinary field but at some point its parent was changed to a map field. Axiom adds new events to the field as an attribute of the parent map field. Events you ingested before the change retain the ordinary structure.
+- **UNUSED** means that the field is configured as a map field but you haven’t yet ingested data into it. Once ingested, data within this field won’t count toward your field limit.
+- **REMOVED** means that the field was configured as a map field but at some point it was changed to an ordinary field. Axiom adds new events to the field as usual. Events you ingested before the change retain the map structure. To fully remove this field, first [trim your dataset](/reference/datasets#trim-dataset) to remove the time period when map data was ingested, and then [vacuum the fields](/reference/datasets#vacuum-fields).

doc-assets/shots/datasets-cribl.png

@@ -11,18 +11,9 @@ In Axiom, an individual piece of data is an event, and a dataset is a collection
 
 ## Datasets overview
 
-When you open the Datasets tab, you see the list of datasets on the left. To explore the fields in a dataset, select the dataset from the list on the left.
+When you open the Datasets tab, you see the list of datasets. To explore the fields in a dataset, select the dataset from the list on the left.
 
-On the right, you see the following:
-- The list of integration dashboards appears on the Datasets overview page. These are prebuilt dashboards automatically generated by Axiom to enhance your experience. For more information, see [Apps](/apps).
-- The list of [starred queries](#starred-queries)
-- The [query history](#query-history)
-
-## Fields list
-
-When you select a dataset, Axiom displays the list of fields within the dataset.
-
-The field types are the following:
+When you select a dataset, Axiom displays the list of fields within the dataset on the left. The field types are the following:
 - String
 - Number
 - Boolean
@@ -31,12 +22,19 @@ The field types are the following:
 
 This view flattens field names with dot notation. This means that the event `{"foo": { "bar": "baz" }}` appears as `foo.bar`. Field names containing periods (`.`) are folded.
 
+On the right, you see the following:
+- [Starred queries](#starred-queries)
+- [Query history](#query-history)
+
 ### Edit field
 
-Click the field name to change the following:
-- Change the field description.
-- Change the field unit. This is only available for number field types.
-- Hide the field. This means that the field is still present in the underlying Axiom database, but it doesn’t appear in the Axiom UI. Use this option if you sent the field to Axiom by mistake or you don’t want to use it anymore in Axiom.
+To edit a field:
+
+1. To the right of the field, click <img src="/doc-assets/icons/ellipsis-vertical.svg" className="inline-icon" alt="Vertical ellipsis icon" /> **More > Manage field**.
+1. Edit the following:
+    - Field description.
+    - Field unit. This is only available for number field types.
+    - Hide the field. This means that the field is still present in the underlying Axiom database, but it doesn’t appear in the Axiom UI. Use this option if you sent the field to Axiom by mistake or you don’t want to use it anymore in Axiom.
 
 ## Quick charts
 
@@ -47,33 +45,34 @@ Quick charts allow fast charting of fields depending on their field type. For ex
 
 ## Virtual fields
 
-Virtual fields are powerful expressions that run on every event during a query to create new fields. The virtual fields are calculated from the events in the query using an APL expression. They’re similar to tools like derived columns in other products but super-charged with an expressive interpreter and with the flexibility to add, edit, or remove them any time.
+Virtual fields are powerful expressions that run on every event during a query to create new fields. The virtual fields are calculated from the events in the query using an APL expression. They’re similar to tools like derived columns in other products but super-charged with an expressive interpreter and with the flexibility to add, edit, or remove them at any time.
 
 To manage a dataset’s virtual fields, click <img src="/doc-assets/icons/virtual-fields.svg" className="inline-icon" alt="Virtual fields icon" /> in the toolbar.
 
+For more information, see [Virtual fields](/query-data/virtual-fields).
+
+## Map fields
+
+Map fields are a special type of field that can hold a collection of nested key-value pairs within a single field. You can think of the content of a map field as a JSON object. The Dataset tab enables you to create map fields, and view unused and removed map fields. For more information, see [Map fields](/apl/data-types/map-fields#create-map-fields).
+
 ## Queries
 
 Every query has a unique ID that you can save and share with your team members. The Datasets tab allows you to do the following:
-- Find a past query.
-- Run previously saved queries.
 - Star a query so that you and your team members can easily find it in the future.
+- Browse previous queries and find a past query.
 
-### Recent queries
-
-To find and run recent queries:
-1. Click **Query library** in the toolbar.
-1. Click the **Recent** tab.
-1. Optional: In the top right, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
-1. Find the query in the list, and then click it to run the query.
+### Starred queries
 
-### Saved queries
+To find and run previously starred queries:
 
-To find and run previously saved queries:
-1. Click **Query library** in the toolbar.
-1. Click the **Saved** tab.
-1. Optional: In the top right, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
+1. Select a dataset.
+1. Optional: In the top right of the **Starred queries** section, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
 1. Find the query in the list, and then click it to run the query.
 
-### Starred queries
+### Query history
+
+To find and run recent queries:
 
-In the **Starred queries** section on the right, you see queries saved for future use. They’re great for keeping a list of useful queries for a dataset. All starred queries are shared with your team.
+1. Select a dataset.
+1. Optional: In the top right of the **Query history** section, select whether to display <img src="/doc-assets/icons/single-user.svg" className="inline-icon" alt="Single user icon" /> your queries or <img src="/doc-assets/icons/many-users.svg" className="inline-icon" alt="Many users icon" /> your team’s queries.
+1. Find the query in the list, and then click it to run the query.

query-data/datasets.mdx

@@ -80,4 +80,4 @@ In **Rule actions**, select the action to send a message to a Lambda function, a
 
 Use the AWS IoT Console, AWS CLI, or an MQTT client to publish messages to the topic that matches your rule. For example, `iot/topic`.
 
-In Axiom, go to the Datasets tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
+In Axiom, go to the Stream tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.

send-data/aws-iot-rules.mdx

@@ -128,4 +128,4 @@ Ensure the log file you upload to the S3 bucket is in the correct format, such a
  ]
 ```
 
-After uploading a test log file to your S3 bucket, the Lambda function automatically processes the log data and sends it to Axiom. In Axiom, go to the Datasets tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
+After uploading a test log file to your S3 bucket, the Lambda function automatically processes the log data and sends it to Axiom. In Axiom, go to the Stream tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.

send-data/aws-s3.mdx

@@ -29,11 +29,7 @@ Open Cribl’s UI and navigate to **Destinations > HTTP**. Click on `+` Add New
 
 - **Name:** Choose a name for the destination.
 
-- In the Axiom UI, click the Datasets tab and create your dataset by entering its name and description.
-
-<Frame caption="Auth overview">
-  <img src="/doc-assets/shots/datasets-cribl.png" alt="Auth overview" />
-</Frame>
+- In the Axiom UI, [create a dataset](/reference/datasets#create-dataset).
 
 - **Endpoint URL:** Input the URL of your Axiom log ingest endpoint. This should be something like `https://api.axiom.co/v1/datasets/DATASET_NAME/ingest`. Replace `DATASET_NAME` with the name of your dataset.