Update map fields #304
Update map fields #304
Conversation
apl/data-types/map-fields.mdx
Outdated
| ## Map fields and flattened fields | ||
|
|
||
| The parent map field and its flattened subfields can appear in the same dataset. For example, ingesting the following JSON adds a map field (`geo`) and two flattened fields (`geo.city`, `geo.country`) to the dataset. Axiom treats them as separate fields and doesn’t maintain a relationship between them. |
There was a problem hiding this comment.
ingesting this doesn't create a map field; just the two flattened fields. to create a map field, you can do so via the api or now, in app.
the point about axiom treating them differently is if you have a map field that shares the same keys/values as a flattened version. theres no conflicts there, like you mentioned
There was a problem hiding this comment.
yes ollie is right about this
the steps to end up in this situation are:
geois not a map field- ingest
{
"geo": {
"city": "Paris",
"country": "France"
}
}this gets put into geo.city and geo.country
- make
geoa map field - ingest the same
{
"geo": {
"city": "Paris",
"country": "France"
}
}this gets put into geo
(or in opposite order)
apl/data-types/map-fields.mdx
Outdated
|
|
||
| - Projecting away all subfields (in this example, `geo.city` and `geo.country`) doesn’t remove the parent map field `geo` unless it is explicitly projected away. | ||
| - Fields metadata may still include entries for projected-away fields if the map field is preserved. |
There was a problem hiding this comment.
i wouldn't note this, those are implementation details i needed at the time but customers dont see it
There was a problem hiding this comment.
i agree, the information below about `['geo']['city'] being different from ['geo.city'] is enough here. imo the above only makes it more confusing to someone who is reading about this for the first time.
apl/data-types/map-fields.mdx
Outdated
|
|
||
| Support for creating your own map fields is coming in early 2025. To express interest in the feature, [contact Axiom](https://axiom.co/contact). | ||
| </Note> | ||
| Axiom automatically creates map fields in datasets that use [OpenTelemetry](/send-data/opentelemetry) and you can create them yourself. |
There was a problem hiding this comment.
| Axiom automatically creates map fields in datasets that use [OpenTelemetry](/send-data/opentelemetry) and you can create them yourself. | |
| Axiom automatically creates the `attributes.custom` map field in datasets that use [OpenTelemetry](/send-data/opentelemetry). You can also create other map fields in any dataset. |
the current "and you can create them yourself" feels unclear imo. not sure my suggestion is exactly right though.
There was a problem hiding this comment.
Thanks! I think it's not just attributes.custom that we create in OTel datasets. I would not specify the actual fields here. But the second part of the sentence could surely be improved, so implemented.
There was a problem hiding this comment.
afaik that's the only one. which other ones do you know of?
There was a problem hiding this comment.
I thought resource.custom could also be one of them? Anyway, it's OK not to specify it here. We do it elsewhere.
apl/data-types/map-fields.mdx
Outdated
| ## Map fields and flattened fields | ||
|
|
||
| The parent map field and its flattened subfields can appear in the same dataset. For example, ingesting the following JSON adds a map field (`geo`) and two flattened fields (`geo.city`, `geo.country`) to the dataset. Axiom treats them as separate fields and doesn’t maintain a relationship between them. |
There was a problem hiding this comment.
yes ollie is right about this
the steps to end up in this situation are:
geois not a map field- ingest
{
"geo": {
"city": "Paris",
"country": "France"
}
}this gets put into geo.city and geo.country
- make
geoa map field - ingest the same
{
"geo": {
"city": "Paris",
"country": "France"
}
}this gets put into geo
(or in opposite order)
apl/data-types/map-fields.mdx
Outdated
|
|
||
| - Projecting away all subfields (in this example, `geo.city` and `geo.country`) doesn’t remove the parent map field `geo` unless it is explicitly projected away. | ||
| - Fields metadata may still include entries for projected-away fields if the map field is preserved. |
There was a problem hiding this comment.
i agree, the information below about `['geo']['city'] being different from ['geo.city'] is enough here. imo the above only makes it more confusing to someone who is reading about this for the first time.
apl/data-types/map-fields.mdx
Outdated
| - `where ['map_field'].property1.property2 == 14` | ||
| - `where ['map_field']['property1']['property2'] == 14` | ||
|
|
||
| If an entity name has spaces (` `), dots (`.`), or dashes (`-`), you can only use index notation for that entity. You can use dot notation for the other entities. For example: | ||
|
|
||
| - `where ['map.field']['property.name1']['property.name2'] == 14` | ||
| - `where ['map.field'].property1.property2 == 14` |
There was a problem hiding this comment.
looks like this was already merged in a previous pr. but as far as i can tell all of these apl examples will not return the desired events.
the only correct syntax is ['map.field']['something.inside.of.it']
see: https://app.dev.axiomtestlabs.co/axiomers-ft83/query?qid=4O0s3Ki3fPa-sxyen5
aside: imo even if there is some other way that works, we should only teach ['map.field']['something.inside.of.it'].
There was a problem hiding this comment.
@c-ehrlich Thanks for this. I don't know if this recently got broken because the original PR underwent a thorough review process: #65
I've tried to implement your suggestions here: 61e9181
The problem is that where ['map.field']['property.name1']['property.name2'] == 14 still doesn't work but then I don't know how else to deal with property names with special characters like dots.
No description provided.