Description
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue, please leave a comment
Terminology
ODM - object-document mapper
Problem
There currently is a low-level DynamoDB client implemented in the SDK, it works with opaque AttributeValue types and direct DynamoDB APIs
aws-sdk-rust/sdk/dynamodb/src/model.rs
Lines 6206 to 6277 in 7e43b19
This low-level SDK crate provides no convenience APIs (batteries) that simplify the regular idiomatic (Best Practices?, single table design?) usage of DynamoDB.
Working with raw AttributeValues, and ad-hoc implementing common workflows is very inconvenient and error-prone.
Solution
I propose we add a new crate that wraps aws-sdk-dynamodb low-level library and exposes the following "batteries-included" APIs (the list can be extended):
ODM
Implement serialization and deserialization (i.e. object-document mapping) of strongly-typed structs and enums (both plain and discriminated unions) into aws_sdk_dynamodb::AttributeValue via proc macros.
We can use serde to do the bulk of the job. I recommend taking over the job done in serde_dynamo crate, and also learn the approaches dynomite crate does.
The latter crate is more popular, but it is not very actively maintained. However, from my viewpoint, abusing serde as much as possible would be a better approach than implementing proc macros for converting between AttributeValue and strongly-typed structs and enums by hand, but that's debatable.
Condition and update expression builder
Condition expressions are used in queries and scans, and update expressions are used in update operations, and they both use custom DynamoDB syntax.
It's okay to use raw strings with the standard Rust format!() macro for simple cases, but sometimes expressions are very dynamic and the expression might depend on lots of different variables and conditions.
Building raw condition expression syntax dynamically is very error-prone, the high-level wrapper crate should expose builders for expressions.
See TypeScript's implementation of this concept in @aws/dynamodb-expressions package on npm.
Projection expression utilities
Add some methods, maybe proc macros to generate the types that represent a projection of different combinations of attributes.
This requires some more thorough design, but the core problems to solve here:
- Prevent the usage of raw strings and raw syntax for building the projection expression
- Prevent working with raw
AttributeValue
The API might look something like:
#[derive(Serialize, Deserialize, Projections)]
struct UserRecord {
partition_key: String,
sort_key: String,
#[project(Projection1)]
name: String,
// Come up with some syntax for nested properties projection (e.g. `ProjectionName = "<prop access>")
#[project(Projection1, Projection2, Projection3 = "[0]")]
departments: Vec<String>,
#[project(Projection1, Projection2)]
birth_date: chrono::NaiveDateTime,
}
// such that `#[derive(Projections)]` generates the following code:
#[derive(Serialize, Deserialize, Projection)]
struct Projection1 {
name: String,
departments: Vec<String>,
birth_date: chrono::NaiveDateTime,
}
#[derive(Serialize, Deserialize, Projection)]
struct Projection2 {
departments: Vec<String>,
birth_date: chrono::NaiveDateTime,
}
#[derive(Serialize, Deserialize, Projection)]
struct Projection3 {
// 0-th element of the original projected departments array
departments: String,
}
// where #[derive(Projection)] implements the `Projection` trait
impl Projection for Projection1 {
const PROJECTION_EXPRESSION: &'static str = "name, department, birth_date"
}
impl Projection for Projection2 {
const PROJECTION_EXPRESSION: &'static str = "departments, birth_date"
}
impl Projection for Projection3 {
const PROJECTION_EXPRESSION: &'static str = "departments[0]"
}Pagination utilities
Implement methods for streaming pagination (see dynomite::DynamoDbExt to learn about existing implementations).
Other utilities for best practices and common workflows
Implement helper utilities according to AWS docs for DynamoDB best practices and single table design.
For example, we've implemented some proc macros for segmented identifiers (described in single table design) in our private repo. We plan to open-source this code, and we may do it earlier to facilitate the development of the high-level DynamoDB crate.
So the list might also include:
- Utilities for working with segmented attributes (see Alex DeBrie talking about this concept at re:Invent).
Additional context
This issue is nowhere an exhaustive description of the desired design for the high-level wrapper crate for aws_sdk_dynamodb, the ideas should be refined and probably extended. However, I think it might be a good starting point to begin the discussion and initiate the work on the MVP subset for the planned high-level APIs (e.g. start with only ODM feature and iterate from that next).
We may decide to separate the described crate to other repo and split the planned features described here into more fine-grained issues if this makes sense to the maintainers.
Waiting for your feedback!