Skip to content

NEOS-1714 add hash transformer #3477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

alishakawaguchi
Copy link
Contributor

No description provided.

@alishakawaguchi alishakawaguchi added the enhancement New feature or request label Apr 8, 2025
@alishakawaguchi alishakawaguchi self-assigned this Apr 8, 2025
Copy link

linear bot commented Apr 8, 2025

Copy link

vercel bot commented Apr 8, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
neosync-docs ⬜️ Ignored (Inspect) Visit Preview Apr 8, 2025 11:34pm

Copy link

github-actions bot commented Apr 8, 2025

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedApr 8, 2025, 10:16 PM

Copy link

github-actions bot commented Apr 8, 2025

Benchstat Geomean Results

1.14% sec/op, -0.00% B/op, 0.00% allocs/op

Benchstat results
                                                                 │ main-benchmark.txt │        branch-benchmark.txt        │
                                                                 │       sec/op       │    sec/op     vs base              │
CleanPostgresType-4                                                      42.66n ±  9%   42.76n ±  2%       ~ (p=0.665 n=6)
CleanMysqlType-4                                                         53.94n ±  0%   54.05n ±  1%       ~ (p=0.608 n=6)
Runner_Single-4                                                          7.701µ ±  3%   7.674µ ±  2%       ~ (p=0.937 n=6)
GenerateBool-4                                                           11.54n ±  1%   11.63n ±  0%       ~ (p=0.056 n=6)
GenerateBusinessName-4                                                   304.1n ±  9%   331.3n ± 10%  +8.96% (p=0.026 n=6)
GenerateCardNumber-4                                                     28.96n ±  6%   28.95n ±  2%       ~ (p=0.509 n=6)
GenerateCategorical-4                                                    93.68n ±  1%   93.54n ±  0%       ~ (p=0.619 n=6)
GenerateCity-4                                                           173.3n ±  2%   177.8n ±  0%  +2.60% (p=0.004 n=6)
GenerateCountry-4                                                        95.17n ±  2%   96.89n ±  0%  +1.80% (p=0.041 n=6)
GenerateEmail-4                                                          1.309µ ±  2%   1.313µ ±  1%       ~ (p=0.675 n=6)
GenerateFirstName-4                                                      203.3n ±  0%   205.3n ±  1%  +0.98% (p=0.002 n=6)
GenerateFloat64-4                                                        26.39n ±  1%   26.73n ±  3%  +1.25% (p=0.026 n=6)
GenerateFullAddress-4                                                    1.411µ ±  0%   1.419µ ±  1%  +0.57% (p=0.004 n=6)
GenerateFullName-4                                                       1.204µ ±  2%   1.323µ ±  3%  +9.93% (p=0.002 n=6)
GenerateGender-4                                                         36.69n ±  0%   37.30n ±  1%  +1.66% (p=0.002 n=6)
GenerateInt64-4                                                          27.02n ±  0%   27.21n ±  1%  +0.70% (p=0.002 n=6)
GenerateInt64PhoneNumber-4                                               51.15n ±  0%   53.19n ±  0%  +3.98% (p=0.002 n=6)
GenerateInternationalPhoneNumber-4                                       174.2n ±  0%   165.6n ±  3%  -4.91% (p=0.002 n=6)
GenerateIpAddress-4                                                      560.0n ±  2%   532.4n ±  1%  -4.95% (p=0.002 n=6)
GenerateLastName-4                                                       189.2n ±  1%   197.7n ±  1%  +4.49% (p=0.002 n=6)
GenerateRandomString-4                                                   264.9n ± 24%   267.9n ±  0%       ~ (p=0.132 n=6)
GenerateSHA256Hash-4                                                     1.004µ ±  1%   1.002µ ±  1%       ~ (p=0.331 n=6)
GenerateSSN-4                                                            265.8n ±  3%   265.1n ±  1%       ~ (p=0.310 n=6)
GenerateState-4                                                          94.77n ±  2%   95.57n ±  0%       ~ (p=0.394 n=6)
GenerateStreetAddress-4                                                  207.5n ±  1%   211.4n ±  1%  +1.88% (p=0.002 n=6)
GenerateStringPhoneNumber-4                                              171.1n ±  0%   166.3n ±  0%  -2.78% (p=0.002 n=6)
GenerateUnixTimestamp-4                                                  77.04n ±  1%   76.96n ±  3%       ~ (p=0.818 n=6)
GenerateUsername-4                                                       346.1n ±  2%   362.9n ±  6%  +4.88% (p=0.002 n=6)
GenerateUTCTimestamp-4                                                   105.8n ±  1%   101.4n ±  0%  -4.11% (p=0.002 n=6)
GenerateUUID-4                                                           732.1n ±  1%   728.6n ±  1%       ~ (p=0.132 n=6)
GenerateZipcode-4                                                        93.42n ±  1%   93.80n ±  0%       ~ (p=0.065 n=6)
TransformCharacterScramble-4                                             159.9n ±  3%   159.0n ±  1%       ~ (p=0.084 n=6)
TransformE164PhoneNumber-4                                               171.6n ±  0%   173.2n ±  1%  +0.96% (p=0.009 n=6)
TransformEmail-4                                                         1.774µ ±  1%   1.751µ ±  0%  -1.30% (p=0.002 n=6)
TransformFirstName-4                                                     233.9n ±  1%   243.5n ±  2%  +4.06% (p=0.002 n=6)
TransformFloat64-4                                                       101.2n ±  1%   105.5n ±  0%  +4.20% (p=0.002 n=6)
TransformFullName-4                                                      1.236µ ±  2%   1.331µ ±  3%  +7.69% (p=0.002 n=6)
TransformInt64-4                                                         31.92n ±  2%   32.03n ±  6%  +0.36% (p=0.041 n=6)
TransformInt64PhoneNumber-4                                              57.88n ±  3%   58.36n ±  2%       ~ (p=0.288 n=6)
TransformLastName-4                                                      194.8n ±  1%   204.1n ±  2%  +4.77% (p=0.002 n=6)
TransformString-4                                                        291.5n ±  1%   294.9n ±  1%  +1.15% (p=0.002 n=6)
TransformStringPhoneNumber-4                                             198.8n ±  1%   188.9n ±  1%  -4.98% (p=0.002 n=6)
TransformUuid-4                                                          32.34n ±  0%   32.89n ±  0%  +1.70% (p=0.002 n=6)
GenerateRandomStringWithInclusiveBounds/SmallRange_1_10-4                82.86n ±  0%   82.47n ±  5%       ~ (p=0.180 n=6)
GenerateRandomStringWithInclusiveBounds/MediumRange_10_100-4             259.2n ±  3%   257.8n ±  1%       ~ (p=0.065 n=6)
GenerateRandomStringWithInclusiveBounds/LargeRange_100_1000-4            1.882µ ±  0%   1.911µ ±  1%  +1.54% (p=0.002 n=6)
GenerateRandomStringWithInclusiveBounds/FixedLength_50_50-4              199.7n ±  0%   199.2n ±  0%       ~ (p=0.468 n=6)
GenerateRandomStringWithInclusiveBounds/ZeroLength_0_0-4                 3.736n ±  2%   3.739n ±  0%       ~ (p=0.394 n=6)
GenerateRandomStringWithInclusiveBounds/Text_1_65535-4                   14.95µ ±  0%   15.22µ ±  5%  +1.78% (p=0.002 n=6)
GenerateRandomStringWithInclusiveBounds/AboveLimit_1_100000-4            14.95µ ±  2%   15.08µ ±  8%  +0.92% (p=0.026 n=6)
GenerateRandomStringWithInclusiveBounds/FixedAboveLimit_100000-4         26.66µ ±  1%   27.10µ ±  2%       ~ (p=0.065 n=6)
GenerateRandomStringWithInclusiveBounds/MaxTextSize_65535-4              26.59µ ±  1%   28.99µ ±  6%  +8.99% (p=0.002 n=6)
TransformHash-4                                                                         326.6n ±  0%
geomean                                                                  240.5n         244.6n        +1.14%

                                                                 │ main-benchmark.txt │         branch-benchmark.txt         │
                                                                 │        B/op        │     B/op      vs base                │
CleanPostgresType-4                                                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
CleanMysqlType-4                                                         0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                                                        2.290Ki ± 1%     2.288Ki ± 3%       ~ (p=0.974 n=6)
GenerateBool-4                                                           0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                                                   304.0 ± 0%       304.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                                                     8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                                                    64.00 ± 0%       64.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                                                           160.0 ± 0%       160.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                                                        24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                                                          385.0 ± 0%       385.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                                                      208.0 ± 0%       208.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                                                        8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                                                    512.0 ± 0%       512.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                                                       418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                                                         16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                                                          7.000 ± 0%       7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                                               8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4                                       40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                                                      31.00 ± 0%       31.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                                                       176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                                                   72.00 ± 0%       72.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                                                     288.0 ± 0%       288.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                                                            46.00 ± 0%       46.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                                                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                                                  240.0 ± 0%       240.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                                              40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                                                  8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                                                       205.0 ± 0%       205.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                                                   24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                                                           80.00 ± 0%       80.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                                                        24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4                                             40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformE164PhoneNumber-4                                               40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                                                         489.0 ± 0%       489.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                                                     224.0 ± 0%       224.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                                                       88.00 ± 0%       88.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                                                      418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                                                         8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                                              8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                                                      176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                                                        88.00 ± 0%       88.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4                                             56.00 ± 0%       56.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                                                          16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/SmallRange_1_10-4                6.000 ± 0%       6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/MediumRange_10_100-4             61.00 ± 0%       61.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/LargeRange_100_1000-4            582.0 ± 0%       582.0 ± 0%       ~ (p=1.000 n=6)
GenerateRandomStringWithInclusiveBounds/FixedLength_50_50-4              64.00 ± 0%       64.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/ZeroLength_0_0-4                 0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/Text_1_65535-4                 34.43Ki ± 0%     34.42Ki ± 0%       ~ (p=0.937 n=6)
GenerateRandomStringWithInclusiveBounds/AboveLimit_1_100000-4          34.44Ki ± 0%     34.39Ki ± 0%       ~ (p=0.368 n=6)
GenerateRandomStringWithInclusiveBounds/FixedAboveLimit_100000-4       64.00Ki ± 0%     64.00Ki ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/MaxTextSize_65535-4            64.00Ki ± 0%     64.00Ki ± 0%       ~ (p=1.000 n=6) ¹
TransformHash-4                                                                           216.0 ± 0%
geomean                                                                             ²                 -0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                                 │ main-benchmark.txt │        branch-benchmark.txt         │
                                                                 │     allocs/op      │  allocs/op   vs base                │
CleanPostgresType-4                                                     0.000 ±  0%     0.000 ±  0%       ~ (p=1.000 n=6) ¹
CleanMysqlType-4                                                        0.000 ±  0%     0.000 ±  0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                                                         24.00 ±  0%     24.00 ±  0%       ~ (p=1.000 n=6) ¹
GenerateBool-4                                                          0.000 ±  0%     0.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                                                  2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                                                    1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                                                   2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                                                          2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                                                       2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                                                         8.000 ±  0%     8.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                                                     2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                                                       1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                                                   10.00 ±  0%     10.00 ±  0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                                                      6.000 ±  0%     6.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                                                        1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                                                         0.000 ±  0%     0.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                                              1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4                                      3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                                                     2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                                                      2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                                                  2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                                                    7.000 ±  0%     7.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                                                           3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateState-4                                                         2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                                                 2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                                             3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                                                 1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                                                      6.000 ±  0%     6.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                                                  1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                                                          3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                                                       2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4                                            3.000 ± 33%     3.000 ± 33%       ~ (p=1.000 n=6)
TransformE164PhoneNumber-4                                              3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                                                        14.00 ±  0%     14.00 ±  0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                                                    3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                                                      3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                                                     6.000 ±  0%     6.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                                                        1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                                             1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                                                     2.000 ±  0%     2.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformString-4                                                       3.000 ±  0%     3.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4                                            4.000 ±  0%     4.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                                                         1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/SmallRange_1_10-4               1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/MediumRange_10_100-4            1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/LargeRange_100_1000-4           1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/FixedLength_50_50-4             1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/ZeroLength_0_0-4                0.000 ±  0%     0.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/Text_1_65535-4                  1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/AboveLimit_1_100000-4           1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/FixedAboveLimit_100000-4        1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
GenerateRandomStringWithInclusiveBounds/MaxTextSize_65535-4             1.000 ±  0%     1.000 ±  0%       ~ (p=1.000 n=6) ¹
TransformHash-4                                                                         7.000 ±  0%
geomean                                                                             ²                +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

Copy link
Member

@nickzelei nickzelei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left a comment regarding supported types.

return v, nil
case []byte:
return string(v), nil
// todo: handle more types
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be difficult to fill in more types here? Ideally this thing works for any type and we can allow it for any or most data types. Unclear how this will appear on the FE atm though.

Copy link
Member

@nickzelei nickzelei Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm moreso wondering fi there is a way to effectively cast any input type to either []byte or string. Whichever is the most convenient for passing in to a hash function. You can hash anything if it's a []byte...maybe casting to string is not the best approach.

}

func hashToString(hasher hash.Hash, input string) (string, error) {
_, err := hasher.Write([]byte(input))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it's better to make the input a []byte then we just convert the value any to a []bte, then we support everything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants