Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use same server info for all map modes #137

Merged
merged 7 commits into from
Jul 18, 2024
Merged

feat: use same server info for all map modes #137

merged 7 commits into from
Jul 18, 2024

Conversation

kohlisid
Copy link
Contributor

@kohlisid kohlisid commented Jul 15, 2024

Sidhant Kohli added 2 commits July 15, 2024 11:54
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
@KeranYang KeranYang requested a review from ayildirim21 July 16, 2024 15:58
Sidhant Kohli added 2 commits July 16, 2024 12:54
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Copy link
Contributor Author

Map Stream with new map mode metadata

no annotations in spec

    - name: flatmap
      limits:
        readBatchSize: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-go/map-flatmap-stream:stable

Map stream server with mapstream.sock

 kubectl logs simple-pl-go-flatmap-0-w5nqq -f
2024/07/16 21:28:04 starting the gRPC server with unix domain socket... /var/run/numaflow/mapstream.sock

Server info file - /var/run/numaflow/mapper-server-info

kubectl logs simple-pl-go-flatmap-0-w5nqq numa -f | grep mapper
2024/07/16 21:27:42 Server info file /var/run/numaflow/mapper-server-info is not ready...


 kubectl logs simple-pl-go-flatmap-0-w5nqq numa -f | grep Info  
2024/07/16 21:28:05 ServerInfo: &{uds go 1.2.0-rc4 v0.7.0-rc2 map[MAP_MODE:stream-map]}


kubectl logs simple-pl-go-flatmap-0-w5nqq numa -f | grep enabled
2024-07-16T21:28:05.294120902Z  INFO    numaflow.MapUDF-processor       udf/map_udf.go:150 
Map mode enabled: Stream Map    {"pipeline": "simple-pl-go", "vertex": "flatmap", "protocol": "uds-grpc-map-udf"}

Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Copy link
Contributor Author

kohlisid commented Jul 17, 2024

Batch Map with new map mode metadata

no annotations in spec

    - name: go-split
      scale:
        min: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-go/batch-map-flatmap:stable1
          imagePullPolicy: Always

Batch Map stream server with batchmap.sock

kubectl logs flatmap-go-split-0-0xcsa -f
2024/07/17 18:50:23 starting the gRPC server with unix domain socket... /var/run/numaflow/batchmap.sock

Server info file - /var/run/numaflow/mapper-server-info

kubectl logs flatmap-go-split-0-0xcsa numa -f | grep mapper 
2024/07/17 18:50:08 Server info file /var/run/numaflow/mapper-server-info is not ready...



 kubectl logs flatmap-go-split-0-0xcsa numa -f | grep ServerInfo:
2024/07/17 18:50:24 ServerInfo: &{uds go 1.2.0-rc4 v0.8.0 map[MAP_MODE:batch-map]}


 kubectl logs flatmap-go-split-0-0xcsa numa -f | grep enabled    
{"level":"info","ts":"2024-07-17T18:50:24.456998519Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:173","msg":"Map mode enabled: Batch Map","pipeline":"flatmap","vertex":"go-split","protocol":"uds-grpc-map-udf"}


@kohlisid
Copy link
Contributor Author

kohlisid commented Jul 17, 2024

Unary Map with new map mode metadata

no annotations in spec

    - name: go-split
      scale:
        min: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-go/map-flatmap:stable
          imagePullPolicy: Always

Unary Map stream server with map.sock

kubectl logs flatmap-go-split-0-mvcdg -f
2024/07/17 18:57:42 starting the gRPC server with unix domain socket... /var/run/numaflow/map.sock

Server info file - /var/run/numaflow/mapper-server-info

kubectl logs flatmap-go-split-0-mvcdg -f numa | grep mapper
2024/07/17 18:57:27 Server info file /var/run/numaflow/mapper-server-info is not ready...




 kubectl logs flatmap-go-split-0-mvcdg -f numa | grep ServerInfo
2024/07/17 18:57:43 ServerInfo: &{uds go 1.2.0-rc4 v0.7.0-rc2 map[MAP_MODE:unary-map]}


 kubectl logs flatmap-go-split-0-mvcdg -f numa | grep enabled   
{"level":"info","ts":"2024-07-17T18:57:43.308422104Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:196","msg":"Map mode enabled: Unary Map","pipeline":"flatmap","vertex":"go-split","protocol":"uds-grpc-map-udf"}



@kohlisid
Copy link
Contributor Author

kohlisid commented Jul 17, 2024

Unary Map with no metadata (and new numaflow)

no annotations in spec

    - name: go-split
      scale:
        min: 1
      udf:
        container:
          image: quay.io/numaio/numaflow-go/map-flatmap:stable
          imagePullPolicy: Always

Unary Map stream server with map.sock

 kubectl logs flatmap-go-split-0-elw3l -f
2024/07/17 19:01:19 starting the gRPC server with unix domain socket... /var/run/numaflow/map.sock

Server info file - /var/run/numaflow/mapper-server-info

 kubectl logs flatmap-go-split-0-elw3l -f numa | grep mapper
2024/07/17 19:01:06 Server info file /var/run/numaflow/mapper-server-info is not ready...




 kubectl logs flatmap-go-split-0-elw3l -f numa | grep ServerInfo
2024/07/17 19:01:19 ServerInfo: &{uds go 1.2.0-rc4 v0.7.0-rc2 map[]}



 kubectl logs flatmap-go-split-0-elw3l -f numa | grep enabled   
{"level":"info","ts":"2024-07-17T19:01:19.426399748Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:196","msg":"Map mode enabled: Unary Map","pipeline":"flatmap","vertex":"go-split","protocol":"uds-grpc-map-udf"}

Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from KeranYang July 17, 2024 19:08
@kohlisid kohlisid marked this pull request as ready for review July 17, 2024 19:09
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested review from KeranYang and whynowy July 17, 2024 21:05
@kohlisid kohlisid requested a review from vigith July 18, 2024 01:27
@kohlisid kohlisid merged commit a2bc6b1 into main Jul 18, 2024
3 checks passed
@kohlisid kohlisid deleted the serverinfo branch July 18, 2024 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Go-SDK single-server info for map
3 participants