[Bug]: milvus load stuck at 0% with querynode error: [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=by-dev-rootcoord-dml_6]

[eskinmi]

Is there an existing issue for this?

• [Bug]: Loading collection stuck at 0% after server's sudden electricity lost #30501

Environment

- Milvus version: 2.5.0
- Deployment mode(standalone or cluster): Cluster (EKS)
- MQ type(rocksmq, pulsar or kafka):  pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.9
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: Xeon Scalable 3.5 GHz 16vCPU & 128Gi
- GPU: N/A

Current Behavior

When I try to load my collection in Milvus, from Attu I can see that the load is stuck at 0%. I have about 20M entities in the collections that I'm trying to load. When I look into the querynode logs, I see:

[2025/01/23 21:42:50.576 +00:00] [ERROR] [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=by-dev-rootcoord-dml_6] [subName=querynode-48-by-dev-rootcoord-dml_6_455282050800631750v0-true] [isMain=true] [error="failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 464:805"] [stack="github.com/milvus-io/milvus/pkg/mq/msgdispatcher.NewDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/dispatcher.go:122\ngithub.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).Add\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:106\ngithub.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*client).Register\n\t/workspace/source/pkg/mq/msgdispatcher/client.go:94\ngithub.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).ConsumeMsgStream\n\t/workspace/source/internal/util/pipeline/stream_pipeline.go:122\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).WatchDmChannels\n\t/workspace/source/internal/querynodev2/services.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).WatchDmChannels\n\t/workspace/source/internal/distributed/querynode/service.go:296\ngithub.com/milvus-io/milvus/pkg/proto/querypb._QueryNode_WatchDmChannels_Handler.func1\n\t/workspace/source/pkg/proto/querypb/query_coord_grpc.pb.go:2020\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ServerIDValidationUnaryServerInterceptor.func7\n\t/workspace/source/pkg/util/interceptor/server_id_interceptor.go:54\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ClusterValidationUnaryServerInterceptor.func6\n\t/workspace/source/pkg/util/interceptor/cluster_interceptor.go:48\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/workspace/source/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34\ngithub.com/milvus-io/milvus/pkg/proto/querypb._QueryNode_WatchDmChannels_Handler\n\t/workspace/source/pkg/proto/querypb/query_coord_grpc.pb.go:2022\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1379\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1790\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1029"]

Expected Behavior

The collection should load successfully.

Steps To Reproduce

No response

Milvus Log

milvus-log.tar.gz

Anything else?

No response

[yanliang567]

/assign @bigsheeper
/unassign

[xiaofan-luan]

@eskinmi

I think this might be a different issue.
The error message showed seek failure. Which mean pulsar report an error on subscribe by-dev-rootcoord-dml_6

1. restart pulsar cluster see it works
2. try to use pulsarctl to connect to pulsar cluster, and subscribe from by-dev-rootcoord-dml_6 this topic, see if can read data
3. upgrade Milvus to 2.5.4

[xiaofan-luan]

If your Pulsar TTL (Time-To-Live) is set to a very short duration, it can also cause similar issues. When the system recovers from a failure, the corresponding position may have already been garbage collected.

[xiaofan-luan]

check which collection watch failed and use birdwatcher to see the checkpoints, if some of the checkpoints is old, we need to reset it to newer timestamp

[eskinmi]

@xiaofan-luan

Thank you very much for your help! Restarting the pulsar cluster did work indeed.
I will make sure to upgrade to the latest version.

🎉

[eskinmi]

@xiaofan-luan btw I can't see the TTL configuration in here: https://milvus.io/docs/configure_pulsar.md

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.