Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: The standalone pod was restarted due to tests executed during the CI pipeline run of the Go SDK #39535

Open
1 task done
zhuwenxing opened this issue Jan 23, 2025 · 6 comments
Open
1 task done

[Bug]: The standalone pod was restarted due to tests executed during the CI pipeline run of the Go SDK #39535

zhuwenxing opened this issue Jan 23, 2025 · 6 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.6.0

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior


2025-01-23T03:05:13Z {container="step-test"} === FAIL: testcases TestSearchGroupByFloatGrowing (223.53s)

2025-01-23T03:05:13Z {container="step-test"} === FAIL: testcases TestSearchGroupByFloatDefault (269.13s)

2025-01-23T03:05:13Z {container="step-test"} === FAIL: testcases TestSearchGroupByFloatDefaultCosine (269.13s)

2025-01-23T03:05:13Z {container="step-test"} 

2025-01-23T03:05:13Z {container="step-test"} DONE 229 tests, 19 skipped, 3 failures in 741.282s

2025-01-23T03:05:15Z {container="prepare"} 2025/01/23 03:05:15 Entrypoint initialization

2025-01-23T03:05:17Z {container="place-scripts"} 2025/01/23 03:05:17 Decoded script /tekton/scripts/script-0-2xvkz

2025-01-23T03:05:17Z {container="place-scripts"} 2025/01/23 03:05:17 Decoded script /tekton/scripts/script-1-j5mnc

2025-01-23T03:05:17Z {container="place-scripts"} 2025/01/23 03:05:17 Decoded script /tekton/scripts/script-2-6xsw2

2025-01-23T03:05:21Z {container="step-check-status"} ----------------Pod Status --------------------------------------------

2025-01-23T03:05:21Z {container="step-check-status"} ms-39529-2-go-pr-etcd-0                                     1/1     Running            0                 15m     10.104.31.26    4am-node34   <none>           <none>

2025-01-23T03:05:21Z {container="step-check-status"} ms-39529-2-go-pr-milvus-standalone-785cfdd7cb-b4lvd         0/1     Running            4 (113s ago)      15m     10.104.31.20    4am-node34   <none>           <none>

2025-01-23T03:05:21Z {container="step-check-status"} ms-39529-2-go-pr-minio-6468d76cff-rx596                     1/1     Running            0                 15m     10.104.31.21    4am-node34   <none>           <none>

2025-01-23T03:05:21Z {container="step-check-status"} ----------------Pod Restarts --------------------------------------------

2025-01-23T03:05:21Z {container="step-check-status"} POD                                                   RESTARTS   LAST TERMINATION REASON

2025-01-23T03:05:21Z {container="step-check-status"} ms-39529-2-go-pr-milvus-standalone-785cfdd7cb-b4lvd   4          Error

failed ci job: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20HA%20CI(go-sdk)/detail/PR-39529/2/pipeline

Expected Behavior

No response

Steps To Reproduce



Milvus Log


No response


Anything else?


No response

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @zhuwenxing
zhuwenxing




            added
      

  kind/bug

  Issues or changes related a bug

      

  needs-triage

  Indicates an issue or PR lacks a `triage/foo` label and requires one.

  labels


      Jan 23, 2025

    

  



    


    

    

      
    

    


      

          @zhuwenxing
zhuwenxing




          added this to the   2.6.0 milestone


      Jan 23, 2025

    

  








      

  
      



  

  @zhuwenxing





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  Author


  


  

    

  





      


        


  
    

      
          
2025-01-23T11:02:37.560113723+08:00 stdout F I20250123 03:02:37.560060  2049 ChunkedSegmentSealedImpl.cpp:172] [SERVER][LoadScalarIndex][CGO_DYN][]segment pk sorted, skip user index loading for primary key field
2025-01-23T11:02:37.560162251+08:00 stdout F [2025/01/23 03:02:37.560 +00:00] [INFO] [segments/segment.go:1157] ["updateSegmentIndex done"] [traceID=9729076588bad8fe923f5a729a8b4d04] [collectionID=455501587754133148] [partitionID=455501587754133149] [segmentID=455501587754138392] [fieldID=100]
2025-01-23T11:02:37.560186392+08:00 stdout F I20250123 03:02:37.560158  2049 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_DYN][]inverted index remove path:/var/lib/milvus/data/indexnode/index_files/455501587754143286/1/
2025-01-23T11:02:37.560309786+08:00 stderr F [2025-01-23T03:02:37Z ERROR tantivy::reader] Error while loading searcher after commit was detected. OpenReadError(FileDoesNotExist("meta.json"))
2025-01-23T11:02:37.560367139+08:00 stdout F  => remove local directory:/var/lib/milvus/data/indexnode/index_files/455501587754143286/1/ failed, error: Directory not empty, files: /var/lib/milvus/data/indexnode/index_files/455501587754143286/1/.tantivy-meta.lock at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:228
2025-01-23T11:02:37.560381019+08:00 stdout F 
2025-01-23T11:02:37.560518021+08:00 stdout F W20250123 03:02:37.560460  2049 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
2025-01-23T11:02:37.560536661+08:00 stderr F terminate called after throwing an instance of 'milvus::SegcoreError'
2025-01-23T11:02:37.560554855+08:00 stderr F   what():   => remove local directory:/var/lib/milvus/data/indexnode/index_files/455501587754143286/1/ failed, error: Directory not empty, files: /var/lib/milvus/data/indexnode/index_files/455501587754143286/1/.tantivy-meta.lock at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:228
2025-01-23T11:02:37.560560859+08:00 stderr F 
2025-01-23T11:02:37.562730451+08:00 stderr F SIGABRT: abort
2025-01-23T11:02:37.562747419+08:00 stderr F PC=0x7ff16ab0c9fc m=701 sigcode=18446744073709551610
2025-01-23T11:02:37.562750462+08:00 stderr F signal arrived during cgo execution
2025-01-23T11:02:37.562752235+08:00 stderr F 
2025-01-23T11:02:37.562755127+08:00 stderr F goroutine 249980 gp=0xc008782e00 m=701 mp=0xc002dc2008 [syscall, locked to thread]:
2025-01-23T11:02:37.583023639+08:00 stdout F I20250123 03:02:37.582885  2021 VectorMemIndex.cpp:238] [SERVER][Load][CGO_LOAD][]construct binary set...
2025-01-23T11:02:37.583048955+08:00 stdout F I20250123 03:02:37.582957  2021 VectorMemIndex.cpp:241] [SERVER][Load][CGO_LOAD][]add index data to binary set: HNSW
2025-01-23T11:02:37.583055263+08:00 stdout F I20250123 03:02:37.582974  2021 VectorMemIndex.cpp:254] [SERVER][Load][CGO_LOAD][]load index into Knowhere...
2025-01-23T11:02:37.586780623+08:00 stdout F I20250123 03:02:37.586673  2021 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][CGO_LOAD] Load index: done (3.583581 ms)
2025-01-23T11:02:37.586819013+08:00 stdout F I20250123 03:02:37.586746  2021 VectorMemIndex.cpp:257] [SERVER][Load][CGO_LOAD][]load vector index done
2025-01-23T11:02:37.58684319+08:00 stdout F I20250123 03:02:37.586778  2021 load_index_c.cpp:336] [SERVER][AppendIndexV2][CGO_LOAD][][collection=455501587754133117][segment=455501587754133580][field=120][enable_mmap=false] load index 455501587754143245 done
2025-01-23T11:02:53.044940065+08:00 stderr F non-Go function
2025-01-23T11:02:53.062028282+08:00 stderr F 	pc=0x7ff16ab0c9fc
2025-01-23T11:02:53.062038096+08:00 stderr F non-Go function
2025-01-23T11:02:53.062042165+08:00 stderr F 	pc=0x7ff16aab8475
2025-01-23T11:02:53.062046244+08:00 stderr F non-Go function
2025-01-23T11:02:53.062049124+08:00 stderr F 	pc=0x7ff16aa9e7f2
2025-01-23T11:02:53.062060034+08:00 stderr F non-Go function
2025-01-23T11:02:53.062063984+08:00 stderr F 	pc=0x7ff16a8ecb9d
2025-01-23T11:02:53.062067126+08:00 stderr F non-Go function
2025-01-23T11:02:53.062070314+08:00 stderr F 	pc=0x7ff16a8f820b
2025-01-23T11:02:53.062072761+08:00 stderr F non-Go function
2025-01-23T11:02:53.062850185+08:00 stderr F 	pc=0x7ff16a8f71e8
2025-01-23T11:02:53.062854462+08:00 stderr F non-Go function
2025-01-23T11:02:53.062857727+08:00 stderr F 	pc=0x7ff16a8f7958
2025-01-23T11:02:53.062862637+08:00 stderr F non-Go function
2025-01-23T11:02:53.0628683+08:00 stderr F 	pc=0x7ff16acb5883


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            

    

      
    

    


      

          @zhuwenxing
zhuwenxing




            added
      

  priority/critical-urgent

  Highest priority. Must be actively worked on as someone's top priority right now.

      

  severity/critical

  Critical, lead to crash, data missing, wrong result, function totally doesn't work.

  labels


      Jan 23, 2025

    

  








      

  
      



  

  @yanliang567





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

  





      


        


  
    

      
          
/assign @congqixia

/unassign

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            


    

    

      
    

    


      

          @yanliang567
yanliang567




            added
      

  triage/accepted

  Indicates an issue or PR is ready to be actively worked on.

    and removed
      

  needs-triage

  Indicates an issue or PR lacks a `triage/foo` label and requires one.

  labels


      Jan 23, 2025

    

  








      

  
      



  

  @zhuwenxing





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  Author


  


  

    

      

      
            zhuwenxing
  

      

      

      commented


        Jan 23, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          
This issue also reproduced in some chaos test with same panic info


querynode-pod-kill-20229-milvus-querynode-cc6fbb8c8-6b9zj panic log


[2025/01/22 07:58:19.519 +00:00] [INFO] [segments/segment.go:1157] ["updateSegmentIndex done"] [traceID=aa9e8e2572256e743948c67079419a83] [collectionID=455483103913911355] [partitionID=455483103913911356] [segmentID=455483103924327448] [fieldID=100]
I20250122 07:58:19.519178  7492 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_DYN][]inverted index remove path:/var/lib/milvus/data/querynode/index_files/455483103924697417/1/
[2025-01-22T07:58:19Z ERROR tantivy::reader] Error while loading searcher after commit was detected. OpenReadError(FileDoesNotExist("meta.json"))
 => remove local directory:/var/lib/milvus/data/querynode/index_files/455483103924697417/1/ failed, error: Directory not empty, files: /var/lib/milvus/data/querynode/index_files/455483103924697417/1/.tantivy-meta.lock at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:228

W20250122 07:58:19.519479  7492 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
terminate called after throwing an instance of 'milvus::SegcoreError'
  what():   => remove local directory:/var/lib/milvus/data/querynode/index_files/455483103924697417/1/ failed, error: Directory not empty, files: /var/lib/milvus/data/querynode/index_files/455483103924697417/1/.tantivy-meta.lock at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:228

SIGABRT: abort
PC=0x7f4be09d59fc m=2109 sigcode=18446744073709551610
signal arrived during cgo execution



failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-cron/detail/chaos-test-cron/20229/pipeline

log:


artifacts-querynode-pod-kill-20229-server-logs.tar.gz


cluster: 4am

ns: chaos-testing


pod info


+ kubectl get pods -o wide
 + grep querynode-pod-kill-20229
 querynode-pod-kill-20229-etcd-0                                   1/1     Running                  0                 45m     10.104.15.47    4am-node20   <none>           <none>
 querynode-pod-kill-20229-etcd-1                                   1/1     Running                  0                 45m     10.104.21.220   4am-node24   <none>           <none>
 querynode-pod-kill-20229-etcd-2                                   1/1     Running                  0                 45m     10.104.26.141   4am-node32   <none>           <none>
 querynode-pod-kill-20229-milvus-datanode-5bcd86cc4c-2kc48         1/1     Running                  3 (44m ago)       45m     10.104.30.154   4am-node38   <none>           <none>
 querynode-pod-kill-20229-milvus-datanode-5bcd86cc4c-xrszq         1/1     Running                  3 (44m ago)       45m     10.104.13.12    4am-node16   <none>           <none>
 querynode-pod-kill-20229-milvus-indexnode-68bd7dfc5b-6czt9        1/1     Running                  3 (44m ago)       45m     10.104.27.225   4am-node31   <none>           <none>
 querynode-pod-kill-20229-milvus-indexnode-68bd7dfc5b-9zmbn        1/1     Running                  3 (44m ago)       45m     10.104.23.201   4am-node27   <none>           <none>
 querynode-pod-kill-20229-milvus-indexnode-68bd7dfc5b-tvdtf        1/1     Running                  3 (44m ago)       45m     10.104.19.159   4am-node28   <none>           <none>
 querynode-pod-kill-20229-milvus-mixcoord-57897895f6-64pvp         1/1     Running                  3 (44m ago)       45m     10.104.30.152   4am-node38   <none>           <none>
 querynode-pod-kill-20229-milvus-proxy-f9c6c476-59xgm              1/1     Running                  3 (44m ago)       45m     10.104.30.151   4am-node38   <none>           <none>
 querynode-pod-kill-20229-milvus-querynode-cc6fbb8c8-6b9zj         0/1     Running                  1 (71s ago)       21m     10.104.21.232   4am-node24   <none>           <none>
 querynode-pod-kill-20229-milvus-querynode-cc6fbb8c8-n5vqk         1/1     Running                  1 (9m43s ago)     21m     10.104.16.24    4am-node21   <none>           <none>
 querynode-pod-kill-20229-milvus-querynode-cc6fbb8c8-q8xt9         1/1     Running                  0                 21m     10.104.19.173   4am-node28   <none>           <none>
 querynode-pod-kill-20229-milvus-streamingnode-7fc4cf868d-hmr8h    1/1     Running                  3 (44m ago)       45m     10.104.30.153   4am-node38   <none>           <none>
 querynode-pod-kill-20229-minio-0                                  1/1     Running                  0                 45m     10.104.27.231   4am-node31   <none>           <none>
 querynode-pod-kill-20229-minio-1                                  1/1     Running                  0                 45m     10.104.15.41    4am-node20   <none>           <none>
 querynode-pod-kill-20229-minio-2                                  1/1     Running                  0                 45m     10.104.34.102   4am-node37   <none>           <none>
 querynode-pod-kill-20229-minio-3                                  1/1     Running                  0                 45m     10.104.33.96    4am-node36   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-bookie-0                        1/1     Running                  0                 45m     10.104.15.46    4am-node20   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-bookie-1                        1/1     Running                  0                 45m     10.104.34.104   4am-node37   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-bookie-2                        1/1     Running                  0                 45m     10.104.26.142   4am-node32   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-bookie-init-nkms8               0/1     Completed                0                 45m     10.104.23.200   4am-node27   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-broker-0                        1/1     Running                  0                 45m     10.104.15.24    4am-node20   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-broker-1                        1/1     Running                  0                 45m     10.104.26.130   4am-node32   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-proxy-0                         1/1     Running                  0                 45m     10.104.9.159    4am-node14   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-proxy-1                         1/1     Running                  0                 45m     10.104.15.25    4am-node20   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-pulsar-init-ktttb               0/1     Completed                0                 45m     10.104.27.224   4am-node31   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-recovery-0                      1/1     Running                  0                 45m     10.104.23.203   4am-node27   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-zookeeper-0                     1/1     Running                  0                 45m     10.104.15.44    4am-node20   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-zookeeper-1                     1/1     Running                  0                 45m     10.104.21.219   4am-node24   <none>           <none>
 querynode-pod-kill-20229-pulsarv3-zookeeper-2                     1/1     Running                  0                 45m     10.104.26.140   4am-node32   <none>           <none>


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @eugen-korentsov





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

      

      
            eugen-korentsov
  

      

      

      commented


        Jan 24, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          
We have same issue with milvus 2.5.3


[2025-01-24T07:15:46Z ERROR tantivy::reader] Error while loading searcher after commit was detected. OpenReadError(FileDoesNotExist("meta.json"))
 => remove local directory:/var/lib/milvus/data/querynode/index_files/455508594323931343/1/ failed, error: Directory not empty at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:220

terminate called after throwing an instance of 'milvus::SegcoreError'
W20250124 07:15:46.880080 14483 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
  what():   => remove local directory:/var/lib/milvus/data/querynode/index_files/455508594323931343/1/ failed, error: Directory not empty at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:220

SIGABRT: abort
PC=0x7f2af6e629fc m=11634 sigcode=18446744073709551610
signal arrived during cgo execution```


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @congqixia





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

      

      
            congqixia
  

      

      

      commented


        Jan 24, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          
@eugen-korentsov

We have some patches about tantivy concurrent file operations issues, could please try the latest 2.5.4 image?

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @eugen-korentsov





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

  





      


        


  
    

      
          
@congqixia updated to 2.5.4, I will update here if it happens again. Thank you.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    
















  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

      

        

  
      @congqixia
  
    congqixia
    

      








      


  


  

    Labels
  



    

      

    kind/bug

  Issues or changes related a bug
  

    priority/critical-urgent

  Highest priority. Must be actively worked on as someone's top priority right now.
  

    severity/critical

  Critical, lead to crash, data missing, wrong result, function totally doesn't work.
  

    triage/accepted

  Indicates an issue or PR is ready to be actively worked on.








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      
    
  
    2.6.0
  





    
      
          


  

    
      

        
    

    Development
  




          
    
No branches or pull requests







    
  




      

    

  
      

  

    

      4 participants
    

    

        
          @zhuwenxing 
        
          @yanliang567 
        
          @congqixia 
        
          @eugen-korentsov