Skip to content

Sina abruptly ends with "Killed" and no information. #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TorkelE opened this issue Jul 7, 2019 · 9 comments
Closed

Sina abruptly ends with "Killed" and no information. #74

TorkelE opened this issue Jul 7, 2019 · 9 comments

Comments

@TorkelE
Copy link

TorkelE commented Jul 7, 2019

So I have some files with rRNAs which I wish to align (in fasta files, different domains and type of rRNAs, but always the same in each file).

I try to use sina and the following command (on ubuntu 18.04, conda version 6.11.4, sina version 1.6.0, python version 3.7.3):

sina -i Archaea/5SrRNAs.fasta -o Archaea/5SrRNAs_aligned.fasta --db SILVA_132_LSUParc_12_12_17_opt.arb

Things then starts up nicely, but in the middle of building index it just ends with the word "Killed", and no information of why. I have tried several times, here is the results:

13:39:21 [famfinder] Option --ptdb deprecated; please use --db instead
13:39:37 [Search (internal)] No cached index found.
Processing: 0 [00:00:14]████████████████| 907382/907382 [00:00:00 / 00:00:00]
Building Index:  75% |█████████████▋    | 684833/907382 [00:22:30 / 00:07:18]Killed
14:06:01 [Search (internal)] No cached index found.
Processing: 0 [00:00:15]██████████████████████████████████████████| 907382/907382 [00:00:00 / 00:00:00]
Killedng Index:  74% |████████████████████████████████▌           | 670767/907382 [00:21:41 / 00:07:39]

(just copied two attempts out from the terminal)

However, I do not actually get any information on what have gone wrong. I would be happ if someone could help me get the program to work.

@epruesse
Copy link
Owner

epruesse commented Jul 7, 2019 via email

@TorkelE
Copy link
Author

TorkelE commented Jul 10, 2019

Thanks for the quick reply (unlike my, I will blame a travel).

I did run top when I did my last tests, %CPU hitting about 125 and %MEM slowly working up to ~50 before the end. At least this is what I remember, might only have been when I was looking at errors in my old sina 1.2 installation.

Anyway, when I am double checking this now the error is gone, and the alignment seem to be successful. I'm confused, but it is good. I will continue to run alignments on all my files now, but hopefully things continue to turn out well.

@TorkelE
Copy link
Author

TorkelE commented Jul 12, 2019

Just updating again. I have tried aligning several datasets (from Archaea, Bacteria, and Eukaryote). All types of rRNAs as well (5S, 5.8S, 16S, 18S, 23S, and 28S). To run the program I use this line (after activating sina in the terminal).

sina -i [Domain]/[Database Nbr]/[rRNA type].fasta -o [Domain]/[Dataset Nbr]/[rRNA type]_aligned.fasta --db [Database]

[Domain] is either Arcahea, Bacteria, or Eukaryote. [Dataset Nbr] just designates one of my datasets), [rRNA type] is either 5S, 16S, or 23S (for Arachaea and Bacteria) or 5S, 5.8S, 18S, or 28S (for Eukaryote). [Database] is the reference database. I have donwloaded these databases from silva. For 16 and 18S I use "SILVA_132_SSURef_NR99_13_12_17_opt.arb" and for 5S, 5.8S, 23S, and 28S I use "SILVA_132_LSURef_07_12_17_opt.arb".

When I run and watch the "top" infromation %CPU hit something like 750 and %MEM peaks at 65, but is usually around 40. If I run dmesg I get this at the end (runaligh.sh is the master script which runs the alignment for all datasets).

[88005.358426] [30337]  1001 30337     4998       68    73728        0             0 runalign.sh
[88005.358430] [30371]  1001 30371   379284     4373   831488        0             0 Web Content
[88005.358434] [30481]  1001 30481  4806717  3262927 26398720        0             0 sina
[88005.358436] Out of memory: Kill process 30481 (sina) score 720 or sacrifice child
[88005.358450] Killed process 30481 (sina) total-vm:19226868kB, anon-rss:13051708kB, file-rss:0kB, shmem-rss:0kB
[88005.827807] oom_reaper: reaped process 30481 (sina), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

(That's the last few lines of a rather large output)

If I try using gdb (on the third datasets that fails, some bacterial 23S rRNAs) I get this outpu:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
15:57:48 [SINA] Aligner ready. Processing sequences
Processing: 0 [00:00:02]█████████████████████████████████████████████████████████████████████| 198843/198843 [00:00:00 / 00:00:00]
[New Thread 0x7ffff4a57700 (LWP 31284)]
[New Thread 0x7ffff3c54700 (LWP 31285)]
[New Thread 0x7ffff3853700 (LWP 31286)]
[New Thread 0x7ffff3051700 (LWP 31289)]
[New Thread 0x7ffff3452700 (LWP 31288)]
[New Thread 0x7ffff2c50700 (LWP 31287)]
Processing:  70% |█████████████████████████████████████████████████████████                        | 482/686 [00:02:31 / 00:01:04]
16:00:20 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
16:00:20 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...
[Thread 0x7ffff3853700 (LWP 31286) exited]
[Thread 0x7ffff4a57700 (LWP 31284) exited]
[Thread 0x7ffff284f700 (LWP 31290) exited]
[Thread 0x7ffff3051700 (LWP 31289) exited]
[Thread 0x7ffff3452700 (LWP 31288) exited]
[Thread 0x7ffff3c54700 (LWP 31285) exited]
[Thread 0x7ffff2c50700 (LWP 31287) exited]
[Inferior 1 (process 31280) exited with code 01]

Next, when I run bt I simply get a response No stack..

When I run the same dataset, but adding the -v option I get a lot of printout, but this is the last bit:

16:08:54 [log] align_filter_slv: 
16:08:54 [log] align_log_slv: shifting bases to fit in 1 bases at pos 83939 to 83939;shifting bases to fit in 1 bases at pos 98552 to 98552;shifting bases to fit in 1 bases at pos 98656 to 98656;shifting bases to fit in 1 bases at pos 99188 to 99188;shifting bases to fit in 2 bases at pos 99468 to 99468;shifting bases to fit in 3 bases at pos 105528 to 105528;shifting bases to fit in 3 bases at pos 113871 to 113871;shifting bases to fit in 1 bases at pos 115049 to 115049;shifting bases to fit in 4 bases at pos 116574 to 116576;shifting bases to fit in 3 bases at pos 118688 to 118688;shifting bases to fit in 3 bases at pos 122481 to 122481;shifting bases to fit in 2 bases at pos 122501 to 122501;shifting bases to fit in 1 bases at pos 127568 to 127568;shifting bases to fit in 1 bases at pos 127576 to 127576;shifting bases to fit in 1 bases at pos 128675 to 128675;shifting bases to fit in 1 bases at pos 128989 to 128989;total inserted bases=87;longest insertion=21;total inserted bases before shifting=1;scoring: raw=-3378.21, weight=-4959.91, query-len=2891, aligned-bases=2891, score=0.681104; 
16:08:54 [log] align_quality_slv: 68
16:08:54 [log] align_startpos_slv: 69268
16:08:54 [log] align_stoppos_slv: 129008
16:08:54 [log] aligned_slv: 2019-07-12 15:08:54
16:08:54 [log] nuc: 2891
16:08:54 [log] turn: turn-check disabled
16:08:54 [log] sequence_number: 490
16:08:54 [log] sequence_identifier: |NZ_JXTV01000014|Genomes/bact/files/GCF_001545275.1_ASM154527v1_genomic.gbff.gz|23S|1|2922
16:08:54 [log] align_bp_score_slv: 119
16:08:54 [log] align_cutoff_head_slv: 0
16:08:54 [log] align_cutoff_tail_slv: 0
16:08:54 [log] align_family_slv: CP014344.698224:858.00 MKAI01000022.368:858.00 MJXK01000006.2787:858.00 MJXK01000007.370:858.00 MJXK01000017.152076:858.00 NAAE01000008.364:858.00 MJXK01000011.42884:858.00 CP010306.400533:858.00 JAKQ01000006.103899:858.00 CP006006.1546603:858.00 NAAG01000001.50601:858.00 CP006006.1242425:858.00 CP014344.394294:858.00 CP014344.39689:858.00 CP010306.41400:858.00 CP006006.880002:858.00 MJXK01000007.294513:858.00 MKHW01000051.160:858.00 MKAI01000018.305898:858.00 AUUI01000015.2761:858.00 MKBN01000015.2763:858.00 CP010306.691819:858.00 JOUH01000014.368:858.00 NAAH01000005.140125:858.00 JAKL01000006.368:858.00 MKAI01000019.2763:858.00 LBEC01000027.369:858.00 MKBK01000048.191536:858.00 LLWL01000010.392:858.00 MKBN01000014.2763:858.00 MKAU01000042.369:858.00 MKAU01000042.301463:858.00 MKBD01000009.103424:858.00 MKBD01000031.51203:858.00 MKBN01000013.2763:858.00 MKBN01000012.368:858.00 MKAU01000041.2761:858.00 LBEN01000019.23:858.00 MKAI01000021.368:858.00 MKBD01000027.2762:858.00 
16:08:54 [log] align_filter_slv: 
16:08:54 [log] align_log_slv: copied alignment from (longer) template sequence CP014344:698224; 
16:08:54 [log] align_quality_slv: 100
16:08:54 [log] align_startpos_slv: 66151
16:08:54 [log] align_stoppos_slv: 129018
16:08:54 [log] aligned_slv: 2019-07-12 15:08:54
16:08:54 [log] nuc: 2921
16:08:54 [log] turn: turn-check disabled
16:08:56 [FASTA I/O] read 490 sequences from 981 lines
16:08:56 [FASTA I/O] wrote 483 sequences (0 excluded, 0 relatives)
Processing:  70% |████████████████████████████████████████████████████████▉                        | 483/688 [00:02:48 / 00:01:11]
16:08:56 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
16:08:56 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...
16:08:57 [Search (internal)] Timings for Kmer Search: 17.420s (408 calls, load query: 0.000s, count kmers: 0.000s, store: 16.907s)

Finally I show the output after the fails from the various files that failed. In each case, if I rerun the program they all fail at a similar place place (I show the output of a second run, hence the dual texts for each dataset).

A set of bacterial 23S rRNAs.

15:14:58 [SINA] Aligner ready. Processing sequences
Processing:  69% |██████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                                | 264/385 [00:00:53 / 00:00:24]
15:15:53 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:15:53 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...
15:49:31 [SINA] Aligner ready. Processing sequences
Processing:  69% |███████████████████████████████████▋                | 264/385 [00:00:59 / 00:00:27]
15:50:29 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:50:29 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...

A set of bacterial 16S rRNAs.

15:16:08 [SINA] Aligner ready. Processing sequences
Processing:  81% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                            | 436/535 [00:00:46 / 00:00:10]
15:16:41 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:16:41 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_SSURef_NR99_13_12_17_opt.arb"' ...
15:50:43 [SINA] Aligner ready. Processing sequences
Processing:  82% |██████████████████████████████████████████▍         | 438/537 [00:00:52 / 00:00:11]
15:51:23 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:51:23 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_SSURef_NR99_13_12_17_opt.arb"' ...

A set of bacterial 23S rRNAs.

15:16:45 [SINA] Aligner ready. Processing sequences
Processing:  70% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                             | 486/690 [00:01:50 / 00:00:46]
15:18:38 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:18:38 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...
15:51:29 [SINA] Aligner ready. Processing sequences
Processing:  70% |████████████████████████████████████▋               | 487/691 [00:02:11 / 00:00:55]
15:53:41 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:53:41 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...

A set of eukaryote 28S rRNAs.

15:18:43 [SINA] Aligner ready. Processing sequences
Processing:   6% |█████████▏                                                                                                                                                  | 13/220 [00:00:08 / 00:02:22]
./runalign.sh: line 84: 29456 Killed                 
15:53:47 [SINA] Aligner ready. Processing sequences
Processing:   6% |███▏                                                 | 13/220 [00:00:10 / 00:02:42]
./runalign.sh: line 84: 31124 Killed 

A set of eukaryote 5.8S rRNAs.

15:18:55 [SINA] Aligner ready. Processing sequences
Processing: 0 [00:00:04]███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 198843/198843 [00:00:00 / 00:00:00]
15:18:55 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:18:55 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...
[*** LOG ERROR ***] [2019-07-12 15:18:56] [Search (internal)] Tried to add incompatible timers

15:54:01 [SINA] Aligner ready. Processing sequences
Processing: 0 [00:00:04]████████████████████████████████████████| 198843/198843 [00:00:00 / 00:00:00]
15:54:01 [SINA] Error during program execution: std::bad_alloc std::bad_alloc
15:54:01 [ARB I/O] Closing ARB database '"sina-1.2.11/NewReferences/SILVA_132_LSURef_07_12_17_opt.arb"' ...
[*** LOG ERROR ***] [2019-07-12 15:54:01] [Search (internal)] Tried to add incompatible timers

(Progress bars probably of uneven size due to different sizes of the terminal winodw when I copied them out)

I hope this can be of some help, if you want I could provide files as well.

@epruesse
Copy link
Owner

epruesse commented Mar 9, 2020

Hi @TorkelE,

sorry for the very late response. The dmesg output is clear - you ran out of memory and the kernel decided that SINA needed to be killed to keep your computer running. Similarly, when you do see the std::bad_alloc, that's almost always also an out-of-memory condition, only that SINA still got to report it.

For the large SILVA SSU reference databases, SINA just needs a lot of memory to fit the database. If I ever find the time, I might write something to use an on-disk database, but that'll be slower.

@TorkelE
Copy link
Author

TorkelE commented Mar 9, 2020

Thank you very much.

@epruesse
Copy link
Owner

How much memory do you have? You might be able to either use the LTP dataset, or you could try downsampling the reference (something like sina -i bigdatabase.arb --prealigned -o downsampled.arb --select-step 2 to make a DB with just every 2nd sequence).

@TorkelE
Copy link
Author

TorkelE commented Mar 10, 2020

At that time I only had about 16GB RAM I believe. I've recently upgraded to a new computer with 128GB Ram (but haven't tried with that). Maybe that should be enough?

@epruesse
Copy link
Owner

That's definitely ample. The 16GB should actually have been enough - my laptop has no more and is working fine for most things.

@epruesse
Copy link
Owner

Closing this in favor of #87

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants