0.17 - 0.19 segfaults on powerpc64le some of the time, can't reproduce in gdb

[infinity0]

With current stable:

(sid_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo$ for i in {1..10}; do RUST_BACKTRACE=1 bin/cargo -V; done
cargo 0.19.0 (28d1d60d4 2017-05-16)
Segmentation fault
Segmentation fault
cargo 0.19.0 (28d1d60d4 2017-05-16)
cargo 0.19.0 (28d1d60d4 2017-05-16)
Segmentation fault
cargo 0.19.0 (28d1d60d4 2017-05-16)
cargo 0.19.0 (28d1d60d4 2017-05-16)
Segmentation fault
Segmentation fault

(sid_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo$ for i in {1..10}; do RUST_BACKTRACE=1 rust-gdb -q -ex 'run -V' -ex bt -ex quit bin/cargo; done                                                                              
Reading symbols from bin/cargo...done.
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
cargo 0.19.0 (28d1d60d4 2017-05-16)
[Inferior 1 (process 21249) exited normally]
No stack.
[.. same result 10 times ..]

Similar sort of thing with Debian 0.17, reported here

[infinity0]

Got a stacktrace from a core dump:

(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ ulimit -c unlimited
(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ cargo/bin/cargo -V
Segmentation fault (core dumped)
(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q cargo/bin/cargo core
Reading symbols from cargo/bin/cargo...done.
[New LWP 15047]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
Core was generated by `cargo/bin/cargo -V '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000025daa598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
(gdb) bt
#0  0x0000000025daa598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
#1  0x0000000025db9b88 in regex_syntax::ExprBuilder::parse::h99b8f77ea0168dc6 ()
#2  0x0000000025d5df94 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
#3  0x0000000025d8aee0 in regex::re_unicode::Regex::new::h499bc833cb50ff25 ()
#4  0x0000000025bcfd20 in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::hf95771c6d594ccc3 ()
#5  0x0000000025f8f058 in std::sync::once::Once::call_inner () at /checkout/src/libstd/sync/once.rs:307
#6  0x0000000025be3d38 in docopt::parse::Parser::parse::h46a0b058a05c5e45 ()
#7  0x0000000025bdeaa4 in docopt::parse::Parser::new::h07ab0f370bc545c9 ()
#8  0x00000000258a2f10 in cargo::call_main_without_stdin::h22117ce339cc3d61 ()
#9  0x00000000258b16d4 in cargo::main::h7ce25b0d0a8f899f ()
#10 0x0000000025f9e21c in core::ops::FnOnce::call_once<fn(),()> () at /checkout/src/libcore/ops.rs:2626
#11 std::panicking::try::do_call<fn(),()> () at /checkout/src/libstd/panicking.rs:454
#12 0x0000000025fa70f0 in panic_unwind::__rust_maybe_catch_panic () at /checkout/src/libpanic_unwind/lib.rs:98
#13 0x0000000025f9efdc in std::panicking::try<(),fn()> () at /checkout/src/libstd/panicking.rs:433
#14 std::panic::catch_unwind<fn(),()> () at /checkout/src/libstd/panic.rs:361
#15 std::rt::lang_start () at /checkout/src/libstd/rt.rs:57
#16 0x00000000259085f8 in main ()

[alexcrichton]

Mind taking a look at the faulting instruction as well? That may illuminate what's going on here.

[infinity0]

I can reproduce it in gdb if I enable ASLR. I observed many different segfaults:

• regex_syntax::parser::Parser::parse_expr
• regex::exec::ExecBuilder::build
• regex_syntax::Expr::simplify::simp
• regex_syntax::parser::Parser::parse_escape::hba8ca67e5013d7ff
• std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::hf95771c6d594ccc3
• regex::re_builder::unicode::RegexBuilder::new::h94340d57c1a8f048

etc etc. The first three are given in more detail below:

(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q -ex 'set disable-randomization off' -ex 'run -V' cargo/bin/cargo
Reading symbols from cargo/bin/cargo...done.
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000045b0a598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
(gdb) bt   
#0  0x0000000045b0a598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
#1  0x0000000045b19b88 in regex_syntax::ExprBuilder::parse::h99b8f77ea0168dc6 ()
#2  0x0000000045abdf94 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
#3  0x0000000045aeaee0 in regex::re_unicode::Regex::new::h499bc833cb50ff25 ()
#4  0x000000004592fd20 in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::hf95771c6d594ccc3 ()
#5  0x0000000045cef058 in std::sync::once::Once::call_inner () at /checkout/src/libstd/sync/once.rs:307
#6  0x0000000045943d38 in docopt::parse::Parser::parse::h46a0b058a05c5e45 ()
#7  0x000000004593eaa4 in docopt::parse::Parser::new::h07ab0f370bc545c9 ()
#8  0x0000000045602f10 in cargo::call_main_without_stdin::h22117ce339cc3d61 ()
#9  0x00000000456116d4 in cargo::main::h7ce25b0d0a8f899f ()
#10 0x0000000045cfe21c in core::ops::FnOnce::call_once<fn(),()> () at /checkout/src/libcore/ops.rs:2626
#11 std::panicking::try::do_call<fn(),()> () at /checkout/src/libstd/panicking.rs:454
#12 0x0000000045d070f0 in panic_unwind::__rust_maybe_catch_panic () at /checkout/src/libpanic_unwind/lib.rs:98
#13 0x0000000045cfefdc in std::panicking::try<(),fn()> () at /checkout/src/libstd/panicking.rs:433
#14 std::panic::catch_unwind<fn(),()> () at /checkout/src/libstd/panic.rs:361
#15 std::rt::lang_start () at /checkout/src/libstd/rt.rs:57
#16 0x00000000456685f8 in main ()
(gdb) disas
Dump of assembler code for function _ZN12regex_syntax6parser6Parser10parse_expr17h69b18aabf3de27d8E:
   0x0000000045b0a580 <+0>:     addis   r2,r12,65
   0x0000000045b0a584 <+4>:     addi    r2,r2,25472
   0x0000000045b0a588 <+8>:     mfcr    r12
   0x0000000045b0a58c <+12>:    mflr    r0
   0x0000000045b0a590 <+16>:    std     r0,16(r1)
   0x0000000045b0a594 <+20>:    stw     r12,8(r1)
=> 0x0000000045b0a598 <+24>:    stdu    r1,-12336(r1)
   0x0000000045b0a59c <+28>:    std     r14,12192(r1)
   0x0000000045b0a5a0 <+32>:    std     r19,12232(r1)
   0x0000000045b0a5a4 <+36>:    std     r24,12272(r1)
   0x0000000045b0a5a8 <+40>:    std     r31,12328(r1)
   0x0000000045b0a5ac <+44>:    li      r19,80
   0x0000000045b0a5b0 <+48>:    li      r5,64
   0x0000000045b0a5b4 <+52>:    li      r14,48
   0x0000000045b0a5b8 <+56>:    li      r31,32
   0x0000000045b0a5bc <+60>:    li      r24,16
   0x0000000045b0a5c0 <+64>:    std     r3,688(r1)
[..]
(gdb) info registers
r0             0x45b19b88       1169267592
r1             0x3fffdf7d05f0   70368198723056
r2             0x45f20900       1173489920
r3             0x3fffdf7d0668   70368198723176
r4             0x3fffdf7d06d0   70368198723280
r5             0x100    256
r6             0x1002ade68b0    1100230846640
r7             0x1002ade67b0    1100230846384
r8             0x0      0
r9             0x3fffa2e50d28   70367182130472
r10            0x0      0
r11            0x0      0
r12            0x22064222       570835490
r13            0x3fffa2f8de70   70367183429232
r14            0x0      0
r15            0x3fffdf7d3b30   70368198736688
r16            0x0      0
r17            0x1002ade6740    1100230846272
r18            0x0      0
r19            0x100    256
r20            0x1002ade679c    1100230846364
r21            0x39     57
r22            0x1002ade6799    1100230846361
r23            0x1002ade6799    1100230846361
r24            0x100    256
r25            0x28     40
r26            0x0      0
r27            0x48     72
r28            0x3fffdf7d0668   70368198723176
r29            0x3fffdf7d06d0   70368198723280
r30            0x3fffdf7d3b28   70368198736680
r31            0x1      1
pc             0x45b0a598       0x45b0a598 <regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8+24>
msr            0x800000010000d033       9223372041149796403
cr             0x22064222       570835490
lr             0x45b19b88       0x45b19b88 <regex_syntax::ExprBuilder::parse::h99b8f77ea0168dc6+840>
ctr            0x3fffa2d31980   70367180953984
xer            0x0      0
orig_r3        0xc0000000006f1300       -4611686018420108544

(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q -ex 'set disable-randomization off' -ex 'run -V' cargo/bin/cargo
Reading symbols from cargo/bin/cargo...done.
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000042a0dcf8 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
(gdb) bt
#0  0x0000000042a0dcf8 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
#1  0x0000000042a3aee0 in regex::re_unicode::Regex::new::h499bc833cb50ff25 ()
#2  0x000000004287fd20 in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::hf95771c6d594ccc3 ()
#3  0x0000000042c3f058 in std::sync::once::Once::call_inner () at /checkout/src/libstd/sync/once.rs:307
#4  0x0000000042893d38 in docopt::parse::Parser::parse::h46a0b058a05c5e45 ()
#5  0x000000004288eaa4 in docopt::parse::Parser::new::h07ab0f370bc545c9 ()
#6  0x0000000042552f10 in cargo::call_main_without_stdin::h22117ce339cc3d61 ()
#7  0x00000000425616d4 in cargo::main::h7ce25b0d0a8f899f ()
#8  0x0000000042c4e21c in core::ops::FnOnce::call_once<fn(),()> () at /checkout/src/libcore/ops.rs:2626
#9  std::panicking::try::do_call<fn(),()> () at /checkout/src/libstd/panicking.rs:454
#10 0x0000000042c570f0 in panic_unwind::__rust_maybe_catch_panic () at /checkout/src/libpanic_unwind/lib.rs:98
#11 0x0000000042c4efdc in std::panicking::try<(),fn()> () at /checkout/src/libstd/panicking.rs:433
#12 std::panic::catch_unwind<fn(),()> () at /checkout/src/libstd/panic.rs:361
#13 std::rt::lang_start () at /checkout/src/libstd/rt.rs:57
#14 0x00000000425b85f8 in main ()
(gdb) disas
Dump of assembler code for function _ZN5regex4exec11ExecBuilder5build17hcca5817b4c570623E:
   0x0000000042a0dce0 <+0>:     addis   r2,r12,70
   0x0000000042a0dce4 <+4>:     addi    r2,r2,11296
   0x0000000042a0dce8 <+8>:     mfcr    r12
   0x0000000042a0dcec <+12>:    mflr    r0
   0x0000000042a0dcf0 <+16>:    std     r0,16(r1)
   0x0000000042a0dcf4 <+20>:    stw     r12,8(r1)
=> 0x0000000042a0dcf8 <+24>:    stdu    r1,-16688(r1)
   0x0000000042a0dcfc <+28>:    std     r15,16552(r1)
   0x0000000042a0dd00 <+32>:    std     r21,16600(r1)
   0x0000000042a0dd04 <+36>:    std     r23,16616(r1)
   0x0000000042a0dd08 <+40>:    ld      r23,16(r4)
   0x0000000042a0dd0c <+44>:    ld      r21,0(r4)
   0x0000000042a0dd10 <+48>:    ld      r15,8(r4)
   0x0000000042a0dd14 <+52>:    cmpldi  r23,0
   0x0000000042a0dd18 <+56>:    std     r24,16624(r1)
   0x0000000042a0dd1c <+60>:    mr      r24,r3
   0x0000000042a0dd20 <+64>:    std     r14,16544(r1)
   0x0000000042a0dd24 <+68>:    std     r16,16560(r1)
[..]
(gdb) info registers
r0             0x42a3aee0       1118023392
r1             0x3fffddf01c40   70368172710976
r2             0x42e70900       1122437376
r3             0x3fffddf01cd0   70368172711120
r4             0x3fffddf01d10   70368172711184
r5             0x0      0
r6             0x0      0
r7             0xffffffffffffffdf       18446744073709551583
r8             0xffffffffffffffe0       18446744073709551584
r9             0x1      1
r10            0x0      0
r11            0x1000d736798    1099737294744
r12            0x24400439       608175161
r13            0x3fff9cc6de70   70367079489136
r14            0x0      0
r15            0x0      0
r16            0x0      0
r17            0x1      1
r18            0xfffffffffffffffc       18446744073709551612
r19            0x0      0
r20            0x0      0
r21            0x1      1
r22            0x2      2
r23            0x3fffddf01f3a   70368172711738
r24            0x3fffddf03398   70368172716952
r25            0x49b7760a37dc72aa       5311844071765734058
r26            0x0      0
r27            0x3fffddf01f38   70368172711736
r28            0x3fffddf01cd0   70368172711120
r29            0x3fffddf01d10   70368172711184
r30            0x3fffddf01e10   70368172711440
r31            0x3fffddf01ec0   70368172711616
pc             0x42a0dcf8       0x42a0dcf8 <regex::exec::ExecBuilder::build::hcca5817b4c570623+24>
msr            0x800000010000d033       9223372041149796403
cr             0x24400439       608175161
lr             0x42a3aee0       0x42a3aee0 <regex::re_unicode::Regex::new::h499bc833cb50ff25+112>
ctr            0x3fff9ca1f7a0   70367077070752
xer            0x0      0
orig_r3        0xc0000000006f1300       -4611686018420108544
trap           0x300    768

(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q -ex 'set disable-randomization off' -ex 'run -V' cargo/bin/cargo
Reading symbols from cargo/bin/cargo...done.
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x000000002ab89d58 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
(gdb) bt
#0  0x000000002ab89d58 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#1  0x000000002ab8ae98 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#2  0x000000002ab8a2d0 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#3  0x000000002ab89fb0 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#4  0x000000002ab8a2d0 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#5  0x000000002ab89fb0 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#6  0x000000002ab8ae98 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#7  0x000000002ab8a2d0 in regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b ()
#8  0x000000002ab89c94 in regex_syntax::ExprBuilder::parse::h99b8f77ea0168dc6 ()
#9  0x000000002ab2df94 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
#10 0x000000002ab5aee0 in regex::re_unicode::Regex::new::h499bc833cb50ff25 ()
#11 0x000000002a99f980 in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::hd8f0a068b027f815 ()
#12 0x000000002ad5f058 in std::sync::once::Once::call_inner () at /checkout/src/libstd/sync/once.rs:307
#13 0x000000002a9b22bc in docopt::parse::Parser::parse::h46a0b058a05c5e45 ()
#14 0x000000002a9aeaa4 in docopt::parse::Parser::new::h07ab0f370bc545c9 ()
#15 0x000000002a672f10 in cargo::call_main_without_stdin::h22117ce339cc3d61 ()
#16 0x000000002a6816d4 in cargo::main::h7ce25b0d0a8f899f ()
#17 0x000000002ad6e21c in core::ops::FnOnce::call_once<fn(),()> () at /checkout/src/libcore/ops.rs:2626
#18 std::panicking::try::do_call<fn(),()> () at /checkout/src/libstd/panicking.rs:454
#19 0x000000002ad770f0 in panic_unwind::__rust_maybe_catch_panic () at /checkout/src/libpanic_unwind/lib.rs:98
#20 0x000000002ad6efdc in std::panicking::try<(),fn()> () at /checkout/src/libstd/panicking.rs:433
#21 std::panic::catch_unwind<fn(),()> () at /checkout/src/libstd/panic.rs:361
#22 std::rt::lang_start () at /checkout/src/libstd/rt.rs:57
#23 0x000000002a6d85f8 in main ()
(gdb) disas
Dump of assembler code for function _ZN12regex_syntax4Expr8simplify4simp17hf9c81225df7c5c8bE:
   0x000000002ab89d40 <+0>:     addis   r2,r12,64
   0x000000002ab89d44 <+4>:     addi    r2,r2,27584
   0x000000002ab89d48 <+8>:     mfcr    r12
   0x000000002ab89d4c <+12>:    mflr    r0
   0x000000002ab89d50 <+16>:    std     r0,16(r1)
   0x000000002ab89d54 <+20>:    stw     r12,8(r1)
=> 0x000000002ab89d58 <+24>:    stdu    r1,-2768(r1)
   0x000000002ab89d5c <+28>:    std     r14,2624(r1)
   0x000000002ab89d60 <+32>:    std     r15,2632(r1)
   0x000000002ab89d64 <+36>:    std     r23,2696(r1)
   0x000000002ab89d68 <+40>:    std     r25,2712(r1)
   0x000000002ab89d6c <+44>:    li      r15,32
   0x000000002ab89d70 <+48>:    li      r14,16
   0x000000002ab89d74 <+52>:    mr      r25,r6
   0x000000002ab89d78 <+56>:    mr      r23,r5
   0x000000002ab89d7c <+60>:    cmpld   r23,r25
   0x000000002ab89d80 <+64>:    std     r30,2752(r1)
[..]
(gdb) info registers
r0             0x2ab8ae98       716746392
r1             0x3fffc2580630   70367709759024
r2             0x2af90900       720963840
r3             0x3fffc25807a8   70367709759400
r4             0x3fffc2580ff0   70367709761520
r5             0x7      7
r6             0xc8     200
r7             0x30     48
r8             0x1      1
r9             0x3fff9af40d28   70367048895784
r10            0x0      0
r11            0x3fffc2582fb0   70367709769648
r12            0x88464c82       2286308482
r13            0x3fff9b07de70   70367050194544
r14            0x10     16
r15            0x20     32
r16            0x10     16
r17            0x10037753ba0    1100442057632
r18            0x3fffc2581770   70367709763440
r19            0x20     32
r20            0x100000112      4294967570
r21            0x1      1
r22            0x1      1
r23            0x6      6
r24            0x100    256
r25            0xc8     200
r26            0x3fffc2581658   70367709763160
r27            0x10037753c50    1100442057808
r28            0x3fffc2581520   70367709762848
r29            0x3771ef9000000000       3995237746208669696
r30            0x3fffc2581420   70367709762592
r31            0xb      11
pc             0x2ab89d58       0x2ab89d58 <regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b+24>
msr            0x800000010000d033       9223372041149796403
cr             0x88464c82       2286308482
lr             0x2ab8ae98       0x2ab8ae98 <regex_syntax::Expr::simplify::simp::hf9c81225df7c5c8b+4440>
ctr            0x2ab8ae44       716746308
xer            0x0      0
orig_r3        0x3fff9ae1e814   70367047706644
trap           0x300    768

[infinity0]

I did not observe this until today, before which I had run cargo many times successfully on the same machine, and it also worked on the build daemons for Debian-experimental. As of today, these segfaults are appearing on these same machines that had run it successfully in the past. I then tried rust's own binary tarball, as well as on previous chroots (of Debian-stable, Debian-testing) that in theory should be the same between today and the previous few days, but the segfaults now apparently also occur there. So I'm thoroughly confused.

[alexcrichton]

It may be worth trying to narrow this down, I highly doubt the bug is actually in Cargo itself. The faulting instruction from that last dump is

stdu    r1,-2768(r1)

which IBM claims is a storage instruction. In that case we can substitute that as:

stdu    70367709759024,-2768(70367709759024)

# or ...

stdu    70367709759024,(70367709756256)

# or ...

stdu    0x3fffc2580630,(0x3fffc257fb60)

The memory address doesn't look misaligned or anything, so looks pretty good to me. It could be that the address is straight up not mapped (this is a bland segfault) perhaps. Either way this sort of smells like an LLVM or an allocator bug?

[infinity0]

Looks like it's a stack overflow, ulimit -s says 8192 (kb) if I increase it to unlimited or 24576 on this machine, the segfaults go away.

[arielb1]

Can you try to find out which function is using so much stack? (print the stack pointer for each frame using frame #FRAME; print $r1 and see where it is increasing)

[infinity0]

It does not look like the stack is going above 8MB though:

(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q -ex 'set disable-randomization off' -ex 'run -V' cargo/bin/cargo
Reading symbols from cargo/bin/cargo...done.
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000051b2a598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
(gdb) bt
#0  0x0000000051b2a598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
#1  0x0000000051b39b88 in regex_syntax::ExprBuilder::parse::h99b8f77ea0168dc6 ()
#2  0x0000000051addf94 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
#3  0x0000000051b0aee0 in regex::re_unicode::Regex::new::h499bc833cb50ff25 ()
#4  0x000000005194fd20 in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::hf95771c6d594ccc3 ()
#5  0x0000000051d0f058 in std::sync::once::Once::call_inner () at /checkout/src/libstd/sync/once.rs:307
#6  0x0000000051963d38 in docopt::parse::Parser::parse::h46a0b058a05c5e45 ()
#7  0x000000005195eaa4 in docopt::parse::Parser::new::h07ab0f370bc545c9 ()
#8  0x0000000051622f10 in cargo::call_main_without_stdin::h22117ce339cc3d61 ()
#9  0x00000000516316d4 in cargo::main::h7ce25b0d0a8f899f ()
#10 0x0000000051d1e21c in core::ops::FnOnce::call_once<fn(),()> () at /checkout/src/libcore/ops.rs:2626
#11 std::panicking::try::do_call<fn(),()> () at /checkout/src/libstd/panicking.rs:454
#12 0x0000000051d270f0 in panic_unwind::__rust_maybe_catch_panic () at /checkout/src/libpanic_unwind/lib.rs:98
#13 0x0000000051d1efdc in std::panicking::try<(),fn()> () at /checkout/src/libstd/panicking.rs:433
#14 std::panic::catch_unwind<fn(),()> () at /checkout/src/libstd/panic.rs:361
#15 std::rt::lang_start () at /checkout/src/libstd/rt.rs:57
#16 0x00000000516885f8 in main ()
(gdb) frame 0
#0  0x0000000051b2a598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
(gdb) print $r1
$1 = 70367743056496
(gdb) frame 16
#16 0x00000000516885f8 in main ()
(gdb) print $r1
$2 = 70367743084176

The machine was recently updated with a kernel security fix for DSA 3886 and some other Debian people think the fix for CVE-2017-1000364 might be the culprit here.

Disabling ASLR with setarch $(arch) -R, without increasing the stack, also avoids these segfaults. BTW, the page size on this machine is 64KB. Not sure if any of this info is useful.

[infinity0]

Not sure if I'm reading this correctly, but it looks like the stack is getting set to be zero-sized? From another run:

(gdb) frame 0
#0  0x00000000290fdcf8 in regex::exec::ExecBuilder::build::hcca5817b4c570623 ()
(gdb) p/x $r1
$1 = 0x3ffffada12a0
(gdb) frame 14
#14 0x0000000028ca85f8 in main ()
(gdb) p/x $r1
$2 = 0x3ffffada3ba0

From /proc/$pid/maps:

3ffffadb0000-3ffffadb0000 rw-p 00000000 00:00 0                          [stack]

[arielb1]

How does maps look like for that executable?

[arielb1]

Fresh off oss-security: http://www.openwall.com/lists/oss-security/2017/06/22/12 - possibly relevant

[infinity0]

Thanks, I can reproduce the segfault with the test program on this segfaulting machine, I will file a bug report to Debian. FWIW I stepped through the entire program and the segfault does in fact occur in the memory allocator but for some reason this wasn't properly captured in the stack traces above:

(experimental_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q -ex 'set disable-randomization off' -ex 'break main' -ex 'run -V' cargo/bin/cargo
Reading symbols from cargo/bin/cargo...done.
Breakpoint 1 at 0x1185e4
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Breakpoint 1, 0x000000002ed085e4 in main ()
(gdb) step
Single stepping until exit from function main,
which has no line number information.
std::rt::lang_start () at /checkout/src/libstd/rt.rs:40
40          sys::init();
[..]
alloc::raw_vec::RawVec<u8>::allocate<u8> () at /checkout/src/liballoc/raw_vec.rs:103
103                 let ptr = if alloc_size == 0 {
(gdb)
110                         heap::allocate(alloc_size, align)
(gdb)
alloc::heap::allocate () at /checkout/src/liballoc/heap.rs:59
59          __rust_allocate(size, align)
(gdb)
alloc_system::__rust_allocate () at /checkout/src/liballoc_system/lib.rs:44
44          unsafe { imp::allocate(size, align) }
(gdb)
alloc_system::imp::allocate () at /checkout/src/liballoc_system/lib.rs:89
89              if align <= MIN_ALIGN {
(gdb)
90                  libc::malloc(size as libc::size_t) as *mut u8
(gdb)

Program received signal SIGSEGV, Segmentation fault.
0x000000002f1aa598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()

[.. then a lot of signal handling code runs before the program actually exits ..]

Hopefully this is indeed the correct fix, but it'll take a while to confirm - I'll get back to this ticket when the relevant people apply the relevant patches.

[infinity0]

Although, their test program still segfaults when I disable ASLR or increase the stack size; whereas cargo's segfault goes away. Do you think this is still compatible with the idea that this linux issue is what's causing the cargo segfaults?

[infinity0]

Unfortunately the issue still persists with a (supposedly) patched kernel. The C code in that oss-security link no longer segfaults, and cargo -V no longer segfaults, however cargo build still segfaults.

I can no longer avoid the segfaults by disabling ASLR, but using a larger stack still works. (edit: corrected 2017-06-30)

[ashleygwilliams]

@arielb1 do you think this is a rust or a cargo issue? looks like the rust issue has more activity- wondering if this can be safely closed? (thanks for filing @infinity0! 👽 👾 🎇 )

[cuviper]

I doubt there's anything particular to Cargo here.

[arielb1]

This is a duplicate of rust-lang/rust#43052

[infinity0]

Should be fixed for the next release yes, closing.