large stack arrays cause segfaults in safe code (even with ulimit unlimited)

[m4b]

Note, first make sure to set ulimit -s unlimited before running the program below.

Something like the following has problems with bad ASM stack offsets/accesses being generated due to 32-bit overflow.

Looks like this is something known in LLVM: https://groups.google.com/forum/#!topic/llvm-dev/g4sF46a47_w

This also only seems to cause segfaults in --release mode

Note however, it's not entirely clear how to easily get a directly heap allocated, large fixed size array without resorting to unsafe, box syntax, or using the vec hack.

I suspect this will come up often as a result, since many people and beginners will attempt to stack allocate a large object only to have it segfault.

Various values can be tweaked to make it not segfault in release or debug, in an entirely unintuitive way, which lends the language a less user friendly feeling, along with violating the principle of least surprise in many cases, imho.

#![feature(box_syntax)]

use std::ops::Index;
use std::ops::IndexMut;
use std::boxed::Box;

// causes problems
//const SIZE: usize = 1300;
const SIZE: usize = 1255;
const BOUND: usize = SIZE;
pub const ARR_SIZE: usize = SIZE * SIZE * BOUND;

struct Arr {
    arr: [[[u8; SIZE]; SIZE]; BOUND]
}

impl Arr {
    pub fn create () -> Arr {
        let x = [0u8; SIZE];
        let y = [x; SIZE];
        let z = [y; BOUND];
// won't cause problems in release
//        Arr { arr: [[[1u8; SIZE]; SIZE]; SIZE] }
        Arr { arr: z }
    }
}

impl Index<usize> for Arr {
    type Output = [[u8; SIZE]; SIZE];

    fn index(&self, _index: usize) -> &Self::Output {
        &self.arr[_index]
    }
}

impl IndexMut<usize> for Arr {
    fn index_mut(&mut self, _index: usize) -> &mut [[u8; SIZE]; SIZE] {
        &mut self.arr[_index]
    }
}

fn main() {
// let mut arr = box Arr::create();
    let mut arr = Arr::create();

    for i in 0..BOUND {
        for j in 0..SIZE {
            for k in 0..SIZE {
                arr[i][j][k] = (i+j+k) as u8;
            }
        }
    }
/*
    for i in 0..SIZE {
        for j in 0..SIZE {
            for k in 0..SIZE {
                println!("{}", arr[i][j][k])
            }
        }
    }
*/
    println!("res: {}", arr[1][0][0]);
    println!("res: {}", arr[1][SIZE-1][SIZE-1]);
    println!("res: {}", arr[1][SIZE-1][SIZE-1]);
}

[retep998]

On 64-bit Windows I compiled the program as provided using rustc -O and upon running it I got thread 'main' has overflowed its stack. I'm not entirely sure what ulimit -s unlimited does so I couldn't do the Windows equivalent (if there even is one). If the issue is that a segfault occurs which isn't a standard stack overflow, it would help if you described which platform/target specifically is running into the issue.

[eefriedman]

(See also #16012.)

[m4b]

Sorry, did a quick write up on Saturday. I'm running Linux:

Linux derp 4.5.4-1-ARCH #1 SMP PREEMPT Wed May 11 22:21:28 CEST 2016 x86_64 GNU/Linux

with 8GB of memory.

So I just receive a segfault; several people on IRC reported the same (one user was on Ubuntu iirc). No warning about main overflowing stack, just a regular segfault.

[m4b]

@retep998 so on Linux the stack size for userland programs is configurable as an environment variable; ulimit -all displays the current values for the current session/shell, and by default, linux sets the stack size to 8KB. So if you have an array (or struct or whatever) larger than 8KB on the stack you'll run into trouble.

Briefly googling, looks like the windows per thread stack is 1MB? https://msdn.microsoft.com/en-us/library/windows/desktop/ms686774(v=vs.85).aspx

If the above is true, I'm not sure how the program is correctly running for you? Since it should have tried to allocate a 1GB multi-dim array on the stack.

[retep998]

@m4b On Windows all functions that use over a page of stack have a stack probe at the beginning. Rust currently doesn't emit stack probes on any other platform, only Windows. This means that if you allocate a giant array on the stack, on platforms other than Windows you can very easily touch a page beyond the guard page of the stack and cause a segfault. On Windows the stack probe will first probe all the space the function needs so it'll hit the guard page before it hits memory after it, thus reliably detecting stack overflow.

So basically what I'm confused about here is whether you're encountering a segfault which is simply a stack overflow that isn't being detected as such due to the lack of stack probes (#16012) or whether this is a genuinely different issue involving incorrect codegen for functions over a certain stack size.

[m4b]

It might be both. IIRC the last explanation we had attempted to come to is that the debug version wasn't segfaulting (and properly printing the values) because it wasn't inlining the array creation code function, which then reduced the total stack size required for main.

However, @eddyb (and I assume the github user @emk (with same irc handle)) did confirm bad codegen. For example, in many cases it generates a positive stack pointer offset:

lea rdi, [rbp + 1073742336]

which then cause bad accesses presumably.

However it wasn't clear why certain values of SIZE (i.e., on my machine 1024) run fine, but when the local variables are switched around, etc., segfaults only occur in release, etc. Again lots of strangeness going on.

[m4b]

Oh, and I should add that there shouldn't be any problems with a 1GB stack on linux (once ulimit is set appropriately). This C program runs fine once ulimit is set.

#include<stdio.h>

int main () {
  int arr[1073741824] = {};
  arr[0x200000] = 0xdeadbeef;
  printf("0x%x\n", arr[0x200000]);
}

(EDIT: when you compile with GCC :P)

[emk]

However, @eddyb (and I assume the github user @emk (with same irc handle)) did confirm bad codegen.

I don't know if you mean me? I have definitely found LLVM bugs in the past, but I don't remember any recently.

[emkw]

@m4b: I'm on IRC under emk- (with minus), and here with w. @emk memory is coherent and in a good shape ;).

Some more insight on this for those interested:
LLVM allocas which sums up over 2GB in a single function starts generating memory access on the wrong end of stack.

diff3 of asm generated from rust code with 1GB, 2GB and 3GB (each minus few hundred bytes), array on stack.

====
1:30c # 1GB - negative offset
    movq    %rsi, -1073741784(%rbp)
2:31c # 2GB - negative offset
    movq    %rsi, -2147483608(%rbp)
3:31c # 3GB - positive offset (wrong side of stack beginning)
    movq    %rsi, 1073741864(%rbp)
====

Clang suffers from the same issue (generating positive offsets over 2GB of alloca in function) diff of clang-generated asm from C code with 1GB vs 3GB array on stack.

[eddyb]

I can't find a LLVM bug report matching this, although who knows how, if ever, this was reported.
cc @rust-lang/compiler

[lilith]

I'm running stable-x86_64-unknown-linux-gnu (default) rustc 1.11.0 (9b21dcd6a 2016-08-15)

For me, this works: let input_bytes = [0u8;1 * 1024 * 1024];

For me, this segfaults: `let input_bytes = [0u8;2 * 1024 * 1024];

(signal: 11, SIGSEGV: invalid memory reference)

It's not possible to catch this with catch_unwind, either. Is it possible to catch any kinds of allocation failure at the moment? I thought this had been fixed with 1.11?

Behaviour above is is the same with nightly-x86_64-unknown-linux-gnu (default) rustc 1.13.0-nightly (e07dd59ea 2016-08-25)

[Aatch]

@nathanaeljones as the name suggests, catch_unwind catches unwinding, and that's all it catches. Can't catch a segfault.

Also, your issue is probably closer to being this one: #36001

[eddyb]

@Aatch Don't think this is related to performance issues of stack arrays.
I haven't investigated this further, I should've tried to report a clang reproduction, my bad.
Here it is: https://llvm.org/bugs/show_bug.cgi?id=29171.

[alexcrichton]

I believe this was fixed with #16012, so closing.