Segfault if used with tarantool debug build

[LeonidVas]

Got segfault on both master and this patchset (#54) on tarantool 2.5.0-62-gbb7c3d167 (debug) when run tests (cd test && ./test-run.py).

#0  0x00007f3f837665f1 in raise () from /lib64/libc.so.6
#1  0x00007f3f8375053b in abort () from /lib64/libc.so.6
#2  0x000056508e4b15b7 in sig_fatal_cb (signo=11, siginfo=0x7f3f6cb7f3f0, context=0x7f3f6cb7f2c0) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/main.cc:301
#3  <signal handler called>
#4  0x00007f3f84767eab in slab_from_ptr (ptr=0x7f3f810300a0, slab_mask=0) at /home/alex/p/tarantool-meta/memcached/third_party/small/small/slab_cache.h:203
#5  0x00007f3f847680ce in mempool_free (pool=0x56508feebb50, ptr=0x7f3f810300a0) at /home/alex/p/tarantool-meta/memcached/third_party/small/small/mempool.h:274
#6  0x00007f3f84768416 in iobuf_delete (ibuf=0x7f3f810300a0, obuf=0x7f3f81040480) at /home/alex/p/tarantool-meta/memcached/memcached/internal/network.c:69
#7  0x00007f3f8476acf8 in memcached_handler (p=0x56508feec040, fd=22) at /home/alex/p/tarantool-meta/memcached/memcached/internal/memcached.c:258
#8  0x000056508e675c6b in lj_vm_ffi_call () at buildvm_x86.dasc:2578
#9  0x000056508e69bfe9 in lj_ccall_func (L=L@entry=0x4149e370, cd=<optimized out>) at lj_ccall.c:1150
#10 0x000056508e6716c8 in lj_cf_ffi_meta___call (L=0x4149e370) at lib_ffi.c:230
#11 0x000056508e673cbb in lj_BC_FUNCC () at buildvm_x86.dasc:809
#12 0x000056508e650a56 in lua_pcall (L=0x4149e370, nargs=<optimized out>, nresults=-1, errfunc=<optimized out>) at lj_api.c:1158
#13 0x000056508e5e4822 in luaT_call (L=0x4149e370, nargs=3, nreturns=-1) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lua/utils.c:1053
#14 0x000056508e5dd150 in lua_fiber_run_f (ap=0x7f3f81001470) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lua/fiber.c:443
#15 0x000056508e4b0e69 in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=0x56508e5dd0dc <lua_fiber_run_f>, ap=0x7f3f81001470)
    at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lib/core/fiber.h:782
#16 0x000056508e606a71 in fiber_loop (data=0x0) at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/src/lib/core/fiber.c:869
#17 0x000056508e7493b4 in coro_init () at /usr/src/debug/dev-db/tarantool-9999/tarantool-9999/third_party/coro/coro.c:110

#54 (comment)

[LeonidVas]

After a brief research of the bug, it seems that the problem is in different versions of the “small” library used in the tarantool and memcached module.
The proofs:

• If the "small" will be checkout to the same commit as in tarantool, the problem will disappear.
• if tarantool will be checkout to tarantool/tarantool@2c27c24, the problem will disappear.

[LeonidVas]

As I understood the problem is this:
Tarantool is export the next symbols:

└──╼ nm -gD ./src/tarantool | grep mempool
00000000007cc1a1 T mempool_alloc
00000000007cbfac T mempool_create_with_order
00000000007cc115 T mempool_destroy
00000000007d1000 T mempool_iterator_create
00000000007d104b T mempool_iterator_next
00000000007cc34d T mempool_stats

Memcached uses the next functions:
mempool_create (defined in small/small/mempool.h) which uses mempool_create_with_order()
mempool_free((defined in small/small/mempool.h)
After tarantool/small@0ce1268 the size of the struct mempool has been changed (was changed the size of mslab_tree_t).

The reason of the bug is the next: when initializing a struct mempool in mempool_create(memcached version) mempool_create_with_order is called(tarantool version with other struct mempool size), where some fields of the struct mempool are assigned. After that, the corrupted struct mempool will be used in mempool_free(memcached version). And, "BAM!", segfault when trying to use a NULL pointer.

[LeonidVas]

Hot fix:
It seems that an updating the version of the "small" library will fix memcached for tarantool> = 1.10 .

[Totktonada]

I vote to workaround this for now for tarantool >= 1.10 and leave the issue open to investigate further.

[Totktonada]

NB: Would some ABI compliance checker (say, this) help us with detecting ABI incompatibility in small?

[LeonidVas]

NB: Would some ABI compliance checker (say, this) help us with detecting ABI incompatibility in small?

If I understand correctly it's not solve the problem, because now we have a situation when the memcached version implicitly depends on the tarantool "small" version. And now it should be synchronously update or rollback.