SEVERE | memory leak in icu_replace.c

[LaurentDhont]

There appears to be a memory leak in icu_replace.c
It would be amazing if this could get fixed, it has been causing tremendous issues in our production server and it took about a year to find the cause. The cause is most likely related to this sqlite extension.

We tested a simple scenario with valgrind and this is the result:

valgrind --leak-check=full --track-origins=yes /usr/bin/sqlite3 test.sqlite
sqlite> create table test (id int, name text);
sqlite> insert into test values (1, "Laurent");
sqlite> insert into test values (2 ,"Sergiu");
sqlite> .load ./icu_replace                                                                                                                                                                                                                                                               
sqlite> select regex_replace("aur", name, "asdfasdfasdf") from test;                                                                                                                                                                                                                      
Lasdfasdfasdfent                                                                                                                                                                                                                                                                          
Sergiu                                                                                                                                                                                                                                                                                    
sqlite>                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                          
==535548==                                                                                                                                                                                                                                                                                
==535548== HEAP SUMMARY:                                                                                                                                                                                                                                                                  
==535548==     in use at exit: 401,868 bytes in 526 blocks                                                                                                                                                                                                                                
==535548==   total heap usage: 1,105 allocs, 579 frees, 998,197 bytes allocated                                                                                                                                                                                                           
==535548==                                                                                                                                                                                                                                                                                
==535548== 1,072 (200 direct, 872 indirect) bytes in 1 blocks are definitely lost in loss record 76 of 137                                                                                                                                                                                
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)                                                                                                                                                                                           
==535548==    by 0x5EADBDC: ???                                                                                                                                                                                                                                                           
==535548==    by 0x5DEE793: ???                                                                                                                                                                                                                                                           
==535548==    by 0x5DEE994: ???                                                                                                                                                                                                                                                           
==535548==    by 0x5EA1EFE: ???                                                                                                                                                                                                                                                           
==535548==    by 0x5EA2458: ???                                                                                                                                                                                                                                                           
==535548==    by 0x5EA7137: ???                                       
==535548==    by 0x5EA8AC6: ???                                       
==535548==    by 0x5EAA9D2: ???                                       
==535548==    by 0x5EAB2D9: ???                                       
==535548==    by 0x5EAB3A3: ???                                       
==535548==    by 0x5C3E681: ???                                       
==535548==                                                            
==535548== 3,492 (200 direct, 3,292 indirect) bytes in 1 blocks are definitely lost in loss record 91 of 137
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==535548==    by 0x5EADBDC: ???                                       
==535548==    by 0x5DEE57B: ???                                       
==535548==    by 0x5EA1F4B: ???                                       
==535548==    by 0x5EA2287: ???                                       
==535548==    by 0x5EA7137: ???                                       
==535548==    by 0x5EA8AC6: ???                                       
==535548==    by 0x5EAA9D2: ???                                       
==535548==    by 0x5EAB2D9: ???                                       
==535548==    by 0x5EAB3A3: ???                                       
==535548==    by 0x5C3E774: ???                                       
==535548==    by 0x5C3EC91: ???                                       
==535548==
==535548== 3,980 (200 direct, 3,780 indirect) bytes in 1 blocks are definitely lost in loss record 92 of 137
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==535548==    by 0x5EADBDC: ???                                       
==535548==    by 0x5DEE793: ???                                       
==535548==    by 0x5DEE994: ???                                       
==535548==    by 0x5EA1EFE: ???                                       
==535548==    by 0x5EA2458: ???                                       
==535548==    by 0x5EA7137: ???                                       
==535548==    by 0x5EA8AC6: ???                                       
==535548==    by 0x5EAA9D2: ???                                       
==535548==    by 0x5EAB2D9: ???                                       
==535548==    by 0x5EAB3A3: ???                                       
==535548==    by 0x5C3E6CD: ???                                       
==535548==                                                            
==535548== 6,940 (200 direct, 6,740 indirect) bytes in 1 blocks are definitely lost in loss record 116 of 137
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==535548==    by 0x5EADBDC: ???                                       
==535548==    by 0x5DEE793: ???                                       
==535548==    by 0x5DEE994: ???                                       
==535548==    by 0x5EA1EFE: ???                                       
==535548==    by 0x5EA2458: ???                                       
==535548==    by 0x5EA7137: ???                                       
==535548==    by 0x5EA8AC6: ???                                       
==535548==    by 0x5EAA9D2: ???                                       
==535548==    by 0x5EAB2D9: ???                                       
==535548==    by 0x5EAB3A3: ???                                       
==535548==    by 0x5C3E635: ???                                       
==535548==                                                            
==535548== 15,356 (200 direct, 15,156 indirect) bytes in 1 blocks are definitely lost in loss record 133 of 137
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==535548==    by 0x5EADBDC: ???                                       
==535548==    by 0x5DEAA9B: ???                                       
==535548==    by 0x5EDA4B6: ???                                       
==535548==    by 0x5DEE46D: ???                                       
==535548==    by 0x5EA1E78: ???                                       
==535548==    by 0x5EA2231: ???                                       
==535548==    by 0x5EA7137: ???                                       
==535548==    by 0x5EA8AC6: ???                                       
==535548==    by 0x5EAA9D2: ???                                       
==535548==    by 0x5EAB2D9: ???                                       
==535548==    by 0x5EAB3A3: ???                                       
==535548==                                                            
==535548== 24,292 (200 direct, 24,092 indirect) bytes in 1 blocks are definitely lost in loss record 135 of 137
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==535548==    by 0x5EADBDC: ???                                       
==535548==    by 0x5DEAA9B: ???                                       
==535548==    by 0x5EDA4B6: ???                                       
==535548==    by 0x5DEE46D: ???                                       
==535548==    by 0x5DEE84A: ???                                       
==535548==    by 0x5DEE994: ???                                       
==535548==    by 0x5EA1EFE: ???                                       
==535548==    by 0x5EA2458: ???                                       
==535548==    by 0x5EA7137: ???                                       
==535548==    by 0x5EA8AC6: ???                                       
==535548==    by 0x5EAA9D2: ???                                       
==535548==
==535548== 32,624 (3,840 direct, 28,784 indirect) bytes in 1 blocks are definitely lost in loss record 136 of 137
==535548==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==535548==    by 0x5EADBDC: ???
==535548==    by 0x5C3EC7E: ???
==535548==    by 0x5C3EDA4: ???
==535548==    by 0x5C5DE49: ???
==535548==    by 0x5C5DF46: ???
==535548==    by 0x5CB6368: ???
==535548==    by 0x4862467: ???
==535548==    by 0x4971660: sqlite3VdbeExec (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==535548==    by 0x4979660: sqlite3_step (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==535548==    by 0x1292D6: ??? (in /usr/bin/sqlite3)
==535548==    by 0x12AB95: ??? (in /usr/bin/sqlite3)
==535548== 
==535548== LEAK SUMMARY:
==535548==    definitely lost: 5,040 bytes in 7 blocks
==535548==    indirectly lost: 82,716 bytes in 38 blocks
==535548==      possibly lost: 0 bytes in 0 blocks
==535548==    still reachable: 314,112 bytes in 481 blocks
==535548==         suppressed: 0 bytes in 0 blocks
==535548== Reachable blocks (those to which a pointer was found) are not shown.
==535548== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==535548== 
==535548== For lists of detected and suppressed errors, rerun with: -s
==535548== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0

[gwenn]

Sorry but I probably won't have time to investigate.

[LaurentDhont]

So sad, would you like a reimbursement to fix the issue?

[gsker]

@LaurentDhont
Perhaps you could test to see if you get the same issue with the upstream ICU extension?

• https://sqlite.org/src/dir/ext/icu

I'm not sure how to test it, but it's in the debian repo as
libsqlite3-ext-icu.

Also you could try this set of extensions instead:

• All the missing SQLite functions
• https://github.com/nalgeon/sqlean

At least it appears to be maintained.

I loaded the binary release and ran this:

sqlite> .load '/tmp/sqlean/regexp'    
sqlite> select regexp_substr('the year is的 °21', '[a-z 的]+');    
the year is的

Just some ideas. I would not be able to help with finding the leak.

(I emailed this comment too, but it didn't show up so it might end up duplicated)

[gwenn]

main.c.txt

cc -Wall -g -o test main.c `pkg-config --libs sqlite3` `pkg-config --libs icu-i18n`
valgrind --tool=memcheck ./test

==8712== Memcheck, a memory error detector
==8712== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==8712== Using Valgrind-3.24.0 and LibVEX; rerun with -h for copyright info
==8712== Command: ./test
==8712==
Lasdfasdfasdfent
Sergiu
==8712==
==8712== HEAP SUMMARY:
==8712==     in use at exit: 89,100 bytes in 46 blocks
==8712==   total heap usage: 416 allocs, 370 frees, 653,148 bytes allocated
==8712==
==8712== LEAK SUMMARY:
==8712==    definitely lost: 0 bytes in 0 blocks
==8712==    indirectly lost: 0 bytes in 0 blocks
==8712==      possibly lost: 0 bytes in 0 blocks
==8712==    still reachable: 89,100 bytes in 46 blocks
==8712==         suppressed: 0 bytes in 0 blocks
==8712== Rerun with --leak-check=full to see details of leaked memory
==8712==
==8712== For lists of detected and suppressed errors, rerun with: -s
==8712== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)