MDEV-39092 Copy Aria data and logs as part of backup by mariadb-andrzejjarzabek · Pull Request #4971 · MariaDB/server

mariadb-andrzejjarzabek · 2026-04-22T08:28:29Z

This is an initial simple implementation which copies all the Aria files in the "end" phase of the backup. Nothing protects the copy from concurrent DDL or DML. Copying only works on MacOS (intended for refactoring to use common file copy method across engines and SQL layer).

CLAassistant · 2026-04-22T08:28:38Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

dr-m · 2026-04-28T06:13:16Z

+    int perform_backup() noexcept
+    {
+      if (scan_datadir())
+        return 1;
+      if (copy_databases())
+        return 1;
+      if (copy_control_file())
+        return 1;
+      if (copy_logs())
+        return 1;
+      return 0;
+    }


This is fundamentally incompatible with the handlerton::backup_step API. Even if you are for now copying everything in handlerton::backup_end, the internal API should be kept as compatible with handlerton::backup_step as possible: copying one file at a time. At least the copy_databases() step must be refactored so that the higher-level API is invoking something that copies one file at a time. Possibly, the copying of log files should be interleaved with that, like innodb_backup_step is doing.

This is a stop-gap to get something working. Eventual final implementation will require re-working the hadlerton API and Sql_cmd_backup to take into account that different subsets of files may be copied under diffefent levels of metadata lock, and also "start" and potentially even "end" may need to be split into different phases with different levels of metadata lock. Given that the stage API is written that way to support multi-threaded copy, which isn't implemented at this time, I propose to merge the change as proposed (I will fix some of the other problems you mentioned) and iteratively improve from there.

This introduces a basic driver Sql_cmd_backup, storage engine interfaces, and basic copying of InnoDB data files. On Windows, we pass a target directory name; elsewhere, we pass a target directory handle. fil_space_t::write_or_backup: Keep track of in-flight page writes and pending backup operation. We must not allow them concurrently, because that could lead into torn pages in the backup. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). TRX_STATE_BACKUP: A special InnoDB transaction state indicating association with BACKUP SERVER, which allows us to pass some context in trx_t from innodb_backup_end() to innodb_backup_finalize(). log_t::backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread(). FIXME: MoveFileEx() after CreateHardLink() fails on Windows TODO: Implement open(O_DIRECTORY) and openat(2) compatible API on Windows, to have a uniform interface (passing a target directory handle, not name). TODO: Implement finer-grained locking around copying page ranges. TODO: Implement other storage engine interfaces. TODO: Implement the necessary locking around backup_end. TODO: Fix the space.get_create_lsn() < checkpoint logic.

backup_target: A structured data type to represent a directory or a stream. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. copy_file(): A file copying service for POSIX systems. On Windows, we will use CopyFileEx().

Fix Windows and macOS

dr-m · 2026-05-21T06:20:11Z

+int copy_entire_file(int src, int dst)
+{
+  return fcopyfile(src, dst, nullptr, COPYFILE_ALL | COPYFILE_CLONE);
+}
+
+extern "C" int copy_file(int src, int dst, off_t)
+{
+  return fcopyfile(src, dst, nullptr, COPYFILE_ALL | COPYFILE_CLONE);
+}


This is adding unnecessary object code duplication. Macros would allow zero-overhead abstraction:

diff --git a/sql/sql_backup_interface.h b/sql/sql_backup_interface.h index d781acff3d9..b7d04705985 100644 --- a/sql/sql_backup_interface.h +++ b/sql/sql_backup_interface.h @@ -21,6 +21,7 @@ # include <copyfile.h> # define copy_file(src, dst, off) \ fcopyfile(src, dst, nullptr, COPYFILE_ALL | COPYFILE_CLONE) +# define copy_entire_file(src, dst) copy_file(src, dst,) #else # ifdef __cplusplus extern "C" @@ -32,4 +33,15 @@ extern "C" @return error code (negative) @retval 0 on success */ int copy_file(int src, int dst, off_t size); + +# ifdef __cplusplus +extern "C" +# endif +/** Copy an entire file. +@param src source file descriptor +@param dst target to append src to +@param size amount of data to be copied +@return error code (negative) +@retval 0 on success */ +int copy_entire_file(int src, int dst); #endif

Outside Windows and Apple, it does make sense to introduce a non-inline function for copy_entire_file() even though it rather small (48 bytes for AMD64). I will include this in the next update of #4817.

Would defining it as an inline function not be better then? Macros have the problem of replacing tokens lexically, so if there's any identifier in code called copy_file and this header is included, it will get messed up. The fact that it would only happen on a Mac makes this problem even worse.

We have a mandatory macOS builder https://buildbot.mariadb.org/#/builders/708 that must pass in order for anything to be merged to a main branch. Hence, compilation failures should only be possible in development branches.

The benefit of defining a macro copy_file() is that the third argument will be guaranteed to be omitted from the generated code. If we had an inline function that ignores the third parameter, some unnecessary code could be emitted related to the evaluation of that parameter.

This is an initial simple implementation which copies all the Aria files in the "end" phase of the backup. Nothing protects the copy from concurrent DDL or DML. Copying only works on MacOS (intended for refactoring to use common file copy method across engines and SQL layer).

Copy non-Aria-specific files *.frm and db.opt as part of Aria backup.

Add MTR test. Copy MyISAM files in Aria plugin so that the MTR works. Perform the end step under maximum level of backup MDL to safely copy non-transactional Aria and MyISAM files.

…kup. (blocking itself not changed yet)

mariadb-andrzejjarzabek requested a review from dr-m April 22, 2026 08:28

gkodinov added the MariaDB Corporation label Apr 23, 2026

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 6 times, most recently from 4ef94be to c143ff2 Compare April 25, 2026 19:02

dr-m reviewed Apr 28, 2026

View reviewed changes

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from efbc62b to c77278b Compare May 4, 2026 14:00

dr-m force-pushed the MDEV-14992 branch from bcbda03 to e0d850e Compare May 5, 2026 12:06

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from e89625e to 06fd556 Compare May 17, 2026 19:19

dr-m force-pushed the MDEV-14992 branch from e0d850e to 0c52540 Compare May 18, 2026 09:40

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from 4d6a19c to d86a6a1 Compare May 20, 2026 05:50

dr-m force-pushed the MDEV-14992 branch from bdec600 to 45e7902 Compare May 20, 2026 06:52

fixup! 45e7902

a556b35

Fix Windows and macOS

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from d86a6a1 to c6f5119 Compare May 20, 2026 08:20

dr-m added 3 commits May 20, 2026 11:32

fixup! a556b35

3ddf047

fixup! 45e7902

f9f85e8

Relax the locking for CreateHardLink()

2256a09

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from c6f5119 to 9ebb23e Compare May 20, 2026 20:24

dr-m reviewed May 21, 2026

View reviewed changes

dr-m added 3 commits May 21, 2026 09:21

Copy enough of the last archived log file

513a54b

Introduce copy_entire_file()

429a295

Rename the local variable backup to innodb_backup to ease debugging

f3d54e6

mariadb-andrzejjarzabek added 4 commits May 21, 2026 10:42

Enable backup for non-Apple systems.

7cf653a

Copy non-Aria-specific files *.frm and db.opt as part of Aria backup.

MDEV-39092 BACKUP SERVER of ENGINE=Aria to the local file system

52df6e2

Add MTR test. Copy MyISAM files in Aria plugin so that the MTR works. Perform the end step under maximum level of backup MDL to safely copy non-transactional Aria and MyISAM files.

Flush Aria WAL log before copying the log file.

6a8e0c2

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from 9ebb23e to 6a8e0c2 Compare May 21, 2026 08:52

Incorporate changes from code review.

04e3bc2

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from 8565956 to 04e3bc2 Compare May 21, 2026 10:53

mariadb-andrzejjarzabek added 2 commits May 27, 2026 07:39

Remove InnoDB table from test.

a235a54

Change BACKUP SERVER plugin API to allow reduced blocking in Aria bac…

185b3b1

…kup. (blocking itself not changed yet)

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from a786a76 to 14ca552 Compare May 29, 2026 09:33

Copy "safe" Aria files under DDL lock only to allow concurrent DML.

c9429eb

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from 14ca552 to c9429eb Compare May 29, 2026 09:47

dr-m force-pushed the MDEV-14992 branch from d595ce0 to c0e48fc Compare May 29, 2026 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MDEV-39092 Copy Aria data and logs as part of backup#4971

MDEV-39092 Copy Aria data and logs as part of backup#4971
mariadb-andrzejjarzabek wants to merge 17 commits into
MariaDB:MDEV-14992from
mariadb-andrzejjarzabek:MDEV-39092

mariadb-andrzejjarzabek commented Apr 22, 2026

Uh oh!

CLAassistant commented Apr 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dr-m Apr 28, 2026

Uh oh!

mariadb-andrzejjarzabek Apr 28, 2026

Uh oh!

dr-m May 21, 2026

Uh oh!

mariadb-andrzejjarzabek May 21, 2026

Uh oh!

dr-m May 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mariadb-andrzejjarzabek commented Apr 22, 2026

Uh oh!

CLAassistant commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dr-m Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

mariadb-andrzejjarzabek Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

dr-m May 21, 2026

Choose a reason for hiding this comment

Uh oh!

mariadb-andrzejjarzabek May 21, 2026

Choose a reason for hiding this comment

Uh oh!

dr-m May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Apr 22, 2026 •

edited

Loading