Migrations should happen in a transaction

[rjmunro]

This is intentionally a reopening of sequelize/sequelize#1882 which was closed automatically by a bot. I think it's a fundamental feature for a migration engine.

What you are doing?

There was a problem applying a migration, due to a issue in the code of the migrations.

What do you expect to happen?

I expect the whole migration to be rolled back.

Nearly all database engines support DDL in transactions. It should be the default way updates are done, unless the user chooses not to.

Even if the database does not fully support transactions for DDL (MySQL) sometimes migrations can make updates to data e.g. adding records to lookup tables, and it should be the default to roll these back.

What is actually happening?

The database ends up in an inconsistent state, with half the changes of the migration applied, but not the other half. I can't re-run the migration, I can't easily go back. I have to manually finish the migration and then manually update the meta tables.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is still an issue, just leave a comment 🙂

[rjmunro]

@Stale This is one of the top 5% of requested features by upvotes - I don't think it should be closed.

[iamjochem]

@rjmunro - I happened upon your issue whilst scanning for transaction-related issues (anything that might explain why I'm seeing sporadic transaction-related failure under load ... in my case it seems as if a given transaction option is being ignored) ...

Regarding MySQL, it does not support transactions for DDL queries at all (maybe the latest version does, or some flavour like MariaDB, but not the version I'm using at). This means automatic transaction support in the sequelize migration tool will not help you (in the case of MySQL).

Having run into pretty much the same problem as you I can up with a couple of pragmatic rules regarding development of migration files:

1. only 1 query (1 up & 1 down) per migration file
2. only use raw queries. (less abstraction, clearer intent)

you end up with potentially a lot of migration files but it's also bullet-proof in terms of rollback.

you could argue that you need to have multiple queries per migration file in order to keep deployments in sync with migrations (i.e. that every release has just one migration). I believe that that is almost impossible to police when there are multiple features being developed by multiple developers, especially when you throw CI & automation in the mix, not to mention that it is likely that most deployments do not have a corresponding migration.

Actually from a rollback POV the 'sequelize-meta' table should be storing batch-numbers along side the migrations that represent which migrations were run for any given migrate up ... additionally I think that it would be smart to store the source-code of the migrations files a. so that there is a record of what was run bundled in the actual DB instance and b. so that it is [technically] possible to perform rollbacks even when the latest migrations are not available (e.g. because you have checked-out/deployed some previous version of your codebase).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is still an issue, just leave a comment 🙂

[rjmunro]

@iamjochem This is good advice for people stuck on MySQL or it's forks. For users of virtually any other databases, transactions are supported for DDL statements, and we should use them.

Transactions even reduce downtime because while all the DDL is going on in the transaction, the old data can still be read and updated.

Splitting migrations into one query each doesn't help you if you make a mistake and the query fails. A transaction will just put the database back to exactly how it was before you started.

[iamjochem]

@rjmunro - forcing migrations to contain only one query mimics transactions ... because the migration becomes all-or-nothing, failure results in a determinate state for the DB and the migrations (if a migration with multiple queries, that does not use a transaction, fails then you are stuck having to manually fix the DB because you can no longer perform an up or a down migration) ... with a failing single-query migration the DB will not be mutated so you have the ability to fix the query and try the up again. obviously this is only pertinant to MySQL & it's derivatives, any backend that supports transactions for DDL mutations should have them forcibly employed by the migration code on a per-migration basis.

[jacqt]

From a cursory look at the sequelize codebase and some following experiments, the following works for in Postgres.

exports.up = function (queryInterface) {
  return queryInterface.sequelize.transaction(function (t) { return Promise.all([
    queryInterface.createTable('audit_logs', {
      ...
    }, {transaction: t})

    queryInterface.createTable('audit_logs2', {
      ...
    }, {transaction: t})
  ]); });
};

I haven't taken a complete look at all the queryInterface methods, but most seem to be passing the options arguments straight to the dialect specific query method (again for Postgres only), so it seems like this would work.

Can any of the maintainers (@sushantdhiman ?) confirm whether this is behavior that we can rely on to remain consistent (and potentially documented)?

[sushantdhiman]

Any thing that touches sequelize.query which includes all query interface api and model public api can pass options.transaction, it will include that query to given transaction, I think its a well known feature, may be docs need clarification.

But its not a correct solution as operation on sequelize_meta wont be covered with this transaction

[jacqt]

Thank you for the quick response and for confirming! Makes sense how it is not a complete solution. I took a deeper look at how it might be possible to share this transaction with the migrator, and I think I have an inkling why this issue is still open :)

Off the top of my head, maybe one way to support this is to modify the cli here (https://github.com/sequelize/cli/blob/master/src/core/migrator.js#L49) to add a 3rd parameter with an unmanaged transaction that the (un)logMigration (https://github.com/sequelize/umzug/blob/master/src/storages/SequelizeStorage.js#L100) would use (and commit if the up method succeeds or rollback if the up method fails). Then it is up to to the developer to use the transaction argument as he wishes.

E.g.

exports.up = function (queryInterface, Sequelize, transaction) {
  return transaction.then((t) { return Promise.all([
    queryInterface.createTable('audit_logs', {
      ...
    }, {transaction: t})

    queryInterface.createTable('audit_logs2', {
      ...
    }, {transaction: t})
  ]); });
};

Some care will need to be done to make sure the transaction is eventually closed or rolled back.

If you think this is worth exploring I'd be happy to tinker with it if I have time or to contribute a bounty for this.

Lastly - I wanted to thank you for maintaining Sequelize! At minimum I would be happy to help write docs to clarify the behavior RE: the options parameter you mentioned in your comment.