rework commands, add logging, add list feature, reimplement some rollback logic with clearing rollbacked states

Author: freenoth

README.md

@@ -3,18 +3,18 @@ This package allow to easy manage apps migrations based on GIT repository (it us
 So you can return to any previous state that is saved in DB by one command.
 
 ## Version
-Current version is `0.2.1`
+Current version is `0.3.0`
 
 It works with:
-- django == 1.11
+- django >= 1.11.3
 - GitPython >= 2.1.8
 - PostgreSQL
 
 ## Installing
 
 First you need to install package with pip:
 ```bash
-pip install git+https://github.com/freenoth/django-rollback.git@0.2.1
+pip install git+https://github.com/freenoth/django-rollback.git@0.3.0
 ```
 
 Then install it to your `INSTALLED_APPS`
@@ -30,48 +30,76 @@ You are also should run `./manage.py migrate` before using additional management
 ## Using
 There are two commands to manage migrations state.
 
-For both commands you can pass the `PATH` argument to specify path to git repository directory (local). Default path is current dir : `'.'`.
+Both commands have common arguments:
+```bash
+  -p PATH, --path PATH  Git repository path.
+  -l LOGGER, --logger LOGGER
+                        Logger name for logging.
+  --log-level LOG_LEVEL
+                        Log level for logging. INFO, DEBUG, etc.
+
+```
+`PATH` argument used to specify path to git repository directory (local). Default path is current dir : `'.'`.
+
+`LOGGER` and `LOG_LEVEL` arguments can be used to setup internal logging. For example, you can use one of django_logging loggers (to push it to slack, write console, file, etc.). There is no default value, so by default additional logging disabled.
 
 ### Saving current state
 ```bash
 ./manage.py save_migrations_state
 ```
 Help message below:
 ```bash
-usage: manage.py save_migrations_state [-p PATH]
+usage: manage.py save_migrations_state [-p PATH] [-l LOGGER]
+                                       [--log-level LOG_LEVEL]
 
-Save migrations state for current commit.
+Save migrations state for current commit. It also check if commit already
+exists and print warning if it`s not the latest state that may be a symptom of
+inconsistent state for migrations.
 
 optional arguments:
   -p PATH, --path PATH  Git repository path.
+  -l LOGGER, --logger LOGGER
+                        Logger name for logging.
+  --log-level LOG_LEVEL
+                        Log level for logging. INFO, DEBUG, etc.
+
 ```
-This command used to save apps migrations state of current commit to DB (it create new or update existing state).
+This command used to save apps migrations state of current commit to DB. It try to create new state, but if already exists it checks is this state the latest.
+If state for current commit is not the latest - it may be a symptom of problems and rollback from current commit will not work for it.
 
 Successful output example below:
 ```bash
 Data = [(4, 'admin', '0002_logentry_remove_auto_add'), (12, 'auth', '0008_alter_user_username_max_length'), (5, 'contenttypes', '0002_remove_content_type_name'), (16, 'django_auto_rollback', '0001_initial'), (17, 'django_rollback', '0001_initial'), (14, 'sessions', '0001_initial')].
 Successfully created for commit "84e47461a95fa325d9e933bbe8cca8c52bbea203".
 ```
 
-### Return to previous state
+### Return to previous state (rollback)
 ```bash
 ./manage.py rollback_migrations
 ```
 Help message below:
 ```bash
-usage: manage.py rollback_migrations [-t TAG] [-c COMMIT] [-p PATH] [--fake]
+usage: manage.py rollback_migrations [-p PATH] [-l LOGGER] [--log-level LOG_LEVEL]
+                                     [--list] [-t TAG] [-c COMMIT] [--fake]
 
 Rollback migrations state of all django apps to chosen tag or commit if
-previously saved.
+previously saved. Also you may not specify commit or tag to rollback, so the
+previous tag will be used. Also it can run in fake mode, only to print
+generated commands for rollback. You also can view current DB state for all
+saved states using list argument.
 
 optional arguments:
+  -p PATH, --path PATH  Git repository path.
+  -l LOGGER, --logger LOGGER
+                        Logger name for logging.
+  --log-level LOG_LEVEL
+                        Log level for logging. INFO, DEBUG, etc.
+  --list                Show the sorted list of all stored states.
   -t TAG, --tag TAG     Git tag to which to rollback migrations.
   -c COMMIT, --commit COMMIT
                         Git commit hash to which to rollback migrations.
-  -p PATH, --path PATH  Git repository path.
   --fake                It allow to only print info about processed actions
                         without execution (no changes for DB).
-
 ```
 
 You can use git commit hash (hex) directly. And you don`t need to specify full commit hash, you just can use first letters.
@@ -87,17 +115,6 @@ Or you can use git tag (it will be translated to related commit).
 ./manage.py rollback_migrations --tag v.0.0.2
 ```
 
-Either tag or commit argument is required:
-```bash
-./manage.py rollback_migrations
-CommandError: Tag or commit should be described by -t or -c arguments.
-```
-And it can`t be used together:
-```bash
-./manage.py rollback_migrations -c 0e02e74  -t v.0.0.1
-CommandError: Tag and commit arguments should not be described together.
-```
-
 Successful output example below:
 ```bash
 ./manage.py rollback_migrations -c c257a23
@@ -112,8 +129,11 @@ Rollback successfully finished.
 
 As you can see above, apps can be rollbacked to `zero` state too, if in previous state this app not used.
 
+After successful rollback, selected state will be selected as current, so all older states will be deleted.
+That`s way current state all the time should be the latest in DB and correspond to current service state.
+
 ### Successful rollback conditions
 So rollback will be successfully finished if two conditions are satisfied:
 - state for current commit was saved (if not - use `./manage.py save_migrations_state` command)
-- state for specified commit or commit which relates to specified tag was saved in the past
- 
+- state for specified commit, commit which relates to specified tag was saved in the past
+- if commit not specified, just the previous state should exists 

django_rollback/consts.py

@@ -0,0 +1,3 @@
+DEFAULT_REPO_PATH = '.'
+COMMIT_MAX_LENGTH = 40
+MIGRATE_COMMAND = 'migrate'

django_rollback/management/base.py

@@ -0,0 +1,155 @@
+import json
+import logging
+from collections import namedtuple
+
+import git
+from django.core.management import call_command
+from django.core.management.base import BaseCommand, CommandError
+from django.db import connection
+
+from django_rollback.consts import DEFAULT_REPO_PATH, COMMIT_MAX_LENGTH, MIGRATE_COMMAND
+from django_rollback.models import AppsState
+from django_rollback.sql import MIGRATIONS_STATE_SQL
+
+MigrationRecord = namedtuple('MigrationRecord', ['id', 'app', 'name'])
+
+
+class BaseRollbackCommand(BaseCommand):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._logger = None
+
+    def add_arguments(self, parser):
+        parser.add_argument('-p', '--path', type=str, default=DEFAULT_REPO_PATH, help='Git repository path.')
+        parser.add_argument('-l', '--logger', type=str, help='Logger name for logging.')
+        parser.add_argument('--log-level', type=str, help='Log level for logging. INFO, DEBUG, etc.')
+
+    def configure_logger(self, options):
+        if options['logger']:
+            logger = logging.getLogger(options['logger'])
+
+            log_level = logging.getLevelName(options['log_level'])
+            if isinstance(log_level, int):
+                logger.setLevel(log_level)
+
+            self._logger = logger
+
+    def handle(self, *args, **options):
+        raise NotImplemented
+
+    def get_current_commit(self, path):
+        try:
+            repo = git.Repo(path)
+            return repo.head.commit.hexsha
+        except ValueError as err:
+            message = f'An error occurred while working with git repo!'
+            self.stdout.write(self.style.ERROR(message))
+            if self._logger:
+                self._logger.error(message + f'\n{__name__}')
+            raise CommandError(err)
+
+    @staticmethod
+    def get_last_apps_state():
+        return AppsState.objects.all().order_by('timestamp').last()
+
+    @staticmethod
+    def get_current_migrations_state():
+        """
+        return a data in format:
+        [(<id> : int, <app> : str, <name> : str), ...]
+        """
+        with connection.cursor() as cursor:
+            cursor.execute(MIGRATIONS_STATE_SQL)
+            return cursor.fetchall()
+
+    def get_apps_state_by_commit(self, commit):
+        queryset = AppsState.objects.filter(commit__istartswith=commit)
+
+        count = queryset.count()
+
+        if count == 0:
+            message = f'Can not find stored data of migrations state for commit `{commit}`.'
+            if self._logger:
+                self._logger.warning(message)
+            raise CommandError(message)
+
+        if count > 1:
+            is_short_commit = len(commit) < COMMIT_MAX_LENGTH
+            message = (f'Found more than 1 ({count}) records for selected commit {commit}.'
+                       f'{" Please clarify commit hash for more identity." if is_short_commit else ""}')
+            if self._logger:
+                self._logger.warning(message)
+            raise CommandError(message)
+
+        return queryset.first()
+
+    def get_migrations_data_by_commit(self, commit):
+        apps_state = self.get_apps_state_by_commit(commit)
+        return json.loads(apps_state.migrations)
+
+    def get_migrations_diff(self, current, other):
+        """
+        current and other has type list of tuples in format:
+        [(<id> : int, <app> : str, <name> : str), ...]
+        it gets from get_migrations_data_from_commit()
+
+        :return list that indicates what migrations should be executed
+        migration_id is useful to detect migration order (from higher to lower)
+        [
+            namedtuple('MigrationRecord', ['id', 'app', 'name']),
+            ...
+        ]
+        """
+        result = []
+
+        current = [MigrationRecord(*rec) for rec in current]
+        other = [MigrationRecord(*rec) for rec in other]
+        other_apps = {migration.app for migration in other}
+
+        # find what is changed in current relative to other
+        diff = set(current) - set(other)
+        for migration in diff:
+            is_new_app = migration.app not in other_apps
+            result.append(MigrationRecord(
+                migration.id,
+                migration.app,
+                'zero' if is_new_app else list(filter(lambda x: x.app == migration.app, other))[0].name,
+            ))
+
+        if self._logger:
+            self._logger.info(f'Migrations diff:\n{result}')
+
+        return result
+
+    def run_rollback(self, migrations_diff_records, fake=False):
+        """
+        migrations_diff_records: List[MigrationRecord], result of get_migrations_diff()
+        sort all migrations by migration.id order from higher to lower and execute migrate command for them
+        """
+
+        if not migrations_diff_records:
+            message = 'There is no migrations to rollback.'
+            self.stdout.write(f'>>> {message}')
+            if self._logger:
+                self._logger.info(message)
+            return
+
+        for migration in sorted(migrations_diff_records, key=lambda r: int(r.id), reverse=True):
+            execute_args = (MIGRATE_COMMAND, migration.app, migration.name)
+            message = f'Executing command: {" ".join(execute_args)}'
+            self.stdout.write(f'>>> {message}')
+            if self._logger:
+                self._logger.info(message)
+
+            if not fake:
+                call_command(*execute_args)
+
+    def make_the_last_state_for_commit(self, commit):
+        apps_state = self.get_apps_state_by_commit(commit)
+        AppsState.objects.filter(id_gt=apps_state.id).delete()
+
+        message = f'state for commit "{commit}" now is the last state in DB'
+        self.stdout.write(f'>>> {message}')
+        if self._logger:
+            self._logger.info(message)