Include filter

[jshank]

I'm working on moving from Duplicati to autorestic and stuck on one item, include filters. I have a backup set that only gets the docker-compose.yaml files in my docker-configs path (roughly 70 files but grows monthly). It appears that restic supports this using --files-from to point to a file that has filepath.Glob patterns.

For Duplicati, I use:

• Source data
  • /path/to/docker-configs
• Filters
  • Include expression: */docker-compose.*

Inside /path/to/docker-configs is the a directory based on the name of the container (plex for example), the docker-compose.yaml file and then any supporting subdirectories configs, keys`, etc. I'd like to avoid going crazy with excludes for all of the possible supporting directories if I can.

[cupcakearmy]

Correct me if i did not understand correctly.

You can pass any flag to restic using the options object.

location:
  foo:
    options:
      backup
        files-from:
          - "*/docker-compose.*"

You can set them also globally
Read more in the docs

[jshank]

Unfortunately files-from reads from a file. You can't pass a pattern directly but putting the pattern into a file identified by files-from does work.

Backend: server                            
> Executing: /usr/local/bin/restic backup --files-from */docker-compose.* --exclude .fuse_hidden* --tag ar:location:docker_configs /mnt/media/docker-configs/       open */docker-compose.*: no such file or directory

[cupcakearmy]

Ah ok I See the Problem now.
That's a tricky one. Tbh I can't think of a clean solution rn. Do you have suggestions?

[cupcakearmy]

I guess one option would be to maintain a list of restic flags that are file arguments and evaluate the path with the same shell autorestic was called by before executing

[jshank]

Getting closer but still having trouble because the from: path is all inclusive by default. I'm getting all of the docker-compose.yaml and docker-compose.yml files that I'm interested in and not getting the deeper folder structure but still getting the files in /mnt/media/docker-configs/path/ that are not docker-compose files.

config.yml

locations:
  docker_configs:
    from:
      - /mnt/media/docker-configs/
    to:
      - server
    options:
      backup:
        files-from:
          - /mnt/media/scripts/restic/docker-configs-include
        exclude:
          - /mnt/media/docker-configs/*/*/*

/mnt/media/scripts/restic/docker-configs-include

/mnt/media/docker-configs/*/docker-compose.yml
/mnt/media/docker-configs/*/docker-compose.yaml

What we get

restic ls -r rest:http://10.1.10.15:8000/shanknas a555b0e9
enter password for repository: 
repository a516a803 opened successfully, password is correct
snapshot a555b0e9 of [<snip> /mnt/media/docker-configs/bwalink/docker-compose.yaml <snip> /mnt/media/docker-configs] filtered by [] at 2021-12-13 17:19:15.301005216 -0800 PST):
/mnt
/mnt/media
/mnt/media/docker-configs
<snip>
/mnt/media/docker-configs/bwalink
/mnt/media/docker-configs/bwalink/.git
/mnt/media/docker-configs/bwalink/.github
/mnt/media/docker-configs/bwalink/.gitignore
/mnt/media/docker-configs/bwalink/Dockerfile
/mnt/media/docker-configs/bwalink/Gemfile
/mnt/media/docker-configs/bwalink/README.md
/mnt/media/docker-configs/bwalink/docker-compose.yaml <-- The only file I want
/mnt/media/docker-configs/bwalink/docker-entrypoint.sh

[jshank]

4-year later, I finally figured this out using a combination of the prevalidate hooks to use find to generate a file including exactly what I wanted, then files-from-verbatim to utilize that file. The trick was getting rid of the required from as it was causing the re-inclusion of the files I didn't want. Feeding it /dev/null did the trick.

  docker_configs:
    from:
    - /dev/null
    to:
    - localbackup
    hooks:
      dir: ""
      prevalidate:
      - find /mnt/media/docker-configs -maxdepth 2 \( -name docker-compose.yaml -o -name "docker-compose.yml" \) 2>/dev/null > /mnt/media/scripts/restic/docker-configs-include
    cron: 2 2 * * *
    options:
      backup:
        exclude:
        - /dev/null
        files-from-verbatim:
        - /mnt/media/scripts/restic/docker-configs-include