Skip to content

Commit debada6

Browse files
someone1someone1
committedJul 14, 2017
Initial Commit
·
v0.4-rc1v0.1-alpha
1 parent 59fc793 commit debada6

30 files changed

+3792
-0
lines changed
 

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
*.dll
44
*.so
55
*.dylib
6+
zfsbackup*
67

78
# Test binary, build with `go test -c`
89
*.test

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
The MIT License (MIT)
2+
3+
Copyright © 2016 Prateek Malhotra (someone1@gmail.com)
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in
13+
all copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21+
THE SOFTWARE.

‎Makefile

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
TARGETS="freebsd/amd64 linux/amd64"
2+
COMMIT_HASH=`git rev-parse --short HEAD 2>/dev/null`
3+
4+
check: get fmt vet lint test test-race
5+
6+
fmt:
7+
@for d in $(DIRS) ; do \
8+
if [ "`gofmt -l $$d/*.go | tee /dev/stderr`" ]; then \
9+
echo "^ improperly formatted go files" && echo && exit 1; \
10+
fi \
11+
done
12+
13+
lint:
14+
@if [ "`gometalinter --config=linter.json ./... | tee /dev/stderr`" ]; then \
15+
echo "^ gometalinter errors!" && echo && exit 1; \
16+
fi
17+
18+
get:
19+
go get -v -t ./...
20+
21+
test:
22+
go test ./...
23+
24+
test-race:
25+
go test -race ./...
26+
27+
vet:
28+
@if [ "`go vet ./... | tee /dev/stderr`" ]; then \
29+
echo "^ go vet errors!" && echo && exit 1; \
30+
fi
31+
32+
build:
33+
go get github.com/mitchellh/gox
34+
${GOPATH}bin/gox -ldflags="-w" -osarch=${TARGETS}
35+
36+
build-dev:
37+
go get github.com/mitchellh/gox
38+
${GOPATH}bin/gox -osarch=${TARGETS} -output="{{.Dir}}_{{.OS}}_{{.Arch}}-${COMMIT_HASH}"

‎README.md

+109
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# ZFSBackup
2+
3+
DISCLAIMER: This is a work in progress - USE AT YOUR OWN RISK!
4+
5+
## Overview:
6+
7+
This backup software was designed for the secure, long-term storage of ZFS snapshots on remote storage. Backup jobs are resilient to network failures and can be stopped/resumed. It works by splitting the ZFS send stream (the format for which is committed and can be received on future versions of ZFS as per the [man page](https://www.freebsd.org/cgi/man.cgi?zfs(8))) into chunks and then optionally compresses, encrypts, and signs each chunk before uploading it to your remote storage location(s) of choice. Backup chunks are validated using SHA256 and CRC32C checksums (along with the many integrity checks builtin to compression algorithms, SSL/TLS transportation protocols, and the ZFS stream format itself). The software is completely self-contained and has no external dependencies.
8+
9+
This project was inspired by the [duplicity project](http://duplicity.nongnu.org/).
10+
11+
### Highlights:
12+
* Written in Go
13+
* Backup jobs are resumeable and resilient to network failures
14+
* Backup files can be compressed and optionally encrypyted and/or signed.
15+
* Concurrent by design, enable multiple cores for parallel processing
16+
* Configurable Operation - Limit bandwidth usage, space usage, CPU usage, etc.
17+
* Backup to multiple destinations at once, just comma separate destination URIs
18+
* Uses familiar ZFS send/receive options
19+
20+
### Supported Backends:
21+
* Google Cloud Storage (gs://)
22+
- Auth details: https://developers.google.com/identity/protocols/application-default-credentials
23+
* Amazon AWS S3 (s3://) (Glacier supported indirectly via lifecycle rules)
24+
- Auth details: https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
25+
26+
27+
### Compression:
28+
The compression algorithm builtin to the software is a parallel gzip ([pgzip](https://github.com/klauspost/pgzip)) compressor. There is support for 3rd party compressors so long as the binary is available on the host system and is compatible with the standard gzip binary command line options (e.g. xz, bzip2, lzma, etc.)
29+
30+
### Encryption/Signing:
31+
The PGP algorithm is used for encryption/signing. The cipher used is AES-256.
32+
33+
## Usage
34+
35+
Full backup example:
36+
37+
$ ./zfsbackup send --encryptTo user@domain.com --signFrom user@domain.com --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc Tank/Dataset@snapshot-20170101 gs://backup-bucket-target
38+
39+
Incremental backup example:
40+
41+
$ ./zfsbackup send --encryptTo user@domain.com --signFrom user@domain.com --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc -i Tank/Dataset@snapshot-20170101 Tank/Dataset@snapshot-20170201 gs://backup-bucket-target,s3://another-backup-target
42+
43+
44+
Full restore example:
45+
46+
$ ./zfsbackup receive --encryptTo user@domain.com --signFrom user@domain.com --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc -d Tank/Dataset@snapshot-20170201 gs://backup-bucket-target Tank
47+
48+
Incremental restore example:
49+
50+
$ ./zfsbackup receive --encryptTo user@domain.com --signFrom user@domain.com --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc -d -F -i Tank/Dataset@snapshot-20170101 Tank/Dataset@snapshot-20170201 gs://backup-bucket-target Tank
51+
52+
Notes:
53+
* Create keyring files: https://keybase.io/crypto
54+
* PGP Passphrase will be prompted during execution if it is not found in the PGP_PASSPHRASE environmental variable.
55+
* `--maxFileBuffer=0` will disable parallel processing, chunking, multiple destinations, and upload hash verification but will use virtually no disk space.
56+
* For S3: Specify Standard/Bulk/Expedited in the AWS_S3_GLACIER_RESTORE_TIER environmental variable to change Glacier restore option (default: Bulk)
57+
58+
Example output for top level command:
59+
60+
```shell
61+
$ ./zfsbackup
62+
zfsbackup is a tool used to do off-site backups of ZFS volumes.
63+
It leverages the built-in snapshot capabilities of ZFS in order to export ZFS
64+
volumes for long-term storage.
65+
66+
zfsbackup uses the "zfs send" command to export, and optionally compress, sign,
67+
encrypt, and split the send stream to files that are then transferred to a
68+
destination of your choosing.
69+
70+
Usage:
71+
zfsbackup [command]
72+
73+
Available Commands:
74+
clean Clean will delete any objects in the destination that are not found in the manifest files found in the destination.
75+
help Help about any command
76+
list List all backup sets found at the provided destination.
77+
receive receive will restore a snapshot of a ZFS volume similar to how the "zfs recv" command works.
78+
send send will backup of a ZFS volume similar to how the "zfs send" command works.
79+
verify Verify will ensure that the backupset for the given snapshot exists in the destination
80+
version Print the version of zfsbackup in use and relevant compile information
81+
82+
Flags:
83+
--compressor string specify to use the internal (parallel) gzip implementation or an external binary (e.g. gzip, bzip2, pigz, lzma, xz, etc. Syntax must be similiar to the gzip compression tool) to compress the stream for storage. Please take into consideration time, memory, and CPU usage for any of the compressors used. (default "internal")
84+
--encryptTo string the email of the user to encrypt the data to from the provided public keyring.
85+
-h, --help help for zfsbackup
86+
--logLevel string this controls the verbosity level of logging. Possible values are critical, error, warning, notice, info, debug. (default "notice")
87+
--manifestPrefix string the prefix to use for all manifest files. (default "manifests")
88+
--numCores int number of CPU cores to utilize. Do not exceed the number of CPU cores on the system. (default 2)
89+
--publicKeyRingPath string the path to the PGP public key ring
90+
--secretKeyRingPath string the path to the PGP secret key ring
91+
--signFrom string the email of the user to sign on behalf of from the provided private keyring.
92+
--workingDirectory string the working directory path for zfsbackup. (default "~/.zfsbackup")
93+
94+
Use "zfsbackup [command] --help" for more information about a command.
95+
```
96+
97+
## TODOs:
98+
* Make PGP cipher configurable.
99+
* Finish the verify command
100+
* Build out more robust restore options (e.g. cascading, parent verification, etc.)
101+
* Refactor
102+
* Test Coverage
103+
* Add more backends (e.g. Azure, BackBlaze, etc.)
104+
* Fix error handling (at least omit panic dumps!)
105+
* Figure out high memory usage during restores
106+
* Add delete feature
107+
* Appease linters
108+
109+

0 commit comments

Comments
 (0)