-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathInstructions-s3cmd-windows.txt
218 lines (157 loc) · 8.26 KB
/
Instructions-s3cmd-windows.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
Deploying to S3 upon Git Push
October 23, 2013 at 4:00 PM PDT
With a simple post-receive hook and using s3cmd, you can have Git deploy to S3 after a pushing to your remote repository. If you're simply interested in the hook code, I have provided it at the bottom of this post.
Setting up s3cmd
To get started, you'll want to configure s3cmd on the user account that is holding the bare repository with your either security credentials of your AWS account or security credentials of an IAM user. I highly recommend creating a dedicated IAM user for s3cmd with an user policy that grants it full control to S3 and use its security credentials rather than giving it unlimited permissions by using your AWS account security credentials.
$ s3cmd --configure
You will be prompted for the access key and secret key:
Access key and Secret key are your identifiers for Amazon S3
Access Key: ACCESSKEY
Secret Key: SECRETKEY
Next, you'll be prompted for a GPG encryption key and the path to GPG that will be used when transferring files to S3. You can leave these blank to not use GPG when transferring.
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
Then, you'll be prompted if you want to use HTTPS when transferring files:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:
If you said no to HTTPS, you will be able to provide a proxy. Leave the proxy name blank if you do not wish to provide a proxy.
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:
HTTP Proxy server port [0]:
You will then have a chance to review what you have provided and to test access with the supplied credentials.
New settings:
#!/bin/sh
Access Key: ACCESSKEY
Secret Key: SECRETKEY
Encryption password:
Path to GPG program: /usr/bin/gpg
#!/bin/sh
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n]
If all goes well, you will be provided with the following:
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Encryption will also be tested. Finally, you will be prompted whether to save the configuration.
Save settings? [y/N] Y
Configuration saved to '/home/git/.s3cfg'
Setting up the hook
Navigate into the working directory of your bare Git repository. Then, open up hooks/post-receive in your favorite text editor. Let's start with the following:
#!/bin/sh
S3_BUCKET=yourbucket
TEMP_DEPLOY_DIR=/tmp/$S3_BUCKET/
These are variables we will be working within the hook. You'll want to set S3_BUCKET to the actual name of your S3 bucket. Currently, we'll be writing to a directory named after the bucket name in /tmp/, however you can change this if necessary.
We will want to ensure the temporary directory is clean and any Git environment variables aren't going to conflict, so we'll add the following to the hook:
# Ensure that the temporary directory is clean and unset potential conflicting
# environment variables
rm -rf $TEMP_DEPLOY_DIR
unset GIT_DIR
unset GIT_WORK_TREE
Now we will want to set up populating the working tree. If you have no submodules in your repository, we will use the following:
# Create a working tree with a bare repo that does not have submodules
mkdir -p $TEMP_DEPLOY_DIR
export GIT_DIR=$(pwd)
export GIT_WORK_TREE=$TEMP_DEPLOY_DIR
git checkout -f
cd $TEMP_DEPLOY_DIR
If you do have submodules, dealing with them using the above method is problematic. I found the best solution is to make an entire clone of the repository in order to get the submodules to initialize and update properly:
# Create a working tree with a bare repo that has submodules
git clone $(pwd) $TEMP_DEPLOY_DIR
cd $TEMP_DEPLOY_DIR
git submodule update --init --recursive
Then, we can now sync the repository with S3:
# Sync with S3
s3cmd sync --delete-removed --acl-public --exclude '.git/*' ./ s3://$S3_BUCKET/
If anything should be preprocessed before syncing with S3, say a Jekyll site, we can build the site and sync only the _site directory:
# Build and sync
jekyll build
s3cmd sync --delete-removed --acl-public --exclude '.git/*' _site/ s3://$S3_BUCKET/
You will want to ensure anything ran from the hook is set up on the remote server, otherwise it will fail.
Finally, we clean up the temporary directory we were using to sync with S3.
# Clean up
cd ..
rm -rf $TEMP_DEPLOY_DIR
That's all there is to it. Git will now deploy to your S3 bucket each time you push to your remote repository.
Example post-receive hook
Here is the complete post-receive hook code.
#!/bin/sh
# post-receive hook that syncs with S3 upon a push
S3_BUCKET=yourbucket
TEMP_DEPLOY_DIR=/tmp/$S3_BUCKET/
# Ensure that the temporary directory is clean and unset potential conflicting
# environment variables
rm -rf $TEMP_DEPLOY_DIR
unset GIT_DIR
unset GIT_WORK_TREE
# Create a working tree with a bare repo that does not have submodules
mkdir -p $TEMP_DEPLOY_DIR
export GIT_DIR=$(pwd)
export GIT_WORK_TREE=$TEMP_DEPLOY_DIR
git checkout -f
cd $TEMP_DEPLOY_DIR
# If the repo has submodules, comment out ore remove the above and uncomment the below:
#
# git clone $(pwd) $TEMP_DEPLOY_DIR
# cd $TEMP_DEPLOY_DIR
# git submodule update --init --recursive
# Sync with S3
s3cmd sync --delete-removed --acl-public --exclude '.git/*' ./ s3://$S3_BUCKET/
# If you use Jekyll, comment out or remove the above line and uncomment the below:
#
# jekyll build
# s3cmd sync --delete-removed --acl-public --exclude '.git/*' _site/ s3://$S3_BUCKET/
-------------------------------------------------------------------------------------------------------------
S3cmd System Requirements: s3cmd required Python 2.4 or greater version to run. We also need to install GPG.
Step 1: Install Python
Download and install python 2.4 or higher version from python official site and install it.
https://www.python.org/downloads/
After installing python, append the Python in Path environment variable. Open command prompt and verify python version.
C:> python --version
Python 2.7.6
Step 2: Install GPG for Windows
Gpg4win (GNU Privacy Guard for Windows) is Free Software to install. Use following link to download and install it.
http://www.gpg4win.org/download.html
Step 3: Download S3cmd and Configure
Download latest s3cmd source code from s3cmd official page and extract to c:s3cmd location.
http://s3tools.org/download
After extracting source code, use following command to setup s3 environment. It will ask for your s3 accounts AccessKey and SecretKey, path to GPG command like below.
C:s3cmd> python s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key: XXXXXXXXXXXXXXXXXXXX
Secret Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: XXXXXXXXX
Path to GPG program: C:Program Files (x86)GNUGnuPGgpg2.exe
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes
New settings:
Access Key: XXXXXXXXXXXXXXXXXXXX
Secret Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Encryption password: XXXXXXXXX
Path to GPG program: C:Program Files (x86)GNUGnuPGgpg2.exe
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)
Save settings? [y/N] Y
Configuration saved to 'C:UsersAdministratorApplication Datas3cmd.ini'
Step 4: Verify S3cmd
To verify s3cmd configuration use following command. It will list the buckets create in your configured s3 account.
C:> python c:s3cmds3cmd ls
2014-02-03 06:37 s3://tecadmin
2014-03-29 07:56 s3://tecadminbackups