Skip to content

Commit d9b2aef

Browse files
dekobondekobon
committed
Refactored docs and moved ubuntu install script to root
1 parent 403600c commit d9b2aef

File tree

5 files changed

+276
-127
lines changed

5 files changed

+276
-127
lines changed

‎README.md

+52-127
Original file line numberDiff line numberDiff line change
@@ -1,157 +1,82 @@
11
# NGINX S3 Gateway
22

3-
This project provides an example of configuring NGINX to act as an authenticating
4-
and caching gateway for read-only requests (GET/HEAD) to the S3 API.
3+
## Introduction
54

6-
## Potential Use Cases
5+
This project provides a working configuration of NGINX configured to act as
6+
an authenticating and caching gateway for to AWS S3 or another S3 compatible
7+
service. This allows you to proxy a private S3 bucket without requiring users
8+
to authenticate to it. Within the proxy layer, additional functionality can be
9+
configured such as:
710

11+
* Listing the contents of a S3 bucket
812
* Providing an authentication gateway using an alternative authentication
913
system to S3
10-
* Caching frequently accessed S3 objects for lower latency delivery and
14+
* Caching frequently accessed S3 objects for lower latency delivery and
1115
protection against S3 outages
1216
* For internal/micro services that can't authenticate against the S3 API
1317
(e.g. don't have libraries available) the gateway can provide a means
14-
to accessing S3 objects without authentication
18+
to accessing S3 objects without authentication
1519
* Compressing objects ([gzip](examples/gzip-compression), [brotli](examples/brotli-compression)) from gateway to end user
1620
* Protecting S3 bucket from arbitrary public access and traversal
1721
* Rate limiting S3 objects
1822
* Protecting a S3 bucket with a [WAF](examples/modsecurity)
19-
* Serving static assets from a S3 bucket alongside a dynamic application
23+
* Serving static assets from a S3 bucket alongside a dynamic application
2024
endpoints all in a single RESTful directory structure
2125

22-
## Usage
23-
24-
Few users will find this project as-is to be sufficient for their use cases. As
25-
such, it is best to borrow from the patterns in this project and build your
26-
own configuration. For example, if you want enable SSL/TLS and compression
27-
in your NGINX S3 gateway configuration, you will need to look at other
28-
documentation because this project does not enable those features of NGINX.
29-
30-
## Examples
31-
32-
In this project, we provide a few configuration examples that extend the
33-
base functionality of the NGINX S3 Gateway.
34-
35-
* [Enabling Brotli Compression in Docker](examples/brotli-compression)
36-
* [Enabling GZip Compression in Docker](examples/gzip-compression)
37-
* [Installing Modsecurity in Docker](examples/modsecurity)
38-
* [Stand-alone Configuration on Ubuntu](examples/ubuntu_install)
39-
40-
### Building the Docker Image
41-
42-
#### NGINX OSS
26+
All such functionality can be enabled within a standard NGINX configuration
27+
because this project is nothing other than NGINX with additional configuration
28+
that allows for proxying S3. It can be used as-is if the predefined
29+
configuration is sufficient, or it can serve as a base example for a more
30+
customized configuration.
4331

44-
In order to build the NGINX OSS container image, do a `docker build` as follows:
45-
```
46-
docker build -f Dockerfile.oss -t nginx-oss-s3-gateway .
47-
```
48-
49-
#### NGINX Plus
50-
51-
In order to build the NGINX Plus Docker image, copy your NGINX Plus repository
52-
keys (`nginx-repo.crt` and `nginx-repo.key`) into the `plus/etc/ssl/nginx`
53-
directory. Then build the container image.
32+
If the predefined configuration does not meet your needs, it is best to borrow
33+
from the patterns in this project and build your own configuration. For example,
34+
if you want to enable SSL/TLS and compression in your NGINX S3 gateway
35+
configuration, you will need to look at other documentation because this
36+
project does not enable those features of NGINX.
5437

55-
If you are using a version of Docker that supports Buildkit, then you can
56-
build the image as follows in order to prevent your private keys from
57-
being stored in the container image.
58-
59-
```
60-
DOCKER_BUILDKIT=1 docker build \
61-
-f Dockerfile.buildkit.plus \
62-
-t nginx-plus-s3-gateway \
63-
--secret id=nginx-crt,src=plus/etc/ssl/nginx/nginx-repo.crt \
64-
--secret id=nginx-key,src=plus/etc/ssl/nginx/nginx-repo.key \
65-
--squash .
66-
```
67-
68-
Otherwise, if you don't have Buildkit available, then build as follows. If you
69-
want to remove the private keys from the image, then you may need to do a
70-
post-build squash operation using a utility like
71-
[docker-squash](https://pypi.org/project/docker-squash/).
72-
73-
```
74-
docker build -f Dockerfile.plus -t nginx-plus-s3-gateway .
75-
```
76-
77-
### Configuration
78-
79-
Environment variables are used to configure this project.
38+
## Usage
8039

81-
* `ALLOW_DIRECTORY_LIST` - Enable directory listing - either true or false
82-
* `AWS_SIGS_VERSION` - AWS Signatures API version - either 2 or 4
83-
* `DNS_RESOLVERS` - (optional) DNS resolvers (separated by single spaces) to configure NGINX with
84-
* `S3_ACCESS_KEY_ID` - Access key
85-
* `S3_BUCKET_NAME` - Name of S3 bucket to proxy requests to
86-
* `S3_DEBUG` - Flag (true/false) enabling AWS signatures debug output (default: false)
87-
* `S3_REGION` - Region associated with API
88-
* `S3_SECRET_KEY` - Secret access key
89-
* `S3_SERVER_PORT` - SSL/TLS port to connect to
90-
* `S3_SERVER_PROTO` - Protocol to used connect to S3 server - `http` or `https`
91-
* `S3_SERVER` - S3 host to connect to
92-
* `S3_STYLE` - The S3 host/path method - `virtual`, `path` or `default`. `virtual` is
93-
the method that that uses DNS-style bucket+hostname:port.
94-
This is the `default` value. `path` is a method that appends the bucket name
95-
as the first directory in the URI's path. This method is used by many S3
96-
compatible services. See this
97-
[AWS blog article](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/)
98-
for further information.
99-
* `PROXY_CACHE_VALID_OK` - Sets caching time for response code 200 and 302
100-
* `PROXY_CACHE_VALID_NOTFOUND` - Sets caching time for response code 404
101-
* `PROXY_CACHE_VALID_FORBIDDEN` - Sets caching time for response code 403
40+
This project can be run as a stand-alone container or as a Systemd service.
41+
Both modes use the same NGINX configuration and are functionally equal in terms
42+
features. However, in the case of running as a Systemd service, other services
43+
can be configured that additional functionality such as [certbot](https://certbot.eff.org/)
44+
for [Let's Encrypt](https://letsencrypt.org/) support.
10245

103-
The above environment variables can be set in a file that is passed to docker
104-
with the `--env-file` flag. The file would look something like
105-
[this example](settings.example).
46+
## Getting Started
10647

107-
The container can be run by (replacing `oss` with `plus` when invoking the NGINX
108-
Plus container):
109-
```
110-
docker run --env-file ./settings -p80:80 --name nginx-oss-s3-gateway nginx-s3-gateway
111-
```
48+
Refer to the [Getting Started Guide](docs/getting_started.md) for how to build
49+
and run the gateway.
11250

11351
## Directory Structure and File Descriptions
114-
115-
```
116-
common/ contains files used by both NGINX OSS and Plus configurations
117-
examples/ contains additional `Dockerfile` examples that extend the base
118-
configuration
119-
oss/ contains files used solely in NGINX OSS configurations
120-
plus/ contains files used solely in NGINX Plus configurations
121-
test/ contains automated tests for validang that the examples work
122-
Dockerfile.oss Dockerfile that configures NGINX OSS to act as a S3 gateway
123-
Dockerfile.plus Dockerfile that builds a NGINX Plus instance that is configured
124-
equivelently to NGINX OSS - instance is configured to act as a
125-
S3 gateway with NGINX Plus additional features enabled
126-
settings.example Docker env file example
127-
test.sh test launcher
128-
```
129-
130-
## Directory Listing
13152

132-
Listing of S3 directories ([folders](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html)) is supported when the `ALLOW_DIRECTORY_LIST` environment variable is set
133-
to `1`. Directory listing output can be customized by changing the [XSL stylesheet](https://www.w3schools.com/xml/xsl_intro.asp): [`common/etc/nginx/include/listing.xsl`](./common/etc/nginx/include/listing.xsl).
134-
If you are not using AWS S3 as your backend, you may see some inconsistency in the
135-
behavior with how directory listing works with HEAD requests. Additionally, due
136-
to limitations in proxy response processing, invalid S3 folder requests will result
137-
in log messages like:
13853
```
139-
libxml2 error: "Extra content at the end of the document"
54+
common/ contains files used by both NGINX OSS and Plus configurations
55+
deployments/ contains files used for deployment technologies such as
56+
CloudFormation
57+
docs/ contains documentation about the project
58+
examples/ contains additional `Dockerfile` examples that extend the base
59+
configuration
60+
oss/ contains files used solely in NGINX OSS configurations
61+
plus/ contains files used solely in NGINX Plus configurations
62+
test/ contains automated tests for validang that the examples work
63+
Dockerfile.oss Dockerfile that configures NGINX OSS to act as a S3 gateway
64+
Dockerfile.plus Dockerfile that builds a NGINX Plus instance that is configured
65+
equivelently to NGINX OSS - instance is configured to act as a
66+
S3 gateway with NGINX Plus additional features enabled
67+
Dockerfile.buildkit.plus Dockerfile with the same configuration as Dockerfile.plus, but
68+
with support for hiding secrets using Docker's Buildkit
69+
Dockerfile.latest-njs Dockerfile that inherits from the last build of the gateway and
70+
then builds and installs the latest version of njs from source
71+
settings.example Docker env file example
72+
standalone_ubuntu_oss_install.sh install script that will install the gateway as a Systemd service
73+
test.sh test launcher
14074
```
14175

142-
Another limitation is that when using v2 signatures with HEAD requests, the
143-
gateway will not return 200 for valid folders.
76+
## Development
14477

145-
## Testing
146-
147-
Automated tests require `docker`, `docker-compose`, `curl` and `md5sum` to be
148-
installed. To run all unit tests and integration tests, run the following command.
149-
If you invoke the test script with the plus parameter, you will need to add your
150-
NGINX repository keys to the `plus/etc/ssl/nginx` directory
151-
152-
```
153-
$ ./test.sh <nginx type - 'oss' or 'plus'>
154-
```
78+
Refer to the [Development Guide](docs/development.md) for more information about
79+
extending or testing the gateway.
15580

15681
## License
15782

‎deployments/ecs/cloudformation/s3gateway.cf

Whitespace-only changes.

‎docs/development.md

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Development Guide
2+
3+
## Extending the Gateway
4+
5+
In the [examples/ directory](/examples), there are `Dockerfile` examples that
6+
show how to extend the base functionality of the NGINX S3 Gateway by adding
7+
additional modules.
8+
9+
* [Enabling Brotli Compression in Docker](examples/brotli-compression)
10+
* [Enabling GZip Compression in Docker](examples/gzip-compression)
11+
* [Installing Modsecurity in Docker](examples/modsecurity)
12+
13+
## Testing
14+
15+
Automated tests require `docker`, `docker-compose`, `curl` and `md5sum` to be
16+
installed. To run all unit tests and integration tests, run the following command.
17+
If you invoke the test script with a plus parameter, you will need to add your
18+
NGINX repository keys to the `plus/etc/ssl/nginx` directory
19+
20+
```
21+
$ ./test.sh <nginx type - 'oss', 'latest-njs-oss', 'plus', or 'latest-njs-plus'>
22+
```

0 commit comments

Comments
 (0)