Skip to content

Minio storage

Jump to bottom
Adrien Pavão edited this page Nov 29, 2019 · 13 revisions

What is Minio?

Minio is "a high performance distributed object storage server, designed for large-scale private cloud infrastructure. Minio is widely deployed across the world with over 104.4M+ docker pulls."

Basically, it allows us to create our own storage servers mocking AWS S3 storage.

Pre-reqs

Install docker:

    $ curl https://get.docker.com | sudo sh
    $ sudo usermod -aG docker $USER
    # reconnect

Open port 80 and optionally 443 for SSL

Minio running

TODO: Add more information here. Do we need to generate keys instead of using "1234567890"?

without SSL

docker run -p 80:80 \
  -e "MINIO_ACCESS_KEY=1234567890" \
  -e "MINIO_SECRET_KEY=1234567890" \
  -v ~/data:/data \
  -v ~/minio_config:/root/.minio \
  -d \
  minio/minio server --address ":80" /data

with SSL

Make sure your SSL certs are named public.crt and private.key, then place them in the ~/minio_config/certs directory.

docker run -p 443:443 \
  -e "MINIO_ACCESS_KEY=1234567890" \
  -e "MINIO_SECRET_KEY=1234567890" \
  -v ~/data:/data \
  -v ~/minio_config:/root/.minio \
  -d \
  minio/minio server --address ":443" /data

Codalab related tasks

  1. create private bucket, I named mine "private"
  2. create public bucket, I named mine "public"
  3. public bucket gets "read" perms
  4. Modify your codalab-competition's .env to use the settings, something like:
    # Minio
    DEFAULT_FILE_STORAGE=storages.backends.s3boto.S3BotoStorage
    AWS_ACCESS_KEY_ID=12345
    AWS_SECRET_ACCESS_KEY=12345
    AWS_STORAGE_BUCKET_NAME=public
    AWS_STORAGE_PRIVATE_BUCKET_NAME=private
    AWS_S3_CALLING_FORMAT=boto.s3.connection.OrdinaryCallingFormat
    
    # Make sure you change "YOURHOST" below to the proper hostname/address!
    AWS_S3_HOST=minio.YOURHOST.com

    AWS_QUERYSTRING_AUTH=False
    S3DIRECT_REGION=us-east-1
    S3_USE_SIGV4=True

Setting policies using minio's "mc" helper:

# We're making the minio docker container and getting a shell into it
$ docker run -it --entrypoint=/bin/sh minio/mc
# export AWS_S3_ENDPOINT_URL=http://172.17.0.1
# export MINIO_ACCESS_KEY=1234567890
# export MINIO_SECRET_KEY=1234567890
# export AWS_STORAGE_BUCKET_NAME=public
# export AWS_STORAGE_PRIVATE_BUCKET_NAME=private
# /usr/bin/mc config host add minio_docker $AWS_S3_ENDPOINT_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY;
# /usr/bin/mc mb minio_docker/$AWS_STORAGE_BUCKET_NAME;
# /usr/bin/mc mb minio_docker/$AWS_STORAGE_PRIVATE_BUCKET_NAME;
# /usr/bin/mc policy set download minio_docker/$AWS_STORAGE_BUCKET_NAME;

SSL

Setup (Certbot)[https://certbot.eff.org/]

    $ sudo apt-get update
    $ sudo apt-get install software-properties-common
    $ sudo add-apt-repository ppa:certbot/certbot
    $ sudo apt-get update
    $ sudo apt-get install certbot

And here are the official Minio SSL install docs

Add DNS TXT record

When you run it, it'll say where to add it and what to put in the TXT record

Run it

    sudo certbot certonly --manual --preferred-challenges dns -d minio.YOURHOST.com --staple-ocsp -m [email protected] --agree-tos

Move + rename files appropriately, they are symlinks so do it like so...

    sudo cp -Lr /etc/letsencrypt/live/minio.YOURHOST.com-0001/fullchain.pem public.crt
    sudo cp -Lr /etc/letsencrypt/live/minio.YOURHOST.com-0001/privkey.pem private.key

    ├── certs
    │   ├── CAs
    │   ├── private.key
    │   └── public.crt
    └── config.json

Renew it

    sudo certbot renew

Mount + format drive

If you need to use a large drive, these commands may be useful:

    # Format drive
    sudo mkfs.ext4 /dev/vdb

    # Mount it on boot, by adding this to /etc/fstab
    /dev/vdb        /mnt/data       ext4    defaults        0 0

    # Do this to make mount look for newly added drives
    sudo mount -a

Copy from * to Minio

Install rclone https://rclone.org/install/

    $ curl https://rclone.org/install.sh | sudo bash
    $ rclone config # add the servers

    # If you need to add more servers, `rclone config` again

    # This next command does the copying from your original server
    # to the new minio server, it may take a while!
    $ rclone copy <server name>:<bucket name> /your/path/to/storage

    # Then if you want to sync to a remote server, you swap source and destination
    $ rclone copy /your/path/to/storage  <server name>:<bucket name>

Backup to remote source cron job script

Edit the variables in the following script, naming it backup_storage.sh

#!/bin/bash
# Local paths to public and private storage
PUBLIC_STORAGE=/data/public
PRIVATE_STORAGE=/data/private

# The storage name you setup and the buckets for public and private storage
REMOTE_PUBLIC_STORAGE=azure-storage:public
REMOTE_PRIVATE_STORAGE=azure-storage:private

rclone copy $PUBLIC_STORAGE $REMOTE_PUBLIC_STORAGE
rclone copy $PRIVATE_STORAGE $REMOTE_PRIVATE_STORAGE

Add this to your @daily or @weekly cron jobs:

# Open cron
crontab -e

# Add this line
@daily /path/to/backup_storage.sh
Clone this wiki locally