This script is used to obtain, download and register into STAC catalogue Landsat Imagery data mainly for uses in CESNET, z.s.p.o.
App is divided into two cooperative parts downloader and http-server.
As the name suggest, this component is responsible for downloading data using USGS M2M API.
The data are downloaded one month (30 days) into the past with one day resolution.
The script runs in while True:
loop, and after downloading all data available at the moment,
it sleeps until the closest 9:00 AM UTC. At this time downloading is re-executed.
-
An account and user login token for USGS M2M API must be created. Please follow the instructions at https://m2m.cr.usgs.gov/.
-
File downloader/config/m2m_config.py must be filled with following information:
api_url = 'https://m2m.cr.usgs.gov/api/api/json/stable/'
username = 'username_used_for_login'
token = 'user_login_token'
- File downloader/config/s3_config.py must be filled with following information:
host_base = "https://s3.cl4.du.cesnet.cz"
access_key = "s3_access_key"
secret_key = "s3_secret_key"
host_bucket = "landsat"
- File downloader/config/stac_config.py must be filled with following information:
stac_base_url = 'https://stac.cesnet.cz/'
username = 'stac.cesnet.cz username'
password = 'stac.cesnet.cz password'
- You may also want to change contents of downloader/config/landsat_config.py, especially
the
s3_download_host
variable:
s3_download_host = "http://chronos.dhr.cesnet.cz:8081/"
The s3_download_host
must correspond to the computer on which the http-server component
is running.
Logging can be altered using downloader/config/landsat_config.py:
log_directory = "log"
log_name = "landsat.log"
log_level = 20
log_logger = "LandsatLogger"
log_directory
can be either relative to downloader/ or absolute.
Log is rotated every day at 12:00 AM UTC.
Powered by Sanic.
HTTP server acts as a relay between asset link published in STAC catalog and S3 storage. Using awscli the script generates a temporary link to download selected asset.
The http-server/.env file must be filled as follows:
AWS_CONFIG_FILE=~/.aws/config
AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials
AWS_ACCESS_KEY_ID=ASDFGHJKLQWERTYUIOP1
AWS_SECRET_ACCESS_KEY=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcd
AWS_DEFAULT_REGION=
AWS_DEFAULT_OUTPUT=text
AWS_DEFAULT_REGION=
got space after =, and it must be there as
said here.
There is not much what can be changed here. See beginning of http-server/main.py.
Lines:
host_name = "0.0.0.0"
server_port = 8080
could be altered, but I recommend to change settings of the Docker and firewall.
Logging can be altered using http-server/main.py lines:
log_logger = "HttpServerLogger"
log_directory = './log'
log_name = 'http-server.log'
log_level = 20
log_directory
can be either relative to http-server/ or absolute.
Log is rotated every day at 12:00 AM UTC.
Package is using Docker. Please see the corresponding docker-compose.yml files for downloader and http-server.
There is not much to change. In fact just the port of http-server in :
http-server:
ports:
- "8080:8080"
To run the package just install docker
and run docker compose up -d
command in both directories.
So to run the downloader in folder landsat/downloader
execute:
docker compose up -d
and do the same in folder landsat/http-server
to execute http-server.
There is also prepared a little script to run both of these docker containers.
Also in both docker-compose.yml files there are flags restart: unless-stopped
, and thus after rebooting the
machine, scripts will restart automatically.
Using m2m-api sources by Angel Farguell licensed under MIT License. Many thanks!