Skip to content

Commit 46474d1

Browse files
neilblairneilblair
committed
Merge commit 'a70776ebf5914071bac202f94292b9c46db0e411' as 'nicsdru-logging'
2 parents 683ce11 + a70776e commit 46474d1

File tree

4 files changed

+394
-0
lines changed

4 files changed

+394
-0
lines changed

‎nicsdru-logging/LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2022 Digital Shared Services
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎nicsdru-logging/README.md

+263
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
# Send Platform.sh logs to Logz.io
2+
3+
Platform.sh log files are stored on disk (in /var/log) and trimmed
4+
to 100 MB automatically. For busy environments this means that logs
5+
may only contain entries for the last 24 hours or even less
6+
depending on the volume of log entries an environment is generating.
7+
By shipping logs to a centralised logging service like Logz.io, it
8+
makes it possible to retain logs for a longer period as well as
9+
having nice tools to search through the logs.
10+
11+
Platform.sh grid hosted environments currently do not support
12+
installation of logging services or daemons. For example, syslog nor
13+
rsyslog are available. To get around this, we can use cron to run a
14+
script which periodically uploads batches of log entries using cURL.
15+
16+
This solution is loosely based on
17+
[this solution for uploading logs to Amazon S3 storage](https://gitlab.com/contextualcode/platformsh-store-logs-at-s3/).
18+
19+
The solution only ever uploads the newest
20+
log entries rather than uploading an entire log file. This is
21+
necessary because log management solutions set a maximum limit on
22+
the size of files that can be uploaded in a single request.
23+
24+
## 1. Add nicsdru-logging repo to the project root using `git subtree`
25+
26+
[git subtree](https://www.atlassian.com/git/tutorials/git-subtree) lets you nest one repository inside another as a
27+
sub-directory.
28+
29+
In your local copy of a platform hosted Drupal project,
30+
run the following command in the project root (which should contain
31+
the `.platform/` directory and the `.platform.app.yaml` file):
32+
33+
```shell
34+
git subtree add --prefix nicsdru-logging [email protected]:dof-dss/nicsdru-logging.git main --squash
35+
```
36+
37+
This will generate a couple of new commits into your project repo.
38+
39+
### Updating the repo
40+
41+
If you want to later update to the latest version of the nicsdru-logging
42+
repo, run the following command in the project root:
43+
44+
```shell
45+
git subtree pull --prefix nicsdru-logging [email protected]:dof-dss/nicsdru-logging.git main --squash
46+
```
47+
48+
## 2. Copy example.cronjob.sh to cronjob.sh
49+
50+
Copy `nicsdru-logging/scripts/example.cronjob.sh` into the project root renaming to `cronjob.sh`.
51+
52+
```shell
53+
cp nicsdru-logging/scripts/example.cronjob.sh cronjob.sh
54+
```
55+
56+
The directory structure for a typical dof-dss D9 project should
57+
end up looking something like this:
58+
59+
```
60+
├── .circleci/
61+
├── .platform/
62+
│ ├── solr_config
63+
│ ├── routes.yaml
64+
│ └── services.yaml
65+
├── config/
66+
├── drush/
67+
├── nicsdru-logging/
68+
│ └── scripts/
69+
│ │ ├── example.cronjob.sh
70+
│ │ └── shiplog.sh
71+
│ └── README.md
72+
├── private/
73+
├── vendor/
74+
├── web/
75+
├── .platform.app.yaml
76+
├── composer.json
77+
├── composer.lock
78+
├── cronjob.sh
79+
├── LICENSE
80+
├── phpcs.sh
81+
└── README.md
82+
```
83+
84+
85+
## 3. Make the new `cronjob.sh` file executable
86+
87+
```shell
88+
chmod +x cronjob.sh
89+
```
90+
91+
## 4. Edit `cronjob.sh` to configure log files to be shipped
92+
93+
`cronjob.sh` contains calls to `shiplog.sh` to ship specifc logs in the following format:
94+
95+
```shell
96+
/bin/bash /app/nicsdru-logging/scripts/shiplog.sh "LOG_NAME" "LOG_PATH" "LOG_DATE_PATTERN" "LOG_TYPE"
97+
```
98+
99+
Where:
100+
101+
- `LOG_NAME` is unique name of a log (e.g "access"). NO SPACES!
102+
- `LOG_PATH` is the full path to source log file to be shipped (e.g. "/var/log/access.log")
103+
- `LOG_DATE_PATTERN` is a date formatted to match datetime stamps in individual log entries
104+
(e.g. "$(date +%d/%b/%Y:)")
105+
- `LOG_TYPE` identifies [the type of log file to Logz.io](https://docs.logz.io/user-guide/log-shipping/built-in-log-types.html) (e.g. "nginx_access"). NO SPACES!
106+
107+
For example, the following line ships the /var/log/access.log generated by nginx.
108+
109+
```shell
110+
/bin/bash /app/nicsdru-logging/scripts/shiplog.sh "access" "/var/log/access.log" "$(date +%d/%b/%Y:)" "nginx_access"
111+
```
112+
113+
*For D9 sites*, with the [filelog contrib module](https://www.drupal.org/project/filelog)
114+
installed, ensure the module is configured to write to logs to `/app/log` and then
115+
uncomment or add the following line to `cronjob.sh`:
116+
117+
```shell
118+
/bin/bash /app/nicsdru-logging/scripts/shiplog.sh "drupal" "/app/log/drupal.log" "$(date +'%a, %d/%m/%Y -')" "drupal"
119+
```
120+
121+
(Note the LOG_TYPE specified is "drupal" which is a custom log type we have asked
122+
Logz.io to configure for us.)
123+
124+
*For D9 multisite projects* using the [filelog contrib module](https://www.drupal.org/project/filelog),
125+
ensure the module is configured on each site to write to logs to a sub-directory of
126+
`/app/log` - for example `/app/log/sitename` and then add a line like the
127+
following to ship the drupal.log for that specific site:
128+
129+
```shell
130+
/bin/bash /app/nicsdru-logging/scripts/shiplog.sh "drupal-sitename" "/app/log/sitename/drupal.log" "$(date +'%a, %d/%m/%Y -')" "drupal"
131+
```
132+
133+
## 5. Create a writable mount in .platform.app.yaml for storing logs
134+
135+
The normal log directory `/var/log` is not writeable. So we create
136+
a writeable mount in `.platform.app.yaml` for the shiplog script to write logs to:
137+
138+
```yaml
139+
mounts:
140+
'/log':
141+
source: local
142+
source_path: 'log'
143+
```
144+
145+
## 6. Add cron job in .platform.app.yaml
146+
147+
```yaml
148+
crons:
149+
# Log shipping cron.
150+
logging:
151+
spec: '*/5 * * * *'
152+
commands:
153+
start: '/bin/bash /app/cronjob.sh'
154+
shutdown_timeout: 290
155+
```
156+
157+
## 7. Create LOGZ_TOKEN and LOGZ_URL environment variables
158+
159+
Obtain an account token from Logz.io. Then create a project
160+
environment variable LOGZ_TOKEN to store the token securely.
161+
162+
```shell
163+
platform variable:create -l environment -e [ENVIRONMENT_NAME] --prefix env: --name LOGZ_TOKEN --value '[LOGZ_IO_TOKEN]' --visible-runtime true --inheritable false --sensitive true
164+
```
165+
166+
**Optionally**, set the "listener URL" for sending logs to
167+
Logz.io. By default the URL is `https://listener-uk.logz.io:8022`
168+
You can override this by setting the project environment variable
169+
LOGZ_URL. The URL should begin with "https://" with no end
170+
forward slash.
171+
172+
```shell
173+
platform variable:create -l environment -e [ENVIRONMENT_NAME] --prefix env: --name LOGZ_URL --value 'https://listener-uk.logz.io:8022' --visible-runtime true --inheritable true --sensitive false
174+
```
175+
176+
## 8. Deploy to Platform.sh and verify logs are being shipped
177+
178+
Cron will run cronjob.sh to ship latest log entries to Logz.io.
179+
You should start to see log entries arrive in Logz.io's Kibana
180+
interface. If you don't, there are a number of things to check.
181+
182+
### Check cron is running
183+
184+
Run this platform CLI command
185+
```shell
186+
$ platform activity:list
187+
```
188+
And gives output like this ...
189+
190+
```shell
191+
Activities on the project NIDirect-D8 (wcjm3mu7bacfm), environment logz (type: development):
192+
+---------------+---------------------------+----------------------------------+----------+----------+---------+
193+
| ID | Created | Description | Progress | State | Result |
194+
+---------------+---------------------------+----------------------------------+----------+----------+---------+
195+
| cc3s4jkfkb5ke | 2022-05-11T16:01:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
196+
| yado3ie3w4rqu | 2022-05-11T15:56:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
197+
| tuckl2vxeecss | 2022-05-11T15:51:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
198+
| z7jx33erazc3a | 2022-05-11T15:46:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
199+
| iq43xajukyn5e | 2022-05-11T15:41:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
200+
| q64gu6qqiw6s4 | 2022-05-11T15:36:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
201+
| dc2hifwrzgm6y | 2022-05-11T15:31:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
202+
| 3l737f5rmp6m4 | 2022-05-11T15:26:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
203+
| kuydxo23aeczy | 2022-05-11T15:21:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
204+
| 6jnwgge2ufivk | 2022-05-11T15:16:27+00:00 | Platform.sh Bot ran cron logging | 100% | complete | success |
205+
+---------------+---------------------------+----------------------------------+----------+----------+---------+
206+
```
207+
208+
### SSH onto the environment and check logs are being generated in the `/app/log` mount
209+
210+
The logging scripts create and rotate their own log files in the `/app/log`
211+
mounted directory.
212+
213+
If the project is a D9 project with the filelog module enabled and you
214+
have configured `cronjob.sh` to ship the access and drupal logs (as described above),
215+
then you should see something like the following in `/app/log`:
216+
217+
```shell
218+
[email protected]:~$ cd /app/log
219+
[email protected]:~/log$ ls -al
220+
total 88
221+
drwxr-xr-x 2 web web 4096 May 11 16:26 .
222+
drwxr-xr-x 18 web web 514 May 11 12:21 ..
223+
-r--r--r-- 1 web web 686 Apr 13 16:41 .htaccess
224+
-rw-r--r-- 1 web web 15743 May 11 12:31 2022-05-11-access.log
225+
-rw-r--r-- 1 web web 8038 May 11 12:31 2022-05-11-drupal.log
226+
-rw-rw-r-- 1 web web 51256 May 11 12:29 drupal.log
227+
[email protected]:~/log$
228+
```
229+
230+
The logs prefixed with a YYYY-MM-DD format date are created by `shiplog.sh`
231+
and rotated daily. The drupal.log is created by the filelog module which
232+
can also be rotated daily (depending on module settings).
233+
234+
### If all else fails, run `cronjob.sh` from the command-line and check the output
235+
236+
```shell
237+
/bin/bash /app/cronjob.sh
238+
```
239+
240+
Healthy output looks like this ...
241+
242+
```shell
243+
[email protected]:~$ /bin/bash /app/cronjob.sh
244+
Shipping log ...
245+
Shipping today's newest log entries from /var/log/access.log ...
246+
Creating 2022-05-11-access.log ...
247+
2022-05-11-access.log already exists
248+
Deleting 2022-05-10-access.log ...
249+
2022-05-10-access.log does not exist
250+
Retrieving latest log entries from /var/log/access.log and writing to 2022-05-11-access.log
251+
Shipping latest log entries from 2022-05-11-access.log to Logz.io using cURL
252+
Log shipping succeeded with: 200
253+
Shipping log ...
254+
Shipping today's newest log entries from /app/log/drupal.log ...
255+
Creating 2022-05-11-drupal.log ...
256+
2022-05-11-drupal.log already exists
257+
Deleting 2022-05-10-drupal.log ...
258+
2022-05-10-drupal.log does not exist
259+
Retrieving latest log entries from /app/log/drupal.log and writing to 2022-05-11-drupal.log
260+
Shipping latest log entries from 2022-05-11-drupal.log to Logz.io using cURL
261+
Log shipping succeeded with: 200
262+
```
263+

‎nicsdru-logging/scripts/example.cronjob.sh

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/usr/bin/env bash
2+
#
3+
# nicsdru-logging/scripts/example.cronjob.sh
4+
#
5+
# Calls shiplog.sh to ship latest log entries from various logs
6+
# to logz.io.
7+
8+
# /bin/bash /app/nicsdru-logging/scripts/shiplog.sh "[LOG_NAME]" "[LOG_PATH]" "[LOG_DATE_PATTERN]" "[LOG_TYPE]"
9+
10+
# Ship /var/log/access.log generated by nginx.
11+
/bin/bash /app/nicsdru-logging/scripts/shiplog.sh "access" "/var/log/access.log" "$(date +%d/%b/%Y:)" "nginx_access"
12+
13+
# Uncomment following line to ship drupal.log generated by D9 filelog module.
14+
# /bin/bash /app/nicsdru-logging/scripts/shiplog.sh "drupal" "/app/log/drupal.log" "$(date +'%a, %d/%m/%Y -')" "drupal"

‎nicsdru-logging/scripts/shiplog.sh

+96
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
#!/usr/bin/env bash
2+
#
3+
# logging/scripts/shiplog.sh
4+
#
5+
# Ship today's log entries from a specified log to logz.io
6+
# Usage: /bin/bash /app/logging/scripts/shiplog.sh [LOG_NAME] [LOG_PATH] [LOG_DATE_PATTERN] [LOG_TYPE]
7+
# eg: /bin/bash /app/logging/scripts/shiplog.sh "access" "/var/log/access.log" "$(date +%d/%b/%Y:)" "nginx_access"'
8+
9+
echo "Shipping log ..."
10+
11+
# LOGZ_TOKEN environment variable is required for this script to run.
12+
if [ -z "$LOGZ_TOKEN" ]; then
13+
echo "LOGZ_TOKEN is not set - exiting"
14+
exit 1
15+
fi
16+
17+
# Set LOGZ_URL if environment variable is not set.
18+
if [ -z "$LOGZ_URL" ]; then
19+
LOGZ_URL="https://listener-uk.logz.io:8022"
20+
fi
21+
22+
# Mount for logs must exist or exit script.
23+
cd /app/log || exit
24+
25+
LOG_NAME=$1
26+
LOG_PATH=$2
27+
LOG_DATE_PATTERN=$3
28+
LOG_TYPE=$4
29+
30+
# Script will be creating logs for today and removing yesterday's logs
31+
TODAY_DATE=$(date +%Y-%m-%d)
32+
YESTERDAY_DATE=$(date --date="yesterday" +%Y-%m-%d)
33+
34+
# Check 4 arguments passed.
35+
if [ $# -ne 4 ]; then
36+
echo "shiplog called with incorrect number of arguments. Expected 4, got $#."
37+
echo 'Usage: ./shiplog.sh [LOG_NAME] [LOG_PATH] [LOG_DATE_PATTERN] [LOG_TYPE]'
38+
echo ' eg: ./shiplog.sh "access" "/var/log/access.log" "$(date +%d/%b/%Y:)" "nginx_access"'
39+
exit 1
40+
fi
41+
42+
if [ -f "$LOG_PATH" ]; then
43+
44+
echo "Shipping today's newest log entries from $LOG_PATH ..."
45+
46+
# Create today's log file.
47+
todays_log="${TODAY_DATE}-${LOG_NAME}.log"
48+
echo "Creating ${todays_log} ..."
49+
if [ ! -f ./"${todays_log}" ]; then
50+
touch ./"${todays_log}"
51+
echo "${todays_log} created"
52+
else
53+
echo "${todays_log} already exists"
54+
fi
55+
56+
# Delete yesterdays log files.
57+
yesterdays_log="${YESTERDAY_DATE}-${LOG_NAME}.log"
58+
59+
echo "Deleting ${yesterdays_log} ..."
60+
61+
if [ -f ./"${yesterdays_log}" ]; then
62+
rm "${yesterdays_log}"
63+
echo "${yesterdays_log} deleted"
64+
else
65+
echo "${yesterdays_log} does not exist"
66+
fi
67+
68+
# Get latest log entries and ship to logz.io.
69+
echo "Retrieving latest log entries from ${LOG_PATH} and writing to ${todays_log}"
70+
cat "$LOG_PATH" | grep "$LOG_DATE_PATTERN" > ./"$LOG_NAME"-latest.log
71+
diff --changed-group-format='%>' --unchanged-group-format='' "$todays_log" "$LOG_NAME-latest.log" > "$LOG_NAME-new.log"
72+
cat "$LOG_NAME-new.log" >> "$todays_log"
73+
echo "Shipping latest log entries from ${todays_log} to Logz.io using cURL"
74+
http_response=$(curl -T "$LOG_NAME-new.log" -s -w "%{response_code}" $LOGZ_URL/file_upload/"${LOGZ_TOKEN}"/"${LOG_TYPE}")
75+
exit_code=$?
76+
77+
# Clean up temporary log files.
78+
rm "$LOG_NAME-latest.log" "$LOG_NAME-new.log"
79+
80+
if [ "$exit_code" != "0" ]; then
81+
echo "The cURL command failed with: $exit_code"
82+
elif [ "$http_response" != "200" ]; then
83+
echo "Log shipping failed with: $http_response"
84+
fi
85+
86+
if [ "$exit_code" != "0" ] || [ "$http_response" != "200" ]; then
87+
exit 1
88+
fi
89+
90+
echo "Log shipping succeeded with: $http_response"
91+
else
92+
echo "${LOG_PATH} does not exist"
93+
exit 1
94+
fi
95+
96+
exit 0

0 commit comments

Comments
 (0)