Cleaning up images

[gedankenstuecke]

By now the repo has grown quite drastically, with the 2025 static folder alone being nearly 250 MB in size, mostly for the image of the BOSC. This is mostly due to people having added full-size images that weigh multiple megabytes each.

Generally, we do not want/need to store, as they are never shown that big anywhere, so we could save a lot of space by replacing these full-size images with lower resolution ones. As the files are already in git, this would require some history rewrite later on, e.g. using https://rtyley.github.io/bfg-repo-cleaner/ which works for this.

So we should:

• Resize those large files with more adequate sizes (generally, I don't think anything larger than 800 pixels on the larger side of an image are needed)
• Clean up the repo history
• Clarify instructions in README & fellowship template to please not upload unnecessarily large images

[peterjc]

I've been using a personal max width/height of 1000 pixels, but 800 is also sensible.

We could also add a large image check as a git pre-commit hook, we use the tool pre-commit for this in Biopython (can be run locally, on https://pre-commit.ci/ for free, or on GitHub Actions etc). That would at least block new big images. I can set that up easily.

There is perhap an 'off the shelf' GitHub Action to update pull requests by resizing large images (which would then need a squash-megre to avoid the history bloat).

[peterjc]

See #134 for catching future excessive images automatically.

Some of the big images are PNG rather than JPG (edits combining two photos), we should convert those as part of resizing.

[nlharris]

Are jpgs more compact than pngs? I didn't realize that!

I apologize for uploading so many large images. As you probably know, I had to do a lot of work on the new website in a short time, and resizing all the images would have added considerably to the time it took. Thank you in advance for the post-commit help making them smaller.

[nlharris]

By the way, I think there are a lot of images that were brought over during the port to the new site that are not actually used anywhere. Is there a way to check for images that are not used anywhere and remove them from this repo?

[gedankenstuecke]

A good alternative format that's designed for the web is https://en.wikipedia.org/wiki/WebP, but not all tools easily support creating it yet I think.

The old images are less problematic, as they are all in the external repo, so most people don't have to interact with them (i.e. don't need to check them out at all).

[peterjc]

Yes - https://github.com/OBF/wp-content has the legacy images from WP which we deliberatly kept separate.

There is value is a little cleanup here in the new site repo (faster loads for people visiting the site) and in cleaning up the git history (for future contributors).

[peterjc]

Oh, the work on #134 will flag big images, but not resize them. That seems possible in theory...

It will fix trailing whitespace, but that's easier.

[nlharris]

i should make some of the images on the bosc site smaller for faster loading. maybe i will tackle that over the winter break. do you use a command-line tool to do that? i have done it by hand (usually in Preview or Mac photos)

[gedankenstuecke]

So, in #135, I've at least identified all images larger than 300kb and then resized them to not be more than 1024 px on their widest side.

I did so using this little bash-oneliner:

find . -type f -size +300k -exec file \{\} \;| awk -F: '{if ($2 ~/image/) print $1}'| while IFS= read -r file_path; do mogrify -resize 1024x1024\> "$file_path";done

The necessary mogrify tool is part of the ImageMagick command line tool.

What I noticed while doing that is that many photos seem to exist in duplicate for some reason. Eg there is CoFest - Ben Busby, Philippe Youkharibache, Jim Proctor on stage - 1.jpeg and CoFest - Ben Busby, Philippe Youkharibache, Jim Proctor on stage.jpeg - and the photos seem identical. @nlharris would you be able to go over the BOSC pages and identify which of the duplicates are actually used and delete the other one?

[nlharris]

wow, that's some pretty impressive command-line fu!

@nlharris would you be able to go over the BOSC pages and identify which of the duplicates are actually used and delete the other one?

maybe in a few weeks

[gedankenstuecke]

Also, I should say that #135 only makes the problem "worse for now", as git will happily store the old large files and the new smaller ones, but I think it's better if we first to do the resizing and deletion of the duplicates before we then do the cleanup that requires a destructive rewrite that removes all no-longer-referenced files