Skip to content

Commit 87d7bd1

Browse files
LaberbearLaberbear
committed
Initial Commit
Add Google Maps -> Exif and Whatsapp Exif Date tools Add dprint Add eslint Add big readme
1 parent 1618ced commit 87d7bd1

20 files changed

+4662
-0
lines changed

‎.gitignore

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
node_modules
2+
testphotolibrary
3+
testwhatsapp
4+
data
5+
bin
6+
location-history
7+
.eslintcache

‎README.md

+128
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Exif Tools
2+
3+
This repository is a collection of the Exif Tools I wrote to manage my photo libraries a bit better.
4+
5+
## Before running any of this, create backups of your photos
6+
7+
I tested these tools on my personal files and all went well, but always be careful when it comes to your precious memories.
8+
9+
## Prerequisites
10+
11+
You need to use Node 20 (best installed using nvm).
12+
Then clone this repository and the install the dependencies with the command:
13+
14+
```
15+
npm ci
16+
```
17+
18+
After that you are ready to get started.
19+
20+
## Current Tools
21+
22+
### Google Maps -> Exif GPS Data
23+
24+
If you took Photos in the past but did not enable Location Data in your Camera app but want it, this might help you.
25+
It analyzes your Google Maps Timeline and if it finds your location for Photos that do not have Exif GPS data yet, it will add it.
26+
27+
The setup for this is a bit more involved due to the sheer amount of data you could get from the Google Maps Takeout.
28+
29+
First get your Google Maps Location History takeout:
30+
https://takeout.google.com/
31+
32+
Location History (Timelime) is what you want.
33+
Deselect all the other options to avoid getting giant export.
34+
35+
After you have received your export, download and extract it.
36+
37+
The data we are interested in is the semantic location history and the raw records.
38+
39+
First you probably have to split up your Records.json file.
40+
Due to limitations in nodejs, each file should not be bigger than 200MBs.
41+
I'd suggest to use a good text editor and just copying/pasting the values in the array into different files.
42+
The outcome files should have the array on a top level. eg. like this
43+
44+
```
45+
[{
46+
latitudeE7: ...,
47+
longitudeE7: ...,
48+
...
49+
}, {...}, {...}]
50+
```
51+
52+
After that we have to seed this data into an SQLite database, since the JSON format is just too inefficient to use it in the application.
53+
54+
Go to seedDatabase.js and modify the paths provided as parameters in loadRecordFile([...]) to point to all recordsX.json files you have created previously.
55+
56+
Then provide the semantic location history folder path to the findLocationFiles(...) function.
57+
58+
After that we are ready to seed the database!
59+
Run the following commands
60+
61+
```
62+
# Create the Empty SQLite database
63+
npm run migrate-db
64+
# Add the JSON data to the database
65+
npm run seed-migration
66+
```
67+
68+
This should take a while depending on the amount of location data available. (I tested it with 14k Place Visits and 1.5 Million raw location records)
69+
70+
When this script is done, the SQLite database is seeded properly with all your location data and we are ready to infuse your photos from that juicy GPS data.
71+
72+
The steps before you now never need to do again, hurray!
73+
74+
To now process your photos and add the GPS data, open up the index.js file.
75+
At the top of the file, add the paths to your photos to the photoLibs array.
76+
The script will check every image in that folder and add GPS data to it if:
77+
78+
- it's a valid image file
79+
- it already has Exif data with the date the photo was taken
80+
- it does not already have GPS data (unless the GPS data comes from this script for reruns)
81+
82+
Now run the script:
83+
84+
```
85+
node index.js
86+
```
87+
88+
Depending on the amount of images, this might take quite a bit of time.
89+
The script will output current status into your terminal.
90+
91+
After it's done, it will also output a summary of the actions it took.
92+
Please check that to make sure nothing went horribly wrong.
93+
94+
Your photos now have GPS Exif tags in them.
95+
96+
### WhatsApp Images Date Taken
97+
98+
By default WhatsApp strips any Exif Tags from Images that you receive/send.
99+
This might be a good thing for privacy, but if you also backup your WhatsApp images it can lead to the photo showing up at random days (depending on createdAt/modifiedAt timestamps).
100+
101+
So this script will find all the WhatsApp Images (based on naming pattern) in the specified folders.
102+
Then for each it will parse the filename and set the Exif.Photo.DateTime tags to the day you have received/sent this image.
103+
It also looks at the modified date of the file and if it matches the date it will take over the time too.
104+
If the modifie date does not match up with the date parsed from the filename, it will default to 12:00.
105+
106+
Right now it also simply hardcodes +02:00 timezone, so if you use this, be sure to change this to something sensible where you live.
107+
108+
Additionally it adds "WhatsApp" as Exif.Image.Make tag, which usually contains the camera/phone model. To easier identify WhatsApp images in your photo library.
109+
110+
Please Note: The original modified date of the file will not be changed, it will add the Exif Tags and then change the modified date to what it was before.
111+
112+
Examples:
113+
114+
- IMG-20170704-WA0000.jpg
115+
modified date 01.01.2019 17:45
116+
=> Exif Date: 04.07.2017 12:00
117+
118+
- IMG-20170704-WA0000.jpg
119+
modified date 04.07.2017 17:45
120+
=> Exif Date: 04.07.2017 17:45
121+
122+
To run the WhatsApp Image tool simply write:
123+
124+
```
125+
node whatsapp.js /PATH/TO/MY/WHATSAPP/FOLDER
126+
```
127+
128+
It should parse all the files in the folder which match the WhatsApp naming pattern and then get going.

‎db.js

+95
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
const { drizzle } = require('drizzle-orm/better-sqlite3');
2+
const Database = require('better-sqlite3');
3+
const {
4+
lte,
5+
desc,
6+
gte,
7+
asc,
8+
and,
9+
} = require('drizzle-orm');
10+
const { records, placeVisit } = require('./schema');
11+
12+
const sqlite = new Database('./data/sqlite.db');
13+
const db = drizzle(sqlite);
14+
15+
async function insertRecords(locations) {
16+
return db.insert(records).values(locations).run();
17+
}
18+
async function insertPlaceVisits(placeVisits) {
19+
return db.insert(placeVisit).values(placeVisits).run();
20+
}
21+
22+
async function removePlaceVists() {
23+
return db.delete(placeVisit).run();
24+
}
25+
26+
async function removeRecords() {
27+
return db.delete(records).run();
28+
}
29+
30+
async function findPlaceVisit(timestamp) {
31+
const fittingPlaceVisit = await db.select().from(placeVisit)
32+
.where(and(
33+
lte(placeVisit.startTimestamp, timestamp),
34+
gte(placeVisit.endTimestamp, timestamp),
35+
))
36+
.limit(1);
37+
if (fittingPlaceVisit.length) {
38+
return {
39+
...fittingPlaceVisit[0],
40+
lat: fittingPlaceVisit[0].lat / 10000000,
41+
lng: fittingPlaceVisit[0].lng / 10000000,
42+
};
43+
}
44+
return null;
45+
}
46+
47+
async function findRecord(timestamp) {
48+
if (!timestamp) {
49+
throw Error('timestamp is not gud');
50+
}
51+
const before = await db.select().from(records)
52+
.where(lte(records.timestamp, timestamp))
53+
.orderBy(desc(records.timestamp))
54+
.limit(1);
55+
const after = await db.select().from(records)
56+
.where(gte(records.timestamp, timestamp))
57+
.orderBy(asc(records.timestamp))
58+
.limit(1);
59+
const beforeRecord = before[0];
60+
const afterRecord = after[0];
61+
let beforeDelta = Number.MAX_SAFE_INTEGER;
62+
let afterDelta = Number.MAX_SAFE_INTEGER;
63+
if (!beforeRecord && !afterRecord) {
64+
return null;
65+
}
66+
if (beforeRecord) {
67+
beforeDelta = timestamp - before[0].timestamp;
68+
}
69+
if (afterRecord) {
70+
afterDelta = after[0].timestamp - timestamp;
71+
}
72+
73+
let selectedDelta = afterDelta;
74+
let [selected] = after;
75+
if (beforeDelta < afterDelta) {
76+
[selected] = before;
77+
selectedDelta = beforeDelta;
78+
}
79+
80+
return {
81+
...selected,
82+
lat: selected.lat / 10000000,
83+
lng: selected.lng / 10000000,
84+
delta: selectedDelta,
85+
};
86+
}
87+
88+
module.exports = {
89+
insertRecords,
90+
findRecord,
91+
insertPlaceVisits,
92+
findPlaceVisit,
93+
removePlaceVists,
94+
removeRecords,
95+
};

‎dprint.json

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"typescript": {
3+
"quoteStyle": "preferSingle"
4+
},
5+
"json": {
6+
},
7+
"markdown": {
8+
},
9+
"excludes": [
10+
"**/node_modules",
11+
"**/drizzle",
12+
"**/*-lock.json",
13+
"**/location-history"
14+
],
15+
"plugins": [
16+
"https://plugins.dprint.dev/typescript-0.93.0.wasm",
17+
"https://plugins.dprint.dev/json-0.19.3.wasm",
18+
"https://plugins.dprint.dev/markdown-0.17.8.wasm"
19+
]
20+
}

‎drizzle.config.json

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"schema": "./schema.js",
3+
"out": "./drizzle",
4+
"dialect": "sqlite",
5+
"dbCredentials": {
6+
"database": "locations",
7+
"url": "./data/sqlite.db"
8+
}
9+
}

‎drizzle/0000_complete_james_howlett.sql

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
CREATE TABLE `records` (
2+
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
3+
`lat` integer NOT NULL,
4+
`lng` integer NOT NULL,
5+
`accuracy` integer NOT NULL,
6+
`source` text NOT NULL,
7+
`timestamp` integer NOT NULL
8+
);
9+
--> statement-breakpoint
10+
CREATE INDEX `timestamp_idx` ON `records` (`timestamp`);

‎drizzle/0001_good_stone_men.sql

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
CREATE TABLE `place_visit` (
2+
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
3+
`lat` integer NOT NULL,
4+
`lng` integer NOT NULL,
5+
`address` text NOT NULL,
6+
`name` text NOT NULL,
7+
`placeId` text NOT NULL,
8+
`location_confidence` real NOT NULL,
9+
`start_timestamp` integer NOT NULL,
10+
`end_timestamp` integer NOT NULL
11+
);
12+
--> statement-breakpoint
13+
CREATE INDEX `place_visit_timestamp_idx` ON `place_visit` (`start_timestamp`,`end_timestamp`);

‎drizzle/meta/0000_snapshot.json

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
{
2+
"version": "6",
3+
"dialect": "sqlite",
4+
"id": "5e12adf9-0b7d-4294-ba94-6b6790c03f5d",
5+
"prevId": "00000000-0000-0000-0000-000000000000",
6+
"tables": {
7+
"records": {
8+
"name": "records",
9+
"columns": {
10+
"id": {
11+
"name": "id",
12+
"type": "integer",
13+
"primaryKey": true,
14+
"notNull": true,
15+
"autoincrement": true
16+
},
17+
"lat": {
18+
"name": "lat",
19+
"type": "integer",
20+
"primaryKey": false,
21+
"notNull": true,
22+
"autoincrement": false
23+
},
24+
"lng": {
25+
"name": "lng",
26+
"type": "integer",
27+
"primaryKey": false,
28+
"notNull": true,
29+
"autoincrement": false
30+
},
31+
"accuracy": {
32+
"name": "accuracy",
33+
"type": "integer",
34+
"primaryKey": false,
35+
"notNull": true,
36+
"autoincrement": false
37+
},
38+
"source": {
39+
"name": "source",
40+
"type": "text",
41+
"primaryKey": false,
42+
"notNull": true,
43+
"autoincrement": false
44+
},
45+
"timestamp": {
46+
"name": "timestamp",
47+
"type": "integer",
48+
"primaryKey": false,
49+
"notNull": true,
50+
"autoincrement": false
51+
}
52+
},
53+
"indexes": {
54+
"timestamp_idx": {
55+
"name": "timestamp_idx",
56+
"columns": [
57+
"timestamp"
58+
],
59+
"isUnique": false
60+
}
61+
},
62+
"foreignKeys": {},
63+
"compositePrimaryKeys": {},
64+
"uniqueConstraints": {}
65+
}
66+
},
67+
"enums": {},
68+
"_meta": {
69+
"schemas": {},
70+
"tables": {},
71+
"columns": {}
72+
},
73+
"internal": {
74+
"indexes": {}
75+
}
76+
}

0 commit comments

Comments
 (0)