Exif errors in the latest RC2 + unusual asset upgrade behaviour

[Koli0842]

Hi,

After some metadata housekeeping, I am trying to redo my immich library and import my 20k+ assets. Looks like photos taken on my Sony Ericsson Xperia X10 have some odd metadata. The photos are readable and editable with ExifTool(GUI), but immich-go logs

error=can't read metadata: exif: decode failed (zero length tag value) (1)

I have uploaded a photo example. Could you please guide me to how to fix the photos in question, or track the issue?
2011-10-21 23.40.40.zip

Maybe worth noting, that I get an error for reading metadata for 3gp files, but there are no fallback currently, so they are uploaded with 1940s timestamps. I can work around this easily by using name based date inference. (2)

I have also noticed that photo stacking might be too sensitive, according to the docs 500 millisecs should be checked, but for me, photos taken within 5 seconds are still stacked in some instances (3)

Additionally, I see some asset upgrade related logs, where in theory, there should not be duplicates, locally I cannot find either a matching timestamp, or a matching filename, only a single asset. Could it be that files are detected multiple times during one import run, or that my server is overloaded with ingest, and reports incorrect data? My import process actually failed with an HTTP 400 Bad Request, related to a file like this. (4)

2024-12-07 19:28:54 INF server's asset upgraded with the input file.FileName=!Telefon:2014/2014-05-11 20.10.23.jpg file.FileDate=2014-05-11 20:10:24 file.Description= file.Title=2014-05-11 20.10.23.jpg file.FileSize=2340335 file.ID= file.CaptureDate=2014-05-11 20:10:23 file.Trashed=false file.Archived=false file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0 file.Longitude=0 reason=An asset with the same name:"2014-05-11 20.10.23.jpg" and date:"2014-05-11 22:10:23" but with smaller size:2.2 MB exists on the server. Replace it.
2024-12-07 19:28:54 INF uploaded file=!Telefon:2014/2014-05-11 20.10.23.jpg
2024-12-07 19:28:54 INF Tagged file=!Telefon:2014/2014-05-11 20.10.23.jpg tags=[!Telefon/2014 {immich-go}/2024-12-07 18:28:11]
2024-12-07 19:28:54 ERR error error=DeleteAsset, DELETE, https://__/api/assets, 400 Bad Request
Bad Request

(I am also unsure about timezone handling being correct, there is timezone metadata on the files, they are in UTC+2, and the file dates on the files have the local timestamp, but the date: logged by immich-go is an extra +2 hours forward to that. On some of my newer assets this works fine, even though I added OffsetTime, OffsetTimeOriginal, OffsetTimeDigitized retroactively to all files, based on my Pixel 7 photos which are handled perfectly. I am still looking into the exact differences, but looks like Nexus 5, ZTE Axon 7, Pixel 1, Pixel 7, it works correctly, and iPhone 3GS, Alcatel OT-995, OT-990, Sony Ericsson X10, Sony Ericsson W595, there are issues. If you happen to have a list of exif tags considered how to handle that, I could probably use that) (5)

Thanks for the great work on this project and the assistance

[simulot]

Thanks for this extended report.

You're right, I have changed time date determination:
Immich-go read the metadata of files for .jpg,mp4,.heic,.dng,cr2,.cr3 files. If valid date is found, the file is passed directly to immich. Otherwise, it tries to infer the date from the path and the file name.
The option --date-from-name, true by default, control this behavior.

The readme is here: https://github.com/simulot/immich-go/tree/next

2011-10-21 23.40.40.zip:
Thanks, I'll test the exif reading with this one.
exiftool does an incredible work to read metadata... Should I use it when it is installed?

3gp files:
I'm surprised by the year...
I don't have sample files of this type. You may share one privately on my discord @simulot

False duplicates:
Could you share the log file?

Time zones:
EXIF data may not have the timezone in the data, specially for old cameras and phones.
Also, some cameras names their files with a timestamp (yea!) in UTC or the local TZ (boo!)

There are some camera that places the TZ in a tag, but there isn't no standard we to do it.

This thread may help you: https://superuser.com/questions/1757307/how-to-set-an-images-date-and-time-with-timezone-with-exiftool

[Koli0842]

Thanks for your quick reply, I will reach out to you on discord, though I can't see it on your profile, but for the sake of completeness:

exiftool and metadata
I have exiftool on my machine and it's available on the path, I assume that's enough for immich-go to pick up and use it?
I also used exiftool in attempt to clean up my metadata and give the best chances of being properly recognized, like I added the timezone fields to old photos that didn't have it. My filenames are a bit of a mess pre-2017, very old assets are named local date (I used a program called Namexif to rename from the old DSCFxxx pattern, when I manually managed my library), and even after that,

3gp and timestamp
As for 3gp files, looks like it's not only those getting a funny timestamp, but for some reason, I have Pixel 7 recorded videos from 1912 and 2038. I also apparently have a photo from year 112 :) At the same time, it looks like a handful of videos only, and others taken with the same device are in the correct spot. Videos taken at a single event with the same device have a spread of 10 years on the timeline which is odd!

duplicate assets
I did some more digging, the root cause of my "false duplicate assets" is the timezone mismatch. For example:
2024-12-07 19:02:33 INF server has a better asset file=!Telefon:2013/2013-08-24 18.15.38.jpg reason=An asset with the same name:"2013-08-24 18.15.38.jpg" and date:"2013-08-24 20:15:38" but with bigger size:1.3 MB exists on the server. No need to upload.
I have a photo at both 18.15.38 and 20.15.38. Locally, these do not conflict, but after immich-go uploads it to immich, the 18.15.38 file receives a 20.15.38 timestamp. At this point, simply the bigger file will get stored. My solution here would probably be to sort out date parsing, I would appreciate some help in that regard, be it a gap in implementation or incomplete metadata in my end.

[GeekSheikh]

I have the same issue with multiple files. Below is the command I used. The worst part is that the upload just freezes when this gets hit and there seems to be no way to allow it to progress. This was from 0.23.0_RC2. Trying again with 0.22.1

immich-go.exe -s http:// -k upload from-google-photos takeout_1.zip

[simulot]

i'm working on @Koli0842 samples. A better version is coming soon.

The upload should not freeze...
Could you share the logs?

[AidanGee7]

I think I just had a similar issue to GeekSheikh, please see the logs below:

│2024-12-13 18:19:36 INF server has same asset file=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426.jpg reason=An asset with the same name:"eyeemfiltered1477073175426.jpg", date:"2016-04-10 14:57:51" and size:│
│2.6 MB exists on the server. No need to upload.                                                                                                                                                                                                       │
│2024-12-13 18:19:36 INF server's asset upgraded with the input file.FileName=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426-EFFECTS-edited.jpg file.FileDate=2024-12-12 18:33:25 file.Description=             │
│file.Title=eyeemfiltered1477073175426-EFFECTS.jpg file.FileSize=1907076 file.ID= file.CaptureDate=2016-04-10 14:57:51 file.Trashed=false file.Archived=true file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0 file.Longitude=0  │
│reason=An asset with the same name:"eyeemfiltered1477073175426-EFFECTS.jpg" and date:"2016-04-10 14:57:51" but with smaller size:1.8 MB exists on the server. Replace it.                                                                             │
│2024-12-13 18:19:36 INF uploaded file=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426-EFFECTS-edited.jpg                                                                                                        │
│2024-12-13 18:19:36 INF server's asset upgraded with the input file.FileName=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426-EFFECTS.jpg file.FileDate=2024-12-12 18:33:14 file.Description=                    │
│file.Title=eyeemfiltered1477073175426-EFFECTS.jpg file.FileSize=2323853 file.ID= file.CaptureDate=2016-04-10 14:57:51 file.Trashed=false file.Archived=true file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0 file.Longitude=0  │
│reason=An asset with the same name:"eyeemfiltered1477073175426-EFFECTS.jpg" and date:"2016-04-10 14:57:51" but with smaller size:1.8 MB exists on the server. Replace it.                                                                             │
│2024-12-13 18:19:36 INF uploaded file=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426-EFFECTS.jpg                                                                                                               │
│2024-12-13 18:19:36 ERR error error=DeleteAsset, DELETE, http://127.0.0.1:2283/api/assets, 400 Bad Request                                                                                                                                            │
│Bad Request                                                                                                                                                                                                                                            
│2024-12-13 18:19:36 INF Stacked file=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426-EFFECTS-edited.jpg                                                                                                         │
│2024-12-13 18:19:36 INF Stacked file=takeout-20241212T173647Z-008:Takeout/Google Photos/Archive/eyeemfiltered1477073175426-EFFECTS.jpg   

After the "Bad Request", two more uploads/lines of text appeared in the log, it then froze and didn't resume. I'm using RC3 and I also used the same command:
immich-go.exe -s http:// -k upload from-google-photos takeout_1.zip

Let me know if you need any more information.

This has happened several times (at different stages of the upload/at different files).

EDIT: Just had another, slightly different Bad Request error which caused the upload to freeze and not resume:

│2024-12-13 20:10:51 INF Stacked file=takeout-20241212T173647Z-007:Takeout/Google Photos/Archive/IMG_20190721_194733-EFFECTS-edited.jpg                                                                                                                │
│2024-12-13 20:10:51 INF Stacked file=takeout-20241212T173647Z-007:Takeout/Google Photos/Archive/IMG_20190101_150800-EFFECTS-edited.jpg                                                                                                                │
│2024-12-13 20:10:51 INF Stacked file=takeout-20241212T173647Z-007:Takeout/Google Photos/Archive/IMG_20190717_211544-EFFECTS-edited.jpg                                                                                                                │
│2024-12-13 20:10:51 INF Stacked file=takeout-20241212T173647Z-007:Takeout/Google Photos/Archive/IMG_20191207_113037-EFFECTS-edited.jpg                                                                                                                │
│2024-12-13 20:10:51 INF Stacked file=takeout-20241212T173647Z-007:Takeout/Google Photos/Archive/IMG_20190421_113304-EFFECTS-edited.jpg                                                                                                                │
│2024-12-13 20:10:51 ERR Can't create stack error=createStack, POST, http://127.0.0.1:2283/api/stacks, 400 Bad Request                                                                                                                                 │
│Bad Request 

[simulot]

Thank you.
The error ERR error error=DeleteAsset, DELETE, http://127.0.0.1:2283/api/assets, 400 Bad Request is new to me

[cappuccino1985]

Hi there! I am also having this DeleteAsset error, the main issue for me is that, as mentioned in a post earlier, the upload stops after this error. Is there any workaround ?
Can I help identify the issue in any way ?

[simulot]

You can enable the --api-trace flag to get the api calls details

[cappuccino1985]

immich-go_2024-12-19_15-27-13.trace.log

Here is the trace file. I dont really know how helpful it will be, since i didnt find any Status 400 on the file.

I dunno if it helps, but here is the last DELETE request on the file

-- response body end --

2024-12-19T15:36:56-03:00 QUERY 14415 DeleteAsset DELETE http://192.168.0.114:2283/api/assets
Content-Type [application/json]
X-Api-Key redacted
-- request JSON Body --
{
"force": true,
"ids": [
"833b8d64-7d00-422e-a210-1abcc7b0b303"
]
}
-- request body end --

[cappuccino1985]

as a temporary measure that I used to continue my uploads (since it was stopping every single time at the same point) i used the parameter --ban-file, and I thought that the program probably checks for the string, and used the folder name as the pattern for the banning. It worked. so, technically you can ban a whole album (and other pictures that by any chance have the album name in its filename)

[simulot]

dirty...

immich-go_2024-12-19_15-27-13.trace.log

Here is the trace file. I dont really know how helpful it will be, since i didnt find any Status 400 on the file.

I dunno if it helps, but here is the last DELETE request on the file

-- response body end --

2024-12-19T15:36:56-03:00 QUERY 14415 DeleteAsset DELETE http://192.168.0.114:2283/api/assets Content-Type [application/json] X-Api-Key redacted -- request JSON Body -- { "force": true, "ids": [ "833b8d64-7d00-422e-a210-1abcc7b0b303" ] } -- request body end --

Could you provide the answer of this QUERY 14415?

[cappuccino1985]

its on the trace file that I uploaded on the last answer. I might actually have posted the wrong delete though

2024-12-19T17:25:37-03:00 RESPONSE 14415 AddAssetToAlbum PUT http://192.168.0.114:2283/api/albums/5c244add-0bd7-438b-b4af-57bba2433ecd/assets
Status: 200 OK
-- response body --
[
{
"id": "e0c80426-6562-4e67-8fc7-75aaa46b1803",
"success": false,
"error": "duplicate"
}
]
-- response body end --

Below I am sure that it is the last delete, because I copied everything from it to the end of the file

2024-12-19T17:25:37-03:00 QUERY 14429 DeleteAsset DELETE http://192.168.0.114:2283/api/assets
Content-Type [application/json]
X-Api-Key redacted
-- request JSON Body --
{
"force": true,
"ids": [
"5694812c-c5c3-464a-a2ad-fb47ddd3602e"
]
}
-- request body end --

2024-12-19T17:25:37-03:00 QUERY 14430 AddAssetToAlbum PUT http://192.168.0.114:2283/api/albums/8212dcb3-21ac-476c-aa56-55fc5b51c8f9/assets
Accept [application/json]
Content-Type [application/json]
X-Api-Key redacted
-- request JSON Body --
{
"ids": [
"f75a502d-3605-42f3-8aaa-c0708d533078",
"215523e5-ad52-476e-8f90-87ff7b83daac"
]
}
-- request body end --

2024-12-19T17:25:37-03:00 RESPONSE 14430 AddAssetToAlbum PUT http://192.168.0.114:2283/api/albums/8212dcb3-21ac-476c-aa56-55fc5b51c8f9/assets
Status: 200 OK
-- response body --
[
{
"id": "f75a502d-3605-42f3-8aaa-c0708d533078",
"success": false,
"error": "duplicate"
},
{
"id": "215523e5-ad52-476e-8f90-87ff7b83daac",
"success": false,
"error": "duplicate"
}
]
-- response body end --

2024-12-19T17:25:37-03:00 QUERY 14431 createStack POST http://192.168.0.114:2283/api/stacks
Accept [application/json]
Content-Type [application/json]
X-Api-Key redacted
-- request JSON Body --
{
"assetIds": [
"f75a502d-3605-42f3-8aaa-c0708d533078",
"215523e5-ad52-476e-8f90-87ff7b83daac"
]
}
-- request body end --

2024-12-19T17:25:37-03:00 RESPONSE 14431 createStack POST http://192.168.0.114:2283/api/stacks
Status: 201 Created
-- response body --
{
"id": "138ea71f-56a8-49f1-993d-9541255040d1",
"primaryAssetId": "f75a502d-3605-42f3-8aaa-c0708d533078"
}
-- response body end --

it appear as that last delete didnt have an answer to the request

[houruomu]

Is there a way to temporarily ban duplicate upgrade and instead skip all duplicate upload?