23.0rc6 freeze on upload error 400 bad request

[4goodluck]

Hitting title error - updates are to a stacked image (possibly from a prior immich-go run). Tried unstacking the image in immich and restarting immich.

command line -
./immich-go upload from-google-photos --log-level=DEBUG --log-file=run5.txt --server=http://10.100.99.37:2283 --api-key=redact --include-unmatched --include-untitled-albums --manage-burst=stack --api-trace=TRUE --manage-heic-jpeg=StackCoverHeic --manage-raw-jpeg=StackCoverJPG /home/mik/Downloads/takeout-*.zip

Log:
2025-01-05 14:31:20 ERR Can't create stack error=createStack, POST, http://10.100.99.37:2283/api/stacks, 400 Bad Request
Bad Request

2025-01-05 14:31:20 ERR upload error file=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438.JPG error=ReplaceAsset, PUT, http://10.100.99.37:2283/api/assets/acd7f801-ebad-4aaf-9ea5-245ea25a924d/original, 400 Bad Request
Bad Request

2025-01-05 14:31:20 DBG file.FileName=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438.JPG file.FileDate=2025-01-04 18:14:22 file.Description= file.Title=IMG_1438.JPG file.FileSize=1068384 file.ID= file.CaptureDate=2011-03-11 14:42:52 file.Trashed=false file.Archived=false file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0.xxxxx file.Longitude=0.xxxxx
2025-01-05 14:31:20 ERR upload error file=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438(1).JPG error=ReplaceAsset, PUT, http://10.100.99.37:2283/api/assets/acd7f801-ebad-4aaf-9ea5-245ea25a924d/original, 400 Bad Request
Bad Request

2025-01-05 14:31:20 DBG file.FileName=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438(1).JPG file.FileDate=2025-01-04 18:14:22 file.Description= file.Title=IMG_1438.JPG file.FileSize=1068384 file.ID= file.CaptureDate=2011-03-11 14:42:52 file.Trashed=false file.Archived=false file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0.xxxxx file.Longitude=0.xxxxx
2025-01-05 14:31:20 ERR upload error file=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438(2).JPG error=ReplaceAsset, PUT, http://10.100.99.37:2283/api/assets/acd7f801-ebad-4aaf-9ea5-245ea25a924d/original, 400 Bad Request
Bad Request

2025-01-05 14:31:20 DBG file.FileName=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438(2).JPG file.FileDate=2025-01-04 18:14:23 file.Description= file.Title=IMG_1438.JPG file.FileSize=1068384 file.ID= file.CaptureDate=2011-03-11 14:42:52 file.Trashed=false file.Archived=false file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0.xxxxx file.Longitude=0.xxxxx
2025-01-05 14:31:20 INF Stacked file=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438.JPG
2025-01-05 14:31:20 INF Stacked file=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438(1).JPG
2025-01-05 14:31:20 INF Stacked file=takeout-20250104T181400Z-001:Takeout/Google Photos/Drop Box/IMG_1438(2).JPG

Api:
2025-01-05T14:31:20-05:00 QUERY 250 createStack POST http://10.100.99.37:2283/api/stacks
X-Api-Key redacted
Accept [application/json]
Content-Type [application/json]
-- request JSON Body --
{
"assetIds": [
"0a278f64-04ca-44b3-9543-3bb01f5dc27e",
"f5883fad-22fc-4c51-975c-e24832cb3961"
]
}
-- request body end --

2025-01-05T14:31:20-05:00 QUERY 251 ReplaceAsset PUT http://10.100.99.37:2283/api/assets/acd7f801-ebad-4aaf-9ea5-245ea25a924d/original
Accept [application/json]
Content-Type [multipart/form-data; boundary=268d1c4520fa783bed5edbc35128a7add5fc544dae3524e59e473a8a034e]
X-Api-Key redacted
asset file name: Takeout/Google Photos/Drop Box/IMG_1438.JPG
-- Empty body or binary body not dumped --
2025-01-05T14:31:20-05:00 QUERY 252 ReplaceAsset PUT http://10.100.99.37:2283/api/assets/acd7f801-ebad-4aaf-9ea5-245ea25a924d/original
Accept [application/json]
Content-Type [multipart/form-data; boundary=773bdaeb0f7ec3da89dc3da79fe208d9286130d4d20dbefff19b4ea00232]
X-Api-Key redacted
asset file name: Takeout/Google Photos/Drop Box/IMG_1438(1).JPG
-- Empty body or binary body not dumped --
2025-01-05T14:31:20-05:00 QUERY 253 ReplaceAsset PUT http://10.100.99.37:2283/api/assets/acd7f801-ebad-4aaf-9ea5-245ea25a924d/original
X-Api-Key redacted
Accept [application/json]
Content-Type [multipart/form-data; boundary=34484d4c85982efc3c741de28e7eb9c961df185d73919d5f027ea4ad9a08]
asset file name: Takeout/Google Photos/Drop Box/IMG_1438(2).JPG
-- Empty body or binary body not dumped --

[simulot]

This one is nasty....
How big is your takeout?

[4goodluck]

It's a single zip about 47g. This happens very early in the up load, about 100 files in. Immich version is current. Input analysis: --------------- scanned image file : 18241 scanned video file : 5241 scanned sidecar file : 19002 discarded file : 6 unsupported file : 6 file duplicated in the input : 0 associated metadata file : 23476 missing associated metadata file : 0 Uploading: ---------- uploaded : 0 upload error : 3 file not selected : 0 server's asset upgraded with the input : 0 server has same asset : 55 server has a better asset : 12

[simulot]

Could you enable the flag --API-TRACE ? and share the log file and the trace file on my discord @simulot ?

[4goodluck]

Let me know if you need anything else.
run6.trace.log.gz
run6.txt.gz

[4goodluck]

I seem to be able to produce bad request errors at will even with RC7. I suspect that any attempt to change files in an external immich library produce this error because every case I can find involves external library files. The immich external libraries are r/w in .env. It's possible I have some other miss configuration on the immich side.

I haven't been able to find associated log events in immich (server, db or redis). I'm also having difficult understanding exactly what immich will and will not do with an external r/w library so this may be my misunderstanding of how immich works.

[simulot]

I seem to be able to produce bad request errors at will even with RC7. I suspect that any attempt to change files in an external immich library produce this error because every case I can find involves external library files. The immich external libraries are r/w in .env. It's possible I have some other miss configuration on the immich side.

That's an important information. I'm building a test system using an external library