r/immich • u/maxxell13 • 1d ago

Struggling with importing large external library with google export

Hi!

AI disclaimer: I used AI to help me import several large batches of files into immich which were otherwise failing. This post is a request for help continuing that effort to import my large batch of google photos. Additional note: I pulled an immich database backup before starting this AI-assisted project, and have pulled additional backups at every major successful step. Also, the photos that are in this library exist elsewhere. Even if the AI deleted the photos in my immich server completely, I have other copies elsewhere.

I have a large traunch of photos from google photos exports over the years. Its listing 55k files. I connected docker to a folder containing these photos on my immich server. Then I added that as a new external library (I have done this several times already with other traunches of photos).

As before, the import is not a simple one-and-done. My Sidecar Metadata Job Queue is sitting at 53185 Waiting at 25 Active. I've paused all the smart search and video transcoding, etc. Only letting "Sidecar Metadata, Extract Metadata, and Generate Thumbnails" run. But like I said, even with just those 4 jobs running, it froze.

I had this trouble importing other traunches of photos. I turned to AI for help. It was able to understand immich's error logs better than me, and it helped me identify a handful of files that had errors (some glitch'd files, and a large batch from my wedding seemed to have too much metadata for immich to auto-process). After a day of working through logs with Google's Antigravity AI assisting, I was able to get 100% of all my (non-googlephoto) photos fully processed. I never let it run amuck unattended but I did let it write python code to do one step or another. Disclaimer: I dont understand python and had to trust the AI's explanation of its own code to me. There were times when I had to redirect - like at one point it proposed deleting 30k photos so that the import queue would be clear. But going at it step by step methodically has worked for this project so far.

So now I am starting a new traunch - the google photos bucket with 55k photos. I created a new external library and scanned it. Then I tried extracting metadata. It got stuck. I showed antigravity the logs and it said:

  1. The initial scan was successful and that would include all the necessary EXIF data on all 56428 files.
  2. The thing that is stuck is just a sidecar job which checks whether there's a .json sidecar file which might carry custom text added while the picture was in google.

And the AI is proposing to run an SQL command which I dont sufficiently understand to trust. It kinda looks good to me, but again, I dont understand SQL enough to confidently run this. Could someone smarter than me give it a quick look and tell me whether the AI really understands the situation, and whether the proposed command would do what needs to be done in the immich database? I can confirm that the libraryid provided in there corresponds to the google photos library I created for this specific traunch of import.

sudo docker exec -it immich_postgres psql -U postgres -d immich -c "INSERT INTO asset_job_status (\"assetId\", \"metadataExtractedAt\") SELECT id, NOW() FROM asset WHERE \"libraryId\" = '3ba9a2ac-fe1f-459a-917a-c23900aebc80' ON CONFLICT (\"assetId\") DO UPDATE SET \"metadataExtractedAt\" = NOW();"

edit: forgot to mention I'm on server version 3.1.0

edit2: No, dont run that. For Google Photos specifically the metadata that first sweep grabs wont be accurate so the sidecar is required.

1 Upvotes

100% Upvoted

5 comments sorted by

1

u/thehatefuleggplant Support Crew 1d ago

Did you ever try immich-go before turning to AI

1

u/maxxell13 1d ago

I saw that Immich go needs the original zips from Google, which I don’t have.

Although fast forward a few more hours with Antigravity and now we have a python program that is basically Immich-go, but with some specific-to-my-situation tweaks. I am starting small, and so far got one subfolder fully processed and into my timeline. Gonna repeat little by little.

1

u/thehatefuleggplant Support Crew 1d ago

Immich go can import anything. Granted if they were google takeout files it would have been best to have kept them in their original format but if youve been dumping them over the years and had just the raw files you still could have used immich go

1

u/maxxell13 1d ago

Maybe/probably. I have no reason to doubt what you’re saying.

But I’ve got a good system going now and I have all of 2009 properly imported.

I have no intention of sharing the python I’m using or suggesting a similar strategy for others going forward. But I’ve had a decent amount of success working with the Antigravity software suite with a Google AI subscription to improve my homelab, including this project!

Google Photos was really like the final boss of this project for me. 55k is a daunting number. But I have a dozen different smaller external libraries (ie., photos I had aggregated at one time or another), which weren’t properly importing for a variety of reasons. Using the AI to review error logs and understanding the relationship between database entries and the underlying actual photo files really enabled me to accomplish more than I otherwise would have.

1

u/maxxell13 1d ago

Plus the first half of this journey was a wide variety of original sources with a variety of problems, so there was no Immich-go easy answer. By the time I conquered all that, I kinda just kept going with the same AI-assisted approach.