Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 383c3205-8faa-44d1-9f34-189b0a9490f6

Status: imported; action: update

Type

asset

Action

update

Status

imported

Items

63

Bytes

12.06 GB

Source host

orbit-2

Manifest sha256

aa0dc86c3244a1b2...

Started

2026-06-30 02:26:58

Completed

2026-06-30 02:36:49

Manifest summary

  • created_at: 2026-06-29T20:05:27Z
  • completed_at: 2026-06-29T20:05:27Z
  • verification.passed: true
  • verification.total_files: 12891
  • prior_sync_id: 4929b08d-0b05-48ac-a3f5-09c880ae8d76

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
52doneimport-feed2026-06-30 02:27:052026-06-30 02:33:16vultr/pid:379506063/632026-06-30 02:36:49-

Items 63

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video7a51e1b3290c7231...6750bc72a7274e05...createdasset:7456-2026-06-30 02:33:25
reel1c63978cea90a58d...9f293357a055477e...createdasset:7457-2026-06-30 02:33:29
reel92d94c1528d471cd...cce4c7c74b6f4040...createdasset:7458-2026-06-30 02:33:33
reel2bece27eb3b0194f...611a16bbf2ff42bf...createdasset:7459-2026-06-30 02:33:36
reele8f08f93c1c8764b...9a04324084d74843...createdasset:7460-2026-06-30 02:33:40
reel630c93e02c7c51ab...6c1d09a8de904c11...createdasset:7461-2026-06-30 02:33:43
reelba3b75243f217019...1347d3ffde9c422f...createdasset:7462-2026-06-30 02:33:47
reelac4fffe97686f732...6d0415779fe7456c...createdasset:7463-2026-06-30 02:33:51
reelbebba8965e8a93da...b11e53307b0046bc...createdasset:7464-2026-06-30 02:33:54
reel0b723ed7794beaf0...95d9bbdfdc1e4059...createdasset:7465-2026-06-30 02:33:58
reelc3011d10288b68e3...030480fcf69142d7...createdasset:7466-2026-06-30 02:34:01
reel159f33f48db02801...b2010ad3ff8441f1...createdasset:7467-2026-06-30 02:34:05
reel0802b96f440270da...3187eb96702c4221...createdasset:7468-2026-06-30 02:34:08
reela75fb66f26f1f7fa...-updatedasset:1120-2026-06-30 02:34:11
reel08e403af15873ad6...3cf087cd99344437...createdasset:7469-2026-06-30 02:34:15
reel596bc8df09a106d1...7171a75238c2480b...createdasset:7470-2026-06-30 02:34:20
reel1fc2a1882121e118...03abefa68ecb4c0c...createdasset:7471-2026-06-30 02:34:23
reel9d8fdbad23906347...85eabea1116249a8...createdasset:7472-2026-06-30 02:34:27
reel64d7a765d3c8f194...de11c8f6ea324f5b...createdasset:7473-2026-06-30 02:34:31
reel087ec78c5dffb0dd...792383850a0148a7...createdasset:7474-2026-06-30 02:34:34
reel7bb32d26f3791707...8b9ff1edab09498a...createdasset:7475-2026-06-30 02:34:38
reel587735249e3ccc7e...3be2fc839d774900...createdasset:7476-2026-06-30 02:34:42
reel9a355a1bf4436e0e...9a617d2b620d49de...createdasset:7477-2026-06-30 02:34:45
reelb6409aeb490d7f72...e004c7a7f4234202...createdasset:7478-2026-06-30 02:34:49
reel8ac7fe31f0d21c2f...da8fca8ef9d54eeb...createdasset:7479-2026-06-30 02:34:53
reel06328a3e6c0506e7...61564bd107d04b40...createdasset:7480-2026-06-30 02:34:56
reeld26a4cd3bcdb7db9...2a86439bcad9433f...createdasset:7481-2026-06-30 02:35:00
reel29896cef47310339...a1f53fb285bc4ade...createdasset:7482-2026-06-30 02:35:03
reel71291ab275e9c0c0...cd813122bfc94744...createdasset:7483-2026-06-30 02:35:07
reel3a5de797fa68b5cc...-updatedasset:1136-2026-06-30 02:35:10
reelbc5782fe8021f208...41119b825a914a5b...createdasset:7484-2026-06-30 02:35:14
reel9118022e58cdd802...-updatedasset:1138-2026-06-30 02:35:17
reelff5f923e75eefa54...-updatedasset:1139-2026-06-30 02:35:19
reel765653a590253501...-createdasset:7485-2026-06-30 02:35:23
reel01caf4b85a1bb1ef...-updatedasset:1141-2026-06-30 02:35:26
reela9db10436988a5f5...5772245528a24bc7...createdasset:7486-2026-06-30 02:35:30
reel1ff7a76d83d7fcfc...c47790cbb50d46f8...createdasset:7487-2026-06-30 02:35:34
reel0fd13c945191c7e4...b1d7e185e8b744ff...createdasset:7488-2026-06-30 02:35:37
reelce63fdc153c5eef2...a259fb5726c1404f...createdasset:7489-2026-06-30 02:35:41
reelbd0766f8ea480426...b5fe958f7a494014...createdasset:7490-2026-06-30 02:35:44
reel5374de7da72bf352...d9bb1f122ec0496d...createdasset:7491-2026-06-30 02:35:48
reel5c70cb155ea509d9...-updatedasset:1245-2026-06-30 02:35:51
reel339d2bb1f0b760eb...63e01de566c94257...createdasset:7492-2026-06-30 02:35:54
reel802cf3facb8dd651...89cddeb851da4d76...createdasset:7493-2026-06-30 02:35:58
reelb9686e1026a295ae...cf49dba2a4b94c51...createdasset:7494-2026-06-30 02:36:02
reel159cfce9a9d4bdee...7a0a6e5261b04e4c...createdasset:7495-2026-06-30 02:36:05
reel1c6d30b296fb0f1f...2f2b6fe7ff4342bf...createdasset:7496-2026-06-30 02:36:09
reel66eb67db2ce78834...d04e17dfba634cf1...createdasset:7497-2026-06-30 02:36:13
reel8b298fb1bb986944...66cfc2521f194808...createdasset:7498-2026-06-30 02:36:16
reel88ff125f155a46b4...-updatedasset:1251-2026-06-30 02:36:19
reel2e4e1ecc16ad9263...-updatedasset:1254-2026-06-30 02:36:23
reelaaf7bc038d6c159d...-updatedasset:1140-2026-06-30 02:36:26
photo9e37d89928971134...4a09fd098dc04be9...createdasset:7499-2026-06-30 02:36:28
photoa91913ccedbe0fba...713b5f03bac84469...createdasset:7500-2026-06-30 02:36:30
photo79fa1c3bb3e5db6e...68f378b2d25c48f3...createdasset:7501-2026-06-30 02:36:33
photo070d30ae548645b7...68da274237324c6b...createdasset:7502-2026-06-30 02:36:35
photo01bd6a899e8654af...8c3a217bdc694970...createdasset:7503-2026-06-30 02:36:37
photo6692c2840d467cf6...b8028ad8eaf4470f...createdasset:7504-2026-06-30 02:36:39
photoeafce2f1d7b34c08...9ee144bf21d8480d...createdasset:7505-2026-06-30 02:36:41
photo7a7dde69a2ed634d...5a5124a96e984973...createdasset:7506-2026-06-30 02:36:43
photo880de80af943a903...ee499bf814894e49...createdasset:7507-2026-06-30 02:36:45
photoe504fcd958115406...c44dae2fa405478a...createdasset:7508-2026-06-30 02:36:47
photo189598e801d0453f...11a61fcb154045c1...createdasset:7509-2026-06-30 02:36:49

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"52","phase":"job_finish","summary":{"items_seen":63,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":54,"deferred":0,"failed":0...2026-06-30 02:36:49
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":54,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":9}}2026-06-30 02:36:49
item_updated{"container":1,"expected_member_count":11,"gallery_hash":"114437b5bedc8889a83727b8bb076371","gallery_id":"39","linked_member_count":11,"missing_member_count":0,"phase":"container_link","title":"source...2026-06-30 02:36:49
item_created{"cover_hash":"11a61fcb154045c19871f615f3450bcd","error":null,"hash":"189598e801d0453f8b44b991ed31ec0a","phase":"item_import","target_ids":{"asset_id":"7509"},"type":"photo"}2026-06-30 02:36:49
item_created{"cover_hash":"c44dae2fa405478a8446bf918ec1fca2","error":null,"hash":"e504fcd9581154063ca042ad2bb0ca28","phase":"item_import","target_ids":{"asset_id":"7508"},"type":"photo"}2026-06-30 02:36:47
item_created{"cover_hash":"ee499bf814894e4995bd4264be43a262","error":null,"hash":"880de80af943a903f360c6bb307e2cfa","phase":"item_import","target_ids":{"asset_id":"7507"},"type":"photo"}2026-06-30 02:36:45
item_created{"cover_hash":"5a5124a96e984973a6e1b5e7a4c331b3","error":null,"hash":"7a7dde69a2ed634dccf50b01b8da2e78","phase":"item_import","target_ids":{"asset_id":"7506"},"type":"photo"}2026-06-30 02:36:43
item_created{"cover_hash":"9ee144bf21d8480da8d96af87db72820","error":null,"hash":"eafce2f1d7b34c089ba8862e2d246f41","phase":"item_import","target_ids":{"asset_id":"7505"},"type":"photo"}2026-06-30 02:36:41
item_created{"cover_hash":"b8028ad8eaf4470fac3e88d5663ace36","error":null,"hash":"6692c2840d467cf6c743ae97cf14381d","phase":"item_import","target_ids":{"asset_id":"7504"},"type":"photo"}2026-06-30 02:36:39
item_created{"cover_hash":"8c3a217bdc694970a6923de432a4ee8e","error":null,"hash":"01bd6a899e8654af117746810462e97d","phase":"item_import","target_ids":{"asset_id":"7503"},"type":"photo"}2026-06-30 02:36:37
item_created{"cover_hash":"68da274237324c6bbdde4872ea96a0d9","error":null,"hash":"070d30ae548645b7af68c3b0c47e0941","phase":"item_import","target_ids":{"asset_id":"7502"},"type":"photo"}2026-06-30 02:36:35
item_created{"cover_hash":"68f378b2d25c48f39a2e2ee8eefa686f","error":null,"hash":"79fa1c3bb3e5db6ed04423ffa7d759a7","phase":"item_import","target_ids":{"asset_id":"7501"},"type":"photo"}2026-06-30 02:36:33
item_created{"cover_hash":"713b5f03bac8446991ae0fee467655a0","error":null,"hash":"a91913ccedbe0fba521e85a7185cac44","phase":"item_import","target_ids":{"asset_id":"7500"},"type":"photo"}2026-06-30 02:36:30
item_created{"cover_hash":"4a09fd098dc04be9b4c13acd6959032c","error":null,"hash":"9e37d8992897113421e46d5309864b4d","phase":"item_import","target_ids":{"asset_id":"7499"},"type":"photo"}2026-06-30 02:36:28
item_updated{"cover_hash":null,"error":null,"hash":"aaf7bc038d6c159d1bd9c88449791df8","phase":"item_import","target_ids":{"asset_id":"1140"},"type":"reel"}2026-06-30 02:36:26
item_updated{"cover_hash":null,"error":null,"hash":"2e4e1ecc16ad9263ddbfd50578badb3d","phase":"item_import","target_ids":{"asset_id":"1254"},"type":"reel"}2026-06-30 02:36:23
item_updated{"cover_hash":null,"error":null,"hash":"88ff125f155a46b445dd65c1f34a3319","phase":"item_import","target_ids":{"asset_id":"1251"},"type":"reel"}2026-06-30 02:36:19
item_created{"cover_hash":"66cfc2521f194808947c18f0e05c7cfa","error":null,"hash":"8b298fb1bb986944b78b8ca0e20ebd97","phase":"item_import","target_ids":{"asset_id":"7498"},"type":"reel"}2026-06-30 02:36:16
item_created{"cover_hash":"d04e17dfba634cf1a7e747a724a941a8","error":null,"hash":"66eb67db2ce78834d07a72ba548cf479","phase":"item_import","target_ids":{"asset_id":"7497"},"type":"reel"}2026-06-30 02:36:13
item_created{"cover_hash":"2f2b6fe7ff4342bf8c0b6d457b69791c","error":null,"hash":"1c6d30b296fb0f1f6f9916519ce84306","phase":"item_import","target_ids":{"asset_id":"7496"},"type":"reel"}2026-06-30 02:36:09
item_created{"cover_hash":"7a0a6e5261b04e4cbeff8fd1c2aeda29","error":null,"hash":"159cfce9a9d4bdee1d7e8170b27da894","phase":"item_import","target_ids":{"asset_id":"7495"},"type":"reel"}2026-06-30 02:36:05
item_created{"cover_hash":"cf49dba2a4b94c51b7537da1024f7c0a","error":null,"hash":"b9686e1026a295ae5da8546a794bb12b","phase":"item_import","target_ids":{"asset_id":"7494"},"type":"reel"}2026-06-30 02:36:02
item_created{"cover_hash":"89cddeb851da4d768e7ffdd7a1273dc6","error":null,"hash":"802cf3facb8dd651b41215a3a36d6ccf","phase":"item_import","target_ids":{"asset_id":"7493"},"type":"reel"}2026-06-30 02:35:58
item_created{"cover_hash":"63e01de566c9425797da2b879a405cf0","error":null,"hash":"339d2bb1f0b760eb1e073359c19b880d","phase":"item_import","target_ids":{"asset_id":"7492"},"type":"reel"}2026-06-30 02:35:54
item_updated{"cover_hash":null,"error":null,"hash":"5c70cb155ea509d9a3030dc0403a54a4","phase":"item_import","target_ids":{"asset_id":"1245"},"type":"reel"}2026-06-30 02:35:51
item_created{"cover_hash":"d9bb1f122ec0496db8ec438d40118c89","error":null,"hash":"5374de7da72bf352b059bbb2bd8abf93","phase":"item_import","target_ids":{"asset_id":"7491"},"type":"reel"}2026-06-30 02:35:48
item_created{"cover_hash":"b5fe958f7a494014ab0d1210f0b6c30a","error":null,"hash":"bd0766f8ea4804266a123a96a28cafb7","phase":"item_import","target_ids":{"asset_id":"7490"},"type":"reel"}2026-06-30 02:35:44
item_created{"cover_hash":"a259fb5726c1404f84726b7ea4e54291","error":null,"hash":"ce63fdc153c5eef27bb66c5b18db65f8","phase":"item_import","target_ids":{"asset_id":"7489"},"type":"reel"}2026-06-30 02:35:41
item_created{"cover_hash":"b1d7e185e8b744ffb6b7b226515f6d1e","error":null,"hash":"0fd13c945191c7e41b7d6eccdcf8929c","phase":"item_import","target_ids":{"asset_id":"7488"},"type":"reel"}2026-06-30 02:35:37
item_created{"cover_hash":"c47790cbb50d46f8be47b625e6bf4815","error":null,"hash":"1ff7a76d83d7fcfce16ebbf2f7a2a49c","phase":"item_import","target_ids":{"asset_id":"7487"},"type":"reel"}2026-06-30 02:35:34
item_created{"cover_hash":"5772245528a24bc79cfc4aba05a400dc","error":null,"hash":"a9db10436988a5f5e322b49b8f716b9c","phase":"item_import","target_ids":{"asset_id":"7486"},"type":"reel"}2026-06-30 02:35:30
item_updated{"cover_hash":null,"error":null,"hash":"01caf4b85a1bb1ef2ba74ec6964fc148","phase":"item_import","target_ids":{"asset_id":"1141"},"type":"reel"}2026-06-30 02:35:26
item_created{"cover_hash":null,"error":null,"hash":"765653a590253501b6be1105b529691b","phase":"item_import","target_ids":{"asset_id":"7485"},"type":"reel"}2026-06-30 02:35:23
item_updated{"cover_hash":null,"error":null,"hash":"ff5f923e75eefa544e4141144c3a8c2d","phase":"item_import","target_ids":{"asset_id":"1139"},"type":"reel"}2026-06-30 02:35:19
item_updated{"cover_hash":null,"error":null,"hash":"9118022e58cdd802def2f17427d17646","phase":"item_import","target_ids":{"asset_id":"1138"},"type":"reel"}2026-06-30 02:35:17
item_created{"cover_hash":"41119b825a914a5ba54295133eb5568b","error":null,"hash":"bc5782fe8021f208d604ba9a8b3f67e8","phase":"item_import","target_ids":{"asset_id":"7484"},"type":"reel"}2026-06-30 02:35:14
item_updated{"cover_hash":null,"error":null,"hash":"3a5de797fa68b5ccc3188e622ed0b5f7","phase":"item_import","target_ids":{"asset_id":"1136"},"type":"reel"}2026-06-30 02:35:10
item_created{"cover_hash":"cd813122bfc94744bdbdb48c4c00c24b","error":null,"hash":"71291ab275e9c0c0cdd111d1a14b0199","phase":"item_import","target_ids":{"asset_id":"7483"},"type":"reel"}2026-06-30 02:35:07
item_created{"cover_hash":"a1f53fb285bc4adeae642abd0487a2bd","error":null,"hash":"29896cef47310339f783f3662f87dd31","phase":"item_import","target_ids":{"asset_id":"7482"},"type":"reel"}2026-06-30 02:35:03
item_created{"cover_hash":"2a86439bcad9433fa3034f2afad37bbb","error":null,"hash":"d26a4cd3bcdb7db9ed8c46fba5cc64cf","phase":"item_import","target_ids":{"asset_id":"7481"},"type":"reel"}2026-06-30 02:35:00
item_created{"cover_hash":"61564bd107d04b409d8eeddc868b396e","error":null,"hash":"06328a3e6c0506e775e316e9922f7170","phase":"item_import","target_ids":{"asset_id":"7480"},"type":"reel"}2026-06-30 02:34:56
item_created{"cover_hash":"da8fca8ef9d54eebbd2d5f610db665d0","error":null,"hash":"8ac7fe31f0d21c2f28f8076c212688ff","phase":"item_import","target_ids":{"asset_id":"7479"},"type":"reel"}2026-06-30 02:34:53
item_created{"cover_hash":"e004c7a7f4234202a0c1c12ba3a8ca6e","error":null,"hash":"b6409aeb490d7f7273bc3d7e2f8cb5b7","phase":"item_import","target_ids":{"asset_id":"7478"},"type":"reel"}2026-06-30 02:34:49
item_created{"cover_hash":"9a617d2b620d49debf0a04f6057e4d7c","error":null,"hash":"9a355a1bf4436e0e28ef5fe412c3549e","phase":"item_import","target_ids":{"asset_id":"7477"},"type":"reel"}2026-06-30 02:34:45
item_created{"cover_hash":"3be2fc839d774900bb487f0bc5cba7ae","error":null,"hash":"587735249e3ccc7e7e742f90fa415e97","phase":"item_import","target_ids":{"asset_id":"7476"},"type":"reel"}2026-06-30 02:34:42
item_created{"cover_hash":"8b9ff1edab09498aafe01e751153905b","error":null,"hash":"7bb32d26f3791707ff17a99dd9ff738b","phase":"item_import","target_ids":{"asset_id":"7475"},"type":"reel"}2026-06-30 02:34:38
item_created{"cover_hash":"792383850a0148a79cea460290b5062c","error":null,"hash":"087ec78c5dffb0dde5739ebd6b9c04e1","phase":"item_import","target_ids":{"asset_id":"7474"},"type":"reel"}2026-06-30 02:34:34
item_created{"cover_hash":"de11c8f6ea324f5b80731a422e01704b","error":null,"hash":"64d7a765d3c8f194d39c506557666119","phase":"item_import","target_ids":{"asset_id":"7473"},"type":"reel"}2026-06-30 02:34:31
item_created{"cover_hash":"85eabea1116249a889a8421a95f66206","error":null,"hash":"9d8fdbad23906347c800b55d27d86a58","phase":"item_import","target_ids":{"asset_id":"7472"},"type":"reel"}2026-06-30 02:34:27
item_created{"cover_hash":"03abefa68ecb4c0c8d6909ddc819a682","error":null,"hash":"1fc2a1882121e118fef7a57dd2d76a73","phase":"item_import","target_ids":{"asset_id":"7471"},"type":"reel"}2026-06-30 02:34:23