Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 4f779eba-810c-4f8d-a552-2749fe9f1750

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

45

Bytes

4.66 GB

Source host

orbit-2

Manifest sha256

f33399b0e2ffe474...

Started

2026-07-02 16:19:29

Completed

2026-07-02 16:39:25

Manifest summary

  • created_at: 2026-07-02T16:17:35Z
  • completed_at: 2026-07-02T16:17:35Z
  • verification.passed: true
  • verification.total_files: 9025
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
75doneimport-feed2026-07-02 16:19:502026-07-02 16:19:51vultr/pid:379506045/452026-07-02 16:39:25-

Items 45

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
videoca296f239c7c08fe...36b9bf216d7b4a04...createdasset:10330-2026-07-02 16:20:41
reelbef23d1ff5ce22f7...a076257c08e94a92...createdasset:10331-2026-07-02 16:21:11
reel7114d6df52edbcba...7ffd175c0eca4c3c...createdasset:10332-2026-07-02 16:21:32
reel645fbfe3802ed281...997a2fbe135445b6...createdasset:10333-2026-07-02 16:22:10
reel7e1655d2c304aa13...bdd16fe1ac1e4340...createdasset:10334-2026-07-02 16:22:41
reel843501686222fdc3...da2fdfc52d534429...createdasset:10335-2026-07-02 16:23:29
reelaef2f1b0411e5f98...cc63e2f429e347cc...createdasset:10336-2026-07-02 16:23:48
reel2ee11b45846bd55a...2d4681419e2d4157...createdasset:10337-2026-07-02 16:24:33
reeldced655a72d1d8f2...81458f8ef98a46e2...createdasset:10338-2026-07-02 16:25:19
reela0c4aebd2c90ad18...4f5ca5628bf640b2...createdasset:10339-2026-07-02 16:25:44
reel3e5ffcdf1aed6183...8f97efc603e04535...createdasset:10340-2026-07-02 16:26:04
reel2696b40480bbfba1...c6358492f79d4978...createdasset:10341-2026-07-02 16:26:29
reel72aaa9cd8804e0de...db72a2633d4747e6...createdasset:10342-2026-07-02 16:27:15
reeleebbaa819e37a67d...cf05b96efb5a4337...createdasset:10343-2026-07-02 16:27:37
reelc94b080b3f71c99c...3ef42edb6fd04558...createdasset:10344-2026-07-02 16:27:56
reel2cecff7b78e82177...0509497a07d6499f...createdasset:10345-2026-07-02 16:28:29
reel743ac113e1375b05...45970195e5464191...createdasset:10346-2026-07-02 16:28:48
reel75fe975cfdbf9f5c...f499786af0e74e47...createdasset:10347-2026-07-02 16:29:22
reelba6d415f9d2c7109...3aba719b981b4d49...createdasset:10348-2026-07-02 16:29:41
reelea0e03aa153b3361...c1edeed1d49e4d3c...createdasset:10349-2026-07-02 16:30:49
reelf4f8b34b7776f4e5...917b321a28854bc5...createdasset:10350-2026-07-02 16:31:36
photo3dd2fe4c9dd1b186...0635eb37884742de...createdasset:10351-2026-07-02 16:31:48
photo76e138829c1ddb55...8bcad6767f6942d7...createdasset:10352-2026-07-02 16:32:22
photoa2876408f6e42515...4c989f1be0514573...createdasset:10353-2026-07-02 16:32:32
photoa99924e61790404a...b766ae50933c4023...createdasset:10354-2026-07-02 16:32:48
photo0ab349c438b295b1...5e1d6c200e7f41bc...createdasset:10355-2026-07-02 16:33:21
photod69afaabc5462c2c...2e38d9ed40164361...createdasset:10356-2026-07-02 16:33:31
photoc7a206910203fd04...9d8e6ef291ad46f0...createdasset:10357-2026-07-02 16:34:07
photo09e60df79122bd07...242df48f0c4a41ca...createdasset:10358-2026-07-02 16:34:21
photob59a9e9eb950ac9d...ee3260796e714c5b...createdasset:10359-2026-07-02 16:34:31
photo7015bbf3488c0f67...4db85847f1f347ee...createdasset:10360-2026-07-02 16:34:40
photoa2db39c3b44709da...1e7e1dcbe9a34c47...createdasset:10361-2026-07-02 16:34:56
photoc3713eaa7dd52259...79cecf3db839440e...createdasset:10362-2026-07-02 16:35:21
photod1c7ad6a47a03268...2c6a353d9c5e45a0...createdasset:10363-2026-07-02 16:35:41
photoa125ff3ca6c63b48...8d839177b83a49f2...createdasset:10364-2026-07-02 16:35:50
photo93f3d19336ba5248...58418acf48f44604...createdasset:10365-2026-07-02 16:35:58
photod28894e05a1a8ab0...d500874b0ea646b1...createdasset:10366-2026-07-02 16:36:10
photo957cbf5b8708dcb1...4b85b68a46384521...createdasset:10367-2026-07-02 16:36:37
photo2a0f391beca7c709...3af474fea4f641f7...createdasset:10368-2026-07-02 16:36:53
photoeeb76d0297ac7f3f...29a3d51c1b4342a9...createdasset:10369-2026-07-02 16:37:19
photo7459cf588396fb76...e89648df47ad4433...createdasset:10370-2026-07-02 16:37:29
photo4fa49d2e4e450fbb...e84cead3bfc74d4e...createdasset:10371-2026-07-02 16:38:08
photo54a6c8201d36363a...2aa1cf0ae1b74724...createdasset:10372-2026-07-02 16:38:24
photo745fc310959e7500...5098153f32da4ace...createdasset:10373-2026-07-02 16:39:14
photo94379da6926f7c53...ae10f39488f74795...createdasset:10374-2026-07-02 16:39:25

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"75","phase":"job_finish","summary":{"items_seen":45,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":45,"deferred":0,"failed":0...2026-07-02 16:39:25
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":45,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-07-02 16:39:25
item_updated{"container":1,"expected_member_count":24,"gallery_hash":"d08ece2c647898abdba602083e067746","gallery_id":"61","linked_member_count":24,"missing_member_count":0,"phase":"container_link","title":"countr...2026-07-02 16:39:25
item_created{"cover_hash":"ae10f39488f74795bd185964974084c8","error":null,"hash":"94379da6926f7c5385eac8707e4acbb3","phase":"item_import","target_ids":{"asset_id":"10374"},"type":"photo"}2026-07-02 16:39:25
item_created{"cover_hash":"5098153f32da4acebf4f60c034e4f61c","error":null,"hash":"745fc310959e7500cf8d29611f70903a","phase":"item_import","target_ids":{"asset_id":"10373"},"type":"photo"}2026-07-02 16:39:14
item_created{"cover_hash":"2aa1cf0ae1b74724aac05d2d8fab3053","error":null,"hash":"54a6c8201d36363abe853359f41eab34","phase":"item_import","target_ids":{"asset_id":"10372"},"type":"photo"}2026-07-02 16:38:24
item_created{"cover_hash":"e84cead3bfc74d4e9b5c430ab2c1c585","error":null,"hash":"4fa49d2e4e450fbb6db47bb19a96a8c1","phase":"item_import","target_ids":{"asset_id":"10371"},"type":"photo"}2026-07-02 16:38:08
item_created{"cover_hash":"e89648df47ad4433bda6758456f8c0da","error":null,"hash":"7459cf588396fb766a18ce2aaf115623","phase":"item_import","target_ids":{"asset_id":"10370"},"type":"photo"}2026-07-02 16:37:29
item_created{"cover_hash":"29a3d51c1b4342a9b551ce24d4b5f10b","error":null,"hash":"eeb76d0297ac7f3f9ef3aa11bdbdbce6","phase":"item_import","target_ids":{"asset_id":"10369"},"type":"photo"}2026-07-02 16:37:19
item_created{"cover_hash":"3af474fea4f641f78edc56ce47511ce9","error":null,"hash":"2a0f391beca7c709ea4419c2a504d5b7","phase":"item_import","target_ids":{"asset_id":"10368"},"type":"photo"}2026-07-02 16:36:53
item_created{"cover_hash":"4b85b68a4638452192396afb5ab2f17a","error":null,"hash":"957cbf5b8708dcb11de8e1af9207ba9c","phase":"item_import","target_ids":{"asset_id":"10367"},"type":"photo"}2026-07-02 16:36:37
item_created{"cover_hash":"d500874b0ea646b1b101e31bf34588bc","error":null,"hash":"d28894e05a1a8ab04569bcfdf19c0240","phase":"item_import","target_ids":{"asset_id":"10366"},"type":"photo"}2026-07-02 16:36:10
item_created{"cover_hash":"58418acf48f4460491ae094f58887fd7","error":null,"hash":"93f3d19336ba524893df0081c7c06bad","phase":"item_import","target_ids":{"asset_id":"10365"},"type":"photo"}2026-07-02 16:35:58
item_created{"cover_hash":"8d839177b83a49f2a5533666ae4083ba","error":null,"hash":"a125ff3ca6c63b48aded34594e6fb6f2","phase":"item_import","target_ids":{"asset_id":"10364"},"type":"photo"}2026-07-02 16:35:50
item_created{"cover_hash":"2c6a353d9c5e45a0bf4f9a763c0a1381","error":null,"hash":"d1c7ad6a47a0326869440d31202d6aa8","phase":"item_import","target_ids":{"asset_id":"10363"},"type":"photo"}2026-07-02 16:35:41
item_created{"cover_hash":"79cecf3db839440ebf87a7c7ac84ee9d","error":null,"hash":"c3713eaa7dd522596d75cb618c27d526","phase":"item_import","target_ids":{"asset_id":"10362"},"type":"photo"}2026-07-02 16:35:21
item_created{"cover_hash":"1e7e1dcbe9a34c4787e5c307fc2c39b2","error":null,"hash":"a2db39c3b44709da2306c27c3914929b","phase":"item_import","target_ids":{"asset_id":"10361"},"type":"photo"}2026-07-02 16:34:56
item_created{"cover_hash":"4db85847f1f347ee896116d909998637","error":null,"hash":"7015bbf3488c0f67c5edf0eac33e1219","phase":"item_import","target_ids":{"asset_id":"10360"},"type":"photo"}2026-07-02 16:34:40
item_created{"cover_hash":"ee3260796e714c5b9baba74e6767c255","error":null,"hash":"b59a9e9eb950ac9d0f82401c14bf8ddd","phase":"item_import","target_ids":{"asset_id":"10359"},"type":"photo"}2026-07-02 16:34:31
item_created{"cover_hash":"242df48f0c4a41cab09550d598870ac8","error":null,"hash":"09e60df79122bd070f48ed3d076777a2","phase":"item_import","target_ids":{"asset_id":"10358"},"type":"photo"}2026-07-02 16:34:21
item_created{"cover_hash":"9d8e6ef291ad46f09cb54ed1a4a9c489","error":null,"hash":"c7a206910203fd0423858eb75dfe076f","phase":"item_import","target_ids":{"asset_id":"10357"},"type":"photo"}2026-07-02 16:34:07
item_created{"cover_hash":"2e38d9ed40164361944c97c9447120df","error":null,"hash":"d69afaabc5462c2c4df5e6484507259c","phase":"item_import","target_ids":{"asset_id":"10356"},"type":"photo"}2026-07-02 16:33:31
item_created{"cover_hash":"5e1d6c200e7f41bc9cb66b1712dde318","error":null,"hash":"0ab349c438b295b18a34ef9975f4a0f8","phase":"item_import","target_ids":{"asset_id":"10355"},"type":"photo"}2026-07-02 16:33:21
item_created{"cover_hash":"b766ae50933c4023931aa09ead195322","error":null,"hash":"a99924e61790404aaf85f3b3232f56b4","phase":"item_import","target_ids":{"asset_id":"10354"},"type":"photo"}2026-07-02 16:32:48
item_created{"cover_hash":"4c989f1be0514573b97459fa4c78c179","error":null,"hash":"a2876408f6e42515b7bbee3e48e5a05a","phase":"item_import","target_ids":{"asset_id":"10353"},"type":"photo"}2026-07-02 16:32:32
item_created{"cover_hash":"8bcad6767f6942d79e0cd2765e5c471f","error":null,"hash":"76e138829c1ddb5519d058a75227cd84","phase":"item_import","target_ids":{"asset_id":"10352"},"type":"photo"}2026-07-02 16:32:22
item_created{"cover_hash":"0635eb37884742deb2ec6fa1f0e21f0d","error":null,"hash":"3dd2fe4c9dd1b18639a18c16ff235c3c","phase":"item_import","target_ids":{"asset_id":"10351"},"type":"photo"}2026-07-02 16:31:48
item_created{"cover_hash":"917b321a28854bc5a9220b8368e4e14b","error":null,"hash":"f4f8b34b7776f4e5b58201923778adbd","phase":"item_import","target_ids":{"asset_id":"10350"},"type":"reel"}2026-07-02 16:31:36
item_created{"cover_hash":"c1edeed1d49e4d3cbacfaf63e6680f58","error":null,"hash":"ea0e03aa153b33617d78132a95168a98","phase":"item_import","target_ids":{"asset_id":"10349"},"type":"reel"}2026-07-02 16:30:49
item_created{"cover_hash":"3aba719b981b4d49b4cbc65456bd116f","error":null,"hash":"ba6d415f9d2c71098ce7bc54a874f3c5","phase":"item_import","target_ids":{"asset_id":"10348"},"type":"reel"}2026-07-02 16:29:41
item_created{"cover_hash":"f499786af0e74e479c81b5fe6406a8e8","error":null,"hash":"75fe975cfdbf9f5c72c5c00ea055bf4d","phase":"item_import","target_ids":{"asset_id":"10347"},"type":"reel"}2026-07-02 16:29:22
item_created{"cover_hash":"45970195e546419193b04c6e663524ad","error":null,"hash":"743ac113e1375b05c773bed9d54f2730","phase":"item_import","target_ids":{"asset_id":"10346"},"type":"reel"}2026-07-02 16:28:48
item_created{"cover_hash":"0509497a07d6499f8effc98389a1c907","error":null,"hash":"2cecff7b78e821779d0c200bd608e9df","phase":"item_import","target_ids":{"asset_id":"10345"},"type":"reel"}2026-07-02 16:28:29
item_created{"cover_hash":"3ef42edb6fd0455893c1f54a574fe9c1","error":null,"hash":"c94b080b3f71c99c9ea25e3f4001c4ad","phase":"item_import","target_ids":{"asset_id":"10344"},"type":"reel"}2026-07-02 16:27:56
item_created{"cover_hash":"cf05b96efb5a433780805997e02ab6ce","error":null,"hash":"eebbaa819e37a67d70919d791caf1e79","phase":"item_import","target_ids":{"asset_id":"10343"},"type":"reel"}2026-07-02 16:27:37
item_created{"cover_hash":"db72a2633d4747e68f139d5819d58fcb","error":null,"hash":"72aaa9cd8804e0de2100d476d3ddaf7a","phase":"item_import","target_ids":{"asset_id":"10342"},"type":"reel"}2026-07-02 16:27:15
item_created{"cover_hash":"c6358492f79d4978bdb399d99074515a","error":null,"hash":"2696b40480bbfba1d38d08c6f198c60b","phase":"item_import","target_ids":{"asset_id":"10341"},"type":"reel"}2026-07-02 16:26:29
item_created{"cover_hash":"8f97efc603e04535881456d39d7253a5","error":null,"hash":"3e5ffcdf1aed618313a6734c9fb9e05d","phase":"item_import","target_ids":{"asset_id":"10340"},"type":"reel"}2026-07-02 16:26:04
item_created{"cover_hash":"4f5ca5628bf640b2815967a3ed3f4653","error":null,"hash":"a0c4aebd2c90ad18147c837f3afb2835","phase":"item_import","target_ids":{"asset_id":"10339"},"type":"reel"}2026-07-02 16:25:44
item_created{"cover_hash":"81458f8ef98a46e2a01c27730ec84350","error":null,"hash":"dced655a72d1d8f2d117b592be0c630b","phase":"item_import","target_ids":{"asset_id":"10338"},"type":"reel"}2026-07-02 16:25:19
item_created{"cover_hash":"2d4681419e2d41578aa311eae797eed2","error":null,"hash":"2ee11b45846bd55a0e96318ed3eaba81","phase":"item_import","target_ids":{"asset_id":"10337"},"type":"reel"}2026-07-02 16:24:33
item_created{"cover_hash":"cc63e2f429e347cc9217bb84c25e69ee","error":null,"hash":"aef2f1b0411e5f9829d4aff1de3161f2","phase":"item_import","target_ids":{"asset_id":"10336"},"type":"reel"}2026-07-02 16:23:48
item_created{"cover_hash":"da2fdfc52d534429950ed30ff3450076","error":null,"hash":"843501686222fdc39f2842666fef13b2","phase":"item_import","target_ids":{"asset_id":"10335"},"type":"reel"}2026-07-02 16:23:29
item_created{"cover_hash":"bdd16fe1ac1e4340afed5cc331f41ecd","error":null,"hash":"7e1655d2c304aa13f9f88b76bab032e1","phase":"item_import","target_ids":{"asset_id":"10334"},"type":"reel"}2026-07-02 16:22:41
item_created{"cover_hash":"997a2fbe135445b6a9e2929eaabb749c","error":null,"hash":"645fbfe3802ed2817fed1bf78432f488","phase":"item_import","target_ids":{"asset_id":"10333"},"type":"reel"}2026-07-02 16:22:10
item_created{"cover_hash":"7ffd175c0eca4c3c9ee5edeb68c68acc","error":null,"hash":"7114d6df52edbcbaae0a0796d8273eab","phase":"item_import","target_ids":{"asset_id":"10332"},"type":"reel"}2026-07-02 16:21:32
item_created{"cover_hash":"a076257c08e94a92924863f52a9656ea","error":null,"hash":"bef23d1ff5ce22f7b4a35fd5219edb49","phase":"item_import","target_ids":{"asset_id":"10331"},"type":"reel"}2026-07-02 16:21:11
item_created{"cover_hash":"36b9bf216d7b4a04a1e5ec5c11f00bc9","error":null,"hash":"ca296f239c7c08fe6ebe9b379351e9a1","phase":"item_import","target_ids":{"asset_id":"10330"},"type":"video"}2026-07-02 16:20:41
import_started{"action":"import","already_terminal_count":0,"container_count":2,"item_count":45,"manifest_sha256":"f33399b0e2ffe474c1f291497ed07e72ec6fa53d01cab9c08787d8c8a3e4602a","max_items":0,"phase":"import"}2026-07-02 16:19:54
job_claimed{"job_id":"75","phase":"job_claim","priority":100,"worker_id":"vultr/pid:3795060"}2026-07-02 16:19:51