Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 28bdae8d-e365-434d-8929-7137d7d059d9

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

54

Bytes

3.22 GB

Source host

orbit-2

Manifest sha256

ab836b808b8ba04c...

Started

2026-06-30 02:26:37

Completed

2026-06-30 02:29:14

Manifest summary

  • created_at: 2026-06-29T21:26:41Z
  • completed_at: 2026-06-29T21:26:41Z
  • verification.passed: true
  • verification.total_files: 5973
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
50doneimport-feed2026-06-30 02:26:432026-06-30 02:26:48vultr/pid:379506054/542026-06-30 02:29:14-

Items 54

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video1afed109d3c69cd0...f3e99d2a95ba4145...createdasset:7281-2026-06-30 02:26:55
reel076f30aa0d78f2c5...d11162061e6f4e10...createdasset:7282-2026-06-30 02:26:58
reel85b828a9a5a3945f...-createdasset:7283-2026-06-30 02:27:00
reelce82d2fcdfa1b541...e63cdd2f4e334272...createdasset:7284-2026-06-30 02:27:02
reeld93a73c9d9c3f9b7...-createdasset:7285-2026-06-30 02:27:04
reelc0e90b9d1f68ecc9...79b78912f44b4e0a...createdasset:7286-2026-06-30 02:27:06
reel812078e3d6d7eacc...-createdasset:7287-2026-06-30 02:27:08
reel1203980f0f82801a...85b5033d5f154a17...createdasset:7288-2026-06-30 02:27:11
reel14eaef158744e9a5...-createdasset:7289-2026-06-30 02:27:13
reel97a10473a4f5e103...bb5d37ba257d42c9...createdasset:7290-2026-06-30 02:27:15
reela5d781fd9bebdb29...-createdasset:7291-2026-06-30 02:27:17
reel81e7964b3b11b0bb...aa379450fee44ebd...createdasset:7292-2026-06-30 02:27:19
reeladf5d841cf368e76...-createdasset:7293-2026-06-30 02:27:21
reelb3a42049ba51cd92...4519c571628348f8...createdasset:7294-2026-06-30 02:27:24
reel0484c3e213f50e64...-createdasset:7295-2026-06-30 02:27:26
reelc43b8bee2b5c9f2a...15d964ccb6cd44b7...createdasset:7296-2026-06-30 02:27:29
reel0a7bdc3006481095...-createdasset:7297-2026-06-30 02:27:31
reeldf0460e18cadd0a9...34697217813d4019...createdasset:7298-2026-06-30 02:27:33
reelcdb73d7cbc820928...-createdasset:7299-2026-06-30 02:27:35
reel1ba4ebcfbcdf9fe4...07b8481eafd54fd0...createdasset:7300-2026-06-30 02:27:38
reel5894041cac461700...-createdasset:7301-2026-06-30 02:27:39
reelfcaa927b518e1a52...6ec20d6d64ac4e08...createdasset:7302-2026-06-30 02:27:42
reel20615df8693a8300...a458fef4ad434a73...createdasset:7303-2026-06-30 02:27:44
reel931d14f6e8396341...134df6b172e44a9c...createdasset:7304-2026-06-30 02:27:47
reel4238979edf4f6830...b44ba6cdba8243a5...createdasset:7305-2026-06-30 02:27:54
reel813e9773f7811406...f90fc78964ce4667...createdasset:7306-2026-06-30 02:28:11
reeld3beb4fc16a94055...414325a642044942...createdasset:7307-2026-06-30 02:28:14
reel0114a101489bff61...63d8e5bcf9ab4ae3...createdasset:7308-2026-06-30 02:28:16
reel1fa08e9dda4821f1...dc33f0b651164e7f...createdasset:7309-2026-06-30 02:28:19
reel2b78e35d0442720a...84d1659ed1b840f7...createdasset:7310-2026-06-30 02:28:21
photo2a9767c6c3ff7deb...e4485c32b7e34a44...createdasset:7311-2026-06-30 02:28:23
photod41bdce8f46f02fe...ca6adae51af64067...createdasset:7312-2026-06-30 02:28:27
photoe612c13695937c01...1eecd2422bab4bfe...createdasset:7313-2026-06-30 02:28:29
photo4e114f7d374e505c...b2355e62ac3f48b9...createdasset:7314-2026-06-30 02:28:31
photo2cc2a1909aaab735...62d89d070010461e...createdasset:7315-2026-06-30 02:28:33
photo12bac72fca4ca671...26b412a35ba94e65...createdasset:7316-2026-06-30 02:28:35
photobe189a77df99411a...1ffcc0facadf4ab4...createdasset:7317-2026-06-30 02:28:37
photo5f88a64e86593f84...49b19f569f3246dc...createdasset:7318-2026-06-30 02:28:39
photo85f0f7337d666ea3...8922e1f83f8245b5...createdasset:7319-2026-06-30 02:28:41
photo6e1c9d2399dac98f...9f9c897a07794757...createdasset:7320-2026-06-30 02:28:43
photocfc298ef45b15762...3b1105c9a9fb4d89...createdasset:7321-2026-06-30 02:28:45
photo498ccf05fe873ec2...6eaebda5b4e647e9...createdasset:7322-2026-06-30 02:28:48
photo871fce8885f05028...cd7e5643abdd4027...createdasset:7323-2026-06-30 02:28:50
photo232502fef10874fd...1851023e3a374132...createdasset:7324-2026-06-30 02:28:52
photo8686b20d0f9f24ce...96a7b32670354227...createdasset:7325-2026-06-30 02:28:54
photod99a94362202b88b...0a421f767f404c79...createdasset:7326-2026-06-30 02:28:56
photo5066947bd70c6c70...26dae2c3586e4c59...createdasset:7327-2026-06-30 02:28:58
photo2c6f0163bf4be53f...9ef1570527364de8...createdasset:7328-2026-06-30 02:29:00
photo841b3166182692d2...1155b8bba9734736...createdasset:7329-2026-06-30 02:29:02
photo5c1fc080f2f44618...967a1cc457544e37...createdasset:7330-2026-06-30 02:29:04
photo3605192fbf4ed7c4...29bcd121fcdf405c...createdasset:7331-2026-06-30 02:29:06
photofe0166f26b514949...0c7cd523c1594688...createdasset:7332-2026-06-30 02:29:09
photo40518d9a83db3253...7eed57f4261d4082...createdasset:7333-2026-06-30 02:29:11
photoc94e4f23f8e9f34a...9defeb7d23fb4412...createdasset:7334-2026-06-30 02:29:14

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"50","phase":"job_finish","summary":{"items_seen":54,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":54,"deferred":0,"failed":0...2026-06-30 02:29:14
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":54,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-30 02:29:14
item_updated{"container":1,"expected_member_count":24,"gallery_hash":"93a09c5953157babce737a64c736b773","gallery_id":"37","linked_member_count":24,"missing_member_count":0,"phase":"container_link","title":"Contes...2026-06-30 02:29:14
item_created{"cover_hash":"9defeb7d23fb4412abbf650465db67f7","error":null,"hash":"c94e4f23f8e9f34af730c5bc1e16a5fb","phase":"item_import","target_ids":{"asset_id":"7334"},"type":"photo"}2026-06-30 02:29:14
item_created{"cover_hash":"7eed57f4261d40829057a34342f43563","error":null,"hash":"40518d9a83db3253f2862e94c87e52c2","phase":"item_import","target_ids":{"asset_id":"7333"},"type":"photo"}2026-06-30 02:29:11
item_created{"cover_hash":"0c7cd523c159468881895fe51d7a7abd","error":null,"hash":"fe0166f26b514949f141bb27af7d7043","phase":"item_import","target_ids":{"asset_id":"7332"},"type":"photo"}2026-06-30 02:29:09
item_created{"cover_hash":"29bcd121fcdf405c8cb91fb6965ba76b","error":null,"hash":"3605192fbf4ed7c4a8ddf3162b9c59d4","phase":"item_import","target_ids":{"asset_id":"7331"},"type":"photo"}2026-06-30 02:29:06
item_created{"cover_hash":"967a1cc457544e37a2019d00c81939e1","error":null,"hash":"5c1fc080f2f44618c6ae3f39328b87d4","phase":"item_import","target_ids":{"asset_id":"7330"},"type":"photo"}2026-06-30 02:29:04
item_created{"cover_hash":"1155b8bba97347369c1ac84003dd2d2c","error":null,"hash":"841b3166182692d2f0239f5cf39ca9ce","phase":"item_import","target_ids":{"asset_id":"7329"},"type":"photo"}2026-06-30 02:29:02
item_created{"cover_hash":"9ef1570527364de8812bf993f7eb9bfe","error":null,"hash":"2c6f0163bf4be53feab7520c95d4769a","phase":"item_import","target_ids":{"asset_id":"7328"},"type":"photo"}2026-06-30 02:29:00
item_created{"cover_hash":"26dae2c3586e4c59b40e93a707384fb0","error":null,"hash":"5066947bd70c6c703043eeaacd110427","phase":"item_import","target_ids":{"asset_id":"7327"},"type":"photo"}2026-06-30 02:28:58
item_created{"cover_hash":"0a421f767f404c79a28f174ae0590d3a","error":null,"hash":"d99a94362202b88b8807a024f286c4d2","phase":"item_import","target_ids":{"asset_id":"7326"},"type":"photo"}2026-06-30 02:28:56
item_created{"cover_hash":"96a7b32670354227b50595bd286a2435","error":null,"hash":"8686b20d0f9f24ce53c8cedc4887353b","phase":"item_import","target_ids":{"asset_id":"7325"},"type":"photo"}2026-06-30 02:28:54
item_created{"cover_hash":"1851023e3a37413296b1ec4c791d7a6f","error":null,"hash":"232502fef10874fdc9f632df8c26f3d4","phase":"item_import","target_ids":{"asset_id":"7324"},"type":"photo"}2026-06-30 02:28:52
item_created{"cover_hash":"cd7e5643abdd40279978f3a43882b766","error":null,"hash":"871fce8885f050285eaeb1ef6492e474","phase":"item_import","target_ids":{"asset_id":"7323"},"type":"photo"}2026-06-30 02:28:50
item_created{"cover_hash":"6eaebda5b4e647e9b7b7d3bdff90446a","error":null,"hash":"498ccf05fe873ec272b3ce3d27ff90c6","phase":"item_import","target_ids":{"asset_id":"7322"},"type":"photo"}2026-06-30 02:28:48
item_created{"cover_hash":"3b1105c9a9fb4d89a5e4619b01cdf3c5","error":null,"hash":"cfc298ef45b157623e47786ae5b16013","phase":"item_import","target_ids":{"asset_id":"7321"},"type":"photo"}2026-06-30 02:28:45
item_created{"cover_hash":"9f9c897a0779475793557f4a1adf02ca","error":null,"hash":"6e1c9d2399dac98f414624af546e9c56","phase":"item_import","target_ids":{"asset_id":"7320"},"type":"photo"}2026-06-30 02:28:43
item_created{"cover_hash":"8922e1f83f8245b59b1784beb16bed89","error":null,"hash":"85f0f7337d666ea361d8451e36ec4a74","phase":"item_import","target_ids":{"asset_id":"7319"},"type":"photo"}2026-06-30 02:28:41
item_created{"cover_hash":"49b19f569f3246dc976e99f338ea8d73","error":null,"hash":"5f88a64e86593f84efe4abe79840da12","phase":"item_import","target_ids":{"asset_id":"7318"},"type":"photo"}2026-06-30 02:28:39
item_created{"cover_hash":"1ffcc0facadf4ab493590d5bd46c3063","error":null,"hash":"be189a77df99411afead380d2eb74bdf","phase":"item_import","target_ids":{"asset_id":"7317"},"type":"photo"}2026-06-30 02:28:37
item_created{"cover_hash":"26b412a35ba94e6582198748c98f0e97","error":null,"hash":"12bac72fca4ca6711e4c5341a805571c","phase":"item_import","target_ids":{"asset_id":"7316"},"type":"photo"}2026-06-30 02:28:35
item_created{"cover_hash":"62d89d070010461eba7a3d0ee2eda8d1","error":null,"hash":"2cc2a1909aaab735f7c067ce4e8b50a5","phase":"item_import","target_ids":{"asset_id":"7315"},"type":"photo"}2026-06-30 02:28:33
item_created{"cover_hash":"b2355e62ac3f48b995b91f6494e19484","error":null,"hash":"4e114f7d374e505c4f99ce228acda16c","phase":"item_import","target_ids":{"asset_id":"7314"},"type":"photo"}2026-06-30 02:28:31
item_created{"cover_hash":"1eecd2422bab4bfe846e3e3a5f9787bc","error":null,"hash":"e612c13695937c01896ea7d649ca583e","phase":"item_import","target_ids":{"asset_id":"7313"},"type":"photo"}2026-06-30 02:28:29
item_created{"cover_hash":"ca6adae51af64067a162acd6adfe0f88","error":null,"hash":"d41bdce8f46f02fecd28bb39830f5a13","phase":"item_import","target_ids":{"asset_id":"7312"},"type":"photo"}2026-06-30 02:28:27
item_created{"cover_hash":"e4485c32b7e34a44bfd82e22ea7aac85","error":null,"hash":"2a9767c6c3ff7deb38c539622f3689af","phase":"item_import","target_ids":{"asset_id":"7311"},"type":"photo"}2026-06-30 02:28:23
item_created{"cover_hash":"84d1659ed1b840f7a0b44d7ff76c2952","error":null,"hash":"2b78e35d0442720a1c9a4fd3807c6d49","phase":"item_import","target_ids":{"asset_id":"7310"},"type":"reel"}2026-06-30 02:28:21
item_created{"cover_hash":"dc33f0b651164e7f9ed95c1649434001","error":null,"hash":"1fa08e9dda4821f19bca4285b6a4edb1","phase":"item_import","target_ids":{"asset_id":"7309"},"type":"reel"}2026-06-30 02:28:19
item_created{"cover_hash":"63d8e5bcf9ab4ae38fba49c876d3822f","error":null,"hash":"0114a101489bff6105b4373fae1ed347","phase":"item_import","target_ids":{"asset_id":"7308"},"type":"reel"}2026-06-30 02:28:16
item_created{"cover_hash":"414325a642044942a4c3a7261e54862c","error":null,"hash":"d3beb4fc16a94055171c001178c4b35a","phase":"item_import","target_ids":{"asset_id":"7307"},"type":"reel"}2026-06-30 02:28:14
item_created{"cover_hash":"f90fc78964ce4667bb52bc7c8f4c4cc0","error":null,"hash":"813e9773f78114061cedf38e4db21e88","phase":"item_import","target_ids":{"asset_id":"7306"},"type":"reel"}2026-06-30 02:28:11
item_created{"cover_hash":"b44ba6cdba8243a58b5b9aed1cb2ae4d","error":null,"hash":"4238979edf4f6830e928698f786c1932","phase":"item_import","target_ids":{"asset_id":"7305"},"type":"reel"}2026-06-30 02:27:54
item_created{"cover_hash":"134df6b172e44a9ca6d2e3b9650a9d34","error":null,"hash":"931d14f6e8396341ce9c0d011d9e2c1f","phase":"item_import","target_ids":{"asset_id":"7304"},"type":"reel"}2026-06-30 02:27:47
item_created{"cover_hash":"a458fef4ad434a738c2154bcadf061ca","error":null,"hash":"20615df8693a8300ca3785dddb930b24","phase":"item_import","target_ids":{"asset_id":"7303"},"type":"reel"}2026-06-30 02:27:44
item_created{"cover_hash":"6ec20d6d64ac4e08a74196c196f155ae","error":null,"hash":"fcaa927b518e1a527645dd22b6599737","phase":"item_import","target_ids":{"asset_id":"7302"},"type":"reel"}2026-06-30 02:27:42
item_created{"cover_hash":null,"error":null,"hash":"5894041cac4617009177d2179996f14c","phase":"item_import","target_ids":{"asset_id":"7301"},"type":"reel"}2026-06-30 02:27:39
item_created{"cover_hash":"07b8481eafd54fd0bab85803a8c48eb6","error":null,"hash":"1ba4ebcfbcdf9fe4a11df693ccde9d76","phase":"item_import","target_ids":{"asset_id":"7300"},"type":"reel"}2026-06-30 02:27:38
item_created{"cover_hash":null,"error":null,"hash":"cdb73d7cbc820928ad34f3059c61b6d7","phase":"item_import","target_ids":{"asset_id":"7299"},"type":"reel"}2026-06-30 02:27:35
item_created{"cover_hash":"34697217813d4019a65eb7f8b1117308","error":null,"hash":"df0460e18cadd0a91d82594eed4de7d7","phase":"item_import","target_ids":{"asset_id":"7298"},"type":"reel"}2026-06-30 02:27:33
item_created{"cover_hash":null,"error":null,"hash":"0a7bdc300648109501481767c3e78dd8","phase":"item_import","target_ids":{"asset_id":"7297"},"type":"reel"}2026-06-30 02:27:31
item_created{"cover_hash":"15d964ccb6cd44b78ea18fcdd90a7880","error":null,"hash":"c43b8bee2b5c9f2ad180498d2d87062f","phase":"item_import","target_ids":{"asset_id":"7296"},"type":"reel"}2026-06-30 02:27:29
item_created{"cover_hash":null,"error":null,"hash":"0484c3e213f50e645704bf55010fc24d","phase":"item_import","target_ids":{"asset_id":"7295"},"type":"reel"}2026-06-30 02:27:26
item_created{"cover_hash":"4519c571628348f8bce7dde23b8d705f","error":null,"hash":"b3a42049ba51cd9226ba0eae0ac99fd2","phase":"item_import","target_ids":{"asset_id":"7294"},"type":"reel"}2026-06-30 02:27:24
item_created{"cover_hash":null,"error":null,"hash":"adf5d841cf368e7649f4109d2e72f9fb","phase":"item_import","target_ids":{"asset_id":"7293"},"type":"reel"}2026-06-30 02:27:21
item_created{"cover_hash":"aa379450fee44ebd956912937b8fce82","error":null,"hash":"81e7964b3b11b0bb0967004590672dda","phase":"item_import","target_ids":{"asset_id":"7292"},"type":"reel"}2026-06-30 02:27:19
item_created{"cover_hash":null,"error":null,"hash":"a5d781fd9bebdb293d3e6b2d82863114","phase":"item_import","target_ids":{"asset_id":"7291"},"type":"reel"}2026-06-30 02:27:17
item_created{"cover_hash":"bb5d37ba257d42c99e3b722e34d92ff3","error":null,"hash":"97a10473a4f5e10346b0699ec914a1ee","phase":"item_import","target_ids":{"asset_id":"7290"},"type":"reel"}2026-06-30 02:27:15
item_created{"cover_hash":null,"error":null,"hash":"14eaef158744e9a52375b2741c5c7e92","phase":"item_import","target_ids":{"asset_id":"7289"},"type":"reel"}2026-06-30 02:27:13
item_created{"cover_hash":"85b5033d5f154a1789732d3a21bc58ab","error":null,"hash":"1203980f0f82801aaa135f1785828418","phase":"item_import","target_ids":{"asset_id":"7288"},"type":"reel"}2026-06-30 02:27:11