Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch e4ba2197-ddfc-4cf4-9660-a65bc000059b

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

58

Bytes

3.34 GB

Source host

orbit-2

Manifest sha256

d5df866c993065cd...

Started

2026-06-30 02:28:27

Completed

2026-06-30 03:00:19

Manifest summary

  • created_at: 2026-06-29T21:48:57Z
  • completed_at: 2026-06-29T21:48:57Z
  • verification.passed: true
  • verification.total_files: 5875
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
56doneimport-feed2026-06-30 02:28:342026-06-30 02:58:07vultr/pid:379506058/582026-06-30 03:00:19-

Items 58

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video931b6496f19224f0...38e15b350dac4043...createdasset:7904-2026-06-30 02:58:15
reel2450e6f456ef428e...835c8b8b550943f4...createdasset:7905-2026-06-30 02:58:18
reel8b1488ae081ad03f...-createdasset:7906-2026-06-30 02:58:19
reel8c18a37507586327...54b379e3f0654691...createdasset:7907-2026-06-30 02:58:22
reel138b7d8d8fe53489...-createdasset:7908-2026-06-30 02:58:24
reel2c9852adb5c3bbea...5c3e3f990776451b...createdasset:7909-2026-06-30 02:58:26
reelc710d9661ab21d74...-createdasset:7910-2026-06-30 02:58:28
reel0459fc5784958dcf...202e4feae73b45f9...createdasset:7911-2026-06-30 02:58:30
reel63e857776c8c316c...-createdasset:7912-2026-06-30 02:58:32
reelb6f4e16c963a4fc8...ca3c91f7902342b3...createdasset:7913-2026-06-30 02:58:35
reel778519b9960a36d1...-createdasset:7914-2026-06-30 02:58:36
reel936cf1327d26b9f5...fc264ae111c34d02...createdasset:7915-2026-06-30 02:58:39
reele2c1a55eabdc3625...-createdasset:7916-2026-06-30 02:58:41
reela37a134d8592a56f...98cee4d50a264ff4...createdasset:7917-2026-06-30 02:58:44
reel1dfc290b2c7e2acf...-createdasset:7918-2026-06-30 02:58:45
reel9c55ff407eb08a17...ea081038e6464262...createdasset:7919-2026-06-30 02:58:48
reel2196206750eaa1e9...-createdasset:7920-2026-06-30 02:58:50
reele763b35d69744a89...c6228280a450429e...createdasset:7921-2026-06-30 02:58:52
reel49d168ecbeeb66af...-createdasset:7922-2026-06-30 02:58:54
reel9ef78834765abf21...a6b8df556ab6459c...createdasset:7923-2026-06-30 02:58:56
reel8cea61ea979b38d4...-createdasset:7924-2026-06-30 02:58:58
reel3557d274433f9799...0c56522f0db04ba8...createdasset:7925-2026-06-30 02:59:01
reelee6769d1bcfc5750...-createdasset:7926-2026-06-30 02:59:03
reel593826c22db55679...79b41bf25b5b4072...createdasset:7927-2026-06-30 02:59:05
reelc9183775a59be8f0...ae9d287281fb45e9...createdasset:7928-2026-06-30 02:59:08
reel866cec93348b00b4...577c7ecca47545d4...createdasset:7929-2026-06-30 02:59:10
reel162b25c94f7cc460...020f439b989643ba...createdasset:7930-2026-06-30 02:59:13
reel2e71d8f3c4feffc2...bda712a729224c3d...createdasset:7931-2026-06-30 02:59:16
reel63a3561f631dc84c...4a094fff8c724275...createdasset:7932-2026-06-30 02:59:18
reel16f33e474c1a8b78...4bfa19401c3946fc...createdasset:7933-2026-06-30 02:59:21
reela8e2f6bcaee92ffb...193d831dc2d744d6...createdasset:7934-2026-06-30 02:59:23
reel63f7d754b2988218...b53590d83b3747e9...createdasset:7935-2026-06-30 02:59:26
reel01da9eec8145af33...8d84a644941f404b...createdasset:7936-2026-06-30 02:59:28
reel524f043becd22f54...0b56b7e844c84ec0...createdasset:7937-2026-06-30 02:59:31
photofb7e81b9680d7e48...99529c6b492a462b...createdasset:7938-2026-06-30 02:59:33
photo595397ce715bdec5...3973643b2a0143fd...createdasset:7939-2026-06-30 02:59:35
photof4b5be311b459978...c05ad3dec0064c53...createdasset:7940-2026-06-30 02:59:37
photo9c3ef9691582689e...68ff1b98adf34466...createdasset:7941-2026-06-30 02:59:39
photo2a68977aa46f6ec3...7fd1114c13b8443e...createdasset:7942-2026-06-30 02:59:41
photo18657609197023f6...badca1ae96724054...createdasset:7943-2026-06-30 02:59:43
photocd42ded3e2cbf96a...9178867efebd4b86...createdasset:7944-2026-06-30 02:59:45
photo93d8516707a0f25f...18e01e4721394904...createdasset:7945-2026-06-30 02:59:47
photoaa6a0cb6f3169e27...92e6e551537347d4...createdasset:7946-2026-06-30 02:59:49
photoaddbeb200c5098a8...fa832d796bb64fe4...createdasset:7947-2026-06-30 02:59:51
photo580d5b7266747691...7263f193f1244d81...createdasset:7948-2026-06-30 02:59:53
photo2404706667f00f03...57f4a181b05448ac...createdasset:7949-2026-06-30 02:59:55
photo05b3468e294bc254...971ec95a7caa44d4...createdasset:7950-2026-06-30 02:59:57
photo465780def52be85a...7e2d106b86d34f12...createdasset:7951-2026-06-30 02:59:59
photobff24ad45358ce3c...578f65dff1f648b4...createdasset:7952-2026-06-30 03:00:01
photo36e684c4633efc15...aed5e80f53514cc0...createdasset:7953-2026-06-30 03:00:03
photo12e33e8057fff0fd...95226c6c0b474714...createdasset:7954-2026-06-30 03:00:05
photo8a1c0ce22c4b96e8...b87c9e1cba63473e...createdasset:7955-2026-06-30 03:00:07
photo11cd182e15592715...f7e4171ee6f448dc...createdasset:7956-2026-06-30 03:00:09
photo617bb42bf59f0d15...42b98991c3e0411c...createdasset:7957-2026-06-30 03:00:11
photo0f13ca98124fcdb0...c9cc2246aab647da...createdasset:7958-2026-06-30 03:00:13
photo34b44cbe099f458f...3ceb1b4bd15f42a8...createdasset:7959-2026-06-30 03:00:15
photoe26fa044bf301086...4a75db9581e8444c...createdasset:7960-2026-06-30 03:00:17
photo9a56a1690a0d28a5...aae3674226a14f35...createdasset:7961-2026-06-30 03:00:19

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"56","phase":"job_finish","summary":{"items_seen":58,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":58,"deferred":0,"failed":0...2026-06-30 03:00:19
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":58,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-30 03:00:19
item_updated{"container":1,"expected_member_count":24,"gallery_hash":"6f63426c8502371eb460badc6d17cd21","gallery_id":"42","linked_member_count":24,"missing_member_count":0,"phase":"container_link","title":"Contes...2026-06-30 03:00:19
item_created{"cover_hash":"aae3674226a14f35b766729dbf9ada4c","error":null,"hash":"9a56a1690a0d28a5df01b626cde71d4c","phase":"item_import","target_ids":{"asset_id":"7961"},"type":"photo"}2026-06-30 03:00:19
item_created{"cover_hash":"4a75db9581e8444c81cb3db9bbada10d","error":null,"hash":"e26fa044bf3010861ce7aceee8e36113","phase":"item_import","target_ids":{"asset_id":"7960"},"type":"photo"}2026-06-30 03:00:17
item_created{"cover_hash":"3ceb1b4bd15f42a8bbf6c7d242710549","error":null,"hash":"34b44cbe099f458f92a29a2467961317","phase":"item_import","target_ids":{"asset_id":"7959"},"type":"photo"}2026-06-30 03:00:15
item_created{"cover_hash":"c9cc2246aab647dab7cf1af1883114ec","error":null,"hash":"0f13ca98124fcdb0a8334a1e69a08f1a","phase":"item_import","target_ids":{"asset_id":"7958"},"type":"photo"}2026-06-30 03:00:13
item_created{"cover_hash":"42b98991c3e0411c89f8814fbeb7ad43","error":null,"hash":"617bb42bf59f0d15e65314a20df6cc00","phase":"item_import","target_ids":{"asset_id":"7957"},"type":"photo"}2026-06-30 03:00:11
item_created{"cover_hash":"f7e4171ee6f448dcba8a516bb287aa6c","error":null,"hash":"11cd182e1559271596ffddf746ceec21","phase":"item_import","target_ids":{"asset_id":"7956"},"type":"photo"}2026-06-30 03:00:09
item_created{"cover_hash":"b87c9e1cba63473e8c4c403baa058e55","error":null,"hash":"8a1c0ce22c4b96e8bc08961506245215","phase":"item_import","target_ids":{"asset_id":"7955"},"type":"photo"}2026-06-30 03:00:07
item_created{"cover_hash":"95226c6c0b47471486705859cac6caaa","error":null,"hash":"12e33e8057fff0fdb1450bfc5eb89d9d","phase":"item_import","target_ids":{"asset_id":"7954"},"type":"photo"}2026-06-30 03:00:05
item_created{"cover_hash":"aed5e80f53514cc08f2fae600e0f76dc","error":null,"hash":"36e684c4633efc155f4a3100a35aae8a","phase":"item_import","target_ids":{"asset_id":"7953"},"type":"photo"}2026-06-30 03:00:03
item_created{"cover_hash":"578f65dff1f648b4b9b8e0cae61656a0","error":null,"hash":"bff24ad45358ce3c99231682ef86082f","phase":"item_import","target_ids":{"asset_id":"7952"},"type":"photo"}2026-06-30 03:00:01
item_created{"cover_hash":"7e2d106b86d34f128e10428f9800f830","error":null,"hash":"465780def52be85af2677f468afa22a6","phase":"item_import","target_ids":{"asset_id":"7951"},"type":"photo"}2026-06-30 02:59:59
item_created{"cover_hash":"971ec95a7caa44d48166f5857a1d0c88","error":null,"hash":"05b3468e294bc25461e05d32b655e06d","phase":"item_import","target_ids":{"asset_id":"7950"},"type":"photo"}2026-06-30 02:59:57
item_created{"cover_hash":"57f4a181b05448acafcb81ee9b80124e","error":null,"hash":"2404706667f00f03f8e02cb3e514ce0b","phase":"item_import","target_ids":{"asset_id":"7949"},"type":"photo"}2026-06-30 02:59:55
item_created{"cover_hash":"7263f193f1244d81b720c966f3392b9d","error":null,"hash":"580d5b72667476916fd37bd110b393ae","phase":"item_import","target_ids":{"asset_id":"7948"},"type":"photo"}2026-06-30 02:59:53
item_created{"cover_hash":"fa832d796bb64fe48b587fb8d8dce782","error":null,"hash":"addbeb200c5098a80b76589ef4738362","phase":"item_import","target_ids":{"asset_id":"7947"},"type":"photo"}2026-06-30 02:59:51
item_created{"cover_hash":"92e6e551537347d4b6763765c7954326","error":null,"hash":"aa6a0cb6f3169e270c09834e4bf6d37d","phase":"item_import","target_ids":{"asset_id":"7946"},"type":"photo"}2026-06-30 02:59:49
item_created{"cover_hash":"18e01e4721394904ab2fd43db8c3c91c","error":null,"hash":"93d8516707a0f25f5a40fa74750c4548","phase":"item_import","target_ids":{"asset_id":"7945"},"type":"photo"}2026-06-30 02:59:47
item_created{"cover_hash":"9178867efebd4b868cc8ea4872ab3e5b","error":null,"hash":"cd42ded3e2cbf96ad8f0f17c5bf8a4ec","phase":"item_import","target_ids":{"asset_id":"7944"},"type":"photo"}2026-06-30 02:59:45
item_created{"cover_hash":"badca1ae96724054b8ab7c04f199b048","error":null,"hash":"18657609197023f67b3031e7323f6d36","phase":"item_import","target_ids":{"asset_id":"7943"},"type":"photo"}2026-06-30 02:59:43
item_created{"cover_hash":"7fd1114c13b8443ea614fefed6c0fc97","error":null,"hash":"2a68977aa46f6ec3a66fc06a39fd0256","phase":"item_import","target_ids":{"asset_id":"7942"},"type":"photo"}2026-06-30 02:59:41
item_created{"cover_hash":"68ff1b98adf34466ae651d8e039608d8","error":null,"hash":"9c3ef9691582689edf4deaf5dac055ea","phase":"item_import","target_ids":{"asset_id":"7941"},"type":"photo"}2026-06-30 02:59:39
item_created{"cover_hash":"c05ad3dec0064c538ecf2db9055af45e","error":null,"hash":"f4b5be311b459978c622211673edaa9b","phase":"item_import","target_ids":{"asset_id":"7940"},"type":"photo"}2026-06-30 02:59:37
item_created{"cover_hash":"3973643b2a0143fd972ef6814124bd5b","error":null,"hash":"595397ce715bdec594694dc9c8f66bbc","phase":"item_import","target_ids":{"asset_id":"7939"},"type":"photo"}2026-06-30 02:59:35
item_created{"cover_hash":"99529c6b492a462b95c34851d4262e9b","error":null,"hash":"fb7e81b9680d7e480d88744b78fb9904","phase":"item_import","target_ids":{"asset_id":"7938"},"type":"photo"}2026-06-30 02:59:33
item_created{"cover_hash":"0b56b7e844c84ec0ad35077533fa6518","error":null,"hash":"524f043becd22f548bd39697cf344b3b","phase":"item_import","target_ids":{"asset_id":"7937"},"type":"reel"}2026-06-30 02:59:31
item_created{"cover_hash":"8d84a644941f404ba5805eb7baa730cc","error":null,"hash":"01da9eec8145af3350ac83e65e27df03","phase":"item_import","target_ids":{"asset_id":"7936"},"type":"reel"}2026-06-30 02:59:28
item_created{"cover_hash":"b53590d83b3747e99ded982ef57f4ed2","error":null,"hash":"63f7d754b29882183bbc3407f3cfa2fb","phase":"item_import","target_ids":{"asset_id":"7935"},"type":"reel"}2026-06-30 02:59:26
item_created{"cover_hash":"193d831dc2d744d6b7cbdc92893daad7","error":null,"hash":"a8e2f6bcaee92ffb22e7b28b927f85b9","phase":"item_import","target_ids":{"asset_id":"7934"},"type":"reel"}2026-06-30 02:59:23
item_created{"cover_hash":"4bfa19401c3946fc8eed900fc5465735","error":null,"hash":"16f33e474c1a8b788a7f38451ac5f22f","phase":"item_import","target_ids":{"asset_id":"7933"},"type":"reel"}2026-06-30 02:59:21
item_created{"cover_hash":"4a094fff8c724275982ff7f87a586c4c","error":null,"hash":"63a3561f631dc84c184a403af8f18411","phase":"item_import","target_ids":{"asset_id":"7932"},"type":"reel"}2026-06-30 02:59:18
item_created{"cover_hash":"bda712a729224c3d927e171c839c99f9","error":null,"hash":"2e71d8f3c4feffc2aab91919564af75e","phase":"item_import","target_ids":{"asset_id":"7931"},"type":"reel"}2026-06-30 02:59:16
item_created{"cover_hash":"020f439b989643babd4b35971842d77f","error":null,"hash":"162b25c94f7cc4604921e9c769d05f9d","phase":"item_import","target_ids":{"asset_id":"7930"},"type":"reel"}2026-06-30 02:59:13
item_created{"cover_hash":"577c7ecca47545d4a37bb010c9df23f5","error":null,"hash":"866cec93348b00b457a4c75061e4b708","phase":"item_import","target_ids":{"asset_id":"7929"},"type":"reel"}2026-06-30 02:59:10
item_created{"cover_hash":"ae9d287281fb45e982ca5fe776b14a65","error":null,"hash":"c9183775a59be8f08638178cc849b789","phase":"item_import","target_ids":{"asset_id":"7928"},"type":"reel"}2026-06-30 02:59:08
item_created{"cover_hash":"79b41bf25b5b4072a6ea69aa96ba7ea8","error":null,"hash":"593826c22db55679c8ddc7dfeb481b9c","phase":"item_import","target_ids":{"asset_id":"7927"},"type":"reel"}2026-06-30 02:59:05
item_created{"cover_hash":null,"error":null,"hash":"ee6769d1bcfc5750b000449bc44ff8b1","phase":"item_import","target_ids":{"asset_id":"7926"},"type":"reel"}2026-06-30 02:59:03
item_created{"cover_hash":"0c56522f0db04ba8b81def73866d1eae","error":null,"hash":"3557d274433f979905eb36bb40d7a759","phase":"item_import","target_ids":{"asset_id":"7925"},"type":"reel"}2026-06-30 02:59:01
item_created{"cover_hash":null,"error":null,"hash":"8cea61ea979b38d4e4f050ee3c46c60b","phase":"item_import","target_ids":{"asset_id":"7924"},"type":"reel"}2026-06-30 02:58:58
item_created{"cover_hash":"a6b8df556ab6459cada3bd0d1babcda7","error":null,"hash":"9ef78834765abf214035783ed2935add","phase":"item_import","target_ids":{"asset_id":"7923"},"type":"reel"}2026-06-30 02:58:56
item_created{"cover_hash":null,"error":null,"hash":"49d168ecbeeb66affedfb5c975f2e825","phase":"item_import","target_ids":{"asset_id":"7922"},"type":"reel"}2026-06-30 02:58:54
item_created{"cover_hash":"c6228280a450429e9802e4903eec3d74","error":null,"hash":"e763b35d69744a8931716757f01825ea","phase":"item_import","target_ids":{"asset_id":"7921"},"type":"reel"}2026-06-30 02:58:52
item_created{"cover_hash":null,"error":null,"hash":"2196206750eaa1e9577226f9d7c06a3c","phase":"item_import","target_ids":{"asset_id":"7920"},"type":"reel"}2026-06-30 02:58:50
item_created{"cover_hash":"ea081038e6464262bfedc82d3682d5ef","error":null,"hash":"9c55ff407eb08a171404de7f7472c84f","phase":"item_import","target_ids":{"asset_id":"7919"},"type":"reel"}2026-06-30 02:58:48
item_created{"cover_hash":null,"error":null,"hash":"1dfc290b2c7e2acf72ecb410a0d436c5","phase":"item_import","target_ids":{"asset_id":"7918"},"type":"reel"}2026-06-30 02:58:45
item_created{"cover_hash":"98cee4d50a264ff4b11c3c7386a74b72","error":null,"hash":"a37a134d8592a56f54ef86601bfa3368","phase":"item_import","target_ids":{"asset_id":"7917"},"type":"reel"}2026-06-30 02:58:44
item_created{"cover_hash":null,"error":null,"hash":"e2c1a55eabdc36256a95d2b22867ea0f","phase":"item_import","target_ids":{"asset_id":"7916"},"type":"reel"}2026-06-30 02:58:41
item_created{"cover_hash":"fc264ae111c34d02899031128560b01c","error":null,"hash":"936cf1327d26b9f591de36c42016e563","phase":"item_import","target_ids":{"asset_id":"7915"},"type":"reel"}2026-06-30 02:58:39