Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch f9e415ed-8a77-493c-b3f2-c4ec6083eb82

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

56

Bytes

3.09 GB

Source host

orbit-2

Manifest sha256

7e80ccd4acd72e6a...

Started

2026-06-30 02:28:45

Completed

2026-06-30 03:07:07

Manifest summary

  • created_at: 2026-06-29T21:13:20Z
  • completed_at: 2026-06-29T21:13:20Z
  • verification.passed: true
  • verification.total_files: 5069
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
58doneimport-feed2026-06-30 02:28:512026-06-30 03:04:58vultr/pid:379506056/562026-06-30 03:07:07-

Items 56

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
videoa0fa74695ce54556...99dedc9f10b94156...createdasset:8047-2026-06-30 03:05:05
reel2f7e547001a446d6...1fb74cd9c2084d17...createdasset:8048-2026-06-30 03:05:08
reel71fa0e38e804fd06...-createdasset:8049-2026-06-30 03:05:10
reel1375c92350891a29...6e39231389b14a45...createdasset:8050-2026-06-30 03:05:12
reelcece9b90eafb9e06...-createdasset:8051-2026-06-30 03:05:14
reelfca1ae8af469abbb...e6a79670dd734f11...createdasset:8052-2026-06-30 03:05:17
reelb6494e013da7cd01...-createdasset:8053-2026-06-30 03:05:19
reel6c479b12ad95ed43...75e9b6586cdd403a...createdasset:8054-2026-06-30 03:05:21
reel8ff28bbe46cf852d...-createdasset:8055-2026-06-30 03:05:23
reel40fa18158e66208b...6d6a4233a0eb4ffe...createdasset:8056-2026-06-30 03:05:26
reel4bc9f661f6f47c3d...-createdasset:8057-2026-06-30 03:05:27
reel73bcb796647f66cd...881c280e759d4d8c...createdasset:8058-2026-06-30 03:05:30
reelc7c0924a8b99ce64...-createdasset:8059-2026-06-30 03:05:32
reel77bb163060a9ac22...ba2353e4792c4b4f...createdasset:8060-2026-06-30 03:05:35
reeld6560f5e27c85a0c...-createdasset:8061-2026-06-30 03:05:36
reel35b5b0ea45258c7e...5a3e74e36a214f7c...createdasset:8062-2026-06-30 03:05:39
reel29b84e20e8ada448...-createdasset:8063-2026-06-30 03:05:41
reel156da4a8fe52cf76...0eab909fcb4e44da...createdasset:8064-2026-06-30 03:05:43
reeldfb6377738f717bd...-createdasset:8065-2026-06-30 03:05:45
reelcc6150abf529b6af...57f85a2115c04c4d...createdasset:8066-2026-06-30 03:05:47
reel4c022a4804b8f527...-createdasset:8067-2026-06-30 03:05:50
reel1b3f47c4b16f4b04...9c480451ae894b36...createdasset:8068-2026-06-30 03:05:52
reel33de1f6af22dff16...-createdasset:8069-2026-06-30 03:05:54
reela72b528eed0e4933...85d6ee543b5141c9...createdasset:8070-2026-06-30 03:05:57
reelff5c6c76eaef80b1...-createdasset:8071-2026-06-30 03:05:59
reelace403455f274d0a...b6223e4dd2c1423c...createdasset:8072-2026-06-30 03:06:01
reel31d054a87237d3b5...-createdasset:8073-2026-06-30 03:06:03
reel0f7efc0fe054814f...3102c0faae8e4f9a...createdasset:8074-2026-06-30 03:06:06
reel6de0dde718ca8516...-createdasset:8075-2026-06-30 03:06:07
reel13d07264da681235...e525063b380b40d4...createdasset:8076-2026-06-30 03:06:10
reelcc56047367549e86...e3614c0df41d49eb...createdasset:8077-2026-06-30 03:06:12
reel49b5f736a7805967...4ad54609ccd24ed1...createdasset:8078-2026-06-30 03:06:15
reelb8591a27c5964e65...d15b25bb23824faf...createdasset:8079-2026-06-30 03:06:18
reel8c872e99a330a716...22f895f20f2941cd...createdasset:8080-2026-06-30 03:06:20
reel03f6df0450ea0f2d...3b943881f6ec4623...createdasset:8081-2026-06-30 03:06:23
reel4e1acfe991873132...24b2278c0f254355...createdasset:8082-2026-06-30 03:06:26
reel5d8a7ad3d5954307...873dff3f087241d7...createdasset:8083-2026-06-30 03:06:28
reel071fa7cdc2f3a66d...79261f975c0145f9...createdasset:8084-2026-06-30 03:06:31
photo1c306b204998d0ea...601da6d303ff463f...createdasset:8085-2026-06-30 03:06:33
photo4fe741c8d5c0a5e3...a69ceee2219f4be8...createdasset:8086-2026-06-30 03:06:35
photo26167c25c986435c...34bf30ab88b44a82...createdasset:8087-2026-06-30 03:06:37
photo40d84e5fe7bec855...30e1afa3388b451d...createdasset:8088-2026-06-30 03:06:39
photo35290bb77864476a...865dd038ef844f22...createdasset:8089-2026-06-30 03:06:41
photobf65535707b5489c...b909510cb62f4a6a...createdasset:8090-2026-06-30 03:06:43
photod69f199561125bc5...0f1aab0fd0134143...createdasset:8091-2026-06-30 03:06:45
photo2af5a024c12a74e0...b56fe7f0535849f6...createdasset:8092-2026-06-30 03:06:47
photofdbc471a7fac3b97...9184cb8eb1c34d73...createdasset:8093-2026-06-30 03:06:49
photo92f22d22da1b26af...254598f46dca42d8...createdasset:8094-2026-06-30 03:06:51
photoc5dc0ae5a6d9d5c1...5102729391684b34...createdasset:8095-2026-06-30 03:06:53
photod6b661740940b495...6a5672558e66482b...createdasset:8096-2026-06-30 03:06:55
photoe0544e10a58cdb80...ce47c8377700432f...createdasset:8097-2026-06-30 03:06:57
photo42e7e43e20b40b51...2ac5c752753c415c...createdasset:8098-2026-06-30 03:06:59
photoc7c90286f34963ba...2569b2826b764931...createdasset:8099-2026-06-30 03:07:01
photoc72a4eedd7800a2b...4a854aa0ce4945ef...createdasset:8100-2026-06-30 03:07:03
photo3527d728547ed53f...48818d51cf334965...createdasset:8101-2026-06-30 03:07:05
photo6d22b54a1ac04550...3c2f4e6e8be4445a...createdasset:8102-2026-06-30 03:07:07

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"58","phase":"job_finish","summary":{"items_seen":56,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":56,"deferred":0,"failed":0...2026-06-30 03:07:07
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":56,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-30 03:07:07
item_updated{"container":1,"expected_member_count":18,"gallery_hash":"262f46a91760545bb2250bc324a5131a","gallery_id":"44","linked_member_count":18,"missing_member_count":0,"phase":"container_link","title":"Contes...2026-06-30 03:07:07
item_created{"cover_hash":"3c2f4e6e8be4445a89ae7f96428a806d","error":null,"hash":"6d22b54a1ac04550204b43645f212cf2","phase":"item_import","target_ids":{"asset_id":"8102"},"type":"photo"}2026-06-30 03:07:07
item_created{"cover_hash":"48818d51cf3349659b8b9140bc4073fd","error":null,"hash":"3527d728547ed53f055214f6ed3770f5","phase":"item_import","target_ids":{"asset_id":"8101"},"type":"photo"}2026-06-30 03:07:05
item_created{"cover_hash":"4a854aa0ce4945efa2120d46105e121d","error":null,"hash":"c72a4eedd7800a2bd22ee9c9c011ae8d","phase":"item_import","target_ids":{"asset_id":"8100"},"type":"photo"}2026-06-30 03:07:03
item_created{"cover_hash":"2569b2826b764931b2437cd39064484c","error":null,"hash":"c7c90286f34963ba18d7d08911a918c8","phase":"item_import","target_ids":{"asset_id":"8099"},"type":"photo"}2026-06-30 03:07:01
item_created{"cover_hash":"2ac5c752753c415cb5836f77e9e6f823","error":null,"hash":"42e7e43e20b40b51312daa72e3fe8a7a","phase":"item_import","target_ids":{"asset_id":"8098"},"type":"photo"}2026-06-30 03:06:59
item_created{"cover_hash":"ce47c8377700432fa4c00a73a1b5de08","error":null,"hash":"e0544e10a58cdb805616e72f3dfa078d","phase":"item_import","target_ids":{"asset_id":"8097"},"type":"photo"}2026-06-30 03:06:57
item_created{"cover_hash":"6a5672558e66482bab95ecb374729de7","error":null,"hash":"d6b661740940b4959ac1fdf3ba2c516c","phase":"item_import","target_ids":{"asset_id":"8096"},"type":"photo"}2026-06-30 03:06:55
item_created{"cover_hash":"5102729391684b34a348c07066d6ddc4","error":null,"hash":"c5dc0ae5a6d9d5c1680a2a6df51af25b","phase":"item_import","target_ids":{"asset_id":"8095"},"type":"photo"}2026-06-30 03:06:53
item_created{"cover_hash":"254598f46dca42d8a30953ccfa0baf54","error":null,"hash":"92f22d22da1b26afce3389e0a53d1139","phase":"item_import","target_ids":{"asset_id":"8094"},"type":"photo"}2026-06-30 03:06:51
item_created{"cover_hash":"9184cb8eb1c34d73b47f0737926ca52e","error":null,"hash":"fdbc471a7fac3b976f57ade6887acf5c","phase":"item_import","target_ids":{"asset_id":"8093"},"type":"photo"}2026-06-30 03:06:49
item_created{"cover_hash":"b56fe7f0535849f698bce020d0f486e7","error":null,"hash":"2af5a024c12a74e0a7e16e04127c0832","phase":"item_import","target_ids":{"asset_id":"8092"},"type":"photo"}2026-06-30 03:06:47
item_created{"cover_hash":"0f1aab0fd01341439ecefbce82722c7b","error":null,"hash":"d69f199561125bc5efd987b578806609","phase":"item_import","target_ids":{"asset_id":"8091"},"type":"photo"}2026-06-30 03:06:45
item_created{"cover_hash":"b909510cb62f4a6abdb7a50a5bdc73d7","error":null,"hash":"bf65535707b5489ce2409ff7039fbe5b","phase":"item_import","target_ids":{"asset_id":"8090"},"type":"photo"}2026-06-30 03:06:43
item_created{"cover_hash":"865dd038ef844f22bde3b85b66e0d456","error":null,"hash":"35290bb77864476ab500168b2bff0ae1","phase":"item_import","target_ids":{"asset_id":"8089"},"type":"photo"}2026-06-30 03:06:41
item_created{"cover_hash":"30e1afa3388b451d94c75929663999f6","error":null,"hash":"40d84e5fe7bec8552d9aa034656c291f","phase":"item_import","target_ids":{"asset_id":"8088"},"type":"photo"}2026-06-30 03:06:39
item_created{"cover_hash":"34bf30ab88b44a82b64a179d4b492573","error":null,"hash":"26167c25c986435c4f4ed39470ff0498","phase":"item_import","target_ids":{"asset_id":"8087"},"type":"photo"}2026-06-30 03:06:37
item_created{"cover_hash":"a69ceee2219f4be89561552b16ad8926","error":null,"hash":"4fe741c8d5c0a5e30797a70561b43338","phase":"item_import","target_ids":{"asset_id":"8086"},"type":"photo"}2026-06-30 03:06:35
item_created{"cover_hash":"601da6d303ff463f82deaec2588d8905","error":null,"hash":"1c306b204998d0ead23c39047168c869","phase":"item_import","target_ids":{"asset_id":"8085"},"type":"photo"}2026-06-30 03:06:33
item_created{"cover_hash":"79261f975c0145f99a9ee019a5ffad21","error":null,"hash":"071fa7cdc2f3a66d5f0ff97a6af09862","phase":"item_import","target_ids":{"asset_id":"8084"},"type":"reel"}2026-06-30 03:06:31
item_created{"cover_hash":"873dff3f087241d79a574884ab967d27","error":null,"hash":"5d8a7ad3d595430731c7ff1b601f2f0c","phase":"item_import","target_ids":{"asset_id":"8083"},"type":"reel"}2026-06-30 03:06:28
item_created{"cover_hash":"24b2278c0f25435580f12cf0e4945fd7","error":null,"hash":"4e1acfe991873132704b4850a2305301","phase":"item_import","target_ids":{"asset_id":"8082"},"type":"reel"}2026-06-30 03:06:26
item_created{"cover_hash":"3b943881f6ec462386d91c9644d4117a","error":null,"hash":"03f6df0450ea0f2d3b1fd26139c6f1f9","phase":"item_import","target_ids":{"asset_id":"8081"},"type":"reel"}2026-06-30 03:06:23
item_created{"cover_hash":"22f895f20f2941cda3f3a268477aa4df","error":null,"hash":"8c872e99a330a716e1e0721ad6809ed9","phase":"item_import","target_ids":{"asset_id":"8080"},"type":"reel"}2026-06-30 03:06:20
item_created{"cover_hash":"d15b25bb23824fafa80ece002b3d1bfa","error":null,"hash":"b8591a27c5964e657d676f7a581094b1","phase":"item_import","target_ids":{"asset_id":"8079"},"type":"reel"}2026-06-30 03:06:18
item_created{"cover_hash":"4ad54609ccd24ed18b897fc7c52afe0c","error":null,"hash":"49b5f736a7805967d5b543c7286ce8df","phase":"item_import","target_ids":{"asset_id":"8078"},"type":"reel"}2026-06-30 03:06:15
item_created{"cover_hash":"e3614c0df41d49eba1a4bff0be524c81","error":null,"hash":"cc56047367549e86ed80fa85e8378384","phase":"item_import","target_ids":{"asset_id":"8077"},"type":"reel"}2026-06-30 03:06:12
item_created{"cover_hash":"e525063b380b40d4aba43e389888d69a","error":null,"hash":"13d07264da68123573948ddaab636d74","phase":"item_import","target_ids":{"asset_id":"8076"},"type":"reel"}2026-06-30 03:06:10
item_created{"cover_hash":null,"error":null,"hash":"6de0dde718ca85166ce2859ed07b2b1a","phase":"item_import","target_ids":{"asset_id":"8075"},"type":"reel"}2026-06-30 03:06:07
item_created{"cover_hash":"3102c0faae8e4f9ab3afd6f0879e2993","error":null,"hash":"0f7efc0fe054814fa17ca9d8da20ebb3","phase":"item_import","target_ids":{"asset_id":"8074"},"type":"reel"}2026-06-30 03:06:06
item_created{"cover_hash":null,"error":null,"hash":"31d054a87237d3b52bc58d3bea10090f","phase":"item_import","target_ids":{"asset_id":"8073"},"type":"reel"}2026-06-30 03:06:03
item_created{"cover_hash":"b6223e4dd2c1423c8791e0c973951c2e","error":null,"hash":"ace403455f274d0a547672eca5f3813b","phase":"item_import","target_ids":{"asset_id":"8072"},"type":"reel"}2026-06-30 03:06:01
item_created{"cover_hash":null,"error":null,"hash":"ff5c6c76eaef80b1993d2fd0bf28fab4","phase":"item_import","target_ids":{"asset_id":"8071"},"type":"reel"}2026-06-30 03:05:59
item_created{"cover_hash":"85d6ee543b5141c9bba42715c8dc066c","error":null,"hash":"a72b528eed0e49337e4746fd34047201","phase":"item_import","target_ids":{"asset_id":"8070"},"type":"reel"}2026-06-30 03:05:57
item_created{"cover_hash":null,"error":null,"hash":"33de1f6af22dff165b3ce9ed98dfd561","phase":"item_import","target_ids":{"asset_id":"8069"},"type":"reel"}2026-06-30 03:05:54
item_created{"cover_hash":"9c480451ae894b368e5546c878257213","error":null,"hash":"1b3f47c4b16f4b04b1726b3a4fa0e854","phase":"item_import","target_ids":{"asset_id":"8068"},"type":"reel"}2026-06-30 03:05:52
item_created{"cover_hash":null,"error":null,"hash":"4c022a4804b8f5276d2ae64267dc0a9e","phase":"item_import","target_ids":{"asset_id":"8067"},"type":"reel"}2026-06-30 03:05:50
item_created{"cover_hash":"57f85a2115c04c4d9ad9a1d5a6292f6e","error":null,"hash":"cc6150abf529b6aff3ac04cda421acf2","phase":"item_import","target_ids":{"asset_id":"8066"},"type":"reel"}2026-06-30 03:05:47
item_created{"cover_hash":null,"error":null,"hash":"dfb6377738f717bdc3618da880a98ad1","phase":"item_import","target_ids":{"asset_id":"8065"},"type":"reel"}2026-06-30 03:05:45
item_created{"cover_hash":"0eab909fcb4e44da8926726756257662","error":null,"hash":"156da4a8fe52cf763f4e85d78c9a9bcc","phase":"item_import","target_ids":{"asset_id":"8064"},"type":"reel"}2026-06-30 03:05:43
item_created{"cover_hash":null,"error":null,"hash":"29b84e20e8ada448979d67f18009a7fe","phase":"item_import","target_ids":{"asset_id":"8063"},"type":"reel"}2026-06-30 03:05:41
item_created{"cover_hash":"5a3e74e36a214f7cba03c30ebbcfdb62","error":null,"hash":"35b5b0ea45258c7e901639c7dcd66576","phase":"item_import","target_ids":{"asset_id":"8062"},"type":"reel"}2026-06-30 03:05:39
item_created{"cover_hash":null,"error":null,"hash":"d6560f5e27c85a0c1ded07b31aa0de9b","phase":"item_import","target_ids":{"asset_id":"8061"},"type":"reel"}2026-06-30 03:05:36
item_created{"cover_hash":"ba2353e4792c4b4fafad68458685573f","error":null,"hash":"77bb163060a9ac228bfa1c457b09ba13","phase":"item_import","target_ids":{"asset_id":"8060"},"type":"reel"}2026-06-30 03:05:35
item_created{"cover_hash":null,"error":null,"hash":"c7c0924a8b99ce646382a15c589b8fd8","phase":"item_import","target_ids":{"asset_id":"8059"},"type":"reel"}2026-06-30 03:05:32
item_created{"cover_hash":"881c280e759d4d8cbb7a02de07496b60","error":null,"hash":"73bcb796647f66cd1ea8c24a93912e0f","phase":"item_import","target_ids":{"asset_id":"8058"},"type":"reel"}2026-06-30 03:05:30
item_created{"cover_hash":null,"error":null,"hash":"4bc9f661f6f47c3db65136f7841af4ee","phase":"item_import","target_ids":{"asset_id":"8057"},"type":"reel"}2026-06-30 03:05:27
item_created{"cover_hash":"6d6a4233a0eb4ffeb741fb704ea4c784","error":null,"hash":"40fa18158e66208badfdfb5fd73ba60f","phase":"item_import","target_ids":{"asset_id":"8056"},"type":"reel"}2026-06-30 03:05:26