Import Media
Queue cloud-sync handover batches and monitor worker runs.
About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.
Purpose
Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).
Flow
- Scan: walks the mount for
.readysentinels and records newingest_batchrows. - Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
- Queue: auto-preflights if needed, then inserts an
import_jobrow. Live progress shows on Import Tasks. - Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.
Example
Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.
Consequences
- The library is site-agnostic. Sites surface library content via
asset_site_membership, gallery/collection membership, content allowlists, and access packages. - VIDEO-SIMPLE syncs import as stable
asset_type=videorows withasset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables. - Cancel is allowed only while a job is queued; once running, only
systemctl stop gpu-admin-panel-import-workerstops it. - Action
removeonly soft-deletes rows viadeleted_at; nothing is hard-deleted. - Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive
Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.
Worker contract
A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).
Sentinel protocol
A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.
Source-of-truth mapping
source/source.json->asset_source(cold provenance).source/data/photo.manifest.json+photo.analysis.json+photo.meaning.json+photo.family.json->asset_ai_runrows with explicitanalysis_kind; unknown filenames are captured withsource_data:<file>.- Bundled
cover_hash->asset_coverlinked to the parent asset. - Containers of type
gallery->asset_gallery+asset_gallery_itemjunctions.
Idempotency + integrity
Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.
Read-only mount + ledger boundary
The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.
Batch 53f64da3-003c-4fd2-a8af-49dca8ee2c71
Status: imported; action: import
Type
asset
Action
import
Status
imported
Items
26
Bytes
3.26 GB
Source host
orbit-2
Manifest sha256
2ba0423250daa666...
Started
2026-06-30 19:10:10
Completed
2026-06-30 19:18:07
Manifest summary
- created_at: 2026-06-30T19:08:38Z
- completed_at: 2026-06-30T19:08:38Z
- verification.passed: true
- verification.total_files: 5810
- prior_sync_id: none
Job history 1
Newest first. The active job (if any) is at the top.
| id | state | requested_by | queued | claimed | worker | progress | finished | last_error |
|---|---|---|---|---|---|---|---|---|
| 67 | done | import-feed | 2026-06-30 19:10:23 | 2026-06-30 19:10:24 | vultr/pid:3795060 | 26/26 | 2026-06-30 19:18:07 | - |
Items 26
Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.
| asset_type | asset_hash | cover_hash | decision | target ids | last_error | updated_at |
|---|---|---|---|---|---|---|
| video | 058f7a2c74f8f738... | 24c30fcd64714d02... | created | asset:9224 | - | 2026-06-30 19:11:21 |
| reel | 2f1a127ecce15371... | e08a581e503f4883... | created | asset:9225 | - | 2026-06-30 19:11:28 |
| reel | 4d6f50ce3820f12f... | 5e2419ef2beb4a4f... | created | asset:9226 | - | 2026-06-30 19:11:40 |
| reel | edd1753bda9abc9f... | 3d6c34088b5e40f8... | created | asset:9227 | - | 2026-06-30 19:11:57 |
| reel | 79916da010175513... | fbaa62b4f4bf4849... | created | asset:9228 | - | 2026-06-30 19:12:10 |
| reel | 78be2c18dfab7dec... | d85ba3e59c4949ba... | created | asset:9229 | - | 2026-06-30 19:12:23 |
| reel | 8f4f0b6340912d48... | 68363ef20cf648b4... | created | asset:9230 | - | 2026-06-30 19:12:38 |
| reel | 643b68d68ed07aff... | 7db54b5f98a442d7... | created | asset:9231 | - | 2026-06-30 19:13:00 |
| reel | 1998699545b82ccb... | 2bfe6252137b470c... | created | asset:9232 | - | 2026-06-30 19:13:13 |
| reel | 5f6546e39f81e65c... | 02e91d35e2d64653... | created | asset:9233 | - | 2026-06-30 19:13:43 |
| reel | 79161d8ce949f414... | b4df5a0bfe174d66... | created | asset:9234 | - | 2026-06-30 19:13:57 |
| reel | fc81b6d7b78d1c5f... | 695b65b708384687... | created | asset:9235 | - | 2026-06-30 19:14:29 |
| reel | e04f42de6f73e230... | 20d60009085d43fc... | created | asset:9236 | - | 2026-06-30 19:14:43 |
| reel | dcc584bf24f56919... | f61eeb592825489e... | created | asset:9237 | - | 2026-06-30 19:15:09 |
| photo | 54160c7441d79ea6... | 772334a265d34024... | created | asset:9238 | - | 2026-06-30 19:15:49 |
| photo | e6a98cf959b3d558... | 49a487c838894f6e... | created | asset:9239 | - | 2026-06-30 19:16:10 |
| photo | 1b49295ff867c836... | 9272b7e6474d43a4... | created | asset:9240 | - | 2026-06-30 19:16:20 |
| photo | 5870310e5c7893fe... | e27a5e1dc47d43ed... | created | asset:9241 | - | 2026-06-30 19:16:26 |
| photo | 56ec5d283aa2fe37... | 74b54e29ebe04119... | created | asset:9242 | - | 2026-06-30 19:16:40 |
| photo | 791c3e5c091a22ca... | 35ad3c50e5e648ef... | created | asset:9243 | - | 2026-06-30 19:16:56 |
| photo | 676f0ad97bfdc5dd... | 13049a2e8acd48c5... | created | asset:9244 | - | 2026-06-30 19:17:13 |
| photo | f38891201c3eecbd... | 1a3211c405184225... | created | asset:9245 | - | 2026-06-30 19:17:27 |
| photo | 80f80ed2cbed1bc8... | 5a8e9d2ee76d41d1... | created | asset:9246 | - | 2026-06-30 19:17:47 |
| photo | 64d9b79b756deb3c... | 20f679f06b8341f1... | created | asset:9247 | - | 2026-06-30 19:17:54 |
| photo | 7186545b5cfbc2a2... | 8f463b326af14f11... | created | asset:9248 | - | 2026-06-30 19:18:01 |
| photo | 85c995aacfba36d9... | 0fabe4a0c7de452d... | created | asset:9249 | - | 2026-06-30 19:18:07 |
Recent events
Append-only audit ledger for this batch (latest 50).
| event_type | payload | created_at |
|---|---|---|
| job_finished | {"error":null,"job_id":"67","phase":"job_finish","summary":{"items_seen":26,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":26,"deferred":0,"failed":0... | 2026-06-30 19:18:07 |
| import_finished | {"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":26,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}} | 2026-06-30 19:18:07 |
| item_updated | {"container":1,"expected_member_count":12,"gallery_hash":"cb684b6c1b787f076eda08020ef73baa","gallery_id":"53","linked_member_count":12,"missing_member_count":0,"phase":"container_link","title":"Contes... | 2026-06-30 19:18:07 |
| item_created | {"cover_hash":"0fabe4a0c7de452dba09fee06578bea4","error":null,"hash":"85c995aacfba36d9052b5bd3a1b1ada3","phase":"item_import","target_ids":{"asset_id":"9249"},"type":"photo"} | 2026-06-30 19:18:07 |
| item_created | {"cover_hash":"8f463b326af14f11bec1812bcbb3b443","error":null,"hash":"7186545b5cfbc2a292535af5db515144","phase":"item_import","target_ids":{"asset_id":"9248"},"type":"photo"} | 2026-06-30 19:18:01 |
| item_created | {"cover_hash":"20f679f06b8341f193e90e1296139847","error":null,"hash":"64d9b79b756deb3c9eb60a706b8fcf9f","phase":"item_import","target_ids":{"asset_id":"9247"},"type":"photo"} | 2026-06-30 19:17:54 |
| item_created | {"cover_hash":"5a8e9d2ee76d41d1a6c02ae3bda71402","error":null,"hash":"80f80ed2cbed1bc86d47a4552bbeda27","phase":"item_import","target_ids":{"asset_id":"9246"},"type":"photo"} | 2026-06-30 19:17:47 |
| item_created | {"cover_hash":"1a3211c4051842258f3f8d999b2a716a","error":null,"hash":"f38891201c3eecbd65a17b4d9b3f5caa","phase":"item_import","target_ids":{"asset_id":"9245"},"type":"photo"} | 2026-06-30 19:17:27 |
| item_created | {"cover_hash":"13049a2e8acd48c5963b18921fc89d07","error":null,"hash":"676f0ad97bfdc5dd9a429f31d63476b8","phase":"item_import","target_ids":{"asset_id":"9244"},"type":"photo"} | 2026-06-30 19:17:13 |
| item_created | {"cover_hash":"35ad3c50e5e648ef8dcea9c16d28f6a2","error":null,"hash":"791c3e5c091a22ca95e337eac96b3e68","phase":"item_import","target_ids":{"asset_id":"9243"},"type":"photo"} | 2026-06-30 19:16:56 |
| item_created | {"cover_hash":"74b54e29ebe04119823f89f88044a40b","error":null,"hash":"56ec5d283aa2fe375338d6b463e8d0c5","phase":"item_import","target_ids":{"asset_id":"9242"},"type":"photo"} | 2026-06-30 19:16:40 |
| item_created | {"cover_hash":"e27a5e1dc47d43ed9e87b1439d79c615","error":null,"hash":"5870310e5c7893feb165a445310540fc","phase":"item_import","target_ids":{"asset_id":"9241"},"type":"photo"} | 2026-06-30 19:16:26 |
| item_created | {"cover_hash":"9272b7e6474d43a49d9a32b26cdc1bd6","error":null,"hash":"1b49295ff867c83681243ec5066eefff","phase":"item_import","target_ids":{"asset_id":"9240"},"type":"photo"} | 2026-06-30 19:16:20 |
| item_created | {"cover_hash":"49a487c838894f6e9ca79105be58676e","error":null,"hash":"e6a98cf959b3d558bfd540a8c5364c08","phase":"item_import","target_ids":{"asset_id":"9239"},"type":"photo"} | 2026-06-30 19:16:10 |
| item_created | {"cover_hash":"772334a265d34024ad995d787a65abcf","error":null,"hash":"54160c7441d79ea6bdb300af85795f1c","phase":"item_import","target_ids":{"asset_id":"9238"},"type":"photo"} | 2026-06-30 19:15:49 |
| item_created | {"cover_hash":"f61eeb592825489eb12cf2d5191ad09b","error":null,"hash":"dcc584bf24f569195f6bdf1e6bfab0ff","phase":"item_import","target_ids":{"asset_id":"9237"},"type":"reel"} | 2026-06-30 19:15:09 |
| item_created | {"cover_hash":"20d60009085d43fc921a97327c8c9fbb","error":null,"hash":"e04f42de6f73e23024d41c189ab36bc3","phase":"item_import","target_ids":{"asset_id":"9236"},"type":"reel"} | 2026-06-30 19:14:43 |
| item_created | {"cover_hash":"695b65b708384687b327ebe71ca8fc07","error":null,"hash":"fc81b6d7b78d1c5f8df113214b0a48b1","phase":"item_import","target_ids":{"asset_id":"9235"},"type":"reel"} | 2026-06-30 19:14:29 |
| item_created | {"cover_hash":"b4df5a0bfe174d6686acc33eaf35ae48","error":null,"hash":"79161d8ce949f4144614df449887c957","phase":"item_import","target_ids":{"asset_id":"9234"},"type":"reel"} | 2026-06-30 19:13:57 |
| item_created | {"cover_hash":"02e91d35e2d64653b5701146d8598516","error":null,"hash":"5f6546e39f81e65c81b42d108b5fb35c","phase":"item_import","target_ids":{"asset_id":"9233"},"type":"reel"} | 2026-06-30 19:13:43 |
| item_created | {"cover_hash":"2bfe6252137b470cbe1baa838ca8cf5b","error":null,"hash":"1998699545b82ccbdb54935fb32968e3","phase":"item_import","target_ids":{"asset_id":"9232"},"type":"reel"} | 2026-06-30 19:13:13 |
| item_created | {"cover_hash":"7db54b5f98a442d78157913624b54a4b","error":null,"hash":"643b68d68ed07aff48d5f4f5ec3f2fa8","phase":"item_import","target_ids":{"asset_id":"9231"},"type":"reel"} | 2026-06-30 19:13:00 |
| item_created | {"cover_hash":"68363ef20cf648b4b21700e2c37b8fef","error":null,"hash":"8f4f0b6340912d488c2e1ae502f37453","phase":"item_import","target_ids":{"asset_id":"9230"},"type":"reel"} | 2026-06-30 19:12:38 |
| item_created | {"cover_hash":"d85ba3e59c4949ba88a837bf4c109c71","error":null,"hash":"78be2c18dfab7dec23b47c8a3d629bc6","phase":"item_import","target_ids":{"asset_id":"9229"},"type":"reel"} | 2026-06-30 19:12:23 |
| item_created | {"cover_hash":"fbaa62b4f4bf4849a87c147bcbc4aa6b","error":null,"hash":"79916da010175513cfa00f4d1e8b5ad9","phase":"item_import","target_ids":{"asset_id":"9228"},"type":"reel"} | 2026-06-30 19:12:10 |
| item_created | {"cover_hash":"3d6c34088b5e40f8a092f58e46c2185b","error":null,"hash":"edd1753bda9abc9fb0263b11f4523fcb","phase":"item_import","target_ids":{"asset_id":"9227"},"type":"reel"} | 2026-06-30 19:11:57 |
| item_created | {"cover_hash":"5e2419ef2beb4a4f89a607de06aa489b","error":null,"hash":"4d6f50ce3820f12f52d854d225c6d96f","phase":"item_import","target_ids":{"asset_id":"9226"},"type":"reel"} | 2026-06-30 19:11:40 |
| item_created | {"cover_hash":"e08a581e503f4883aa83a622f35fcbb1","error":null,"hash":"2f1a127ecce15371d495ede76aa4cd18","phase":"item_import","target_ids":{"asset_id":"9225"},"type":"reel"} | 2026-06-30 19:11:28 |
| item_created | {"cover_hash":"24c30fcd64714d02ae6f5c9da9a519a1","error":null,"hash":"058f7a2c74f8f738803366cbe5155f1c","phase":"item_import","target_ids":{"asset_id":"9224"},"type":"video"} | 2026-06-30 19:11:21 |
| import_started | {"action":"import","already_terminal_count":0,"container_count":2,"item_count":26,"manifest_sha256":"2ba0423250daa666deb0b37ad4dc72bf0b99c2df887bdf5786cba5e6ea61c702","max_items":0,"phase":"import"} | 2026-06-30 19:10:26 |
| job_claimed | {"job_id":"67","phase":"job_claim","priority":100,"worker_id":"vultr/pid:3795060"} | 2026-06-30 19:10:24 |
| job_queued | {"job_id": "67", "requested_by": "import-feed", "items_total": "26"} | 2026-06-30 19:10:23 |
| preflight_finished | {"counts":{"will_create":26,"will_fail":0,"will_link":0,"will_skip":0,"will_update":0},"failure_samples":[],"item_count":26,"item_type_counts":{"photo":12,"reel":13,"video":1},"ok":1,"phase":"prefligh... | 2026-06-30 19:10:23 |
| preflight_started | {"action":"import","container_count":2,"item_count":26,"item_type_counts":{"photo":12,"reel":13,"video":1},"manifest_sha256":"2ba0423250daa666deb0b37ad4dc72bf0b99c2df887bdf5786cba5e6ea61c702","phase":... | 2026-06-30 19:10:10 |
| scan | {"source": "import-feed", "manifest_sha256": "2ba0423250daa666deb0b37ad4dc72bf0b99c2df887bdf5786cba5e6ea61c702", "sync_type": "asset", "action": "import", "item_count": "26"} | 2026-06-30 19:10:10 |