Import Media
Queue cloud-sync handover batches and monitor worker runs.
About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.
Purpose
Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).
Flow
- Scan: walks the mount for
.readysentinels and records newingest_batchrows. - Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
- Queue: auto-preflights if needed, then inserts an
import_jobrow. Live progress shows on Import Tasks. - Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.
Example
Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.
Consequences
- The library is site-agnostic. Sites surface library content via
asset_site_membership, gallery/collection membership, content allowlists, and access packages. - VIDEO-SIMPLE syncs import as stable
asset_type=videorows withasset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables. - Cancel is allowed only while a job is queued; once running, only
systemctl stop gpu-admin-panel-import-workerstops it. - Action
removeonly soft-deletes rows viadeleted_at; nothing is hard-deleted. - Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive
Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.
Worker contract
A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).
Sentinel protocol
A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.
Source-of-truth mapping
source/source.json->asset_source(cold provenance).source/data/photo.manifest.json+photo.analysis.json+photo.meaning.json+photo.family.json->asset_ai_runrows with explicitanalysis_kind; unknown filenames are captured withsource_data:<file>.- Bundled
cover_hash->asset_coverlinked to the parent asset. - Containers of type
gallery->asset_gallery+asset_gallery_itemjunctions.
Idempotency + integrity
Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.
Read-only mount + ledger boundary
The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.
Batch deeae029-439e-4538-8f5f-c3b44eb1ded3
Status: imported; action: import
Type
asset
Action
import
Status
imported
Items
39
Bytes
3.04 GB
Source host
orbit-2
Manifest sha256
f9518ef9eb0ab116...
Started
2026-07-02 17:00:32
Completed
2026-07-02 17:02:03
Manifest summary
- created_at: 2026-07-02T16:59:32Z
- completed_at: 2026-07-02T16:59:32Z
- verification.passed: true
- verification.total_files: 5807
- prior_sync_id: none
Job history 1
Newest first. The active job (if any) is at the top.
| id | state | requested_by | queued | claimed | worker | progress | finished | last_error |
|---|---|---|---|---|---|---|---|---|
| 76 | done | import-feed | 2026-07-02 17:00:37 | 2026-07-02 17:00:41 | vultr/pid:3795060 | 39/39 | 2026-07-02 17:02:03 | - |
Items 39
Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.
| asset_type | asset_hash | cover_hash | decision | target ids | last_error | updated_at |
|---|---|---|---|---|---|---|
| video | 97a21488df1bcca2... | 7f48afaaa943456c... | created | asset:10375 | - | 2026-07-02 17:00:48 |
| reel | a7e545cac88b9246... | - | created | asset:10376 | - | 2026-07-02 17:00:50 |
| reel | d599e25f824e1d67... | 45317e37870c4bb7... | created | asset:10377 | - | 2026-07-02 17:00:52 |
| reel | 00ab0bc84c12f22d... | - | created | asset:10378 | - | 2026-07-02 17:00:54 |
| reel | 352f84c2b21ee3be... | - | created | asset:10379 | - | 2026-07-02 17:00:56 |
| reel | 6c63a81da30ad5aa... | - | created | asset:10380 | - | 2026-07-02 17:00:58 |
| reel | 720c9de52e7949fa... | - | created | asset:10381 | - | 2026-07-02 17:00:59 |
| reel | 6ecf33387d930bd6... | - | created | asset:10382 | - | 2026-07-02 17:01:01 |
| reel | 880af4a0b8089d25... | - | created | asset:10383 | - | 2026-07-02 17:01:03 |
| reel | 9f1a0ed15c66ef37... | - | created | asset:10384 | - | 2026-07-02 17:01:04 |
| reel | f2d5795142db81ad... | - | created | asset:10385 | - | 2026-07-02 17:01:06 |
| reel | 52d7a6c0faef1517... | - | created | asset:10386 | - | 2026-07-02 17:01:08 |
| reel | 613233915cfab6a8... | - | created | asset:10387 | - | 2026-07-02 17:01:10 |
| reel | 500ceb5e964c9ceb... | - | created | asset:10388 | - | 2026-07-02 17:01:12 |
| reel | 19248ca632512860... | - | created | asset:10389 | - | 2026-07-02 17:01:13 |
| photo | 53327024c6890bbb... | ec186dc5786c45be... | created | asset:10390 | - | 2026-07-02 17:01:15 |
| photo | dc3696cf4d25f1ce... | f133a16ba2734af3... | created | asset:10391 | - | 2026-07-02 17:01:17 |
| photo | d1eff59857259cf8... | 8450701a0842414c... | created | asset:10392 | - | 2026-07-02 17:01:20 |
| photo | 523dd7b2e8c83660... | d050ee3d13f94abb... | created | asset:10393 | - | 2026-07-02 17:01:22 |
| photo | 30fe9933e96a1665... | 543fc8799ded469e... | created | asset:10394 | - | 2026-07-02 17:01:24 |
| photo | 088c4ffb618e93f4... | 3d1fd49fb1eb440a... | created | asset:10395 | - | 2026-07-02 17:01:26 |
| photo | 7dfe1aa27cdc0995... | bdb7a10c694d4460... | created | asset:10396 | - | 2026-07-02 17:01:28 |
| photo | e59d849fb19855a7... | 2346d77832ea4f51... | created | asset:10397 | - | 2026-07-02 17:01:30 |
| photo | 1100c35554cb1ac3... | 8803ca27781a4b53... | created | asset:10398 | - | 2026-07-02 17:01:33 |
| photo | 4fcfd1eae766c6a9... | b39bdd15b5a744a5... | created | asset:10399 | - | 2026-07-02 17:01:35 |
| photo | b64c05c8958ac162... | 386c11a1982e41ad... | created | asset:10400 | - | 2026-07-02 17:01:37 |
| photo | 8458ae2990400d6e... | 0f93f30c61e54e31... | created | asset:10401 | - | 2026-07-02 17:01:39 |
| photo | 1fb434e3f645a0f7... | 130a4ef8c63b4c88... | created | asset:10402 | - | 2026-07-02 17:01:41 |
| photo | 6c937b120b852b70... | 74a22879f828427c... | created | asset:10403 | - | 2026-07-02 17:01:43 |
| photo | d3f5e84f38d3b04d... | 90a0dbdeddc14d83... | created | asset:10404 | - | 2026-07-02 17:01:45 |
| photo | d10118cac6ab64b3... | 4397652a58e747a8... | created | asset:10405 | - | 2026-07-02 17:01:47 |
| photo | ffef1b7584434f24... | 02aff9118c41447b... | created | asset:10406 | - | 2026-07-02 17:01:49 |
| photo | fb494d38260d362e... | cbe4f0bdb1814dbd... | created | asset:10407 | - | 2026-07-02 17:01:51 |
| photo | a2c34264dc237381... | a8322ca58fc84463... | created | asset:10408 | - | 2026-07-02 17:01:53 |
| photo | dd92698154540670... | ee46ce5f86044577... | created | asset:10409 | - | 2026-07-02 17:01:55 |
| photo | ac45c0222b65726b... | 241269066a934f12... | created | asset:10410 | - | 2026-07-02 17:01:57 |
| photo | dc32c7226fcb461a... | d7a06eb7d48c415a... | created | asset:10411 | - | 2026-07-02 17:01:59 |
| photo | 2e67d993736c4911... | 4db58254da714ffc... | created | asset:10412 | - | 2026-07-02 17:02:01 |
| photo | a91d8acf8620511f... | 1f71cc8fe98a498d... | created | asset:10413 | - | 2026-07-02 17:02:03 |
Recent events
Append-only audit ledger for this batch (latest 50).
| event_type | payload | created_at |
|---|---|---|
| job_finished | {"error":null,"job_id":"76","phase":"job_finish","summary":{"items_seen":39,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":39,"deferred":0,"failed":0... | 2026-07-02 17:02:03 |
| import_finished | {"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":39,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}} | 2026-07-02 17:02:03 |
| item_updated | {"container":1,"expected_member_count":24,"gallery_hash":"4680a104e3b65dfddf0656d99378baa0","gallery_id":"62","linked_member_count":24,"missing_member_count":0,"phase":"container_link","title":"countr... | 2026-07-02 17:02:03 |
| item_created | {"cover_hash":"1f71cc8fe98a498d8527c9174c841df6","error":null,"hash":"a91d8acf8620511f808054e173a19a74","phase":"item_import","target_ids":{"asset_id":"10413"},"type":"photo"} | 2026-07-02 17:02:03 |
| item_created | {"cover_hash":"4db58254da714ffca485255497474f76","error":null,"hash":"2e67d993736c491116ce42e9747b5e69","phase":"item_import","target_ids":{"asset_id":"10412"},"type":"photo"} | 2026-07-02 17:02:01 |
| item_created | {"cover_hash":"d7a06eb7d48c415a83047c87c953aa79","error":null,"hash":"dc32c7226fcb461a95c0fd1d287160af","phase":"item_import","target_ids":{"asset_id":"10411"},"type":"photo"} | 2026-07-02 17:01:59 |
| item_created | {"cover_hash":"241269066a934f12a64010d11e1b3616","error":null,"hash":"ac45c0222b65726b3443b602d6766a0c","phase":"item_import","target_ids":{"asset_id":"10410"},"type":"photo"} | 2026-07-02 17:01:57 |
| item_created | {"cover_hash":"ee46ce5f860445779489ede5a55d84da","error":null,"hash":"dd926981545406704833e4bbe0247940","phase":"item_import","target_ids":{"asset_id":"10409"},"type":"photo"} | 2026-07-02 17:01:55 |
| item_created | {"cover_hash":"a8322ca58fc844638d981351e84b297b","error":null,"hash":"a2c34264dc237381c60720e33d17ca38","phase":"item_import","target_ids":{"asset_id":"10408"},"type":"photo"} | 2026-07-02 17:01:53 |
| item_created | {"cover_hash":"cbe4f0bdb1814dbd97a08d26f4cecb39","error":null,"hash":"fb494d38260d362e714259dbd4756cbb","phase":"item_import","target_ids":{"asset_id":"10407"},"type":"photo"} | 2026-07-02 17:01:51 |
| item_created | {"cover_hash":"02aff9118c41447b9dfd2c01a223d3d9","error":null,"hash":"ffef1b7584434f24188389e029080e65","phase":"item_import","target_ids":{"asset_id":"10406"},"type":"photo"} | 2026-07-02 17:01:49 |
| item_created | {"cover_hash":"4397652a58e747a8aa69125527350e66","error":null,"hash":"d10118cac6ab64b3f3546068e72e2b8e","phase":"item_import","target_ids":{"asset_id":"10405"},"type":"photo"} | 2026-07-02 17:01:47 |
| item_created | {"cover_hash":"90a0dbdeddc14d838dc94a04af6ff9bf","error":null,"hash":"d3f5e84f38d3b04d3220ec4e204d8d09","phase":"item_import","target_ids":{"asset_id":"10404"},"type":"photo"} | 2026-07-02 17:01:45 |
| item_created | {"cover_hash":"74a22879f828427cb2100b96dc7d98a7","error":null,"hash":"6c937b120b852b7032e0db21b37b1d9a","phase":"item_import","target_ids":{"asset_id":"10403"},"type":"photo"} | 2026-07-02 17:01:43 |
| item_created | {"cover_hash":"130a4ef8c63b4c889cd14defc7249553","error":null,"hash":"1fb434e3f645a0f7a6b044159aab0436","phase":"item_import","target_ids":{"asset_id":"10402"},"type":"photo"} | 2026-07-02 17:01:41 |
| item_created | {"cover_hash":"0f93f30c61e54e31a6c7e98ffe244e04","error":null,"hash":"8458ae2990400d6ea37c4dd1ad3a6213","phase":"item_import","target_ids":{"asset_id":"10401"},"type":"photo"} | 2026-07-02 17:01:39 |
| item_created | {"cover_hash":"386c11a1982e41adba1486e4905e411c","error":null,"hash":"b64c05c8958ac162f8870ecc2a20e9c9","phase":"item_import","target_ids":{"asset_id":"10400"},"type":"photo"} | 2026-07-02 17:01:37 |
| item_created | {"cover_hash":"b39bdd15b5a744a59e2750a52d37bdc0","error":null,"hash":"4fcfd1eae766c6a99f51358b66e6edd2","phase":"item_import","target_ids":{"asset_id":"10399"},"type":"photo"} | 2026-07-02 17:01:35 |
| item_created | {"cover_hash":"8803ca27781a4b538b649af4e96741ff","error":null,"hash":"1100c35554cb1ac3c300392867c1a13b","phase":"item_import","target_ids":{"asset_id":"10398"},"type":"photo"} | 2026-07-02 17:01:33 |
| item_created | {"cover_hash":"2346d77832ea4f51980b48078524c9b7","error":null,"hash":"e59d849fb19855a7f9d464d192f66017","phase":"item_import","target_ids":{"asset_id":"10397"},"type":"photo"} | 2026-07-02 17:01:30 |
| item_created | {"cover_hash":"bdb7a10c694d4460a2b96c066bd8a8b3","error":null,"hash":"7dfe1aa27cdc09956423d5884987c374","phase":"item_import","target_ids":{"asset_id":"10396"},"type":"photo"} | 2026-07-02 17:01:28 |
| item_created | {"cover_hash":"3d1fd49fb1eb440aa3b1e8e0004b9916","error":null,"hash":"088c4ffb618e93f477031ec440ddbcdb","phase":"item_import","target_ids":{"asset_id":"10395"},"type":"photo"} | 2026-07-02 17:01:26 |
| item_created | {"cover_hash":"543fc8799ded469eab725205a0fc6ea0","error":null,"hash":"30fe9933e96a1665db00c03147feac58","phase":"item_import","target_ids":{"asset_id":"10394"},"type":"photo"} | 2026-07-02 17:01:24 |
| item_created | {"cover_hash":"d050ee3d13f94abb9282c4f098c23b66","error":null,"hash":"523dd7b2e8c83660efe404bce7ff9139","phase":"item_import","target_ids":{"asset_id":"10393"},"type":"photo"} | 2026-07-02 17:01:22 |
| item_created | {"cover_hash":"8450701a0842414c8d0537b4a860a783","error":null,"hash":"d1eff59857259cf884ee30a5a2f5fa6b","phase":"item_import","target_ids":{"asset_id":"10392"},"type":"photo"} | 2026-07-02 17:01:20 |
| item_created | {"cover_hash":"f133a16ba2734af387bddd8d49443ca8","error":null,"hash":"dc3696cf4d25f1ce59e1a8306de00162","phase":"item_import","target_ids":{"asset_id":"10391"},"type":"photo"} | 2026-07-02 17:01:17 |
| item_created | {"cover_hash":"ec186dc5786c45beb6d652c99045ed52","error":null,"hash":"53327024c6890bbb0609c8a5ff67c27b","phase":"item_import","target_ids":{"asset_id":"10390"},"type":"photo"} | 2026-07-02 17:01:15 |
| item_created | {"cover_hash":null,"error":null,"hash":"19248ca632512860419e079f90442851","phase":"item_import","target_ids":{"asset_id":"10389"},"type":"reel"} | 2026-07-02 17:01:13 |
| item_created | {"cover_hash":null,"error":null,"hash":"500ceb5e964c9ceb55783095cb32d09a","phase":"item_import","target_ids":{"asset_id":"10388"},"type":"reel"} | 2026-07-02 17:01:12 |
| item_created | {"cover_hash":null,"error":null,"hash":"613233915cfab6a8abf9e0475903fcf6","phase":"item_import","target_ids":{"asset_id":"10387"},"type":"reel"} | 2026-07-02 17:01:10 |
| item_created | {"cover_hash":null,"error":null,"hash":"52d7a6c0faef1517934ea515e3dd9ff1","phase":"item_import","target_ids":{"asset_id":"10386"},"type":"reel"} | 2026-07-02 17:01:08 |
| item_created | {"cover_hash":null,"error":null,"hash":"f2d5795142db81adb0a8ee508e2ebb1f","phase":"item_import","target_ids":{"asset_id":"10385"},"type":"reel"} | 2026-07-02 17:01:06 |
| item_created | {"cover_hash":null,"error":null,"hash":"9f1a0ed15c66ef37559aeeb9dfe07613","phase":"item_import","target_ids":{"asset_id":"10384"},"type":"reel"} | 2026-07-02 17:01:04 |
| item_created | {"cover_hash":null,"error":null,"hash":"880af4a0b8089d251641b1aeb0df7137","phase":"item_import","target_ids":{"asset_id":"10383"},"type":"reel"} | 2026-07-02 17:01:03 |
| item_created | {"cover_hash":null,"error":null,"hash":"6ecf33387d930bd646e2cff90ce05aba","phase":"item_import","target_ids":{"asset_id":"10382"},"type":"reel"} | 2026-07-02 17:01:01 |
| item_created | {"cover_hash":null,"error":null,"hash":"720c9de52e7949fad0ba4fa5b4d81f8f","phase":"item_import","target_ids":{"asset_id":"10381"},"type":"reel"} | 2026-07-02 17:00:59 |
| item_created | {"cover_hash":null,"error":null,"hash":"6c63a81da30ad5aa579884db9182b792","phase":"item_import","target_ids":{"asset_id":"10380"},"type":"reel"} | 2026-07-02 17:00:58 |
| item_created | {"cover_hash":null,"error":null,"hash":"352f84c2b21ee3be512b1e9506cf3822","phase":"item_import","target_ids":{"asset_id":"10379"},"type":"reel"} | 2026-07-02 17:00:56 |
| item_created | {"cover_hash":null,"error":null,"hash":"00ab0bc84c12f22d81c40e7abae30608","phase":"item_import","target_ids":{"asset_id":"10378"},"type":"reel"} | 2026-07-02 17:00:54 |
| item_created | {"cover_hash":"45317e37870c4bb7b73f1f829bcf517c","error":null,"hash":"d599e25f824e1d67c8ba59668d14a804","phase":"item_import","target_ids":{"asset_id":"10377"},"type":"reel"} | 2026-07-02 17:00:52 |
| item_created | {"cover_hash":null,"error":null,"hash":"a7e545cac88b9246039b5266c0e66a46","phase":"item_import","target_ids":{"asset_id":"10376"},"type":"reel"} | 2026-07-02 17:00:50 |
| item_created | {"cover_hash":"7f48afaaa943456cad41051319992f3e","error":null,"hash":"97a21488df1bcca213c8767072af4d11","phase":"item_import","target_ids":{"asset_id":"10375"},"type":"video"} | 2026-07-02 17:00:48 |
| import_started | {"action":"import","already_terminal_count":0,"container_count":2,"item_count":39,"manifest_sha256":"f9518ef9eb0ab1166ad9115ca8bcf14b4c79e55788fb0224b822a482e53be88b","max_items":0,"phase":"import"} | 2026-07-02 17:00:41 |
| job_claimed | {"job_id":"76","phase":"job_claim","priority":100,"worker_id":"vultr/pid:3795060"} | 2026-07-02 17:00:41 |
| job_queued | {"job_id": "76", "requested_by": "import-feed", "items_total": "39"} | 2026-07-02 17:00:37 |
| preflight_finished | {"counts":{"will_create":39,"will_fail":0,"will_link":0,"will_skip":0,"will_update":0},"failure_samples":[],"item_count":39,"item_type_counts":{"photo":24,"reel":14,"video":1},"ok":1,"phase":"prefligh... | 2026-07-02 17:00:37 |
| preflight_started | {"action":"import","container_count":2,"item_count":39,"item_type_counts":{"photo":24,"reel":14,"video":1},"manifest_sha256":"f9518ef9eb0ab1166ad9115ca8bcf14b4c79e55788fb0224b822a482e53be88b","phase":... | 2026-07-02 17:00:32 |
| scan | {"source": "import-feed", "manifest_sha256": "f9518ef9eb0ab1166ad9115ca8bcf14b4c79e55788fb0224b822a482e53be88b", "sync_type": "asset", "action": "import", "item_count": "39"} | 2026-07-02 17:00:32 |