Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch c726a848-3a65-4294-9f46-838dbe7a57da

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

58

Bytes

6.18 GB

Source host

orbit-2

Manifest sha256

ed46cd8cd5a1b290...

Started

2026-07-02 15:21:40

Completed

2026-07-02 15:44:59

Manifest summary

  • created_at: 2026-06-27T08:35:39Z
  • completed_at: 2026-06-27T08:35:39Z
  • verification.passed: true
  • verification.total_files: 9329
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
74doneimport-feed2026-07-02 15:22:122026-07-02 15:22:17vultr/pid:379506058/582026-07-02 15:44:59-

Items 58

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video7e3c15781e0aaf81...45969574e7f34c57...createdasset:10272-2026-07-02 15:23:12
reelbf2378f4e44a3261...-createdasset:10273-2026-07-02 15:23:28
reel87bbf282f673225b...-createdasset:10274-2026-07-02 15:23:41
reel0ed838bf6b15d138...-createdasset:10275-2026-07-02 15:24:05
reel1be97e5953c9b446...-createdasset:10276-2026-07-02 15:24:30
reeldaaafe9a650ac0ec...-createdasset:10277-2026-07-02 15:24:53
reelf43fe8caeeb42a19...-createdasset:10278-2026-07-02 15:25:19
reelbe561801ea265be1...-createdasset:10279-2026-07-02 15:25:31
reel8f4ab029ccb41713...-createdasset:10280-2026-07-02 15:25:46
reel634d1ba0e0e59f54...-createdasset:10281-2026-07-02 15:26:13
reel5628983b9211af93...b64756dc12da4284...createdasset:10282-2026-07-02 15:26:34
reelac05be1c038d0ab3...559b884169d14adb...createdasset:10283-2026-07-02 15:27:01
reelf480758f7830f746...50a1d410ee0f4eb6...createdasset:10284-2026-07-02 15:28:20
reel79e0c0014ced2a70...-createdasset:10285-2026-07-02 15:28:57
reel465fb112a2fdee41...-createdasset:10286-2026-07-02 15:29:37
reel9b8a3f3fab42da0b...ddea6954524c4a34...createdasset:10287-2026-07-02 15:30:05
reel47086b6f0eb2dfd9...7ccb899f30ff487b...createdasset:10288-2026-07-02 15:30:21
reel7fc56d05c7d25d16...1bc69207f7584c08...createdasset:10289-2026-07-02 15:30:47
reel77b7bb0f6007e892...a18ff065f31044a8...createdasset:10290-2026-07-02 15:31:06
reelae4d3808dc154f36...-createdasset:10291-2026-07-02 15:31:21
reel6d6e728ced9fe93f...9f5eb656b9c340a3...createdasset:10292-2026-07-02 15:31:51
reele1ac867d0df7cb0a...-createdasset:10293-2026-07-02 15:32:09
reel1564a445bb16c2d8...-createdasset:10294-2026-07-02 15:32:42
reela9a536c017a6b710...3c702274dec84552...createdasset:10295-2026-07-02 15:33:03
reelff81454e6b6d4cba...ac3a972b49944b6e...createdasset:10296-2026-07-02 15:33:30
reel992e062fbf37f5e7...4fe7fcb13b0d4d2f...createdasset:10297-2026-07-02 15:33:46
reela0f33ad0cdf61c54...1dca5e158d8941a7...createdasset:10298-2026-07-02 15:34:04
reelf2b3aa87278100f6...bb5f2f97b5694c6c...createdasset:10299-2026-07-02 15:34:38
reelf25c0454e8a466f2...2b08baa194114629...createdasset:10300-2026-07-02 15:34:55
reelb157f031e4beb3e9...3389a5ba86604bd4...createdasset:10301-2026-07-02 15:35:14
reele434aa20801114b6...4db6535bce9f40a8...createdasset:10302-2026-07-02 15:35:48
reeleddc94b84c7015ad...5350e75e2b634f26...createdasset:10303-2026-07-02 15:36:39
reel83ee210fca4afb22...-createdasset:10304-2026-07-02 15:37:23
reel0b591879825aff6b...-createdasset:10305-2026-07-02 15:37:40
reelbcf859ec13226e98...-createdasset:10306-2026-07-02 15:37:51
photo428cac71e60a4777...2989a69b39b64a6d...createdasset:10307-2026-07-02 15:38:09
photo0974778a66c0a19e...95fe414572454cce...createdasset:10308-2026-07-02 15:38:23
photoe4f3ebba523d0d4d...a0c1a311dff545b5...createdasset:10309-2026-07-02 15:38:38
photo81ea093141bffdbd...c2f4ac8dc18c4bb7...createdasset:10310-2026-07-02 15:38:44
photob61e383236e2b671...9c25dc656aca4af5...createdasset:10311-2026-07-02 15:38:53
photo70ecaec40a20ce9a...82c61ba6cca34580...createdasset:10312-2026-07-02 15:39:12
photo52b833cf40fb6899...dfc9f8bc78794efc...createdasset:10313-2026-07-02 15:39:32
photo8364a24746205fa7...768cdaab12694ed1...createdasset:10314-2026-07-02 15:39:44
photo5fa5cb1c45c3d729...344867874a814cbb...createdasset:10315-2026-07-02 15:39:59
photoe6098301303f569a...78c187d90e5747c9...createdasset:10316-2026-07-02 15:40:08
photodeec83ae9d97b4cd...2db61f7da5e14e09...createdasset:10317-2026-07-02 15:40:52
photo2d213f396f9eff95...0e640f483a974810...createdasset:10318-2026-07-02 15:41:02
photo6a2b217cd89820ea...1c9ae353afa2467c...createdasset:10319-2026-07-02 15:41:24
photo8e829e4c40d824df...b50fa5b9d1dc4932...createdasset:10320-2026-07-02 15:41:51
photobeafdea5f2d162cf...9c4b52f90bb148fa...createdasset:10321-2026-07-02 15:42:09
photoa81d3097dddd7425...2f02668495284151...createdasset:10322-2026-07-02 15:42:46
photob5e631cd7064d7c2...3577a4343b6c448c...createdasset:10323-2026-07-02 15:42:51
photob8619b66bd425d1f...dbb947837e314315...createdasset:10324-2026-07-02 15:42:57
photo0e5e9a5a9bc06e33...a4505d5bad8b4eaf...createdasset:10325-2026-07-02 15:43:39
photode9f5ecd0927a093...4d34799d06614822...createdasset:10326-2026-07-02 15:43:53
photod0ea6354a89acf3d...3e57b62c237c4e46...createdasset:10327-2026-07-02 15:44:24
photof3e57daa8c1374f7...c3a7dcd9a71d4f3e...createdasset:10328-2026-07-02 15:44:40
photo14ecd251f6647e70...729b1b001f8f415b...createdasset:10329-2026-07-02 15:44:59

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"74","phase":"job_finish","summary":{"items_seen":58,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":58,"deferred":0,"failed":0...2026-07-02 15:44:59
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":58,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-07-02 15:44:59
item_updated{"container":1,"expected_member_count":23,"gallery_hash":"bd33e2b9eea97b4522ce8296053be837","gallery_id":"60","linked_member_count":23,"missing_member_count":0,"phase":"container_link","title":"countr...2026-07-02 15:44:59
item_created{"cover_hash":"729b1b001f8f415bb98968cc840221e7","error":null,"hash":"14ecd251f6647e705946083c1c865cd1","phase":"item_import","target_ids":{"asset_id":"10329"},"type":"photo"}2026-07-02 15:44:59
item_created{"cover_hash":"c3a7dcd9a71d4f3e82671af61f8415af","error":null,"hash":"f3e57daa8c1374f7a7640bb19292b27e","phase":"item_import","target_ids":{"asset_id":"10328"},"type":"photo"}2026-07-02 15:44:40
item_created{"cover_hash":"3e57b62c237c4e468261d3ce4f86b110","error":null,"hash":"d0ea6354a89acf3dbfd897c78c6ce1f8","phase":"item_import","target_ids":{"asset_id":"10327"},"type":"photo"}2026-07-02 15:44:24
item_created{"cover_hash":"4d34799d06614822844fd2d9073959c7","error":null,"hash":"de9f5ecd0927a0930bfa68a5acc0a579","phase":"item_import","target_ids":{"asset_id":"10326"},"type":"photo"}2026-07-02 15:43:53
item_created{"cover_hash":"a4505d5bad8b4eafb71695024aff6d76","error":null,"hash":"0e5e9a5a9bc06e33002120081664868d","phase":"item_import","target_ids":{"asset_id":"10325"},"type":"photo"}2026-07-02 15:43:39
item_created{"cover_hash":"dbb947837e314315bde2c40292eb7c22","error":null,"hash":"b8619b66bd425d1fcf9bd67097583869","phase":"item_import","target_ids":{"asset_id":"10324"},"type":"photo"}2026-07-02 15:42:57
item_created{"cover_hash":"3577a4343b6c448ca0767a08190b174a","error":null,"hash":"b5e631cd7064d7c2a707e8cbf77f95a0","phase":"item_import","target_ids":{"asset_id":"10323"},"type":"photo"}2026-07-02 15:42:51
item_created{"cover_hash":"2f02668495284151b9ef3e15756fe266","error":null,"hash":"a81d3097dddd7425e7740ea1a5e0c275","phase":"item_import","target_ids":{"asset_id":"10322"},"type":"photo"}2026-07-02 15:42:46
item_created{"cover_hash":"9c4b52f90bb148faba3a21192a3f28bd","error":null,"hash":"beafdea5f2d162cf6bab362c1564b05a","phase":"item_import","target_ids":{"asset_id":"10321"},"type":"photo"}2026-07-02 15:42:09
item_created{"cover_hash":"b50fa5b9d1dc49329dbb7ca91f103a98","error":null,"hash":"8e829e4c40d824df46a75061dcc071ea","phase":"item_import","target_ids":{"asset_id":"10320"},"type":"photo"}2026-07-02 15:41:51
item_created{"cover_hash":"1c9ae353afa2467c9fb8f12991f01e72","error":null,"hash":"6a2b217cd89820eae986f32da0f82299","phase":"item_import","target_ids":{"asset_id":"10319"},"type":"photo"}2026-07-02 15:41:24
item_created{"cover_hash":"0e640f483a974810ba0b579cf1c3e7a3","error":null,"hash":"2d213f396f9eff95e5b5b1e4a9649a00","phase":"item_import","target_ids":{"asset_id":"10318"},"type":"photo"}2026-07-02 15:41:02
item_created{"cover_hash":"2db61f7da5e14e09a04e6f7d61425572","error":null,"hash":"deec83ae9d97b4cd266853a955465875","phase":"item_import","target_ids":{"asset_id":"10317"},"type":"photo"}2026-07-02 15:40:52
item_created{"cover_hash":"78c187d90e5747c9bfe984605ecb6175","error":null,"hash":"e6098301303f569ad6509bcfd796b693","phase":"item_import","target_ids":{"asset_id":"10316"},"type":"photo"}2026-07-02 15:40:08
item_created{"cover_hash":"344867874a814cbb89bcfa4c166e2209","error":null,"hash":"5fa5cb1c45c3d729304a1166b25fdfc4","phase":"item_import","target_ids":{"asset_id":"10315"},"type":"photo"}2026-07-02 15:39:59
item_created{"cover_hash":"768cdaab12694ed191f0f7f34ec247f0","error":null,"hash":"8364a24746205fa755309f3073ebe434","phase":"item_import","target_ids":{"asset_id":"10314"},"type":"photo"}2026-07-02 15:39:44
item_created{"cover_hash":"dfc9f8bc78794efc80b2b5c9e9efde10","error":null,"hash":"52b833cf40fb68998efbed71a5740c0a","phase":"item_import","target_ids":{"asset_id":"10313"},"type":"photo"}2026-07-02 15:39:32
item_created{"cover_hash":"82c61ba6cca34580bccddeeabe351395","error":null,"hash":"70ecaec40a20ce9ab38b25087738c177","phase":"item_import","target_ids":{"asset_id":"10312"},"type":"photo"}2026-07-02 15:39:12
item_created{"cover_hash":"9c25dc656aca4af58d3eeecc265e2569","error":null,"hash":"b61e383236e2b67184da8b2428e79488","phase":"item_import","target_ids":{"asset_id":"10311"},"type":"photo"}2026-07-02 15:38:53
item_created{"cover_hash":"c2f4ac8dc18c4bb7a73b255d6cf8f83a","error":null,"hash":"81ea093141bffdbd9a10b537b3fb7535","phase":"item_import","target_ids":{"asset_id":"10310"},"type":"photo"}2026-07-02 15:38:44
item_created{"cover_hash":"a0c1a311dff545b5a9c075381014167d","error":null,"hash":"e4f3ebba523d0d4d9e9421afa28ef26e","phase":"item_import","target_ids":{"asset_id":"10309"},"type":"photo"}2026-07-02 15:38:38
item_created{"cover_hash":"95fe414572454ccead54000705bd7bf3","error":null,"hash":"0974778a66c0a19e568905505fb5b36d","phase":"item_import","target_ids":{"asset_id":"10308"},"type":"photo"}2026-07-02 15:38:23
item_created{"cover_hash":"2989a69b39b64a6d829f4e29da999413","error":null,"hash":"428cac71e60a4777db50ff5a50fa488f","phase":"item_import","target_ids":{"asset_id":"10307"},"type":"photo"}2026-07-02 15:38:09
item_created{"cover_hash":null,"error":null,"hash":"bcf859ec13226e98161f93763588e56e","phase":"item_import","target_ids":{"asset_id":"10306"},"type":"reel"}2026-07-02 15:37:51
item_created{"cover_hash":null,"error":null,"hash":"0b591879825aff6bbb9363c01e3bd57f","phase":"item_import","target_ids":{"asset_id":"10305"},"type":"reel"}2026-07-02 15:37:40
item_created{"cover_hash":null,"error":null,"hash":"83ee210fca4afb226c3f0a904333ac11","phase":"item_import","target_ids":{"asset_id":"10304"},"type":"reel"}2026-07-02 15:37:23
item_created{"cover_hash":"5350e75e2b634f26b7bc2bf7ebf4f5be","error":null,"hash":"eddc94b84c7015ad55d7cef35efa2645","phase":"item_import","target_ids":{"asset_id":"10303"},"type":"reel"}2026-07-02 15:36:39
item_created{"cover_hash":"4db6535bce9f40a8a14d3a6302a9306d","error":null,"hash":"e434aa20801114b6f586e82766e4c4ad","phase":"item_import","target_ids":{"asset_id":"10302"},"type":"reel"}2026-07-02 15:35:48
item_created{"cover_hash":"3389a5ba86604bd49e051cd488379f80","error":null,"hash":"b157f031e4beb3e99510e693bcd9b7c1","phase":"item_import","target_ids":{"asset_id":"10301"},"type":"reel"}2026-07-02 15:35:14
item_created{"cover_hash":"2b08baa194114629992b15960707687e","error":null,"hash":"f25c0454e8a466f28e3df4ffec7b701b","phase":"item_import","target_ids":{"asset_id":"10300"},"type":"reel"}2026-07-02 15:34:55
item_created{"cover_hash":"bb5f2f97b5694c6cb885b20234fbbeb1","error":null,"hash":"f2b3aa87278100f6319db4ecf661daa4","phase":"item_import","target_ids":{"asset_id":"10299"},"type":"reel"}2026-07-02 15:34:38
item_created{"cover_hash":"1dca5e158d8941a7af11a0a3370c0406","error":null,"hash":"a0f33ad0cdf61c549c983a0da119f689","phase":"item_import","target_ids":{"asset_id":"10298"},"type":"reel"}2026-07-02 15:34:04
item_created{"cover_hash":"4fe7fcb13b0d4d2f870973e9c78b6cf2","error":null,"hash":"992e062fbf37f5e76d561004c84336a5","phase":"item_import","target_ids":{"asset_id":"10297"},"type":"reel"}2026-07-02 15:33:46
item_created{"cover_hash":"ac3a972b49944b6e951d6e4f38b0b625","error":null,"hash":"ff81454e6b6d4cbadff04607a9965a92","phase":"item_import","target_ids":{"asset_id":"10296"},"type":"reel"}2026-07-02 15:33:30
item_created{"cover_hash":"3c702274dec84552b951ea6e99b8315e","error":null,"hash":"a9a536c017a6b7108feb6593c570f321","phase":"item_import","target_ids":{"asset_id":"10295"},"type":"reel"}2026-07-02 15:33:03
item_created{"cover_hash":null,"error":null,"hash":"1564a445bb16c2d8e6c40d537eaf4a1c","phase":"item_import","target_ids":{"asset_id":"10294"},"type":"reel"}2026-07-02 15:32:42
item_created{"cover_hash":null,"error":null,"hash":"e1ac867d0df7cb0a79ae2b2c7fb9ead7","phase":"item_import","target_ids":{"asset_id":"10293"},"type":"reel"}2026-07-02 15:32:09
item_created{"cover_hash":"9f5eb656b9c340a392b1f6d707cd2bf3","error":null,"hash":"6d6e728ced9fe93f7f02e68086491bff","phase":"item_import","target_ids":{"asset_id":"10292"},"type":"reel"}2026-07-02 15:31:51
item_created{"cover_hash":null,"error":null,"hash":"ae4d3808dc154f362d491a112878b254","phase":"item_import","target_ids":{"asset_id":"10291"},"type":"reel"}2026-07-02 15:31:21
item_created{"cover_hash":"a18ff065f31044a8b82fef5941df3ce0","error":null,"hash":"77b7bb0f6007e8925678e1045dc1e3bf","phase":"item_import","target_ids":{"asset_id":"10290"},"type":"reel"}2026-07-02 15:31:06
item_created{"cover_hash":"1bc69207f7584c08a06a358223793e5a","error":null,"hash":"7fc56d05c7d25d16f69c3ee0b0c33d62","phase":"item_import","target_ids":{"asset_id":"10289"},"type":"reel"}2026-07-02 15:30:47
item_created{"cover_hash":"7ccb899f30ff487babd4e77795ce22de","error":null,"hash":"47086b6f0eb2dfd95a907ba387892693","phase":"item_import","target_ids":{"asset_id":"10288"},"type":"reel"}2026-07-02 15:30:21
item_created{"cover_hash":"ddea6954524c4a34800d1acec307c002","error":null,"hash":"9b8a3f3fab42da0b3da33f89232738a6","phase":"item_import","target_ids":{"asset_id":"10287"},"type":"reel"}2026-07-02 15:30:05
item_created{"cover_hash":null,"error":null,"hash":"465fb112a2fdee41fcaf499a02d6d6be","phase":"item_import","target_ids":{"asset_id":"10286"},"type":"reel"}2026-07-02 15:29:37
item_created{"cover_hash":null,"error":null,"hash":"79e0c0014ced2a7000298a5cde82cb57","phase":"item_import","target_ids":{"asset_id":"10285"},"type":"reel"}2026-07-02 15:28:57
item_created{"cover_hash":"50a1d410ee0f4eb68ce9921c1d4c1339","error":null,"hash":"f480758f7830f746c18a6e7db5d34560","phase":"item_import","target_ids":{"asset_id":"10284"},"type":"reel"}2026-07-02 15:28:20
item_created{"cover_hash":"559b884169d14adb8cb8b5cf417ea4ca","error":null,"hash":"ac05be1c038d0ab350fc6d2151ff9543","phase":"item_import","target_ids":{"asset_id":"10283"},"type":"reel"}2026-07-02 15:27:01