Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 321d2e4b-df84-4150-9698-c1eaae52d528

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

50

Bytes

4.76 GB

Source host

orbit-2

Manifest sha256

aa1fda2fb8261251...

Started

2026-07-02 17:55:27

Completed

2026-07-02 17:58:03

Manifest summary

  • created_at: 2026-07-02T17:54:37Z
  • completed_at: 2026-07-02T17:54:37Z
  • verification.passed: true
  • verification.total_files: 9515
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
77doneimport-feed2026-07-02 17:55:322026-07-02 17:55:35vultr/pid:379506050/502026-07-02 17:58:03-

Items 50

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video78f9a4c00c00eccc...c5eabc072fa84e54...createdasset:10414-2026-07-02 17:55:43
reel0ae0fdfc66af604f...ec0ad896e2794bd0...createdasset:10415-2026-07-02 17:55:46
reel7efd7ff91817bfba...58fbce63ec2e42c0...createdasset:10416-2026-07-02 17:55:50
reel45985a1b61a7bca5...5895558c16424ad1...createdasset:10417-2026-07-02 17:55:53
reelf375afe94893d9c5...a1e07af3fbec4d2b...createdasset:10418-2026-07-02 17:55:57
reel871c03a9a8c82a7d...77a136f76d8f429e...createdasset:10419-2026-07-02 17:56:01
reelf867f3b3126a053e...610da469994a4cf8...createdasset:10420-2026-07-02 17:56:04
reelbd7d4213ef18bfa3...75b09164450c41d1...createdasset:10421-2026-07-02 17:56:08
reel5063cd208224c33b...216b9703c2a94372...createdasset:10422-2026-07-02 17:56:12
reel02f151ff301f28e4...08b1f3beeedb4e3c...createdasset:10423-2026-07-02 17:56:15
reel67bf94b88eff5e25...e182d334f14a407f...createdasset:10424-2026-07-02 17:56:19
reelacd10f0c87291e52...49c7655fff0f4e44...createdasset:10425-2026-07-02 17:56:22
reel65d41c1aeeb04b2b...2e91a20181764dc3...createdasset:10426-2026-07-02 17:56:26
reela93978709b871e16...dc03e3f8a0664845...createdasset:10427-2026-07-02 17:56:30
reelab0df869f1b3c0cf...9dddc95bd8954f52...createdasset:10428-2026-07-02 17:56:33
reel7256c898f6bee8ad...688cbac18cd1475e...createdasset:10429-2026-07-02 17:56:37
reela5065f0328fd0efb...08f5190889254dd8...createdasset:10430-2026-07-02 17:56:40
reel8df05a6170d07b01...8a855f406270454b...createdasset:10431-2026-07-02 17:56:44
reeled360cb3ee4f6c34...13b83d8292b74dc0...createdasset:10432-2026-07-02 17:56:48
reel5ee15eb19d2b1e3b...45782eb4dc824055...createdasset:10433-2026-07-02 17:56:52
reel3baa65a109c6fb60...8d9a2a81b2ce4d82...createdasset:10434-2026-07-02 17:56:55
reel5693c97b73d79807...2c9b1b73208143a2...createdasset:10435-2026-07-02 17:56:59
reel1e373b62fc50b3db...499b3194280a4568...createdasset:10436-2026-07-02 17:57:03
reeldd45ad485e0c8938...b33ce6a87f314334...createdasset:10437-2026-07-02 17:57:07
reel7a8f9e04ba9575cb...df827a318af34752...createdasset:10438-2026-07-02 17:57:10
reel74eb2d571a13fc4a...c74ffa3a88db4b74...createdasset:10439-2026-07-02 17:57:14
photoc7be37eecc9972d5...a23b4abb92bd4b4d...createdasset:10440-2026-07-02 17:57:16
photo9af54986888a2344...d7333057de464a1e...createdasset:10441-2026-07-02 17:57:18
photo919dae4d44c88449...b422fd5ec34b421e...createdasset:10442-2026-07-02 17:57:20
photo0ab650d38b143564...d8bf953379fb430c...createdasset:10443-2026-07-02 17:57:22
photoe94b80d9b60f26e4...bf4537afae0749b4...createdasset:10444-2026-07-02 17:57:24
photo08c258f1a88ecfac...69f1b48bb024419d...createdasset:10445-2026-07-02 17:57:26
photo3f33955ec1ae87fd...2c448aa208304b36...createdasset:10446-2026-07-02 17:57:28
photo0e1297ee41f16816...f778c61906d94764...createdasset:10447-2026-07-02 17:57:30
photo9e383c3ccadfce22...16ecd7f660fa4b85...createdasset:10448-2026-07-02 17:57:32
photodefc34faa35a1c2a...d177673ab68a4b6c...createdasset:10449-2026-07-02 17:57:34
photo9118c74c5a4961cf...7de21148a46148c3...createdasset:10450-2026-07-02 17:57:36
photo6d00fb353084321a...9e57289cb1d3466f...createdasset:10451-2026-07-02 17:57:38
photoaa31c6d8fbc479ae...6ea00248e9034c82...createdasset:10452-2026-07-02 17:57:39
photo3a016ac960828983...7393e6b35c38458f...createdasset:10453-2026-07-02 17:57:42
photodf3b52c4e8f7ae3c...13d3a6ea08de4bd9...createdasset:10454-2026-07-02 17:57:44
photo32f4953052cf5a71...02f2100139054556...createdasset:10455-2026-07-02 17:57:46
photo669a8c92379eb67c...b73e64014f4146f3...createdasset:10456-2026-07-02 17:57:48
photo4a6da2236923ea63...28f575daff9941a0...createdasset:10457-2026-07-02 17:57:50
photo3f802223f23c6cc6...c713cb940d594667...createdasset:10458-2026-07-02 17:57:52
photoe57196d60e28602e...f603e654c6ea4c26...createdasset:10459-2026-07-02 17:57:54
photo502d16aec464ba69...1cdbda1fcf784534...createdasset:10460-2026-07-02 17:57:57
photofead41f3644f14f8...91643b60f06e45c8...createdasset:10461-2026-07-02 17:57:58
photo4f59dfd7106c4623...0f7e3501deb84754...createdasset:10462-2026-07-02 17:58:01
photo82130a65609bbe45...b34d7fe656b3430b...createdasset:10463-2026-07-02 17:58:03

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"77","phase":"job_finish","summary":{"items_seen":50,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":50,"deferred":0,"failed":0...2026-07-02 17:58:03
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":50,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-07-02 17:58:03
item_updated{"container":1,"expected_member_count":24,"gallery_hash":"3299189e9d6d77cd885fccf8f625e66a","gallery_id":"63","linked_member_count":24,"missing_member_count":0,"phase":"container_link","title":"countr...2026-07-02 17:58:03
item_created{"cover_hash":"b34d7fe656b3430b8c8836aaf8deff34","error":null,"hash":"82130a65609bbe450cd3b6139d7306ea","phase":"item_import","target_ids":{"asset_id":"10463"},"type":"photo"}2026-07-02 17:58:03
item_created{"cover_hash":"0f7e3501deb8475480609b796a13ff0b","error":null,"hash":"4f59dfd7106c46237bd8901d06a9c778","phase":"item_import","target_ids":{"asset_id":"10462"},"type":"photo"}2026-07-02 17:58:01
item_created{"cover_hash":"91643b60f06e45c88b95a8ff5923c970","error":null,"hash":"fead41f3644f14f8d0720f815612c9b9","phase":"item_import","target_ids":{"asset_id":"10461"},"type":"photo"}2026-07-02 17:57:58
item_created{"cover_hash":"1cdbda1fcf78453485e035858c85e6c8","error":null,"hash":"502d16aec464ba6994df57f69383050e","phase":"item_import","target_ids":{"asset_id":"10460"},"type":"photo"}2026-07-02 17:57:57
item_created{"cover_hash":"f603e654c6ea4c26b50c0e0f81c342c6","error":null,"hash":"e57196d60e28602eee8cd11684a79435","phase":"item_import","target_ids":{"asset_id":"10459"},"type":"photo"}2026-07-02 17:57:54
item_created{"cover_hash":"c713cb940d59466784330502f0fc55f9","error":null,"hash":"3f802223f23c6cc63021f71aa13a7f5b","phase":"item_import","target_ids":{"asset_id":"10458"},"type":"photo"}2026-07-02 17:57:52
item_created{"cover_hash":"28f575daff9941a089f17831465049c2","error":null,"hash":"4a6da2236923ea633a7bd52a2eea70dc","phase":"item_import","target_ids":{"asset_id":"10457"},"type":"photo"}2026-07-02 17:57:50
item_created{"cover_hash":"b73e64014f4146f385064fbc6bd3a8f8","error":null,"hash":"669a8c92379eb67c718763d3e8a17273","phase":"item_import","target_ids":{"asset_id":"10456"},"type":"photo"}2026-07-02 17:57:48
item_created{"cover_hash":"02f2100139054556a5b1739a7d4562cb","error":null,"hash":"32f4953052cf5a71583e125ec7bef89f","phase":"item_import","target_ids":{"asset_id":"10455"},"type":"photo"}2026-07-02 17:57:46
item_created{"cover_hash":"13d3a6ea08de4bd9a43bd4846b3941f7","error":null,"hash":"df3b52c4e8f7ae3cc6a47d0a5922df1d","phase":"item_import","target_ids":{"asset_id":"10454"},"type":"photo"}2026-07-02 17:57:44
item_created{"cover_hash":"7393e6b35c38458fa83b3d16e750276b","error":null,"hash":"3a016ac960828983d81cfcb217a58ff3","phase":"item_import","target_ids":{"asset_id":"10453"},"type":"photo"}2026-07-02 17:57:42
item_created{"cover_hash":"6ea00248e9034c828ee73cdbb3bff345","error":null,"hash":"aa31c6d8fbc479aef60159b39777a288","phase":"item_import","target_ids":{"asset_id":"10452"},"type":"photo"}2026-07-02 17:57:39
item_created{"cover_hash":"9e57289cb1d3466faed72a831b4baa7d","error":null,"hash":"6d00fb353084321a41733f18ad3a0b57","phase":"item_import","target_ids":{"asset_id":"10451"},"type":"photo"}2026-07-02 17:57:38
item_created{"cover_hash":"7de21148a46148c3938f53399605628d","error":null,"hash":"9118c74c5a4961cff8c24d04c1ca589c","phase":"item_import","target_ids":{"asset_id":"10450"},"type":"photo"}2026-07-02 17:57:36
item_created{"cover_hash":"d177673ab68a4b6c99bb5a2c0d5e30f3","error":null,"hash":"defc34faa35a1c2abf749b53bcb24c1b","phase":"item_import","target_ids":{"asset_id":"10449"},"type":"photo"}2026-07-02 17:57:34
item_created{"cover_hash":"16ecd7f660fa4b85b5aec6a99dac8c46","error":null,"hash":"9e383c3ccadfce22446b1bcef659aabe","phase":"item_import","target_ids":{"asset_id":"10448"},"type":"photo"}2026-07-02 17:57:32
item_created{"cover_hash":"f778c61906d947649b108fadeefd181e","error":null,"hash":"0e1297ee41f1681628257229377b24ba","phase":"item_import","target_ids":{"asset_id":"10447"},"type":"photo"}2026-07-02 17:57:30
item_created{"cover_hash":"2c448aa208304b36b1aad09a44290c52","error":null,"hash":"3f33955ec1ae87fd2b1b5fc354a562eb","phase":"item_import","target_ids":{"asset_id":"10446"},"type":"photo"}2026-07-02 17:57:28
item_created{"cover_hash":"69f1b48bb024419d87e5efcbcba46646","error":null,"hash":"08c258f1a88ecfac4a51331a86cdf494","phase":"item_import","target_ids":{"asset_id":"10445"},"type":"photo"}2026-07-02 17:57:26
item_created{"cover_hash":"bf4537afae0749b48ca823c355dad3f3","error":null,"hash":"e94b80d9b60f26e4852be689ac37f8eb","phase":"item_import","target_ids":{"asset_id":"10444"},"type":"photo"}2026-07-02 17:57:24
item_created{"cover_hash":"d8bf953379fb430c9a9dfc62de0508a1","error":null,"hash":"0ab650d38b1435649f8c45110f41916d","phase":"item_import","target_ids":{"asset_id":"10443"},"type":"photo"}2026-07-02 17:57:22
item_created{"cover_hash":"b422fd5ec34b421e8dadc9b28876111e","error":null,"hash":"919dae4d44c8844949c8a742e65efce0","phase":"item_import","target_ids":{"asset_id":"10442"},"type":"photo"}2026-07-02 17:57:20
item_created{"cover_hash":"d7333057de464a1eb0bfdd610a43eac4","error":null,"hash":"9af54986888a23449f823a2c37a331c1","phase":"item_import","target_ids":{"asset_id":"10441"},"type":"photo"}2026-07-02 17:57:18
item_created{"cover_hash":"a23b4abb92bd4b4dab628bb32c3463b7","error":null,"hash":"c7be37eecc9972d565073fbc9c662eac","phase":"item_import","target_ids":{"asset_id":"10440"},"type":"photo"}2026-07-02 17:57:16
item_created{"cover_hash":"c74ffa3a88db4b74b1a50fb8030c2d61","error":null,"hash":"74eb2d571a13fc4a4b2f8dbd54bbd9e5","phase":"item_import","target_ids":{"asset_id":"10439"},"type":"reel"}2026-07-02 17:57:14
item_created{"cover_hash":"df827a318af3475298c25b47ff3b924c","error":null,"hash":"7a8f9e04ba9575cb8a844cd446410510","phase":"item_import","target_ids":{"asset_id":"10438"},"type":"reel"}2026-07-02 17:57:10
item_created{"cover_hash":"b33ce6a87f3143349353bc928b406920","error":null,"hash":"dd45ad485e0c8938162588b2c04d7f9a","phase":"item_import","target_ids":{"asset_id":"10437"},"type":"reel"}2026-07-02 17:57:07
item_created{"cover_hash":"499b3194280a4568a3b67ab8644c3698","error":null,"hash":"1e373b62fc50b3db3d04228169e7ba81","phase":"item_import","target_ids":{"asset_id":"10436"},"type":"reel"}2026-07-02 17:57:03
item_created{"cover_hash":"2c9b1b73208143a29661312fb1d7de2c","error":null,"hash":"5693c97b73d7980737e24b4207538996","phase":"item_import","target_ids":{"asset_id":"10435"},"type":"reel"}2026-07-02 17:56:59
item_created{"cover_hash":"8d9a2a81b2ce4d828b1b2de21245d394","error":null,"hash":"3baa65a109c6fb601190f32909d051d3","phase":"item_import","target_ids":{"asset_id":"10434"},"type":"reel"}2026-07-02 17:56:55
item_created{"cover_hash":"45782eb4dc824055989c05cce29fdb07","error":null,"hash":"5ee15eb19d2b1e3b09e7d9fe2bd26466","phase":"item_import","target_ids":{"asset_id":"10433"},"type":"reel"}2026-07-02 17:56:52
item_created{"cover_hash":"13b83d8292b74dc0a637ecb8137f118c","error":null,"hash":"ed360cb3ee4f6c34beb85ba5f210e072","phase":"item_import","target_ids":{"asset_id":"10432"},"type":"reel"}2026-07-02 17:56:48
item_created{"cover_hash":"8a855f406270454b9732bad00a82668d","error":null,"hash":"8df05a6170d07b01b1d99dca4ff36dc8","phase":"item_import","target_ids":{"asset_id":"10431"},"type":"reel"}2026-07-02 17:56:44
item_created{"cover_hash":"08f5190889254dd8a7903125bac41f8e","error":null,"hash":"a5065f0328fd0efbedca6b4da1bb39e0","phase":"item_import","target_ids":{"asset_id":"10430"},"type":"reel"}2026-07-02 17:56:40
item_created{"cover_hash":"688cbac18cd1475e93efa4f17aabac8a","error":null,"hash":"7256c898f6bee8ad6d24cac698e250fb","phase":"item_import","target_ids":{"asset_id":"10429"},"type":"reel"}2026-07-02 17:56:37
item_created{"cover_hash":"9dddc95bd8954f52822a8c353dc39684","error":null,"hash":"ab0df869f1b3c0cfc0391d484c62a36a","phase":"item_import","target_ids":{"asset_id":"10428"},"type":"reel"}2026-07-02 17:56:33
item_created{"cover_hash":"dc03e3f8a0664845aee63a70505775e5","error":null,"hash":"a93978709b871e1663a644881f8cf93b","phase":"item_import","target_ids":{"asset_id":"10427"},"type":"reel"}2026-07-02 17:56:30
item_created{"cover_hash":"2e91a20181764dc3ad20680b791517bf","error":null,"hash":"65d41c1aeeb04b2b0de232aae0341d33","phase":"item_import","target_ids":{"asset_id":"10426"},"type":"reel"}2026-07-02 17:56:26
item_created{"cover_hash":"49c7655fff0f4e44bc97f4ea0e005fbc","error":null,"hash":"acd10f0c87291e52f4342b5ec8960e60","phase":"item_import","target_ids":{"asset_id":"10425"},"type":"reel"}2026-07-02 17:56:22
item_created{"cover_hash":"e182d334f14a407f9cfe14fdaab157d7","error":null,"hash":"67bf94b88eff5e252dd0c424b969bfe4","phase":"item_import","target_ids":{"asset_id":"10424"},"type":"reel"}2026-07-02 17:56:19
item_created{"cover_hash":"08b1f3beeedb4e3cbddbb787e9b3838a","error":null,"hash":"02f151ff301f28e4fa42b239c41d1d94","phase":"item_import","target_ids":{"asset_id":"10423"},"type":"reel"}2026-07-02 17:56:15
item_created{"cover_hash":"216b9703c2a943728ea9a12e84c47b33","error":null,"hash":"5063cd208224c33be5585495eaaeb074","phase":"item_import","target_ids":{"asset_id":"10422"},"type":"reel"}2026-07-02 17:56:12
item_created{"cover_hash":"75b09164450c41d19818f66c141a6372","error":null,"hash":"bd7d4213ef18bfa3bb095fa9388413ad","phase":"item_import","target_ids":{"asset_id":"10421"},"type":"reel"}2026-07-02 17:56:08
item_created{"cover_hash":"610da469994a4cf89f430792ada95f36","error":null,"hash":"f867f3b3126a053e71c7543c1fab7aca","phase":"item_import","target_ids":{"asset_id":"10420"},"type":"reel"}2026-07-02 17:56:04
item_created{"cover_hash":"77a136f76d8f429e8675cced91048a33","error":null,"hash":"871c03a9a8c82a7db1de2f929d1b493e","phase":"item_import","target_ids":{"asset_id":"10419"},"type":"reel"}2026-07-02 17:56:01
item_created{"cover_hash":"a1e07af3fbec4d2b848653690d8c729c","error":null,"hash":"f375afe94893d9c51f8be8394596be0b","phase":"item_import","target_ids":{"asset_id":"10418"},"type":"reel"}2026-07-02 17:55:57
item_created{"cover_hash":"5895558c16424ad181679e60ca530370","error":null,"hash":"45985a1b61a7bca5b06a605d4fa28031","phase":"item_import","target_ids":{"asset_id":"10417"},"type":"reel"}2026-07-02 17:55:53