Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch c02133da-24fb-4053-857e-dbc36a880067

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

56

Bytes

5.30 GB

Source host

orbit-2

Manifest sha256

cfc9a0367d274b71...

Started

2026-07-02 18:57:59

Completed

2026-07-02 19:14:53

Manifest summary

  • created_at: 2026-07-02T18:57:06Z
  • completed_at: 2026-07-02T18:57:06Z
  • verification.passed: true
  • verification.total_files: 10716
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
79doneimport-feed2026-07-02 18:58:052026-07-02 19:11:56vultr/pid:379506056/562026-07-02 19:14:53-

Items 56

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
videof6c97655d3d67609...3fb30e246d90437a...createdasset:10721-2026-07-02 19:12:04
reel9e6a6cec657175f2...fe833661dd0a4b30...createdasset:10722-2026-07-02 19:12:07
reeld797c32d9a2b450a...c41f387f5d8d49db...createdasset:10723-2026-07-02 19:12:11
reel029e06e00b7ba6d7...d33b5e88c55b4a65...createdasset:10724-2026-07-02 19:12:15
reelb7d77f0ee28c20c3...4ed2b6803e7d4b23...createdasset:10725-2026-07-02 19:12:18
reela39a28fb9bd3542a...c5befacbb1dc400d...createdasset:10726-2026-07-02 19:12:22
reel29fabef55277d04a...a22e36a6cacb4d0c...createdasset:10727-2026-07-02 19:12:26
reelb0228cbeb360381f...d4bc8b6dfee34226...createdasset:10728-2026-07-02 19:12:30
reel27c392dd81c0701f...20dce8d6ac7c4a69...createdasset:10729-2026-07-02 19:12:33
reele5b605e5027fee29...19c3300162af4f1b...createdasset:10730-2026-07-02 19:12:37
reela9ef57b147e4095a...894c67e9666e455d...createdasset:10731-2026-07-02 19:12:41
reel55002572dd438973...2497c1fee3a44544...createdasset:10732-2026-07-02 19:12:45
reel19238d59f7dfd702...92f4a3703d4446f2...createdasset:10733-2026-07-02 19:12:49
reelac2dceb587205584...c9bd1bb154e2482b...createdasset:10734-2026-07-02 19:12:52
reel12e175f8ff922ec1...2c64907fa28748ee...createdasset:10735-2026-07-02 19:12:56
reel7e9757641be918dc...bb920ded0834402c...createdasset:10736-2026-07-02 19:13:00
reelf6d930aa0a6d44ff...b2c068330a734c6f...createdasset:10737-2026-07-02 19:13:04
reel19bfad0a5d1ead38...500fa569b2b1475f...createdasset:10738-2026-07-02 19:13:07
reelcae0f1ee1488dbac...542f59210fea45b2...createdasset:10739-2026-07-02 19:13:11
reelc1711bbe6cda9fc8...ac1d586c68194c6c...createdasset:10740-2026-07-02 19:13:14
reel69ea3c092fcb3e2c...048cc0fce7254b26...createdasset:10741-2026-07-02 19:13:18
reel403c3e05d0f77341...e6afb1a9348c408b...createdasset:10742-2026-07-02 19:13:22
reelc1f82061da5d5097...016bd6b230f64df4...createdasset:10743-2026-07-02 19:13:26
reelb46e9b27dd25d918...3c64930408144b76...createdasset:10744-2026-07-02 19:13:30
reele77c92228a40564d...b6d0626ad981437d...createdasset:10745-2026-07-02 19:13:33
reel0b53daa5d52b4d4f...72ee8fdf021449f8...createdasset:10746-2026-07-02 19:13:38
reel40b5ce4570ad7e17...10d515c9288e46f0...createdasset:10747-2026-07-02 19:13:42
reel1aba19a938989eab...b3dc74d181a04197...createdasset:10748-2026-07-02 19:13:45
reel445d2aee55cc3d97...8bf9d2a09700496a...createdasset:10749-2026-07-02 19:13:50
reele1a62322fb510f4d...f494929f8a7c4eb6...createdasset:10750-2026-07-02 19:13:54
reel5cd0462aa93ec36d...8523d4ac03024ae9...createdasset:10751-2026-07-02 19:13:58
reelaa5fc8757460ec2a...fcf2bbc76b8f4f2b...createdasset:10752-2026-07-02 19:14:01
photo6a6b3e363c668526...3dc7c7a682004d61...createdasset:10753-2026-07-02 19:14:03
photob7f53539228fba75...29a8c6348aaa44b9...createdasset:10754-2026-07-02 19:14:05
photo9dcdb2de4d6e0aa8...92bc09d84328446c...createdasset:10755-2026-07-02 19:14:07
photoab1e68885242fd36...2152be2663434c69...createdasset:10756-2026-07-02 19:14:09
photo8102a418607094a4...e34dcd8ef4474a70...createdasset:10757-2026-07-02 19:14:11
photo7c979f60705cd98b...0a8a329f4f9c4e16...createdasset:10758-2026-07-02 19:14:13
photofc0fe80ceb8997dd...69a6ab35534f4861...createdasset:10759-2026-07-02 19:14:15
photo8ff6d7702e287538...b7becb01ee6c4bab...createdasset:10760-2026-07-02 19:14:17
photo1352f72a00688a9d...b58773941c904388...createdasset:10761-2026-07-02 19:14:19
photod83b8ad60aaa4031...1696234f588e44bb...createdasset:10762-2026-07-02 19:14:21
photo0605795ec9ace0f0...bc7f8427db5f4329...createdasset:10763-2026-07-02 19:14:24
photo2a29c9a90892057c...53b4d393c5954bcf...createdasset:10764-2026-07-02 19:14:27
photoeda04853e97d22cb...488f0d14e8314bac...createdasset:10765-2026-07-02 19:14:29
photobbd3c8ac6e4d231f...3a083d5546234b49...createdasset:10766-2026-07-02 19:14:31
photo3a44a58449260c52...b77f21b723b74aed...createdasset:10767-2026-07-02 19:14:33
photoba4ba97eadd9e4cf...1edf9de98d594905...createdasset:10768-2026-07-02 19:14:35
photo9afd0539c255adc1...e541d00e382b4266...createdasset:10769-2026-07-02 19:14:38
photo876c04dca997c349...bfa0c126d92b45e2...createdasset:10770-2026-07-02 19:14:40
photoa8f361c9a33b91e6...cafc36eb39934b00...createdasset:10771-2026-07-02 19:14:42
photo7935b344471a86bb...93ed781653924a4f...createdasset:10772-2026-07-02 19:14:44
photoc3f658caacb624ae...7b62e9ea00444f45...createdasset:10773-2026-07-02 19:14:46
photoec4d8a306bb4c474...27642f2a96c24c8d...createdasset:10774-2026-07-02 19:14:48
photoa0ac9ed1da7b0ebc...761dc32503884763...createdasset:10775-2026-07-02 19:14:50
photo57d135cd745d9cc6...0d56a8a98ab3418b...createdasset:10776-2026-07-02 19:14:53

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"79","phase":"job_finish","summary":{"items_seen":56,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":56,"deferred":0,"failed":0...2026-07-02 19:14:53
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":56,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-07-02 19:14:53
item_updated{"container":1,"expected_member_count":24,"gallery_hash":"6d99ea5c5020358088ad9f0ea7e4006b","gallery_id":"65","linked_member_count":24,"missing_member_count":0,"phase":"container_link","title":"countr...2026-07-02 19:14:53
item_created{"cover_hash":"0d56a8a98ab3418bbb486304a4e9a0de","error":null,"hash":"57d135cd745d9cc60f26df4933db0a11","phase":"item_import","target_ids":{"asset_id":"10776"},"type":"photo"}2026-07-02 19:14:53
item_created{"cover_hash":"761dc32503884763af9d30ea52cc849a","error":null,"hash":"a0ac9ed1da7b0ebc6ce185daa4170918","phase":"item_import","target_ids":{"asset_id":"10775"},"type":"photo"}2026-07-02 19:14:50
item_created{"cover_hash":"27642f2a96c24c8d94727b0bbd319cee","error":null,"hash":"ec4d8a306bb4c4749f0271b87dc845ef","phase":"item_import","target_ids":{"asset_id":"10774"},"type":"photo"}2026-07-02 19:14:48
item_created{"cover_hash":"7b62e9ea00444f45ba4548e3207e2c81","error":null,"hash":"c3f658caacb624aec247090040f1c271","phase":"item_import","target_ids":{"asset_id":"10773"},"type":"photo"}2026-07-02 19:14:46
item_created{"cover_hash":"93ed781653924a4f8fdd40ada0ab8de7","error":null,"hash":"7935b344471a86bb11281a5492214aa8","phase":"item_import","target_ids":{"asset_id":"10772"},"type":"photo"}2026-07-02 19:14:44
item_created{"cover_hash":"cafc36eb39934b00970ae849d00e4206","error":null,"hash":"a8f361c9a33b91e6a1649d44552fb249","phase":"item_import","target_ids":{"asset_id":"10771"},"type":"photo"}2026-07-02 19:14:42
item_created{"cover_hash":"bfa0c126d92b45e297d355b4427d9310","error":null,"hash":"876c04dca997c349cb89d0c1e0ef1406","phase":"item_import","target_ids":{"asset_id":"10770"},"type":"photo"}2026-07-02 19:14:40
item_created{"cover_hash":"e541d00e382b4266b39f3a4f6d831ac3","error":null,"hash":"9afd0539c255adc19559e368872aefff","phase":"item_import","target_ids":{"asset_id":"10769"},"type":"photo"}2026-07-02 19:14:38
item_created{"cover_hash":"1edf9de98d5949059ebe6de474c6d031","error":null,"hash":"ba4ba97eadd9e4cf23d2f216bd3a91ad","phase":"item_import","target_ids":{"asset_id":"10768"},"type":"photo"}2026-07-02 19:14:35
item_created{"cover_hash":"b77f21b723b74aed9804cae639e30c0c","error":null,"hash":"3a44a58449260c522bad76e727e6e625","phase":"item_import","target_ids":{"asset_id":"10767"},"type":"photo"}2026-07-02 19:14:33
item_created{"cover_hash":"3a083d5546234b498b527ec2901363e7","error":null,"hash":"bbd3c8ac6e4d231f2164ea89333d5319","phase":"item_import","target_ids":{"asset_id":"10766"},"type":"photo"}2026-07-02 19:14:31
item_created{"cover_hash":"488f0d14e8314bacacbd5862338151e9","error":null,"hash":"eda04853e97d22cb74f374fb5405891a","phase":"item_import","target_ids":{"asset_id":"10765"},"type":"photo"}2026-07-02 19:14:29
item_created{"cover_hash":"53b4d393c5954bcf9c01d45658d61752","error":null,"hash":"2a29c9a90892057cd74929aa1c56570c","phase":"item_import","target_ids":{"asset_id":"10764"},"type":"photo"}2026-07-02 19:14:27
item_created{"cover_hash":"bc7f8427db5f4329852f704403316daa","error":null,"hash":"0605795ec9ace0f0e6c27b9f5d0b8954","phase":"item_import","target_ids":{"asset_id":"10763"},"type":"photo"}2026-07-02 19:14:24
item_created{"cover_hash":"1696234f588e44bbad32260830d18cb4","error":null,"hash":"d83b8ad60aaa403171133236c5015577","phase":"item_import","target_ids":{"asset_id":"10762"},"type":"photo"}2026-07-02 19:14:21
item_created{"cover_hash":"b58773941c904388abbbc825cc6fda76","error":null,"hash":"1352f72a00688a9d63762b1ea0a4d7f6","phase":"item_import","target_ids":{"asset_id":"10761"},"type":"photo"}2026-07-02 19:14:19
item_created{"cover_hash":"b7becb01ee6c4bab86da252341182dfe","error":null,"hash":"8ff6d7702e2875383cc5367c01875616","phase":"item_import","target_ids":{"asset_id":"10760"},"type":"photo"}2026-07-02 19:14:17
item_created{"cover_hash":"69a6ab35534f48618f0c2ebed9f83b83","error":null,"hash":"fc0fe80ceb8997dd26f15be082324781","phase":"item_import","target_ids":{"asset_id":"10759"},"type":"photo"}2026-07-02 19:14:15
item_created{"cover_hash":"0a8a329f4f9c4e16a5c5ae3edb6511a4","error":null,"hash":"7c979f60705cd98b7cfa27b5d0d54658","phase":"item_import","target_ids":{"asset_id":"10758"},"type":"photo"}2026-07-02 19:14:13
item_created{"cover_hash":"e34dcd8ef4474a709d88aa8d43e0d32a","error":null,"hash":"8102a418607094a4dfacc23123b3deb6","phase":"item_import","target_ids":{"asset_id":"10757"},"type":"photo"}2026-07-02 19:14:11
item_created{"cover_hash":"2152be2663434c699d09993b97fc84a3","error":null,"hash":"ab1e68885242fd3647d5579cc00dcb99","phase":"item_import","target_ids":{"asset_id":"10756"},"type":"photo"}2026-07-02 19:14:09
item_created{"cover_hash":"92bc09d84328446cb616b36f2f6cf22a","error":null,"hash":"9dcdb2de4d6e0aa87b0439725a16c8eb","phase":"item_import","target_ids":{"asset_id":"10755"},"type":"photo"}2026-07-02 19:14:07
item_created{"cover_hash":"29a8c6348aaa44b9848f58eff63f818b","error":null,"hash":"b7f53539228fba75b1828c32a29d4b1f","phase":"item_import","target_ids":{"asset_id":"10754"},"type":"photo"}2026-07-02 19:14:05
item_created{"cover_hash":"3dc7c7a682004d61b674b11c61964d7d","error":null,"hash":"6a6b3e363c6685268df62283059f2b52","phase":"item_import","target_ids":{"asset_id":"10753"},"type":"photo"}2026-07-02 19:14:03
item_created{"cover_hash":"fcf2bbc76b8f4f2b9a5097dff6767eff","error":null,"hash":"aa5fc8757460ec2a13092ce48616895d","phase":"item_import","target_ids":{"asset_id":"10752"},"type":"reel"}2026-07-02 19:14:01
item_created{"cover_hash":"8523d4ac03024ae9915ebcbaf228cf70","error":null,"hash":"5cd0462aa93ec36d776ce0559b668c9a","phase":"item_import","target_ids":{"asset_id":"10751"},"type":"reel"}2026-07-02 19:13:58
item_created{"cover_hash":"f494929f8a7c4eb6ac51ed3612b110fd","error":null,"hash":"e1a62322fb510f4d4646f1768a51b320","phase":"item_import","target_ids":{"asset_id":"10750"},"type":"reel"}2026-07-02 19:13:54
item_created{"cover_hash":"8bf9d2a09700496aae392021e6d02c9b","error":null,"hash":"445d2aee55cc3d973bba2146fe103198","phase":"item_import","target_ids":{"asset_id":"10749"},"type":"reel"}2026-07-02 19:13:50
item_created{"cover_hash":"b3dc74d181a0419794ab75b8ac9e9b79","error":null,"hash":"1aba19a938989eab2eee41bb17c47c98","phase":"item_import","target_ids":{"asset_id":"10748"},"type":"reel"}2026-07-02 19:13:45
item_created{"cover_hash":"10d515c9288e46f0b706745b40ff9833","error":null,"hash":"40b5ce4570ad7e175a1335a1e0fb8079","phase":"item_import","target_ids":{"asset_id":"10747"},"type":"reel"}2026-07-02 19:13:42
item_created{"cover_hash":"72ee8fdf021449f8bc98b8e2422ed1dc","error":null,"hash":"0b53daa5d52b4d4f5ba046f0b2722dd8","phase":"item_import","target_ids":{"asset_id":"10746"},"type":"reel"}2026-07-02 19:13:38
item_created{"cover_hash":"b6d0626ad981437da7d400adae2b0f23","error":null,"hash":"e77c92228a40564d2f1e9b568d930289","phase":"item_import","target_ids":{"asset_id":"10745"},"type":"reel"}2026-07-02 19:13:33
item_created{"cover_hash":"3c64930408144b76824173712c69be74","error":null,"hash":"b46e9b27dd25d91827bdc3cf97fc46fb","phase":"item_import","target_ids":{"asset_id":"10744"},"type":"reel"}2026-07-02 19:13:30
item_created{"cover_hash":"016bd6b230f64df4aa4e53695a3dd18a","error":null,"hash":"c1f82061da5d5097a01c07222d4e1700","phase":"item_import","target_ids":{"asset_id":"10743"},"type":"reel"}2026-07-02 19:13:26
item_created{"cover_hash":"e6afb1a9348c408b9984eb4c11a0727a","error":null,"hash":"403c3e05d0f77341fb54ff2c9d9eeb85","phase":"item_import","target_ids":{"asset_id":"10742"},"type":"reel"}2026-07-02 19:13:22
item_created{"cover_hash":"048cc0fce7254b26968a21a05cf5d20b","error":null,"hash":"69ea3c092fcb3e2cbd2c0ac997584c49","phase":"item_import","target_ids":{"asset_id":"10741"},"type":"reel"}2026-07-02 19:13:18
item_created{"cover_hash":"ac1d586c68194c6cad49a546e5c18aef","error":null,"hash":"c1711bbe6cda9fc897c6564f33cf6c00","phase":"item_import","target_ids":{"asset_id":"10740"},"type":"reel"}2026-07-02 19:13:14
item_created{"cover_hash":"542f59210fea45b28268825f3a3d9be4","error":null,"hash":"cae0f1ee1488dbace53e88d392c11871","phase":"item_import","target_ids":{"asset_id":"10739"},"type":"reel"}2026-07-02 19:13:11
item_created{"cover_hash":"500fa569b2b1475f93c877890ac83918","error":null,"hash":"19bfad0a5d1ead38b4588f44b3834a3b","phase":"item_import","target_ids":{"asset_id":"10738"},"type":"reel"}2026-07-02 19:13:07
item_created{"cover_hash":"b2c068330a734c6f8179bac45be38655","error":null,"hash":"f6d930aa0a6d44ff34f6b9699996d4d1","phase":"item_import","target_ids":{"asset_id":"10737"},"type":"reel"}2026-07-02 19:13:04
item_created{"cover_hash":"bb920ded0834402ca173217a0b7b295a","error":null,"hash":"7e9757641be918dc9067cb0bc069ef6f","phase":"item_import","target_ids":{"asset_id":"10736"},"type":"reel"}2026-07-02 19:13:00
item_created{"cover_hash":"2c64907fa28748eebc775a119cb32361","error":null,"hash":"12e175f8ff922ec13d5751c82a1d10fa","phase":"item_import","target_ids":{"asset_id":"10735"},"type":"reel"}2026-07-02 19:12:56
item_created{"cover_hash":"c9bd1bb154e2482b94ef338734bd1ded","error":null,"hash":"ac2dceb587205584784e2b2cf34fa83c","phase":"item_import","target_ids":{"asset_id":"10734"},"type":"reel"}2026-07-02 19:12:52
item_created{"cover_hash":"92f4a3703d4446f2ab2e89ec2ab1fdc5","error":null,"hash":"19238d59f7dfd702cf97a425e58adb08","phase":"item_import","target_ids":{"asset_id":"10733"},"type":"reel"}2026-07-02 19:12:49
item_created{"cover_hash":"2497c1fee3a44544bb8e1946ab3d26f9","error":null,"hash":"55002572dd438973f1d99b775549d46c","phase":"item_import","target_ids":{"asset_id":"10732"},"type":"reel"}2026-07-02 19:12:45
item_created{"cover_hash":"894c67e9666e455d871e80c985f37c2c","error":null,"hash":"a9ef57b147e4095a8db66f937adbcda8","phase":"item_import","target_ids":{"asset_id":"10731"},"type":"reel"}2026-07-02 19:12:41
item_created{"cover_hash":"19c3300162af4f1b8fcfb7723f25b114","error":null,"hash":"e5b605e5027fee291ebedb64a305f798","phase":"item_import","target_ids":{"asset_id":"10730"},"type":"reel"}2026-07-02 19:12:37