# MP3 master and laptop sync Endpoint: **https://mp3-sync.wdms.ca**. Protocol version 1. Python 3.10+. This service exchanges complete, closed MaintPrice3 **data snapshots**, including `Data/Price.mdb`, `Data//Survey.mdb`, the other MDBs, and optional `Reports/`. The server's `master` is an atomic pointer to the latest accepted generation. Every accepted generation and every conflicting upload is retained. A publish validates the archive and JobInfo/Survey IDs before changing the pointer. **Initial state: awaiting the current laptop data.** Old server exports are not the current master. In particular, the COR export lacks laptop jobs 396–415. Bootstrap once from the laptop with MP3 closed. Then reconcile server packages by job ID/UniqueID, never by replacing the laptop's whole index with an old one. ## Data contract ZIP paths are relative to the MaintPrice3 install/data root, with no enclosing `MaintPrice3/` directory: ``` Data/Price.mdb Data/419/Survey.mdb Data/PTable.mdb Tables.mdb Wsh.mdb HHData/HHJobs.mdb Reports/COR/COR_419_Pricing.pdf ``` Managed paths: root `*.mdb`, any `*.mdb` under `Data/` or `HHData/`, and PDF/CSV/JSON/XLSX/TXT/MD under `Reports/`. All other files (including executable, DLL, INI, `.rpt` templates, credentials, client state, lock and temporary files) are outside this format. App installation and machine configuration stay local. Keep separate local backup directories OUTSIDE the app/data tree. All path matching and conflict comparisons must respect Windows case rules. ZIP paths may not contain traversal, symlinks, duplicate paths, case collisions, Windows device names or unsafe characters. Maximum 20,000 members, 2 GiB expanded and 2 GiB uploaded. Compression is ZIP deflate. Every file has a SHA-256 manifest. The server accepts Jet MDBs and requires each indexed JobID to have a survey with the same UniqueID; it rejects duplicate IDs/UIDs and empty job indexes. This checks database readability and identity, not pricing correctness. ## API Every `/v1/` request requires `Authorization: Bearer ` over HTTPS. The public `/healthz` and `/setup/` contain no databases or credentials. Tokens grant read/write access to this MP3 store only, not shell access. | Method / path | Result | |---|---| | GET `/healthz` | Service health and protocol version | | GET `/v1/head` | Current revision and manifest, or `revision: null`, `bootstrap_required: true` | | GET `/v1/history` | Accepted revisions and preserved conflicts, newest first | | GET `/v1/revisions//manifest.json` | Immutable revision metadata | | GET `/v1/revisions//snapshot.zip` | Immutable complete data snapshot | | GET `/v1/conflicts//snapshot.zip` | Preserved conflicting upload | | GET `/v1/conflicts//manifest.json` | Its metadata and reason | | GET `/v1/packages` | Historical generated packages with names, lengths and checksums | | GET `/v1/packages/.zip` | Download a historical package for reconciliation | | POST `/v1/snapshots` | Validate and conditionally publish a complete snapshot | POST raw ZIP bytes (not multipart), with `Content-Length`, `Content-Type: application/zip`, `X-MP3-SHA256: `, `X-MP3-Base-Revision: ` (`none` ONLY for bootstrap), and `X-MP3-Client: `. Revisions are opaque 32-character lowercase hex IDs. A successful response is HTTP 201 with the new manifest. An identical snapshot based on the current revision returns HTTP 200 and the current manifest. HTTP 409 means the current revision changed OR files disappeared without explicit deletion review. The uploaded snapshot is saved under `conflict_id`; the current master is untouched. Never retry a stale upload with a freshly fetched revision without first reconciling both versions. Preserve the old common ancestor. An explicitly reviewed deletion may use `X-MP3-Allow-Delete: yes`, still requiring the correct base revision. No timer should automatically send this header. HTTP 401 = authentication failure; 413 = size limit; 422 = invalid archive or database; 428 = missing/invalid revision precondition; 507 = low free disk space. After a network timeout, fetch head/history and compare the file manifest to the submitted snapshot: a publish may have succeeded before the response was lost. Never assume the upload failed and discard the local snapshot. ## Laptop client algorithm (to implement locally) Persist a common-ancestor revision and full file manifest outside the live tree. Use SHA-256 content, not modification time, to detect changes. Poll about every 60 seconds, while logged in. Local MP3 remains usable offline; synchronize after it closes. Close-time sync and a "Sync, then open MP3" desktop shortcut are useful. 1. Hold one OS mutex shared by the task, manual sync and launcher. If MP3 or Access is running, or any `.ldb`/`.laccdb` exists, defer. Check actual executable names and the real data path, including Windows VirtualStore. Don't assume the installed Program Files copy is the one MP3 writes. Never delete locks blindly. 2. First run: make a complete local backup. If server is empty, snapshot the actual laptop data and bootstrap. If both sides already contain data and there is no baseline, reconcile explicitly; never assume one is newer from timestamps. 3. Snapshot closed local data into a private staging tree, checking stable hashes before and after. If local and remote match, record the remote baseline. If neither changed since the ancestor, do nothing. If only local changed, POST with the ancestor revision. If only remote changed, download that specific revision, verify ZIP checksum AND complete manifest, and prepare installation. 4. Before installing, re-check the local manifest, app processes, and lock files. Prevent MP3 from starting during installation (launcher/mutex plus appropriate Windows process/file guards). Take a full backup outside the live directory. Apply the whole managed set with a durable journal and rollback; never let the launcher open MP3 after a partial/interrupted apply. Recover the journal first on restart. Preserve all unmanaged app files and machine settings. Do not just perform an unguarded sequence of copies over the live install. 5. If both sides changed, preserve local staging + remote revision + common ancestor, display a visible conflict, and stop automatic replacement. Even disjoint jobs often edit the SAME binary Price.mdb. Reconcile JobInfo rows by JobID and UniqueID using a real MDB reader/writer, with matching survey folders; never concatenate MDBs, choose by timestamp, or fabricate new identities. 6. Only update the baseline after successful upload acknowledgement/verified recovery, or complete verified local installation. Keep logs, tray/status feedback and a last-success timestamp. Keep credentials out of logs/arguments. This is eventual synchronization of closed datasets, not simultaneous multi-user editing. Competing offline edits can require reconciliation. Microsoft likewise [recommends avoiding opening Access databases directly from cloud sync locations](https://support.microsoft.com/en-us/access/ways-to-share-an-access-desktop-database). ## Reference Python transport Download `client.py` and `core.py` from `/setup/` into the SAME directory. No pip dependencies. This reference can publish a staged tree and download into a new directory. It deliberately does not install snapshots over a running Windows app. The local integration must provide the process guards, scheduling, rollback and UI. ``` python client.py --token-file /private/token status python client.py --token-file /private/token publish /closed/staging --bootstrap python client.py --token-file /private/token checkout /new/staging/path python client.py --token-file /private/token publish /edited/checkout ``` `checkout` writes `.mp3-sync-state.json` containing its ancestor. `publish` uses that recorded revision; never replace it to bypass a conflict. The initial `--bootstrap` refuses a nonempty server. `publish --allow-delete` is for reviewed removal only. Direct API use is fine if it preserves the same contract. ## Server operator / future Codex sessions Source: `/home/ubuntu/projects/mp3_injest/sync/`. Runtime code: `/opt/mp3-sync/`. Service: `mp3-sync.service`, dedicated `mp3sync` user. State: `/srv/mp3-sync/`; canonical read-only tree: `/srv/mp3-sync/master/`. Credential: `/etc/mp3-sync/token` (private; readable by ubuntu for local publishing). HTTP listens ONLY on `127.0.0.1:8150`; nginx supplies HTTPS. The certificate uses the normal certbot renewal timer and existing nginx deploy hook. Do not modify `master`, `revisions`, old snapshots, or the pricing SQLite catalog in place. Create a checkout for each job batch, use its CURRENT Price.mdb as the builder input, create its matching surveys, validate pricing, then publish once. The old 2017 catalog remains a historical analysis artifact. Rebuild a separate catalog from the current master when a task needs current rates. ``` cd /home/ubuntu/projects/mp3_injest python3 sync/client.py status python3 sync/client.py checkout /tmp/mp3-work-UNIQUE # Build jobs against /tmp/mp3-work-UNIQUE/Data/Price.mdb and current templates. # Put resulting files into the checkout, preserving existing jobs. python3 sync/client.py publish /tmp/mp3-work-UNIQUE ``` For an existing generated package, append ONLY selected NEW JobInfo rows: ``` python3 sync/import_package.py --checkout /tmp/mp3-work-UNIQUE \ --package cor_mp3/output/COR_MP3_Jobs_416-419 \ --output /tmp/mp3-merged-UNIQUE --jobs 416 417 418 419 python3 sync/client.py publish /tmp/mp3-merged-UNIQUE ``` The importer uses Java/Jackcess to copy the current index and append selected rows, retaining unrelated jobs. It refuses existing JobIDs/UIDs, schema mismatches, missing surveys, and source changes. Existing jobs need explicit reconciliation. It copies package documents into Reports// automatically. It stages a new tree and validates all job identities before exposing the result. Recovery: download an old revision into a NEW checkout, compare with current, restore/reconcile the required data, then publish against the actual current base. Keep both copies; no API deletes history. For a full deliberate rollback, stage the old snapshot and explicitly record the reviewed current base before publishing. Do not silently reset a client's base revision. ## Operations and backups ``` systemctl status mp3-sync journalctl -u mp3-sync --since today df -h /srv/mp3-sync sudo nginx -t python3 -m unittest discover -s tests/sync -v ``` All revisions and conflicts are retained without automatic pruning. Watch disk usage; requests stop before space is exhausted. A full snapshot is transmitted per change, trading bandwidth for a consistent complete generation (current datasets are modest). `/srv/mp3-sync` contains the ZIPs, extracted trees and atomic master link; include the entire directory plus `/etc/mp3-sync`, `/opt/mp3-sync`, the systemd unit and nginx config in host backups. Restore those, permissions and the service, then verify manifest hashes before resuming client writes. Version history here is on the SAME server disk. It protects against replaced files and conflicts; it is not itself an independent backup. On 2026-09-09 we verified completed AWS EBS snapshots of this server's root volume `vol-007ba2f9bcc14dd82` in `ca-west-1` on September 7, 8 and 9. The latest was `snap-0b33073f0f4a96900`, completed with start time 2026-09-09 11:12 UTC, before this service was installed. `/srv/mp3-sync` is on that root volume. Check that a subsequent successful snapshot includes the new service and incoming laptop data; backup policy/retention and restore testing were not audited. End-to-end laptop validation still needs to be performed by the local setup session. Rotate token by atomically replacing `/etc/mp3-sync/token`, preserving permissions, then updating authorized clients. The server reads it per request. Never put it in URLs, source control, screenshots, command-line arguments or public setup files. To deploy source updates: run the tests, install server.py/core.py into `/opt/mp3-sync/` as root-owned files, update public client/docs as needed, restart `mp3-sync`, and check HTTPS health and unauthorized rejection. To disable service, `sudo systemctl disable --now mp3-sync`; stored generations are unaffected.