Frigate Fork — Modifications

This page documents the diffs between orangesurf/frigate and upstream sparrowwallet/frigate. The fork exists to make Frigate’s index usable directly from a zero-trust browser scanner — which, in upstream, is not possible because there’s no raw-batch HTTP endpoint and no compressed-key column.

Fork state: rebased onto sparrowwallet/frigate v1.4.1 (upstream tip e556c0f) with two commits on top — c59379c (port of browser-scanner support) and 247eb9d (a startup-recursion fix). The original per-feature fork commits (c1c0677, f8402cb, …) are preserved on the backup/pre-rebase branch/tag if you want to see them individually.

1. Why fork

Upstream Frigate is an Electrum server. Its clients (Sparrow Wallet, other BIP-352-aware wallets) speak Electrum RPC and optionally hand over ephemeral scan keys so Frigate can run the match query server-side using its unified ufsecp DuckDB extension (Metal / CUDA / OpenCL / CPU). That architecture is a nice power/privacy tradeoff, but it implies the server is in the trust path for that session.

The browser scanner takes a different position: “keys never leave the tab, even ephemerally.” For that to work, Frigate has to expose the raw index over HTTP in a shape the browser can stream efficiently. The fork adds that, plus a few infrastructure changes to make the indexer practical to run on a single mid-range machine.

2. UTXO-only indexing mode

c1c0677 Add UTXO-only indexing mode for Silent Payments

Upstream Frigate has one index schema, a per-transaction tweak table that never deletes rows:

-- upstream
CREATE TABLE tweak (
  txid       BLOB,        -- 32 bytes
  height     INTEGER,
  tweak_key  BLOB,        -- 64 bytes
  outputs    BIGINT[]     -- array of 8-byte x-coord prefixes
);

This is correct and complete — every SP-eligible transaction is preserved forever — but for a single-user wallet it’s wasteful. You mostly care about unspent outputs. The fork adds a new IndexMode enum with two values:

// frigate/src/main/java/com/sparrowwallet/frigate/index/IndexMode.java
public enum IndexMode {
    FULL,        // upstream behavior — the tweak table
    UTXO_ONLY    // new — the utxo table, per-output, prunable
}

In UTXO_ONLY mode, Frigate creates a different table:

-- fork, UTXO_ONLY mode
CREATE TABLE utxo (
  txid                  BLOB,
  output_index          INTEGER,
  height                INTEGER,
  tweak_key             BLOB,     -- 64 bytes
  compressed_tweak_key  BLOB,     -- 33 bytes (see §4)
  output_hash_prefix    BIGINT,   -- signed i64
  value                 BIGINT,
  PRIMARY KEY (txid, output_index)
);

Differences from the upstream schema that matter:

• Granularity is the individual output, not the transaction. A tx with three eligible Taproot outputs gets three rows, keyed by (txid, output_index). This enables per-output deletion.
• Rows are pruned when the output is spent. During block ingestion Frigate walks every input and issues DELETE FROM utxo WHERE txid = ? AND output_index = ? for any outpoint that matches an existing row. The table therefore reflects the actual current UTXO set (modulo the SP-eligibility filter), not a historical log.
• Each row carries a value field so the browser can display balances without round-tripping to mempool.space.
• A utxoMinValue config option (default 1000 sats) filters dust that is unlikely to be an intentional SP payment — this meaningfully shrinks the index.

Config keys, in upstream’s nested TOML format (~/.frigate/config.toml):

[index]
mode = "UTXO_ONLY"     # or "FULL"
utxoMinValue = 1000    # sats, output-level dust filter

Legacy JSON configs (pre-1.4.0) are migrated on first run and the old file is kept as config.bak.

3. Bootstrap from dumptxoutset

f8402cb Add UTXO bootstrap from dumptxoutset for fast initial sync

Naive first-sync means Frigate replays every block from Taproot activation (709 632) to the current tip, scanning each transaction. That’s many hours to days on a regular disk. The fork adds a --bootstrap CLI flag that instead asks Bitcoin Core for the current UTXO set via the dumptxoutset RPC, parses the resulting snapshot, and builds the index from just the outputs that are unspent right now.

New files:

Order of operations (timings in parentheses are what the fork docs project; actual measured times on this box were ~3× faster — see footnote):

1. bitcoind dumptxoutset ~/.frigate/frigate-bootstrap-utxos.dat (serialized by Bitcoin Core, ~9 GB, fork docs say 5–15 min)
2. Parser streams the file, filters to P2TR outputs meeting utxoMinValue (~15 sec)
3. Group surviving outputs by txid, then batch-fetch full transactions via getrawtransaction (parallel stream, 100 txs per batch) to recover their inputs
4. For each, compute tweak_key and insert into utxo (fork docs: ~90 min on a commodity machine)
5. Record the snapshot block height in config; the next startup resumes ordinary block-by-block indexing from snapshot_height + 1

Measured on this box (bitcoind 30.2.0, local NVMe, txindex=1, Apple Silicon, 165 213 395 total UTXOs → 6 735 888 P2TR candidates → 5 611 253 unique txs → 3 975 362 with computable tweaks):

• Step 1 (dumptxoutset): 2 min 51 s
• Step 2 (parse): 14 s
• Step 3 (group by txid): < 1 s
• Step 4 (fetch + compute): 25 min 46 s
• Total: 28 min 52 s, producing 12 015 272 rows

Rule of thumb: on a local bitcoind with a warm disk and txindex=1, budget ~30 minutes, not 90.

Prerequisites: txindex=1 in bitcoin.conf (required for step 3’s getrawtransaction), and the bootstrap only makes sense in UTXO_ONLY mode since the snapshot only contains unspent outputs.

4. Compressed tweak-key column

Upstream stores tweak_key in a 64-byte raw format that packs x and y little-endian. This is optimal for the DuckDB secp256k1 extension — it skips the point-decompression step that normally precedes every scalar multiply. But it’s terrible for HTTP transport: 64 bytes base64s to 88 chars, and the browser needs the 33-byte compressed form anyway because that’s what tiny-secp256k1 accepts.

The fork stores both: tweak_key stays as 64-byte raw (for any future server-side scan queries), and a new compressed_tweak_key BLOB column holds the 33-byte form. Index.java populates both columns during ingestion via a compressRawKey() helper.

Why this matters for the browser:

• Bandwidth: 33 → 44-char base64 vs 64 → 88-char base64 is a 50 % reduction per row. At the current ~12M rows that’s hundreds of megabytes of transport saved per full scan.
• CPU: the worker can feed the key straight into secp.pointMultiply(). No parsing, no little-endian-to-big-endian swap, no parity bit reconstruction.
• Backwards compatibility: the worker still supports the 64-byte path (rawToCompressedBytes()) for databases built against upstream.

The /api/info response exposes a has_compressed_column flag so the browser can pick the right field automatically.

5. HTTP batch API

New file http/HttpApiServer.java.

Upstream Frigate speaks Electrum RPC. The fork adds a plain HTTP server on port 8081, loopback-only, CORS-enabled, serving two endpoints:

GET /api/info

{
  "table": "utxo",
  "total_records": 12021187,
  "batch_size": 20000,
  "key_format": "compressed",
  "key_size": 33,
  "encoding": "base64",
  "mode": "duckdb",
  "has_compressed_column": true
}

Schema advertisement. The browser uses this to decide which fields exist, which key format to expect, and how big the table is (for progress calculations).

GET /api/batch

Parameters:

Response shape (in scan=1 mode, which the browser uses):

{
  "offset": 0,
  "count": 20000,
  "total": 12021187,
  "has_more": true,
  "query_ms": 30.57,
  "records": [
    {
      "k": "AikSKQswQz/2WW4bC0yxsT34gZ5YRoWYR0rc8gFeq2da",  // tweak_key
      "p": "1918350031342774883",                           // prefix (string!)
      "t": "oKbKPek9NVZumRWZ64ZbJ8El9ZIWmf4NUkjZ9t0ZHwQ=",  // txid
      "o": 0,                                               // vout
      "h": 934358,                                          // height
      "v": 1197086                                          // value, sats
    },
    ...
  ]
}

Notes on the encoding:

• k and t are base64 of the raw bytes — 44 chars for 33-byte keys, 44 chars for 32-byte txids. More compact than hex.
• p is a string, not a number. The stored value is a signed i64 whose absolute value can exceed JavaScript’s 2⁵³ safe integer range. Serializing as a string preserves precision; the browser wraps it in BigInt() before comparing.
• Responses are gzipped automatically — ~70 % size reduction.

The HTTP server is started from Frigate.java alongside the Electrum server, so both are available at once and there’s no config knob to disable either.

6. Other changes

Fork-specific additions

• --bootstrap CLI arg added in Args.java, wired into Frigate.java to short-circuit normal startup.
• blockchain.silentpayments.features RPC (electrum/SilentPaymentsFeatures.java) exposes index.mode and index.utxoMinValue to Electrum clients so wallets can detect whether the server is running in a mode that supports their query.
• index.lastIndexedBlockHeight persisted in the TOML config, updated by the indexer every block so bootstrap progress and normal block replay survive restarts.
• TxEntry carries output_index so the UTXO_ONLY scan path can return per-output matches.

Inherited from upstream during the v1.4.1 rebase

Everything listed below is upstream work that the fork now carries because we rebased onto the current v1.4.1 tip rather than staying on an older base:

• Unified ufsecp DuckDB extension. Upstream consolidated the separate secp256k1 + cudasp extensions into a single ufsecp.duckdb_extension.gz per platform (linux amd64/arm64, macos amd64/arm64). The fork now ships these gzipped artifacts under src/main/resources/native/ instead of separate raw binaries.
• Metal / OpenCL / CUDA scan backends. The new ufsecp extension auto-detects GPU capability and exposes ufsecp_backend(). On Apple Silicon the Metal kernel handles SP scanning directly. On Linux with NVIDIA it uses CUDA; with AMD/Intel it uses OpenCL. Selectable via scan.computeBackend = "auto" | "gpu" | "cpu".
• TOML configuration with JSON migration. Upstream refactored io/Config.java to use Jackson + TomlMapper with a nested schema ([core] [index] [scan] [server] [database]). Legacy JSON configs are migrated on first startup and preserved as config.bak. The fork’s new keys (index.mode, index.utxoMinValue, index.lastIndexedBlockHeight) are added to the same migration path.
• Configurable Electrum port. server.port replaces upstream’s hard-coded 50001. This box runs it on 57001.
• DuckDBReadPool for concurrent reads. Upstream added a pool of read-only DuckDB connections (index/DuckDBReadPool.java) so concurrent Electrum scans don’t serialize on a single connection.
• Scan audit mode. If FRIGATE_AUDIT_SCAN_KEY and FRIGATE_AUDIT_SPEND_KEY env vars are set at startup, the indexer itself computes P_k for that wallet on every indexed transaction and stores it alongside the on-chain hash prefix. Useful for cross-checking a live index against a known wallet.
• macOS tray icon (control/TrayManager.java) — upstream provides a system-tray status indicator. The code is present in our fork but the launchd-managed headless process never creates the tray (which requires an Aqua session).
• Better operational error messages. Frigate.getOperationalErrorMessage() recognizes connect-refused, port-in-use, missing txindex, missing bitcoind cookie, authentication failure, and DB lock conflicts, and logs a single explanatory line for each instead of a wall of stack trace.
• Dependency updates. Java 25.0.2, DuckDB 1.4.4, ufsecp v3.64 (“fix rare correctness issues”), plus the usual library bumps.

7. Commit summary

Active on master

Preserved on backup/pre-rebase

The original five per-feature fork commits are kept on a backup branch (pushed to origin) as backup/pre-rebase and backup/pre-rebase-2026-04-15. Useful if you want to see the per-feature diffs without the v1.4.1 noise:

For the system-level picture of how these pieces plug into the browser, see System Docs.