perf(byte-storage): write inputs under 128 B as a literals-only block (LAB-12921)

README.md

@@ -341,7 +341,7 @@ Single-threaded time per call of cachekit-core's own operations, and the payload
 | `encrypt_aes_gcm` | 526 ns · 1.95 GB/s | 13.79 µs · 4.75 GB/s |
 | `decrypt_aes_gcm` | 568 ns · 1.80 GB/s | 13.26 µs · 4.94 GB/s |
 
-- `store` compresses with LZ4 (an input under 64 bytes is written as one literals-only LZ4 block, without running the compressor; with a format of at most 255 bytes it goes straight into the envelope, so its store allocates only the returned bytes; from 64 bytes to under 4 KiB the compressor reuses one 8 KiB hash table and one 4,524-byte output buffer per thread, kept for the life of the thread), checksums with xxHash3-64 and wraps both in the MessagePack envelope. `retrieve` checks the envelope, decompresses and verifies the checksum.
+- `store` compresses with LZ4 (an input under 128 bytes is written as one literals-only LZ4 block, without running the compressor; with a format of at most 255 bytes it goes straight into the envelope, so its store allocates only the returned bytes; from 128 bytes to under 4 KiB the compressor reuses one 8 KiB hash table and one 4,524-byte output buffer per thread, kept for the life of the thread), checksums with xxHash3-64 and wraps both in the MessagePack envelope. `retrieve` checks the envelope, decompresses and verifies the checksum.
 - `encrypt_aes_gcm` and `decrypt_aes_gcm` are `ZeroKnowledgeEncryptor`'s AES-256-GCM calls with a 34-byte AAD.
 - The payload is MessagePack-encoded records (`benches/common/mod.rs`), the corpus `hot_path` and `perf_ir` also use. LZ4 compresses it to 0.57 of its size at 1 KiB and to 0.38 at 64 KiB.
 - Each figure is the median of 21 timed batches, from one run on the CPU named. Two runs of this code on the EPYC 7763 agreed within 2% on every cell but one (`decrypt_aes_gcm` at 64 KiB, 23%); three arm64 runs varied by up to 30% on a cell.
@@ -371,7 +371,7 @@ The Criterion suite in `benches/hot_path.rs` needs the `encryption` feature (`ma
 | `tenant_keyring_decrypt/<size>` | `TenantKeyring::decrypt_indexed`, key derived once at `for_tenant` |
 | `hkdf` | `derive_domain_key` (takes no payload, so it has no size) |
 
-Sizes are 64 B, 1 KiB and 64 KiB. `store`, `retrieve` and `retrieve_data` also run at 40 B, below the 64-byte literals-only block threshold, and at 1 MiB. `ByteStorage` switches to its copying allocation pattern once the envelope or the compressed payload reaches 64 KiB. The 64 KiB payload compresses to a 24,702-byte envelope and never gets there; the 1 MiB one compresses to 383,893 bytes. One case at a time: `make perf-ir ARGS="--case store/1024"`. The bench binary is built with fat LTO, so its cases shape each other's code. Because both `retrieve` and `retrieve_data` call the envelope decode, it stays out of line, and `retrieve` costs 26 to 27 Ir/op more at 40 B to 1 KiB (2.8% at 40 B, rustc 1.99.1) than in a binary that calls `retrieve` alone. A consumer that calls only one of the two gets the decode inlined.
+Sizes are 64 B, 1 KiB and 64 KiB. `store`, `retrieve` and `retrieve_data` also run at 40 B and at 128 B, the first size the LZ4 encoder takes (40 B and 64 B sit below the 128-byte literals-only block threshold), and at 1 MiB. `ByteStorage` switches to its copying allocation pattern once the envelope or the compressed payload reaches 64 KiB. The 64 KiB payload compresses to a 24,702-byte envelope and never gets there; the 1 MiB one compresses to 383,893 bytes. One case at a time: `make perf-ir ARGS="--case store/1024"`. The bench binary is built with fat LTO, so its cases shape each other's code. Because both `retrieve` and `retrieve_data` call the envelope decode, it stays out of line, and `retrieve` costs 26 to 27 Ir/op more at 40 B to 1 KiB (2.8% at 40 B, rustc 1.99.1) than in a binary that calls `retrieve` alone. A consumer that calls only one of the two gets the decode inlined.
 
 **Method** (`benches/perf_ir.rs`): each case runs in its own process under `valgrind --tool=cachegrind --cache-sim=no`, once with N calls and once with none, after the same setup and one warm-up call. N is 1,000, or 20 for a 1 MiB case. Ir/op is `(Ir[N] - Ir[0]) / N`, so process start, setup and exit cancel. The counts are exact, so N sets the run time, not the precision. The measured process gets an empty environment, so neither the shell nor a `VALGRIND_OPTS` changes what is counted.
 
@@ -469,12 +469,17 @@ it. Measure as [Instruction counts](#instruction-counts) describes, and put the
 pull request.
 
 **Give small values their own path.** Most cached values are small, and a fixed per-call cost
-dominates them. Below 64 B, `store` writes one literals-only LZ4 block instead of running the
-compressor, which zeroes its hash table on every call: −80.8% instructions on a 63 B store
-([#128]). The trade: compressible inputs under 64 B are stored uncompressed and grow. From 64 B
-to under 4 KiB, the compressor reuses one hash table and one output scratch per thread: −27.5% on
-a 64 B store ([#144]). `make perf-ir` has no case below 64 B or above 64 KiB, so a change that
-branches on size also measures each side of its threshold and 1 MiB, as [#128] and [#144] did.
+dominates them. Below 128 B, `store` writes one literals-only LZ4 block instead of running the
+compressor, which zeroes its 8 KiB hash table on every call and then searches for matches: −80.8%
+instructions on a 63 B store ([#128]), and −95.1% on a 64 B store (10,551 → 515 Ir/op) when the
+threshold rose from 64 B to 128 B ([#156]). The trade: compressible inputs under 128 B are stored
+uncompressed and grow. In the examples measured, small JSON and MessagePack records grew 16% to
+84% and padding and byte runs up to about 4.5 times; no repeated-byte input from 64 B to 127 B grew
+by more than 117 B (`LITERAL_BLOCK_THRESHOLD`'s rustdoc). 128 B is also the most the one-pass
+literal envelope writer allows. From 128 B to under 4 KiB, the
+compressor reuses one hash table and one output scratch per thread ([#144]). A change that
+branches on a size threshold measures each side of it, adding `make perf-ir` cases where the
+[size list](#instruction-counts) misses one, as [#128], [#144] and [#156] did.
 Per-thread reuse has its own traps, all from [#144]:
 
 - Reuse stopped paying at about 6 KiB, because the copy out grows with the block while the saving
@@ -550,6 +555,7 @@ change. [#131] changed `store` and [#132] `retrieve`. [#144] says wasm32 was not
 [#131]: https://github.com/cachekit-io/cachekit-core/pull/131
 [#132]: https://github.com/cachekit-io/cachekit-core/pull/132
 [#144]: https://github.com/cachekit-io/cachekit-core/pull/144
+[#156]: https://github.com/cachekit-io/cachekit-core/pull/156
 [cachekit-rs#185]: https://github.com/cachekit-io/cachekit-rs/pull/185
 
 ## License

SECURITY.md

@@ -55,7 +55,7 @@ This crate protects against:
 This crate does **not** protect against:
 
 - Side-channel attacks on the host system
-- Payload bytes left in process memory: `store` compresses inputs from 64 B to under 4 KiB through a per-thread output buffer (4,524 B) that is not cleared between calls, so it holds bytes of earlier payloads for the life of the thread, as freed buffers do until they are reused
+- Payload bytes left in process memory: `store` compresses inputs from 128 B to under 4 KiB through a per-thread output buffer (4,524 B) that is not cleared between calls, so it holds bytes of earlier payloads for the life of the thread, as freed buffers do until they are reused
 - Compromise of the master key
 - Denial of service via resource exhaustion (partial protection only)
 - Attacks requiring physical access
@@ -113,7 +113,7 @@ the multiply, and a release build panicked when the compressed output outgrew
 the wrapped bound. This crate requires 0.13, which computes the bound in `u64`;
 the `wasm32` CI job stores and retrieves an input one byte past that boundary.
 
-Below 64 bytes, `StorageEnvelope::new` does not call the encoder at all. It
+Below 128 bytes, `StorageEnvelope::new` does not call the encoder at all. It
 writes the input as one literals-only LZ4 block, which every LZ4 block decoder
 reads.
 

benches/perf_ir.rs

@@ -76,13 +76,14 @@ const MIB: usize = 1024 * 1024;
 /// A small value, a typical value, a large object (as `hot_path`).
 const SIZES: &[usize] = &[64, 1024, 64 * 1024];
 /// ByteStorage picks its strategy by size, so store and retrieve (and
-/// retrieve_data, which decodes the same envelope) also measure
-/// 40 B, under the literals-only block threshold (64 B), and 1 MiB. Its
+/// retrieve_data, which decodes the same envelope) also measure 40 B, 128 B
+/// and 1 MiB. 40 B and 64 B sit under the literals-only block threshold
+/// (128 B), 128 B is the first size the LZ4 encoder takes. Its
 /// large-payload switch compares the envelope (store) or the compressed
 /// payload (retrieve) with 64 KiB: the 64 KiB payload stores as a
 /// 24,702-byte envelope and never reaches it, the 1 MiB one as 383,893 bytes,
 /// past glibc's default mmap threshold too.
-const STORAGE_SIZES: &[usize] = &[40, 64, 1024, 64 * 1024, MIB];
+const STORAGE_SIZES: &[usize] = &[40, 64, 128, 1024, 64 * 1024, MIB];
 
 /// Operations and their payload sizes. HKDF derives from the master key and
 /// tenant id alone, so it has one case with no size.

benches/perf_ir_baselines.json

@@ -18,21 +18,24 @@
         "prescan/1024": 910,
         "prescan/64": 815,
         "prescan/65536": 864,
-        "retrieve/1024": 7409,
-        "retrieve/1048576": 6679353,
-        "retrieve/40": 2637,
-        "retrieve/64": 2630,
-        "retrieve/65536": 391975,
-        "retrieve_data/1024": 7243,
-        "retrieve_data/1048576": 6679184,
-        "retrieve_data/40": 2471,
-        "retrieve_data/64": 2464,
-        "retrieve_data/65536": 391809,
-        "store/1024": 22934,
-        "store/1048576": 11912874,
-        "store/40": 1186,
-        "store/64": 10551,
-        "store/65536": 772704,
+        "retrieve/1024": 5739,
+        "retrieve/1048576": 6677798,
+        "retrieve/128": 1304,
+        "retrieve/40": 1005,
+        "retrieve/64": 1007,
+        "retrieve/65536": 390383,
+        "retrieve_data/1024": 5573,
+        "retrieve_data/1048576": 6677632,
+        "retrieve_data/128": 1138,
+        "retrieve_data/40": 839,
+        "retrieve_data/64": 841,
+        "retrieve_data/65536": 390217,
+        "store/1024": 22781,
+        "store/1048576": 11912713,
+        "store/128": 11082,
+        "store/40": 533,
+        "store/64": 515,
+        "store/65536": 772551,
         "tenant_keyring_decrypt/1024": 5608,
         "tenant_keyring_decrypt/64": 2623,
         "tenant_keyring_decrypt/65536": 235224

src/byte_storage.rs

@@ -72,17 +72,36 @@ const MAX_COMPRESSION_RATIO: u64 = 1000;
 pub(crate) const LARGE_PAYLOAD: usize = 64 * 1024;
 
 /// Inputs shorter than this are written as one literals-only LZ4 block instead
-/// of going through `lz4_flex::compress`.
+/// of going through the LZ4 encoder.
 ///
-/// `lz4_flex::compress` zeroes an 8 KiB hash table on every call, which costs
-/// more than the whole store for a tiny value, and below LZ4's 13-byte minimum
-/// its output is literals-only anyway. The price is stored size: a compressible
-/// input under the threshold is no longer shrunk; its block is the input plus
-/// 1 byte (plus 2 from 15 bytes up). Compressed bytes are not canonical
-/// (protocol `spec/wire-format.md`, "Compressed-byte reproducibility"), and
-/// every reader decodes a literals-only block.
+/// The encoder zeroes an 8 KiB hash table on every call, even the reused one
+/// (`compress_into_with_table` clears it), then searches for matches. For a
+/// small value both cost more than the rest of the store, and below LZ4's
+/// 13-byte minimum its output is literals-only anyway. The price is stored
+/// size: a compressible input under the threshold is no longer shrunk; its
+/// block is the input plus 1 byte (plus 2 from 15 bytes up). Compressed bytes
+/// are not canonical (protocol `spec/wire-format.md`, "Compressed-byte
+/// reproducibility"), and every reader decodes a literals-only block.
+///
+/// From 64 to 127 bytes the literal block takes about a fifth of the encoder
+/// path's wall time, and a compressible value pays for it in size. Measured
+/// against the encoder, small structured values grew 16% to 84% (a JSON
+/// record, feature flags, a MessagePack list of maps, a repeated UUID),
+/// padding and byte runs up to about 4.5 times, and random bytes not at all.
+/// Envelope lengths vary by a few bytes with the checksum, whose bytes
+/// MessagePack writes in 1 or 2 bytes each, so these are examples: 127 bytes of
+/// one byte went from 34 to 36 bytes to 151 to 158. The growth itself is at
+/// most 117 bytes per value, the most a sweep of every repeated-byte input
+/// from 64 to 127 bytes reached. The bench corpus (MessagePack records about
+/// 130 bytes long, `benches/common`) shows only +0.2% below 128 bytes,
+/// because a slice that short sits inside one record; past one record it
+/// shows +4% at 144 bytes and +28% at 256.
+///
+/// 128 is also the most `encode_literal_envelope`'s fixed widths allow (the
+/// `<= 0x80` assert below): a higher threshold needs a wider envelope writer
+/// as well as a size decision, so do not tune it up on corpus figures alone.
 #[cfg(feature = "compression")]
-const LITERAL_BLOCK_THRESHOLD: usize = 64;
+const LITERAL_BLOCK_THRESHOLD: usize = 128;
 
 // The literal block writes the length in at most one extra byte, which holds
 // for literal lengths up to 15 + 254.
@@ -211,7 +230,8 @@ fn compress_with_thread_table(data: &[u8]) -> Result<Vec<u8>, ByteStorageError>
 // `retrieve`, `retrieve_data` and `validate` decode into its private borrowing twin,
 // `borrowed::StorageEnvelope`, whose `decode_one_pass` hard-codes this field
 // order and these encodings. `store` encodes from another, `EnvelopeRef`,
-// and writes the same bytes by hand for an input under 64 bytes with a format
+// and writes the same bytes by hand for an input under
+// `LITERAL_BLOCK_THRESHOLD` with a format
 // of at most 255 bytes (`encode_literal_envelope`, sized by
 // `envelope_encoded_len`). Change all of them together;
 // `literal_envelope_matches_rmp_serde`, `encoded_len_matches_rmp_serde` and
@@ -236,6 +256,9 @@ pub struct StorageEnvelope {
 impl StorageEnvelope {
     /// Create new envelope with data compression and checksum.
     ///
+    /// An input under 128 bytes is not compressed: it is written as one
+    /// literals-only LZ4 block, the input plus 1 or 2 bytes.
+    ///
     /// Takes the input by shared slice: it is only compressed and hashed (both
     /// borrow), never retained, so there is no reason to own it. Avoids a
     /// full-payload copy (up to `MAX_UNCOMPRESSED_SIZE`) on the write path.
@@ -319,7 +342,7 @@ impl StorageEnvelope {
 
 /// `data`'s LZ4 block and checksum, with the input and block size checks.
 /// `StorageEnvelope::new` and `store`'s general path call it; the
-/// sub-64-byte envelope `encode_literal_envelope` writes is too small to need
+/// sub-[`LITERAL_BLOCK_THRESHOLD`] envelope `encode_literal_envelope` writes is too small to need
 /// them.
 #[cfg(all(feature = "compression", feature = "checksum"))]
 fn compress_and_checksum(data: &[u8]) -> Result<(Vec<u8>, [u8; 8]), ByteStorageError> {
@@ -449,11 +472,11 @@ fn envelope_encoded_len(
 /// written in one pass into a `Vec` allocated once at its final length: the
 /// literals-only block goes straight into the `bin` field, so no block is
 /// allocated and copied. The bytes are `rmp_serde`'s for the same envelope
-/// (`literal_envelope_matches_rmp_serde`). The envelope is at most 343 bytes,
+/// (`literal_envelope_matches_rmp_serde`). The envelope is at most 407 bytes,
 /// far below every size cap, so it needs none of `compress_and_checksum`'s
 /// checks.
 ///
-/// `None` for an input of 64 bytes or more, or a format string longer than
+/// `None` for an input of [`LITERAL_BLOCK_THRESHOLD`] bytes or more, or a format string longer than
 /// 255 bytes: `store` then takes its general path, which writes the same
 /// bytes.
 #[cfg(all(feature = "compression", feature = "checksum", feature = "messagepack"))]
@@ -593,7 +616,10 @@ impl ByteStorage {
 
     /// Store arbitrary bytes with compression and checksums
     ///
-    /// Returns serialized StorageEnvelope bytes
+    /// Returns serialized StorageEnvelope bytes. An input under 128 bytes is
+    /// not compressed: it is written as one literals-only LZ4 block, the input
+    /// plus 1 or 2 bytes, which is faster than running the compressor and
+    /// costs at most 117 bytes of envelope over it.
     #[cfg(all(feature = "compression", feature = "checksum", feature = "messagepack"))]
     pub fn store(&self, data: &[u8], format: Option<String>) -> Result<Vec<u8>, ByteStorageError> {
         // Security: Check input size before processing
@@ -1090,7 +1116,7 @@ mod tests {
             LITERAL_BLOCK_THRESHOLD - 1,
             LITERAL_BLOCK_THRESHOLD,
             LITERAL_BLOCK_THRESHOLD + 1,
-            100,
+            200,
             1000,
             4103,
         ] {
@@ -1114,7 +1140,7 @@ mod tests {
         // class input: the shared table must not have been upgraded by it.
         let sizes = [
             LITERAL_BLOCK_THRESHOLD,
-            100,
+            200,
             1024,
             2048,
             TABLE_REUSE_LIMIT - 1,
@@ -1544,7 +1570,14 @@ mod tests {
         }
         // `store` itself, either side of its hand-written literal envelope.
         let storage = ByteStorage::new(None);
-        for len in [0, 1, 40, 63, 64, 1024] {
+        for len in [
+            0,
+            1,
+            40,
+            LITERAL_BLOCK_THRESHOLD - 1,
+            LITERAL_BLOCK_THRESHOLD,
+            1024,
+        ] {
             let stored = storage.store(&xorshift_bytes(len), None).unwrap();
             assert!(
                 borrowed::StorageEnvelope::decode_one_pass(&stored).is_some(),
@@ -1783,8 +1816,8 @@ mod tests {
         // (input, whether its envelope is at or above LARGE_PAYLOAD)
         for (data, large) in [
             (vec![], false),
-            (vec![b'a'; 63], false),
-            (vec![b'a'; 64], false),
+            (vec![b'a'; LITERAL_BLOCK_THRESHOLD - 1], false),
+            (vec![b'a'; LITERAL_BLOCK_THRESHOLD], false),
             ((0..70_000).map(|i| (i * 7919 % 251) as u8).collect(), false),
             // either side of the gate: LZ4 adds ~257 length bytes to a literal run
             (xorshift_bytes(LARGE_PAYLOAD - 512), false),
@@ -1869,7 +1902,15 @@ mod tests {
         let storage = ByteStorage::new(None);
         // Warm this thread's LZ4 table and scratch, allocated on first use
         storage.store(&xorshift_bytes(1024), None).unwrap();
-        for len in [0, 14, 15, 40, 63, 64, 1024] {
+        for len in [
+            0,
+            14,
+            15,
+            40,
+            LITERAL_BLOCK_THRESHOLD - 1,
+            LITERAL_BLOCK_THRESHOLD,
+            1024,
+        ] {
             let data = xorshift_bytes(len);
             for format in [None, Some("msgpack".to_string())] {
                 let (stored, requested) = bytes_requested(|| storage.store(&data, format).unwrap());

tests/compress_bound_32_bits.rs

@@ -1,6 +1,6 @@
 //! The LZ4 encoder's output bound on 32-bit targets.
 //!
-//! `StorageEnvelope::new` sends every input of 64 B or more through
+//! `StorageEnvelope::new` sends every input of 128 B or more through
 //! `lz4_flex`, which sizes its output as `16 + 4 + input_len * 110 / 100`.
 //! lz4_flex 0.12 computed that product in `usize`, so on a 32-bit target it
 //! overflowed for any input above 39,045,157 B, far inside the crate's 512 MiB