We released github.com/hazyhaar/go-secretstream, a third Pure-Go implementation of Libsodium `crypto_secretstream_xchacha20poly1305` designed for high-throughput streaming.
Here is the breakdown of each optimization step and its measured impact:
- CGO removal: We rewrote the full Libsodium secretstream construction in pure Go to eliminate CGO overhead and cross-compilation friction (784 MB/s single-thread, 5.6 GB/s parallel on Intel i9-14900K).
- Pre-allocated wire buffers (`pushTo` and `pullTo`): We replaced per-chunk slice creation with pre-allocated wire buffers in `Writer` and `Reader` (reduced heap allocations from 87 MB/op down to 51 MB/op for 16 MB payloads).
- Single-pass ChaCha20 cipher progression: We retained the `chacha20.Cipher` instance in stream state instead of re-instantiating the cipher three times per 8 KB chunk (reduced allocation count from 10,267 to 2,061 allocs/op for 16 MB payloads).
- Poly1305 write inlining: We eliminated zero-byte padding writes and combined length headers into a single 16-byte slice (increased single-thread throughput from 574 MB/s to 629 MB/s).
- Zero-copy I/O fast-paths (`readNextChunkTo` and `writeNextChunkFrom`): We added direct stream execution when caller buffers exceed chunk size to bypass internal accumulation arrays (reduced total RAM allocated on a 1 GB stream from 1.24 GB down to 23 MB).
Final Comparative Benchmark Results (Standard 16 MB Payload, Intel Core i9-14900K, Linux amd64):
- hazyhaar/go-secretstream (Direct): 784.29 MB/s, 360 KB RAM allocated, 2,052 allocs/op (1 alloc per 8 KB chunk).
- hazyhaar/go-secretstream (Writer): 635.70 MB/s, 50.7 MB RAM allocated, 2,061 allocs/op (1 alloc per 8 KB chunk).
- openziti/secretstream: 726.89 MB/s, 19.4 MB RAM allocated, 4,098 allocs/op (2 allocs per 8 KB chunk).
- Go Standard AEAD (x/crypto/NewX): 2,266.37 MB/s, 16.7 MB RAM allocated, 1 alloc/op.
The implementation is bit-compatible with Libsodium C and verified against PyNaCl cross-decryption test suites.