Photon Ring

Broadcast messaging for Rust, where every subscriber sees every message and no subscriber can slow another one down.

Who this is for

You have one producer (or a few) and several consumers that must each see the whole stream — market data fanned out to a pricing engine, a risk check and a recorder; telemetry to several sinks; a staged pipeline. The messages are small and fixed-shape, they arrive faster than a lock-based channel can move them, and you care about the nanoseconds.

If instead each message should be handled by exactly one of N workers, you want a work queue, not this. If your consumers are async tasks that idle most of the time, use your runtime's own channel.

What you get that you would otherwise build yourself

Consumers with different delivery contracts, on the same ring. Backpressure exists so a consumer that must not lose messages can stop the producer. But a consumer that is only observing should never be able to do that:

let (mut tx, subs) = channel_bounded::<Order>(1024, 0);

let mut risk      = subs.subscribe();        // gates the publisher, loses nothing
let mut telemetry = subs.subscribe_lossy();  // never gates it, drops when behind

risk keeps its no-loss guarantee. telemetry is invisible to the publisher's backpressure scan, so however slow it gets it cannot stall order flow — and when it falls behind it reports Lagged { skipped } with an exact count rather than silently missing data. Both read the same sequence numbers, so an observation can be tied to the message the risk engine processed.

A consumer that dies releases the ring instead of wedging it: its subscriber drops as the thread unwinds, and the publisher continues. Consumers can attach to a ring that is already running, so a debug tap goes on and comes off a live system without perturbing it.

Both properties fall out of the design — each slot carries its own validation stamp, so subscribers never share a barrier and therefore never contend.

Two payload models. channel() and friends take T: Pod — fixed-shape plain data, validated optimistically, no copying beyond the message itself. event_channel() takes any Send + Sync type, including String, Vec, enums and Option: slots own their values and are mutated in place, so steady-state publishing allocates nothing.

It is no_std compatible with alloc. The concurrency protocols are gated in CI by Miri and loom; a TLA+ model of the seqlock lives in verification/ and is checked by hand rather than on every push.

Important

The default channel() is lossy on overflow: the publisher never blocks, and slow subscribers detect drops via TryRecvError::Lagged. If lossless delivery matters more than raw latency, use channel_bounded().

Quick start

use photon_ring::{channel, channel_mpmc, Photon};

// SPMC channel (single producer, multiple consumers)
let (mut pub_, subs) = channel::<u64>(1024);
let mut sub = subs.subscribe();
pub_.publish(42);
assert_eq!(sub.try_recv(), Ok(42));

// MPMC channel (multiple producers, clone-able)
let (mp_pub, subs) = channel_mpmc::<u64>(1024);
let mp_pub2 = mp_pub.clone();
mp_pub.publish(1);
mp_pub2.publish(2);

// Named-topic bus
let bus = Photon::<u64>::new(1024);
let mut p = bus.publisher("prices");
let mut s = bus.subscribe("prices");
p.publish(100);
assert_eq!(s.try_recv(), Ok(100));

Choosing a ring

Every ring broadcasts: each subscriber sees each message, with a private cursor and no shared read barrier. What varies is the payload family and the delivery contract.

Constructor	Payload	Delivery contract
channel	T: Pod	Lossy. The publisher never blocks; a subscriber that falls behind observes Lagged { skipped }. The fastest path.
channel_bounded	T: Pod	Per consumer. subscribe() gates the publisher and loses nothing; subscribe_lossy() taps the same ring and can never stall it.
channel_mpmc	T: Pod	Lossy, many producing threads. There is no bounded MPMC ring — see design constraints.
event_channel	Any T: Send + Sync (String, Vec, enums)	Every subscriber gates the publisher. Slots are factory-built once and mutated in place; steady state copies and allocates nothing.
Photon<T> / TypedBus	T: Pod	String-keyed topics, each an independent lossy ring. Photon fixes one payload type for the whole bus; TypedBus allows one per topic.
topology::Pipeline / topology::Consumer	T: Pod	Dedicated OS threads wired with lossy rings between stages; shutdown, drain, and panic capture handled.

Sequence numbers are shared by every subscriber on a ring, so consumers with different contracts — a gating risk engine, a lossy telemetry tap — can correlate observations of the same message.

Installation

[dependencies]
photon-ring = "3.0.0"

Optional features:

• derive: enables #[derive(photon_ring::DerivePod)] for user-defined Pod types.
• hugepages: enables Linux memory controls such as mlock, prefault, and NUMA helpers.
• atomic-slots (default): the data-race-free slot implementation, which decomposes the payload into AtomicU64 stripes (stepping down through AtomicU32/U16/U8 for a trailing partial stripe) instead of write_volatile/read_volatile. Zero performance cost on x86-64. On ARM64 both paths pay the same reader-side acquire fence, so atomic-slots costs nothing extra there either. Eliminates the formal undefined behavior the default path carries under the Rust abstract machine, for payloads with no padding bytes. Its multi-threaded tests run under Miri in CI, so the claim is machine-checked rather than asserted. default-features = false selects the older volatile implementation instead, which is a formal data race under the Rust memory model and is only worth taking if you have measured a difference on your hardware. Padding is the remaining gap: a type such as (u8, u64) has 7 uninitialized bytes, and reading those as part of an atomic word is itself undefined. Use #[repr(C)] with explicit padding fields (as the examples do) so every byte is initialized.

Rust 1.94+ is supported. For best performance, compile with -C target-cpu=native to enable PREFETCHW and other CPU-specific optimizations.

The problem

Inter-thread communication is often the dominant cost in concurrent systems. Traditional messaging designs usually pay for at least one expensive property on the hot path:

Approach	Write cost	Read cost	Allocation
std::sync::mpsc	Lock + CAS	Lock + CAS	Per-message
Mutex<VecDeque>	Lock acquisition	Lock acquisition	Dynamic growth
crossbeam-channel	CAS on head	CAS on tail	None
LMAX Disruptor pattern	Sequence claim + barrier	Sequence barrier spin	None

The Disruptor showed that pre-allocated rings can remove allocator overhead and drive very low latency, but its shared sequence barriers still create cache-line contention between producers and consumers.

The solution: seqlock-stamped slots

Photon Ring moves synchronization into each slot. Every slot carries its own seqlock stamp beside the payload, so readers validate the data they just loaded instead of bouncing a shared barrier cache line.

                        64 bytes (one cache line)
    +-----------------------------------------------------+
    |  stamp: AtomicU64  |  value: T                      |
    |  (seqlock)         |  (Pod - all bit patterns valid)|
    +-----------------------------------------------------+
    For T <= 56 bytes, stamp and value share one cache line.
    Larger T spills to additional lines (still correct, slightly slower).

Write protocol

1. stamp = seq * 2 + 1     (odd = write in progress)
2. fence(Release)          (stamp visible before data)
3. write_volatile(slot.value, data)
4. stamp = seq * 2 + 2     (even = write complete, Release)
5. cursor = seq            (Release - consumers can proceed)

Read protocol

1. s1 = stamp.load(Acquire)
2. if odd -> spin
3. if s1 < expected -> Empty
4. if s1 > expected -> Lagged
5. value = read_volatile(slot)    (direct read, T: Pod)
6. fence(Acquire)                 (payload read completes before re-check)
7. s2 = stamp.load(Relaxed)
8. if s1 == s2 -> return
9. else -> retry

Why this is fast

1. No shared mutable state on the read path. Each subscriber keeps its own local cursor. Readers do not publish progress into a shared hot cache line unless bounded backpressure tracking is enabled.
2. Stamp and payload are co-located. For T <= 56 bytes, the stamp check and payload read hit the same cache line.
3. No allocation on publish or receive. The ring is fixed at construction time, and hot-path operations are direct slot reads and writes.
4. T: Pod makes torn reads safe to reject. Every bit pattern is valid, so an optimistic torn read is harmless and discarded by the stamp re-check.
5. Single-producer SPMC avoids write-side contention. Publisher::publish takes &mut self, so the type system enforces one producer without CAS. MpPublisher adds an MPMC path when you need multiple concurrent writers.

Benchmarks

Measured with Criterion on an Intel i7-10700KF (8C/16T, 3.80 GHz, Linux 6.8, Rust 1.93.1) and Apple M1 Pro (8C, macOS 26.3, Rust 1.92.0), --release, 100 samples, no core pinning unless stated.

Note

These numbers use Pod payloads and compare concrete implementations, not abstract algorithms. Scheduler noise, pinning, CPU generation, and payload layout all matter, so treat them as reproducible snapshots rather than universal constants.

Fanout scaling

Delivering one message to N consumers is O(N) work somewhere. The question is where it lands, and whether the producer pays it.

Time to publish one message and have all N consumers observe it, in nanoseconds. Single-threaded and in-cache, so this measures protocol overhead rather than real cross-core fanout latency:

N consumers	1	2	4	8	16	32	marginal
Photon Ring	3.9	5.9	8.5	14.9	25.9	50.2	1.5 ns/consumer
tokio::sync::broadcast 1.53	47.7	65.5	99.2	167.3	306.9	579.6	17.2 ns/consumer
crossbeam-channel 0.5, one per consumer	22.3	44.3	87.8	176.5	351.3	701.9	21.9 ns/consumer
flume 0.11, one per consumer	27.4	53.9	108.7	217.6	428.4	859.3	26.8 ns/consumer

The marginal figure is what matters. A subscriber here costs about one cursor read and one stamp check, because subscribers share no state — so the producer's work does not grow with the audience. The other shapes pay per-consumer coordination: a shared-ring broadcast clones through common state, and a point-to-point queue is not broadcast at all, so fanning out means the producer sends once per consumer.

Read that last row as what broadcast costs on a queue that does not do broadcast, not as a race those libraries lost — they solve the different and equally real problem of exactly one receiver owning each message. Reproduce with cargo bench --bench fanout_scaling.

Cross-thread fanout

The table above isolates protocol cost in cache. This one is the deployed shape: one producer, N consumer threads on their own cores, 100k messages, every consumer accounting for all of them. Total milliseconds, lower is better:

	semantics	N=1	N=2	N=4	N=8
Photon Ring	lossless	0.49	0.52	0.58	2.62
disruptor 4.0	lossless	0.45	2.94	6.99	12.15
crossbeam-channel 0.5, one per consumer	lossless	1.39	14.90	27.92	46.79
tokio::sync::broadcast 1.53	lossy	8.23	8.61	33.86	119.47

At a single consumer disruptor is faster — its consumers coordinate through a shared sequence barrier, and with one consumer there is nothing to coordinate, so the barrier costs nothing while this crate still pays for its per-slot stamps. From two consumers onward the positions reverse: the producer must fold every consumer's sequence into a minimum before it can publish, so that cost grows with the audience, while a stamped slot is read independently by each subscriber. Photon stays close to flat through N=4.

Two honest caveats. The jump in photon's own N=8 figure is not established as architectural: that run puts nine threads on a sixteen-thread machine that was not otherwise idle, and it needs a pinned rerun on a quiet box before anyone leans on it. And the lossy row is not comparable to the three lossless ones — it drops under pressure instead of applying backpressure, so compare it against this crate's lossy channel() rather than against channel_bounded().

Reproduce with cargo bench --bench fanout_threaded.

Core operations

                                    i7-10700KF     M1 Pro
                                    ──────────     ──────
  Publish only                        2.8 ns      2.4 ns
  Roundtrip (1 sub, same thread)      2.7 ns      8.8 ns
  Fanout (10 independent subs)       17.0 ns     27.7 ns
  MPMC (1 pub, 1 sub)                12.1 ns     10.6 ns
  Empty poll                          0.9 ns      1.1 ns
  Batch 64 + drain                    158 ns      282 ns
  Struct roundtrip (24B Pod)          4.8 ns      9.3 ns
  Cross-thread roundtrip               95 ns      130 ns
  One-way latency (RDTSC)             48 ns p50     —

Throughput

• Sustained throughput: about 300M msg/s on Intel and 88M msg/s on M1 Pro
• Payload scaling: at cache-line-sized payloads the copy is a few percent of latency — cross-core cache-coherence transfer dominates. The copy only becomes co-dominant in the KiB range; see docs/payload-scaling.md

Delivery contracts in detail

The guarantees sketched above have edges worth knowing before you rely on them.

A dying consumer releases the ring provided its Subscriber drops with it — automatic when the consumer thread owns the subscriber. One parked in long-lived shared state (an Arc'd registry, a supervisor struct) outlives its consumer and keeps gating the publisher, so don't do that with a tracked subscriber. A merely wedged consumer still applies backpressure; that is the guarantee working.

The no-loss guarantee belongs to each tracked subscriber's lifetime, not to the ring. If the last tracked subscriber goes away while lossy ones remain, nothing gates the publisher any more and the bounded ring behaves like a lossy one.

subscribe_from_oldest() is the exception on a bounded ring: it starts at a sequence the publisher was already entitled to overwrite, so its history may be lapped, and it registers a full ring behind — which gates the publisher until it drains. Use subscribe() unless you specifically want the retained history.

cargo run --release --example degradation demonstrates the slow-observer and dead-consumer scenarios; tests/degradation.rs asserts them.

At a glance

Capability	Photon Ring
Delivery	Broadcast — every subscriber sees every message
Publish	2.8 ns lossy, 7.96 ns bounded with a live consumer
Cross-thread roundtrip	95 ns
Throughput	~300M msg/s sustained
Per-consumer contracts	Gating and non-gating subscribers on one ring
Consumer failure	A dead consumer releases the publisher
Hot attach	Subscribe to and detach from a running ring
Topologies	Pipelines, fan-out, managed terminal consumers
Topic bus	Named topics, and a typed bus for per-topic payload types
Backpressure	Optional, per subscriber
no_std	Yes, with alloc
Multi-producer	Yes

Photon Ring is for streams where every subscriber should observe every message with minimal coordination between them. If instead each message should be owned by exactly one receiver, a work queue is the better shape.

API overview

Channels are the lowest-level interface. channel::<T>(capacity) creates the fastest single-producer path and returns a Publisher<T> plus a cloneable Subscribable<T>. channel_bounded::<T>(capacity, watermark) adds optional backpressure; Publisher::try_publish returns PublishError::Full(value) instead of overwriting unread slots. channel_mpmc::<T>(capacity) returns MpPublisher<T>, which is Clone + Send + Sync and uses atomic sequence claiming for concurrent producers. On the write side, the important APIs are publish, publish_with to build the value in the caller's closure, publish_batch on Publisher, and published/capacity for lightweight counters.

Subscribers are independent and contention-free by default. Subscribable::subscribe() starts from future messages only, while subscribe_from_oldest() starts at the oldest message still retained in the ring. Subscriber<T> exposes try_recv, recv, recv_with, latest, pending, recv_batch, and drain, plus observability counters through total_received, total_lagged, and receive_ratio.

For topic routing, Photon<T> provides a string-keyed bus where all topics share the same payload type, and TypedBus allows a different T: Pod per topic. Both lazily create topics and expose publisher, try_publisher, subscribe, and subscribable. publisher() will panic if the publisher for that topic was already taken, and TypedBus also panics on type mismatches for an existing topic.

Pipelines build dedicated-thread processing graphs on supported OS targets. topology::Pipeline::builder().capacity(...).input::<T>() returns an input publisher plus a typed builder; .then(...) chains stages, .fan_out(...) creates a diamond, .then_a(...) and .then_b(...) extend either branch, and .build() returns the final subscriber plus a Pipeline handle. then_with(f, WaitStrategy) (and then_a_with, then_b_with) lets you configure the wait strategy for each pipeline stage. The handle supports shutdown, join, panicked_stages, is_healthy, and stage_count. For manual shutdown outside topology, use Shutdown.

The #[derive(photon_ring::DeriveMessage)] macro supports a #[photon(as_enum)] attribute for fields whose types are #[repr(u8)] enums. Unrecognized types without this attribute now produce a compile error instead of being silently assumed to be enums.

Ring capacity accepts any integer >= 2. Power-of-two capacities use bitwise seq & mask for zero-overhead indexing; arbitrary capacities use Lemire reciprocal-multiply fastmod (~1.5 ns).

Wait behavior is explicit. recv_with accepts WaitStrategy::BusySpin, YieldSpin, BackoffSpin, Adaptive, or MonitorWaitFallback depending on whether you want the absolute lowest wakeup latency or better core sharing. MonitorWaitFallback uses Intel TPAUSE (Alder Lake+) for near-zero power wakeup (~30 ns), with automatic fallback to PAUSE on older x86 or WFE on ARM. On supported platforms, the crate also includes affinity helpers for CPU pinning; with the hugepages feature on Linux, you can use Publisher::mlock, Publisher::prefault, and mem::{set_numa_preferred, reset_numa_policy} to reduce page-fault and NUMA noise.

Companion crates

• photon-ring-async — Runtime-agnostic async wrappers. AsyncSubscriber with yield-based polling and configurable spin budget. Works with tokio, smol, embassy, or any executor.
• photon-ring-metrics — Observability wrappers with SubscriberMetrics (snapshot/delta tracking) and PublisherMetrics. Framework-agnostic — bring your own prometheus/opentelemetry.

Design constraints

Constraint	Rationale
T: Pod	Every bit pattern must be valid, which makes optimistic torn reads safe to reject.
Capacity >= 2	Any capacity works. Power-of-two uses seq & mask; arbitrary capacity uses Lemire fastmod (~1.5 ns, zero-division).
Single producer by default	The fastest path relies on &mut self rather than write-side atomics.
Lossy overflow by default	The publisher never blocks; subscribers detect drops through Lagged.
MPMC is lossy-only	Backpressure gates the publisher on per-subscriber trackers, which the multi-producer claim path does not consult. For lossless delivery use channel_bounded (single producer).
64-bit atomics required	The core algorithm depends on AtomicU64.
64-bit sequence numbers	Stamp encoding seq * 2 + 2 overflows at u64::MAX / 2 (~9.2 × 10^18 messages). At 1 billion msg/s this would take ~292 years.

Platform support

Platform	Core ring	Affinity	Topology	Hugepages
x86_64 Linux	Yes	Yes	Yes	Yes
x86_64 macOS / Windows	Yes	Yes	Yes	No
aarch64 Linux	Yes	Yes	Yes	Yes
aarch64 macOS (Apple Silicon)	Yes	Yes	Yes	No
wasm32	Yes	No	No	No
FreeBSD / NetBSD / Android	Yes	Yes	Yes	No
32-bit ARM (Cortex-M)	No	No	No	No

Soundness and Pod

The Pod trait

The Pod trait means more than Copy: every possible bit pattern of the payload must be valid. This is required because the stamp-based read protocol may speculatively read bytes from a slot while a writer is updating it. If a torn bit pattern could be invalid for T, the read would be undefined behavior before the stamp check could discard it.

Primitive numerics, arrays of Pod, and the zero- and one-element tuples are already supported. Larger tuples are not: repr(Rust) may pad them, and Pod forbids padding. For your own structs, use #[repr(C)], stick to Pod fields, and implement Pod manually or via the derive feature when appropriate.

Type	Why it is not Pod	Use instead
bool	Only 0 and 1 are valid	u8
char	Must be a valid Unicode scalar	u32
NonZero<u32>	0 is invalid	u32
Option<T>	The discriminant has invalid patterns	Sentinel integer
Rust enum	Only declared variants are valid	u8 or u32
&T, &str	Pointers must be valid	Value types only
String, Vec<_>	Heap-owning, has Drop	Fixed [u8; N] buffer

Formal soundness

Photon Ring offers two slot implementations, selectable at compile time:

	Default (atomic-slots)	default-features = false
Mechanism	AtomicU64::store/load(Relaxed) stripes	write_volatile / read_volatile
Formal status	Formally sound — no data races	Data race under Rust abstract machine (practical UB)
Miri	Passes, enforced in CI	Flags multi-threaded tests
x86-64 cost	Identical MOV instructions; measured indistinguishable	Baseline
ARM64 cost	One DMB ISHLD reader fence (both paths)	Same fence — no additional cost
Precedent	Same pattern as Linux kernel seqlocks (20+ years)	Per-word atomic decomposition, as in atomic-memcpy

Note

Both implementations place an Acquire fence between the payload read and the stamp re-check — the same barrier the Linux kernel's read_seqcount_retry() carries as smp_rmb(). Without it an acquire load is one-way and a weakly ordered CPU may satisfy the payload read after the re-check has validated, which would return data from a later overwrite. On x86 the fence emits no instruction — TSO already orders load-load — but it is still a compiler barrier, and measured at roughly +1.2 ns on the same-thread roundtrip microbenchmark because it forbids reordering the optimiser was otherwise free to do. That is the price of the guarantee, on every architecture.

With that fence in place, the default volatile implementation produces correct results on real hardware; what remains is that it is a data race under Rust's abstract machine, which Miri reports and no compiler has yet exploited. Enable atomic-slots for a build free of that race — machine-checked in CI, and free on x86-64.

Tip

Keep rich domain types at the edges and publish compact Pod messages in the middle. Convert enums, Option, booleans, and strings into explicit numeric fields or fixed-size buffers before calling publish.

Examples of safe boundary conversions:

• bool -> u8 (0 = false, 1 = true)
• enum Side { Buy, Sell } -> u8 (0 = Buy, 1 = Sell)
• Option<u32> -> u32 (0 = None, nonzero = Some)
• String / &str -> [u8; N]

Running

cargo test
cargo bench
cargo bench --bench payload_scaling
cargo +nightly miri test --test correctness -- --test-threads=1
RUSTFLAGS="--cfg loom" cargo test --release --test loom_mpmc --test loom_backpressure  # exhaustive interleaving checks
cargo run --release --example market_data
cargo run --release --example pipeline
cargo run --release --example backpressure

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.