Case studies

Every layer of the storage stack. One specialty. Storage.

From raw NVMe commands to the GPU's data path, our engineers have shipped production code at each layer. When storage is the bottleneck, we can follow the problem through every layer ourselves.

  • Built production code our engineers wrote
  • Upstream features in the open-source project's own codebase
  • Deployed & tuned installed, benchmarked, handed over
Where we've worked, GPU to NVMe7 layers

↑ application / GPU

  1. Workloads HPC · AI · Kubernetes

    • Go CSI driver serving block volumes to containers and KubeVirt VMs (Built)
    • Slurm scheduling and xCAT provisioning for HPC clusters (Deployed & tuned)
  2. Observability eBPF

    • Fleet-scale eBPF tracing we built (Built)
  3. Caching & tiering KV cache · cold tier

    • S3 garbage collector keeping the cold tier consistent (Built)
    • LMCache KV-cache tiers for LLM inference (Deployed & tuned)
  4. Filesystems & protocols Lustre · Ceph · pNFS

    • BeeGFS core development with ThinkParQ: metadata search, mirroring, CLI (Upstream)
    • Lustre, Ceph and pNFS deployment and tuning (Deployed & tuned)
  5. Control plane & consensus Raft · RocksDB

    • Raft-replicated, RocksDB-backed metadata service with leases (Built)
  6. Data servers & protection replication · EC · recovery

    • Chunk replication, erasure coding, background GC and recovery (Built)
  7. Kernel, block & devices io_uring · ublk · NVMe

    • ublk block driver (DKMS), io_uring reactor, raw NVMe passthrough (Built)
    • Filesystem client compatibility through Linux 6.12 (Upstream)

↓ NVMe / devices

Instruments

We build the tools when the standard ones run out

We show exactly which kernel function, I/O path and node is the bottleneck, without redeploying, recompiling or instrumenting your application. Then we build eBPF tools so your team can find the next bottleneck.

trace.yaml · what to trace

target: vfs_read
measure: latency
output: histogram
nodes: storage-*

vfs_read.bt · generated (illustrative)

kprobe:vfs_read /comm != "bpftrace"/ {
  @start[tid] = nsecs;
}

kretprobe:vfs_read /@start[tid]/ {
  @latency_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}

A runnable bpftrace script, with kprobe, uprobe or tracepoint attachment chosen automatically, scheduled across the fleet.

Custom built

yapio

A parallel I/O benchmark we wrote in C, where groups of processes run different I/O patterns across nodes at the same time.

Custom built

eBPF tracing

Follows slow I/O from application calls through the kernel to the device.

Run at scale

The standard toolchain

  • IOR / mdtest
  • fio
  • IO500
  • TLA+ / TLC
  • bpftrace
  • Ansible
  • pytest
  • Grafana

Failure testing

Proven in the lab, ready for production

Distributed systems fail in ways that only real clusters reveal. We build deterministic automation that stands up real multi-node clusters, injects exact failures, and gates every release on the result.

  • Node crash

    Clients reconnect, no acknowledged write is lost

    ● encoded in tests

  • Network partition

    Consensus holds, no split-brain, clean rejoin

    ● encoded in tests

  • Disk failure

    Chunk recovery completes, every checksum matches

    ● encoded in tests

  • Failure mid-resync

    Mirrored metadata survives a second failure

    ● encoded in tests

  • Live upgrade

    Two major versions mount side by side; rollback is safe

    ● encoded in tests

  • Pod restart

    CSI block devices keep serving in-flight I/O

    ● encoded in tests

Find the layer that's costing you throughput.

Send us your topology, workload and a benchmark run. We will get back to you.