How We Made Our Text-to-Speech API Free: The Inference Engineering Behind S2.1 Pro

TL;DR

• Fish Audio S2.1 Pro is now available as a free text-to-speech API: 83 languages, no hard usage cap, model string s2.1-pro-free. Free access is available for an initial period; we'll communicate any changes with advance notice. For more information, view here.
• End-to-end inference optimization reduced per-GPU request capacity by 4×: the same workload that previously required four H200s now runs on one
• Key levers: custom CUDA kernels (2.1–4.3× over cuBLAS at decode shapes), continuous batching (52× throughput scaling c=1→64), GPU utilization lifted from ~50% to ~90%+
• The same stack powers our voice cloning API — same model weights, same infrastructure, free tier included. S2.1 Pro builds on Fish Speech, our open-weight foundation model.
• Core kernel library fish-scales-ops is open source

See why developers are switching from other platforms → Start building free

July 2026 | Fish Audio rebuilt the inference stack and made S2.1 Pro free for every developer

---

The Real Reason Voice AI Costs Money

Voice AI inference has a structural problem that pricing pages don't explain: autoregressive decode is GPU-utilization hostile by default.

Every TTS API call drives a decode loop that generates audio tokens one step at a time. Each step is a small matrix multiply — M ≤ 128 in practice — followed by a memory read of the KV cache for that sequence. The CUDA cores sit mostly idle while the memory subsystem catches up. On a $30,000 H200, you're paying to compute and burning it on memory stalls.

At low concurrency, this gets worse. A single-request decode loop occupies the GPU in 10–20μs bursts separated by scheduling overhead. GPU utilization on a naively-served TTS endpoint at c=1 can sit below 10%. The hardware is present; the work isn't filling it.

The industry's answer has been to pass the cost to developers through per-character pricing. Our answer was to fix the utilization problem at the infrastructure layer. When a single GPU can do the work that previously required four, the economics of a free tier become viable.

Here's what that required.

---

End-to-End Optimization: The Full Stack

No single change gets you to a 4× efficiency improvement. The gains came from optimizing every layer simultaneously and letting them compound:

• Custom CUDA kernels — eliminating per-token compute waste
• FP8 quantization — cutting memory bandwidth pressure in half
• Continuous batching — filling the GPU across concurrent requests
• GPU scheduling — eliminating idle capacity between workloads
• Owned network and storage — removing cloud infrastructure markup

Each layer is described below. The throughput and cost numbers at the end are the product of all of them together.

---

Custom CUDA Kernels: fish-scales-ops

Why cuBLAS Wasn't Enough

The decode shapes in TTS serving (M ≤ 128) are not the shapes that cuBLAS, PyTorch's GEMM kernels, or even torch.compile are optimized for. Those libraries target training and large-batch prefill — M in the thousands. At small M, they leave memory bandwidth on the table and miss fusion opportunities that only exist at decode batch sizes.

The specific gap: for a SwiGLU MLP forward at M=1 (single-request decode), cuBLAS scaled_mm issues three separate kernel launches — activation, gate multiply, down projection — with intermediate results written to and read back from HBM between each. At M=1, that round-trip is the bottleneck, not the compute.

We built fish-scales-ops to close this gap: a production-grade FP8 GEMM and FlashAttention library targeting NVIDIA Hopper (H200, sm_90a) and Blackwell (sm_120a) architectures, open source.

FP8 Quantization Strategy

We implemented two quantization schemes for different hardware generations:

bsgemm (128×128 block-scaled FP8) for H200/Hopper. Block scaling at 128×128 granularity gives good numeric stability without per-tensor quantization's accuracy risk.

mxfp8 (1×32, OCP UE8M0 format) for Blackwell (RTX 5090, RTX 6000 PRO). Microscaling with 32-element shared exponents — the OCP MX standard — gives finer-grained numeric fidelity at the scales that matter for audio token generation.

One non-obvious bug in the mxfp8 path took significant debugging: quantize_blockscale.py materialized weight_scale tensors with .contiguous() for safetensors compatibility, but linear_mxfp8_raw reads scales with K-major stride (1, N). Loading a row-major contiguous [N, K/128] made every GEMM produce incorrect logits at full kernel speed — downstream sampling mode-collapsed to a single repeating semantic token, which looks like a model quality issue but is actually a memory layout bug. The fix was re-striding in MXFP8LinearMethod.process_weights_after_loading. Post-fix, greedy decode matches bf16 token-exact on the first five audio frames.

Kernel Benchmark Results

On decode shapes (M ≤ 128), internal benchmarks on equivalent hardware — full methodology in the fish-scales-ops repo:

• MXFP8 path beats torch.nn.functional.scaled_mm on 8 of 10 square shapes from 1024³ to 16384³, up to +21%
• End-to-end Qwen3-4B SwiGLU MLP forward: 1.7–6.2× over raw cuBLAS scaled_mm at every M from 1 to 4096
• Against torch.compile-fused cuBLAS at decode shapes: 2.1–4.3× faster
• MXFP8 FlashAttention on sm_120a: 7–19× over PyTorch SDPA decode, 1.2–2.7× over FlashInfer BF16

The two fused kernels driving most of the gain: one fuses activation quantize + UE8M0 scale-pack into a single launch; the other fuses the SwiGLU prologue directly into the down-GEMM activation quantize, eliminating a full BF16 intermediate write+read that PyTorch otherwise materializes between the two ops.

---

Batch Inference Architecture: The Latency–Throughput Trade-off

Continuous Batching for Autoregressive TTS

Kernel efficiency improves per-token cost. Batching is what converts per-token efficiency into GPU utilization across the request stream.

Static batching — pad all sequences to the same length, run as a fixed batch — waste compute on padding and blocks new requests until the slowest sequence in the batch finishes. For TTS with variable output lengths, the tail latency penalty is significant.

We use continuous batching adapted from sglang: new requests join the active decode batch as slots free up, without waiting for a batch boundary. The GPU decode loop never stalls on a slow outlier request.

DualAR: S2-Specific Batching Constraints

S2.1 Pro's DualAR architecture generates audio tokens in two stages — coarse codebooks followed by fine codebooks — with a sliding previous_tokens window that conditions each step. This creates a correctness constraint that standard LLM batching doesn't have: the prev_tokens_shift_inplace kernel requires a strict shape/dtype contract (prev int32 [bs, W, cw], next_tokens int32 [bs, cw]). Any shape-massaging in the caller — .unsqueeze(), implicit dtype promotion — corrupts the window and produces degraded audio that's hard to distinguish from a model quality regression.

For voice cloning workloads, this constraint is especially important: speaker conditioning flows through the same decode path, and window corruption manifests as speaker identity drift across long outputs rather than obvious audio artifacts.

Throughput Results

H200, bsgemm FP8, concurrency sweep:

Throughput scales ~52× from c=1 to c=64, while TTFB increases by 36.6ms — 73ms to 110ms. The GPU does 52 times the work for a latency penalty that's imperceptible in most production workloads.

At c=64, a single H200 sustains 8,006 tok/s. That number is the direct mechanism behind the free tier: more throughput per GPU means lower cost per request.

These are the numbers running behind every free API call. Start building →

Real-Time Mode: Architectural Approach for Low-Latency Workloads

For voice agents and turn-taking applications, we expose a separate real-time endpoint optimized for time-to-first-audio rather than aggregate throughput.

The core architectural decision: prefill happens at connection open, off the latency-critical path entirely. The clock for streaming latency starts at first audio chunk send, not at connection establishment. This means the prefill cost, which scales with prompt length, doesn't appear in the TTFA number at all.

At the scheduler level, a pingpong design runs decode and prefill on separate threads with no per-cycle host barrier. The barrier removal is the primary TTFA improvement at c≥4: in a naive implementation, the host-side synchronization after each decode step serializes what should be concurrent work. Pipelined decode loop and pinned H2D memory transfers address TTFB and jitter at high concurrency.

The full design and latency breakdown are in arXiv:2603.08823.

---

Quality Validation: How We Verified Optimization Didn't Degrade Audio

Quantization and batching can introduce quality regressions that are difficult to detect with throughput benchmarks alone — the GPU runs at full speed producing incorrect outputs.

We caught one instance of this during mxfp8 development: the weight layout bug described above produced ~53% relative logit error while kernel latency was completely normal. The regression only showed up in output token analysis, not in any performance metric.

Our validation pipeline has two gates:

Correctness gate: Post-quantization greedy decode must match bf16 token-exact on the first five audio frames for a fixed reference input. Token-exact match at the first frames catches systematic logit corruption — if the quantized model produces different probability distributions, it will diverge from bf16 immediately on deterministic inputs.

Performance regression gate: Per-request decode_tps must stay within 5% of frozen baseline numbers at every concurrency point. Any change that degrades throughput beyond 5% at any concurrency — c=1, c=16, c=64 — requires explicit justification before merge. This gate protects against correctness fixes that silently add synchronization to the decode loop.

Both gates run on every change to the inference stack. The result: the S2.1 Pro Free model weights are identical to the paid tier, and the quantized serving path is validated against the unquantized baseline on every deployment.

---

Voice Cloning at Scale: Why Inference Efficiency Matters

Fish Audio's voice cloning has always been free — and it remains, by our own assessment and developer feedback, the strongest in its class. Speaker consistency across 83 languages, natural prosody, and stable performance across accents aren't features we added to a paid tier. They're the baseline. S2.1 Pro builds on Fish Speech S2 Pro, our open-weight foundation model released earlier this year.

What inference optimization changes is the economics of maintaining that commitment at scale. Voice cloning workloads carry higher per-request cost than standard TTS: every call must condition on a reference speaker embedding, maintain speaker identity across variable-length outputs, and do both consistently across languages. Without the efficiency gains described in this post, free voice cloning at production scale would require subsidizing a structurally expensive workload. With them, it doesn't.

Three optimizations matter specifically for voice cloning:

FP8 preserves speaker conditioning. The token-exact correctness validation — which catches logit-level corruption — directly validates that quantization doesn't degrade the speaker embedding signal. Speaker consistency in cloned outputs matches the bf16 baseline.

Continuous batching handles mixed request types. Production voice cloning traffic mixes reference-conditioned and unconditioned requests, varying output lengths, varying languages. Continuous batching fills the decode batch regardless of request type — there's no penalty for workload heterogeneity.

52× throughput scaling makes bulk generation viable. At 8,006 tok/s on a single H200, generating large cloned-voice audio libraries — audiobooks, game dialogue, localized content at scale — is economically viable within the free tier.

The free tier doesn't include voice cloning because we optimized our way into affording it. We include it because that's always been the product — and the engineering work described here is what lets us keep that promise as usage scales. For integrating voice cloning into agent workflows, see our MCP and agent skills support.

Clone a voice in under 60 seconds → Try free

---

GPU Infrastructure: Owning the Stack

Hardware and Procurement

Our GPU cluster runs approximately 500 cards across multiple data centers. The multi-DC topology was a procurement decision before it was a reliability one: GPU supply and pricing are volatile, and single-vendor lock-in means absorbing that volatility directly. Multi-vendor, multi-DC sourcing hedges price risk and supply constraints simultaneously, at the cost of operational complexity.

For inference, we run a mixed fleet. H200 SXM5 handles high-concurrency production serving where bsgemm FP8 aggregate throughput dominates. RTX 5090 and RTX 6000 PRO (sm_120a) serve edge endpoints — at c=64, 5090 mxfp8 achieves 5,869 tok/s, approximately 71% of H200 bsgemm throughput at substantially lower hardware cost per card.

The sm_120a deployment required one non-obvious fix: the original cuda_graph_max_bs = 24 cap for 32GB GPUs produced a severe throughput cliff at high concurrency — c=32 fell to ~31 tok/s/req, c=64 to ~35 tok/s/req. Lifting the cap to min(max_running_requests, 64) recovered the full throughput curve. The underlying issue: CUDA graph capture at bs=24 was leaving the GPU underutilized at the decode batch sizes that actually appear at c=32+.

GPU Scheduling: Eliminating Idle Capacity

Baseline GPU utilization on an inference-only cluster is approximately 50%. Inference traffic is inherently bursty; the GPU sits idle during off-peak hours, between training runs, and during dataset loading phases.

We run two parallel scheduling systems — Kubernetes for inference, Slurm for training — with a shared principle: data cleaning and preprocessing jobs run as elastic fill work at the lowest priority tier in both clusters. Cleaning jobs occupy idle GPU capacity whenever inference or training isn't using it. They're checkpointable and immediately preemptible, so they disappear without correctness cost when real work arrives.

Priority ordering: inference > cleaning in the K8s cluster; training > cleaning in Slurm. The autoscaler on the inference side uses real-traffic signals — not time-based heuristics — so cleaning jobs get evicted as soon as load ramps.

Result: GPU utilization from ~50% to ~90%+. At cluster scale, that delta roughly halves the effective cost per inference request — the hardware budget is fixed, but twice as many requests are served against it.

Network Infrastructure

We operate our own network: direct ISP bandwidth procurement, ASN, 300Gbps connectivity, and Cloudflare peering. Two cost implications for TTS specifically.

Egress at TTS scale is non-trivial. Cloud provider egress pricing is structurally high-margin; owning the network layer removes that markup entirely.

Latency is the second reason. Every millisecond of network round-trip appears in user-perceived TTFB. Cloudflare peering puts Fish Audio endpoints close to users routed through Cloudflare's network — a substantial fraction of global API traffic. Multi-DC deployment means most requests reach a nearby inference node without traversing the full public internet.

Storage: Tiered Architecture

Training data, model checkpoints, and inference cache have different access patterns and cost requirements. We use three-tier architecture:

• Hot (NVMe flash): active training data, model weights for serving
• Warm (mixed flash/HDD): recent checkpoints, preprocessed datasets
• Cold (self-hosted Ceph HDD cluster): historical training data, long-term storage

The cold tier replaced third-party object storage for bulk data. A local cache layer in front of cold storage means repeated reads — common in training data pipelines — are served from NVMe rather than cold HDD.

---

The Result: What the Stack Produces

Across all layers:

• Custom FP8 kernels: 2.1–4.3× faster than standard cuBLAS at decode shapes
• Continuous batching: 52× throughput scaling from c=1 to c=64, 37ms latency increase
• GPU scheduling: utilization ~50% → ~90%+, roughly halving cost per request
• Owned infrastructure: cloud markup removed from compute, network, and storage

Combined, the same request volume that previously required four H200s now runs on one. The 4× efficiency improvement is what makes the free tier economically viable — not a subsidy, but a structural cost reduction. The free access period reflects that the unit economics work; see Pricing for current availability.

---

How Fish Audio's Inference Stack Compares

Infrastructure data based on publicly available documentation, GitHub repositories, and engineering blogs as of June 2026.

The pattern is consistent: Fish Audio is the only major TTS provider that has publicly documented its inference architecture, open-sourced its kernel library, and published concrete throughput numbers. "Not disclosed" isn't a criticism — it's an observation about what's verifiable. For an independent assessment of voice quality across providers, see our blind TTS provider comparison.

---

What's Open Source

fish-scales-ops is available now: FP8 GEMM and FlashAttention for Hopper and Blackwell, production-grade, CUDA Graph capture-safe.

If you're building inference infrastructure for autoregressive models on H200 or RTX 5090/6000 PRO, the decode-shape performance gap versus standard libraries is real and the kernels are there to close it.

What's in the library:

• FP8 GEMM: bsgemm (1×128 act × 128×128 wgt) for Hopper; MXFP8 1×32 (OCP UE8M0) for Blackwell
• MXFP8 FlashAttention for sm_120a: contiguous prefill, paged-prefill (extend), paged decode with native GQA via stride-0 K/V broadcast
• CUDA Graph capture-safe throughout

---

What's Next

B300 support is on the roadmap. Native FP8 tensor cores and NVLink 5 bandwidth on Blackwell will push the throughput curve further.

Inference optimization continues. Frozen benchmark numbers are a regression gate, not a target. Each concurrency point is something to beat.

More of the serving stack is a candidate for open-sourcing as it stabilizes — the sglang_lite modifications, DualAR-specific batching logic, sm_120a CUDA graph handling. The infrastructure that makes the free tier sustainable is the same infrastructure we'd want the community to build on.

---

Getting Started with the Free TTS API

Model string: s2.1-pro-free. One header change from any existing Fish Audio API call.

python

import httpx

body = {
    "text": "Hello, world!",
    "reference_id": "your_model_id",
    "format": "mp3",
}

with httpx.Client() as client:
    res = client.post(
        "https://api.fish.audio/v1/tts",
        headers={
            "Authorization": "Bearer <YOUR_API_KEY>",
            "Content-Type": "application/json",
            "model": "s2.1-pro-free",
        },
        json=body,
    )

res.raise_for_status()

with open("output.mp3", "wb") as f:
    f.write(res.content)

JavaScript

import { writeFile } from "fs/promises";

const body = {
  text: "Hello, world!",
  reference_id: "your_model_id",
  format: "mp3",
};

const res = await fetch("https://api.fish.audio/v1/tts", {
  method: "POST",
  headers: {
    Authorization: "Bearer <YOUR_API_KEY>",
    "Content-Type": "application/json",
    model: "s2.1-pro-free",
  },
  body: JSON.stringify(body),
});

if (!res.ok) {
  throw new Error(`TTS request failed: ${res.status} ${await res.text()}`);
}

const buffer = Buffer.from(await res.arrayBuffer());
await writeFile("output.mp3", buffer);

83 languages. No hard usage cap. No credit card. Voice cloning included.

Every other state-of-the-art TTS API charges from the first token. S2.1 Pro doesn't.

Clone a voice in under 60 seconds →

Full API Documentation →