Four ways to make a machine write music, what each one costs, and why I built the smallest one

1. The four families

Almost everything shipping in generative music falls into one of four architectural families. They differ in what the model actually predicts, and that single choice determines size, latency, cost, controllability and whether the thing can run on a phone.

Symbolic models: predict notes

The model emits a stream of musical events — pitch, duration, position, chord — and a separate synthesiser turns those into sound. This is the oldest line of work, running from Magenta's MelodyRNN and PerformanceRNN through folk-rnn to Music Transformer and REMI.

Because a minute of music is a few hundred tokens rather than a few thousand frames of audio, these models are tiny. Overtone is 3.3M parameters. The trade is total: you get perfect editability and near-zero cost, and you get no timbre, no production, no vocals. The model writes the score; something else has to perform it.

Autoregressive audio tokens: predict compressed audio

A neural codec — EnCodec, DAC — compresses waveforms into a discrete token stream, and a transformer predicts those tokens exactly as a language model predicts words. Meta's MusicGen is the reference implementation, at 300M, 1.5B and 3.3B parameters, operating on a 32 kHz EnCodec with four codebooks at roughly 50 tokens per second of audio.

This produces real audio with real timbre. It is also where the cost appears: generating a minute means predicting thousands of tokens autoregressively, and quality below about 1B parameters degrades fast.

Latent diffusion: denoise a compressed representation

Instead of predicting tokens left to right, compress audio into a continuous latent space and train a diffusion model to denoise it, conditioned on a text embedding. Stable Audio Open (~1.2B parameters) pairs a VAE, a T5 text encoder and a diffusion transformer.

The practical advantage is that diffusion generates the whole clip in parallel over a fixed number of denoising steps, so it does not get slower the way autoregression does. ACE-Step, the strongest open full-song model, reports synthesising up to four minutes of music in about twenty seconds on an A100 — roughly fifteen times faster than LLM-based baselines. The classic weakness is long-range structure: it is easier to make a good-sounding thirty seconds than a song with a real second chorus.

Commercial hybrids: both, behind an API

Suno, Udio and Google's Lyria are closed, but the public technical picture is consistent: a diffusion transformer over a compressed audio latent, conditioned by a text encoder, with autoregressive components handling structure and lyric alignment. Udio has described its system as a latent diffusion transformer. These are the only models that reliably produce a complete song with coherent vocals.

The row that matters for product strategy is editable output. Audio models return a mixed, finished stereo file. If your user wants the bassline two semitones down, there is nothing to edit — you regenerate and hope. Symbolic models return structure, so every downstream feature that needs to know what the notes are is available to you.

2. What it actually costs

I measured the hosted option rather than guessing. Google's Lyria 3 is exposed through the Gemini API in two variants, and I ran both.

Two things stood out. First, five concurrent clip requests all returned in about nine seconds, so the service parallelises cleanly. Second, the Pro model returns a timestamped lyric alignment alongside the audio:

[[A0]]
[0.0:] The porch light stayed on for a year after you left
[:] I kept paying the bill like a promise I never said
[[B1]]
[30.3:] So burn it down slow, let the moths have the glow
[[A2]]
[60.6:] I packed up the summer in cardboard and dust

The letter is section identity and the number is the ordinal, so A0 B1 A2 B3 is verse / chorus / verse / chorus. That is a machine-readable song form arriving free with every generation, and almost nobody builds on it. Synced lyrics, section-aware regeneration and structural editing are all sitting there in an API response that most integrations throw away.

Now the arithmetic that decides business models. At $0.04 a clip, a user who generates forty drafts in an evening costs you $1.60 — and generation is exactly the behaviour a music tool encourages, because nobody keeps the first result. Self-hosting an open model instead: ACE-Step's reported four minutes of audio per twenty seconds of A100 time works out to roughly $0.008 per four-minute track on a spot A100 around $1.50/hour, before utilisation losses. An order of magnitude cheaper, and you own the latency.

Overtone's marginal cost per minute is $0.00, because inference happens on the listener's device. That is not a clever optimisation, it is a direct consequence of choosing to predict notes instead of audio. A 3.3 MB download replaces a per-request bill forever.

3. The part nobody demos

Every commercial music model is either in litigation or has recently settled. As of July 2026:

• Warner settled with both Suno (November 2025) and Udio, converting lawsuits into licensing deals.
• Universal settled with Udio in October 2025 and announced a licensed platform, while continuing to litigate against Suno.
• Sony remains in active litigation against both.
• As part of its Warner settlement, Suno agreed to deprecate models trained on unlicensed music and to restrict downloads on free accounts.
• A summary-judgment hearing in the Massachusetts case against Suno was set for July 2026, with Suno arguing training is transformative fair use. Labels moved to add tens of thousands of recordings after fingerprinting evidence.

Read that list as an engineering constraint rather than news. A model whose training set you cannot document is a model you may have to deprecate. Suno has already agreed to do exactly that. If you are building on top of a third-party model, your product inherits its provenance risk; if you are training your own, provenance is a design decision you make on day one and cannot retrofit.

I hit this myself, immediately

The obvious corpus for this project was The Session's data dump: 54,878 traditional tunes in ABC notation with clean genre, metre and mode metadata. Nearly perfect. Then I read the licence:

"You may not use, adapt, modify, or process the material in any way with Large Language Models. This includes but is not limited to training Large Language Models…"

So I did not use it. Not because a court would necessarily agree that a music transformer is a Large Language Model, but because the entire argument of this project is that provenance is checkable, and you do not get to make that argument on a corpus that told you no. I used the Nottingham Music Database instead — traditional folk repertoire, public-domain compositions, the standard benchmark for exactly this task. It is fifty times smaller. That is the trade, and it is the right one.

4. How Overtone was built

Representation

ABC notation is parsed into events and re-encoded in a REMI-flavoured token language. Time is measured in ticks where a quarter note is twelve, because twelve divides evenly by two, three and four — so eighths, sixteenths and triplets are all exact and nothing is lost to quantisation.

<BOS> STYLE_jig METER_6/8 MODE_maj KEY_0
BAR POS_0 CHORD_G NOTE_67 DUR_6 POS_6 NOTE_69 DUR_6 …
BAR POS_0 CHORD_C NOTE_72 DUR_12 …

Every tune is normalised to C major or A minor first. A 3.3M parameter model should not spend capacity relearning the same melody in twelve keys. Transposition is then reapplied as augmentation, tagged with a KEY_ token, which restores key conditioning and multiplies the corpus twelvefold without adding information the model must memorise.

Tunes parsed

1,034

Musical events

106,036

Vocabulary

225 tokens

Training tokens

4,401,552

Held out

51 whole tunes

The validation split holds out whole tunes, not random windows. If you split by window, fragments of every tune appear in both sets and your validation loss becomes a memorisation score.

Model and training

A plain decoder-only transformer: four layers, 256-dimensional, eight heads, 512-token context, tied input and output embeddings, pre-LayerNorm. 3,339,008 parameters. Trained on one laptop GPU with AdamW, cosine decay and gradient clipping. Nothing here is novel — that is the point. The interesting claim is that this size is sufficient, and the way to support that claim is to make the artefact playable rather than to describe it.

Export and the parity check

Weights are quantised to int8 with one float32 scale per output row and written to a flat binary: 3.39 MB. Per-row scaling matters — a single scale for the whole tensor visibly degrades sampling, because the embedding matrix has very uneven row norms.

A hand-written forward pass and a lossy export are two places where a bug produces output that still sounds plausible, so the two implementations are compared directly on identical inputs:

max abs logit difference : 0.051
mean abs difference      : 0.014
argmax agreement         : 100.0%
top-5 overlap ≥ 4/5      : every position

The browser runtime

No framework, no WASM, no inference library. The model is dequantised into typed arrays and run with ordinary matrix-vector products and a KV cache, which is all single-token decoding needs. It lives in a Web Worker so that sampling never delays audio scheduling. Measured in Chrome: weights load in about 100 ms and sampling runs at 180–280 tokens per second. A bar of this music costs about eighteen tokens, so at a jig tempo the model composes roughly twenty times faster than the music plays — which is why there is always a buffer of written-but-unheard bars to the right of the playhead.

Constrained decoding

The token language has a strict shape: a note is always followed by its duration, positions never move backwards within a bar, and control tokens are legal only in the prefix. An unconstrained sample violates all three occasionally — during development I watched it emit a stray STYLE_jig in the middle of a tune.

So the sampler carries a small grammar that masks illegal tokens before sampling. This makes malformed output structurally impossible rather than merely unlikely, and it costs one pass over a 225-entry vocabulary per token. It is the cheapest quality win in the entire build.

Playing it without glitches

Audio uses the standard lookahead pattern: a timer wakes every 25 ms and schedules any note beginning within the next 200 ms directly against the AudioContext clock, so timing is sample-accurate even when the main thread is busy. Two details that matter in practice — the context window slides by re-priming with the control prefix plus a tail of recent bars when the model approaches 512 positions, and key changes are applied at schedule time rather than generation time, so switching key takes effect on the next bar instead of after the whole buffer drains.

5. Provenance

Training data: the Nottingham Music Database, a long-standing collection of British and American traditional folk tunes in ABC notation, used here via the cleaned machine-readable edition published by Jukedeck. The compositions are traditional repertoire in the public domain and the collection has been the standard benchmark for symbolic music modelling for over a decade.

Honest caveats, because this section is worthless if it only contains the convenient facts. A minority of entries carry contributor or arranger attributions in their source metadata rather than being purely anonymous traditional settings. No audio recordings were used at any point — the corpus is notation only, and the model has never been exposed to a performance. Nothing from The Session was used, for the reason given above.

Novelty is measured rather than assumed: generated token streams are compared against the full training corpus for verbatim n-gram overlap and longest exact common run, so "it learned the style, it did not memorise the tunes" is a number and not a hope.

6. Where the openings are

If you are building a product here and you are not going to out-spend Google on foundation models, the survey above points at four gaps.

Real-time is structurally unavailable to the giants

A 30-second Lyria clip takes nine seconds and a full song takes forty. That is fine for "write me a song" and useless for anything interactive. You cannot put a forty-second render inside a live set, a game engine, or a instrument. Small local models are the only way to get latency below perception, and this demo is an existence proof that the runtime side is tractable. Interactive generation is a different product category, not a faster version of the same one.

Structure is the moat, not timbre

Audio models return a finished mix. The moment a user wants to change one element, you are regenerating the whole thing. Anything that keeps symbolic structure alongside the audio — stems, MIDI, section maps, the lyric alignment Lyria already hands you — enables an editing product rather than a slot machine.

Provenance will become a purchasing requirement

Brands, broadcasters, game studios and sync licensors need to know what a track was trained on. Suno has already agreed to deprecate models trained on unlicensed music. A generator that ships a documented, licence-clean corpus is not making an ethical gesture; it is removing a procurement blocker that currently disqualifies the whole category from a large amount of commercial work.

Cost per generation shapes the whole funnel

Free-tier generosity is what drives adoption in creative tools, and it is precisely what a per-generation bill makes impossible. Any architecture that moves marginal cost toward zero — local inference, symbolic representation, aggressive caching — buys you a product decision that competitors cannot copy without changing their architecture.

7. What this model cannot do

It writes monophonic melody with chord symbols in the idiom of British and Irish traditional dance music, because that is what 1,034 tunes of that repertoire contain. It will not write you a trap beat. It does not sing, it has no timbre of its own, and the sound you hear is a small synthesiser in the page interpreting its notes. It has a 512-token context, so its sense of long-range form comes from re-priming rather than from genuinely holding a whole piece in mind.

What it demonstrates is the part that transfers: corpus diligence, tokenisation, a training loop, an honest validation split, quantised export verified against the reference implementation, a dependency-free client runtime, constrained decoding, and a scheduler that keeps audio clean while a neural network runs in the same tab. Swap the corpus and the same pipeline produces a different idiom. Swap the representation for audio tokens and the same product scaffolding holds, with a GPU bill attached.

The demo is on the front page. It starts making music about a hundred milliseconds after you press the button, and it never plays the same thing twice.