A working monograph
Notes on the method.
The design, training, and evaluation of eight generations of melody models: the evolution from a baseline LSTM to a 34-million-parameter Transformer with REMI tokenisation and BPE compression, and the observations gathered along the way.
Symbolic, not audio
Tools like Suno and Udio generate finished audio: a complete waveform, like a rendered image. You can listen to it, but you can't reach in and move a note, change a chord, or rewrite the bridge. The audio is the final product.
The Melograph generates symbolic music: MIDI sequences where every note is a discrete event with a pitch, a start time, a duration, and a velocity. It is the difference between a screenshot and an SVG: one is flat pixels, the other is individual shapes you can select, move, and restyle. You can transpose sections, swap instruments, adjust timing, and drop the output straight into a DAW like Ableton or Logic as a starting point for real composition.
Formally, the task is next-token prediction over symbolic music sequences: given a sequence of tokens representing musical events, the model predicts a distribution over the vocabulary for the next token, and novel pieces are sampled from it autoregressively. The model has never heard audio. It learns structure (which notes tend to follow which, how phrases build, where tension resolves) entirely from the symbolic representation. The audio on this site is just a MIDI-to-WAV render of what the model actually writes.
Prior work
Attention mechanisms for music
Huang et al. (2019) demonstrated that self-attention captures long-range musical structure more effectively than recurrent networks. Their Music Transformer used relative positional encoding to learn interval relationships rather than absolute positions, which is musically motivated: a perfect fifth is a perfect fifth regardless of the starting pitch.
This motivated the transition from LSTM to Transformer in op. 7. The implementation here uses Rotary Position Embeddings [Su et al., 2021] instead of the original relative-attention scheme: RoPE encodes position by rotating query and key vectors, producing a natural attention decay with distance at lower cost.
REMI tokenisation
Huang and Yang (2020) introduced REMI (REvamped MIDI-derived Events): musical events as typed tokens: Bar, Position, Pitch, Velocity, Duration, Tempo, Chord. A note is no longer a single pitch token but a structured group encoding when it occurs, how loud it is, and how long it lasts. Adopted in op. 6 via the MidiTok library, with 32 velocity levels, chord detection, tempo tokens, and rest tokens.
Subword compression for music
Fradet et al. (2023) applied Byte Pair Encoding [Sennrich et al., 2016] to REMI sequences: the most frequent token pairs merge into single tokens, compressing common note events into atomic units and increasing the musical context visible in a fixed window. Adopted in op. 7 (512-token vocabulary), expanded to 1,024 in op. 8.
Architectural components from language modelling
The Transformer borrows standard components from LLM research: SwiGLU feed-forward layers [Shazeer, 2020], RMSNorm [Zhang and Sennrich, 2019], weight tying [Press and Wolf, 2017], and warmup-then-cosine learning-rate decay [Loshchilov and Hutter, 2016].
How music becomes tokens
Three tokenisation strategies across the eight studies, each encoding progressively more musical information.
Op. 1 was the first experiment, a proof of concept since lost to the ether. Everything here starts from op. 2.
Pitch strings
Notes as music21 pitch names ("C4", "F#3"), chords as dot-separated integers. No timing, no velocity. Temporal structure learned purely from position in the sequence.
59–1,279
tokens
REMI
Typed tokens: Bar, Position, Pitch, Velocity (32 levels), Duration, Tempo, Chord, Rest. Vocabulary decreased despite encoding strictly more information, because the typed scheme beats enumerating every observed combination.
362
tokens
REMI + BPE
BPE merges frequently co-occurring token groups, often a full note event, into single tokens. A 256-token window covers roughly two to three times more music than raw REMI.
512–1,024
tokens
The instruments
MelodyLSTM
OPP. 1–6
Three-layer LSTM. The early studies used hidden units of [256, 512, 256] with float-normalised pitch input; op. 5 widened to [512, 512, 512]; op. 6 added a learned 128-dimensional embedding layer, replacing float normalisation, and was the first version trained natively in PyTorch (earlier studies were TensorFlow/Keras, later converted).
The LSTM reads sequences of length 100 with stride 1 during training. At inference, a seed phrase is extended autoregressively through a sliding window.
MusicTransformer
OPP. 7–8
An 8-layer decoder-only Transformer: 8 attention heads, 512 model dimension, 2048 feed-forward dimension. About 34 million parameters, with causal masking and KV caching at inference.
RoPE
Position is encoded by rotating query and key vectors, so attention decays naturally with distance.
SwiGLU
Gated feed-forward: W₂(SiLU(xW₁) ⊙ xW₃). Three projections give finer control of information flow.
RMSNorm
Pre-norm without mean centering, before both attention and feed-forward sublayers.
Weight tying
Input embeddings share weights with the output projection: fewer parameters and built-in regularisation.
Training
The data pipeline
MIDI files split 90/10 into training and validation before any augmentation, so no transposed copy of a validation piece can leak into training. Augmentation is pitch transposition within ±6 semitones, up to two copies per file, which roughly triples the data while preserving every structural relationship.
Token sequences are sliced into fixed windows: length 100, stride 1 for the LSTM (single next-token target); length 256, stride 64 for the Transformer (causal language-modelling loss at every position).
Hyperparameters
LSTM · OP. 6
TRANSFORMER · OPP. 7/8
Training data
| Study | Genre | Tracks | Tokeniser | Vocab |
|---|---|---|---|---|
| Op. 2 | R&B / 90s hip-hop | 24 | Pitch strings | 59 |
| Op. 3 | Dance | ~200 | Pitch strings | 635 |
| Op. 4 | Jazz | ~120 | Pitch strings | 1,279 |
| Op. 5 | Mixed | 275 | Pitch strings | 629 |
| Op. 6 | Mixed | 275 | REMI | 362 |
| Op. 7 | Mixed | 275 | REMI + BPE | 512 |
| Op. 8 | Mixed + classical | 275 + MAESTRO | REMI + BPE | 1,024 |
Op. 8 adds the MAESTRO dataset (v3): 1,276 competition piano performances.
Controlling randomness
0.8
0.1 – 2.0
Temperature
Divides logits before softmax. Below 1.0 the distribution sharpens and the music plays it safe; above 1.0 it flattens and the music takes chances.
50
0 – 500
Top-K
Keeps only the k highest-probability tokens and discards the rest, which prevents sampling from the incoherent long tail.
0.95
0.01 – 1.0
Top-P (nucleus)
Includes tokens until cumulative probability exceeds p. Adapts dynamically: fewer candidates when the model is confident, more when it is uncertain.
Measuring the music
Perplexity
exp(cross-entropy) on held-out validation data: how well the model predicts the next token of pieces it never saw.
N-gram repetition
Measured at 2, 4, 8, and 16-gram lengths over generated sequences. High repetition means the model is stuck in loops.
Compression ratio
Unique n-grams over total n-grams in generated output. Near 1.0 is diverse; low is heavy repetition. A quick training-time diagnostic.
Pitch distribution
Histogram comparison against training data: catches mode collapse and out-of-range drift. A well-trained model approximates the distribution without memorising it.
The studies
Eight studies trained in sequence, each changing one or two variables from the last.
Baseline LSTM
LSTM [256, 512, 256] · float input · TF/Keras
Initial proof of concept: float-normalised pitch input, no embedding layer. Vaguely musical output without coherent phrasing. Since lost, but it established the approach was viable.
Genre-specific data
LSTM [256, 512, 256] · float input · TF/Keras
Trained on 24 R&B/hip-hop tracks. Vocabulary of 59 tokens.
More data
LSTM [256, 512, 256] · float input · TF/Keras
Around 200 dance tracks; vocabulary 635. Better diversity, but repetitive loops, as pitch strings struggled with the harmonic variety.
Jazz
LSTM [256, 512, 256] · float input · TF/Keras
Around 120 jazz tracks pushed vocabulary to 1,279; the combinatorial explosion of chord voicings became a clear bottleneck.
Wider, mixed-genre
LSTM [512, 512, 512] · float input · TF/Keras
All 275 tracks combined; hidden units widened. Acceptable output, but the ceiling of pitch-string encoding had been reached.
PyTorch, embeddings, REMI
LSTM [512, 512, 512] · embed 128 · REMI · PyTorch
Rewritten in PyTorch with learned embeddings and REMI tokenisation. Vocabulary fell from 629 to 362 while encoding strictly more. Output improved substantially, confirming the input representation had been the real constraint.
The Transformer
Transformer 8L/8H/512d · REMI + BPE 512
RoPE, SwiGLU, RMSNorm, weight tying; BPE over REMI; sequence length 256. Early-stopped at epoch 45, best validation loss 0.0952.
Bigger vocabulary, bigger data
Transformer 8L/8H/512d · REMI + BPE 1024
Same architecture; BPE doubled to 1,024; MAESTRO added. Ran all 100 epochs without early stopping, a sign there was finally enough data to keep learning throughout.
Findings
Tokenisation mattered more than model capacity
The move from op. 5 to op. 6 changed framework, input representation, and tokenisation at once, yet the decisive shift was tokenisation: vocabulary fell from 629 to 362 while encoding strictly more musical information. As in language modelling, representation beat scaling at this size.
Genre-specific training had diminishing returns at small scale
With fewer than 300 files in total, combining genres (op. 5) beat the genre-focused models (opp. 2–4). Genre focus becomes worthwhile again at thousands of tracks per genre, which is exactly what the ninth study prepares.
Widening the LSTM bought little
The wider hidden layers of op. 5 improved output modestly; the pitch-string representation and the recurrent bottleneck were the true limits. Attention removed that bottleneck in op. 7 by connecting any pair of positions directly.
BPE bought context almost for free
A 256-token BPE window covers two to three times the music of raw REMI at negligible cost. Op. 8 doubled the vocabulary to absorb the more complex patterns in MAESTRO.
Augmentation was necessary at this data scale
Transposition within ±6 semitones roughly tripled the data and loses nothing musically. Without it, the models overfit within a few epochs.
Op. 8 never early-stopped
Op. 7 stopped at epoch 45. Op. 8, the same architecture with more data, used all 100 epochs. Data, not capacity, set the ceiling.
Limitations, and the ninth study
Single-voice generation
The models write one melodic line. Multi-track generation (melody, bass, chords, drums) would need interleaved token streams or a multi-stream architecture.
Training data, then and next
The first eight studies trained on 275 files plus MAESTRO, which are full arrangements rather than melodies. The corpus for the ninth study is being assembled now: over 20,000 true single-voice melodies from POP909, Wikifonia lead sheets, and skyline-extracted Lakh MIDI, tagged by genre.
Conditioning arrives with op. 9
Key, tempo, and style are accepted by the API today but ignored by the models. The ninth study trains style condition tokens in, so that one model can serve every genre in the dropdown.
Evaluation is automated
Perplexity and repetition metrics are useful diagnostics but do not measure musicality. Human evaluation, or a learned critic, remains open work.
No music-aware positions
RoPE encodes sequence position rather than beat or bar structure. Combining it with music-aware relative encoding could improve harmonic coherence.
No feedback loop
The songbook stores compositions but not opinions of them. Ratings could one day provide signal for preference-based fine-tuning.
Bibliography
Huang, C-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A., Hoffman, M., Dinculescu, M., Eck, D. (2019). Music Transformer: Generating Music with Long-Term Structure. arXiv
Huang, Y-S., Yang, Y-H. (2020). Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions. arXiv
Fradet, N., Gutowski, N., et al. (2023). Byte Pair Encoding for Symbolic Music. arXiv
Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C-Z. A., Dieleman, S., Elsen, E., Engel, J., Eck, D. (2019). Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset. arXiv
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv
Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv
Zhang, B., Sennrich, R. (2019). Root Mean Square Layer Normalization. arXiv
Press, O., Wolf, L. (2017). Using the Output Embedding to Improve Language Models. arXiv
Sennrich, R., Haddow, B., Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv
Loshchilov, I., Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv