ML instruction sets considered orthognal

2026, Aug, 18

For today's ML compilers, an internal ISA may be RISC or CISC. Just as with traditional computing, CISC, aka. torch/xla/opentvm, is more widely used. Meanwhile RISC (tinygrad, other solutions) is promising yet currently holds a weaker foothold.

ML computing being orthognal to traditional computing is a fact. It's orthogonaliltiy is rather understood: Massively data parallel architectures, tangible arithmetic intensity, and hardware-etched tiled matmuls. If one assumes a Transformer or quasi-Transformer architecture would pervail, the space of possible opeartions shrink even further (Albeit, unless done in a very certain way, etching the transformer arch into an hardware chip may not be the right approach, decode is memory bottlenecked!).

I argue the analogy to traditional computing breaks. contemporary AI is

1) non-turing complete 2) highly speicalized, closer to a DSP than a CPU.

Consequentely, there exists a space for a new instruction kind: Call it OISC: Operation Instruction Set Compute. FlashAttention, in it's modern incarnations is one of the most important kernels in present LLMs, it treats (multi-head) attention as one operation. The same is true for optimized RMSNorm, Attention Residuals, Linear Attention, Embedding, SwiGlu, and on and on.

The above emergent fact can lead to the design & implementation of vastly more performant, and perhaps simpler, ML systems. If done with efficacy, it can also enable greater inter-accelerator interopabillity; It's a daunting task to implement thousands of ops for a new accelerator, many of which lack necessary documentation. It's a vastly less daunting task to implement a dozen ops, alas in that case, one is at the mercy of the (hopefully search) optimizer.

It's a well tuned task to implement fused, optimized, implementation for core transformer building blocks, dependent on an intricate knowledge of the accelerator's inner working.

* The definition of a Megakernel is, as of writing, fuzzy: is it merely in-gpu scheduling? More aggresive kernel fusion? and therefore not taken into account
* An "operation kernel" does not merely intent a forward pass, but a backward pass and autograd too.
* One can argue the analogy between Hardware ISAs and internal compiler intermidiate representations is incomplete. I think it's flawed.