AI models are trained and served on fleets of accelerators that cost enormous sums to build and run, and how the software around those chips is configured has a large effect on how much work they get done. That's the problem in AI infrastructure I find most interesting right now, and I think the most promising way to solve it is an LLM agent trained with reinforcement learning on measured performance.

This isn't a neglected problem. Published accounts from Meta, Google, and AMD describe the work today as careful manual tuning, trial and error, and costly search, and recent work from Stanford, Berkeley, and others applies LLMs to systems performance problems.

Full disclosure, I don't work at an AI lab and have no insider knowledge of how any lab handles this internally. I just have a deep interest in this at the moment. Everything here comes from public papers and reports, my own research, and my opinions and thoughts on the matter.

How much compute goes unused

The usual way to measure this is model FLOPs utilization, or MFU. Every chip has a maximum amount of math it can do per second, and MFU is the share of that maximum that actually goes into the model's own work, rather than waiting on memory, waiting on other chips, or sitting idle. Even carefully tuned training runs at the largest scale use only around half of what their chips can do. When Meta trained Llama 3, one of its largest models, on up to 16,384 GPUs, it reported an MFU of 38 to 43 percent, and NVIDIA's own Megatron-LM team reached 52 percent.

Better configuration can recover a meaningful part of that. AMD's tuning agent found a setup that got 27 percent more work out of the same four servers with no new hardware, though its run used a faster number format than the reference it was compared against.

Running a trained model for users, called serving, has the same problem. When a chatbot answers you, it keeps a running memory of the conversation so far, called the KV cache. The Berkeley team behind vLLM found that earlier serving systems wasted most of that memory, and that managing it better let the same hardware do 2 to 4 times more work. Changing how requests are scheduled helps too, and Sarathi-Serve, from Microsoft Research and Georgia Tech, delivered up to 2.6 times more capacity than vLLM on the same chip.

A large model doesn't fit on one chip, so it has to be split across many, and there are several ways to split it (by layers, by slicing each layer, by giving copies of the model different data, and more) that are usually combined. On top of that come choices like how much data each chip processes at once, how much memory goes to the KV cache, how incoming requests are grouped and scheduled, which low-level GPU programs (called kernels) run the math, and how chips share results over the network. Each of these changes how much useful work the same hardware delivers.

Why the gap persists

The number of possible combinations is enormous, and the settings affect each other. AMD's engineers note that teams still search for good setups through trial and error, burning expensive GPU hours on runs that crash, and NVIDIA's Megatron-LM work showed that the right combination depends on both the model and the hardware it runs on.

The best answer also keeps moving. Vidur, a simulator from Microsoft Research and Georgia Tech, found that the best setup depends on the kind of traffic a model serves, so a setup tuned for one workload can cost up to twice as much on another.

Alpa, from Berkeley and Google, planned how to split models across chips as well as hand-tuned systems did. But Google's ASAP team describes today's options for training as manual tuning or costly search that tests settings without understanding why they work, and says both struggle to keep up with how quickly AI models change.

Why an LLM, and why reinforcement learning

The agent is an LLM that reads a description of the job, the hardware, the current settings, and whatever the system's monitoring tools and logs report, then proposes a change. Each change is measured for throughput (how much work gets done), latency (how long each request takes), and cost. Those measurements become the reward in reinforcement learning, a training method where a model tries things, gets scored, and is updated to do more of what scored well. Over time it learns which proposals actually work, rather than which ones merely sound plausible.

I think an LLM is a strong starting point because most of what an engineer reads while tuning these systems is text, whether that's configuration files, logs, timing reports, error messages, or documentation, and a model can read all of it and reason about why a run was slow. In AMD's tuning agent, the LLM proposed combinations of settings that a tool adjusting one setting at a time couldn't reach.

A closely related task shows what training adds. Stanford's KernelBench found that top AI models, used out of the box, matched the speed of standard code in fewer than 20 percent of cases. Training on measured speed changes that. With reinforcement learning, Kevin, an AI model from Cognition, went from kernels about half as fast as the standard code to slightly faster ones, and CUDA-L1 reached about three times faster on average. Closer to this problem, Notre Dame's AutoLLMResearch uses reinforcement learning to train an AI agent that sets up expensive AI experiments by learning from cheap ones, though it tunes model design and training choices rather than the systems the models run on.

Berkeley researchers argue that performance problems are well suited to AI-driven discovery because answers can be checked, since a proposed fix can be run on a real system or a simulator and measured. Google DeepMind's AlphaEvolve showed what that kind of loop can find, a scheduling rule that recovers an average 0.7 percent of the computing power across Google's fleet and a change that cut Gemini's training time by 1 percent.

Why this hasn't been done yet

If this approach is worth pursuing, the obvious question is why the labs haven't done it already. The honest answer is that I don't know whether they have, since labs don't publish everything. What the public record does show is that the pieces this approach depends on are recent. DeepSeek's GRPO, from 2024, made reinforcement learning for LLMs cheaper by scoring a group of the model's answers against each other instead of training a separate model to judge them. Kevin and CUDA-L1, which showed that LLMs trained on measured speed can write faster kernels, came out in 2025. And simulators checked against real hardware, like Vidur from 2024 and the projection engine behind AMD's tuning agent from 2026, make trials far cheaper than running them for real, which matters because a single search on real hardware can cost about $218,000.

As for why so much of this is still done by hand, the published accounts point to three reasons. The settings interact so tightly that whether a setup even fits in memory can come down to a single byte, so blind search burns GPU hours on runs that crash. Automated search is slow and expensive, and as noted earlier, Google's ASAP team says both manual tuning and black-box search struggle to keep up with how quickly AI models change. And engineers need to understand what's changing on systems this expensive, which is part of why Google DeepMind chose readable code over a reinforcement learning model for Google's scheduler. That concern applies less to an LLM agent like the one I'm describing, because what it produces is a configuration change written as text, which an engineer can read and review before it runs. That said, AI is advancing fast, and it's only accelerating. It's possible the labs are already working on this problem, but I haven't seen evidence of it.

The reinforcement learning in published work so far is also aimed at narrower targets. Kevin and CUDA-L1 train LLMs to write kernels, not to configure the job itself. Learn to Shard, from Yale and Microsoft Azure, uses reinforcement learning to decide how to split AI models for serving, but it trains a purpose-built network rather than an LLM. Google's ASAP and AMD's agent take a different route, using LLMs as they come, guided only by instructions.

What hasn't been published is an LLM, trained with reinforcement learning, making these configuration decisions itself across how models are split across chips, how requests are grouped, how memory is used, and how chips communicate, for both training and serving. That's the question I'm pursuing. Each piece it depends on has now been shown to work on its own, and the open challenges are what make putting them together hard.

Open challenges

Safety on live systems. Changing the settings of a running system can break it, and large training runs already fail often. Meta logged 419 unexpected interruptions over 54 days of Llama 3 training. AMD's agent reduces the risk by rejecting setups that won't fit in memory before running them.

Reward design. A model trained to chase a score can find ways to raise the score without doing the real task, which is known as reward hacking. Kevin's authors saw this happen, with smaller models copying the reference code instead of writing their own. Measuring speed fairly is also hard. AutoTuneBench found that the best result in its study looked 10.6 times faster against a weak comparison but only about 2 times faster against a fair one.

Generalization. The goal is an agent whose lessons carry over to new models and new hardware. CUDA-L1 reports that its kernels carry over across GPU generations, but whether a configuration agent can move to a new generation of chips with little retraining is something I'll have to test.

Evaluation cost. Simulators are what make enough trials affordable. Vidur predicts serving speed to within 9 percent and found the best setup for a large model in one hour on a regular processor, with no GPUs. ASTRA-sim does the same kind of modeling for training, and AMD's projections stay within about 10 percent of real measurements. Its authors note that the predictions only need to rank setups in the same order the real hardware would.

Measuring success

  • Throughput, latency under a set response-time target, and cost per token (the unit of text a model reads or writes), compared against default settings, standard search methods, a strong off-the-shelf model given the same tools, and specialized search tools.
  • How many real measurements the agent needs to reach a good configuration, since fewer expensive trials is the whole point.
  • How often it proposes something that crashes or runs out of memory.
  • How well it performs on models, workloads, and hardware it never saw during training.
  • Whether gains found in simulation hold up on real GPUs.
  • All of it under a fixed measurement process the agent can't modify, with fair comparisons and repeated runs.

If it can't beat a well-tuned conventional search under those conditions, the approach doesn't hold up.

Starting point

The natural starting point is serving, since Vidur makes it possible to run many experiments without large-scale hardware. That means building the test environment and setting fair baselines first, then training a small open-weight model (one whose trained weights are publicly available) with GRPO to see whether it beats them, with the best configurations confirmed on real GPUs. Training jobs come next, using ASTRA-sim.

References

  1. Grattafiori, A., Dubey, A., et al. "The Llama 3 Herd of Models." arXiv, 2024. arxiv.org/abs/2407.21783
  2. Narayanan, D., Shoeybi, M., Casper, J., et al. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." SC, 2021. arxiv.org/abs/2104.04473
  3. Raina, A., Razaghi, P., Chen, Y., et al. "Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training." NeurIPS, 2026. arxiv.org/abs/2610.19423
  4. Kwon, W., Li, Z., Zhuang, S., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP, 2023. arxiv.org/abs/2309.06180
  5. Agrawal, A., Kedia, N., Panwar, A., et al. "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve." OSDI, 2024. arxiv.org/abs/2403.02310
  6. Agrawal, A., Kedia, N., Mohan, J., et al. "Vidur: A Large-Scale Simulation Framework for LLM Inference." MLSys, 2024. arxiv.org/abs/2405.05465
  7. Zheng, L., Li, Z., Zhang, H., et al. "Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning." OSDI, 2022. arxiv.org/abs/2201.12023
  8. Ding, Y., Chen, X., Zhang, X., Zhou, Z. "ASAP: An Agentic Solution to Auto-optimize Performance of Large-Scale LLM Training." NeurIPS Workshop on ML for Systems, 2025. arxiv.org/abs/2511.03844
  9. Ouyang, A., Guo, S., Arora, S., et al. "KernelBench: Can LLMs Write Efficient GPU Kernels?" arXiv, 2025. arxiv.org/abs/2502.10517
  10. Baronio, C., Marsella, P., Pan, B., Guo, S., Alberti, S. "Kevin: Multi-Turn RL for Generating CUDA Kernels." arXiv, 2025. arxiv.org/abs/2507.11948
  11. Li, X., Wang, A., Wang, G., Li, J., Shum, C. "CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning." ICLR, 2026. arxiv.org/abs/2507.14111
  12. Guo, T., Chawla, N. V., Wiest, O., Zhang, X. "AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration." arXiv, 2026. arxiv.org/abs/2605.11518
  13. Cheng, A., Liu, S., Pan, M., et al. "Barbarians at the Gate: How AI is Upending Systems Research." arXiv, 2025. arxiv.org/abs/2510.06189
  14. Novikov, A., Vũ, N., Eisenberger, M., et al. "AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery." arXiv, 2025. arxiv.org/abs/2506.13131
  15. Shao, Z., Wang, P., Zhu, Q., et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv, 2024. arxiv.org/abs/2402.03300
  16. Yin, R., Mishra, S. D., Zuo, X., et al. "Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference." arXiv, 2025. arxiv.org/abs/2509.00217
  17. Chen, L. "AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines." arXiv, 2026. arxiv.org/abs/2609.18123
  18. Won, W., Heo, T., Rashidi, S., et al. "ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale." ISPASS, 2023. arxiv.org/abs/2303.14006