跳到正文
DEV Community· Bohdan Hordii·· 2026-10-19AI 評分32

Architectural Deep-Dive: Language Models Learn to Improve Language Models

Architectural Deep-Dive: Language Models Learn to Improve Language Models

AI 導讀

LM2LM 以研究世界模型(RWM)和雙迴路驗證,為透過強化學習優化模型、提示詞或神經網絡架構的智能體提供執行結果錨定的回饋機制。凍結的 LLM 根據上下文中的實驗紀錄預測提案收益,並定期以硬件執行結果更新紀錄,校正預測偏差。此方法以代理預測篩走底部 90% 的候選項,只對排名靠前的提案進行實際驗證,以減少計算開支。

正文

Bohdan Hordii

Breaking Recursive Self-Improvement: How Research World Models Eliminate GPU Bottlenecks

Autonomous engineering systems that optimize their own code, prompts, or neural architectures inevitably encounter the same structural dead-end: the feedback cost dilemma.

To train an agentic optimizer via reinforcement learning (RL), you need reward signals for tens of thousands of generated proposals. If each proposal requires a physical compilation, benchmark suite, or model-training run, reward collection costs spiral into hundreds of thousands of compute hours. Conversely, substituting real execution with a lightweight neural surrogate triggers Goodhart's Law. As the optimizer shifts its proposal distribution, it exploits blind spots in the surrogate, yielding high predicted rewards that catastrophically fail in physical production.

The LM2LM framework introduces a production-ready blueprint that solves this failure mode through active contextual grounding.


The Architecture: Dual-Loop Grounding

Rather than training a monolithic reward model that rapidly drifts, LM2LM structures the feedback pipeline into an active feedback loop:

  +-------------------------------------------------------------+
  |                      LM2LM OPTIMIZER                        |
  |  Generates LM / System Optimization Proposals via RL        |
  +------------------------------+------------------------------+
                                 | Proposal Candidate
                                 v
  +-------------------------------------------------------------+
  |                RESEARCH WORLD MODEL (RWM)                   |
  |  Frozen In-Context Predictor (Real Execution History Cache)  |
  +------------------------------+------------------------------+
                                 |
             +-------------------+-------------------+
             | Top-Ranked Candidates                 | Sub-optimal
             v                                       v
  +----------------------+                 [ Discard / Deprioritize ]
  | HARDWARE EXECUTION   |
  | Isolated Sandbox     |
  +----------+-----------+
             | Ground Truth Telemetry
             v
  [ Append to RWM Context ] ---> Closes the Distribution Shift Loop

1. In-Context Research World Model (RWM)

Instead of fine-tuning a rigid regressor, a frozen LLM functions as an empirical predictor. Real past experimental records (hyperparameters, structural changes, and measured delta gains) are injected directly into its context window. It predicts the expected gain of new proposals based on demonstrated trajectory history.

2. Periodic Empirical Synchronization

To prevent the RL policy from drifting out-of-distribution (OOD) where the RWM might overestimate gains, the system enforces periodic hardware checkpoints. Proposals ranking above an empirical confidence threshold are dispatched to real sandbox runs. The measured metrics are dynamically appended to the RWM's context buffer, immediately correcting calibration drift.


Production Takeaways for Autonomous Agent Stacks

  1. Stop Blind Prompt and Code Self-Tuning: If your agent writes optimization code or refactors pipeline configurations without an execution-anchored reward model, your policy will quickly learn to game static unit tests.
  2. Contextual History as an Adaptive Value Function: Leveraging high-capability frozen models with rigorous execution logs in-context provides better domain adaptation than fine-tuning small reward models on stale offline traces.
  3. Bounded Physical Verification: Reserve physical execution strictly for the top percentile of candidate mutations. Use surrogate prediction to filter the bottom 90% of architectural noise, cutting compute expenditure while maintaining strict convergence guarantees.

This balance of surrogate filtering and deterministic validation represents the baseline required for reliable, autonomous recursive engineering.


Originally published at https://bohdanhordii.dev

Author: Bohdan Hordii | Website: bohdanhordii.dev | Contact: @bohdan_hordii

來源:DEV Community · dev.to