Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Abstract

Prompt learning is a popular parameter-efficient method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks. In this paper, we introduce a Diffusion Meta-Prompt (DMP) model, a framework that models the distribution of learned prompts using diffusion models. DMP is trained only on a repository of previously learned prompts to synthesize new prompts conditioned on natural language task descriptions, without access to the task data.

To improve sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes. DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training.

Key Results: DMP reduces storage and inference costs by over 90% compared to prompt retrieval methods. For composite classification, DMP achieves up to 2.0% average gain over prior meta-learning methods across 55 pairs of datasets with gains as high as 8.5% on specific pairs. DMP also enhances cross-task generalization with ~2-9% improvement for hierarchical classification tasks.

Key Contributions

• Diffusion Meta-Prompting Framework: A meta-learning approach that treats learned prompts as samples from a shared distribution, learned with diffusion models. Unlike existing methods, DMP is trained once on a prompt repository and synthesizes new task-specific prompts without access to task data or losses.
• Test-Time Steering: A novel steering mechanism that produces prompts with better generalization (up to 2.0%) without additional training, using training-selected anchors to guide diffusion sampling.
• Theoretical Guarantee: We provide a bound on the expected task loss of prompts sampled from a DMP in terms of the empirical loss of the prompt repository and the denoising error.
• State-of-the-Art Results: Extensive experiments show DMP improves generalization in challenging regimes including cross-task and composite classification (up to 2.0% gains), while reducing storage and inference costs by over 90%.
• Representation-Agnostic: DMP applies beyond prompts to foundation-model representations, demonstrated through sparse multi-view retrieval experiments (+3.44 mAP improvement).

Method

Figure 2: DMP framework for Text-to-Prompt synthesis (left) and Prompt Variation synthesis (right).

Diffusion Meta-Prompting

DMP models the distribution of learned prompts using diffusion models. Given a repository of prompts learned via standard prompt learning techniques, DMP trains a conditional generator p_θ(x | y) that synthesizes new prompts conditioned on natural language task descriptions.

Test-Time Steering

To improve sampling stability, we introduce a steering mechanism that biases diffusion sampling toward high-performing prompts. For each dataset, we select the prompt with highest training accuracy as an anchor. During sampling, the predicted clean prompt is interpolated with this anchor at each denoising step, stabilizing generation without requiring test data access.

Concept Composition

By framing diffusion models as Energy Based Models, DMP enables task composition through classifier-free guidance. Multiple concepts can be combined by averaging noise predictions, allowing for flexible prompt combinations at inference time.

Results

Composite Classification

DMP achieves +4.4% average gain over CoOp across 55 dataset pairs, with improvements of +4.3% on base classes and +5.7% on novel classes.

Hierarchical Classification

DMP prompts generalize better than BPL with ~2-9% accuracy gains for cross-task generalization on taxonomic classification.

Variation Synthesis

DMPVariation demonstrates superior robustness and generalization compared to Textual Inversion, effectively adapting to diverse contexts while preserving subject identity.

Efficiency Comparison

Qualitative Results

Concept Composition

DMP enables seamless composition of multiple concepts (e.g., identity + attributes) without explicit training.

Negative Prompting

DMP supports negative prompting, generating images with attributes opposing specified concepts.

Cross-Context Generalization

DMPVariation prompts generalize to diverse contexts while Textual Inversion tends to overfit.

Comparison with Prior Work

Citation

@inproceedings{sridhar2026dmp,
  title={Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation},
  author={Sridhar, Deepak and Li, Yi and Bhardwaj, Kartikeya and Liu, Shuangjun and Jing, Taotao and Li, Yuan and Zhang, Shuai and Lyu, Jiancheng and Gao, Dashan and Vasconcelos, Nuno},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2026}
}

Code & Models

Code and trained models will be released publicly. Stay tuned!

Acknowledgements

This work was supported by Qualcomm Technologies. We thank the NeurIPS reviewers for their valuable feedback.