LLMCompass: Evaluation and Resource-Aware Model Selection

Thu, 01 Jan 2026 00:00:00 +0000

University of Central Florida — ongoing

Research question: how can an AI system select a model that satisfies task-specific capability requirements while jointly accounting for quality, uncertainty, latency, memory, throughput, and monetary or computational cost?

Curate skill-oriented evaluations for summarization, question answering, reasoning, instruction following, and related capabilities; compare rankings across datasets, prompts, judges, and deployment settings.
Study generalization of model rankings to unseen tasks and evaluate whether intent-to-skill mappings provide reliable evidence for model recommendation.
Develop reproducible inference and data infrastructure as a byproduct of the research, including model execution, metadata capture, ranking, and deployment-plan generation.

Model Selection |

LLMCompass: Evaluation and Resource-Aware Model Selection