• Classified by Topic • Classified by Publication Type • Sorted by Date • Sorted by First Author Last Name • Classified by Funding Source •
Centralized Critics for N-Agent Ad Hoc Teamwork.
Andrew McAlinden, Caroline
Wang, Steven Shi, Arrasy Rahman, Elad
Liebman, and Peter Stone.
Technical Report arXiv e-Prints 2610.34799,
arXiv, 2026.
arXiv version
(unavailable)
Centralized training with decentralized execution (CTDE) has become the dominant paradigm in multi-agent reinforcement learning (MARL), motivating extensive study of what privileged information should be made available to the critic during training. While prior work has examined the role of global state in fully cooperative settings, the design of centralized critics for N-agent ad hoc teamwork - in which dynamically varying set of learning agents must coordinate with an arbitrary set of uncontrolled teammates drawn from an unknown distribution - remains poorly understood. We study this question by comparing two natural classes of advantage estimators: controlled history estimators, which condition only on the observation and action histories of the learner's own agents, and joint history estimators, which additionally condition on the histories of the uncontrolled teammates. We prove that both estimators are unbiased and characterize the conditions under which the joint history estimate attains strictly lower variance than its controlled counterpart. Empirically, joint history critics match or outperform controlled history critics across a range of NAHT tasks, yielding improved sample efficiency and final returns. Together, these results provide both a theoretical justification and an empirical recommendation for incorporating uncontrolled agent histories into centralized critics for N-agent ad hoc teamwork.
@Techreport{mcalinden26,
author = {Andrew McAlinden and Caroline Wang and Steven Shi and Arrasy Rahman and Elad Liebman and Peter Stone},
title = {Centralized Critics for N-Agent Ad Hoc Teamwork},
institution = "arXiv",
number = "arXiv e-Prints 2610.34799",
year = {2026},
month = {October},
abstract = {
Centralized training with decentralized execution (CTDE)
has become the dominant paradigm in multi-agent
reinforcement learning (MARL), motivating extensive
study of what privileged information should be made
available to the critic during training. While prior
work has examined the role of global state in fully
cooperative settings, the design of centralized critics
for N-agent ad hoc teamwork - in which dynamically
varying set of learning agents must coordinate with an
arbitrary set of uncontrolled teammates drawn from an
unknown distribution - remains poorly understood. We
study this question by comparing two natural classes of
advantage estimators: controlled history estimators,
which condition only on the observation and action
histories of the learner's own agents, and joint history
estimators, which additionally condition on the
histories of the uncontrolled teammates. We prove that
both estimators are unbiased and characterize the
conditions under which the joint history estimate
attains strictly lower variance than its controlled
counterpart. Empirically, joint history critics match or
outperform controlled history critics across a range of
NAHT tasks, yielding improved sample efficiency and
final returns. Together, these results provide both a
theoretical justification and an empirical
recommendation for incorporating uncontrolled agent
histories into centralized critics for N-agent ad hoc
teamwork.},
wwwnote={<a href="https://arxiv.org/abs/2610.34799">arXiv version</a>},
}
Generated by bib2html.pl (written by Patrick Riley ) on Sun Nov 01, 2026 16:10:06