Robotic systems that are developed for social and dynamic environments require adaptive
mechanisms to successfully operate. Consequently, learning from rewards has provided meaningful results in
applications involving human-robot interaction. In those cases where the robot’s state space and the number
of actions is extensive, dimensionality becomes intractable and this drastically slows down the learning
process. This effect is specially notorious in one-step temporal difference methods because just one update
is performed per robot-environment interaction. In this paper, we prove how the action-based learning of a
social robot can be improved by combining classical temporal difference reinforcement learning methods,
such as Q-learning or Q(λ), with a probabilistic model of the environment. This architecture, which we
have called Dyna, allows the robot to simultaneously act and plan using the experience obtained during real
human-robot interactions. Principally, Dyna improves classical algorithms in terms of convergence speed and
stability, which strengthens the learning process. Hence, in this work we have embedded a Dyna architecture
in our social robot, Mini, to endow it with the ability to autonomously maintain an optimal internal state while
living in a dynamic environment.