Reinforcement Learning for Optimal HVAC Control: from Theory to Real-world Applications

The HVAC system accounted for a significant portion
of the building’s energy consumption, resulting in enormous
CO2 emissions. Among the numerous HVAC control methods,
reinforcement learning (RL) gives the ability to control complex
systems without requiring an explicit model of the building’s
thermal dynamics. This study conducted a concise review of
previous research on the application of RL to HVAC systems
in buildings, it offered a thorough explanation of the theoretical
foundations of RL and a summary of several recent RL studies
that employ a particular variant of each main component of
the RL: environment, state-space, action-space, rewards function,
number of time steps and training episodes. Most studies
construct the training environment as a stationary MDP due
to the use of a predefined single sequence of transitions for
non-action-controllable state vector components (e.g., outdoor
temperature and occupancy schedule). This type of MDP is solved
using tabular RL and DRL algorithms. Future research should
focus on using the Meta-RL approach for HVAC systems, which
solves the problem of non-stationarity in the environment (nonstationarity-
MDP) where the sequence of transitions for (outdoor
temperature and occupancy schedule) are changeable, as is the
case during the real-world implementation of the RL controller
in buildings.