CS285 DRL Notes-Lecture 1 Introduction

This is the note of Berkeley CS285 taught by Sergey Levine. Lecture 1 introduces reinforcement learning (RL) as a framework for learning decisions from experience.

The whole lecture can be understood through one idea:

A policy does not merely predict an answer. Its action changes the world, which changes the data the policy will see next.

RL is the study of learning inside this feedback loop.

RL is an agent-environment feedback loop

At time step $t$:

• The agent receives a state $s_t$, or an observation $o_t$ when the full state is hidden.
• A policy chooses an action $a_t$.
• The environment produces a reward $r_t$ and the next state $s_{t+1}$.

The lecture first writes a policy as a state-to-action map:

$$ \pi_\theta:s_t\to a_t. $$

The parameter vector $\theta$ is what learning changes. A stochastic policy generalizes this map to $a_t\sim\pi_\theta(\cdot\mid s_t)$. The environment dynamics determine what happens after an action:

$$ s_{t+1}\sim p(\cdot\mid s_t,a_t). $$

This abstraction covers very different domains. A robot action may be a motor torque, an inventory action may be a purchase, and a language-model action may be the next token. The physical meaning changes, but the feedback structure does not.

Reward is evaluative, not a correct-action label. It says how desirable an outcome is according to the task objective. It may be immediate or delayed, dense or sparse, and it does not necessarily tell the agent which earlier action caused the outcome.

A policy creates a trajectory distribution

Lecture 1 summarizes one interaction sequence and its goal as

$$ (s_1,a_1,r_1,\ldots,s_T,a_T,r_T), \qquad \text{maximize}\ \sum_t r_t. $$

This is the essential RL problem: choose a policy whose whole sequence receives high total reward. The equations below are the probabilistic version used in later lectures. They use a fully observed state; in a partially observed problem, the policy conditions on observations or their history.

With $T$ decisions, write the variables needed by the dynamics as

$$ \tau=(s_1,a_1,s_2,a_2,\ldots,s_T,a_T,s_{T+1}). $$

Assuming a Markov environment and policy, its probability or density factors as

$$ p_\theta(\tau) =p(s_1)\prod_{t=1}^{T} \pi_\theta(a_t\mid s_t)\, p(s_{t+1}\mid s_t,a_t). $$

This identity exposes the feedback loop:

• The policy controls the action factors $\pi_\theta(a_t\mid s_t)$.
• The environment controls the transition factors $p(s_{t+1}\mid s_t,a_t)$.
• Changing the policy changes the probability of entire future trajectories.

For a stochastic policy or environment, the lecture’s total reward becomes an expected objective:

$$ J(\theta) =\mathbb E_{\tau\sim p_\theta(\tau)} \left[\sum_{t=1}^{T}r_t\right]. $$

This expectation is an explanatory probabilistic extension of the lecture’s $\max_\theta\sum_t r_t$. Later lectures may write a reward function $r(s_t,a_t)$ and introduce discounting; here $r_t$ is simply the reward observed from the environment. The central idea is unchanged: adjust the policy so that high-reward trajectories become more likely.

Why RL is not just supervised learning

In supervised learning, someone usually provides input-target pairs $(x_i,y_i)$, and the model learns to predict $y_i$ from $x_i$. In RL, the agent chooses actions, observes their consequences, and must improve from rewards rather than action labels.

Question	Supervised learning	Reinforcement learning
What is the signal?	Target $y$ for each input $x$	Reward evaluating behavior
Where does data come from?	Usually an externally collected dataset	Often from the current policy’s interaction
Does the output affect the next input?	Usually no	Yes, through the environment
What is optimized?	Prediction loss	Expected cumulative reward
Main extra difficulty	Generalization	Exploration and temporal credit assignment

The phrase “RL data is not i.i.d.” is useful but incomplete. Sequential data can also appear in supervised learning, and RL can learn from a fixed offline dataset. The structural distinction is that an RL action has consequences and the objective evaluates a sequence of decisions rather than matching a provided correct answer.

Two difficulties follow immediately:

• Exploration: the agent must choose actions to discover which outcomes are good.
• Credit assignment: a delayed reward must be attributed to the earlier decisions that produced it.

Why reinforcement learning can discover new behavior

Supervised learning says, in effect, “behave like these targets.” RL says, “find behavior that scores highly under this objective.” The reward specifies what is preferred without fully specifying how to achieve it.

That freedom can produce solutions that are absent from demonstration data. It can also produce reward hacking: behavior that scores well according to the written reward but violates the designer’s real intention.

Optimization is therefore both the strength and the danger of RL: optimized behavior follows the specified objective, not the designer’s unstated intention. This is why reward design is part of the problem, not a minor implementation detail.

What “deep” adds to reinforcement learning

The RL framework does not require neural networks. Classical RL can use tables or simple function approximators. Deep RL uses neural networks to represent objects such as policies, value functions, or environment models.

This division of labor is helpful:

• Deep learning provides scalable representation learning from images, language, and other high-dimensional observations.
• RL provides an objective and algorithms for improving sequential decisions.

A neural network does not remove the RL difficulties. Exploration, delayed credit, unstable targets, and costly interaction remain. “Deep” increases representational power; it does not make the sequential optimization problem disappear.

One framework, many applications

An application is an RL problem only after its observation, action, reward, and interaction process are defined.

System	Observation	Action	Possible reward
Robot	Camera image, joint state	Motor torque	Task success, speed
Inventory system	Stock and demand	What to purchase	Profit minus cost
Language model	Prompt and generated prefix	Next token	Human preference score
Image generator	Prompt and current sample	Denoising decision	Alignment or quality score

The table also reveals a limitation: saying “use RL” does not define the problem. The chosen state, action space, reward, and data-collection mechanism determine what can actually be learned.

Why real-world RL is difficult

The same feedback loop that makes RL powerful creates its main challenges:

• Sample efficiency: interaction may be expensive, slow, or dangerous.
• Exploration: useful experience may require trying uncertain actions.
• Credit assignment: rewards may arrive long after the important action.
• Distribution change: improving the policy changes the data distribution.
• Optimization stability: policy updates change both behavior and future training data.
• Reward specification: the measurable reward may not match the real goal.
• Safety: a poor exploratory action can have real consequences.

These are not separate accidents. Most are consequences of learning from data that the learner itself helps generate.

Common confusions

Is reward the same as a label?

No. A label tells the model the desired output for a particular input. A reward evaluates the consequence of behavior and may arrive much later.

Is every control problem an RL problem?

No. A controller can be designed from known dynamics without learning. It becomes an RL problem when behavior is improved from experience using a reward or return objective.

Does RL always require online interaction?

No. Offline RL learns from previously collected interaction data. The policy-dependent nature of the data still matters because the learned policy may visit states not covered by that dataset.

Does a high return imply intelligent behavior?

Only relative to the defined environment and reward. A policy can maximize the wrong reward very effectively.

My takeaway

The cleanest definition of RL is not “learning by trial and error” or “machine learning without labels.” RL is optimization over a feedback loop: the policy selects actions, actions shape future experience, and the objective scores whole trajectories.

Deep RL combines this loop with neural representation learning. Its promise is that a general learner can turn rich observations into complex decisions. Its difficulty is that data collection, optimization, and behavior are now coupled together.

References

• CS285: Deep Reinforcement Learning
• Lecture 1: Introduction (PDF)
• Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction, 2nd edition.