Reinforcement Learning: Learning by Reward

Learning the way you trained a pet

In supervised learning you hand the model the right answers. Reinforcement Learning (RL) is different: nobody tells the agent the correct move. Instead it tries an action, sees what happens, and gets a reward (a treat) or a penalty (a telling-off) — then it adjusts so it earns more reward next time. It is exactly how you might train a dog to sit.

This connects straight back to the agent model from Unit 2: an agent senses its environment and acts on it. RL just adds one thing — a reward signal that tells the agent how well it is doing.

The five words you need

• State — the situation the agent is in right now (e.g. which square it stands on).
• Action — a move it can make from that state (up, down, left, right).
• Reward — a number the environment gives back: positive for good, negative for bad.
• Policy — the agent’s strategy: which action to pick in each state.
• Q-value — a score for an (state, action) pair: “how much total reward do I expect if I take this action here?”

Learning means improving the policy by updating Q-values from experience, so over time the agent prefers high-reward actions.

The agent–environment loop

RL repeats one loop over and over, sometimes for millions of rounds:

1. Look at the current state.
2. Choose an action (early on, partly at random, to explore).
3. Do the action; the environment returns a reward and the new state.
4. Update the Q-value for the action you just took, nudging it toward the reward you received.
5. Repeat from step 1 in the new state.

# The RL loop in pseudo-Python (the learning lives in update_Q)
state = start_state
while not done:
    action = choose_action(state)          # explore or exploit
    reward, next_state = environment(state, action)
    update_Q(state, action, reward, next_state)   # learn from it
    state = next_state

One Q-learning update by hand

The famous rule for updating a Q-value is Q-learning. In words: nudge the old estimate toward the reward you just got plus the best you expect from the next state. The formula uses two dials — the learning rate alpha (how big a step to take, here 0.5) and the discount gamma (how much future reward counts, here 0.9):

# Q-learning update for the action we just took
# new_Q = old_Q + alpha * (reward + gamma * best_next_Q - old_Q)

old_Q       = 0.0    # what we thought this action was worth
reward      = 10     # the treat we just received
best_next_Q = 4.0    # best Q-value available from the next state
alpha, gamma = 0.5, 0.9

new_Q = old_Q + alpha * (reward + gamma * best_next_Q - old_Q)
print('updated Q =', new_Q)

The explore-vs-exploit tension

An RL agent faces a dilemma every step: exploit (pick the action it already thinks is best) or explore (try something new that might be even better). Too much exploiting and it never discovers good moves; too much exploring and it never cashes in. Real agents usually explore a lot early on, then settle down.

Where RL is used