a custom gymnasium environment that pits a classical destination dispatching algorithm against a ppo-trained agent, built to answer one question: can reinforcement learning discover a better elevator dispatch policy from experience alone, with no hand-coded routing logic?
short answer: yes. the ppo agent cut average passenger wait time by 84.5% (601 → 93 steps) while increasing throughput by ~7%, both agents evaluated over 1,000 independent episodes under identical traffic.
the problem
elevator dispatch is a classic operations-research problem. real buildings run rule-based systems — scan, collective control, destination dispatching — deterministic algorithms that assign cars by proximity and direction. they work, but they can't adapt to learned patterns. the question was whether a policy trained purely from reward could beat a reasonable hand-coded baseline in a controlled simulation.
what i built
a two-phase simulation modelling a 20-floor building with 4 independent elevators, 10,000 steps per episode.
- phase 1 — a passenger appears on a floor with a directional call (up/down). call buttons are modelled as booleans, not counters: a real hall button tells you someone is waiting, not how many. i kept the sim grounded in the information a real system actually has.
- phase 2 — on boarding, a destination is sampled in the travel direction. the system doesn't know where you're going until you press a button inside the car.
traffic is probabilistic with time-of-day modifiers — a 3× morning up-rush from the lobby, a 2× evening down-rush — so the agent has to handle shifting demand, not a static distribution.
observation space (92 values): per-elevator position, direction and load, plus per-floor up/down waiting flags and how long each has been waiting. i deliberately included elevator load: the classical agent reads targetFloors directly when it decides, so hiding equivalent state from ppo would have rigged the comparison.
action space: MultiDiscrete([3, 3, 3, 3]) — every step the agent sets each of the 4 elevators to idle / up / down.
the hard part: reward design
this is where the actual ml work lived. stable learning didn't happen until the fifth reward iteration.
- raw negative wait time → values hit -6.4M, gradients too large,
explained_variance ≈ 0. - normalized wait penalty → stable but no positive signal; the agent learned to idle, not to serve.
- pickup bonus → it picked passengers up and never delivered them. pickup became the goal.
- dropoff-weighted → partial win, then a bug: the reward block sat inside the 20-floor loop and was recomputed 20× per step, corrupting the signal entirely.
- shaping reward (final):
# dense per-step nudge: each occupied elevator moving toward its target
for elevator in elevators:
if elevator.targetFloors and moving_toward(elevator, next_target):
reward += 0.01
# after all floor processing:
reward += dropoffs - min(avg_wait, 100) / 100the unlock was dense feedback. without the shaping term the agent could go hundreds of steps with no meaningful signal; with it, every step it moved an occupied car the right way earned a small positive reward. that's what made learning stable.
training
MlpPolicy (two 64-unit hidden layers), learning_rate=0.0001, ent_coef=0.05, total_timesteps=5,000,000.
the high entropy coefficient was deliberate. at ent_coef=0.001 the policy collapsed to near-deterministic after ~500k steps and lost its exploration before finding good strategies — visible as entropy_loss cratering from -4.0 toward 0. bumping it kept the agent exploring.
| milestone | steps | ep_rew_mean | explained_variance |
|---|---|---|---|
| start | 10,240 | -1,220 | 0.39 |
| midpoint | 2,004,992 | +1,810 | 0.72 |
| final | 5,001,216 | +1,940 | 0.64 |
reward crossed from negative to positive early and held. at 2M steps it was clearly improving but not plateaued; by 5M it had stabilized.
results
both agents evaluated over 1,000 independent episodes of exactly 10,000 steps, terminated disabled so every episode runs full length, ppo run with deterministic=True for direct comparison against the deterministic baseline.
| metric | classic | ppo | change |
|---|---|---|---|
| mean avg reward | -0.67 | +0.14 | +0.81 |
| mean avg wait (steps) | 600.61 | 93.06 | −84.5% |
| dropoffs / episode | 2,664 | 2,855 | +7.2% |
| pickups / episode | 2,664 | 2,866 | +7.6% |
the 84.5% wait-time cut is the headline, and it isn't reward hacking — pickups and dropoffs stay nearly equal (2,866 vs 2,855), so the agent isn't shuttling passengers on fake short trips to farm dropoff bonuses. it genuinely serves people faster and moves more of them. the mechanism: instead of reacting to existing calls like the baseline, it positions cars proactively.
what i learned
the policy was almost entirely a function of the reward signal, not the network. four of my five attempts failed not because the agent couldn't learn, but because i was telling it to learn the wrong thing — and one of those failures was a plain bug (the reward recomputed inside a loop) that no amount of training would have fixed. dense, well-shaped feedback beat clever architecture every time. the debugging discipline — watching explained_variance and entropy_loss to diagnose why learning stalled — mattered more than any hyperparameter.
honest limitations
i'm not claiming a production dispatcher. the baseline is a simplified destination-dispatching implementation with no persistent assignments and no look-ahead — a fair, reasonable opponent, not a state-of-the-art commercial system. the time model treats one step as one floor traversal, so the wait-time numbers are in steps, not seconds: real acceleration curves, door cycles and dwell times would shift both agents' numbers. there are no capacity limits and it's a single 4-car bank. realistic kinematics and a stronger baseline are what i'm building next — a tougher comparison is a more honest one.