Tesla AI Day

Deep Understanding Tesla FSD Part 3: Planning & Control

From Theory to Reality Analyze the Evolution of Full Self-Driving from Tesla AI Day

Jason Zhang
10 min readNov 5, 2021


From Tesla AI Day

This is the third article in my series on Deep Understanding Tesla FSD.

  1. Deep Understanding Tesla FSD Part 1: HydraNet
  2. Deep Understanding Tesla FSD Part 2: Vector Space
  3. Deep Understanding Tesla FSD Part 3: Planning & Control
  4. Deep Understanding Tesla FSD Part 4: Labeling, Simulation, etc

In the previous articles, we discussed the whole architecture of Tesla Vision.

From Tesla AI Day

In this article, we will talk about: Planning & Control.

Planning & Control

This part is introduced by Ashok Elluswamy who is director of autopilot software at Tesla.

The vision networks(See the previous article for details) take dense video and then compress it down into a 3D Vector Space. The role of the planner now is to consume this Vector Space and get the car to the destination, while maximizing the safety, comfort, and efficiency of the car. I have to say that as a human driver with more than ten years of experience, my driving style is to pursue the perfect combination of these three points.

The early version of Tesla Autopilot, when it was not called FSD(Full Self Driving), already had a good performance in most of the highway driving scenes, it was able to stay in the lanes, make lane changes as necessary, and take exits off the highway. (I remembered there is a Youtube video in 2016 showing that “Self Driving Tesla Drives It’s Owner To Work & Then Finds A Parking Spot”, you can watch it at end of this article.) The Tesla FSD requires the planner to have the same excellent performance on city street driving.