Recently, I took (and of course finished) the Self-driving Cars specialization offered by the University of Toronto on Coursera. My main motivation: explore something new, get a few more ideas. Well, it reminded me of my experience with UPenns Robotics Specialization on Coursera. So, I got a few more ideas and especially a deeper understanding why so many self-driving car companies are struggling so much with making significant progress. Instead of writing a small review about every single course on Coursera, I decided to write a more comprehensive review of the whole specialization.

Note: Unlike Udacity’s approach to share solutions on GitHub and writing detailed blog posts about projects, the Coursera honor code does not allow any sharing of solutions. Therefore, I’m not going to answer any requests for sharing my solutions. If you are struggling with some projects, then get the skills to overcome your struggles.

Contents

General review of this specialization

Let me start with a general review of this specialization before commenting on the individual courses. The promise of the specialization is as follows:

This Specialization gives you a comprehensive understanding of state-of-the-art engineering practices used in the self-driving car industry. By interacting with real data sets from an autonomous vehicle (AV), you’ll implement methods for static and dynamic object detection, localization and mapping, behaviour and maneuver planning, and vehicle control ― all through hands-on projects using the open source simulator CARLA. You’ll learn from a highly realistic driving environment that features 3D pedestrian modeling and environmental conditions. When you complete the Specialization successfully, you’ll be able to build your own self-driving software stack and be ready to apply for jobs in the autonomous vehicle industry.

– Description on the certificate of this specialization

The specialization consists of four courses:

  1. Introduction to Self-Driving Cars
  2. State Estimation and Localization
  3. Visual Perception for Self-Driving Cars
  4. Motion Planning for Self-Driving Cars

The Coursera specialization setting is as follows. You have to pay for a subscription on a monthly basis. While paying, you can access all courses but you’ll lose it once you stop paying. Therefore, it is important to download all course material while paying to revisit them later. I monitored the specialization for quite a while but I decided to take it once it was clear that all courses are available. I saw and still see no reason to start with something that is not completely available but I would have to pay until it is. I started this specialization in late April when the last course (motion planning) was not available et but announced for recent publication.

Let’s have a look at common things of all courses. First of all, this specialization feels like a standard university course (probably consisting of 20-30 ETCS/6 months of work). What I mean with standard university course? Well:

  • assignments are unclear (and sometimes it looks like wrong answers are required to pass)
  • theory on stuff that hardly anyone uses anymore (at least if they are sane ;))
  • really awful forum support
  • lots of guessing with the assignments (I love hard challenges, but I hate spending time to figure out what to do because of a lack of documentation)
  • gear controller for cars with ICEs - welcome to the future ;)
  • assingments are somewhat disconnected (the SDC simulation software CARLA offers a lot of potential to build a full, virtual SDC over the series of all four courses).

Similar to most university lectures I enjoyed, this course has a focus on reading recommended papers (e.g. on quaternion kinematics for Error-State Extended Kalman Filter (ES-EKF) which consists of 95 pages full of (good) math - isn’t fun to read but much more useful for the final task than the “lectures”). Similar to the robotics specialization: what is the value of prepared material if it hardly helps for the projects (we’re talking about a life-long learning type of course not something to torture first year students with ;)). A short summary upfront: if that mindset is present in most self-driving car companies, then it is no wonder that Tesla beats them all. Don’t get we wrong here. It is valueable, but “classical” robotics. That stuff is great to build something showable fast, but really aweful if our goal is building a real autonomous car. Perhaps most importantly, all that stuff is much more deterministic and therefore “understandable” by a broader audience than real probabilistic neural networks.

When it comes to quizzes, there are some typical problems on Coursera. Sometimes wrong anwers are considered correct and vice versa. Furthere, there are some cultural challenges ;). If you grew up in a certain country in Europe with the Autobahn and much less relaxed drivers compare to the rest of the world, you’ll end up answering some questions quite a bit differently than what the lectures tell you (e.g. if the light turns yellow, I expect acceleration and no breaking ;)). The programming assignments have many ups and downs. Their major disadvantage is that their documentation varied a lot. Further, source code quality varies depending on the assignment and probably depending on who programmed it. One or two programming assignments contain(ed?) incorrect example functions. I’m still not sure if they have been fixed. In general, I would recommend to download each programming assignment (inline shell functions in Jupyter notebooks are helpful for this) and treat every parameter selection task as an optimization problem. This is not part of the course, however it:

  1. Saves a lot of time
  2. Provides a more stable solution
  3. If you are new to optimization algorithms, then you learn something that you can use for pretty much every problem that is about selection a set of parameters. This might be even more valuable than all the rest of the course since you can use it for everything in your life.
    Moreover, both assignments that require CARLA benefit a lot from rewriting some parts with Cython.

Warning: The courses uses the CARLA self-driving car simulator. Make sure that your hardware is able to use it before you start the course.

Since all courses are online now, I would estimate that everybody with a proper background in physics (1st year university level) and Python programming knowledge should be able to finish this specialization within a month or two (I finished it within 1.5 month of part part part part time work on it;)). NB! There were quite many forum posts by people complaining about their non-existing mechanical engineering/physics knowledge. Seriously, if they really hold degrees in computer science, then they should be able to look up some basic engineering mechanics - especially since their math skills should be higher sufficient.

Forum support? Well, it seems like the courses are somewhat abandoned once they are available. This is different to courses I took at Udacity. It might be a Coursera thing since that is what a observed with almost all course except the deeplearning.ai specialization. However, I ended up with an experience close to my robotics experience. Yes, there is some forum support, but it certainly looks like it is not of any high priority. On the other hand, we have to keep in mind that course fees are much loweer than e.g. at Udacity.

Individual courses

Okay, let’s have a look at each course of this specialization.

Introduction to Self-Driving Cars

The introduction courses covers most things you would expect in such an introductory course. The official course description looks as follows:

This course will introduce you to the terminology, design considerations and safety assessment of self-driving cars. By the end of this course, you will be able to:

  • Understand commonly used hardware used for self-driving cars

  • Identify the main components of the self-driving software stack

  • Program vehicle modelling and control

  • Analyze the safety frameworks and current industry practices for vehicle development

For the final project in this course, you will develop control code to navigate a self-driving car around a racetrack in the CARLA simulation environment. You will construct longitudinal and lateral dynamic models for a vehicle and create controllers that regulate speed and path tracking performance using Python. You’ll test the limits of your control design and learn the challenges inherent in driving at the limit of vehicle performance.

– Intro to Self-Driving Cars course description

Okay, let’s work through this step by step. The first part covers basic computational hardware (unfortunately no design of ASICs ;)) and sensors for perception including

  • Cameras,
  • LiDAR,
  • RADAR,
  • Ultrasonics,
  • GNSS,
  • IMU,
  • and Wheel odometry.

The general focus of this course and the state estimation course seems to be on IMU, GNSS and LiDAR. IMHO this concept is very questionable since it somewhat restricts a potentially autonomous system to something automated that works in specific environments only due to real-world physics (EM wave propagation through different media and EM wave reflection properties).

The other 5/7th of the course cover vehicle dynamics and control theory. Nothing special, just plain bread and butter stuff. However, I noticed two things:

  1. They considered car with ICEs (internal combustion engines) which adds some complexity to controllers (including gears dynamics, etc.). However, ICEs in 2019? Really? No electric cars?
  2. PID control? Who in his right mind uses that anymore? It simply adds so much instability in a real-world setting that I wouldn’t recommend using this.

In general I would conclude that it is clear that this course has more a deterministic/classical robotics approach aiming at “hand-engineering” a system. This is a great approach to generate some early results but in long-term it loses against every “intelligent”/self-learning system. A general downside of this course is that it is more a presentation of concepts and less a teaching of concepts.

A little advice for the final project. PID or MPC controller are probably the easiest to implement. Using behavioral cloning would probably take too much time and (probably) results in too much deviation from the waypoints that have to be followed. My first attempt was with constant full throttle and no breaking resulted in this:

A suitable solution (with breaking and throttle control) should look like this after some fine-tuning (which can be automated ;)):

In the simulator it looks like this:

State Estimation and Localization

Let’s have a look at the second course of this specialization. It covers:

This course will introduce you to the different sensors and how we can use them for state estimation and localization in a self-driving car. By the end of this course, you will be able to:

  • Understand the key methods for parameter and state estimation used for autonomous driving, such as the method of least-squares

  • Develop a model for typical vehicle localization sensors, including GPS and IMUs

  • Apply extended and unscented Kalman Filters to a vehicle state estimation problem

  • Understand LIDAR scan matching and the Iterative Closest Point algorithm

  • Apply these tools to fuse multiple sensor streams into a single state estimate for a self-driving car

For the final project in this course, you will implement the Error-State Extended Kalman Filter (ES-EKF) to localize a vehicle using data from the CARLA simulator.

– State Estimation and Localization for Self-Driving Cars course description

In other words, this course is about pose recovery from different sensors and sensor fusion. Again, this course consists of very superficial presentation of various topics (I wouldn’t call it teaching). IHMO it is not okay to simply refer to some publications explaining it. Seriously, if I take such a course (and pay for it!), then I want to learn something in a condensed but understandable way to safe time. Otherwise, I could teach it myself by using a ton of publications. For those interested in understanding Kalman filters I can recommend this series on YouTube.

Unfortunately, RADAR is left out and visual odometry (odometry using cameras) is part of course no. 3. This course has a minor programming exercise, which is not soooo easy, and a final project. The final project covers sensor fusion and ES-EKFs:

From raw IMU data




to an acceptable error using LiDAR and GNSS data:










LiDAR, IMU and GNSS data originates from a simulation performed in CARLA. I would have prefered it to do an assignment doing exact that inside the simulation environment instead of working with extracted data only. A word to the wise. This assignment requires a lot of parameter tweaking. I recommend writing an optimization pipeline for it.

Visual Perception for Self-Driving Cars

This course will introduce you to the main perception tasks in autonomous driving, static and dynamic object detection, and will survey common computer vision methods for robotic perception. By the end of this course, you will be able to work with the pinhole camera model, perform intrinsic and extrinsic camera calibration, detect, describe and match image features and design your own convolutional neural networks. You’ll apply these methods to visual odometry, object detection and tracking, and semantic segmentation for drivable surface estimation. These techniques represent the main building blocks of the perception system for self-driving cars. For the final project in this course, you will develop algorithms that identify bounding boxes for objects in the scene, and define the boundaries of the drivable surface. You’ll work with synthetic and real image data, and evaluate your performance on a realistic dataset.

– Visual Perception for Self-Driving Cars course description

The course itself covers a lot of classical computer vision and a tiny bit of modern computer vision using neural networks. Unfortunately, the latter is not required in any programming assignment. When it comes to programming assignments, a major portion of programming assignments consists of fighting the poor and completely outdated OpenCV documentation. I’m not new to OpenCV, I use it very frequently but it still surprises me on a weekly basis ;). I want to point that this is not a problem of this course but and OpenCV problem. The grader is a mess. I’m really not sure if the staff of this course is to blame for that or if it is a coursera issue. In the end it turned out that many errors are caused by the Jupyter notebook (who knows why). A general recommendation for all programming assignments in this course is to download the exercises and work on your local machine. However, running the algorithms locally on my workstation (even without Cython) increased prototyping speed quite a lot. In the end I used some Cython to get testing done even quicker.

As mentioned in the review of the previous course, this courses covers visual odometry. This means that we localize our car from images and end up with a trajectory like this:

It was really disappointing to me that no deep learning was used in any of the programming assignments of this course, especially since the last assignment deals with semantic segmentation. Lane line detection using classical computer vision certainly is not state-of-the-art. I would have expected much more advanced stuff to be covered in a final assingment on such a topic.

Motion Planning for Self-Driving Cars

This course was released really late (end of April 2019).

Let’s have a look at the 4th and final course of this specialization.

This course will introduce you to the main planning tasks in autonomous driving, including mission planning, behavior planning and local planning. By the end of this course, you will be able to find the shortest path over a graph or road network using Dijkstra’s and the A* algorithm, use finite state machines to select safe behaviors to execute, and design optimal, smooth paths and velocity profiles to navigate safely around obstacles while obeying traffic laws. You’ll also build occupancy grid maps of static elements in the environment and learn how to use them for efficient collision checking. This course will give you the ability to construct a full self-driving planning solution, to take you from home to work while behaving like a typical driving and keeping the vehicle safe at all times. For the final project in this course, you will implement a hierarchical motion planner to navigate through a sequence of scenarios in the CARLA simulator, including avoiding a vehicle parked in your lane, following a lead vehicle and safely navigating an intersection. You’ll face real-world randomness and need to work to ensure your solution is robust to changes in the environment.

– Motion Planning for Self-Driving Cars

This course is all about planning. It emphasises a hierarchical approach to planning and covers different planning objectives on different levels of such a hierarchy. Unfortunately, there is some focus on HD maps. Again, most stuff is classcial robotics thinking applied to the goal of building a self-driving car.

The final project was tougher than it looked. Well, the programming assignment itself was really not challenging but fine-tuning everything. My manual parameter tuning approaches attracted the car to street light poles like moths to a flame. I recommend rewriting for-loops etc using either Numba or Cython. I did that and used a grid search appraoch to find suitable parameters automatically. I passed using that setup, but it still required some manual fine tuning to pass. In the end, I was very close to rewrite all client side functions in Julia to increase prototyping speed, enable multiple CPU cores and use more advanced optimization methods. Even after passing the assignment, I’m still surprised how the motion planner came up with certain trajectories. In retrospective, it seems like a lot of the weird behavior is caused by the nature of PID controllers and some properties of a spiral path optimizer. I don’t think anyone should use them anymore unless you want to build something that drives like a drunken 3 year old ;). There are some other things that remain unclear to me like why sidewalks, light poles etc. are not considered as a non-driveable area in the planner.

Here are some impressions of the final project:

The drunken three year old ;)

How to hit a trash barrel and drive around a stop sign

Driving a bit more around a stop sign

A more plesant way of driving

While debugging a few attempts manually, I found quite some interesting behavior. Sometimes the planner came up with complete mirages ;). Again, this concept is classical robotics, so it provides okayisch results fast, but many things will break in slightly different situations. Some of the weird behavior could be caused by CARLA itself as well. CARLA is still undergoing heavy development and somethings seem to be fixed in newer versions.

Final words

I’m not someone participating in today’s “positivism/sugarcoating”. Beside taking courses for fun (gaining knowledge), I’m interested in gaining new insights on things that I can apply today or I keep it in mind to revisit it some time in the future. Did I gained any new knowledge? Yes, I learned a few new things and more importantly about perspectives of building such a self-driving car. The courses definitely motivates me to continue to work on my technical note on RADAR vs. LiDAR (I will link it here once it is published). Further, I’m going to add a robotics/autonomous systems section to my literature lists. I collected so much over the last 2-3 years on this topic and this course provided some additional literature that I was not aware of.

Did the staff a bad job? I wouldn’t say so. They designed a course covering topics and concepts how things are done at (probably) most companies in the field of autonomous systems (cars, robots, multi-copters, submarines etc.). Since they advertise this specialization as a qualification to apply for jobs in the self-driving car industry, I conclude that it probably qualifies for work in most self-driving car companies that apply these more classical methods. That means, that you can probably apply everywhere but at Tesla and Comma.ai ;).

Would I recommend taking this course? Well, I would say so because:

  1. It provides an extremely important overview of the mechanical/electrical engineering part of SDC which is something that many computer science people lack completely.
  2. It helps understanding how “more traditional” engineers think. This can help to resolve arguments.
  3. The course fees are really low (compared to e.g. Udacity’s Self-Driving Car ND) so it won’t set you back with too much if you don’t like/finish it (there is a free (7day?) trial phase at the beginning).

Suggested improvements

I don’t want to complain too much without providing some suggestions (in case anyone of the staff reads this). First, hardware requirements for this course are quite high anyhow. Hence, that could be utilized. The amazing thing of a simulator such as CARLA is that the simulator provides ground truth data. This makes it really easy to apply machine learning to solve problems. This could include building models that perform sufficiently on edge devices. For example lane line detection (or driveable area detection), which was part of course 3, is so much more stable using a deep learning approach than classical computer vision. Perhaps a fith course would be useful to introduce all the modern stuff - not just on a few slides but as programming assignments. This includes other approaches to motion planning as well (e.g. various flavors of reinforcement learning, value iteration networks, etc.).