NuScenes is a dataset released by a self-driving car company called NuTonomy, which itself is part of Aptiv now. However, it seems like NuTonomy remains sowhat independent. While there are many self-driving car related datasets available, this one is unique. It contains RADAR readings. Let’s have a look at it.

Introduction to NuScene

This dataset is formally introduced in Caesar et al. (2019): nuScenes: A multimodal dataset for autonomous driving. We’ll find a few example scenes and videos are available on YouTube. Further, there is an interactive exploration tool available to look at the data without downloading it. AFAIK, it only supports annotations, images and LiDAR previews - RADAR is missing.

The dataset itself was released under the CC BY-NC-SA 4.0 license.

The sensor suite consists of 13 sensors:

  • 6 cameras (3 forward facing, 3 backward facing, 16 Hz sample rate)
  • 5 long-range RADAR sensors operating at 77 GHz (13 Hz sample rate)
  • LiDAR on top (20 Hz sample rate)
  • IMU

We can find the exact sensor layout here.

Tutorial walk-through

tutorial.ipynb

The NuScenes devkit requires the following folder/data structure:

$ tree -d .
.
├── data
│   └── sets
│       └── nuscenes
│           ├── maps
│           ├── samples
│           │   ├── CAM_BACK
│           │   ├── CAM_BACK_LEFT
│           │   ├── CAM_BACK_RIGHT
│           │   ├── CAM_FRONT
│           │   ├── CAM_FRONT_LEFT
│           │   ├── CAM_FRONT_RIGHT
│           │   ├── LIDAR_TOP
│           │   ├── RADAR_BACK_LEFT
│           │   ├── RADAR_BACK_RIGHT
│           │   ├── RADAR_FRONT
│           │   ├── RADAR_FRONT_LEFT
│           │   └── RADAR_FRONT_RIGHT
│           ├── sweeps
│           │   ├── CAM_BACK
│           │   ├── CAM_BACK_LEFT
│           │   ├── CAM_BACK_RIGHT
│           │   ├── CAM_FRONT
│           │   ├── CAM_FRONT_LEFT
│           │   ├── CAM_FRONT_RIGHT
│           │   ├── LIDAR_TOP
│           │   ├── RADAR_BACK_LEFT
│           │   ├── RADAR_BACK_RIGHT
│           │   ├── RADAR_FRONT
│           │   ├── RADAR_FRONT_LEFT
│           │   └── RADAR_FRONT_RIGHT
│           ├── v1.0-mini
│           ├── v1.0-test
│           └── v1.0-trainval
└── nuscenes-devkit

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from nuscenes.nuscenes import NuScenes

path = "./data/sets/nuscenes"
nusc = NuScenes(version='v1.0-mini', dataroot=path, verbose=True)

======
Loading NuScenes tables for version v1.0-mini...
23 category,
8 attribute,
4 visibility,
911 instance,
12 sensor,
120 calibrated_sensor,
31206 ego_pose,
8 log,
10 scene,
404 sample,
31206 sample_data,
18538 sample_annotation,
4 map,
Done loading in 0.9 seconds.
======
Reverse indexing ...
Done reverse indexing in 0.1 seconds.
======

The v1.0-mini subset contains 10 scenes from Boston and Singapore:

nusc.list_scenes()

scene-0061, Parked truck, construction, intersectio... [18-07-24 03:28:47]   19s, singapore-onenorth, #anns:4622
scene-0103, Many peds right, wait for turning car, ... [18-08-01 19:26:43]   19s, boston-seaport, #anns:2046
scene-0655, Parking lot, parked cars, jaywalker, be... [18-08-27 15:51:32]   20s, boston-seaport, #anns:2332
scene-0553, Wait at intersection, bicycle, large tr... [18-08-28 20:48:16]   20s, boston-seaport, #anns:1950
scene-0757, Arrive at busy intersection, bus, wait ... [18-08-30 19:25:08]   20s, boston-seaport, #anns:592
scene-0796, Scooter, peds on sidewalk, bus, cars, t... [18-10-02 02:52:24]   20s, singapore-queensto, #anns:708
scene-0916, Parking lot, bicycle rack, parked bicyc... [18-10-08 07:37:13]   20s, singapore-queensto, #anns:2387
scene-1077, Night, big street, bus stop, high speed... [18-11-21 11:39:27]   20s, singapore-hollandv, #anns:890
scene-1094, Night, after rain, many peds, PMD, ped ... [18-11-21 11:47:27]   19s, singapore-hollandv, #anns:1762
scene-1100, Night, peds in sidewalk, peds cross cro... [18-11-21 11:49:47]   19s, singapore-hollandv, #anns:935

my_scene = nusc.scene[0]
my_scene

{'token': 'cc8c0bf57f984915a77078b10eb33198',
 'log_token': '7e25a2c8ea1f41c5b0da1e69ecfa71a2',
 'nbr_samples': 39,
 'first_sample_token': 'ca9a282c9e77460f8360f564131a8af5',
 'last_sample_token': 'ed5fc18c31904f96a8f0dbb99ff069c0',
 'name': 'scene-0061',
 'description': 'Parked truck, construction, intersection, turn left, following a van'}

Let’s visualize it:

first_sample_token = my_scene['first_sample_token']
nusc.render_sample(first_sample_token)

This is pretty much everything we have. I’m going to investigate the RADAR data and post an update (or write a paper).