Event Camera Benchmark Dataset

This is the dataset for the paper

Experimental Comparison between Event and Global Shutter Cameras.
Holešovský, O., Škoviera, R., Hlaváč, V., Vítek, R. (2021).
Sensors (Basel, Switzerland), 21. https://doi.org/10.3390/s21041137

If you use any of this data in an academic context, please cite the following publication:

@Article{Holesovsky2021sensors,
author = {Holešovský, Ondřej and Škoviera, Radoslav and Hlaváč, Václav and Vítek, Roman},
title = {Experimental Comparison between Event and Global Shutter Cameras},
journal = {Sensors},
volume = {21},
year = {2021},
number = {4},
article-number = {1137},
url = {https://www.mdpi.com/1424-8220/21/4/1137},
issn = {1424-8220},
doi = {10.3390/s21041137}
}

Sample Python scripts

We provide two Python scripts which demonstrate how to read the dataset. Before executing any of the scripts, set the value of the global variable hdf5_root in the header of the script to the path of the directory containing the downloaded dataset or its parts (hdf5 files and/or unpacked tar/zip archives, see below).

python sample_dot.py loads all rotating dot datasets available in hdf5_root recorded by each camera (ATIS, DVS240, Basler). For each camera available, it calibrates the encoder-camera position ground truth and shows a sample event/frame image of the dot with the ground truth position marked by a green cross.

python sample_ballistic.py loads the ballistic dataset (ballistic.hdf5). It shows a sample image of each of the three included projectile types. The sample image displays ATIS events and a Photron intensity frame, both transformed into the same metric world coordinates using the camera calibration parameters stored in the dataset.

We tested the scripts on a GNU/Linux platform with these software dependencies:

ffmpeg 4.1.3
imageio 2.5.0
imageio-ffmpeg 0.3.0
h5py 2.10.0
matplotlib 3.1.0
numpy 1.17.4
opencv 4.1.0
python 3.7.3
scipy 1.6.1

Dataset files

In the table below, choose and download the dataset components you wish to use and place (or extract) them all in the same root directory (called hdf5_root in the sample Python scripts). The two event cameras are ATIS and DVS240, the frame cameras are Basler and Photron.

File	Size	Description
aruco_atis.hdf5	3.8 GiB	Rotating ArUco markers. ATIS.
aruco_dvs240.hdf5	1.8 GiB	Rotating ArUco markers. DVS240.
aruco_basler.tar	1.9 GiB	Rotating ArUco markers. Basler.
dot_atis.hdf5	765 MiB	Rotating dots. ATIS.
dot_dvs240.hdf5	235 MiB	Rotating dots. DVS240.
dot_basler.tar	2.0 GiB	Rotating dots. Basler.
ballistic.hdf5	512 MiB	All ballistic recordings including calibration parameters and images.
ballistic_calib_images.zip	6.0 MiB	Ballistic camera calibration images as png files.
atis_dvs240_config.zip	8 KiB	Bias configuration files of the ATIS and DVS240 event cameras.
sample_dot.py	10 KiB	Demo script: show rotating dot images or events.
sample_ballistic.py	5 KiB	Demo script: show ballistic projectile images and events.
LICENSE.txt	1 KiB	The software license for the two Python scripts (MIT License).
sha256sums.txt	1 KiB	Checksums (sha256sum) of all the files above.

In addition to trying the sample Python scripts above, you can use HDFView to browse the HDF5 files. The most recent 3.1.2 version we tested worked well, similarly to version 2.9 (from Ubuntu 16.04 package repositories). However, version 2.11 (in 18.04 and 20.04 Ubuntu repos) could not load the files properly.

License

This dataset is released under the Creative Commons license (CC BY-NC-SA 4.0), Attribution-NonCommercial-ShareAlike, which is free for non-commercial use (including research).

Rotating disk

HDF5 tree structure

marker type (aruco or dot)
- camera (atis or dvs240)
  - sensor illuminance (20lux, 80lux, 400lux or 2000lux)
    - events
    - encoder
    - params
marker type (aruco or dot)
- camera (basler)
  - sensor illuminance (5lux, 20lux, 80lux, 400lux or 1000lux)
    - encoder
    - params

Datasets

event camera datasets (for atis, dvs240)
- encoder - N_s x 2 int32 array, N_s is the number of stored encoder records
  - encoder[i,0] - timestamp in microseconds
  - encoder[i,1] - encoder position (in encoder ticks)
- events - N_e x 4 int32 array, N_e is the number of stored events
  - events[i,0] - timestamp in microseconds
  - events[i,1] - horizontal pixel coordinate (x)
  - events[i,2] - vertical pixel coordinate (y)
  - events[i,3] - event polarity (1 means ON/positive, -1 means OFF/negative)
- Encoder and event timestamps were generated by the same time base.
frame camera datasets (for basler)
- encoder - N_s x 1 int32 array, N_s is the number of stored encoder records
  - encoder[i,0] - encoder position (in encoder ticks) at the start of exposure of every frame stored in the associated mp4 video file

Attributes

params group contains attributes based on camera type
- all
  - cpr - the number of encoder counts (ticks) per disk revolution (constant)
  - w - image width in pixels
  - h - image height in pixels
  - illuminance - sensor illuminance in lux
  - radius - distance of the markers from the center of the disk in meters (constant)
- basler only (global shutter frame camera)
  - exposure_time - global shutter exposure time in microseconds
  - frame_rate - global shutter frame rate in frames per second
sensor illuminance group contains attributes based on camera type
- basler only
  - config - relative path to the Basler camera configuration text file (.pfs) used for the recording
  - video - relative path to the recorded video file (.mp4)
- atis only
  - config - relative path to the ATIS camera configuration text file (.bias) used for the recording (contains the camera biases)
  - sensitivity - the ATIS pixel sensitivity value which follows from the used bias configuration

Ballistic experiments

The ATIS camera used biases biases/ATIS_gen3_EM_faster_s40.bias with sensitivity 40 in all the ballistic recordings.

HDF5 tree structure

ballistic
- callibration
  - atis
    - image_sequence
  - photron
    - image_sequence
- projectile type (9x19mm_low_charge, 9x19mm_standard_charge or 7.62x51mm)
  - shot number
    - photron
      - image_sequence
      - params
    - atis
      - events
      - params

Datasets

image_sequence - uint8 array, shape: N_images x image width x image height
- its attributes
  - t0 - timestamp of the start of exposure of the first image in the sequence in microseconds (applies to all Photron image sequences except calibration images)
events - N_e x 4 int32 array, N_e is the number of stored events
- events[i,0] - timestamp in microseconds
- events[i,1] - horizontal pixel coordinate (x)
- events[i,2] - vertical pixel coordinate (y)
- events[i,3] - event polarity (1 means ON/positive, -1 means OFF/negative)

Attributes

calibration/atis or calibration/photron group
- camera_matrix - 3x3
- radial_distortion - three radial lens distortion parameters
- tangential_distortion - two tangential lens distortion parameters
- rotation - 3x3 rotation matrix
- translation - 1x3 translation vector
- world_units - units of the world coordinate frame
params group
- exposure_time - in microseconds (Photron only)
- frame_rate - in frames per second (Photron only)
- illuminance - sensor illuminance in lux (ATIS only)
- w - image width in pixels
- h - image height in pixels

Camera calibration

We calibrated the cameras in Matlab 2020a Camera Calibrator using the images of chessboard patterns stored in ballistic/calibration/[atis or photron]/image_sequence. Both stationary cameras simultaneously captured images of the chessboard at 17 different positions. Therefore these 17 image pairs can also be used for calibrating a Photron-ATIS stereo camera.

The Photron calibration images are 1024 x 1024 pixels in size, whereas the Photron ballistic recordings are just 640 x 280. The smaller images are simply center crops from the full 1024 x 1024 pixel Photron image.

The sample Python script demonstrates how to transform ATIS events and Photron images of a flying projectile to the common world plane using the calibration parameters stored in the HDF5 (see the attributes of calibration/atis or calibration/photron above). The 3D rotation and translation determine the position of each camera with respect to the first calibration chessboard plane (the first image in the calibration image_sequence).

The transformation between world coordinates (X,Y,Z) and undistorted image coordinates (x,y) is

$s[\begin{array}{ccc} x & y & 1 \end{array}] = [\begin{array}{cccc} X & Y & Z & 1 \end{array}] [\be...$

where K is the 3x3 camera_matrix

$[ \begin{array}{ccc} f_x & 0 & c_x\\ 0 & f_y & c_y\\ 0 & 0 & 1 \end{array} ]$

R is the 3x3 rotation matrix, t is the 1x3 translation vector and s a scale factor, which can be fixed by setting Z=0 (the depth coordinate of the calibration plane). c_x , c_y are the pixel coordinates of the optical center; f_x , f_y is the focal length expressed in horizontal and vertical pixel units.