giacomoi / event-based_vision_resources

Event-based Vision Resources

Table of Contents:

• Devices

• Neuromorphic Systems

• Algorithms

  • Feature Detection and Tracking
  • Depth Estimation (3D Reconstruction)
    • Monocular
    • Stereo
  • Optical Flow Estimation
  • Intensity-Image Reconstruction
  • Localization and Ego-motion estimation
  • Visual Odometry and SLAM (Simultaneous Localization And Mapping)
  • Visual-Inertial Odometry
  • Visual Stabilization
  • Video Processing
  • Pattern recognition
  • Control

• Datasets and Simulators

• Software

  • Drivers
  • Calibration
  • Algorithms
  • Utilities

• Neuromorphic Processors and Platforms

• Workshops and Tutorials

• People / Organizations

• Contributing

---

Devices & Companies Manufacturing them

• DVS (Dynamic Vision Sensor): Lichtsteiner, P., Posch, C., and Delbruck, T., A 128x128 120dB 15μs latency asynchronous temporal contrast vision sensor, IEEE J. Solid-State Circuits, 43(2):566-576, 2008.
  • Product page at iniLabs. Buy a DVS
  • Introductory videos about the DVS
  • iniLabs invents, produces and sells neuromorphic technologies for research.
• DAVIS (Dynamic and Active-Pixel Vision Sensor) : Brandli, C., Berner, R., Yang, M., Liu, S.-C., Delbruck, T., A 240x180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor, IEEE J. Solid-State Circuits, 49(10):2333-2341, 2014.
  • Product page at iniLabs. Buy a DAVIS
  • Color-DAVIS: Li, C., Brandli, C., Berner, R., Liu, H., Yang, M., Liu, S.-C., Delbruck, T., Design of an RGBW Color VGA Rolling and Global Shutter Dynamic and Active-Pixel Vision Sensor, IEEE Int. Symp. Circuits and Systems (ISCAS), Lisbon, 2015, pp. 718-721.
• ATIS (Asynchronous Time-based Image Sensor): Posch, C., Matolin, D., Wohlgenannt, R. (2011). A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS, IEEE J. Solid-State Circuits, 46(1):259-275, 2011.
  • Chronocam
• Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., Delbruck, T.,
  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output,
  Proc. IEEE (2014), 102(10):1470-1484.
• Samsung's DVS (Gen2)
  • Son, B., et al., 4.1 A 640×480 dynamic vision sensor with a 9µm pixel and 300Meps address-event representation, IEEE Int. Solid-State Circuits Conf. (ISSCC), San Francisco, CA, 2017, pp. 66-67.
  • Slides and Video by Yoel Yaffe, Samsung Israel Research Center, Samsung Electronics.
• CeleX (Hillhouse Technology, Singapore). YouTube
• Insightness AG. The Silicon Eye Technology
  • Slides and Video by Christian Brandli, CEO and co-founder of Insightness.
• iniVation invents, produces and sells neuromorphic technologies with a special focus on event-based vision into business.

Neuromorphic Systems

• Delbruck, T., Frame-free dynamic digital vision*, Int. Symp. Secure-Life Electronics, Advanced Electronics for Quality Life and Society, University of Tokyo, Tokyo, Japan, Mar. 6-7, 2008, pp. 21-26. Introduces the software architecture of jAER and shows examples of several event-based processing algorithms.
• Liu, S.-C. and Delbruck, T., Neuromorphic sensory systems, Current Opinion in Neurobiology, 20:3(288-295), 2010.
• Delbruck, T., Fun with asynchronous vision sensors and processing. Computer Vision - ECCV 2012. Workshops and Demonstrations. Springer Berlin/Heidelberg, 2012. A position paper and summary of recent accomplishments of the INI Sensors' group.
• Liu, S.-C., Delbruck, T., Indiveri, G., Whatley, A., Douglas, R., Event-Based Neuromorphic Systems, Wiley. ISBN: 978-1-118-92762-5, 2014.
• Delbruck, T., Neuromorophic Vision Sensing and Processing (Invited paper), 46th Eur. Solid-State Device Research Conference (ESSDERC), Lausanne, 2016, pp. 7-14.

Algorithms

Feature Detection and Tracking

• Litzenberger, M., Posch, C., Bauer, D., Belbachir, A. N., Schon. P., Kohn, B., Garn, H.,
  Embedded Vision System for Real-Time Object Tracking using an Asynchronous Transient Vision Sensor,
  IEEE 12th Digital Signal Proc. Workshop and 4th IEEE Signal Proc. Education Workshop, Teton National Park, WY, 2006, pp. 173-178.
• Litzenberger, M., Kohn, B., Belbachir, A.N., Donath, N., Gritsch, G., Garn, H., Posch, C., Schraml, S.,
  Estimation of Vehicle Speed Based on Asynchronous Data from a Silicon Retina Optical Sensor,
  IEEE Intelligent Transportation Systems Conf., Toronto, Ont., 2006, pp. 653-658. PDF
• Drazen, D., Lichtsteiner, P., Haefliger, P., Delbruck, T., Jensen, A.,
  Toward real-time particle tracking using an event-based dynamic vision sensor,
  Experiments in Fluids (2011), 51(1):1465-1469. PDF
• Ni, Z., Pacoret, Benosman, R., Ieng, S., Reginer, S.,
  Asynchronous event-based high speed vision for microparticle tracking,
  J. Microscopy, 245(1):236-244.
• Ni, Z., Bolopion, A., Agnus, J., Benosman, R., Regnier, S.,
  Asynchronous event-based visual shape tracking for stable haptic feedback in microrobotics,
  IEEE Trans. Robot. (2012), 28(5):1081-1089.
• Piatkowska, E., Belbachir, A. N., Schraml, S., Gelautz, M.,
  Spatiotemporal multiple persons tracking using Dynamic Vision Sensor,
  IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Providence, RI, 2012, pp. 35-40. PDF
• Ni, Z.,
  Asynchronous Event Based Vision: Algorithms and Applications to Microrobotics,
  Ph.D. Thesis, Université de Pierre et Marie Curie, Paris, France, 2013.
• Borer, D., Rosgen, T.,
  Large-scale Particle Tracking with Dynamic Vision Sensors,
  ISFV16 - 16th Int. Symp. Flow Visualization, Okinawa 2014. Project page, PDF
• Lagorce, X., Meyer, C., Ieng, S. H., Filliat, D., Benosman, R.,
  Live demonstration: Neuromorphic event-based multi-kernel algorithm for high speed visual features tracking,
  IEEE Biomedical Circuits and Systems Conference (BioCAS), Lausanne, 2014, pp. 178-178.
• Lagorce, X., Meyer, C., Ieng, S. H., Filliat, D., Benosman, R.,
  Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual Features Tracking,
  IEEE Trans. Neural Netw. Learn. Syst. (2015), 26(8):1710-1720.
• Clady, X., Ieng, S.-H., Benosman, R.,
  Asynchronous event-based corner detection and matching,
  Neural Networks (2015), 66:91-106.
• Lagorce, X., Meyer, C., Ieng, S. H., Filliat, D., Benosman, R.,
  Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual Features Tracking,
  IEEE Trans. Neural Netw. Learn. Syst. (2015), 26(8):1710-1720.
• Ni, Z., Ieng, S. H., Posch, C., Regnier, S., Benosman, R.,
  Visual Tracking Using Neuromorphic Asynchronous Event-Based Cameras,
  Neural Computation (2015), 27(4):925-953.
• Barranco, F., Teo, C. L., Fermüller, C., Aloimonos, Y.,
  Contour Detection and Characterization for Asynchronous Event Sensors,
  IEEE Int. Conf. Computer Vision (ICCV), 2015, Santiago, Chile, pp. 486-494. PDF
• Liu, H., Moeys, D. P., Das, G., Neil, D., Liu, S.-C., Delbruck, T.,
  Combined frame- and event-based detection and tracking,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Montreal, QC, 2016, pp. 2511-2514.
• Tedaldi, D., Gallego, G., Mueggler, E., Scaramuzza, D.,
  Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS),
  IEEE Int. Conf. Event-Based Control Comm. and Signal Proc. (EBCCSP), Krakow, Poland, 2016. PDF, YouTube
• Braendli, C., Strubel, J., Keller, S., Scaramuzza, D., Delbruck, T.,
  ELiSeD - An Event-Based Line Segment Detector,
  Int. Conf. on Event-Based Control Comm. and Signal Proc. (EBCCSP), Krakow, Poland, 2016. PDF
• Glover, A. and Bartolozzi, C.,
  Event-driven ball detection and gaze fixation in clutter,
  IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Daejeon, South Korea, 2016, pp. 2203-2208. YouTube, Code
• Vasco, V., Glover, A., Bartolozzi, C.,
  Fast event-based Harris corner detection exploiting the advantages of event-driven cameras,
  IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Daejeon, South Korea, 2016, pp. 4144-4149. YouTube, Code
• Zhu, A., Atanasov, N., Daniilidis, K.,
  Event-based Feature Tracking with Probabilistic Data Associations,
  IEEE Int. Conf. Robotics and Automation (ICRA), Singapore, 2017. YouTube
• Mueggler, E., Bartolozzi, C., Scaramuzza, D.,
  Fast Event-based Corner Detection,
  British Machine Vision Conf. (BMVC), London, 2017.

Depth Estimation (3D Reconstruction)

Monocular Depth Estimation

• Rebecq, H., Gallego, G., Scaramuzza, D.,
  EMVS: Event-based Multi-View Stereo,
  British Machine Vision Conf. (BMVC), York, 2016. PDF, YouTube, 3D Reconstruction Experiments from a Train using an Event Camera

Monocular Depth Estimation using Structured Light

• Brandli, C., Mantel, T.A., Hutter, M., Hoepflinger, M.A., Berner, R., Siegwart, R., Delbruck, T.,
  Adaptive Pulsed Laser Line Extraction for Terrain Reconstruction using a Dynamic Vision Sensor,
  Front. Neurosci. (2014) 7:275. PDF, YouTube
• Matsuda, N., Cossairt, O., Gupta, M.,
  MC3D: Motion Contrast 3D Scanning,
  IEEE Conf. Computational Photography (ICCP), Houston,TX, 2015, pp. 1-10. PDF, YouTube, Project page

Stereo Depth Estimation

• Kogler, J., Sulzbachner, C., Kubinger, W.,
  Bio-inspired stereo vision system with silicon retina imagers,
  Int. Conf. Computer Vision Systems (ICVS), 2009, pp. 174-183. PDF
• Schraml, C., Schon, P., Milosevic, N.,
  Smartcam for real-time stereo vision - address-event based embedded system,
  Int. Conf. Computer Vision Theory and Applications (VISAPP), Barcelona, Spain, 2007, pp. 466-471.
• Schraml, S., Belbachir, A. N., Milosevic, N., Schon, P.,
  Dynamic stereo vision system for real-time tracking,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Paris, 2010, pp. 1409-1412.
• Kogler, J., Sulzbachner, C., Humenberger, M., Eibensteiner, F.,
  Address-Event Based Stereo Vision with Bio-Inspired Silicon Retina Imagers,
  Advances in Theory and Applications of Stereo Vision (2011), pp. 165-188.
• Kogler, J., Humenberger, M., Sulzbachner, C.,
  Event-Based Stereo Matching Approaches for Frameless Address Event Stereo Data,
  Int. Symp. Visual Computing (ISVC) 2011, Advances in Visual Computing, pp. 674-685.
• Benosman, R., Ieng, S. H., Rogister, P., Posch, C.,
  Asynchronous Event-Based Hebbian Epipolar Geometry,
  IEEE Trans. Neural Netw. (2011), 22(11):1723-1734.
• Lee et. al., ISCAS 2012
• Rogister, P. , Benosman, R., Ieng, S.-H., Lichtsteiner, P., Delbruck, T.,
  Asynchronous Event-Based Binocular Stereo Matching,
  IEEE Trans. Neural Netw. Learn. Syst., 23(2):347-353, 2012.
• Carneiro, J., Ieng, S.-H., Posch, C., Benosman, R.,
  Event-based 3D reconstruction from neuromorphic retinas,
  Neural Networks (2013), 45:27-38.
• Carneiro, J.,
  Asynchronous Event-Based 3D Vision,
  Ph.D. Thesis, Université de Pierre et Marie Curie, Paris, France, 2014.
• Piatkowska, E., Belbachir, A. N., Gelautz, M.,
  Asynchronous Stereo Vision for Event-Driven Dynamic Stereo Sensor Using an Adaptive Cooperative Approach,
  IEEE Int. Conf. Computer Vision Workshops (ICCVW), Sydney, NSW, 2013, pp. 45-50.
• Piatkowska, E., Belbachir, A. N., Gelautz, M.,
  Cooperative and asynchronous stereo vision for dynamic vision sensors,
  Meas. Sci. Technol. (2014), 25(5).
• Lee, J. H., Delbruck, T., Pfeiffer, M., Park, P. K. J., Shin, C.-W., Ryu, H., Kang, B. C., Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas,
  IEEE Trans. Neural Netw. Learn. Syst. (2014), 25(2):2250-2263.
• Camuñas-Mesa, L. A., Serrano-Gotarredona, T., Ieng, S. H., Benosman, R. B., Linares-Barranco, B.,
  On the use of orientation filters for 3D reconstruction in event–driven stereo vision,
  Front. Neurosci. (2014) 8:48.
• Camuñas-Mesa, L. A., Serrano-Gotarredona, T., Linares-Barranco, B., Ieng, S., Benosman, R.,
  Event-Driven Stereo Vision with Orientation Filters,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Melbourne VIC, 2014, pp. 257-260.
• Belbachir, A. N., Schraml, S., Mayerhofer, M., Hofstatter, M.,
  A Novel HDR Depth Camera for Real-time 3D 360-degree Panoramic Vision,
  IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), 2014, pp. 419-426. PDF
• Eibensteiner, F., Kogler, J., Scharinger, J.,
  A High-Performance Hardware Architecture for a Frameless Stereo Vision Algorithm Implemented on a FPGA Platform,
  IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Columbus, OH, 2014, pp. 637-644.
• Schraml, S., Belbachir, A. N., Bischof, H.,
  Event-Driven Stereo Matching for Real-Time 3D Panoramic Vision,
  IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp. 466-474. PDF. Slides.
• S. Schraml, A. N. Belbachir, Bischof, H.,
  An Event-Driven Stereo System for Real-Time 3-D 360° Panoramic Vision,
  IEEE Trans. Ind. Electron. (2016), 63(1):418-428.
• Firouzi, M. and Conradt, J.,
  Asynchronous Event-based Cooperative Stereo Matching Using Neuromorphic Silicon Retinas,
  Neural Processing Letters, 2016, 43(2):311-326. PDF
• Osswald, M., Ieng, S.-H., Benosman, R., Indiveri, G.,
  A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems,
  Scientific Reports 7, Article number: 40703 (2017).

Optical Flow Estimation

• Cook et. al. IJCNN 2011,
  Interacting maps for fast visual interpretation.
  Joint estimation of optical flow, image intensity and angular velocity with a rotating event camera.
• Benosman, R., Ieng, S.-H., Clercq, C., Bartolozzi, C., Srinivasan, M.,
  Asynchronous Frameless Event-Based Optical Flow,
  Neural Networks (2012), 27:32-37.
• Benosman, R., Clercq, C., Lagorce, X., Ieng, S.-H., Bartolozzi, C.,
  Event-Based Visual Flow,
  IEEE Trans. Neural Netw. Learn. Syst. (2014), 25(2):407-417.
  • Code (jAER): LocalPlanesFlow
• Orchard, G., Benosman, R., Etienne-Cummings, R., and Thakor, N,
  A Spiking Neural Network Architecture for Visual Motion Estimation,
  IEEE Biomedical Circuits and Systems Conf. (BioCAS), Rotterdam, 2013, pp. 298-301.
• Tschechne, S., Sailer R., Neumann, H.,
  Bio-Inspired Optic Flow from Event-Based Neuromorphic Sensor Input,
  IAPR Workshop on Artificial Neural Networks in Pattern Recognition (ANNPR) 2014, pp. 171-182.
• Barranco, F., Fermüller, C., Aloimonos, Y.,
  Contour motion estimation for asynchronous event-driven cameras,
  Proc. IEEE (2014), 102(10):1537-1556. PDF
• Barranco, F., Fermüller, C., Aloimonos, Y.,
  Bio-inspired Motion Estimation with Event-Driven Sensors,
  Int. Work-Conf. Artificial Neural Networks (IWANN) 2015, Advances in Computational Intelligence, pp. 309-321.
• Conradt, J.,
  On-Board Real-Time Optic-Flow for Miniature Event-Based Vision Sensors,
  IEEE Int. Conf. Robotics and Biomimetics (ROBIO), Zhuhai, China, 2015, pp. 1858-1863.
• Brosch, T., Tschechne, S., Neumann, H.,
  On event-based optical flow detection,
  Front. Neurosci. (2015), 9:137.
• Kosiorek, A., Adrian, D., Rausch, J., Conradt, J.,
  An Efficient Event-Based Optical Flow Implementation in C/C++ and CUDA,
  Tech. Rep. TU Munich, 2015.
• E. Mueggler, C. Forster, N. Baumli, G. Gallego, D. Scaramuzza,
  Lifetime Estimation of Events from Dynamic Vision Sensors,
  IEEE Int. Conf. Robotics and Automation (ICRA), Seattle (WA), USA, 2015, pp. 4874-4881. PDF, Code
• Rueckauer, B. and Delbruck, T.,
  Evaluation of Event-Based Algorithms for Optical Flow with Ground-Truth from Inertial Measurement Sensor,
  Front. Neurosci (2016). 10:176.
  • Code (jAER)
• Bardow, P. A., Davison, A. J., Leutenegger, S.,
  Simultaneous Optical Flow and Intensity Estimation from an Event Camera,
  IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016. YouTube
• Liu, M., Delbruck, T.,
  Block-Matching Optical Flow for Dynamic Vision Sensors: Algorithm and FPGA Implementation,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Baltimore, MD, 2017.

Intensity-Image Reconstruction from events

• Cook, M., Gugelmann, L., Jug, F., Krautz, C., Steger, A.,
  Interacting maps for fast visual interpretation,
  Int. Joint Conf. on Neural Networks (IJCNN), San Jose, CA, 2011, pp. 770-776. YouTube
  • Martel, J. N. P., Cook, M.,
    A Framework of Relational Networks to Build Systems with Sensors able to Perform the Joint Approximate Inference of Quantities,
    IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Workshop on Unconventional Computing for Bayesian Inference, 2015, Hamburg. PDF
  • Martel, J. N. P., Chau, M., Dudek, P., Cook, M.,
    Toward joint approximate inference of visual quantities on cellular processor arrays,
    IEEE Int. Symp. Circuits and Systems (ISCAS), Lisbon, 2015, pp. 2061-2064.
• Kim, H., Handa, A., Benosman, R., Ieng, S.-H., Davison, A. J.,
  Simultaneous Mosaicing and Tracking with an Event Camera, British Machine Vision Conference, 2014. PDF, YouTube.
  • Code for intensity reconstruction.
• Barua, S., Miyatani, Y., Veeraraghavan, A.,
  Direct face detection and video reconstruction from event cameras,
  IEEE Winter Conf. Applications of Computer Vision (WACV), Lake Placid, NY, 2016, pp. 1-9. YouTube
• Bardow et. al. CVPR 2016,
  Simultaneous Optical Flow and Intensity Estimation from an Event Camera.
• Reinbacher, C., Graber, G., Pock, T.,
  Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation,
  British Machine Vision Conf. (BMVC), York, 2016. PDF, YouTube, Code
• Moeys, D. P., Li, C., Martel, J. N. P., Bamford, S., Longinotti, L., Motsnyi, V., Bello, D. S. S., Delbruck, T.,
  Color Temporal Contrast Sensitivity in Dynamic Vision Sensors,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Baltimore, MD, 2017. PDF.

Localization and Ego-Motion Estimation

• Cook et. al. IJCNN 2011,
  Interacting maps for fast visual interpretation.
  Joint estimation of optical flow, image intensity and angular velocity with a rotating event camera.
• Weikersdorfer, D. and Conradt, J.,
  Event-based particle filtering for robot self-localization,
  IEEE Int. Conf. on Robotics and Biomimetcs (ROBIO), Guangzhou, 2012, pp. 866-870. PDF
• Censi, A., Strubel, J., Brandli, C., Delbruck, T., Scaramuzza, D.,
  Low-latency localization by Active LED Markers tracking using a Dynamic Vision Sensor,
  IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Tokyo, 2013. PDF, Slides
• Mueggler, E., Huber, B., Scaramuzza, D.,
  Event-based, 6-DOF Pose Tracking for High-Speed Maneuvers,
  IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Chicago, IL, 2014, pp. 2761-2768. PDF, YouTube
• Gallego, G., Forster, C., Mueggler, E., Scaramuzza, D.,
  Event-based Camera Pose Tracking using a Generative Event Model,
  arXiv:1510.01972, 2015.
• Mueggler, E., Gallego G., Scaramuzza, D.,
  Continuous-Time Trajectory Estimation for Event-based Vision Sensors,
  Robotics: Science and Systems XI (RSS), Rome, Italy, 2015. [PDF]
• Gallego, G., Lund, J.E.A., Mueggler, E., Rebecq, H., Delbruck, T., Scaramuzza, D.,
  Event-based, 6-DOF Camera Tracking for High-Speed Applications,
  (Under review), 2016. YouTube
• Reinbacher, C., Munda, G., Pock, T.,
  Real-Time Panoramic Tracking for Event Cameras,
  IEEE Int. Conf. Computational Photography (ICCP), Stanford, CA, USA, 2017, pp. 1-9. PDF, YouTube, Code
• Mueggler et. al. IJRR 2017.
  The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM.
• Vasco, V., Glover, A., Mueggler, E., Scaramuzza, D., Natale, L., Bartolozzi, C.
  Independent Motion Detection with Event-driven Cameras,
  Int. Conf. Advanced Robotics (ICAR), Hong Kong, 2017. PDF

Visual Odometry and SLAM (Simultaneous Localization And Mapping)

• Weikersdorfer, D., Hoffmann, R., Conradt. J.,
  Simultaneous localization and mapping for event-based vision systems.
  Int. Conf. Computer Vision Systems (ICVS), 2013, pp. 133-142. PDF, Slides
• Censi, A. and Scaramuzza, D.,
  Low-latency Event-based Visual Odometry,
  IEEE Int. Conf. Robotics and Automation (ICRA), Hong-Kong, 2014, pp. 703-710. PDF, Slides
• Weikersdorfer, D., Adrian, D. B., Cremers, D., Conradt, J.,
  Event-based 3D SLAM with a depth-augmented dynamic vision sensor,
  IEEE Int. Conf. Robotics and Automation (ICRA), Hong-Kong, 2014, pp. 359-364.
• Weikersdorfer, D.,
  Efficiency by Sparsity: Depth-Adaptive Superpixels and Event-based SLAM.
  Ph.D. Thesis, Technical University of Munich, Munich, Germany, 2014. PDF
• Kueng, B., Mueggler, E., Gallego, G., Scaramuzza, D.,
  Low-Latency Visual Odometry using Event-based Feature Tracks,
  IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Daejeon, South Korea, 2016, pp. 16-23. PDF. YouTube
• Kim, H., Leutenegger, S., Davison, A.J.,
  Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera,
  European Conference on Computer Vision (ECCV), 2016, pp. 349-364. PDF, YouTube
• Rebecq, H., Horstschaefer, T., Gallego, G., Scaramuzza, D.,
  EVO: A Geometric Approach to Event-based 6-DOF Parallel Tracking and Mapping in Real-time,
  IEEE Robotics and Automation Letters (RA-L), 2:2(593-600), 2017. PDF, Youtube.
• Gallego, G. and Scaramuzza, D.,
  Accurate Angular Velocity Estimation with an Event Camera,
  IEEE Robotics and Automation Letters (RA-L), 2:2(632-639), 2017. PDF, Youtube.
• Mueggler et. al. IJRR 2017.
  The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM.

Visual-Inertial State Estimation

• Mueggler, E., Gallego, G., Rebecq, H., Scaramuzza, D.,
  Continuous-Time Visual-Inertial Trajectory Estimation with Event Cameras,
  (Under review), 2017.
• Mueggler et. al. IJRR 2017.
  The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM.
• Zhu, A., Atanasov, N., Daniilidis, K.,
  Event-based Visual Inertial Odometry,
  IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 2017.
• Rebecq, H., Horstschaefer, T., Scaramuzza, D.,
  Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization,
  British Machine Vision Conf. (BMVC), London, 2017.

Visual Stabilization

• Delbruck, T., Villanueva, V., Longinotti, L.,
  Integration of dynamic vision sensor with inertial measurement unit for electronically stabilized event-based vision,
  IEEE Int. Symp. Circuits and Systems (ISCAS) 2014, 2636-2639. YouTube

Video Processing

• Brandli, C., Muller, L., Delbruck, T.,
  Real-time, high-speed video decompression using a frame- and event-based DAVIS sensor,
  IEEE Int. Symp. on Circuits and Systems (ISCAS), Melbourne VIC, 2014, pp. 686-689.

Pattern Recognition

• Lee, J., Delbruck, T., Park, P. K. J., Pfeiffer, M., Shin, C. W., Ryu, H., Kang, B. C.,
  Live demonstration: Gesture-Based remote control using stereo pair of dynamic vision sensors,
  IEEE Int. Symp. Circuits and Systems (ISCAS) 2012, Seoul, South Korea, pp. 736-740. PDF, YouTube
• Barua et. al. WACV 2016. Face recognition.
• Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N., Benosman, R.,
  HFIRST: A Temporal Approach to Object Recognition,
  IEEE Trans. Pattern Anal. Machine Intell. (TPAMI), 2015, 37(10):2028-2040. PDF
  • Code: HFIRST: A simple spiking neural network for recognition based on the canonical frame-based HMAX model.
• Moeys, D., Corradi F., Kerr, E., Vance, P., Das, G., Neil, D., Kerr, D., Delbruck, T.,
  Steering a Predator Robot using a Mixed Frame/Event-Driven Convolutional Neural Network,
  IEEE Int. Conf. Event-Based Control Comm. and Signal Proc. (EBCCSP), Krakow, Poland, 2016. PDF, YouTube 1, YouTube 2
• Lagorce, X., Orchard, G., Gallupi, F., Shi, B., Benosman, R.,
  HOTS: A Hierarchy Of event-based Time-Surfaces for pattern recognition,
  IEEE Trans. Pattern Anal. Machine Intell. (TPAMI), 2017, 39(7):1346-1359.
• Lungu, I.-A., Corradi, F., Delbruck, T.,
  Live Demonstration: Convolutional Neural Network Driven by Dynamic Vision Sensor Playing RoShamBo,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Baltimore, MD, 2017. YouTube, Slides 36-39

Control

• Delbruck, T. and Lichtsteiner, P.,
  Fast sensory motor control based on event-based hybrid neuromorphic-procedural system,
  IEEE Int. Symp. Circuits and Systems, New Orleans, LA, 2007, pp. 845-848.
• Conradt, J., Cook, M., Berner, R., Lichtsteiner, P., Douglas, R. J., Delbruck, T.,
  A Pencil Balancing Robot Using a Pair of AER Dynamic Vision Sensors,
  IEEE Int. Symp. Circuits and Systems (ISCAS) 2009, pp. 781-784, 2009. PDF, Poster, Project page, YouTube 1, YouTube 2, YouTube 3
• Conradt, J., Berner, R., Cook, M., Delbruck, T.,
  An embedded AER dynamic vision sensor for low-latency pole balancing,
  IEEE Int. Conf. Computer Vision Workshops (ICCVW), Kyoto, Japan, 2009. PDF
• Delbruck, T. and Lang, M.,
  Robotic Goalie with 3ms Reaction Time at 4% CPU Load Using Event-Based Dynamic Vision Sensor,
  Front. Neurosci. (2013) 7:223. PDF, YouTube
• Censi, A.,
  Efficient Neuromorphic Optomotor Heading Regulation,
  American Control Conference (ACC), Chicago, IL, 2015, pp. 3854-3861.
• Mueggler, E., Baumli, N., Fontana, F., Scaramuzza, D.,
  Towards Evasive Maneuvers with Quadrotors using Dynamic Vision Sensors,
  Eur. Conf. Mobile Robots (ECMR), Lincoln, 2015. PDF
• Delbruck, T., Pfeiffer, M., Juston, R., Orchard, G., Mueggler, E., Linares-Barranco, A., Tilden, M. W.,
  Human vs. computer slot car racing using an event and frame-based DAVIS vision sensor,
  IEEE Int. Symp. Circuits and Systems (ISCAS), Lisbon, 2015, pp. 2409-2412. YouTube 1, YouTube 2
• Moeys et. al. EBCCSP 2016. VISUALISE Predator/Prey Dataset.

Datasets and Simulators (sorted by topic)

• Several datasets from the Sensors group at INI (Institute of Neuroinformatics), Zurich:
  • DVS128 Dynamic Vision Sensor Silicon Retina data

Optical Flow

• DVS/DAVIS Optical Flow Dataset associated to the paper Rueckauer and Delbruck, FNINS 2016.
• Binas et. al. ICML 2017. DDD17: End-To-End DAVIS Driving Dataset.

Visual Odometry and SLAM

• Combined Dynamic Vision / RGB-D Dataset associated to the paper Weikersdorfer et. al. ICRA 2014.
• Barranco, F., Fermuller, C., Aloimonos, Y.,
  A Dataset for Visual Navigation with Neuromorphic Methods,
  Front. Neurosci. (2016), 10:49.
• E. Mueggler, H. Rebecq, G. Gallego, T. Delbruck, D. Scaramuzza,
  The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM,
  Int. J. Robotics Research, 36:2, pp. 142-149, 2017. PDF, PDF IJRR, Dataset.
• Binas, J., Neil, D., Liu, S.-C., Delbruck, T.,
  DDD17: End-To-End DAVIS Driving Dataset,
  Int. Conf. Machine Learning, Sydney, Australia, PMLR 70, 2017. Dataset

Recognition

• Orchard, G., Jayawant, A., Cohen, G.K., Thakor, N.,
  Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades,
  Front. Neurosci. (2015), 9:437. YouTube
  • Neuromorphic-MNIST (N-MNIST) dataset is a spiking version of the original frame-based MNIST dataset (of handwritten digits). YouTube
  • The Neuromorphic-Caltech101 (N-Caltech101) dataset is a spiking version of the original frame-based Caltech101 dataset. YouTube
• VISUALISE Predator/Prey Dataset associated to the paper Moeys et. al. EBCCSP 2016
• Hu, Y., Liu, H., Pfeiffer, M., Delbruck, T.,
  DVS Benchmark Datasets for Object Tracking, Action Recognition, and Object Recognition,
  Front. Neurosci. 10:405. Dataset

Software

Drivers

• jAER (java Address-Event Representation) project. Real time sensory-motor processing for event-based sensors and systems. github page. Wiki
• caer (AER event-based framework, written in C, targeting embedded systems)
• libcaer (Minimal C library to access, configure and get/send AER data from sensors or to/from neuromorphic processors)
• ROS (Robotic Operating System)

Calibration

• Lens focus adjustment or this other source.
• For the DAVIS: use the grayscale frames to calibrate the optics of both frames and events.
  • ROS camera calibrator (monocular or stereo)
  • kalibr software by ASL - ETH.
• For the DAVIS camera and IMU calibration: kalibr software by ASL - ETH, using the grayscale frames.
• For the DVS (events-only):
  • Calibration using blinking LEDs or computer screens by RPG - UZH.
  • DVS camera calibration by G. Orchard.
  • DVS camera calibration by VLOGroup at TU Graz.

Algorithms

• Several event-processing filters in the jAER (java Address-Event Representation) project

• Optical Flow

  • LocalPlanesFlow, inspired by the paper Benosman et. al. TNNLS 2014.
  • Several algorithms compared in the paper by Rueckauer and Delbruck, FNINS 2016.
  • Event-Lifetime estimation, associated to the paper Mueggler et. al. ICRA 2015.

• Intensity-Image reconstruction

  • Code for intensity reconstruction, inspired by the paper Kim et. al. BMVC 2014.
  • DVS reconstruction code associated to the paper Reinbacher et. al. BMVC 2016.

• Localization and Ego-Motion Estimation

  • Panoramic tracking code associated to the paper Reinbacher et. al. ICCP 2017.

• Pattern Recognition

  • A simple spiking neural network for recognition associated to the paper Orchard et. al. TPAMI 2015.

Utilities

• Process AEDAT: useful scripts to work with data from jAER and cAER.
• Matlab functions in jAER project
• AEDAT Tools: scripts for Matlab and Python to work with aedat files.
• Matlab AER functions by G. Orchard. Some basic functions for filtering and displaying AER vision data, as well as making videos.
• Python code for AER vision data by G. Orchard.
• edvstools, by D. Weikersdorfer: A collection of tools for the embedded Dynamic Vision Sensor eDVS.

Neuromorphic Processors and Platforms

• Dynamic Neuromorphic Asynchronous Processor (DYNAP) by iniLabs
  • Qiao, N., Mostafa, H., Corradi, F., Osswald, M., Stefanini, F., Sumislawska, D., Indiveri, G.,
    A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses,
    Front. Neurosci. (2015) 9:141. PDF
  • Indiveri, G., Qiao, N., Corradi, F.,
    Neuromorphic Architectures for Spiking Deep Neural Networks,
    IEEE Int. Electron Devices Meeting (IEDM), Washington, DC, 2015, pp. 4.2.1-4.2.4. PDF
• Wiesmann, G., Schraml, S., Litzenberger, M., Belbachir, A. N., Hofstatter, M., Bartolozzi, C.,
  Event-driven embodied system for feature extraction and object recognition in robotic applications,
  IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Providence, RI, 2012, pp. 76-82.
• Galluppi, F., Denk, C., Meiner, M. C., Stewart, T. C., Plana, L. A., Eliasmith, C., Furber, S., Conradt, J.,
  Event-based neural computing on an autonomous mobile platform,
  IEEE Int. Conf. Robotics and Automation (ICRA), Hong Kong, 2014, pp. 2862-2867. PDF

Workshops and Tutorials

• ICRA 2015 Workshop on Innovative Sensing for Robotics, with a focus on Neuromorphic Sensors.
• Event-Based Vision for High-Speed Robotics (slides) IROS 2015, Workshop on Alternative Sensing for Robot Perception.
• ICRA 2017 First International Workshop on Event-based Vision.
• The Telluride Neuromorphic Cognition Engineering Workshops.
• Capo Caccia Workshops toward Cognitive Neuromorphic Engineering.

People / Organizations

• Institute of NeuroInformatics (INI) of the University of Zurich (UZH) and ETH Zurich.
• iniLabs (Comerzialization of neuromorphic technology from INI).
• Dynamic Vision Sensor (DVS) - asynchronous temporal contrast silicon retina
• Robotics and Perception Group (RPG-UZH).

Contributing

Please see CONTRIBUTING for details.

---