Instructors of record: Yann LeCun, Denis Zorin
Coordinator: Alfredo Canziani
Credits: 2
Grading: Pass/Fail

Offered: Spring 2021

This team-taught course provides a high-level overview of the keyideasand technologies that lead to revolutionary changes in Artificial Intelligence (AI) and to the explosive growth in practical applications of AI.Taught by a team of NYU's top experts in artificial intelligence lead by the Turing award winner Yann LeCun, the course will introduce students to a range of topics in fundamentals of AI and its key sub-areas including machine learning, natural language processing, computer vision, as well as its applications in several domains.

The course will be complimentary with respect to “Thinking, Learning, and Consciousness in Humans and Machines”, which considers the impact of AI technologies on society and relationship between human and artificial intelligence. Our course focuses on the key scientific ideas underlying revolutionary advances in AI technology.

The course will consist of a sequence of 14 lectures, approximately 45 min in duration each. The lectures will be pre-recorded and watched asynchronously. There will be a weekly discussion session with the presenters, and an online discussion board. Assessment will be based on brief weekly online questionnaires.


Syllabus of Lectures (tentative)

  1. Introduction; Deep Learning revolution in AI. (Yann LeCun)
  2. AI for understanding visual information: computer vision. Machine learning foundations of computer vision (Rob Fergus)
  3. Vision continued (Laurens van der Maaten)
  4. Natural language processing: machine learning foundations, language generation, dialog(Kyunghyun Cho)
  5. Engineering behind AI dialog systems: (Emily Dinan)
  6. Speech understanding and synthesis (Michael Picheny)
  7. Autonomous robots, reinforcement learning with robotics applications robotics. (Lerrel Pinto)
  8. Robotics (Andy Zheng)
  9. AI and games: how machines learned to beat humans in games (Noam Brown)
  10. AI for medicine: saving lives with machine learning. (Rajesh Ranganath)
  11. Machine learning in medical imaging: challenges and opportunities (Krzysztof J. Geras)
  12. Fairness in AI: (Emily Denton)
  13. Conclusion: What is next in AI? (Yann LeCun)

Lecture title:  Deep Learning revolution in AI.

Lecture topics:  Brief history of AI,   What AI can do,   basic ideas of ML, supervised learning, gradient descent,  multilayer networks, ConvNets

Reading materials:

  • LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. nature, 521(7553), 436-444.

Lecturer: Yann LeCun,  Silver Professor of Computer Science, Data Science, and Neural Science.  VP and Chief Scientist, Facebook; ACM Turing Award Laureate

Yann LeCun is VP & Chief AI Scientist at Facebook and Silver Professor at NYU affiliated with the Courant Institute of Mathematical Sciences & the Center for Data Science. He was the founding Director of Facebook AI Research and of the NYU Center for Data Science. He received an Engineering Diploma from ESIEE (Paris) and a PhD from Sorbonne Université. After a postdoc in Toronto he joined AT&T Bell Labs in 1988, and AT&T Labs in 1996 as Head of Image Processing Research. He joined NYU as a professor in 2003 and Facebook in 2013. His interests include AI machine learning, computer perception, robotics and computational neuroscience. He is the recipient of the 2018 ACM Turing Award (with Geoffrey Hinton and Yoshua Bengio) for "conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing", a member of the National Academy of Engineering and a Chevalier de la Légion d’Honneur.


Lecture title:  AI for understanding visual information:  computer vision.  Machine learning foundations of computer vision

Lecture topics:  Computer vision,  object recognition, generative image models,, self-supervised learning.

Reading materials:  

Imagenet classification with deep convolutional neural networks, Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, NeurIPS 2012. 

Visualizing and Understanding Convolutional NetworksMatthew D Zeiler, Rob Fergus, ECCV 2014.

Deep Residual Learning for Image Recognition, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. CVPR 2015. 

Mask R-CNN, Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick, ICCV 2017. 

A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. ICML 2020. 

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Alec Radford, Luke Metz, Soumith Chintala, ICLR 2016. 

Analyzing and Improving the Image Quality of StyleGAN, Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila

Lecturer: Rob Fergus,  Professor of Computer Science + DeepMind

Rob Fergus is a Professor of Computer Science at NYU. He is also a research scientist at Google DeepMind. He has degrees from Cambridge, Oxford and Caltech, all in Electrical Engineering. Before coming to NYU, he spent two years as a postdoc at MIT. His research focuses on machine learning, computer vision and deep learning.

Lecture title: Natural language processing: machine learning foundations, language generation, dialog

Lecture topics:  NLP,   language generation, machine translation, what happens in dialog systems like GPT.

Lecturer: Kyunghyun Cho, Associate Professor of Computer Science and Data Science

Kyunghyun Cho is an associate professor of computer science and data science at New York University and CIFAR Associate Fellow. He was a postdoctoral fellow at University of Montreal until Summer 2015 under the supervision of Prof. Yoshua Bengio, after receiving PhD and MSc degrees from Aalto University April 2011 and April 2014, respectively, under the supervision of Prof. Juha Karhunen, Dr. Tapani Raiko and Dr. Alexander Ilin.  Cho and his collaborators have extensively investigated natural language processing and machine translation. This has resulted in an attention mechanism for artificial neural networks and a new paradigm in machine translation, called ‘neural machine translation.’ It has not only advanced research, but has also been adopted by industry to produce better machine translation systems.  He tries his best to find a balance among machine learning, natural language processing, and life, but almost always fails to do so.

Lecture title:  AI for Speech Processing: Overview and Recent Developments

Lecture topics: The lecture will cover the basics of four key long-standing problems in speech processing today: Speech Recognition, Speech Synthesis, Speaker Verification, and Spoken Language Understanding, and how these areas have been revolutionized through breakthroughs in Deep Learning  

Background Readings:

Chorowski, Jan K., Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. "Attention-based models for speech recognition." In Advances in neural information processing systems, pp. 577-585. 2015.

Wang, Yuxuan, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang et al. "Tacotron: Towards end-to-end speech synthesis." arXiv preprint arXiv:1703.10135 (2017).

Li, Chao, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu. "Deep speaker: an end-to-end neural speaker embedding system." arXiv preprint arXiv:1705.02304 650 (2017).

Serdyuk, Dmitriy, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu, and Yoshua Bengio. "Towards end-to-end spoken language understanding." In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5754-5758. IEEE, 2018.

Lecturer: Michael  Picheny,  Research Professor, Computer Science and Data Science,   Formerly IBM T. J. Watson Research Center

Michael Picheny received his B.S., M.S., and Ph.D. degrees from the Massachusetts Institute of Technology, Cambridge. He recently joined New York University in the Courant-Computer Science and the Center for Data Science as a research professor after many years as the senior manager of speech technologies in IBM Research AI at the IBM T.J. Watson Research Center, Yorktown Heights, New York. He has worked in the speech recognition area since 1981, when he joined IBM. He served as an associate editor of IEEE Transactions on Acoustics, Speech, and Signal Processing from 1986 to 1989 and as the chair of the Speech Technical Committee of the IEEE Signal Processing Society from 2002 to 2004. He was a member of the board of the International Speech Communication Association from 2005 to 2013 and was named an International Speech Communication Association fellow in 2014. He was the general chair of the 2011 IEEE Automatic Speech Recognition and Understanding Workshop. He is currently a distinguished industry speaker of the IEEE Signal Processing Society. He has published numerous papers in journals and at conferences on nearly every aspect of speech recognition. He is a Fellow of the IEEE.

Lecture title:   Autonomous robots: teaching robots to drive and fly safely

Lecture topics: This lecture will cover broad applications of machine learning techniques for a variety of real-world applications such as robotic manipulation, navigation, and self-driving cars. We will explore the core technology and ideas that makes these applications possible and discuss areas that require additional research. Topics will include: neural networks for predicting robotic action; reinforcement learning for dexterous manipulation; imitation learning from humans.

Reading materials:

  1. Robots Use AI to Learn to Clean Your House 
  2. AI Researchers Devise Cheap Data Collectio Method to Scale Training Robots
  3. Deep Learning Robot Takes 10 Days to Teach Itself to Grasp

Lecturer: Lerrel Pinto, Assistant Professor of Computer Science

Lerrel Pinto  is  an assistant professor of Computer Science.   His research interests focus on mearning and computer vision for robots. He received a PhD degree from CMU in 2019; prior to that he received an  MS degree from CMU in 2016, and a B.Tech in Mechanical Engineering from IIT-Guwahati. His work on large-scale learning for grasping received the Best Student Paper award at ICRA 2016. Several of his works have been featured in popular media like TechCrunch, MIT Tech Review and BuzzFeed among others.


Lecture title: AI and games:  how machines learned to beat humans in games. 

Lecture topics: Adversarial search and learning, regret minimization, AI for Go, AI for poker.

In order for an AI bot to be successful in the real world, it must be able to understand how humans and other agents might adapt to the behavior of the bot. This ability to adapt poses a serious challenge to learning techniques designed for stationary, single-agent environments. This lecture will focus on adversarial multi-agent settings and describe techniques used to compute equilibria in perfect-information and imperfect-information games. These techniques have led to the creation of champion AI agents in popular games such as Go and Poker.

Reading materials:

A Simple Alpha(Go) Zero Tutorial 

Intro to Regret Minimization  


AlphaGo Zero Paper 

Pluribus Paper

Lecturer: Noam Brown,  Research Scientist, Facebook AI Research NYC  

Noam Brown is a Research Scientist at Facebook AI Research.  He earned his PhD at Carnegie Mellon University. His research interests focus on multi-agent artificial intelligence and computational game theory. He has applied this research toward creating Libratus and Pluribus, the first AIs to defeat top humans in no-limit poker. Pluribus was named one of Science Magazine’s Top Ten Breakthroughs of the Year for 2019 and Libratus was a finalist for the same award in 2017. Noam received a NeurIPS Best Paper award in 2017, the 2017 Allen Newell Award for Research Excellence, and the 2019 Marvin Minsky Medal for Outstanding Achievements in AI. He was named a 2019 Innovator Under 35 by MIT Tech Review. 

Lecture title:   AI for medicine: saving lives with machine learning

Lecture topics:  Healthcare data, models for medical data,  sepsis,  covid

Lecturer: Rajesh Ranganath,  Assistant Professor of Computer Science

Lecture title:  Machine learning in medical imaging: challenges and opportunities

Lecture topics:
Explainable AI, learning from very large images, applications in medical imaging, breast cancer diagnosis

Longer description: Although deep neural networks have already achieved a good performance in many medical image analysis tasks, their clinical implementation is slower than many anticipated a few years ago. One of the critical issues that remains outstanding is the lack of explainability of the commonly used network architectures imported from computer vision. In my talk, I will provide an overview of the challenges and opportunities in machine learning applications in medical imaging. I will also explain how to train deep neural networks, tailored to medical image analysis, in which making a prediction is inseparable from explaining it.

Reading materials:

Wu et al. Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE TMI, 2019. 

Blog post summarizing this paper

Shen et al. An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization.

Shamout et al. An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department.

Additional resources:

Lecturer: Krzysztof J. Geras, Assistant Professor of Radiology

Krzysztof is an assistant professor at NYU School of Medicine and an affiliated faculty at NYU Center for Data Science. His main interests are in unsupervised learning with neural networks, model compression, transfer learning, evaluation of machine learning models and applications of these techniques to medical imaging. He previously did a postdoc at NYU with Kyunghyun Cho, a PhD at the University of Edinburgh with Charles Sutton and an MSc as a visiting student at the University of Edinburgh with Amos Storkey. His BSc is from the University of Warsaw.