CILVR SEMINAR: World Knowledge in the Time of Large Models

This talk will discuss the massive shift that has come about in the vision and ML community as a result of the large pre-trained language and language and vision models such as Flamingo, GPT-4, and other models. We begin by looking at the work on knowledge-based systems in CV and robotics before the large model revolution and discuss the impact it had. This impact can be broken down into three areas in which world knowledge should be studied in the context of these new models: evaluation, harnessing large models, and building outside knowledge. First, evaluating world knowledge is even more important as the large model revolution gives more easy access to world knowledge. Next, we discuss recent work in harnessing models such as Flamingo and Chinchilla for visual and procedural knowledge. Finally, the talk discusses how, by focusing on knowledge acquisition as an agent-centric problem, we can make developments in retrieving and collecting world knowledge.