NVIDIA Artificial Intelligence plays Minecraft, and wins a prize (but needed 750 GPUs)

MineDojo is an Artificial Intelligence (AI) program created by NVIDIA that can play Minecraft. To teach MineDojo how to play Minecraft, the devs analysed over 730,000 YouTube videos, 7,000 wiki pages, 340,000 Reddit posts, and 6.6 million Reddit comments explaining the game.

Based on this information, the researchers developed the MineCLIP interaction model, which correlates video clips with certain Minecraft in-game actions. As a result, a player can instruct MineDojo what to do in-game using plain language, such as "discover a pyramid in the desert" or "construct a nether portal and enter," and MineDojo will execute the necessary series of actions to make it happen.

As expected, they had to train the AI with tens of thousands of hours of the aforementioned content and hundreds of NVIDIA GPUs to do this. There were a total of 32 NVIDIA A100 and 720 NVIDIA Tesla V100 GPUs to make this happen. All of their efforts paid off when the AI was able to construct a diamond pickaxe in less than four minutes.

The objective of MineDojo is to watch and learn in order to develop and implement actions in the game subsequently. According to NVIDIA, "While academics have trained other AIs in gaming contexts such as StarCraft, Dota, and Go for decades, they always have specialised in only a few skills. Therefore, NVIDIA researchers went to Minecraft, the world's most popular game, to construct a scalable, taught AI capable of executing a wide variety of jobs successfully."

The IA MINEDOJO paper has been awarded by NeurlPS 2022. Linxi Fan and Guanzhi Wang of NVIDIA, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar from a variety of academic institutions are its authors.

On its official website, you can view MineDojo in action, and the source code for MineDojo and MineCLIP is accessible on GitHub.