3D Scanning the Virtual World
There’s a lot of photogrammetry apps out there for 3D scanning the real world. Those techniques could also apply in the virtual world. For example, by taking screenshots of a scene in a video game, it should be possible to extract a 3D model, without decoding the native files.
Here, I 3D scanned Harvest Moon: A New Beginning (Harvest Moon is one of my favorite video game series).
Extracting a scene from Harvest Moon: A New Begining is perfectly challenging. The game is played on a Nintendo 3DS, which has no native connections to a computer. To get the native 3D models, (I’m not even sure if this is possible), you probably would have to reverse engineer the ROM/game data. Luckily, you could install a homebrew 3DS video capture board onto the 3DS to output video to a USB connection. The outputted video could be used for 3D scanning!
The game also has a 3D world with some camera rotation movement. Camera rotation is crucial because otherwise there would not be enough information captured by the images. The game is also shaded in a cartoon-like way. This style of shading (which I love) will pose challenges to 3D scanning, which is expecting real-world light/color conditions.
Unfortunately, I don’t have a capture board. I could theoretically use my camera to take photos of the screen, but there would be a lot of noise. The next best thing was a Youtube video I found (by Thomal9) that had 2 seconds of footages I could use (Youtube video: time 7:41-7:42).
Above is one of the images I used for my 3D scanning. I used a total 12. They are not high in resolution.
I used VisualFSM and Meshlab, and followed Jesse’s Open Source Photogrammetry guide and got the following result with the default Poisson surface reconstruction (the same in the gif up top).
*One note about using VisualFSM on Windows, which differs from the instructions, you need all the files from the CMVS-PMVS folder for Yasutaka Furukawa’s patch-based dense reconstruction, and not just pthreadVc2.dll.
Here is an alternate reconstruction using 4 for both octree depth and solver divide. The house is much better formed but the detail of the mailbox and the farmer are complete lost.
Given the low resolution of the screenshots and the lack of diversity of the scene, the 3D scanned model of Harvest Moon: A New Beginning came out pretty great. As a proof-of-concept, it shows that 3D scanning virtual worlds is viable. With a little cleanup, these 3D scanned models can be 3D printed, to create custom figurines when the native models aren’t available.
In the future, I’m going to try 3D scanning a high-resolution native PC game to get better results.
Caret Dash Caret 