As a research institute of Toei Company, the Tsukun Laboratory explores how to integrate digital technology into film culture under the concept of "virtual production." Among their key research themes is the "digital set," which involves reconstructing studio sets and filming locations using 3DCG technology to revolutionize the workflow of video production. Traditionally, digital sets were created using photogrammetry, but since 2018, they have introduced 3D laser scanners. By utilizing Leica Geosystems' imaging laser scanner, the Leica BLK360, they have added accuracy, speed, and flexibility to digital set production, streamlining the workflow. This approach to film production holds significant potential for creating new forms of entertainment.
Photogrammetry Alone Cannot Meet the Increasing Demands
Toei Company produces a wide variety of visual works, including movies and TV dramas. Within Toei's Tokyo Studio in Nerima, Tokyo, the Toei Digital Center, which consolidates the post-production activities of the Toei Group, and the Tsukun Laboratory, a research institute, are located. The mission of the Tsukun Laboratory is to "Design the Future of Visual Production." Their development of "virtual production" aims to eliminate time and distance constraints from the filming process, thereby broadening the scope of expression.
"For example, controlling the weather at a filming location. If it rains, the director, actors, and everyone else are stalled, or they have to set up the camera and wait for the right clouds to drift into place. With a digital set, we can create a stadium filled with spectators without gathering large numbers of extras, and provide precise direction. It allows for seamless filming of microscopic journeys through the human body or sword fights inside a burning castle, using various virtual production techniques. Dangerous shoots, like explosions, can be conducted safely," says Shingo Kobayashi, VFX Supervisor at Tsukun Laboratory. "Virtual production" liberates traditional production methods from their constraints, with the "digital set" playing a crucial role as a virtual art set.
The Evolution of Digital Set Creation in Three Stages
From around 2009 to 2010, when the Tsukun Laboratory was established, all locations and props were measured manually with a tape measure, and the data was inputted into a computer to create digital sets using CG. However, creating accurate and high-quality models with this method required a great deal of effort.
By around 2014 to 2015, they began using photogrammetry software that automatically analyzed and synthesized photos taken with digital cameras to generate 3D models. This method significantly improved the expressiveness and quality of digital sets, and also greatly reduced the effort and time required for production.
"However, since photos don't have absolute coordinates, it's impossible to get accurate sizes. When synthesizing a vast number of photos to create 3D models, distortions inevitably occur," says Mr. Kobayashi. It was also time-consuming to take hundreds to thousands of overlapping photos with slight shifts. Moreover, the accuracy of the analysis depended on the quality of the photos, requiring good lighting and proper techniques for capturing edges.
The demand for digital sets ranges from small objects like stuffed animals to entire rooms and even street scenes. It became difficult to meet these needs with photogrammetry alone. Therefore, they started paying attention to 3D laser scanners as a means to obtain accurate, distortion-free, and deep 3D data. They began full-scale test trials in the spring of 2018.
The Encounter with BLK360 Transformed Digital Set Production
"We first borrowed a 3D laser scanner from another company to try it out. Our impression was, 'It's slow! It's heavy!'" Mr. Kobayashi laughs.
One scan took 15 minutes, and it took an entire day to scan the exterior of a single house. While the data obtained was high-precision and could generate accurate 3D models, the reproduction of textures was unsatisfactory. For example, weathered tile roofs are better captured with photos. This meant that neither the 3D laser scanner nor photos alone were sufficient. They sought a workflow that combined both, aiming to get the best of both worlds.
"While we understood the direction we should go in, we couldn't find any precedents in the visualization field. During our search, we accidentally came across a blog and photos of someone using the BLK360. 'Is this small, black device really a 3D laser scanner?' Astonished, we investigated and eventually found our way to the BLK360," says Mr. Kobayashi. They quickly borrowed a BLK360 and the 3D point cloud data processing software「Leica Cyclone REGISTER 360」from Leica Geosystems to test it at Toei Kyoto Studio Park.
"The previous 3D laser scanner we tested was difficult for me to carry around. But the BLK360 is 'light! Fast!' It's easy to carry in one bag and simple to assemble. In Low Density mode, focused on scanning speed, each scan takes less than three minutes, and even in Middle Density mode, it's less than five minutes. This made the repeated operation of 'place, scan, move, and scan again' very easy," says Ms. Yume Kari Tsuji, a CG designer at Tsukun Laboratory.
Photogrammetry can't capture photos in dark places, but a 3D laser scanner can scan even in relatively low light, which is a significant advantage. Moreover, the BLK360 offers just the right balance of mobility and scanning speed. Despite its low cost, it provides high-accuracy shape acquisition. It can be connected to an iPad via WiFi, allowing the scanning process to be monitored in real-time.
"We compared it with higher-precision models from Leica Geosystems, but our goal isn't to accurately replicate structures. Our focus is to create digital sets with the quality needed for the director's vision in the shortest possible time. In that sense, the BLK360 is the ideal product for cinematic virtual production. To say the least, in 2018 it was 'the one and only choice,'" Mr. Kobayashi states. After confirming its smooth integration with various tools and its ability to combine with photogrammetry, they purchased the BLK360 at the end of 2018.
Dramatic Evolution in Expression and Quality Through the Fusion of Photogrammetry and 3D Scanning
Let's introduce three examples of utilizing the BLK360.
Example 1: Digital Set Creation of Street Scenes at Toei Uzumasa Eigamura
The first example is the challenge of creating a digital set of street scenes at Toei Uzumasa Eigamura. In this vast movie village, they modeled three blocks of buildings, including small alleys, to create a system that can be used for future events and shooting simulations.
The 3D scanning covered 30 to 40 locations, and 4,000 to 5,000 photos were taken, some of which were also used as video data.
BLK360 scanned and meshed image of buildings in Toei Movie Village
A digital set that combines 3D scan data and photographic data and reconstructs the real building in a richly expressive manner by "combining the best of both worlds".
Example 2: Komine Castle in Shirakawa, Fukushima Prefecture
Komine Castle was burned down in the Boshin War and later abandoned, with only parts of the stone walls and moat remaining. However, in the early Heisei period, the three-story turret and front gate were restored based on Edo-period illustrations and excavation surveys, reminiscent of their former appearance. At the Komine Castle Historical Museum within the premises, visitors can enjoy various seasonal views of the Edo-period Komine Castle and samurai figures in a VR space using the "VR Telescope."
They can also experience the vastness of Edo-period Komine Castle in an immersive "3-Screen VR Theater."
Additionally, within the Shiroyama Park, an "AR App" allows visitors to view the castle's gates and halls on their smartphone screens, enjoying both past and present views. "We photographed and scanned the extensive castle ruins surrounded by stone walls with a team of nine people. We conducted 3D scanning at 70 to 80 locations using the BLK360," says Tsuji. The use of 3D scan data enabled efficient and accurate modeling of the vast stone walls, composed of uniquely shaped and sized stones. The castle structures located on top of the stone walls were modeled based on reference materials, creating the images for the museum's "3-Screen VR Theater," "VR Telescope," and "AR App."
Data Processing Workflow
Here's an overview of the data processing workflow that combines 3D scanning and photogrammetry.
The set of photographs is analyzed, synthesized, and meshed using the photogrammetry software RealityCapture. The data obtained from the BLK360 is processed using point cloud synthesis software like Leica Cyclone REGISTER 360 and Autodesk ReCap Pro. These tools organize, stitch, and synthesize multiple point cloud datasets.
The synthesized point cloud data and the photo data are imported into Reality Capture.
Here, the data is segmented into parts according to each building and meshed. The meshed data is imported into the 3DCG software Autodesk Maya. CG designers remodel and reshape the data into a lighter form, optimizing it for further use. The reshaped model is retextured in RealityCapture. Final touches to textures and material qualities are applied using the 3D painting tool Substance Painter. The completed digital set data is output as an asset for physical-based rendering (PBR) and is then ready for use in virtual production.
Digital Set Workflow, Utilization Examples, and Overall Concept of Virtual Production
One method of utilizing digital sets is the Tsukun Research Institute's real-time compositing system, "LiveZ studio."
"LiveZ studio" links the movements of a real camera with those of a virtual camera in a virtual space, allowing real-time compositing and display of actors and digital set images on the same screen. Actors can check their movements against the composited digital set on a monitor screen, and staff can plan and proceed smoothly with the shooting by confirming the composited CG background on the spot.
"LiveZ studio" was exhibited at the "Castle EXPO" held at Pacifico Yokohama in December 2019, as a corner where visitors could experience simulated filming in an Edo-period street scene. When visitors acted in front of a green screen, the composited video of them wielding a sword amid falling cherry blossoms in an Edo street scene was displayed on the monitor. This service, which allowed visitors to take home the composited video and post it on their SNS, was very well received.
Future Prospects for High-Quality Imaging
Photogrammetry has the advantage of being able to reproduce colors, textures, and atmospheres, but it also has weaknesses such as distortion in shapes and sizes when multiple photos are stitched together. On the other hand, 3D scanning can quickly obtain accurate shape data and has a sense of speed, but it struggles with reproducing colors, textures, and atmospheres. By combining both methods, the Tsukun Research Institute is creating a workflow that produces movie-level digital sets in a short time and utilizes them in various ways.
LiveZ studio" links the movements of the actual camera with the movements of the camera that exists in the virtual space.
Actors can check their movements while viewing the synthesized image on a monitor screen.
Movie of digital set and "LiveZ studio" by Toei Zukun Laboratory
The potential to meet the increasingly advanced demands for video quality.
Photogrammetry has its weaknesses such as distortions in shape and size when stitching together multiple photos, but it also has strengths in reproducing colors, textures, and atmosphere.
On the other hand, 3D scanning can immediately acquire accurate shape data with speed, but it struggles with reproducing colors, textures, and atmosphere. At Tsuukun Institute, we are developing workflows that combine both methods to efficiently create and utilize movie-quality digital sets in a short timeframe.
"3D scanning data has minimal errors, typically within a few millimeters. It is overwhelmingly accurate, so we use it as the foundational data and platform for adding CG processing," Mr. Tsuji stated.
Mr. Ichida also points out, "3D scanning is effective for quality control in producing high-quality data regardless of the creator's skill level. Even beginner-level CG creators can create accurate CG models by modeling according to the scanned data."
As television broadcasting evolves to 4K/8K resolution, the demand for higher quality levels in visual content will continue to escalate. There is also increasing demand for 3D visualization using VR/AR, such as archiving world heritage sites and precious cultural assets, and creating new entertainment that delights tourists from overseas.
Digital sets offer rich potential to meet these needs, as they can be reused indefinitely, do not degrade over time, provide a wide range of artistic applications and expressive choices, and can integrate with other elements and new technologies. They are well-positioned to cater to these demands with their promising capabilities for the future.
"However, to effectively utilize virtual production, it is necessary to allocate time differently than traditional methods. It is important to recognize that pre-production tasks may take longer than before," Mr. Kobayashi stated. The approach to direction, actors, art, props, costumes, and more will change, affecting the allocation of time, costs, and the overall workflow.
"When considering the long-term benefits of utilizing advanced digital set technologies and how to promote them, it's crucial to consider how we can transform industry norms. This effort is also a significant part of our work," added Mr. Kobayashi emphatically.
※The product names and service names mentioned in this article (including diagrams) are trademarks or registered trademarks of their respective companies.
