Digital modeling is one of the most widely used tools for bringing bodies to life in 3D. Created from thousands of everyday images and videos, 3D generative models employ artificial intelligence to help us understand the structure of animals and humans. These models are essential for a wide range of real-world applications, including biological research and surgical planning. Existing generative models, however, have limitations as they rely on training data that consists of fixed, typical skeletal structures—and nature is anything but typical.
Researchers at The University of Texas at Austin recently received support from the National Science Foundation (NSF) to develop a new class of digital shape models that are capable of accurately representing animal and human forms with skeletal structures that deviate from the norm. “This project enables adaptive, learnable models that can accommodate diverse and atypical anatomies,” says Georgios Pavlakos, assistant professor in the Department of Computer Science, and the project's principal investigator. “By supporting flexible modeling across a broad spectrum of species and conditions, this work has the potential to advance research across multiple disciplines, including advancements in biology, medicine, and paleontology.”
Working with Volkan Isler, professor in UT Computer Science and a core researcher in Texas Robotics, and Qixing Huang, an associate professor in Computer Science, the team aims to develop a new and versatile methodology for 3D/4D generative models that can accurately capture the structure of deformable objects, such as animal bodies and human hands, from standard images and videos. The proposed model will be skeleton-driven, but, unlike current models, the skeleton and the embedded mesh topology will not be limited to a single, fixed version. When the skeleton changes, the embedded mesh topology will also change. The model will also be able to synthesize shapes with new skeletons that are not present in the training data.
Collaborating with Experts in Medicine and Geoscience
3D modeling plays a crucial role in surgical planning. For example, in the surgical management of congenital hand differences, these models enable surgeons to meticulously plan and practice the complex procedures needed to remove extra digits and reconstruct the remaining anatomy for optimal function and appearance. Today, the estimated rate of newborns with congenital hand differences—such as more than five fingers present on one hand—is between 0.4%–0.8%. For these children, effective surgeries after birth are critical, yet current tools are unable to model hands with more than five fingers (as seen in the image below.) The challenge is that it is difficult to use 3D scanners to reconstruct the hands of newborns because they are moving. Video-based reconstructions guided by hand templates are viable options, but current template models depend on a fixed (five-fingered) skeletal structure.
In collaboration with Dr. Fang Yu, a surgeon from Xiangya Hospital of Central South University in Changsha, China, the research team will evaluate the applicability of their approach in reconstructing the hands of newborns and whether it is accurate enough for medical applications. Doctors may one day use the new model for surgical preplanning and customized treatment to reduce risks and improve surgical precision.
In collaboration with Christopher Bell, a geoscientist at UT’s Jackson School of Geosciences, the team will apply the new approach to both discover and verify scientific insights and correlations among species. Modeling animal structures—including extinct species—has significant value for biology, paleontology, and education. The research will enable automated analysis of animal skeletal variations across a wide range of species, supporting research on the form and function of an animal—why and how it looks the way it does—and its connections to other species, past and present. With adaptable 3D modeling, this research can help document species diversity, assist in anatomical studies, and expand access to digitized biological collections. In addition, the researchers aim to help automate the reconstruction of fossilized species to support paleobiologists in understanding skeletal variation across evolutionary timelines.
By bridging the gap between current 3D models and the varied anatomies of the natural world, this research can change how we visualize and interact with biological forms. Whether helping a physician navigate complex and life-changing surgeries, or assisting a paleontologist in reconstructing the movements of an extinct species, these adaptive 3D models offer a new and important level of precision and efficiency. Ultimately, this work will ensure that as our digital tools evolve, they no longer require life to fit a standard mold, but instead adapt to represent the true, complex variety of every living form.