High definition 3D graphics at your fingertips – Tamy Boubekeur, Télécom Paris

It’s an old dream in the process of becoming reality: the first “3D cameras”, allowing you to adjust the angle long after taking the shot, are finally emerging from laboratories. Expectations are huge from professionals in cinema, computer-aided design and computer games. But the technology is not yet fully mature: the data obtained is so large that it is impossible to exploit directly in existing applications. Research teams from IMT, including the Télécom Paris-based Computer Graphics group led by Tamy Boubekeur, are working to tackle this issue. The researchers are developing innovative tools, capable of processing raw data to obtain new types of media.

High-Definition 3D Capture

The creation of images using 3D models has developed significantly since the 1980s. The initial objective in this era was the achievement of photo-realism, to make synthetic images indistinguishable from photographs. “Nowadays it is possible… if one has the time and resources available for processing!” explains Tamy Boubekeur. Until only recently, the creation of the underlying 3D models was very tedious. A single car could require months of work, with hordes of computer graphics designers having to meticulously design all the digital models, piece by piece, in addition to their movement.

3D capture changes everything. It has been acclaimed by archaeologists and urban planners, for example, since the early 2000s, who can “photograph” a 3D model of a fragile object or a complex outdoor scene such as a road traffic accident, in an instant, explains Tamy Boubekeur. “They can then study, modify and examine it at their leisure, whilst the original is protected in a museum, or the road around a structure doesn’t have to be closed.”

But digitally capturing an immobile statue is very different from tracking a tennis player in action, and computer graphics researchers have more recently begun tackling the issue of “dynamic” capture, “There was a great leap forward – capture increased from 40 points using motion capture markers for localisation, to several million points using current performance capture methods, which take advantage of the countless recent advancements in 3D reconstruction and vision, using no contact and no markers. Subtle movements are detected, even down to the creases in the fabric of a T-shirt!” And there is now no need to spend a whole morning in a dedicated room for each recording, sessions can be more improvised. Everything has become more precise and easier, and there is now an extensive field of new, notably mobile, applications emerging.

This progress is not down to the devices used, (“a single webcam is sufficient“) but from progress made in 3D algorithms in recent years. Whilst public attention is often focused on advances in the equipment, it is primarily improvements in digital movement and patterns of models which make these new feats possible. “Incidentally, there is a lot of mathematics involved!” comments Tamy Boubekeur. The themes of computer graphics and computer vision are coming together to gradually give rise to a calculative science for “visual” applications based on digital geometric data: visual geometry.

An avalanche of data points

Yet the extreme power of these new capture technologies has its downside, which is in fact fairly extensive! When a computer graphics designer comes up with a digital model, the definition may be limited but the result is highly structured and can be easily controlled. Tamy Boubekeur draws attention to the fact that “data produced by capture is extremely dense. It doesn’t have any explicit structure!” It thus takes up an enormous amount of memory, and above all makes it impossible to modify or re-use in actual software. “For animations, for example, it is only possible to replay the same film.” And yet industrial applications must have the capability to, for example, make a computer game character reflect the movements of a real actor, or simplify a Paris street scan for exploration via a mobile platform.

Tamy Boubekeur has launched a number of projects with his team, with the aim of making thus ream of data easier to manipulate. One of these projects addresses animations and has led to the creation of CageR, a system capable of confining a digitalised 3D animation in a “cage” in order for it to be manipulated. “We can see cages as the development of skeletons which could enable us to manipulate old 3D models, while also allowing for natural control of volume.” The volume of data is also reduced to a fraction of its original abundance. The data captured can thus finally be modified to achieve the desired goal, to create a cinematic special effect, for example.

At such a level of detail available, it becomes often difficult for a human operator to use these digital models efficiently. For this reason, Tamy Boubekeur’s team are also working on intelligent modeling tools: “Some tools were in great demand, but no-one had yet succeeded in producing them. For example, we created, in a collaboration with New York University, Toronto University and the University of Tokyo, a 3D equivalent of the “copy-paste” brush of digital imaging software: it picks up the relief of an object and applies it to the surface of another.” Tamy Boubekeur demonstrates on a digital model of a lizard with smooth skin: preparing a sample of the cracked surface of a ball on his screen, he applies it instantly to the lizard model.

Moving towards instant 3D

In light of this progress, 3D digital models are moving from a status as rare and precious resources to a widespread commodity whose abundance will need to be managed – the Big Data trend is not far. For Tamy Boubekeur, the Internet will soon offer an extensive palette of 3D models which designers can dip into at their leisure. The final challenge will be to navigate amid this abundance of data and options. “3D capture will become something that is easy to use and is here to stay, much like digital photography today. It will be impossible to manually label all 3D models, to say that one is a spade, the other is a tree etc.” For this reason, he is working, in collaboration with TU Berlin,  on a pattern recognition tool: just give him a freehand sketch and he can find corresponding digital models in his large database! “We can already furnish and fill rooms in 3D in just a few minutes.”

If fully exploited, these “visual geometry” algorithms have amazing potential: “In the future, we can envisage computer games taking place in cities like Paris, the true Paris captured in real time by satellite, at the exact time of playing the game!” For Tamy Boubekeur, this new capacity doesn’t make the work of designers less valuable; on the contrary: “3D digitalisation and computer-aided design are complementary, he explains, we have to take the best of them both, by making 3D design as natural as possible and leaving the human operator to focus on what she does best: being creative! The computer graphics designer becomes an artist with digitalised geometry as his palette and the computer as his paintbrush.”

Tamy Boubekeur leads a 3D computer graphics team at the LTCI (Laboratoire Traitement et Communication de l’Information – Laboratory of Information Communication and Processing), a CNRS and Télécom Paris joint research unit. This doesn’t mean that researchers model 3D objects, on the contrary! As Tamy Boubekeur puts it himself: “I quickly became aware that I found it more interesting to create tools which serve to model than to do the modeling myself!” After a thesis on 3D rendering and modeling at  the INRIA (National Institute for Research in Computer Science and Control) South West Bordeaux centre in 2007, he obtained accreditation to supervise research (HDR – Habilitation à diriger les recherches) in 2012 at the University of Paris-Sud. After working as a research associate at the Technische Universität Berlin, he headed up the Computer Graphics group at Télécom Paris in 2008. “To be recruited to an organisation where my topics were not represented was a great opportunity for me. Nowadays, the school is a breeding ground for students capable of combining Computer Science, Maths and Physics, which fits perfectly with my research.”  In just five years the group has built a strong reputation in its field, establishing partnerships with major companies including Dassault Systèmes, Disney Research, and Orange Labs.