Interview with Omer Shapira

Omer Shapira is an Artist and a Programmer, exploring trust between humans and machines with Virtual Reality and Robotics. Before working at NVIDIA, he was a Technical Director at Fake Love and Framestore VR. Omer talks about technology like you talk about your ex.

P!: You grew up in Israel where you majored in math and worked as a film editor. Did your experience play a role in shaping your early ideas and approach in design?

O: I was 15 when I started working as an editor, and was still using analog tools (“A/B Roll”), which are all pretty much ungoogleable now. Those were just a few tape machines and a keyboard, and the practicum was selecting a piece of footage and watching it get copied from one tape to the other in real-time, erasing whatever was on the other tape. This is destructive media, like clay. Not being able to undo makes you think of the object before you as a final form. But unlike clay, electronic media contains hidden state you have to keep in your head. One of the earliest things that all analog editors learned was to understand the cost of making a mistake, and to plan as far ahead as possible.

Moving to digital meant we had an ability to undo, and an ability to look at the timeline, so that hidden state became less hidden. Naturally, seeing what you’re doing means you can work faster. Oddly enough, I found the opposite to be true: editors who started out using non-linear digital tools were slower, because they never had to learn to plan a few moves head. A few years later, preview tools got a little better, and the practice changed: making complex sequences is now a single editor’s job, not an entire crew. That’s all due to the evolution of the tool. By the way, from the early 2000s it was apparent the editing style became more idiosyncratic. For a while, it didn’t feel like an intentional aesthetic… I think it is related to that editors starting at the time didn’t have the same inhibitions, so they could explore, but they also made a lot of mistakes that were finalized. I still think to this day that editing is the only thing I was ever fully comfortable with, and it shows how important the tools are.

P!: It sounds strangely similar to how physical models are in architecture.

O: I think so. In parametric software, there is often a problem where many of the design choices are made by negligence as a result of changing variables without any visual continuum between their values. Often there aren’t visual ways to find a ‘sweet spot’ in a collection of sliders in Grasshopper. This problem is not unique to design – There’s a fundamental problem in Numerical Optimization fields, like Deep Learning: when searching for local maxima in a high-dimensional function, a common task is figuring out which parameters to change at a given point in parameter space get a better result. That’s like being somewhere on a mountain, and not knowing where the peak is.

Commonly what you’d do is some guesswork: you know what looks “slightly” better in your next step, but maybe that leads to a dead end in the overall solution? Most of the the time it’s guesswork – you just have the computer walk around the mountain, because the mountain is hard to visualize. In both cases, the problem is not compute power, but interaction design: how do you visualize the problem so there’s no need to tweak parameters and get serendipitous results.

P!: We use certain definitions of identity to find a “sweet spot” in architecture. Is there an identity that you try to create for your users in VR?

O: We’re not there yet. For example, take the problem of usability as a construct to explore sweet spot in. Any VR interaction device ranges on the continuum between ‘toy’, ‘tool’ and ‘instrument’. A toy is self-explanatory, seldom useful, and typically doesn’t store states or innate qualities. It’s meant to be predictable for beginners to use. Instruments are the other end: they require skill, and can sometimes hide function and state – but at the right hands, can create amazing things. Think about the closest thing you have to an instrument in VR – I can think of Tilt Brush, but it’s neither as simple as a toy nor as precise as an instrument – so we’re stuck in the middle, around “what can an average person do to be somewhat expressive and not frustrated”. That’s a far cry from what I would like to see as identity.

That’s a stark contrast to what we can already do well in VR, which is spatial design. In VR, you apply the same basic set of tools as you do in architecture to make the space coherent for humans to use, except you’re designing a ride, not just a building, which is an important distinction.

One thing that I see a lot is that architects and interior designers working in VR will not make tiny rooms; they will make rooms with long vanishing points – because hey, real buildings have corridors! But people without that experience will make novice mistakes like constructing open spaces with trees all around at a radius from the user, and no clear sense of perspective. Obviously, that’s pointless. We all want our eyes to be guided when reading a space. In that sense there’s nothing new about about designing for VR – every nuance in spatial practices still applies. As far as interaction design goes, we have a very very long way to go to be in the same place.

P!: You studied neuroscience and senses in order to gain access into VR. I wonder what you think the relationship between neuro-realism, hyper reality and architecture is?

O: I didn’t go to school for neuroscience – I just have some PhD friends with patience for my questions! But I really learn a lot from the basic textbooks in perception. There are lot of hacks that our perceptual system uses to interpret reality coherently, and we haven’t built them into our creation tools yet. This is the first emergence of VR where we have accessible tools, but neither of them [Unreal Engine and Unity] none of them were built for this use. Those tools are primarily designed for consoles, phones – basically 2D screens, not for displays mapping around your body. Screens have years of perception research embedded in their design. For example, ‘MIP Mapping’, that effect of blurring and scaling down images that should appear further away to make the approach to them to appear continuous – that’s used everywhere, from Google Maps to 3D Shooters – that’s a result of understanding photons should reach our eyes, and how to represent reality to us in 2D pixels. It’s been around since the early 90s, so we’re about 25 years away from this feeling entirely new. That research for immersive media is new and isn’t as extensive. We all know our bodies really well and you don’t really need to be a neuroscientist to understand we’re lacking in tools.

Once we have those frameworks in place, as an artist you’ll have to think about those less – the tools will be designed to do the right thing.

For example, we still have a problem of mapping a person’s physical space to their virtual space, so they neither hit walls, nor need to turn around. One way of solving it is diverting the user’s walking path slightly at the right time, so they experience is continuous. This is called “Redirected Walking”. This isn’t yet well-understood, nor a part of every designer’s toolkit. Another problem is the that even though the hardware for computer vision is getting faster and smaller, there’s no intuitive design approach yet to map real objects to virtual objects. As a design practicum, virtual and real spaces do not coexist peacefully yet – but the tools are getting there, with Apple’s ARKit and Google’s Core AR.

Nowadays, lack of frameworks means a lot of work falls on VR creators because you’re both the artist and the tool maker. This is a huge strain on the artform – like, how many animators do you know that write their own software? This is the trap I’m in right now because at Nvidia I sometimes focus on making tools, which means that I can’t always work on the most beautiful things. When I try making beautiful things, I realize I need new tools. I hope for a day when we’ll be able to abstract most of it away and leave you doing VR the same way you’d be designing immersive theatre; not thinking about the machine, but thinking about the human.

P!: I wonder if it is because the tools are already set in architecture, so we only need to focus on how we think instead of how we feel. Does that present a loss in connection between the human bodies, mind and how architecture operates in the physical world?

O: Let me take a step back: I think architectural tools are a little behind. For instance, when you’re designing a bathroom, you have to reach for a hidden tool – or think very hard – before placing the door so it doesn’t hit the toilet. That’s fairly basic – now imagine having to figure out what congestion looks like near an elevator bank at lunchtime? Predicting edge cases should be the default behavior.

As a toolmaker, I’m really inspired by David Rutten, the author of Grasshopper. He was needed a form of expressing himself in ways that other tools were – at best – awkward at. There’s a very tangible contribution that Grasshopper has made to the world of design – it changed the verbs we use.

I feel like architecture needs a lot of new verbs. Models are often shown by default from above – a view which no one will see. Clients expect to see shiny buildings – and often get what they want – so architects are never forced to think about their model 20 years in the future, when it’s covered in gum and grime. Simulation technology can solve that. In industrial design, people work at a scale that they understand even if the product’s damaged. The scale allows the designer to check what happens if the product is thrown in dirt or be covered with grease. You can make it imperfect and test it against the real world. In architecture, that type of thinking is mostly speculative.

Large-scale experiments in architecture have gone massively wrong thanks to lack of foresight. Joshua Walton (formerly LAB at Rockwell, Microsoft) once told me that he loves working with architects because no matter where you are in the world, architects are trained to collaborate on a professional level. I guess it’s because when you graduate and start working in a practice, you design door knobs and stairwells for a while, and learn to integrate.

The video games industry suffers from the same problem of large amounts of repetitive work – and solves it in a very different way: A large games studio will have internal teams writing artists tools for everything. The better the tools are, the larger the problems you can tackle. If you read the credits for WATCH_DOGS, there are thousands of people signed on the final product – but they didn’t create a single skyscraper, they created all of Chicago, with cars, and planes, and physics, and humans walking around talking to each other – and they made this entire process easily repeatable, so WATCH_DOGS 2 could do the same thing, but with all of the Bay Area. The ideas of skill integration and quality assurance seem very different in architecture than in games. In architecture there’s a perception that there’s not enough money to build a good tool chain, which I think is false. The evolution of tools for planning VR experiences may benefit architecture. We’re solving the same problems, we need to understand some coarse human metrics.

l don’t want to be a techno-optimist here, but I think there’s a potential to automate some of the shortcomings in architectureal design nowadays. Similar to a skyscraper design, the cost for large games are in the hundreds of millions, but unlike skyscrapers they can scale infinitely. For example, In Gensler’ design for Nvidia’s HQ Building, programmers built a tool to simulate the path of the sun inside the building, saw it was generating too much heat – and ended up adjusting the angle of the skylights early in the design process. If small firms had that as a tool, imagine how their ability to dare make unique designs increases.

P!: There is always the debate on the architect being an artist, a designer, or a collaborator. How does that interface work for you as an artist and program developer?

O: I don’t feel that I have done as much as I should’ve in making tools yet. The major leaps in simplicity and expressiveness are not there yet. If you think about the bigger picture, it’s not just VR: There is continuum between AR on a mobile phone and full Virtual Reality – it’s our spatial perception and proprioception, and we need to further explore the continuum as designers.

It’s important to agree on these paradigms early. Our environment is becoming automated. We already have lacking paradigms in home automation. When level 5 self-driving cars are here, they’ll need to interact with humans reliably and efficiently to keep them safe. Same goes for delivery drones and assistive robots, which will be all around us, so the elderly can have a dignified life and we can have a convenient one.

To work on this platform, which is really just an extension of the human body, we need to understand human instincts, fears and wants.

If I am designing for a human, does it matter if it’s in reality or VR? Robots are already modifying our reality, and VR is just an analog of that. I try to look at human robot interaction and social robotics. [pulls out “Robot Sex” and “Affective Computing”]. We’ll soon have robots mimicking human functions, because we’re all alone at some point of our day. We need to augment our reality so we can be physically close regardless of distance. These are my main points of inspiration. A lot of my mental framework are still from my days of editing: The major aspect being I can control someone’s emotional progression through experience within a short amount of time.

P!: Lastly, do you have any reactions to the Design Objectives?

O: The list is not wrong, but the practicum is naïve. Things scale at a pace that humans cannot predict without automation. We cannot simply say “This building did not take off because too many people were using it.” We need to get away from designing spaces to be static. I feel like architects love designing pavilions because they can be creative while still maintaining absolute control and understanding of every human scenario happening in it. We are not at the age where you can make Villa Rotonda supported by a patron, for a handful members of elite to see. We are making things used by hundreds of thousands of people a year, and have to consider congestion, erosion, and disorder. If we care about people, we really need to embrace chaos in planning.