Mixed Reality, Design, and the Future of Perception

More Topics


Q&A with Freeman Design Leadership Council member Sergei Gepshtein

A vision scientist at the Salk Institute for Biological Studies, Sergei Gepshtein studies visual experiences in the natural world and in built environments — such as in an architectural ensemble, a city, or a trade show.

As a member of the Freeman Design Leadership Council, Sergei brings his unique point of view to our intersectional thinking approach. We recently caught up with him to chat about why and how his visual research and the future of brand experience are intertwined. Here’s what he had to say. 

Q: As a vision scientist or sensory scientist, what are your thoughts on perception?

SG: There are many ways of thinking about perception. One approach, which is particularly close to my heart, is to think of perception as a creative process. Let me explain this concept.

Your sensory organs gather information about the world in a raw format. In vision, for example, the light reflected from objects forms two slightly different images in your eyes. On the back of each eye, there is a sensory surface called the retina. The image projected on the retina is captured by a great number of tiny receptors. Each receptor is like a pixel on your display, but here every “pixel” is used to pick a small bit of information and send it down to your brain. From a myriad of such tiny bits of information on your retina, your brain constructs your experience of the visual world: the dynamic objects at different distances, full of color and texture.

Similarly, mechanical vibrations in the air engage your hearing system, through receptors sitting deep inside your ears, from which the brain derives the acoustic world, etc. All this information is then combined in the brain to create what you experience as a meaningful multi-sensory world. 

You have the illusion that this multi-sensory world stretches right in front of you, so you have direct access to it. Open your eyes and you see it, stretch your hand and you feel it. But in fact, these experiences are created by your brain from the raw energies that activate your sensory organs. You can think of any given perceptual experience as a simulation that runs on the hardware of your brain to give you the best account of what happens “outside” your brain, based on the raw sensory energies.  

One way to think of design... is to ask how the designed environment, including all its objects — the walls, the colors, the fabric, the furniture — tells a story.

Sergei Gepshtein

Q: Why do you think visual science relates so well to brand experiences?

SG: Anytime you design a large-scale environment, you are faced with a myriad of possibilities for creating unique experience: within the moment and also over time, as you move through the environment. To me, your day-to-day work offers a fantastic playground for experimentation on the scale that is inaccessible in the scientific laboratory.

For example, I currently study how one experience follows another as you walk through an environment. That is, I study sequences of experience. I am trying to predict such sequences from what we know about perception. The ability to predict is important because, when you design the environment, you cannot try every possibility. You cannot do your work by trial and error. Your customers traverse the environment in many directions; they look where they want, and they change their gaze and attention any way they like. Taking the predictive approach allows me to build comprehensive maps of potential experiences. Such maps can help you organize the environments so that your audience will have the desired experiences by following particular trajectories in the environment.

One way to think of design in this case is to ask how the designed environment, including all its objects — the walls, the colors, the fabric, the furniture — tells a story. The predictive power helps ensure that the story works in every case: for any direction your customer walks or turns her gaze. 

Q: Can you give us an example of how your research may impact events in the future?

SG: Absolutely! In this interview published in a Scientific American blog and in Pacific Standard, I described a scenario that could naturally translate to the sort of spaces you are working in. Imagine a large digital display board that presents information that has different meanings to people looking at it from different distances. The meaning accessible to you depends on your distance from the board and also on how quickly you are moving. From the long distance, for example, you could read the time and location for the general session. And up close, you could learn detailed information about the speaker, their background, and what happens after the general session. Paired with additional data and technology, those up-close messages might be personalized to the audience type or even to the individual, so every person receives messages customized for her alone.

One of my goals is to develop algorithms for such customization so that, one day, these methods would become commonplace — something that a designer can just plug into his software and run with. 

Q: That’s really fascinating — and could have a huge influence on our industry. But let’s backtrack a bit. Tell us about what you had recently described as the three revolutions in visual representation.

SG: Well, the first revolution occurred with the invention of linear perspective at the beginning of the European Renaissance in the 15th century. This invention allowed artists to represent three-dimensional space on a two-dimensional surface in a way that makes the space look vividly “real.”  When you glance at the scene painted in perspective, you have the experience of looking through a window opening onto a “real” three-dimensional scene. (Such paintings are often called “Alberti’s windows” after the Italian painter and architect Leon Alberti, who was one of the co-inventors of linear perspective.)

The second revolution occurred in the early 20th century at the invention of the moving picture. When we added motion to a picture, space became alive. It became narrative. But when we go to the cinema, we are still looking through Alberti’s “window.” There is a wall separating the audience and the represented world.

Now we are in the midst of the third revolution. The wall, that boundary between the audience and the represented world, is being dissolved by new technologies. Let me explain. If you walk in front of a painting, you may have a sense of space stretching out behind the surface of the painting. But the painting doesn’t respond to you. For example, moving left or right will not let you see what happens behind the trunk of a painted tree. Now, imagine taking the painting from the wall, making it smaller, and strapping it to your head. And instead of pigments painted onto a canvas, imagine a digital display that represents three-dimensional worlds that respond to your movements. What we’ve just described is virtual reality. Now you can walk around that tree and choose which side of the world you want to see. 

Q: So what makes this third revolution so significant, particularly to brand experiences?

SG: For one thing, we are shifting from a person passively following a story through an Alberti’s window to a person who can actively explore the story space. For example, painters share the way they see the world: they give an opportunity to see the world their way. Filmmakers know how to control where you are looking and what you are seeing, so you experience the story their way. That is all traditional storytelling.

But when you are surrounded by a story space, you can look anywhere and walk anywhere — you become a participant in creating the story. The transformation brings about many exciting possibilities, but also there are some challenges. For example, when we translate this approach to brand experiences, we need to be careful not to become too obtrusive. We will have to learn to balance various ways of attracting and engaging the audience.

Q: How are people’s expectations of a personalized experience affecting your area of expertise?

SG: Let us recall, first, that the experience doesn’t need to be completely virtual. A head-mounted display could be a semitransparent lens. Or you could use augmented reality so the individual will still have a sense of grounding in the physical reality. Then the person could be among a group, which will have the experience together, such as watching a short film on the wall. It could be the same film for everyone, or it could vary depending on who is looking at it. At a trade show, for example, the film could be customized based on the audience type or persona.

Taking it a step further, when paired with tracking technologies I mentioned earlier, the display that appears on the wall could grow in its degree of sophistication, using individually customized messages and other fine-tuned visuals, effectively transforming the space into something intimately personal. Such interactive, responsive augmented reality doesn’t have to involve head-mounted displays. The images can be beamed right into your eyes by the tiny projectors embedded in the environment.

Such technologies are just arriving. But their potential for brand experiences is breathtaking, and it can come become reality within only a few years.


What you need to know to stay ahead of the ever-changing experiential marketing curve.

Order About Solutions Insights Work Contact Careers