Make It So: Interaction Design Lessons from Science Fiction with Nathan Shedroff & Chris Noessel

Sean Carmichael

October 24th, 2012

Play

[ Transcript Available ]

Nathan Shedroff Chris Noessel

Science fiction films often take liberties with the technology that they display. After all, it is fiction. Though they can make up essentially whatever they want, technologies still need to be somewhat realistic to the audience. This influences the way that sci-fi technology is presented in film, but in turn, it’s how sci-fi influences technological advances in the real world.

Nathan Shedroff, Chair of the MBA in Design Strategy Program at California College of the Arts, and Chris Noessel, Managing Director at Cooper, took it upon themselves to study the lessons that can be learned from science fiction. They analyzed a variety of interfaces from all different time periods of film and television. They discovered that when new technologies are developed and released to the market, people already have expectations of how it should work. This is based upon having already seen a similar, fictional technology.

Of course, there are instances where the technology in film is all but an impossibility, or at least impractical in real life. This changes as gestural and voice recognition technologies become more advanced, but a lot of interfaces in sci-fi are developed simply for the “cool” factor. Even then, looking to these interfaces as a reference point can help focus a design.

Nathan and Chris join Jared Spool to discuss their Rosenfeld Media book, Make It So: Interaction Design Lessons from Science Fiction in this podcast.

Make it So Cover

As always, we love to hear what you’re thinking. Share your thoughts in our comments section.

Recorded: October, 2012
[ Subscribe to our podcast via Use iTunes to subscribe to UIE's RSS feed. ←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]

Full Transcript.


Jared Spool: Hello, everyone. I’m here today with Nathan Shedroff and Chris Noessel. We are talking today about their brand new book called “Make It So: Interface Design Lessons from Sci-Fi,” which is published by the amazing folks over at Rosenfeld Media.

It’s a fabulous book. Personally I think it’s one of the best ones that Rosenfeld has come out with, because it’s just chock full of all this really cool stuff about science fiction and how design interacts with science fiction. I’m really happy we have them today.

Hey, Nathan. Hey, Chris.

Chris Noessel: Howdy, howdy.

Nathan Shedroff: Hi.

Jared: Cool. Here’s the thing. In my life at least, I’ve got my work, and then I’ve got my sci-fi fetish. I’ve always sort of kept them separate. I think a lot of people do that. It’s cool and I read all these books and I watch all these movies and I see all of this stuff, but thinking about how does that translate back into my work is hard.

What is it that these film and TV special effect artists are doing that really makes a difference to us, today? I don’t know. Nathan, why don’t you start with that?

Nathan: Well, one of the things that they’re doing is that they’re portraying visions of technology and interfaces and application to audiences which just so happen to be users as well. So they’re setting expectations about what to expect in terms of use, what’s right or wrong, what works and doesn’t, regardless of the technology that gets implied by some science fiction.

We see examples of how the world should be, and we react to that.

Jared: So is it the case that the sci-fi stuff is actually driving designs that we’re seeing, or is it more of dealing with just the background? It sits in the back of my head, and I’m thinking about it all the time?

Nathan: I wouldn’t use the word “drive,” but I would definitely use the word “influence.”

Jared: Can you give me an example?

Nathan: Yeah, the Star Trek communicator from the original series, the sort of flip out communicator was something that was seen starting in 1966. We’ve had conversations with people that are anecdotal evidence that that solution was discussed by the product development team at Motorola that worked on the StarTAC, get it? StarTAC mobile phone that was the first flip phone.

Not only was it an influence on audiences about what to expect in terms of this is what mobile communications will be like in the future even though the use of the communicator was more sort of walkie talkie like than phone like, but it was an influence on the engineers and developers who created it. In fact the phone that they created, the version that they created before that flipped the wrong way and when they showed it to people, people would say, “This is great but you’ve got it wrong.”

They would say, “Well, what do you mean?”

And they would say, “Well, it flips the wrong way.”

When the engineers said, “How do you know what the right way is?” people would say, “Well, like on Star Trek.”

Jared: That’s really interesting, right? Because it became this frame of reference. But the guys at Star Trek, they weren’t thinking, “We’re out to design a phone for everybody to use.” They were just making a prop.

Nathan: Of course. But they were still trying to solve a problem to whatever extent they could tell. They were still trying to make it realistic. They were quickly assuming technology constraints and human body constraints and whatnot and coming up with something that they thought was cool and realistic at the same time.

Chris: This is Chris. I do think that it’s worth noting there are certain films and television shows in the world that really try and put a stake in the ground as far as what will the future be like.

Three that just come to mind are “Destination Moon” from the 1950s, which was an older sci-fi film, about space travel before we had done it. “2001 A Space Odyssey”–we know that Kubrick employed a lot of consultants in order to get this picture right. The same thing with Spielberg when he made “Minority Report.” There’s lots of interfaces in that film beyond the gestural one, and they put a lot of work in there to say what’s realistic and what’s cool.

Jared: So, Chris, you’re working at Cooper these days, and running how that place is working with clients. How is all this sci-fi stuff working its way into the designs that you’re doing with your clients? Do you see an influence there or is it separate?

Chris: Well, I certainly see that there are two kinds of influence. One is actually just as a means of communication. For a client we had recently–and I don’t think we have permission yet to speak publicly about their name, but it was a voice interface for a piece of medical technology.

A lot of times we reference sci-fi just as a way to talk about it. Oh, well, this would kind of be like the talking computer in “Star Trek.”

And it was just a way that we could talk to each other about the features and functions of what it was. You know, is this more like 1960s “Star Trek,” or are we talking more like “Iron Man’s” voice interface inside of his suit?

But a second way that it’s actually influenced is my sense of where technology is and where it’s headed. Having been immersed in science fiction and thinking deeply about these things, I tend to look a little bit forward, I think, than our clients do about what it is that things could and should do.

One simple example is the notion of agency. Can you guys remember the name of the butler in “Iron Man?” The name of the interface inside of his suit?

Nathan: Oh, wait a minute.

Chris: Jarvis, Jarvis.

Jared: Jarvis, that’s right.

Chris: Just having that on hand, this notion of “Hey, technology is pretty near agentive already.” It’s one of the things that I end up talking to my clients sooner rather than later.

Jared: I remember years and years ago talking with Pattie Maes on agency. She was at the MIT Media Lab, and I think she still is. She was doing a lot of work on what agents were about. She talked about this idea of the butler specifically, and this notion that butlers should be present but almost invisible.

You know, it’s like when you’re at a dinner and you’re having this intimate conversation. Suddenly the waiter butts in and starts adding their opinion, and you didn’t invite them into the conversation. It’s a little awkward.

It feels to me like those are types of lessons that sometimes these films get right, but sometimes they break those rules for storytelling effect. Is it possible that you can get the wrong design out of film?

Nathan: Oh, yeah, absolutely.

Chris: Oh, yeah.

Nathan: We see lots of examples in science fiction of things that would not work for us in the real world but work in science fiction because they wow audiences and they seem to be cool. A good example is the gesture interface in “Minority Report” that everyone looks at, especially if they’re in interaction design they reference…

Interaction designers reference this as being really cool but the reality is that they don’t work very well for humans. They have very specialized uses at all and because it requires us to have this sort of upper body movement these gesture interfaces are actually really tiring.

We see this in voice recognition, voice response systems, as well, where all through Star Trek you can talk to the computer via voice and have a conversation. We see it in lots of science fiction, in fact. But the reality is it’s not going to fit our world unless you happen to have a private office because most people who work in open office environments or out on the street, they don’t want everyone to know what they’re doing with their computer and the fact that they’re on Facebook and tweeting as opposed to working on their spreadsheet.

There are definitely portrayals of interfaces that work perfectly fine in science fiction because there’s such a limited portrayals that won’t ever really work, at least not in the way they’re portrayed for us in the real world.

Jared: That’s interesting, right? Because I remember watching “Star Trek: The Next Generation” and they had this communicator that they would use to communicate with people all over the ship or sometimes talk to the computer. They would just tap this emblem on their clothing, this piece of jewelry, and then suddenly they’d be instantly connected to someone else. But somehow or other, it always knew who to connect to almost before they said who they wanted to talk to.

Nathan: They certainly always specified who they were trying to contact but it was so instantaneous that, yeah, it was on the edge of credibility, at least for us at that time. But if you look at how quickly a Google search happens and it comes back even though it’s hitting 15 servers and reconstituting answers and coming back. It’s almost incredible in itself.

Jared: Yeah, but…I mean Picard would touch his shirt. He would say, “Picard to Riker,” and somehow Riker’s jewelry would immediately just say, “Picard to Riker” and nobody else’s jewelry would say that. Somehow it was almost as if the system preemptively knew that that’s the connection that they wanted so that they could establish it.

Something about it always struck me as a little odd. How would you implement that without the seeming delays that these things have. And, of course, no one else in the room can talk at that time. That’s the other…

Chris: The needs of narrative produce some pretty interesting interface moments. One of the things that we noticed fairly early on in working through communication interfaces is that you don’t ever see volume controls.

Jared: Oh yeah!

Chris: You have sound engineers, right? Nobody wants to sit through the very mundane task of going, “Oh, that’s too loud.” There’s one exception which is the dial in “Contact” that Ella uses but that was specifically, again, for narrative effect. To show how she was concentrating.

There are lots of examples like that where it’s just like narrative has a story behind it and a lot of the quotidian stuff that we need in our everyday lives is just too boring for screen.

Jared: I have never heard anyone use the word quotidian in a sentence before.

Chris: [laughs] Word nerd has helped immensely in writing this book.

Jared: Let’s go back to gestures for a second. Is there a way to take apart the gestures that we’ve seen? In the book you mention “Forbidden Planet,” right? That was a movie when? In the ’50s.

Chris: Yeah, 1956, I think.

Jared: Yeah. The gestures that were in the ships then, long before we had Kinect and Nintendo and all these sort of gesture based full body experiences. And then the little swipes that we get on our mobile devices.

What can we learn from what science fiction built in? How do we take it apart and say, “This is actually something that is applicable to us.”

Nathan: This is actually…I just wanted to make a note. Not “Forbidden Planet” but “The Day the Earth Stood Still.

Jared: Damn. “The Day the Earth Stood Still.” OK.

Chris: Both actually had some gestural interfaces. Wow. I’m really totally taxed to try to remember…Oh, Alta. The daughter in “Forbidden Planet” uses gestural interfaces to summon Robbie the Robot.

Jared: Yes! I knew it.

Chris: “The Day the Earth Stood Still” has…Klaatu has gestural interfaces aboard his ship. That one was an earlier film. That was in 1951.

Jared: Got it. OK, so there we go. All the way back to ’51 and “The Day the Earth Stood Still,” which apparently was the day I forgot about that movie. Nonetheless, it seems that we have these gestural interfaces. There’s all this sort of gesturing and swiping and very artistic dance-like moves that immediately get interpreted as control sequences.

How do we take that apart? Walk me through this.

Chris: Well, I think one of things that it ends up doing is similar to the StarTAC phone, which had pre-trained audiences on how to use a flip phone device for 30 years before it hit the market.

Gestural interfaces that people see in film are doing some of the same psychological pre-training for what it means to use a gestural interface to talk to a computer system.

Of course, over the course of our chapter, the gestural interfaces chapter, we have done the analysis to find seven meanings that are emergent in sci-fi film and television.

The fun thing to me is that there isn’t a gestural czar out there who’s been appointed in Hollywood to make sure these things are all the same and all work the same way. So this is really just a lot of production designers thinking about the problem and well, what makes sense?

Even though they don’t do some things, similar to the fact that we never see a volume control mechanically, we don’t ever see a gesture for volume control, either. It’s a lot of the same thing, right? Like it is setting expectations, giving us some early notions, and maybe even setting them wrong, which is part of why we wanted to write the book.

These metaphors that are being put out there need to be examined critically from a designer’s eye, before we let them influence us unduly.

Jared: So you’ve got these different gestures that you found. Does that mean that that becomes a gestural framework that we can apply and then we can use these examples out of the films for that?

Chris: I certainly think we can. If Hollywood keeps using gestural interfaces, and I don’t see them stopping for a while, we’re going to run into troubles if we have a different meaning for the same gestures that people have seen in TV and movies. Only because it will confound their expectations.

Gestural interfaces are kind of like the old DOS interfaces in that they have almost no affordance. There are few biological meanings. I think actually that’s where those seven pidgin gestures that we identified in the book are coming from. Very universal, biological meanings.

Beyond that, Hollywood’s kind of all over the board, and certainly when it comes down to doing daily work, or even just using it for entertainment, we’re going to have some challenges ahead of us to fill out things around those seven gestures.

Nathan: In essence, collectively, but without any real collaboration or cooperation. We’re creating that language now or “we” meaning a bunch of production designers out in Hollywood and New York and London, et cetera, are creating a shared gesture language that we are training to audiences of science fiction, whether we like it or not.

Jared: Well, that’s interesting, because language has always evolved by use, and if you look at, for instance, the number of idioms and quotes that people use that just came out of Shakespeare’s plays, right? There are all these phrase and things that all started in Shakespeare’s plays. Hundreds of them.

Chris: Yeah, “lovely” is one of those that I still am blown away by.

Jared: Yeah.

Chris: He just made that word up. I was like, “Yeah, we kind of needed it.”

Jared: So is this sort of that same thing, where in order to tell the story, they had to create gestural words and phrases that just make sense? Since they were the ones who coined them, that’s what gets ported into real life?

Chris: I think that’s a great metaphor. I hadn’t actually thought about that particularly before, but I don’t see why it’s not true.

I do think that unlike language, if you take any of…let’s take turn to rotate, which I think is the third gesturial pidgin that we had identified. That one has a basis in the world, right? You would grab the opposite edges of a cube that was sitting in front of you in order to turn it. We see that in sci-fi. It just makes sense.

Others of them are arbitrary. Wave to activate? Who says that’s a wave as opposed to a poke or opposed to any of a number of a series of gestures. I think a few of them are just like that. Some arbitrary thing that makes sense that’s going to end up staying with us for a long time.

Jared: I talk with my hands so much. I just could imagine being in somebody’s house in their kitchen and start waving my arms around and all of a sudden the blender and the microwave are engaging.

Chris: There’s that great moment in Minority Report where Daniel Witwer is coming in to meet Tom Cruise’s character for the first time. And Agent Anderton, Tom Cruise’s character, actually turns to shake his hand and almost erases all of his work in the process. It’s actually a great moment in the film that reminds us that these interfaces aren’t going to be perfect.

It’s like we’ve been collecting interface jokes of late. That’s actually one of my favorite ones. That is working something out in the film that we’re going to have to deal with if and when that technology becomes commonplace.

Jared: Yeah. It does seem like it’s ripe for error and things like that. Have you guys looked at what people have been doing with Kinect and other sort gestural heavy interfaces there? Have they pushed beyond what’s in the films in some ways or is it mapping pretty close?

Nathan: I have to say I can’t say that I’m an expert on everything that everyone’s doing in Kinect but, by all means, people have pushed beyond the simple pidgin language that we’ve identified, which is really just six or seven gestures by necessity because they’re trying to do real things. Those real things are requiring them to do more than swipe to activate or pinch to scale, et cetera.

So by all means they’re inventing gesture commands beyond what we’ve seen in sci-fi.

Jared: So, one of the theories I’ve always had, and I’m going to come back to this, that you were just saying, in a second. But one theory I’ve always had is in my days of going to academic conferences where people were exploring user interface in any number of ways like the CHI Conference or the Whist Conference, you would see something like a gesture based interface like what Myron Kruger had for his virtual reality stuff back in the mid ’80s. He was at the University of Connecticut and did all these incredible virtual reality things with very low budget.

And then you’d see that in movies five or 10 years later, almost identical to what’s there. In the book you point out a couple of places where stuff that was in research ended up directly in the films.

And then it becomes sort of some dude says, “I can build the Minority Report interface for $100 using these homemade tools.” The next thing you know you’ve got a gesture based thing using the built in camera on your Mac and all these other things.

Is there a virtuous cycle that goes back between what the effects people are doing and what we’re doing in research and what people are doing in hacker/production environments?

Nathan: Yeah. By all means, we’ve seen and identified examples where production designers, directors, producers see something in the real world and it inspires them to create something in science fiction.

That could be as simple as it’s an image somewhere that they just rip off as scratch and put it into a screen somewhere by coloring it blue and making it glow to outright homage where they’ve seen something rather sophisticated and either reproduce it or actually improved on it.

And then we see examples of science fiction influencing the design of stuff we find in the real world.

Jared: Can you mention an example?

Nathan: Yeah, I mean, the best example, I think, that we found in the book is Douglas Caldwell at the Department of Defense seeing this map planning tool at the end of “X-Men” and being inspired by it, because he works in the Army Corps of Engineers for the office that creates what are still called “sand tables,” which are essentially the same thing but much less sophisticated.

So he goes back to work the next day, writes a new RFP referencing this scene in this super-hero movie. This is a DOD RFP, mind you. Ultimately hires a company that three years later produces the best example or the best fit of what we can do to reproduce what he saw in the film with real world technology, and actually extending it even further to create the first dynamic sand table.

Chris: Interestingly enough, we found out later from–not Bryan Singer himself, but somebody who worked on the movie, that Singer had been influenced by one of those little pin boards that you see in the malls, like at Sharper Image, where you push your hand up in it, and you see an image in the pins.

So it’s an interesting long chain of events that lead to that dynamic sand table.

Nathan: Is this just one of several examples of reciprocal relationship between what we see in science fiction and what we see in the real world and how they come to be.

Jared: That’s really cool. Yeah, there’s a similar project I saw in a TED talk where a dude at Stanford has created a navigation map system for the blind that uses a similar technology to those pin things that’s air pressured on the back, and can draw the road ahead of you and you can actually drive a car keeping your hand on this thing and feeling where you are relative to obstacles on the road and things of that sort. I wonder if that was also influenced by thinking about that interface.

Now this gets me to this other thing, right? So every designer has been in a room where someone has leaned forward and said, why can’t we make this like Minority Report?

Chris: [laughs]

Jared: I just saw Iron Man the other night and it was really cool the way he talked to his fire extinguisher. Why can’t we talk to fire extinguishers that way?

At what point does all of this science fiction complicate the understanding of what we can and can’t do today. Is that an issue or is that in fact to our advantage, that maybe we should be questioning that?

Chris: I certainly think it’s the better problem to have, trying to convince clients to just improve their thing a little bit as opposed to a lot. I would much rather have a client come to me and we have to reign them back, and that’s so rarely the case.

So it actually provides sort of a service for us that way. I think the one danger that I’ve seen, certainly at Cooper, is where people are asking for the wrong future interface, so when we are working on the next fire extinguisher and somebody mentions Iron Man’s Jarvis interface as opposed to one that might be a better fit.

But at least they broached the topic and they’re looking forward and we get to be able to say, “hey, that’s a cool idea, but what if?” And then try and bring up a better one that they may have seen.

Nathan: One of the instances where this can become a problem is that it creates expectations beyond what we can technologically or feasibly create. If it’s our clients, we run the risk of dissatisfying them, because they saw something in a science fiction film that wowed them and they want us to recreate it, but we really don’t have the technology to make it happen correctly.

And on the other side of that, science fiction often raises the expectations in audiences about what can be or should be. And again, if we try to follow those expectations but we can’t make good on them, then we really run the risk of failing in a way that we wouldn’t if we just didn’t raise those expectations or try to meet those expectations in the first place.

Jared: Well, I’m wondering if things like Siri are good examples of where the expectations are somehow miscast for some people, because they do believe that it’s this perfect butler type experience. Siri often falls short and you have to repeat things. There’s issues that, for some reason, you never see them doing a lot of in the movies.

Chris: Yeah, certainly. I think that we talk a bit in the book in certainly the anthropomorphism chapter about the dangers of raising some expectations in your audience. And anthropomorphism is one of those things that’s really tricky to work around.

As soon as something begins to adopt human-like behavior or language, people drop into this mode of “Oh, well, you have fully human capabilities.” And of course, technology is nowhere near there. We haven’t quite hit the singularity yet.

I mean, it’s just not going to meet that expectation, so it is something of a danger.

Jared: Well, I think that what you guys have done here has been really incredible. The collection of all these examples and the analysis you’ve done of them is really–how many hours of watching movies did you have to go through to… [laughter]

Nathan: And are still going through.

Chris: Yeah. We’re still going through.

Nathan: They keep making more.

Chris: Yeah. We told them to stop, and they didn’t listen to us. Wow. I don’t think I’ve ever tallied it up. Have you, Nathan?

Nathan: I don’t think we could. I mean, …

Chris: We have been working officially on the project for about six years now, but we entered into it partially because we are fans of the genre, and we’ve been working on it all our life. They’re not going to stop making sci-fi any time soon, and we’re not going to stop using sci-fi, either. I think we’re in this for the long haul.

But the other thing that we could do, and actually it’s partially a plug, is that we really hope that if there are others who want to join us on this, we’ve got a website up called scifiinterfaces.com, where we’re slowly unveiling the database that we used to write the book. Hopefully others can join in to either just use that database in order to ask their own questions of science fiction interfaces, or to help us build it out with their own favorite TV shows and movies.

Jared: What are two movies that you felt are unlikely, in the more obscure movie places, that people should see that they probably haven’t, what would you recommend?

Nathan: Chrysalis.

Jared: Chrysalis, yeah. You talked about that at Web Visions. I know nothing about this film. I never heard of it.

Nathan: It’s a very beautiful French film.

Jared: Oh, need to see this. OK.

Chris: Yeah, and they have a beautiful gestural interface in there that is used by a doctor for tele-surgery. So she is this surgeon in Paris. Am I right, Nathan?

Nathan: I think so, yeah.

Chris: Yeah, I think she’s in Paris, and her patient is on a table I think somewhere in Latin America and she’s still having to do heart surgery on this little girl. Not only is it pretty well done, but it’s beautiful to look at and uses gesturals appropriately and looks like it has a lot of medical… It’s something that I suspect…Is tele-doctors a word yet?…That tele-doctors will really look to as a model.

If I was going to add one I would say Sleep Dealer is a…

Nathan: …spatial volumetric projection.

Chris: …oh yeah. Yeah, thank you. That’s actually what makes Chrysalis so beautiful.

In Sleep Dealer it’s actually kind of an unknown Mexican sci-fi film which tells the story of a fellow who lives in Mexico City but works in, I think it’s LA, through the gestural control and the immersive experience of being a robot. So he’s essentially a construction worker but the way he does the work is he sort of gets into this harness and then remotely controls the construction robot across the border.

It’s complicated by these military drones that are heading over where he lives and it’s not only a well-told story about the likely use of technology, but they get a lot of the things right and raise all of the social issues around it that are sort of dangerous. It’s not really well known and I highly recommend it.

Jared: OK. So those are Chrysalis and Sleep Dealer. Those are on my list now.

Chris: [laughs]

Jared: I’m going to put them out there.

Nathan: There’s a ton more.

Chris: Yeah, if we had more time.

Jared: Yes, OK. So, guys, this has been fabulous. Love the book and I think everybody should get a copy and read it right away.

We have Chris Noessel today who’s been talking to us. He’s a Managing Director at Cooper and his co-author Nathan Shedroff, who is the Chair of the MBA in Design Strategy Program at the California College of Arts in San Francisco. And the two of them have just published this fabulous book called “Make it So: Interface Design Lessons from Science Fiction.” You need to go out, read this book, and create cool stuff.

Chris and Nathan, thank you so much for playing along with me today and answering all my silly questions about the cool stuff you’ve been doing.

Chris: Sure, thank you.

Nathan: Yes, thank you very much.

Jared: And I want to thank our audience again for listening, and as always for encouraging our behavior. Take care. We’ll talk to you next time.

Add a Comment