Josh Clark – Discoverability in Designing for Touch

Sean Carmichael

January 5th, 2012

Play

[ Transcript Available ]

While the traditional “mouse and cursor” interfaces are still in use, many of us are becoming familiar with touch-based interactions. The power and capabilities of mobile and tablet devices are growing. Often, these devices are the more convenient alternative for users to access your content. But beyond accessing your information, how are they interacting with your design?

Josh Clark, the author of Tapworthy, offers the notion that buttons are a hack. Touchscreen devices allow users to manipulate content with more than just their index finger. Multi-touch gestures can be used in many apps, in some case as the equivalent of keyboard shortcuts on the desktop. It’s a great way to create a fluid and deeply engaging interface.

The problem? Gestures are invisible. This leads to discoverability problems because it’s not clear what a certain gesture accomplishes, and they’re not the same in every app. Because there is no pattern library for gestures, it takes something like word of mouth for a gesture to catch on, such as the “pull down to refresh” gesture.

Josh shares his thoughts on designing for touch with Jared Spool in this podcast. And if you need more from Josh, you won’t want to miss his January 12, 2012 virtual seminar, Buttons are a Hack: The New Rules of Designing for Touch.

As always we love to hear your thoughts. Please share with us in our comments section.

Recorded: December, 2011
[ Subscribe to our podcast via Use iTunes to subscribe to UIE's RSS feed. ←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]

Full Transcript.


Jared Spool: Hello, everyone. Welcome to another episode of the Spoolcast. I am very excited to have Josh Clark with us here today.

Josh has been speaking at our Web App Masters tour for the last couple years, and he’s doing a virtual seminar for us on January 12th of 2012, called “Buttons are a Hack: The New Rules for Designing for Touch.”

And I wanted to sort of get into the background of that with Josh today, and he’s joining us all the way from beautiful Amsterdam, across the wonders of the Skype.

Josh, how you doing there?

Josh Clark: I’m doing great. I’m tempted to decide to call Skype a Dutch word, but I don’t actually know if that’s true.

Also, you know, hey, I wanted to say, by the way, so, we’ve got this virtual seminar coming up on January 12th, and I wanted to be sure that, for the folks who come, the folks who listen in and watch the webinar, definitely wear your party hats, because it’s my birthday.

Jared: Oh, I didn’t realize that.

Josh: Yeah. That’s right. I will be, I guess, what, 23 years old, as far as you know.

Jared: Well, yes, in Internet years.

Josh: Yes. That actually makes me 150, I think, but that’s…

Jared: It could be, it could be. That’s really awesome, happy birthday. I think if we wanted to make Skype a Dutch word, we’d have to use a J.

Josh: Right. I’m staying here on the Skypeingrocht. Skypeinstrasse.

Jared: Yes, indeed.

So, you’re talking about the world of touch. And this world sort of just sprung out at us, as far as I can tell, really fast. It reached out to us really fast, maybe, is the right way to put it.

And you went on and wrote a fabulous book called “Tapworthy,” and have some other stuff in the works right now. But you’ve sort of immersed yourself in this world where we interact by gesturing on plates of glass.

Josh: It’s true. I mean, you’re right, that it has kind of come out of nowhere. The world of touch was, for a long time, limited to technological curiosities, Microsoft Surface, coffee table, for lack of a better word.

Jared: Well, airport kiosks, right? I mean, it was this really, sort of, clunky museum, big button, you just press really hard, and hopefully, it’s calibrated enough so that it actually presses the thing you want.

Even ATM machines were only touch sometimes, right? Often times, they used hardware buttons, because they couldn’t count on touch actually working.

Josh: Right. And it’s amazing, it’s gotten so much better. You still run into that, I would say, bank machines. I use Bank of America, I hate to say. And their ATM machines all touch screen based, and they’re all calibrated wrong.

They’re always, like, at your waist level, so your angle is weird, so it looks like you’re touching at one point that you’re not. I miss all the time. I mean, here I am, right, I’m supposed to be some touch screen expert, yada yada. I can’t even make my ATM machine work.

So, we’re still, clearly, learning how to use touch and design for field of view, design for ergonomics, design for different stances. You know, the stuff that I work with is primarily handheld, phones and tablets. But the fact is that we’re starting to see touch show up and get refined in kiosks where they’ve been a long time, as you’ve seen.

But also, on desktop screens, you know, with the early word from Microsoft is that Windows eight is going to be a touch based UI, where you flip through screens very similar to you do on the Windows phone, and on the Zune player before that, which Windows phone was heavily influenced by.

So, here’s this touch interface that, and we’ll see how this works out, because it’s kind of jammed together in sort of an unholy alliance with old Windows. You’ll be swiping through and tapping through stuff, and all of a sudden, whoa, here’s a Microsoft Excel spreadsheet from 10 years ago, and now you’ve got to figure out how to go back to keyboard and mouse.

It’s a little bit of a Frankenstein thing going on there, potentially.

Jared: I’m imagining some sort of mash-up between Minority Report and Dance Dance Revolution.

Josh: You know what? If it turns out to be that, I think that we will all be incredibly happy Windows users. You know, I think that probably a lot of the folks that listen in to this, sort of, the folks who do design work and UX work are probably Mac folks. But you know what? You just described the recipe for getting everyone back to Windows, the whole world.

Jared: That could be it, that could be it. And I’d probably lose 10 pounds. Or at least, temporarily misplace it, which is what I usually do with weight.

Josh: Oh, here it is.

Jared: Exactly, exactly. I know it’s around here somewhere, I probably just left it in the bedroom.

Josh: It always rears its head this time of year.

Jared: Yeah, yeah, it seems to find its way back.

So, lots of people are taking things they’ve been building for that sort of single click, index finger mouse world. And now, trying to bring it to the glass plated gesture touch world.

I’m curious if you’ve run into any sort of examples of things where that transition hasn’t gone as smoothly as it probably could have.

Josh: Yeah. I think, in general, we have a broad problem with what’s called discoverability. You know, gestures are not labeled. They are invisible. You have to find them.

And so, I see things go in two directions. One is, there’s no cue at all, you know, there’s no effort from the part of designers to help people find these gestures, which are powerful shortcuts, in many cases.

So, the new Twitter app, for example, which, for their mobile apps, people kind of got upset about it. And particularly I’ve seen a lot of conversation around the iPhone app, where, sort of, once core features, like direct messages and account switching, for people who use multiple accounts, got pushed a layer deeper into the app, sort of, hidden underneath one of the tabs, the me tab or your profile information is kept.

And they give you shortcuts, though, to get quick access to that, they swipe up from the lower right corner, from the me tab, that will trigger your direct messages and get them to you instantly rather than having to poke around.

And so, this is the kind of thing that’s like, wow, this is great, except that they aren’t discoverable. It’s something that’s unlikely for someone to try on their own. So, they have to be bold about it.

Jared: Yeah, I was told by a Twitter employee. My guess is that all Twitter employees are now responsible for spreading this. I call these word of mouth functionality, right? Where the only way you can learn about it is through word of mouth.

Josh: And, you know, fair enough, that’s the way a lot of expertise is conveyed. But I think that it’s something that the application can actually teach you in context and lead you through that, as well.

I don’t want to put in too many spoilers, because this is actually the meat of the seminar, is how do you design these gestures and put them to use and make them discoverable? That’s really something that I’ll be talking about a lot is really practical strategies and techniques that you can do to use some of these next generation interface interactions.

But in ways where people will find them effortless and easy to learn and won’t, two years later, after using your apps, are you kidding me? That’s been there all along? Which, I think, is often the way it is with those kinds of shortcuts, you know. You’re glad to learn them, but you’re a little bit angry that it wasn’t easier to find.

Jared: Yeah, I was going to say, these things have sort of been around for a while, because, you know, if you think about desktops, drag and drop isn’t that discoverable, either, right? I mean, if you’re familiar with it, you might experiment with it.

But, for example, you can click on an image in many browsers and drag it someplace, like on a desktop and it will do something with it. And what it does, you can’t tell until you’ve done it once. The fact that you can pick it up and drag it is not an obvious thing unless you do it by accident, you know, you’re holding the mouse button down when you didn’t mean to and move your hand, and suddenly, oh, wait, I just dragged that.

And so, this sort of behavior of things that you can’t discover and have these sort of hidden semantics to them, has really been around for a long time. It’s not new with gestures, but I’m wondering if somehow, because the screens are smaller and therefore, we’re more constrained as to what we can put on the surface if we’re more likely to run into this in the touch world.

Josh: Well, you know, I think if there is, and I think it’s not just small screens, I think it works for tablets, as well. I think it’s twofold. And one is that, even though the screens are nearly the same size as tablets, you still want big chunky elements and generous white space, because that’s what makes them ergonomically friendly. Which means that it does take up more room to have controls.

And in general, the whole trade off of interface design, since the get go, has traditionally been, the more features, the more functionality, the more capability that any particular screen has, typically, the trade off is more and more chrome. You know, I think, Jared, you use in your talks a lot, you showed Microsoft Word with all of its toolbars and ribbons shown. It literally fills up the whole screen.

But that makes visible all of the functionality, all the huge functionality of Word. And one of the things that’s sort of interesting about gestures is that now we’re able to use non-visual language and controls to replace some of that really dense and frankly, confusing chrome.

But now, you’re right, we have to also take into account techniques that reveal those, the presence of those gestures. And there a whole bunch of techniques that I’ll be looking at from coaching to animation to all kinds of little things.

Sometimes, they’re so subtle that they won’t even get your users’ attention consciously, but will be enough to get them to almost subconsciously try to do things. It’s the Manchurian candidate approach of interface design.

I just made that up now, by the way.

Jared: I like that, I like that.

Josh: You can have it for free.

Jared: OK. Manchurian candidate approach. That’s really cool. Everything just suddenly changed.

Josh: Oh, but, I’m sorry, one other thing, you asked me about, sort of, examples where we go awry. A big popular thing is to use screens that are themselves, sort of, 3D clones of a physical device. You know, sort of skeuomorphic design.

And the advantage of this is, wow, if it looks like a physical object and it behaves like that physical object, I will understand how to navigate it. If it looks like a book, I will understand that I can swipe to turn to the next page, to get more content.

But I think a problem that you see, a lot of times, is that people don’t follow through on those metaphors. So, you have things that look like a book, but turns out, if you still punch in buttons, flipping pages doesn’t actually work. You see that in Apple’s context app for iPad, for example. And until iOS five came out, in their calendar app, as well.

And so, the thing is that, we’ve used these tricks and stunts forever in print and, more recently, in desktop software, as eye candy, pretty up the design, you know, you see this in OS 10 all over the place now, there’s sort of all this gratuitous stuff that looks like 3D objects. And it’s just eye candy, has no function.

But when you put that into a handheld device, suddenly there’s a real expectation, and this just goes to the different way that touch interfaces tickle our brains, there’s a real expectation that, because I’m holding this physical object, if it looks like a physical object, I would expect it to act like one.

And so, there’s a real thing now that you have to follow through on your interface metaphor.

Jared: I think that, this is interesting to me, because this, to me, feels a little bit like history repeating itself. Because I remember the days of, in the ’90s, when Microsoft Bob came out with these sort of pseudo skeuomorphic designs where you had a physical desk as your desktop. And you manipulated books on a bookshelf to get to data, and you had a file cabinet to get to your files and you had a phone to operate your modem.

And then again, when the web first came out, there was Southwest Airlines, their first website, they thought, OK, we’re going to be really clever. So, you had to walk up to the ticket desk, and you interacted with an agent at the ticket desk. And there were e-commerce sites where you had to physically place things in shopping bags, and the shopping bags got more and more bloated as you shopped more.

And there was always this sort of, in some ways, awkward skeuomorphism. Some of it had to do with the fact that the resolution and the number of colors you had and stuff couldn’t do photo realism. But even in interfaces that could do photo realism, it always felt, I don’t know, kind of Uncanny Valley-ish.

Josh: Yeah, well, right. So, a couple of things. And one is, we’re always dealing with illusion, some sort of metaphor, some sort of interface metaphor that borrows, in some way, from something we already know.

So, you described a whole range of experiments in there of trying to create new metaphors, when the web came out. And in the traditional desktop also had things like that, well, you mentioned Microsoft Dot.

But, when you look at what we did arrive at, which is, here’s some pictures of folders and here are some pictures of documents and we’ve got a little trash can over here that bulges when you put stuff into it. I mean, we still did wind up with some sort of metaphor of using the real world to express notions of containers and files.

And, of course, there’s no such thing as a file on your hard drive, it’s all a bunch of little magnetic impulses, zeros and ones, essentially. This is this mess of stuff that your operating system is somehow making sense of.

So, I think that it’s important to realize that whatever we come up with for metaphor, it’s illusion. I mean, that is what we do. That is what we do as interface designers, we create illusions and abstractions that stand in for the manipulation of content and information. And touch is giving us, now, a new way to do that.

And so, it is natural, I think, to explore as we get a new way to interact, to explore fresh metaphors that go away from our traditional GUI desktop. But, yeah, I think it’s important to remember that things can veer pretty quickly into kitsch. And we do see that a lot.

You know, I think that the Yahoo entertainment’s iPad app, when it first came out, was sort of this really, sort of impressive looking living room set that you sort of see from above, and you can manipulate the stuff in the living room to do the kinds of things, very much like the Southwest waiting room ticket office that you described.

But I think when we see things like books, like, all right, that’s a pretty straightforward metaphor that we should at least be able to follow up on. But you still see a lot of things, book interfaces, that don’t let you turn the page, which is on Apple’s own apps. Again, the contacts app is like that, it’s an address book that works with desktop style buttons.

Jared: Yeah. Yeah, I think there is, really, something about thinking about taking that all the way through and making sure that you’ve got all the gestures the way it works. And I guess some of that comes from making sure you’re spending the time.

You know, the lens of the world I always use comes from a user research perspective, right? So, if you go out and you do your user research, and you actually watch people interact with these things, at some point, you should see someone gesture in a way that makes a lot of sense, that they start to manipulate the real world.

You know, there’s that You Tube meme that’s floating around, of the video of the three-year-old, or two-year-old, who’s trying to use a print magazine like an iPad, because she has these gestures that she’s learned worked on the iPad, and now, she’s trying to get pages to turn by swiping them instead of picking the page up and moving it.

Josh: Right. I think it’s titled, “Magazine is a broken iPad,” I think. Well, and there’s a similar thing, you know, there’s a video making the rounds right now with a lizard playing a game, oh, what’s it called? Ant Crossroad, I think, is the name of the game, or ants crawl across the screen and it’s trying to zap them with it’s tongue.

I guess my point with this stuff is, if you do go the physical route, of aping a physical object, you should make sure that it behaves like that object, so that you’re giving people useful cues of how to use it. Because it’s not just eye candy when it’s in a handheld device, it really is a cue for how you’re supposed to use it.

But, on the other hand, you don’t necessarily have to do that, either. You know, you look at something like Windows Phone, which is not a realistic interface, you know, it’s very two dimensional, these tiles that move around the screen.

And so, it’s not trying to be anything that we know from the physical world, but it operates as if it were physical. Which is to say, that there are these tiles with physics that you slide around. And so, while it’s not necessarily something that you recognize, it, as an object from the real world, it is something that responds very physically.

Jared: It was actually interesting, this idea that you sort of have to create physics properties for your virtual space. And say, you know, if I point and I move my finger to the right, that’s going to create a momentum for this set of pixels to actually react to that and have physical properties, so that if I do it fast, they should move fast, and if they do it slow, they should move slow.

And if they don’t quite complete the gesture, maybe they should snap back into place. Is that what you’re saying? That having those?

Josh: Yeah. I mean, just in general, this notion that there’s a physical response, creating the illusion of a physical response to your touch. That yeah, and all those things that you talked about, the snap back, and sort of the inertial reaction of things slowing down or bouncing around the screen, those things are hard to get right.

And this is one of the very subtle, I think, polish and finish differences that you see between Android and iPhone, for example. Where iPhone, they really paid a lot of attention to it. And for Android, depending on the hardware you’re on, it can feel really stuttery, it brings you out of that illusion.

But I think one thing that you mentioned earlier is how we’re manipulating things under glass. And that’s the fundamental illusion that we’re able to create, right? It’s manipulating flat objects under glass, two dimensional things. There are opportunities where we can create illusions of depth or of texture. And those can also be inviting and give cues for where and how to touch the screen.

But ultimately, at the basic level, the Windows Phone approach of having tiled content that you slide around and move through and tap and touch is sort of the most honest approach, right?

I’m not saying that one is better than the other, I think that a lot of people are fed up with, sort of, the skeuomorphism as a visual design trend. And fair enough, I think we’ve seen a lot of it.

But I think it’s also really useful to help people turn the corner into getting used to using these objects. And Apple’s definitely been a big proponent of doing it, and its touch interfaces. And increasingly, in its desktop interfaces, too.

And I think that, it’s hard to say that it hasn’t been a success when you see newcomers to computing, either, typically, on either side of age spectrum, either toddlers or seniors, how quickly they’re able to pick up and understand this through the direct interaction and often through these real world cues. You know, that is a powerful teaching mechanism.

Jared: Yeah. I think it is. And yet, there’s still things that are difficult to manipulate. You know, so, for example, I have trouble, I don’t know, maybe you do, too, with the video slider in video player on the iPhone. So, being able to, you know.

I want to go back, you know, somebody just said something interesting and I was only half paying attention, and I want to go back and hear them repeat it. And I find it to be a really clumsy act to just somehow get back a few seconds so I can hear what they just said without going back and having to listen to the last 10 minutes.

Josh: Yeah. So, you know, what’s interesting about this, is that there actually is an adjustment for this. But this, again, is something that’s poorly explained and it’s not easily discoverable, is that if you touch and hold that slider and then pull your finger way down the screen and slide from there, that you’re going to get, sort of, a more fine control on the scrubber control.

Jared: Oh, interesting.

Josh: Which is not obvious, but, sort of, when you think about it a little bit, it’s almost like, you were getting, like, leverage, if you think of it as a whole, as like, it sort of has a physical corollary. But, in any case, it’s not something that is easily discovered.

So, one of the things that I’ll talk about in the seminar is how to improve stuff like that. That means, great, that there is consideration for things like the shortcuts that I mentioned earlier, or different ways to use it to get finer control over the scrubber control, like you mentioned.

But we need to do a better job. We have to go beyond just coding that stuff in and doing more to help lead people by the hand, literally. How to touch these interface controls and how to use them. And how to learn them gradually.

You know, one of the things that I’ll talk about in the seminar is all the ways that we try to teach people to do it that really are failing, that are failing our users, and, in a way, then, they’re failing all of our efforts to make great software.

Jared: Yeah. I think it’s interesting. And I think that people who are sort of really been immersed in a desktop way of thinking have to have some sort of way of stepping back and attacking the problem a little differently.

You know, I’m thinking about, for instance, one of the banking apps that I use is from Chase, Chase online banking. And their controls, the way I manipulate my bank account, and this is on my desktop, through a browser. The way I manipulate my bank account is through this list of functionality that is all the line height of text, separated from each other.

And if they tried to move that to my iPad, you know, if I try and use the website for my iPad, it’s virtually impossible, because I can’t get, with my fingertip, that granularity of pointing control. And I’m often selecting the wrong function, because my fingers are just too wide, and they cover up what I’m touching. And all sorts of things that are not an issue when I’m using a mouse.

Josh: You want to know one of the really, and we talk about, sort of, irony, I guess. I think this qualifies as irony. Is that, actually, the harder that you try and concentrate to hit a specific point, on iOS, specifically, the more likely you are to miss. And the reason is, because of the shape of your finger, the part that it looks like you’re touching, to you, is actually above where the actual meat of your finger touches the glass, because of the curve of your finger.

So, they have adjusted the code downward. So, in other words, you touch a point on the glass, it will actually sort of give you the credit for touching a few pixels above. So, if you’re really trying to just zero in and touch with just the very tip of your fingertip, you will miss.

Jared: Yeah, so, I end up playing all these games with pinch zoom and trying to zoom into the text so that it’s big enough that it’s the right size for my finger. But then, the screen messed up and it doesn’t always work. It, sometimes, for whatever reason, the browser resets its page back to the original size before I can do that. And oh, it frustrates me.

Josh: I mean, this is why all of us in this industry got to get better at building websites that look great on every device, you know. And all the great work that folks like Ethan Mark Cohod and Erik Gustafson have been doing the last year to explain responsive and adaptive web design. So important right now, so that we can have those big chunky touch targets for mobile, but that sort of shrink down to more reasonable visual size on desktop.

That’s the big challenge for us, is to be a bit more flexible and clever about adapting our designs, or really, more specifically adapting our content to different sizes of screens. I don’t mean, what kind of content we show. I strongly believe that the same content should be shown on all platforms. But you know, how we display it.

Jared: Right. Right. I think that there’s lots of challenges, because I think it’s really easy to go overboard in creating all these gestures, so that, you know, if you sweep to the left and then move to the right and then do a little dance, that’s going to be open file.

Josh: Right. Or at least, hokey pokey.

Jared: Yes. either way, it’s what it’s all about.

Josh: Yeah, that’s right. Well, and I think that it’s important to realize that it’s for complex gestures, like what you’re talking about. Those have got to be power shortcuts, those have got to be for power users, right? Those are the keyboard shortcuts of touch.

So, just building a shortcut like that doesn’t mean you can eliminate the long way around. There still has to be some evident visual cue for any basic function. And so, that does mean that we need to have buttons in appropriate places.

But I think it’s important to remember that buttons are a hack, and I mean that in both the real world as well as the virtual world. And I’ll explain what I mean by that in the virtual seminar.

Jared: Yeah, no, I think you’re absolutely right. We’ve created this, sort of, monstrosity because of our fingers, and because of the way we’re built.

You know, Bill Buxton used to only half joke that if alien archaeologists land on our planet, you know, 20,000 years from now and dig up our remains of today’s society and find the computers of today, they would think that we only had one finger.

Josh: Right.

Jared: Because, up until recently, the devices we had, at least, the technology devices we had, sort of assumed that you only did one finger at a time, that you didn’t have feet, that you could move independently, and you didn’t have multiple fingers.

And I think that the gesture world pushes us beyond that, so that, you know, you don’t play a piano with one finger if you’re a successful pianist. You move independently across your hands and even your feet.

And it feels to me like these new technologies are helping us see into that world a little bit more and take advantage of what we can do. But at the same time, we have to create a whole new language for this.

Josh: Yeah, that’s right. And we’re just starting to get into that multi touch world. I mean, we’ve had multi touch on phones for nearly five years now. But only, sort of, in theory.

Most of us haven’t used multi touch, except for the occasional pinch, right? In general, most of us use just one digit, our thumb, to use the phone. You’re holding it in one hand, unless you’re holding it in some kind of crazy claw. You’re using your thumb to tap away.

And so, it’s really, with tablets, especially, now that we have a larger screen with enough room to put all of our fingers on the screen, and a form factor that you pretty much have to use it with two hands, means that you’ve always got a hand free to do more complex gestures on it.

So, you’re right, though, now that we’ve opened up this possibility and started to get some willingness to do it, we don’t have a very understood gesture vocabulary, at all. Because the iPad is, very literally, the very first truly mainstream far and wide, multi touch device where people will actually use multi touch on it.

And so, we’re in this horrible situation, not horrible, it’s an awkward situation for both users and designers, where we have to figure out what does a three finger swipe mean? And it means one thing on another app, and another thing on another.

So, we have a lack of confidence, as users, and as designers. And there’s nobody who’s really showing much leadership in it right now. I mean, my big concern, actually, is that it’s going to be the toolmakers who decide this stuff.

You know, as more and more people use, for example, Adobe’s toolkit, magazine makers and publishers, I’m thinking of, particularly, here, to export their print magazines into more fancy iPad designs. By using Adobe’s tools, they do Adobe’s interactions.

And you know, I give a lot of props to Adobe for a lot of stuff that they do, but I’m not necessarily sure that their interaction design is the best.

And so, I want to make sure, and it’s part of the reason that I like to talk about this stuff a lot and do the virtual seminar like we’re going to be doing, is to talk in a more thoughtful way about, what should these standards be? And how can we arrive at them as a community, instead of just inheriting them de facto from the big boys?

Jared: Well, that’s interesting. Because, you know, take something simple like pull to refresh. That just showed up in, I think it was a Twitter app, initially.

Josh: That’s right. It was Tweetie, what became the official Twitter app, that’s right.

Jared: Yeah. And then, all of a sudden, it’s everywhere. And it becomes this general thing. And there was no standards body that said, you know, that the UN did meet and say, “We hereby declare that pull to refresh will be the official way to refresh data for the rest of humanity.”

Though that would be cool if they did. Particularly with whatever accent I just made up there.

Josh: I really like that, I think it was kind of quasi-European, but also, vaguely gnomish. I don’t know.

Jared: Yes, it was very gnomish.

So, it’s interesting to me that that standard just sort of became the standard. And you’re right, I think some popular tool vendor is going to pick something, and everyone’s going to say, oh, that’s what we’ll use, and they’ll start borrowing and stealing it. And the next thing you know, it’ll spread like wildfire through the design community. And I wonder how we make sure that we don’t have regrets about what we settle on too quickly.

Josh: Yeah. I think that that’s a real challenge, right now, and a real worry that I have.

I mean, on the one hand, we need to standardize as soon as we reasonably can, because that’s what’s holding us back right now, is sort of a sense of confusion of, well, what does a three finger swipe mean? You know, what does that mean? Do I have license to do that, or am I just going to confuse people? So, it’s sort of a holding area that we’re in now, holding pattern, in that regard.

But on the other hand, you know, right, we don’t want to do it too fast. I mean, I think the pull to refresh thing is a great example of both the good and the bad of this. On the one hand, when Loren Brichter, the developer who made Tweetie, he sort of invented that gesture and put it into the Tweetie app, that then became the official Twitter app.

You know, he put a lot of thought into how it is that that should work, why that works. And, in fact, in the version before, sort of a quick, sort of, interaction design history of this app. In the version before he introduced pull down to refresh, he just had a refresh button at the very top.

And the thinking was, wow, all right, when you’re going to see what your latest tweets are, you scroll up to the top of the screen. What better to do than have a refresh button waiting there for you when you got there?

And as an exercise, I was like, well, wait a second, I don’t even need to do that at all, right? So, it has the instruction, pull it down, you snap it back, and the thing reloads.

And so, that is a great interaction. It’s this thing that is just sitting there waiting for you right when you need it. It’s piggybacking on top of the known behavior and gesture that you do to scroll to the top.

But what you see, interestingly, is that, other apps, where that behavior isn’t part of it, Foursquare, for example, for iPhone, uses that pull down to refresh gesture for locations, even though there’s no chronological aspect to it. And so, it’s something that you wouldn’t discover on your own unless you were a fan of other apps that use that, and you just naturally associate that with the refresh button.

This is also an iOS only behavior, you know, you don’t see this much in Android apps, this has sort of been an iOS community thing. So, we have this issue, not only of, right, do people understand the intent and context of the gesture to use it in the way that it was originally intended?

So, I think it’s great that the pull down to refresh gesture spread far and wide. But sometimes it’s spread too far into context where I’m not sure that it works as well.

And then, we also have cases of, well, all right, so what’s the Android community thing going to be? You know, as we develop platforms and software ecosystems, apart from each other, you know, is it going to be like developing ecosystems in the real world, where we have all this crazy stuff done in Australia?

Sorry, Australians, you’ve got some crazy animals there. Do you know the platypus actually has, like, electric shocks? Anyway.

Jared: Huh.

Josh: That we’re going to have, sort of, that.

Jared: Seriously? Like an electric eel?

Josh: Yeah, they’ve got, like, fangs and they’re poisonous and they have an electric field. Anyway, I learned this recently, I’m fascinated by it. But it’s like.

Jared: Are they like a renewable energy source? Could we, like, hook platypuses up to our toasters?

Josh: Yeah. I mean, I think that would be adorable. You’d just have to watch out for the poison.

Jared: A platypus powered toaster.

Josh: I’m also going to be doing another virtual seminar on the zoology of Australia coming up after this.

But no, but that’s my concern, it’s like, oh great, in a way, the pull down to refresh gesture is the platypus of iOS that doesn’t exist on other platforms.

And so, I guess I worry about this slightly, right, how do we manage these expectations of what is standard in terms of just good practice touch screen and what do we understand as working well on an Android tablet, or an iPhone tablet?

We’ve always had this kind of problem with operating systems, and we had it with Windows and Mac, they broach differently, similarly enough, but enough to confuse you when you tried to switch. And that was generally OK, because usually you were an all Mac or an all Windows person.

But now we have all these different touch screens in our lives that run different systems. And so, there’s, I think, a real need just to share and talk and it’s really, I think, a time to be generous with our ideas about what’s working. And I mean that even with companies that are competitors.

I mentioned publishing earlier. I think it’s crazy that you have to learn completely different gestures to read a magazine in different magazine apps. You know, I feel like, wow, all right, Conde Nast, Time Inc., get together and figure out how you want it to work. That’s not a competitive thing, that’s just basic infrastructure.

And I think that we as designers and developers and IAs need to reach out to our peers and say, hey, why did you make that decision? I’m thinking of doing this. Do you think that would work for your app, too? Would you be willing to sort of give this a try so we can sort of try to develop some standards?

But in a way, that’s a community way that comes from the right source, sort of, the creative source and from close up views of our users and how they’re using our apps, rather than top down from the toolmakers.

Jared: It would seem to me that the first step of that would be to start putting together some way of seeing what all these gestures that are evolving are and who’s using them where.

So, back in the ’80s, I worked for a company, Digital Equipment Corporation, and we made computers within our own product line. We had something like 24 different operating systems for different architectures of hardware that we worked.

And it wasn’t like the thousand versions of Android that are out there. I mean, these were radically different interaction models depending on the technology we were using.

And, of course, you know, a common thing across each of these things was being able to edit a document, usually a program file, right? So, you had to write programs, and you had to edit them, so you needed some sort of editing tool.

And so, there were 14 different editors from Emax to various proprietary editor tools to word processors to all sorts of different things. And I created this table, and I was the first person to ever do this in the company, because I was writing a brand new family of word processors. I wrote one of the first PC based word processors for this company.

And I created this table of all the different ways you could do something like cut and paste. And there were myriads of gestures and myriads of keystrokes and myriads of commands. And some had physical keys that were literally on the keyboard, labeled cut and paste. And others had control keys or control shift keys.

Or there was one program that had a meta key. So, it was meta shift control. And the left shift key and the right shift key would do something different, so it was like, meta left shift control right forearm would generate.

Josh: You’re making me dizzy.

Jared: Yeah. But we had to create this table of all the different ways to do common functionality, and there was no consistency across it. And I’m wondering if anyone’s doing a sort of similar thing today across the popular apps and just trying to catalog this.

Josh: So, I haven’t seen anything, but this is something that I have been wanting to do for awhile, and if anybody out there is interested in collaborating with me, I’d love to, is to start putting together a gesture pattern library.

This is still pretty new, just in general, just getting mobile pattern libraries together, and there are a few really good ones out there. But there hasn’t been a lot in terms of how are gestures being used, and how can we start to use a collection like that, as you said, have a conversation about what is it that’s working well and is working best? Because we’re still finding our way, you know, with that stuff.

And you’ll see apps take right turns, too, like the new Twitter app that I mentioned before, used to have this thing where you could swipe left to right across the title bar, the navigation bar in iOS, and doing that, if you were drilled down into the app, would just spring you back up to the top of your timeline. They did away with that in their new version, and now you get at it by tapping the home tab again.

So, you know, there’s still a lot of trial and error and experimentation within apps, let alone across them. And I think that that’s healthy to a certain degree, but it’s also frustrating. It’s, like you say, it’s a balance between getting to standards as quickly as we can, but with enough time and thought that we’re arriving at the right ones.

Jared:Wow. Well, you’re going to talk a lot about this further in your virtual seminar, which is going to be January 12th.

Josh: My birthday.

Jared: And your birthday, Josh’s birthday. “Buttons Are a Hack: The New Rules of Designing for Touch.” Josh Clark, thank you very much for joining us today.

Josh: Thanks for having me.

Jared: And thanks to everyone for listening. And once again, thank you for encouraging our behavior. We’ll talk to you later.

Add a Comment