, , ,

Part II:  Recent User Experience Changes on Books, Music and Movies

In the first post in this series, we looked at recent technological changes; in particular, we focused on touch and intelligent voice interface. In this, second in the series, we’ll examine how these changes may play out with regard to media focusing on books with a brief look at music, and movies. We’ll look at the current implications of these changes and some potential unrealized. We’ll also look at use of voice and touch to do things other than make stuff happen, in others, look beyond simply seeing them as a replacement for keyboards and mice. Some changes, of course, are already underway. By the way, most of the examples of what I’ll be discussing come from Amazon. It, along with Microsoft (and especially XBox’s Kinect) are doing the most consumer based innovation on interacting with content. So what are the themes as we see technology change our interactions with content, especially regarding books:

  1. We’ll use whatever works best, including combinations of interfacing with it. 
  2. We’ll start doing some new things like bimodal reading
  3. We will do more sharing, commenting and creating of content in reaction to the content we consume.

Current State: Navigating and Syncing with Touch and Voice

If we look at e-readers, we see that voice and touch both already play a role. Starting with the Kindle Touch, we obviously see touch interface with the reader. This continued through the Kindle Paperwhite and the Voyage. Clearly, there are other e-readers that perform similarly.


What the Kindle readers also bring is Whispersync for Voice. Now this is different from a typical voice interface since this involves syncing between audiobooks (hence, the narrator’s voice, not yours) and e-books. So, it’s neither an intelligent interpretation of words nor a way to directly interact with the book that were reviewing here. Rather, it’s using location recognition within a book that is available for both voice and E text to sync with one another so that you may start the e-book from the last place you listened to the audiobook and vice versa. (More on which here.) So it’s helping navigate, sure enough, in the sense that it’s getting you to the point that you want to be, but not based on voice commands.


New rule – no rules, mix, match, notes, research: use touch, voice and everything else as needed. 

So that leads us to our first point, which is not all the updates using touch or voice are all about navigating or commanding. Our second point is that you should not limit yourself to one interface even with the same object. This is the “mixed martial arts” version of an interface, use what works. So it could be that you take a combination of pen entry and/or freehand drawing voice and text all to make your point, or convey information. As one of the beauties about Microsoft OneNote is the product’s built-in ability to use as touch, pen, voice, or keyboard – whatever works. That’s really what this is all about making stuff work making it work more easily make it work more quickly being more productive. OneNote can even take an image with text embedded in it and OCR that text to the point where it’s editable. It’s a little like Johnny Bench, it will take whatever you throw at it any way that you throw it.


So, we don’t use new styles of navigation simply for the sake of being new, but they’re actually helpful. Changes in e-book interface using voice and touch are established. There’s also some other sensory data that play a role, for example, on the Voyage there’s a sensor that allows it to recognize the ambient light around the reader and adjust the light accordingly. So the interface is made brighter based on the environment. Also if we turn the device the print will translate to landscape. One of the nice things I like and using the Microsoft Surface tablet in the Kindle app is to put it in portrait mode for comics and there’s a feature where you can tap through sub- pieces of the panel within the comic and it will enlarge that area so it recognizes that you may want a slightly larger viewing area than the whole page and so it will expand out that part and dam and make smaller the rest of the comic so that it’s a little bit more clear and you can actually navigate from sub-panel the subpanel without ever going to the full panel. Practical and brilliant use of touch interface.

Screenshot (20)

We also must recognize what works well and voice and touch have their limitations. In the Voyage, you have the option of using the screen to navigate or using tactile presses called PagePress that are along the sides of the bezel. While the initial push was to mimic the process of reading a book on paper as much as possible, over time, we recognize this may not be the best way to experience an e-book. That sometimes means recognizing that it will involve a different process than paper. In the case of the Voyage, you see a step seemingly backward by now allow separate button-like pushes to navigate. This may simply be recognizing that while similar to a print books, it is different and some things work better than others in that medium. All this is to say touch and voice are already playing a role in how we interact with books, albeit, the voice interaction isn’t our voice directing things.


We Can Navigate – What’s Next?

But where could this be going? Well there’s some real clear ways you can see this process moving forward in terms of taking notes. One of the great benefits of having it on Kindle as you can note some highlighted passage and get back to that later for review or research. It would be terrific, if you could simply speak out your notes and their translated into textual notes and saved associated with that passage ala Dragon NaturallySpeaking. Enabling intelligent voice search within a Kindle would be terrific as well and then could be extended to intelligent voice search within an audio book. I can see having a more tightly integrated audible interface into the Amazon Echo, so that you can search, start, stop, and bookmark audible books via the Echo. All of this would simply be an evolution of what’s already there for touch and voice. Amazon also experimented with an artificial voice reader of text for those who have difficulty reading. This was never a substitute for a narrated book, but rather a stop-gap for those who could not read the text well. The Kindle Fire allows for binaural reading, which essentially has the text and the audio portion synced together as your reading. So the text dynamically follows along at the same pace as the voice. There are some studies that suggest the content is retained at a greater level when binaural reading is used. So, touch and audio aren’t necessarily all flash and amazing intelligence; they may work in such a natural and embedded way that you don’t even notice that they’re involved in your interface.

Inside Echo Sound

Inside Echo Sound

From an audio perspective, obviously audio is already deeply involved with books. There are even experiments in providing a more immersive experience with books via background music and noises; a sort of soundtrack for the book. See, for example, Booktrack and read A Scandal in Bohemia complete with music, sounds of horse hooves along cobbled stone Victorian streets, the bell at 221B being rung and footfalls up the stairs. Also check out Soundtrack to a Book. While this may distract as much as immerse, it’s a whole new way to interact with books. For myself, I prefer imagination and a little book music.



Listening to music is another place where voice and, even touch, have made little change.  Other than generic searches that lead to music, we don’t see much in the way of voice being used to access any of your playlist until Echo came along and changed that to some degree. Now voice can be used to search for music search for artists, request an album, and request a song via Amazon’s Echo. Clearly that’s a bit of a boon to those who are visually challenged, but it does take a practiced effort to get what you want out of the Echo. One of the things that I’ve noticed, for example, is that the search results returned an item as opposed to a list of items among which to choose. So if it’s not the version of Vivaldi’s Four Seasons that you desire, you don’t have an easy recourse other than more specific searching. In other words, were a long way from using natural English to elegantly interface with music players or anything else. While we have some touch-enabled interface, for example iTunes, Beats Music, Spotify or Xbox Music players on smartphones and tablets. So there’s nice ways to navigate there is well-presented albums and their respective covers, it still mostly around allowing touch is a way to navigate with your music in very similar manner as using keyboard and mouse. It’s nice that they’ve finally gotten to the point where they incorporate the idea that I don’t want to scroll through hundreds and hundreds of things to get to what I want and that search isn’t always the way want to go either. So like this idea of navigating to an album by clicking on the letter brings up the alphabet, then selecting the album beginning with that letter. On the PC it’s even a little bit weaker we see Xbox Music allow you to use touch interface is just as easy to do the same thing with a mouse why then select yellow submarine.

Amazon Fire TV Android App

Amazon Fire TV Android App

In other words, so far, there’s been relatively little effort into how we might interface with our music with touch beyond navigation. Do we want to do some mash-ups. Are there ways that we can use touch to more dynamically build playlists. How about I literally shuffle images to build a shuffle playlist. I honestly don’t think we’ll see a new shift in how we interact with interfaces via touch and tell we have some ability to mimic 3-D, then the possibilities will open up. Are there ways that we can visually represent the music via touch more readily or more creatively at least. So one of the things that I think will see changes not only is touch fused to navigate with our media, potential be used interact with our media. What if I want to build a video/slideshow by grabbing bits of music and pictures and even other video and mashing them up. I suspect there’s a way to do this with multiple fingers in such a way that it’s really creative and fun almost a game. Similarly with voice, not only do we went to navigate via our voice or issue commands, but possibly use a tone for voice to indicate a beginning of a song that we want to pull into a playlist. Hum a song and have it identified on shazam. Build riffs off of current songs that were listening to and place them within another context. Indeed, I think we’ll see that the next step in touching voice is less about navigation commands and making things happen, and more about interacting with the media itself.

XBox 360 with Kinect

XBox 360 with Kinect

Of course, of recent innovations with how we interact with content, XBox’s Kinect where motion is used to interact with the user interface as well as games. Windows also uses movement and gestures to interact with the OS; many people find this interface unintuitive, but it is innovative nonetheless.

Viloin with Books

We’re Creating, Not Merely Consuming

The technological impact on consuming media then it goes well beyond navigating and initiating via voice and touch. We see the enabling of information the idea of x-ray or IMDb together more information why you’re consuming books and or movies, the ability to insert your own notes and highlights and come back to those in the ability to share through things like good reads your experience within that content. In other words creating content sharing via various social media writing your notes bringing it together for a reading group or movie club are all going to change Artie have changed the ways that we consume the content. Will look at embedded trailers before we go to the movie or view it online. We’ll look at reviews before we purchase a book. All of that information changes, at least a little bit, how we interact with our content.

Example Kindle notes

Example Kindle notes

In other words creating content sharing via various social media writing your notes bringing it together for a reading group or movie club are all going to change Artie have changed the ways that we consume the content. Will look at embedded trailers before we go to the movie or view it online. We’ll look at reviews before we purchase a book. All of that information changes, at least a little bit, how we interact with our content.

X-Ray Example

X-Ray Example

A similar notion, using IMDB, holds for movies:


X-Ray for movies