In my experience, engineers often develop primarily for themselves. Until I began my design research, I never seriously considered how other people would perceive, utilise or interact with my programs. So long as they made sense to me, they were great. It was when I started developing ubiquitous computing systems that the concepts of usability and user-centred and participatory design were introduced to me by my advisor Margot Brereton and Danish user-centred designer Jacob Buur. When I was writing my dissertation on design it truly startled me to realise how sheltered I’d been from human-computer interaction best practices.  According to much of the literature and commercial practice, user-centred design and usability are now thought to be a common and necessary practice, but it still isn’t a core component to engineering education.

My own experience was a fairly typical engineering education which was a computer systems engineering degree at The University of Queensland. During my studies, I interned at Compaq, I continued programming for fun and competition, and I completed an honours thesis designing a mobile payment system.  The fact that someone immersed in software engineering hadn’t heard about user-centred design (or considered the user perspective in his own time) speaks volumes for traditional design education for engineers. This type of education, and the resulting world view, has ramifications for how user-centred and participatory design is approached within commercial design practice. It would seem (from my experience) that the way engineering is taught with a single-mindedly technical (or rather, problem solving) focus. This makes sense of course. Engineering is all about solving hard problems — engineers are given a requirement for solving a problem, and it is pretty darn satisfying finding the best way to solve it. This is seen in sites like StackOverflow and TopCoder. Engineers also tend to hold a strong personal interest in their field and in deriving optimal solutions for it. In adhering to the design specification given, whether it is then in turn usable to others, is not a primary consideration.

When supposing how people might use their software, many software engineers create a solution that works for them or an idealised version of the user.  Otherwise, the responsibility is left to a design team – who again, often idealise how a user might interpret and interact with the design.  Both react to later data on use and usually a few iterations are made.   In the past, the engineer would be the first step, but a greater proportion of systems design now attempts to create the interface as the first step in the design process (which causes its own problems of essentially asking the engineers for a pony – you can ask for whatever you want, but it doesn’t mean you can have it).  Either way, it is not until after several iterative cycles that a user-friendly and technically possible interface takes shape.

I first noticed the problems with these approaches when I started my PhD program. My initial role was of the computer engineer to implement technical solutions for “innovative means of multimodal interaction” (specifically, developing a standalone gesture recognition system that could be worn on a hand in an unobtrusive way), and I would be working closely a multi-disciplinary team of designers, with backgrounds in design, engineering and computer science, all with an interest in exploring different human-centred design approaches.

The project was described as requiring a researcher to

“investigate and design ways of interacting with the information infrastructure that maintain natural social interactions, take advantage of physical space and utilize our extensive human abilities and recognize and manipulate physical objects.”

and my main interest was the use of ubiquitous computing to provide alternative modalities for computer input in an as-yet-unspecified context. I wanted to develop embedded devices that afforded new interaction modalities for computing. I soon found that this required considerations of problems that went beyond recognition algorithms and new hardware. I became involved in methods and ideas completely foreign and new to me, such as ethnography and user participation. At first, I couldn’t see their benefit and felt we were “wasting our time” and should press on at the “true” problem at hand, of building a working system.

In other words, I was doing a pretty great job at fulfilling the stereotype that engineers “don’t play well with others” and tend to exclude non-technical designers. That said, there can also be difficulties in accommodating technical members to a team well-versed with qualitative design practices. On my research team, I tended to retain a deeper technical focus than others I collaborated with. New and unfamiliar concepts that I did not agree with meant there was occasional friction during design activities or when contributing to academic papers, and my technical inclination affected integration within the design team. Ultimately though, it was in understanding and accommodating these different perspectives on design that it became possible for me as a ‘classically’ trained engineer to rethink my contribution and involvement in the design process.

My initiation to both methodological and technical considerations was at one of Jacob Buur’s workshops. Previous to the workshop I thought of design as the implementation of a system to solve a specific problem. My view was that in creating such a system, the problem’s requirements would be defined both abstractly (i.e., as I came to realise, without a holistic consideration of the context of use) and subjectively by engineers, who then set about solving the problem. Buur’s workshop impressed upon me the importance of user engagement and expanding the design requirements based on a detailed consideration of the context of use. During the first workshop, when reviewing videos of design studies, I critiqued the products being presented. Buur critiqued the design process taking place.

(Actually, I should also point out, that although it seems like I’m saying in a fairly self-loathing way “Ugh, engineers never think about the user and that’s a huge mistake,” I am not advocating against pure engineering research. Such research provides technical advancement that plays an invaluable part of design — it is the actual implementation/deployment of new technology that is problematic.)

My initial efforts during my PhD were to appropriate and improve gesture technology (the initial prototype I worked on was designed and built by two friends of mine, Michael Day and Sarah Alexander who did an amazing job cobbling together a portable system using accelerometers for sensors and neural networks for rapid recognition). However my advisor in the meantime was encouraging me to explore different domains for potential new use cases. Even at this point I still had a strong disconnect between the technology and its application. I saw ethnographic studies as something I merely “had to do as part of the research”. I initially did not consider ethnography as part of the design process.  So instead, the first eighteen months of my research were spent learning about neural networks, methods of pattern recognition, and how to interface sensors to learning networks. It became clear to me during this time that the scope of developing a more accurate system would require me to focus on technical breakthroughs and exploring the field of artificial intelligence. However, from my earlier undergraduate studies with handwriting and speech recognition on personal digital assistants, I knew embedded pattern recognition was already a mature field. I had seen firsthand what was possible with existing technology, and observed recognition systems which worked with a high rate of recognition in the laboratory which had not been implemented for a variety of reasons. Knowing this, I changed tack and focussed instead on why these existing systems were not being used and to investigate means of integrating them into a system in a manner that made them both usable and useful. Instead of technical development, I began to focus on what the user required and how their needs could be met with adapting off-the-shelf technology.

In terms of technical maturity, gesture recognition, 8 years after I started my PhD, is still a developing field (with the exception being touch-screen, or two-dimensional gestures, for example the Apple iPhone and iPad), while in comparison, handwriting and speech recognition are both fairly mature in their content and application. While handwriting recognition was not directly useful for the work I was studying, I hoped to incorporate the technical lessons learned for two-dimensional space to three-dimensional space. Speech is almost always used in tandem with gesture by people while communicating, offering a further avenue of recognition for enhancing new systems. Speech recognition can also be deployed to support interaction in an ambient fashion, and is a well studied and technically advanced field of research. Based on its possibilities, I planned to augment the gesture recognition systems I was working on with off-the-shelf speech recognition. The focus of my research switched to creating a ubiquitous computing system to support work practice, while focussing on usability and limiting the time required for technical development.

Through my exposure to and understanding of participatory and user-centred design, I also wanted to develop a system that satisfied the user from a personal and social perspective. The system prototyped needed to be integrated with the practitioner’s work context, while supporting ready appropriation by an individual user (for example, supporting accent, word choice and functional expectations of such a system). I needed to consider localisation of the system, the context it was to be deployed to (to accommodate both the unique challenges of the environment it was used in and the expected interaction paradigm) in addition to the technical challenges faced. I began to realise that while engineers and designers may restrict themselves to a particular field, there was potential for overlap between the two.

Engineering can be seen as the devising and analysis of systems of technical systems to solve problems, while design is the speculative and synthetic process to develop new products and services. Where these meet is in human-centred and participatory approaches which focus on human-experience and acknowledge human agency in human-computer systems. Having to consider such a comprehensive range of constraints affected the design process. I realised it was not merely enough to provide a more technically advanced method of pattern recognition that afforded new interaction paradigms, I also needed to adapt and configure such a system for its context of use. The outcomes of my prototyping and design methodology are discussed in my dissertation. The culmination of these concerns led to my ultimate research question:

How may engineers, designers and practitioners be better involved in and served by a design process for complex information systems that adequately addresses the needs of the practitioner?

(n.b. ‘practitioner’ is participatory design parlance for the end-user)

The answer, in a nutshell was what became a design-manifesto of sorts for me:

“In order to create both usable and useful design it is necessary to respect the tacit knowledge of the user, while using user-focussed design techniques to tailor a system to be its most effective for a particular work context.”

This led to a thesis which ended up exploring the gap between engineering design and human use, and identifying principles for allowing engineers to connect this gap.

I like to think that these principles allow for improving the integration of engineers in a design process which emphasises usability and participant empowerment, and to prove it, I built a prototype system for performing periodontal charting with a group of dentists in Australia and New Zealand.  In the process I discovered a new sense for what was possible with design and the actors within and newfound sense of how important usable design is for people to support what their actions.

I’ll further discuss these methods and how the design proceeded in another essay…

I might try and figure out a way to show this over time.  The Palo Alto/Mountain View runs build up during winter, and then during Summer I do far more runs by the bay (when I don’t need street lights to run after work).

Here’s the trip as a whole:

This is the airport we left from:

Flying down the strip was pretty cool:

The landing site (top down view here) and picnic area:

I was pretty amazed by the terrain data in Google Earth.  It’s been a while since I had a good explore through it (I also noticed a lot more 3D buildings too, kudos Google for all the great recent updates).

Since I moved to the US I’ve really enjoyed having a Skype phone as my main phone at home.  It has been incredibly cheap, and obviated any of the hassles involved with a regular land line (such as installation and the fact you can only make/receive calls in one place!).  However lately I’ve been enjoying Skype video calls.  The last few weeks I have combined Skype with streaming video to watch the game played in heaven, rugby union, with my friends back home.

First I used my Dell M1330 which has a built-in microphone and webcam.  I picked up this laptop for a measly $530 from Dell on sale which was an amazing deal.  Next I used my Mac Mini hooked up to my HDTV to stream the game.  I paid $4.99 to watch the game live from www.mediazone.com and then used VMWare Fusion to allow me to watch the DRMed file using Windows Media Player on OS X.

Minor points that made this experience really great:

• The fullscreen high def video through Skype was excellent quality and not at all choppy.  When the camera was turned towards the game I could see the play perfectly, albeit at 5 frames per second.
• Skype have really improved their echo cancellation technology in the last few releases.  Using the “speaker phone” mode of Skype was great as it was like a virtual conference call.
• Foxtel IQ on my brother’s end allowed him to pause the game enough to get it perfectly in sync, again contributing to the feeling I was there
• Wireless laptops meant the camera could get moved around, and I was able to have individual conversations with people in the room.  If only I had been hooked up via ConnectR… but it was a good approximation

The other great use of Skype video calling was the ability to play Rock Band more collaboratively (the game lacks any method to interact with the other players, not even voice within the game menus!).  Sadly the experience there wasn’t as immersive as there is always a 1 second delay on the audio which is quite distracting.  Otherwise, the future is here!  Who knew when video calling came it would be completely free (aside from equipment)…

My first example is the Nike Plus running kit. How does this fit ubiquitous computing?

1. Embedded, perceptually invisible computing
2. Functionally invisible
3. Accountable
4. Inexpensive

Perceptually Invisible
The Nike Plus running kit has two parts to it.  The first is the receiver that attaches to your iPod.  It is relatively compact, and many armbands and pouches for the iPod have been designed to accommodate it, so it is not noticeable.  The second part is the sensor for your shoe.  If you buy a pair of Nike Plus shoes, it already has a space in the shoe for you to insert your sensor, where you’ll never think about it again.  If you have other shoes, you can buy a “shoe wallet” that holds the sensor.  Again this is unobtrusive, and I never even think about the fact that I have the sensor in my shoe.

One nice touch with regard to the sensor is that they spent a lot of time building smart energy saving routines into it.  This means that although the battery is not user replaceable, it should last years (a new kit can be had for as little as $15 on eBay anyway).

Functionally invisible
Another clever part of the software Nike developed was calibration routines.  Out of the box, the device is fairly accurate, and after a single calibration session, I found it to be within 2% accuracy of my GPS running watch.  What this means is you don’t even notice the fact it is a pedometer – to the user it just works, and gives you your distance.

Another way in which the Nike Plus kit is functionally invisible is its integration with the iPod and iTunes.  I use my iPod as I normally would while running.  When I sync my songs on iTunes, in the background it uploads and formats my run information to the Nike Plus website.  If I am entered in any competitions it updates those.  I choose when I want to look at my stats, and I never have to think about collating them or accessing them (unlike Garmin’s efforts).

Accountable
By “accountable” I’m referring to a property of design which I defined in my thesis.  For a design to be accountable, it appropriately presents information about itself and how it works to the user.  One difficulty in ubiquitous computing is providing embedded computing that automagically performs actions, while not obscuring how it works to the user.  Users rarely use designs in the way the designer intended, and by exposing how something functionally works you can assist the user’s understanding of the design, and assist with their appropriation.

The Nike running kit is very open with how it works (an accelerometer and wireless system), provides the data in an easy to read format (it has already been used in many academic projects), and allows for calibration of the sensor (while not requiring it).

Inexpensive
This is simply – the kit costs only $30 SRP, but can be had for far cheaper on eBay or sites like Eastbay.  It is amazing how many people I know (I can think of four in my extended family alone!) who have bought the kit after finding out how cheap it is.

But the real question is – is it useful and does it work?  I was talking with a colleague yesterday about his running, and he was saying that after a coming competition was over, he knew his motivation would flag.  Since using the iPod running kit, I found my running increased by a factor of 4 thanks to the competitive motivation.  I get to “go running” with my brother who lives in Australia.  Everyone I know who has one is still using it.  I’m here writing about it on my  blog!  That to me seems like success.

Link

For those of you who can’t watch the video, it’s of a new type of projector calibration.  By embedding fiber optic sensors into the edges of an object, any standard commercial projector can then be automatically calibrated to perfectly project an image onto that object.  It is quite amazing to watch.

Normally I’m a philosophy-first ubicomp kind of guy, and prefer projects that focus on the human effects of ubiquitous computing.  However, I’m not an idealist and I realise that technical innovation is a fundamental requirement of the field.  However I still believe some of the best technical achievements are in reusing existing technology in novel ways.  This is a perfect example of this.  In particular, there are three things that I think are done right:

1. Keep the functionality simple
2. Keep the technology smart but simple
3. Use off-the-shelf-technology

First of all, they focused on a single problem at hand.  Achieving computing potential embedded invisibly still requires a means to interact with that potential.  Finding new ways of getting information displayed on everyday objects is a huge step forward, and previously was a pretty hard task.  It required custom screens, or complex manual configuration.  Solving a single problem provides a design pattern for others to use and extend upon (and then worry about the user experience).

Secondly, the technology itself is simple.  Fibre optic sensors mean the system should be robust and cheap.  There are not a lot of different sensors which could break, nor is there a complex system with fragile dependencies.  Most of the magic is done in the software which allows for further improvement and customization, as seen by the later project:

Link

Finally, the technology is off-the-shelf.  This project used a single embedded chip, plus a regular projector and some custom software.  Sounds like something both easy to hack up yourself, and to commercialize for other people.  This is a very nice comparison to a system like Microsoft Surface, which is full of proprietary components.

This functional and technical simplicity in turn achieves two things.  One – it means the technology itself is cheap, and two, it is reproducible.  Ubicomp needs to drastically lower the cost of entry to continue rapid expansion and adoption.

With micro-projectors becoming more popular, I’m really looking forward to commercial implementation of such a system.   While there are some shortcomings (such as the brightness of the projected image), this is still a lot more immersive than fiducial markers.  This is the exact type of technology needed to allow ubiquitous computing to be useful to mainstream, commercial applications.