Today was day 1 of my trip to the SigCHI 2007 conference in San Jose. This is my first time attending this conference, and so far I've been pleasantly surprised. The conference organizers are doing a good job of balancing Academic, Design, and Business interests at the show.

Usability & Product Development: an Introductory Usability Course for Management

A few interesting stats:

• 63% of projects exceed cost estimates


• the average project is 189% over budget


• the average project is 222% behind schedule


• the average project has only 61% of specified features


• 50% of the development time and money is spent in user interfacec design and development


• most flaws originate in product definition


• 80% of software costs occur after the release

Top reasons for cancelling projects:

• incomplete user requirements
  
• lack of user involvement

Top reasons for exceeding cost estimates:

• user requests for changes


• overlooked user tasks


• users' lack of understanding of own requirements, etc.

Who Killed Design

• There are no tech companies that have a track record for new product development
• ”The only designers worth studying are the repeat offenders.” (This one is less impressive as a quote, but extremely compelling as an idea… more later…)
• ”Designers suck at understanding themselves.” (and especially the process of design)
• ”The era of the Renaissance Man is dead. But the Renaissance itself has forever changed the process of Design.”

There were many more, but I’ll have to have some time to pull them from my notes…

A few other provocative tid-bits (not sure who said them, but they’re worth recording)…

• Design - moving from an existing circumstance to a preferred one.


• How many designers can recognize that the problem they are addressing has no “solution?” How many of those who can know what to do in that circumstance?


• If you’re interested in sustainability, you want the fur coat and not the Mac.

I also attended some paper sessions on Navigation and Expert/Novice studies (separate talks). Not much to new to report.

More tomorrow…

SigCHI 2007 – day 2

The first session I attended today was a talk about “Programming by Professionals.” The first paper, presented by Cherubini, was about how developers use whiteboards to communicate. Although one audience member rightly pointed out that this study was somewhat limited in that is only utilized employees of Microsoft, some of the conclusions by the author are interesting.

For example, the author suggests that there are 9 primary types of use of whiteboard diagrams by developers:

1. understanding existing code
2. ad-hoc meeting
3. design / re-factoring
4. design review
5. onboarding – or bringing a new person up-to-speed
6. explaining to a secondary stakeholder
7. explaining to customers
8. hallway art
9. technical documentation

A few other general observations from this talk:

• Each type of use had different levels of formality and need for accuracy/precision.
• They also suggested that the diagrams were less important than the conversations around them – which makes sense.
• Developers did not follow any graphical standard in their diagrams (such as UML)
• Many design decisions are made in 1-to-1 meetings
• Engineers need microscopic and macroscopic views of the code. Zooming between these levels happens extremely fluidly in these whiteboard discussions
• Some plasticity in the drawing tool is required (to support the flexibility of the conversation)
• Architects tend to keep many versions of drawings/plans even as they evolve. The metamorphosis of a plan or design is important to capture both so that everyone understands “how we got here” and because it can be used post-hoc to help the designer better understand and improve their own process. Programmers, however, can express themselves in code – a process which in-and-of itself does not lend itself to plastic “redrawing.” This suggests that having a rapid prototyping process in an organization can capture alternate approaches early, as well as allow designers/engineers to express themselves in a manner appropriate to their work.

Usable Tools to Support Code Refactoring

• Error messages are often used to help a developer systematically review affected spots in the code after a refactoring gesture is made. In other words, developers don’t seem to sit around and think… I wonder where all this change might cascade or how many instances of this wildcard I have used in the application. Instead, they make the change, and then use the error messages to get a feel for the reach of the change. In Eclipse this is relatively efficient, since you can get code errors and warnings without the need for a full re-compile.
• End programmers are not compiler experts; they don’t see their programs as abstract syntax trees. Tools to support their refactoring have to support the way they visualize their code/application

…. Games Panel writeup forthcoming ….

There was a Games panel on Tuesday that had some interesting and solid papers. The first was a paper from the Project Massive (of projectmassive.com) researcher, regarding they motivations for game play in the context of the "gaming addiction" debate. The talk started poorly, by quoting gaming industry statistics on the demographics of game players -- "look... it's not just 15 year-old boys anymore....blah blah blah." For the record, there is NO good reason to trust these numbers, especially since the methods AND actual detailed stats are proprietary. I once tried to buy data or reports (or any information) from NPD Funworld as an academic, and they wouldn't even talk to me about the possibility for the very reason that I was an academic.

Can we please, as a Games Research community, stop regurgitating these numbers as if they are scientific fact?

Anyway -- I got sidetracked -- despite touching a nerve with me in the first few slides, this talk had a lot of interesting numbers and a rather solid experimental approach. The N in this study was VERY large (something like 500 subjects from an original sample of nearly 5000 participated in multiple waves), and the battery of survey questions given to each participant was impressive. I would LOVE to dig into this amount and quality of data -- kudos to project massive on this front.

The one caveat I'll point out was raised by a commenter during Q&A, namely that the study or followups would benefit from using more than self-report measures of amount of game play and its effect on quality of life, work, social life, etc. Surveys and/or digests from family and friends might have revealed a slightly more nuanced picture about the "appropriateness" of the amount of game playing on the subject in question.

The Life and Death of Online Gaming Communities
Bravo. This was another fascinating study of World of Warcraft players from Duchenault et al. at PARC and Stanford. I highly recommend taking at look at this paper. The approach they use if technically not all that difficult -- extending the WoW client software (which has an open API) to dump simple /who data is just elegant. I'm curious about if this type of automated data mining is actually allowed by the eula of the client API, as I know that some open client software (like Second Life's, I believe) has clauses disallowing the use of customized client software for mining of data that is considered proprietary.

My main concern with this presentation was that the authors wanted to generalize their findings about the average size and characteristics of successful (meaning long-lasting) guilds to other types of organizations. There are MANY formal factors influencing the scale of coordination in WoW that may or may not translate to different interfaces, organizations, and most importantly -- different TASKS. It is reasonable to assume that a task that does not require or favor a team/group coordination would not elicit group clustering, loyalty, etc. like an RPG game does. For comparison's-sake it would be useful to look at something like Second Life social networks in order to try and infer guild-like clustering of communication and behavior.

Ongoing work... thought I would post an updated screenshot...

Jeff Veen formerly of Adaptive Path -- now Google -- giving a talk about Web 2.0. Veen had a greatslideshow that he's been generous enough to post on his personal website (here). In this talk, he covers historical precursors to the boom-bust cycles we seem to be experiencing in the tech world over the past twenty years or so. The comparison to the tulip frenzy in Holland in the mid 17th century (Tulip Craze) was interesting but not altogether enlightening about the state of the current tech industry. Perhaps it was just that he didn't have enough time to fully develop this thesis during the talk. I appreciate the sentiment that there's not much new about "new media," but I didn't feel that pointing out market frenzies of yore (another example he cites is the steam engine) lends itself to a precise historical comparison with the state of tech markets today.

Veen argued that one of the major themes of "Web 2.0" is that of giving up control. By this I believe he is referring to the nature of interactive media that gives users control over their own data. Some examples include Digg, MySpace, Flickr, YouTube, del.icious, etc. In each case, the traditional hierarchical forms of data have given way to tagging and categorization of data based on social attributes / markup. He argues that the key design concerns are how to blend editorial control and structure along with rich mechanisms for user participation.

A few provocative items from this talk:

• Build Trust -- users are peers
• 1/20th of a second is all it takes for users to develop an opinion about your brand/site's trustworthiness
• In the era of tagging, curation is done by algorithms

Although the talk was very engaging, and the slides were especially well done, I was left wishing for a little more precision of terms and academic rigour. All in all it felt more like a pump-you-up keynote than a critical examination of whatever "Web 2.0" has come to mean.

In my opinion, the term "Web 2.0" lost its meaning long ago and should be put to rest like so many buzzwords from the late 90's.

My latest update adds a grid analysis to the player navigation, so you can calculate stats about amount of the game map explored, hot-spots or areas of "camping," etc. It's also a precursor to doing flow analysis -- which looks a lot like a vector field. I'm not the first person to do this sort of analysis, but I have a few ideas up my sleeve for how to extend this approach into uncharted territory (pardon the pun).

Interesting interview with Jakob Nielsen (of Nielsen-Norman Group) about Usability.

here

more...

Added:

• grid 'time in cell' histogram
• time vs. sample tally metrics on grid
• additional grid-based stats
• ability to export stats to CSV

This treemap is truly remarkable. Changed my understanding of the casualties of this war -- I won't soon forget it.