« Attention through Status Message Update (SMU) | Main | Persuasion through Status Message Update "SMU" on Facebook »
May 20, 2008
My response to a PhD proposal
A colleague from another country asked me to review and respond to a student's dissertation proposal. Overall, it was well done. The student showed strong skills in many areas.
But they wanted feedback to make the proposal and research better. So I typed out a variety of my responses. Below are excerpts.
(I believe I've removed all the personally identifying markers)
This is an unusual type of blog post for me. See what you think.
--BJ Fogg
---------------------------
-----Focus on an EASY behavior change
The proposal uses exercise as the DV, the behavior change of interest. I would suggest picking something easier, a behavior that's fairly easy to achieve. Exercise is not one of those. (I realize the program and school may have reasons for picking exercise, and previous work may suggest this. One good thing about being an outsider in viewing this proposal is that those things don't enter into my evaluation.)
Why not exercise? It's tough to get people to do this. They don't like sweating, it takes time, they've been prompted to do this for ages. People are probably resistant to new appeals to exercise.
Instead, I would pick something simple, like writing for 5 minutes in a journal each day, or texting a friend a compliment each day.
The thesis is not about exercise. It's about principles and systems that lead to behavior change. If the behavior change is too demanding, then you won't be able to see the effects of the variables. Let me push this to extremes . . . imagine the student was proposing a study where donating $1,000,000 was the target behavior. The student would do the research -- perhaps great research -- and find no impact. The conclusion would then be "arguments systems don't work" or "characters don't work." The problem, of course, is that the behavior is exceptionally hard to achieve.
If you pick an easy behavior to change (like 5 minutes of journal writing each day), then the variable effects of the experiments are more likely to show up. Once you learn about which principles are most powerful, THEN you can apply those principles to tougher behaviors, if you want to have impact in health change.
-----Focus on real outcomes, not proximal measures
The proposal has quite a few measures that are NOT directly assessing attitude or behavior changes. They are proximal or mediating steps to persuasion. I would try to measure persuasion (attitude & behavior change) and reduce the focus on mediating steps to persuasion.
The last part of the proposal worries me because I see various measure proposed and none are measuring persuasion (attitude or behavior change), just approximations or mediating steps to persuasion (like moving forward on the stages of change).
I suggest taking a look at all the proposed measures. Do away with those that are intervening steps to persuasion and measure persuasion itself -- attitudes via self-report, behaviors via record keeping, self-report or direct observations.
-----Don't assume previous research was well done
Personally, I pay attention only to research that works, not to studies that don't work. Studies fail for many reasons, not just because the concepts were wrong. Often the stimuli is quite bad. For example, in testing the persuasive impact of video, a researcher may have created a poor video. In looking at the persuasive effect of images on web sites, the researchers may have chosen corny photos that are not credible (of the 16 in the proposal, some of those photos look fake -- a marketing photo done by actors, not real people). It takes a great designer to make an effective persuasive technology. Most researchers are not great designers. It's a problem to draw general conclusions based on stimuli created by researchers.
(I like that the proposal talks about pre-testing the images and such. This is very important. In fact, the most important testing in persuasion experiments happens before the actual experiment begins. You need to make sure your stimuli has impact with your audience.)
I assume that most studies that fail were poorly designed, operationalized, or executed. I don't rule out possibilities based on failed studies.
The proposal seems to take all research findings at face value, weighing the failures against the successes. If we took this approach to creating new businesses, we would overwhelmingly conclude that businesses don't work. Well, they do. It's just that most people don't know how to run businesses well.
Lots of the research cited in the proposal focused on difficult behavior changes. I would include these studies in the lit review, but I would not let failures here sway research. I think we're early enough in persuasive tech that it's hard to do good experiments in this area. The operationalization and measurement issues are difficult. Build and focus on studies that have worked. Assume most failures were the result of problems in the research method, not in the concept. Again, there are many reasons a study fails to show results.
-----Comparing Genres is really hard to do well. I advise against it
For years I've spoken against studies that attempt to compare the persuasive impact of different genres. For example, some try to compare the impact of video against web sites. The problem is one of media sampling. What video do you select? What web site? What if you select a great video and a crummy web site? Then the video wins. What if you do the opposite? Then the web site wins. How do you select equivalent media in each genre? I believe you cannot. That's why these studies will always be flawed. (You can select the "best" example in each genre, but then the study is about the "state of the art" and not some generalizable principle)
In the same way I'm not optimistic about trying to compare genres of persuasion, such as advice versus argumentation. What advice system will you choose? What argumentation system? It poses the same problem as video versus web site. The study results will hinge mostly on the media samples you choose or create. Often the results will be misleading because people will generalize your findings widely, well beyond the scope of the research performed.
Comparing features within a genre are good: multiple photos, depth of argument, tailoring of message, etc.
------Getting clear on Motivation versus Ability
The student should get clear about the roles of motivation and ability in behavior change. Certain studies and variables focus on increasing motivation as a step toward behavior change. Others focus on ability (meaning either training/education or reducing barriers).
I will be speaking about this important conceptual distinction the at the doctoral consortium at Persuasive 08. I won't type it all here, hoping the student will be at the DC. The past research reviewed and the proposed research will be much clearer when viewed from a behavior change framework that unpacks motivation and ability. More in Oulu.
Posted by BJ Fogg at May 20, 2008 12:04 PM