Somewhere between 1.7% and 43.5% of success is just showing up
Inspired by this quickie analysis in which Fabio Rojas finds that 27% of variance in exam grades is explained simply by class attendance, and also by the recent end of my semester, I decided to take a look at how attendance affects grades in my big undergrad class. His class is on social theory, and I assume it’s just one thing; mine, by contrast, is a graphic design class made up of split but related lecture and lab components. I have a formal attendance score for lab, but only a proxy for lecture — how many quizzes they took.
The results? Pre-midterm quiz-taking has no significant relationship with scores on the midterm, which covers only lecture material; it accounts for only 1.7% of variance. On the other hand, lab attendance is hugely predictive of total points earned on lab projects, accounting for 43.5% of variance. I’m not terribly surprised by this, since lecture material is much easier to get outside of class than lab material is (primarily from the textbook), and because quiz-taking is an imperfect proxy than only takes into account who showed up at the beginning of class on quiz days. The crummy nature of the variable is somewhat confirmed by the fact that lab attendance is a significant predictor of midterm score, accounting for 7.2% of variance.
Filed: Leave Them Kids Alone || 12:53, December 20 || View Comments
How is TV formed?
Of the pre-release excerpts from Walter Isaacson’s biography of Steve Jobs, the one that intrigued me the most was the one that related Jobs having “finally cracked” the next-generation television set. It was tightly related to the then-upcoming iCloud, but this bit of narration seems like the money quote: “No longer would users have to fiddle with complex remotes for DVD players and cable channels.”
Now rumors are rampant that Apple will soon unveil its TV, most recently suggesting it will come in three sizes (32″ at the low end, 55″ at the high end) and that the next iMac will bridge the gap by offering TV functionality. Key to all the rumors is the clear notion that Apple’s television set, unlike the extant Apple TV, will be an all-in-one set, not a set-top box. Jobs seemed pretty set in his belief that a set-top box wouldn’t work, even though one could do all the functionality he envisioned, and that Apple already sells a set-top box.
For a couple reasons, I’m not sure what to make of this. First, most households already have at least a couple of set-top boxes — the cable/satellite receiver and a video game console, which doubles as a media player. My living room has three set-top boxes (a DirecTV receiver, a PlayStation 3 and a Wii), and that’s not counting the VGA cable that sometimes connects my laptop to the TV. The second issue follows directly from this separation of functionality — I need to be able to update or replace my TV distinctly from my set-top boxes, or the entire enterprise becomes cost-prohibitive. The all-in-one model that was attractive to computer consumers who flocked to the iMac shouldn’t look nearly as viable in a market a) full of people who are already comfortable plugging cables into their TVs, and b) lacking in big educational customers who want all-in-one devices for ease of maintenance.
At the same time, Jobs was also thinking about interface complexity — getting rid of all your complicated remotes. Part of this is about getting Siri on your TV and using voice commands to control everything, but I’m pretty skeptical that most TV consumers will want to do that. Imagine that you just want to channel-surf on a Saturday afternoon; do you want to keep saying, “next channel,” until you find something worth stopping for? Or you’re flipping through several football games to follow your fantasy team; is repeatedly calling out channel numbers an attractive control scheme?
Of course, none of this takes content availability into account. I really wonder whether the success Jobs had manhandling the big record labels left him overconfident in his ability to bend TV and movie studios to his will. The reason why most people have multiple set-top boxes is that they want access to multiple, exclusive content libraries. I want a broad selection of channels, including premium channels, to be available to me live; I need a cable or satellite receiver. I want to play Little Big Planet; I need a PlayStation 3. I want to play Super Mario Galaxy; I need a Wii. For other things I have other choices — if I want to play Blu-rays or watch Netflix, I can use my PS3 rather than add another device — but even the hyper-converged, iCloud-driven Apple TV isn’t going to get rid of the basic functionality of my existing equipment. So I’m still left wondering — unlike with the iPod, iPhone or iPad, but much like the current Apple TV — what the point of this device is. Voice commands and better user interfaces can be added to any existing box, and streaming video will be a central component of every TV device that’s ever designed from here on out. What benefit do I get from buying a TV that’s hardwired into that content control functionality?
Filed: aka Syscrusher || 15:50, December 9 || View Comments
Sports, perception and sample-size bias
New York Rangers goalie Henrik Lundqvist is the Tim Tebow of hockey. Actually, he's even better, having won nine of his last ten. Photo by Robert Kowal.
Apologies in advance, but if you don’t follow sports this post may not make much sense.
This afternoon, the Denver Broncos picked up their sixth win in seven games this season with Tim Tebow starting at quarterback. If you haven’t heard, Tebow has what might be called non-traditional passing mechanics, but as many commentators have noted, he “just wins.” There’s a lot that could be said, and already has been said, about the strange way in which quarterbacks are credited with team success in football, but that’s not really the point of this post. Rather, I want to point out how odd it is that seven games — even seven games that include six wins — can be considered so meaningful in football.
This stretch has turned Denver’s season around, to be sure. They lost four of their first five, but now find themselves tied for the lead in their division. But this is only possible because of the NFL’s relatively tiny schedule. Consider that, for a hockey goaltender — probably the only every-game player in North American major team sports with as much impact as a quarterback — six wins in seven games is barely noticeable; it’s less than a tenth of the season. For a baseball player (where there isn’t such a great analogue, since starting pitchers only go every fifth game), six wins in seven games is a good week. You could win your league’s player of the week award in May and be sent down to the minors in June. For Tebow, six wins in seven games is two months and half the season, and it’s especially significant when one of your divisional rivals (San Diego) is imploding at the same time.
But this is all perception; if we’re trying to think about what this seven-game sample means in terms of predicting the larger population of games that is a player’s career, seven tells us nothing. It doesn’t matter that an NFL season is only 16 games long; seven games don’t provide enough observations to reduce the error to an acceptable level. If we were to look at a proportionally similar number of baseball games — 70 — we’d keep the proportion the same, but reduce the sampling error by examining nominally more cases.
So where does this perception error come from? Is it just the kind of rank innumeracy we see in many contexts? Maybe, but I suspect there’s also an important media effect as well. Sports media — both reporters and game broadcasters — and the sports culture they’re embedded in frequently express hostility toward data-driven strategy. Narratives and tradition rule in sports, and when data contradict them it’s because data can’t possibly figure out the relevant “intangibles.” Noting that a seven-game span isn’t really an illuminating sample gets in the way of a lot of narrative structure.
Filed: Science Is Real || 21:49, December 4 || View Comments
Who cues?
At the MAPOR conference last weekend, I presented a study on how partisan media strengthens belief gaps. The belief gap idea, first identified by Doug Hindman a couple years ago, is an extension of the knowledge gap, a theory with over 40 years of work behind it. Whereas the knowledge gap hypothesis suggests that education predicts differential gains of knowledge about political issues — everyone learns, but high-education people learn more, creating a widening gap — the belief gap hypothesis suggests that ideology is better than education for such predictions. It’s called the “belief” gap because the conceptualization of beliefs better fits the context in which facts and knowledge are politically contested.
My paper (with students Delwar Hossain and Ben Lyons) took the initial findings and expanded them in three ways. First, we examined partisanship in addition to ideology and found that it’s consistently a better predictor of beliefs. We attribute this to both being essentially markers of group affiliation, but partisanship being a clearer one for both researchers and survey respondents. Ideology has long been conceptualized as a coherent belief system that drives opinion-formation, but most research suggests few people actually have this kind of formal ideology. Instead, we use cues from elites to guide our opinions, attitudes and beliefs.
Second, we examined the role of partisan traditional and social media in the belief gap process. Despite concern that social media are politically polarizing and insular, we found that partisan traditional media are far stronger drivers of partisan beliefs. There is a structural explanation for this — cable TV and radio have far larger audiences than do blogs and pundits’ social media outlets — as well as a psychological one — we’re exposed to more elite opinion through these outlets, whether those elites are elected officials or opinion-leading commentators.
Finally, we examined belief gaps in five issues — two science-related issues that had previously been studied by Hindman (climate change and abstinence-only sex education), two evidence-free rumors about President Obama (he’s a Muslim, he was born outside the U.S.) and one factual economic issue (whether most Americans’ taxes have gone up during the Obama Administration). Each of these issues has a correct answer by consensus of relevant authorities, but each is also highly politicized. We found belief gaps for each, with largely similar patterns of partisan media influence.
I lay all this out because thinking about our findings in the context of the other presentations in the belief gap panel — from Hindman; Ken Blake and Misa Culley; and Rob Daves, Allen White and Stephen Everett — led me to thinking a lot about the broader, more abstract facets of this idea.1 To my mind, there are two big questions to be answered. First, we need to think about what things a person can have “beliefs” about. During the panel, Rob Daves talked about “verifiable” issues and referenced the work of Cecilie Gaziano in this area, but I think we can think of this in cognitive terms. Given that the belief gap idea grow out of the knowledge gap, I suggest that we look towards the cognitive structure of knowledge to understand what we mean by “belief.” Presumably we are thinking of issues about which the believer can feel that their beliefs are “correct,” even if all evidence and authoritative consensus suggests otherwise, even if there is no consensus to draw on, and even if the answer exists but is unknowable. We may further want to separate issues that are retrospective (about which verification may already be possible), prospective (about which verification can’t be done yet) and ongoing (about which verification may be ephemeral or in constant dispute). These orthogonal issue dimensions would co-exist with the dimension already in use in existing research, politicization.2 The typology might look something like this — consistent with a seat-of-the-pants typology, the examples are the results of just some quick thinking on this and may not fit all that well:
| Politicized | Non-politicized | |||||
| Retrospective | Prospective | Ongoing | Retrospective | Prospective | Ongoing | |
| Consensus | Obama born in U.S. | Global temperatures will rise | Climate change | Lincoln killed by Booth | Vaccines and autism | |
| Disputed | Roe v. Wade lowered crime rate | Economy will improve next year | Gun ownership and safety | JFK killed by Oswald | ||
| Unknowable | 2000 election stolen | Jesus will return someday | Alien life exists | |||
If we believe that the process observed in the belief gap phenomenon is one of elite cuing by like-minded political leaders (consistent with the work of, e.g., John Zaller), the next question is who does the cuing across the range of this issue typology. For politicized issues, we’ve got a pretty strong hypothesis that political elites provide the most relevant cues, but who those elites are might vary by issues. Particularly for issues that are politicized along evangelical/non-evangelical religious lines, we might expect to see different people and sources playing important roles in mass-opinion formation. Maybe economic, defense and science issues all have different arrays of influential elites; still, we’re probably talking about a relatively narrow band of elites that cue beliefs across a lot of political issues.
But what about beliefs for which elite political cues are not relevant? With the possible exception of Michele Bachmann, nobody’s politicizing childhood vaccinations. So who cues beliefs about vaccinations? Is it scientific consensus (as reported by news media)? Jenny McCarthy? Oprah? If we can explain how non-political beliefs are cued, we may go a long way toward identifying the underlying cognitive and social psychological processes of political belief formation.
1. I should also acknowledge the suggestions of several members of the SIUC political science department (particularly Tobin Grant and Scott McClurg) during a preliminary presentation of this work, which I subsequently incorporated into the final product, and which have informed my ongoing thinking about this topic.
2. There’s another wrinkle here, which is the concept of issue domains and the cognitive work that goes into connecting our attitudes on related issues. For example, in the data used in our paper above, beliefs that tax cuts encourage job creation and that federal deficits discourage job creation were strongly correlated, even though tax cuts help to increase deficits. Additionally, our respondents also anticipate strong inflation over the next year, even though we’ve been in a period of historically low inflation during the global recession. What it looks like is that, instead of considering each issue on its own, there’s a relationship between all these economic issues and general economic attitudes — that is, the economy is bad, and inflation is bad, so we’re in an inflationary period. Job creation is good and tax cuts are good, so they must go together. Probably relevant, but also probably not worth getting into until the first level of questions have been worked out.
Filed: Science Is Real || 20:11, November 21 || View Comments
Gordian knots in research methods
Whenever I have a project in mind that involves Facebook, there’s a methodological stumbling block that almost always comes up: Most of what’s interesting isn’t accessible unless you are friends with the people you’re trying to study. So maybe you rework the research questions, or you come up with a way to address them using survey data, etc.
But now I see that I was overlooking the obvious solution: Just create fake profiles to friend people with, as a group of four researchers at the University of British Columbia did. For them, it was entirely necessary, as they were studying the vulnerability of online social networks to malicious bots, so they basically created their own benign bots and observed what they accomplished. The very first phase resulted in about a 20% friend-acceptance rate, so if you’ve got a good sampling method, this is looking decent enough as a way of getting real, live Facebook content.
Filed: Science Is Real || 23:10, November 6 || View Comments
Open tabs, October 21
- The study I’m presenting at MAPOR next month does a bit of work examining the differences and overlaps between partisanship and ideology, so I was quite interested to read Peter Frase’s essay on how the 1% and the 99% orient toward these concepts. My only quibble with his take is that he suggests that young adults will have an experience with politics “in which ‘left versus right’ is used interchangeably with ‘Democrats versus Republicans.’” It’s not just the young people — ideology, for all that it might be in play among political and financial elites, is not something that most people have beyond the group-affiliation marker of “liberal,” “moderate” or “conservative.” If we consider ideology to be a set of organizing principles from which we derive our decisions and opinions about political issues, research suggests only about 5% of Americans have one, if that.
- Paul Waldman’s got a really incisive post on the way the culture wars have silently expanded to consume the entire range of political issues. It’s an extension of that ideological group membership phenomenon, and it’s why one of the results I’ll present from that paper is a relationship between Christian media use and disbelief in anthropomorphic climate change. Ideological and partisan groups don’t have perfect overlap, but they each have sets of beliefs that are required of members and occasionally contested, requiring members to shift when called upon.
- Matthew Yglesias makes note of the structural changes in the blogosphere — that it’s really not about outsiders anymore, as most of the outsiders have been brought mostly into the mainstream media-political tent — and his place in it.
- How many photos have ever been taken? An analysis by 1000memories suggests the answer is about 3.5 trillion; 140 billion are hosted by Facebook.
Filed: Open Tabs || 22:54, October 21 || View Comments
Crazy different
I don’t really have anything of value to add to all the commentary on the death of Steve Jobs, other than to note that I don’t think its volume can be attributed solely to the fact that everyone knew it was coming. Though his revolutionary work lasted three and a half decades, he really became a titanic figure in the last 15 or so years. Calling him the 21st century’s Edison is probably not overselling his impact and stature.
My first computer was an Apple //c like the one seen here. Remember the closed Apple key? Good times. Photo by Mike Maginnis.
What struck me reading the MetaFilter obituary thread was how many people’s experience mirrored mine — starting with a classroom Apple ][ Plus running BASIC, secretly coveting your friend’s IIgs, eventually moving into the more open design and development world of the early Macs — and how many didn’t. There was something really communal in reading people describe the same path, but starting with an iMac G3 and ending with a MacBook Pro and an iPad. As tough as it is to measure the impact of Apple (and Pixar, let’s not forget) on society in general, it may be even tougher to figure at the individual level. There are whole industries full of people whose lives have been altered by Jobs and his determination to pursue the path as he saw it.
And it probably won’t get much press this week, but Jobs credits that vision to his 70s-era use of LSD and other psychedelics, which he called “one of the two or three most important things” he had done. In a very real way, the revolutionary promise of the counterculture was finally brought to bear by Jobs and his unusual and uncanny ability to understand how people relate through information. It’s no accident that Jobs referred to the original Mac as “insanely great” or that the first ad campaign after his return to Apple was the “Think Different” series, featuring a monologue beginning, “Here’s to the crazy ones.” His legacy is one of having the foresight to understand what most of us could only grasp looking back, and of having his contemporaries perched on his rear-view mirror, desperate to figure it out for themselves.
Filed: aka Syscrusher || 16:03, October 6 || View Comments
Civility out of context
I fielded a survey recently, and this out of context result really intrigues me. For the non-quantitative people, these are regression results that show how each variable predicts the outcome variable with all the others controlled. The outcome here is agreement with the statement, “When most Americans debate issues facing the country, they are more civil today compared to ten years ago.” I’m looking at blog use, ideology and partisanship in this study, and here are the predictive results (the ones with the footnote symbols are statistically significant):
| Beta | |
| Gender (Female) | .03 |
| Age | -.05 |
| Race (White) | -.19*** |
| Education | -.11* |
| Income | .01 |
| Partisanship (Republican) | .10† |
| Ideology (Conservative) | -.04 |
| Conservative Blog Use | .23** |
| Liberal Blog Use | .03 |
| *** p < .001, ** p < .01, * p < .05, † p < .1 | |
I’m pretty sure that the partisanship and conservative blog use results are manifestations of those individuals remembering 2001 as a time when everybody was being so mean to George W. Bush all the time. It’s likely also the reverse — Democrats seeing the current environment as severely uncivil — but the distribution of the blog use data suggests to me it’s more the former than the latter. More on this at MAPOR next month.
Filed: Science Is Real || 19:53, October 1 || View Comments
Peer-to-peer review
I’m in my tenth year as an academic, which means that my view of how things worked in the world of research pre-2002 is based purely on output — that is, the papers and books I’ve read that were published before my time. As a result, I really don’t know if the phenomenon I’ve been noticing lately is new or not, but man, there are unpublished papers everywhere. Conference papers, working papers, executive reports, etc., are posted all over scholars’ web sites, their academia.edu profiles, departmental sites, and then frequently logged in Google Scholar and/or touted in news releases. They may or may not make an academic splash, though there are certainly a number of recent conference papers in political science and mass comm that used the web to gain a lot of notice, but news organizations and commentators don’t operate with the same filters as academics do. This University of Washington study (including one author who is a former colleague of mine at Wisconsin) is a good example. It was linked by Talking Points Memo, in a way that seems fairly typical of how unpublished work is disseminated through blogs and other online political commentary. Its conclusion makes the paper appealing, as it seems to inject some empirical evidence into the debate over whether “Twitter revolutions” really have anything to do with Twitter, which has already gone back and forth in the press.
At the same time, it’s a study that hasn’t gone through the peer review process. Maybe it’s applying theory in an unconventional way, or maybe there’s something odd about the data, or maybe it’s exactly right (and I should note I’m only highlighting this study because it’s the most recent example I’ve come across — there are dozens more). But the widespread availability of unreviewed research presents a twist to the science news cycle model, which relies on carefully considered and reviewed conclusions to be reined in. If we’re inserting more and more research from earlier stages of the process into our discussions of public policy, current events, etc., we could see major challenges to the peer review model.