Archive | Uncategorized RSS feed for this section

Research ethics: Gender as a variable in NLP

4 Apr

I’ve previously posted on the talk I’m giving today (April 4) at EACL in Valencia. This post provides the slides with the accompanying notes. (If you are reading this before US Eastern bedtime on April 4, it may not be the final version, as I’m editing the slides while watching the earlier presentations during the day.) Here, again, for your reading pleasure, is the abstract of the paper I submitted at EACL, but note that the last couple slides go far afield of the specific paper…

Researchers in natural-language processing (NLP) and related fields should at- tend to ethical principles in study design, ascription of categories/variables to study participants, and reporting of findings or results. This article offers an ethical framework for using gender as a variable in NLP studies and proposes four guidelines for researchers and practitioners. The principles outlined here should guide practitioners, researchers, and peer reviewers, and they may be applicable to other social categories, such as race, applied to human beings connected to NLP research.

Pre-print of the EACL paper also appears on the previous post.

I welcome your thoughts.



“Gender as a variable in writing studies: Ethics and methodology” at WRAB in Bogotá

27 Nov

Organizers of the Writing Research Across Borders (WRAB) IV conference in Bogotá Colombia in February 2017 accepted my proposal to present a paper. I’ve paid my registration and booked my travel. I intend to have a near-final draft of a journal article ready for the conference; I hope to make a few final edits if I get good feedback there and ship it off the week after I return. I’m also looking forward to a couple days of kicking around with my spouse in Bogotá, which looks like an amazing city.

Here is the abstract for my paper:

Gender as a variable in writing studies: Ethics and methodology

This presentation uses results of a study where participants identified their own genders to illustrate ethical and methodological problems. It makes normative claims about gender as a variable in studies of written communication, including composition studies, technical and computer-mediated communication, and natural language processing.

Theories of gender and communication include early gender-difference/dominance views, social role theory, standpoint theory, and queer theory. Nevertheless, empirical researchers often use gender as a variable without explaining how they ascribe it to participants or what they intend it to mean. For example, Tebeaux and Allen performed studies in technical communication with gender as a variable but without explaining how they assigned this category to participants. Herring and Paolillo assigned author-gender labels using a qualitative heuristic. Yan and Yan and Rao and colleagues used automated heuristics to code author gender.

I argue on ethical grounds that (1) researchers should avoid using gender as a variable in their work unless it is necessary to answer their research questions; (2) researchers using gender as a variable should make explicit their methods for assigning gender categories; and (3) researchers should respect difficulties of research participants when asking them to self-identify for gender.

Works cited

Allen, Jo. “Women and Authority in…Communication Scholarship….” Technical Communication Quarterly 3.3 (1994): 271.

Herring, Susan C., and John C. Paolillo. “Gender and Genre Variation in Weblogs.” Journal of Sociolinguistics 10.4 (2006): 439–459.

Rao, Delip et al. “Classifying Latent User Attributes in Twitter.” Proceedings of the 2nd International Workshop on Search and Mining User-Generated Contents. Toronto, ON, Canada: ACM, 2010. 37–44.

Tebeaux, Elizabeth. “Toward an Understanding of Gender Differences in Written Business Communications…” Journal of Business and Technical Communication 4.1 (1990): 25–43.

Yan, Xiang, and Ling Yan. “Gender Classification of Weblog Authors.” AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs. 2006. 228–230.


Use What You Choose, article posted on ACM

18 Nov

ACM has published the Proceedings of the 34th ACM International Conference on the Design of Communication (September 2016 SIGDOC ’16), including our article, “Use What You Choose: Applying Computational Methods to Genre Studies in Technical Communication.” My co-authors are William Hart-Davidson, Kenneth C. Walker, Douglas M. Walls, and  Ryan Omizo.

Our article is available for free download here:

Does it matter to the legal profession and pedagogy that men and women didn’t write differently?

3 Nov

I gave a talk this week at a law school regarding my article in Written Communication from October 2016, “Gender/Genre: The lack of gendered register in texts requiring genre knowledge.” The article reports the results of an empirical study, but it does so with reference to theories from corpus linguistics and relevance-theoretic pragmatics, not the sort of thing that most law faculty are interested in. Instead, I wanted to emphasize for them the implications of my study in the legal profession and pedagogy and to situate it within a conversation about gender differences more broadly.

The article is one voice in a cultural and scholarly discussion about gender difference in communication. The conversation about gender differences has a folk component and a scholarly component. My article cites just three news stories—from The Boston Globe, CBS News, and The New Republic—but Google “gender difference” in Google news entries for a plethora.

Despite frequent studies that seem to show gender differences, my study showed that where men and women received similar training for a less than a year and set to a similar writing task in a profession to which they are socializing, their writing was statistically and practically indistinguishable.

This is important because difference discourse enables deficiency discourse. In other words, folks who talk of gender differences often end up talking about how some task may be more suited to men than to women. While there is evidence that women communicate differently than men in some contexts, particularly ones like social media without the constraints of professional conventions, there is not so much about the professional context and even less about professional writing.

Destabilizing folk beliefs about gender differences serves an important purpose in the legal profession, because we don’t want those folk beliefs to affect the confidence placed in female attorneys. In other words, we want to prevent gender discrimination in the legal workplace.

I used Biber’s involved/informational dimension[1] to identify variables to look for in a corpus of nearly 200 law-student papers. Several studies of gender differences in writing have used the involved—informational dimension. (See the article for discussion and citations. I won’t repeat that discussion here.)

The involved end of the dimension is frequently associated with women’s writing and the informational end with mens’. For example, Biber and colleagues examined a 123,000-word corpus of letters written by men (n = 187) and women (n = 51) between the 17th and 20th centuries.[2] Using the involved–informational dimension, they concluded that letters women wrote showed higher prevalence of involved characteristics than those men wrote. They also concluded that letters women wrote to other women were more involved than those they wrote to men, and letters men wrote to men were more informational than those they wrote to women (pp. 219-220).

Note that this suggests there is a sensitivity in the writers to the gender, and perhaps the discursive expectations of the audience. Thus, though men wrote more informationally than women, they did so to a lesser extent when writing TO women.

So, I wondered if men and women would abandon habitual, gendered communication practices when they wrote for an audience with more specific expectations for the writing in question. I collected students’ responses and final writing projects from their first-year legal writing course at two law schools in the Midwest. There were 197 participants of whom 193 indicated their gender. I asked students to indicate their gender in a free-form field, for reasons set out in the article and in some forthcoming work of mine. Based on their responses, I classed them into two gender categories, Gender M and Gender F.

None of the informational features varied significantly (p<0.05) with the gender of the author. Three of the involved features varied statistically significantly, but even they did not confirm an association between Gender F writers and the involved style, as the gender using these three involved features more frequently was Gender M, which tends to disconfirm a correlation between involved register and female authors in this sample.

Here, all three significant differences also had small effect sizes. (See the article for a discussion of this concept.) Two other features had small effect sizes, and all the rest had negligible effect sizes (none with r>.10). In short, it would be nearly impossible for a human reader to detect any differences based on author gender, consciously or unconsciously.

So in this study, linguistic register did not vary with gender where novice legal writers, after receiving less than a year of professional training, wrote in a form they understood to be convention-bound.

So what? The article makes its own arguments about its theoretical significance. What about significance to the legal profession and pedagogy?

The significance of my findings for the legal profession is simple: Law firms should expect men and women to communicate similarly, at least in writing. My study is one piece of evidence that women adapt to one genre of legal communication very similarly to men. Further research may reveal others.

None of this, however, means that judges or law-firm hiring partners, even female ones, are judging female associates according to the same standards as male ones. For example, in another recent voice in the scholarly debate, Professor Michael Higdon discussed the public distress over the use by women of “vocal fry,” a low tone of the voice that results in a gravelly sort of sound.[3] Apparently, it’s commonly used by the Kardashians, and many folks consider it an abomination in women’s speech. But I noted in a recent review of prize-winning Three-Minute Thesis presentations that male presenters make frequent use of vocal fry. I don’t know if anyone was complaining, but the speakers did after all win prizes for their performances.[4]

The bind, of course, is that women are told they should lower their voices to sound more authoritative. Their different—i.e., non-masculine—voices are seen as deficient, and the remedy is to make them sound more masculine. But they have to be careful not to go too far. This is similar to the bind where women, told to be more assertive like male counterparts are judged when they do so to be bossy or bitchy.

The significance of the conversation for legal educators is more complex. It fits into a broader conversation about women, communication, and the law. My study showed that men and women did not communicate distinguishably on the involved/informational dimension in first-year legal writing. I suspect that other characteristics, like vocal fry, are plenty common among men and disfavored mostly when women use them. With such characteristics, we need to let our female students know when they are doing it and that some folks find it annoying. The choice whether to use it or not must be theirs, and we should respect that choice.

We need to let ALL our students know first that many folk beliefs about gender differences in communication are ungrounded, and even some studies that purport to measure them are flawed (because, for example, they look at texts where men and women are not writing for the same purposes or with the same training); and second that some judgments are applied disproportionately to women.

Finally, we need to model the kind of leadership for them that we expect them to take up in the profession. When they become the hiring partners and judges of the future, they should not assume that a woman using vocal fry is a “valley girl” or “Kardashian wannabe”—in other words, they should withhold judgment about the person based on this superficial characteristic. But as good mentors, they may want to alert a female colleague to the rhetorical implications of her speech patterns, just as we as good teachers might do.

One professor in my audience noted that the first half of my pedagogical treatment puts a special burden on female students. Telling them that vocal fry represents a rhetorical danger requires them to add another thing to their checklist of things to do or not do based on gender. I acknowledge that is true. But until the second half of my pedagogical treatment is widely practiced and a new generation of decision-makers takes power in the profession, women will be subject to these kinds of judgments, and they deserve to know the risks so they can make their own choices.

[1] Douglas Biber, Variation across speech and writing 105 (1988).

[2] Douglas Biber, Susan Conrad & Randi Reppen, Corpus linguistics: Investigating language structure and use (1998).

[3] Michael J. Higdon, Oral Advocacy and Vocal Fry: The Unseemly, Sexist Side of Nonverbal Persuasion, 13 Legal Comm’n & Rhetoric: JALWD 1 (2015).

[4] I’m grateful to have taken part in a Facebook discussion of Prof. Higdon’s work hosted by JALWD on October 18. It helped me refine my thoughts on pedagogy relating to this topic.

Hahn: The Stactive Style

3 Oct

I enjoyed getting to know Dr. Ed Hahn during our time in Minnesota’s PhD program in Rhetoric and Scientific and Technical Communication. He always struck me as a very smart and thoughtful guy. In the most recent issue of Rhetoric Society Quarterly, he demonstrated that amply with this essay:

Hahn, E. (2016). The Stactive Style: Whiteness and the Rhetoric of History. Rhetoric Society Quarterly, 46(4), 331–350.

In it, Hahn identifies a stylistic symptom of a challenge that justice-oriented scholars with commitments in postmodern philosophy must face. On the one hand, they need to document historical instances of injustice, for example by showing patterns of racism, in order to argue in a rhetorically effective way to remedy that injustice. On the other hand, they carry with them the postmodern skepticism toward historical narratives (grand or otherwise).

The symptom, according to Hahn, is the “stactive” sentence style. I’m normally not fond of portmanteau words, and stactive is one. Its parents are “stative,” referring to sentences (often constructed with the copula) that “express states of being rather than action” (e.g., “I am hungry.”); and “active,” referring to sentences where a change in state is described (e.g., “I ate lunch.”). Hahn argues stactive sentences have aspects of both parents, that they suggest a historicity and change while concealing any details (dates, agents, etc.) about the process. In short, they serve a stative function using active verbs.

Hahn gives numerous examples of writers who nod to the necessity of historical processes resulting in present-day statuses but obscure actors, objects, dates, and details of the process stylistically. In an essay in Rhetoric Review (2005), Kennedy, Middleton, and Ratcliffe write that “‘the logic of white supremacy,’ for instance, ‘emerged to justify the existence of slavery as well as the oppression of slaves, Chinese immigrants, American Indians, Jewish people, etc.'” (emphasis by Hahn; Hahn at p. 337, qtng. Kennedy, Middleton, and Ratcliffe).  Here, Hahn argues that “emerged” conceals the processes and any detailed account of how white supremacy emerged, despite the authors’ call for a ‘historicization’ of whiteness.

My gloss on this: Accepting postmodernism in this sense deprives us of confidence in tools (like critically rational discourse and empirical evidence) that rhetoric tells us we need to actually solve problems in the world. I agree.

Two additional thoughts:

  1. In my mind, this connects with work in cognitive linguistics showing that people who hear a story told with indirect verbs (passives, middle voice, nominalizations) tend to attribute less culpability to the human actors in the story. (See Fausey, C. M., & Boroditsky, L. (2010). Subtle linguistic cues influence perceived blame and financial liability. Psychonomic Bulletin & Review, 17(5), 644–650.)
  2. This makes me think of the concept in linguistics of the ‘middle voice,’ which comes somewhere between the ‘active’ and ‘passive’ voices. For example, if y0u say, “The coffee brewed,” the coffee is the subject of the verb, but it is not really an agent or a patient in this construction. The agent is removed. (I’m veering into stuff I don’t remember that well from linguistics, so take it with a grain of salt.) Middle voice also appears in English with reflexive pronouns. E.g., “He laid himself down,” or “She got herself out of there.” In some languages it is (or was) very common. (I remember a lot of it in Old Norse class.)

My Written Communication article has dropped…

24 Sep

My article in Written Communication is now available online. (The print version comes out next month.) You can read it on the journal’s website only if you have a subscription to the journal. You can read the version I submitted here (not formatted as it appears in the journal, but same content other than a few typo corrections):

Gender/Genre: The lack of gendered register in texts requiring genre knowledge

You can also contact me directly for a copy of the PDF from the journal (with the official page numbers).

Here’s the abstract:

Some studies have found characteristics of written texts that vary with author gender, echoing popular beliefs about essential gender differences that are reinforced in popular works of some scholarly authors. This article reports a study examining texts (N = 193) written in the same genre—a legal memorandum—by women and men with similar training in production of this type of discourse— the first year of U.S. law school—and finds no difference between them on the involved–informational dimension of linguistic register developed by Biber. These findings provide quantitative data opposing essentialist narratives of gender difference in communication. This essay considers relevance theory as a framework for understanding the interaction, exhibited in this and previous studies, of genre knowledge and gendered communicative performances.

Use what you choose presentation at SIGDOC ’16

23 Sep

I’m presenting the paper below at SIGDOC ’16 in Silver Spring, Maryland today. The paper is the outgrowth of an RSA workshop that Bill Hart-Davidson and Ryan Omizo led in Madison, WI, in 2015. Here’s the abstract:

This paper reports on the results of an intensive application development workshop held in the summer of 2015 during which a group of thirteen researchers came together to explore the use of machine-learning algorithms in technical communication. To do this we analyzed consumer electronic product customer reviews to reevaluate a central concept in North American Genre Theory: stable genre structures arise from recurring social actions. We discovered evidence of genre hybridity in the signals of instructional genres embedded into customer reviews. Our paper discusses the creation of a prototype web application, “Use What You Choose” (UWYC), which sorts the natural language text of Amazon reviews into two categories: instructionally-weighed reviews (e.g., reviews that contain operational information about products) and non-instructionally-weighed reviews (those that evaluate the quality of the product). Our results contribute to rhetorical genre theory and offer ideas on applying genre theory to inform application design for users of information services.

Here is the full paper:

And here are my slides from today, along with my speaker notes:

Qualitative empirical research in legal communication (LWI 2016)

12 Jul

This post will ultimately include the presentation slides, handout, and annotated bibliography for my presentation/workshop at the Legal Writing Institute in Portland OR on July 13, 2016. (Until the session is over, only the bibliography will appear here.)

Presentation slides (with notes)

This PDF includes my slides and my (sometimes extensive) notes on them. Try to ignore the little cues on them reminding me how to do the slide animations and when to move to the [NEXT] slide.

Presentation handouts

This PDF includes all the handouts from the session.

Draft annotated bibliography

I meant this bibliography to be more complete and more annotated than it is. Nevertheless, this is the best I was able to do given the time constraints.


Reflections on the Western Front (Part 2)–by the numbers

3 Jul

In Europe this month, there are numerous commemorations of the 1916 Battle of the Somme. Meanwhile, I’m distressed that so little attention has been paid in the U.S. to the centennial of World War I. The importance of that war in setting the stage for the 20th Century is hard to overestimate–the rise of Hitler’s Germany, the reluctance of the other powers to check him, etc. (Of course, events like the Congress of Vienna 100 years earlier can be seen as setting the stage for WW I. Such is history.) But for Americans, reflecting on WWI seems to be of little consequence. Perhaps the numbers can help tell the story why.

Consider the following table. Note that Civil War dead (750,000 or so) account for more than half of the 1.35 million military dead from ALL U.S. wars right up to the present, combined. So we can see why the Civil War is such a popular topic of interest in the U.S., even today. World War I, by contrast, represents less than 10% of those losses.

War Nation Military deaths Population Mil. deaths as % of population
American Revolution U.S. 25,000 3,900,000 0.64%
Civil War U.S.: Union and Confederate 750,000 31,400,000 2.39%
World War I U.S. 116,500 92,200,000 0.13%
World War II U.S. 405,400 142,100,000 0.29%
Viet Nam U.S. 58,200 213,300,000 0.03%
World War I U.K. and colonies 830,000 45,400,000 1.83%
World War I France 1,800,000 39,600,000 4.55%
World War I Germany 2,000,000 64,900,000 3.08%
World War I Russia 2,000,000 175,000,000 1.14%
World War I Austria-Hungary 1,350,000 51,400,000 2.63%
World War I Canada 60,000 7,200,000 0.83%
World War I Australia 60,000 5,000,000 1.20%
World War I New Zealand 17,500 1,100,000 1.59%

(Data from Wikipedia (U.S. casualties in all wars and other countries’ in WW I) and U.S. Census.  (census). Compare Wikipedia data on WW II casualties from all countries.)

From these numbers, it’s pretty easy to see why the Civil War looms large in the American consciousness, and even why World War II is so much more important than World War I for most Americans. (World War II is probably the moment when it became clear that the U.S. was a super-power, too.) In a way different than these other wars, WW I was not America’s war. We spent most of 1914-1918 avoiding getting into the war. The U.S. did not declare war against Germany until April 2017. By the time a million Americans were present in Europe, in 1918, more than 1.5 million Frenchmen had already died defending their country.

When folks characterize the French as being prone to surrender, I’m wondering what they are talking about. By the time WW I was over, the French had sacrificed the lives of more than 4.5% of their population (that does not even include war wounded)–a whole generation of young men wiped out. The same generation of the U.K. and its possessions was also decimated. It’s not hard to see why popular opinion (and thus politicians) in Europe did not support taking more aggressive action against Hitler as Germany positioned itself for the next world conflict.

It’s also easy to see why World War I still echoes so loudly along the Western Front today, and perhaps why we can barely hear those echoes here.

Reflections on the Western Front (WW I), Part 1

27 Jun

My spouse and I took a tour along part of the Western Front of WW I in Belgium and France this month. I did not, as I had hoped, post reflections along the way, as it turned out to be more work bicycling 40-70 km per day than I thought it would, especially on chilly, rainy days. I posted the following materials, making up Part 1 of my reflections, on Facebook during the journey. I’m gathering them here for friends and colleagues who are not Facebook “friends.” Part 2 and maybe subsequent parts will come later. (All photos are ours, Copyright 2016 Brian Larson and Robert Tendal, unless otherwise indicated. Contact us for permission before using.)

In and near Ypres

On June 16 and 17, we were in and near Ypres/Ieper, a location important throughout the war. For those of you who know the poem “In Flanders Fields,” these are those fields. We were in the “In Flanders Fields” museum and attended the daily Last Post ceremony under the Menin Gate, pictured here, on June 16 which happens at precisely 20:00hr every day.

DSCF0994 Menin gate

It is a quiet, solemn occasion, with hundreds of folks respectfully witnessing. A single lector recites lines from “For the Fallen,” by Laurence Binyon:

“They shall grow not old, as we that are left grow old:
Age shall not weary them, nor the years condemn.
At the going down of the sun and in the morning,
We will remember them.”

The deep voice of the crowd repeats the poem’s last line:
“We will remember them.”

DSCF1009 Menin crowd 2

Perhaps the most famous war poem ever, John McRea’s “In Flanders Fields” commemorates the deaths of many young men. Writing in 1915, McRea could have had no idea how many more young men the poppies would later commemorate in their blowing. McRae’s language is lovely, but I feel somehow that it is still glorifying war: “If ye break faith with use who die….” In fact, it was used as a public relations tool in increase volunteer enlistment throughout the British Empire and Commonwealth. I’ll post a couple other poems that I think are equally beautiful but show the more complex face of the war. Still, McRea’s words deserve a place of honor:

In Flanders fields the poppies blow
Between the crosses, row on row,
That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.

We are the Dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved and were loved, and now we lie
In Flanders fields.

Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high.
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields.

Looking at Ypres now, you could not possibly imagine the waste in which it lay at the end of the War. The pictures you see here are an idyll, a medieval town reconstructed in its pre-war image.

DSCF1006 Menin from Belfrey

DSCF1007 From Ypres Belfrey

It’s beautiful, and yet it’s a haunting reminder of the way that modern warfare begs us to imagine ourselves in a time before, a time that maybe never really existed, but that comforts us nonetheless.

Rubble, with parts of some buildings still standing
Public Domain,

The Menin Gate Memorial, however, helps to shatter that illusion. Among the 54,000 names on the Menin Gate memorial in Ypres, names of men whose bodies were never found after WWI, is that of T.P. Ward. I’m no relation to him (other than as a human being). Someone thought to decorate it with a poppy.

DSCF0995 Poppy

Near Ypres and the Menin Gate is the Tyne Cot Cemetery and Memorial, where many men are buried, but another 35,000 are commemorated on the wall whose graves are not known. Remember that World War I was described during its era as the “war that would end all wars.” This epitaph on a soldier’s tombstone–provided by his parents after the War–at the Tyne Cot cemetery is a fitting rebuttal of that ‘argument’: “Sacrificed to the fallacy that war can end war.”

DSCF1068 fallacy