Dead Sheepdog: Ethical Dilemmas in Software Development

Ethics is the consideration of what one ought to do in a given circumstance within a context of society’s and one’s own values.

Ethics within a given discipline takes into account considerations specific to that discipline and values shared among trained and experienced practitioners of good will within that discipline.

“(Ethics) refers to moral values that are sound, actions that are morally required (right) or morally permissible (all right), policies and laws that are desirable.”

Ethical considerations are often hard to navigate. Not just because of the consequences of certain actions but because of ambiguity and dissensus over the given situation and anticipated outcomes.

In this, ethics shares qualities with software development itself.

As Fred Brooks describes, software development is suffused with essential complexity. Attempts to “abstract away its complexity often abstract away its essence.”

Martin, M. and Schinzinger, R., Ethics in Engineering, McGraw Hill, Boston, 2005, pg 8.

“No Silver Bullet – Essence and Accident in Software Engineering”, Brooks, F. P., Proceedings of the IFIP Tenth World Computing Conference, pp. 1069-1076, 1986.

Copyright ©, Ken H. Judy 2008, All rights reserved.

Haidt and Joseph identify five psychological systems, each with its own evolutionary history, that gives rise to moral intuitions across cultures.

Haidt and Joseph’s Five Psychological Systems

Foundation Origin/Expression
Harm/Care People have a sensitivity to cruelty and harm, they feel approval toward those who prevent or relieve harm, and this approval is culturally codified in virtues such as kindness and compassion, and in corresponding vices such as cruelty and aggression.
Fairness/Reciprocity The long history of alliance formation and cooperation among unrelated individuals in many primate species has led to the evolution of a suite of emotions that motivate reciprocal altruism, including anger, guilt, and gratitude (Trivers, 1971).
Ingroup/loyalty People value their ingroups, they also value those who sacrifice for the ingroup, and despise those who betray or fail to come to the aid of the ingroup, particularly in times of conflict. Most cultures therefore have constructed virtues such as loyalty, patriotism, and heroism
Authority/respect People often feel respect, awe, and admiration toward legitimate authorities, and many cultures have constructed virtues related to good leadership, which is often thought to involve magnanimity, fatherliness, and wisdom. Bad leaders are despotic, exploitative, or inept. Conversely, many societies value virtues related to subordination: respect, duty, and obedience.
Purity/sanctity In most human societies disgust has become a social emotion as well, attached at a minimum to those whose appearance or occupation makes people feel queasy. In many cultures, disgust goes beyond such contaminant-related issues and supports a set of virtues and vices linked to bodily activities in general, and religious activities in particular.

These five psychological systems operate at both a conscious and subconscious level. What we feel to be a rationally derived moral judgement may be an instinctive reaction justified by a reasoned argument.

One study examined people’s responses to two dilemmas: A runaway trolley is headed for five people who will be killed if it continues on its present course. In the first case, the only way to save them is to hit a switch that will turn the trolley onto an alternate set of tracks where it will kill one person but save five. In the second case, the only way to save the five people is to push an innocent stranger off a bridge, onto the tracks below killing him but halting the trolley.

The study reveals that most people would hit the switch but not push the person. Though they believe this difference is based in reason the two situations are morally equivalent. fMRI results demonstrate a strong visceral response to pushing a person to their death. This revulsion colors their reasoned response.

Even after the rational arguments for their decision are logically refuted, people embrace their original responses and remain convinced that there is some rational basis for the difference.

Having established that the psychological underpinnings of moral thought are at some level subconscious and irrational, Hait goes on to demonstrate that though the five psychological systems are in part innate and culturally determined, they also vary among individuals within a society. Different people balance these five systems differently and these differences drive different conclusions about appropriate moral action.

As support for this claim, one study established that within American culture, political liberals place more weight in considerations of harm, care, fairness and reciprocity than ingroup loyalty, authority, respect, purity and sanctity. Political conservatives balance these concerns more equally.

The result of Hait’s work is that we can expect one person’s consideration of what is right to differ from another person of good will even in situations where the consequences are clear. Hence, essential complexity. Reducing ethical decision making to rules abstracts away the essence of what is really going on in people’s psyches.
Hait, J. and Graham J.,”When Morality Opposes Justice: Conservatives Have Moral Intuitions that
Liberals may not Recognize”, Social Justice Research, 2007.

Greene, J.D., Sommerville, R.B., Nystrom, L.E., Darley, J.M., & Cohen, J.D. (2001). An fMRI
investigation of emotional engagement in moral Judgment. Science, Vol. 293, Sept. 14, 2001, 2105-2108.

Hait, J. and Graham J.,”When Morality Opposes Justice: Conservatives Have Moral Intuitions that Liberals may not Recognize”, Social Justice Research, 2007.

Copyright ©, Ken H. Judy 2008, All rights reserved.