Datamining

People sometimes ask me what does it mean when sociopaths say they datamine. Basically they are collecting information about you in an effort to predict your future behavior and what might please or displease you. In this way, they are not much different than Google, Facebook, Orkut, or the other programs that kids are into these days. This was a good description of how these programs work, from Mind Hacks:

The Economist has an excellent article that discusses the increasingly diverse ways in which information from your social network – drawn from services like Facebook, or from telephone calls or payment patterns – are being used to obtain personal information about you.

This is not information which you have explicitly stated or included, but which can be found out or ‘mined’ from your patterns of behaviour and your connections to other people.

And from the Economist article mentioned, the fascinating way in which phone companies target trend-setting customers:

Telecoms operators naturally prize mobile-phone subscribers who spend a lot, but some thriftier customers, it turns out, are actually more valuable. Known as “influencers”, these subscribers frequently persuade their friends, family and colleagues to follow them when they switch to a rival operator. The trick, then, is to identify such trendsetting subscribers and keep them on board with special discounts and promotions. People at the top of the office or social pecking order often receive quick callbacks, do not worry about calling other people late at night and tend to get more calls at times when social events are most often organised, such as Friday afternoons. Influential customers also reveal their clout by making long calls, while the calls they receive are generally short.

Similarly, sociopaths watch your behavior to figure out who you are. It can be something as small as the way you grip a steering wheel when you drive or whether you break prolonged eye contact and when. The sociopath collects all of this information about you and mentally references it to the thousands of other people he has collected information from, coming up with a rough sketch of who you are. As marketers have known for centuries, people that like certain things will probably like other similar things.

It's not hard to collect this information, the sociopath is paying attention to these little behavioral responses anyway to make sure that he is remaining undetected. And it's hard not to notice certain very common human behavioral patterns, once you've been made aware of them.

After the sociopath has collected all of this information, he can use it in various ways. He can use it to better construct his own masks to stay hidden. He can use it to anticipate your every need and desire. Or he can use it to get into your mind and plant yet another type of mine. That's the mining that you really should be worried about, and the only way that the sociopath can set traps in your mind is if you have weaknesses or needs that you refuse to address yourself.

Apart from that, datamining of any type is relatively harmless. It's basically just catering to your expectations.

Knowing you better than you know yourself

I saw this in a recent comment: "How could someone who feels no empathy for others possibly understand how to treat a child?"

It reminded me of something that one of my professors said once in the mid 2000s. He said that sometimes he would be in a bookstore, see a book he wanted to buy, but would put off buying it until he got home and could order it on Amazon. Why? Not because he wanted to get it for cheaper, but because he wanted Amazon to know about the purchase so it would be better at recommending books to him in the future. When I watch Netflix, I look at their recommendations for me, and their "star" ratings guess for how much I would like a particular film/television show. They're pretty accurate. And it doesn't take that many data points to pick you out from an otherwise anonymized list of a half million other Netflix subscribers. It's crazy. It turns out that we really all are special little snowflakes.

Has this never happened to you? That someone knew better than you what you would like? Is it empathy that helps them do it? Probably not, right? Because isn't empathy allegedly the ability to feel what another person is feeling? Does Netflix feel what I feel? Not likely, right? But Netflix still does a great job predicting what people want to watch. If I collect a bunch of data points on you, or a child, or your dog, or anything else, will I also understand how you would like to be treated? Will I know perhaps even better than you know yourself? My loved ones feel this way about me.

Maybe you're not the type of person who would go home to order a book on Amazon rather than buy it in the store (or one of the people making up the statistic that 75% of Netflix streaming selections come from recommendations from the site), but there are plenty of people who would value that unique service quite highly. This is particularly true of children who often have their true feelings and wants/needs superseded and/or ignored by the adults in their lives. And by the way, can empaths really understand what/how children think/feel? Unless they have a lot of time recently around people in that age, I have found that most adults are pretty bad at understanding or even caring about how kids feel.

Identifying sociopaths (and then what?)

This was an interesting post from author of Liars and Outliers, Bruce Schneier, about the identification of sociopaths. He first quotes Scott Adams (Dilbert comic author) predicting the future identification of "sociopaths and terrorists":

My hypothesis is that science will someday be able to identify sociopaths and terrorists by their patterns of Facebook and Internet use. I'll bet normal people interact with Facebook in ways that sociopaths and terrorists couldn't duplicate.

Anyone can post fake photos and acquire lots of friends who are actually acquaintances. But I'll bet there are so many patterns and tendencies of "normal" use on Facebook that a terrorist wouldn't be able to successfully fake it.

Adams says that the reason that it would work is the same reason that fraud detection programs work: "[C]rooks don't know there is a normal pattern and so they don't know when they violate it. I think the same would be true for Facebook. There must be dozens of normal Facebook patterns that sociopaths and terrorists wouldn't know about, and therefore couldn't fake." This does not seem implausible, and is one of several reasons why I think that remaining completely anonymous and undetectable is not going to work for the rising generation of sociopaths.

Schneier has another criticism:

Okay, but so what? Imagine you had such an amazingly accurate test...then what? Do we investigate those who test positive, even though there's no suspicion that they've actually done anything? Do we follow them around? Subject them to additional screening at airports? Throw them in jail because we know the streets will be safer because of it? Do we want to live in a Minority Report world? 

The problem isn't just that such a system is wrong, it's that the mathematics of testing makes this sort of thing pretty ineffective in practice. It's called the "base rate fallacy." Suppose you have a test that's 90% accurate in identifying both sociopaths and non-sociopaths. If you assume that 4% of people are sociopaths, then the chance of someone who tests positive actually being a sociopath is 26%. (For every thousand people tested, 90% of the 40 sociopaths will test positive, but so will 10% of the 960 non-sociopaths.) You have postulate a test with an amazing 99% accuracy -- only a 1% false positive rate -- even to have an 80% chance of someone testing positive actually being a sociopath.

He ends with this thought: "Many authors have written stories about thoughtcrime. Who has written about genecrime?"

The comments are also really worth reading for their intelligence and insight. For instance, there was this comment:

First you'll need a useful definition of "sociopath" that is not, for practical purposes, equivalent to "non-conformist" or "adherent to a religion not on the approved list", etc. 

Followed by this comment:

You may underestimate the ability to create tautological tests. If you define a sociopath as someone who fails the sociopathy test, then the sociopathy test is 100% accurate in identifying sociopaths. 

That's all well and good, until people begin to think that an attribute thus defined is useful for anything.

Edward Snowden and independent thinking

The reaction to Edward Snowden coming forward as the source for the NSA leaks has been interesting and varied, from clear signs of support to accusations of him being a traitor. I think the most interesting (and possibly the most prevalent reaction) is a little bit of fear mixed in with some what-is-he-thinking-could-he-really-be-that-naive and a lot of judgment guised as that's-not-how-I-would-have-done-it (this last one is the most hilarious to me -- you would never have done it, so any analysis of how you would have done it in a non-existent reality goes beyond mere speculation to pure fantasy). Like Monday morning quarterbacks, these people have questioned his decisions from things like his choice of an extradition-lite hideaway to his decision to come forward (as if him outting himself affects in anyway whether the U.S. government knows who he is and is trying to track him down) to whether he was able to save enough money to live on or if he is now completely unemployable for the rest of his life.


For as popular as Robert Frost's poem "The Road Not Taken" is, particularly during graduation speech season, there is a pretty clear social norm that favors steady job white picket fence 2 1/2 children and people that dare violate this norm can be very polarizing. On the one hand, most seem to acknowledge that there are great men and women that have bucked the trends and led to important advancements to us as a species. On the other hand, people who buck the trend present a lot of problems for society, at the very least because there is no comfortable pigeonhole to confine them to. This discomfort is often expressed in terms of these people being "unpredictable" or "untrustworthy". From Megan McArdle with the Daily Beast, "Whistleblowers Are Weird":

Human institutions, from the family to the government, are founded by trust.  You need to be able to trust the people you work with, at least to the extent of being able to predict their future behavior.  You may think you don't trust that rat down the hall, but in fact, you do trust him quite a lot: not to come into work with a machete and hack you to death in order to secure your superior office chair, not to start randomly swearing at clients, and so forth.

Some of that trust is enforced by fear of the consequences.  But a lot of our ability to make a credible committment to be trustworthy comes from the fact that we are hard wired to be loyal. . . . You would feel bad about yourself if you [broke that trust].

That's why psychopaths are so dangerous: they don't have any of the internal brakes, the shame and guilt, that keep the rest of us from blatantly violating the trust of people around us.  Oh, of course we do betray people from time to time--we break promises, forget to call our grandmothers, and engage in the guilty pleasure of gossiping about friends.  But the hallmark of these betrayals is that they are impulsive and unjustified.  Psychopaths feel no guilt about doing these things--or stealing your money, your wife, and your dog.  They are fundamentally untrustworthy, though also, thankfully rare.

This reasoning seems flawed -- the reason why the rat down the hall doesn't machete you is because he is hardwired to be loyal? She's correct that if you're worried about someone harming you, the world of possibilities includes both intentional bad behavior (which she suggests that only sociopaths commit because they don't feel shame and guilt and these "brakes" on bad behavior are both apparently necessary to avoid bad behavior and also infallible?) and the unintentional bad behavior of everyone else (which she suggests is always impulsive). She says that we need to trust people to predict their future behavior (says no statistician, behavioral economist, or psychologist ever because currently the best predictor of future behavior is not trust or loyalty but past behavior). But of the two possibilities of bad behavior, which is more predictable? The sociopath's? (Who is almost the quintessential economic rational actor.) Or the "impulsive and unjustified" (i.e. no apparent or reasonable triggers or other identifiable causes) behavior of the non-sociopath?

Nor does someone's trust or the apparent level of predictability of a person constrain his behavior. You can trust the rat down the hall all you want, but that doesn't mean that your trust in him will keep him from someday putting a machete in your back. The best you could say about him is that his past behavior doesn't indicate an above average risk of being a murderer and/or that the chance is already so low and there are so many competing dangers vying for our attention that it's simply not worth the effort of thinking about who exactly could kill you. At this point, though, we are simply talking about probabilities of behavior and "impulsive and unjustified" are almost by definition random and unpredictable.

What is apparently happening here is that McArdle and many others intuit that there is something particularly unknowable about whistleblowers and sociopaths. Uncertainty like this does in fact increase both actual and perceived risk. But there's nothing terribly "unknowable" about them. In fact, in a lot of ways they are less complicated than most people -- the one hyper-rational ruled by self-interest and the other an ideologue that can't be bought off by even a $200,000 a year cushy government salary. Which makes me think this is the real crux of the issue. The scary thing about a whistleblower or any other independent thinker is that he is not as constrained by social norms. This is evidenced for the whistleblower by his rejection of the picket fence and golden retriever lifestyle. For most people, it's possible to constrain their behavior with so-called golden handcuffs -- all the materialistic trappings of a comfortable life in exchange for unquestioning loyalty, including submitting one's will to one's employer, one's government, the police, and even one's parent teacher association. The norm is enforced as heartily as it is because although the majority acknowledges that they can gain big from independent thinking, they can't have everyone constantly questioning the most basic of social rules as it would lead to chaos and a weakening of the social contract. And people are incredibly and irrationally loss averse, so given the choice they would rather keep what they have than chance it on someone who who already comes off as a bit of an outsider (how do we even know he has our best interests at heart? how can we "trust" him to make the right decisions? isn't this why we have a government heavily dominated by administrative agencies so we could sub-contract out important decisions like this and never have to consider them ourselves?).

Thus, the uncertainty lies not in the behavior of the independent thinker, which is actually quite predictably independent, but whether he is right or not. The problem is that although we say that everyone is free to act according to his own best judgment, the "right" thing to do with that freedom as far as the majority is concerned is to marry, own a house, be gainfully employed, and have at least two children. Only then are you considered sufficiently invested in society that you become "predictable" to us, in that we know you have too much to lose to take any major risks ("freedom's just another word for nothing left to lose"). And if its one thing the financial crisis has taught us, it is that one person's actions can ripple through the entire global economy. Of course that applies for good things as well as bad, but how can we know ahead of time which is which? This is particularly a problem when (most?) people (inaccurately and unreliably) gauge the rightness or wrongness of decisions by imagining what they themselves would have done in that situation, and the thoughts and decisions of an independent thinker are difficult for many to fathom. (Of course, despite this inherent uncertainty, there are people who feel very comfortable assigning themselves the role of arbiter of good and bad. These people form the rank and file of the social norm police).

Still McArdle does make an interesting acknowledgment that the things that make people seem different and even unappealing to us are the very traits that make them socially beneficial:

We may well end up grateful to Edward Snowden, and also find that we don't like him very much. Of course, Edward Snowden probably doesn't care. After all, if he cared about people liking him as much as the rest of us do, he probably wouldn't have been able to do with he did.

Book appendix (part 3)

From an interview with a friend:


I like talking to you because you are like a stockpile of knowledge with the capability to process important components of that knowledge and to assimilate them into an intelligent decision—the best decision.  Whereas I feel like I might fall on to decision that is fourth best, even though I have been exposed to the same data.  But I have forgotten that information in the meantime, and am unable to pull it forward when the time to make the decision arises.  And you even take into account my personal preferences.  I don’t know how, I guess because you know me now.  But something I find very humorous is that when I start explaining emotionally frustrating things to you, maybe about my marriage, and you’ll say “That’s because he __________” and I am always wondering why you have so much insight into my emotional life.  Insight that I didn’t have—like I am still hashing through the ideas emotionally and haven’t been able to reach any conclusion, but you have been able to reach a conclusion by just listening to me for a minute.  Sometimes I discount your conclusions, I will be honest.  At those times I generally conclude that you didn’t input the right information.  Other times I will be surprised at how spot on you are.  It seems like you know my husband better than I know him. I’m always surprised with your assessments of people, because you can kind of sum them up, taking this vast amount of data—a person—and you break it down into the important bits for that output.  You tell me, “well of course that is what happened because of these few things.”  

Also, you’re blatantly honest.  At first I was scared and there were moments in this house in which I was afraid that you would provoke fights in social situations. Then I started finding the humor in it.  Now sometimes I will use it to find out things I really want to know by just asking you, although I can still get angry at some of the things you say.  Overall, though, it is refreshing, and I have a much harder time getting offended at anything you say than I used to.  Even now telling you these things, it’s odd because I think now you will understand me so much better and when I come to you with another emotional problem you will say, “Oh, it’s because of this,” or “something something something” and I will feel ok.  

When I come to you with an emotional problem though, I don’t feel like you give empathy or emotional support.  Sometimes you will say, “that’s just because your husband's a retard, sorry.”  So maybe that is empathy.  Maybe it is refreshing to hear that it comes down to something that isn’t emotional—that my problems aren’t fundamentally an emotional issue, but something separate that can be intellectualized.  It takes out the sting in the hurt.  

I remember one time you were talking to me in the car and you said something like, “I don’t think I want to marry a guy who is as intelligent as me.”  And I asked you, “someone more like me.”  You said “no, not really.”  And I thought, oh ok, smarter than me then.  

I think you’re a better computer than I am.  If you had learned all of the stuff that I learned in college, I think you could do so much better with it than I can.  But that’s alright, I supposed I have other skills.  You’re like a data processor, but better because you can also process emotional inputs.  You can’t ask Google why my husband did something.  It’s like the best thing—kind of like a fun toy.

Unflappable?

Very few things bother me, but sometimes I am bothered when there is an unexpected reaction or consequence to something I've done. My friend says that I'm never so mad as when I think I have been doing things on the up and up, but someone still chastises me.

The other day I was traveling and had to rent an automobile. I parked in a particular location that turned out to be in front of somebody's garage. They left a note for me saying that they needed to get their car our of their garage and were going to tow my car, etc. I grabbed the note and hurried away, I was late for something, but the note continued to bother me. How could I have not seen the garage, I wondered? What would I have done if they had towed it, let the rental agency deal with it? How had I let this almost happen? My mind wouldn't let it go.

I read a good description of this type of reaction in this comment:

Any time a kink happens in my social interactions, whether it's a slip of my tongue or an unexpectedly aggressive reply, I dwell on it. It replays in my mind, and I dissect it to find out what I could have done differently. Did I misread the person's intentions? Was I not forthcoming enough? It's not that I truly care how people perceive me; I don't hunger for their acceptance or praise. But I very carefully cultivate my outward persona: it is charming, it is witty, and it is benign. So when it fails to work as planned, it's a serious problem. It throws into question all of the hard work I've put into it. 

If I make someone cry, I'm not disturbed because I've caused them pain. I'm disturbed because I don't mean to be seen as a negative source--now I have to apologize or feign sincerity, or all my effort to appear as a sympathetic and trustworthy person, and the emotional power it gives me over that person, vanishes. 

I'm disturbed because I control everything, all the time, and for me to not do that, or stumble--it's unacceptable. 


With regard to my parking incident, I drove back the next day to that same neighborhood to investigate. There was no cutaway from the curb. The "garage" was covered in ivy and not clearly either a garage or functional. It was as hidden from sight as the Batcave. I was at glad to see that my mind hadn't slipped as much as I thought. And I started to wonder at how often people park in front of their garage. Do they deal with this every day? Could they put up a sign? Or perhaps paint the curb a different colour? I was angry at them, for setting me up for failure--for trapping me and acting like they had some sort of moral or legal high ground. I left them the note they wrote, secured in their door. I don't know why, but I thought it was vaguely threatening, like letting them know that I knew where they lived. And they shouldn't leave notes on my automobile. Or something...

Anxiety vs. fear (part 2)

There was an interesting article in the NY Times about the difference between fear and anxiety a little while ago.  Here is how they described it:

You are taking a walk in the woods ― pleasant, invigorating, the sun shining through the leaves. Suddenly, a rattlesnake appears at your feet. You experience something at that moment. You freeze, your heart rate shoots up and you begin to sweat ― a quick, automatic sequence of physical reactions. That reaction is fear.

A week later, you are taking the same walk again. Sunshine, pleasure, but no rattlesnake.  Still, you are worried that you will encounter one. The experience of walking through the woods is fraught with worry. You are anxious.

Human anxiety is greatly amplified by our ability to imagine the future, and our place in it.

What is the difference between anxiety and fear?

Scientists generally define fear as a negative emotional state triggered by the presence of a stimulus (the snake) that has the potential to cause harm, and anxiety as a negative emotional state in which the threat is not present but anticipated. We sometimes confuse the two: When someone says he is afraid he will fail an exam or get caught stealing or cheating, he should, by the definitions above, be saying he is anxious instead.
***
The automatic nature of the activation process reflects the fact that the amygdala does its work outside of conscious awareness. We respond to danger, then only afterward realize danger is present.


Every animal (including insects and worms, as well as animals more like us) is born with the ability to detect and respond to certain kinds of danger, and to learn about things associated with danger.  In short, the capacity to fear (in the sense of detecting and responding to danger) is pretty universal among animals.  But anxiety ― an experience of uncertainty ― is a different matter. It depends on the ability to anticipate, a capacity that is also present in some other animals, but that is especially well developed in humans.  We can project ourselves into the future like no other creature.

While anxiety is defined by uncertainty, human anxiety is greatly amplified by our ability to imagine the future, and our place in it, even a future that is physically impossible.  With imagination we can ruminate over that yet to be experienced, possibly impossible scenario. We use this creative capacity to great advantage when we envision how to make our lives better, but we can just as easily put it to work in less productive ways — worrying excessively about the outcome of things. Some concern about outcomes is essential to success in meeting life’s challenges and opportunities. But at some point, most of us probably worry more than we need to.  This raises the questions: How much fear and worry is too much? How do we know when we have skipped the line from normal fear and anxiety to a disorder?


And of course the line between fear and anxiety is not always clear either.

I thought that the article made an interesting point about the human ability to predict the future.  It's odd that I have cast myself in the part of oracle in my life -- an amateur fortune teller.  I guess it's because I thought it would be powerful to know the future.  I've gotten better over the years to the point where now every time that I get burned in a prediction it's been because I've failed to take into account how truly unpredictable other human behavior can be.  The more burned I become, the more reluctant I am to stick my hand in the fire.  I can't decide whether that is a good thing or a bad thing.