Failed Mary

Why physicalists should be ambivalent about Mary and what they should actually say about her.

This post is in response to Pete Mandik’s Hail Mary.

I had to write something about this because this is an argument I’ve had with Pete on a number of occasions before on Twitter/BlueSky, and it’s a rare area where I strongly disagree with him, though in general I really appreciate Pete’s views.

I’ll assume the reader is familiar with Mary the colour scientist, and if not go and read Pete’s post. Read Pete’s post anyway!

The disagreement centers on what physicalists should say about Mary. Pete thinks that Mary will necessarily be able to figure out what red is like. I favour an alternative response known as The Ability Hypothesis.

But let’s get out of the way first the many points where I do agree with Pete. The main thing is that I agree with him that Mary does not disprove physicalism. Great! But furthermore, assuming physicalism is true, then:

1. This is a really far-out thought experiment, requiring Mary to perform superhuman feats of understanding and deduction that are so unrealistic as to make our intuitions about what Mary would or would not know fairly worthless. (Pete hasn’t emphasised this point in his post but has elsewhere.)

2. For some experiences, ordinary humans can know what they are like without having experienced them, by interpolating or composing other experiences. Hume’s missing shade of blue is a case of interpolation. Pete’s Japanese flag is a case of composition.

3. For any experience, is always possible in principle for at least some extraordinary agent to fully know what it is like without having experienced it, as in Pete’s Swamp Mary thought experiment.

4. Mary would be able to know or deduce all the effable facts about what red is like. You can quiz her as much as you like about the experience of seeing red, and she will be able to give you all the right answers.

5. Nothing mental is fundamental. All of our experiences are complex.

To be clear, I’m not asserting that she won’t be able to know what red is like. Maybe she will. Who knows (see point 1)? My issue is with thinking that it has to be possible for her to know what red is like. I’m arguing for open-mindedess. That physicalism is compatible with either experimental outcome. For what it’s worth, despite not trusting my intuitions, I guess that she would not know what red is like, at least if her brain is organised anything like a human’s.

Let’s look at the above points in turn, and see how they address the question of whether physicalism is compatible with Mary not being able to know what red is like just from reading books and thinking hard.

Point 1 above clearly favours open-mindedness. If our intuitions are unreliable, and the case is far-out, then it’s hard to say what is possible. We cannot with any confidence say what Mary’s reaction is going to be to seeing red for the first time.

The second and third points demonstrate that it is possible in some circumstances to KWIL without having had the experience. But that doesn’t mean that it is always possible to KWIL just by reading books and thinking hard.

Hume’s missing shade of blue is one thing, but what about all the missing shades that are completely out of distribution? I don’t think I can imagine the missing shades perceived by a tetrachromat. I don’t think someone with red-green colour blindness can really imagine distinct shades of red and green. OK, these are ordinary people we’re talking about, not Mary the super-genius. But the point stands that just because an agent can interpolate doesn’t mean an agent can extrapolate.

Similar points apply to composition. I can imagine a Japanese flag without seeing it because I have conscious access to all the parts of which it is composed. It doesn’t follow that I can always compose any arbitrary experience in my imagination, because I may not always have conscious access to the necessary parts. To borrow an example from Lewis, I don’t know what marmite tastes like, and I’m probably not going to get there by being told to combine various tastes together in my imagination. So, for ordinary humans, some novel experiences are composable from familiar ones and others are not.

But, again, Mary is no ordinary human. She’s a super-genius. Very extraordinary agents such as Swamp Mary can certainly know what an experience is like without having experienced it. Mary is extraordinary in virtue of her great intelligence and knowledge, but as I understand it, she is otherwise supposed to be as close to an ordinary human as possible. She doesn’t just know what red is like by coincidence, like Swamp Mary. She’s supposed to know or not know it based only on her thinking about propositions. So the question is whether Mary can extrapolate out of distribution, or compose familiar experiences of non-colours into novel experiences of colour in her imagination, or otherwise know what red is like, purely in virtue of the fact that she knows a lot of stuff and is very smart despite being otherwise normal.

Since we agree that experience isn’t strictly required for knowledge, then it follows that with all her superhuman knowledge and limitless material resources, Mary could design and program a nanorobot neurosurgeon that could go into her brain and physically rewire it so that she knows what red is like without her having had the experience. But that goes beyond just reading books and thinking hard. If we confine ourselves to reading books and thinking hard, then I don’t see much reason for confidence that Mary will necessarily be able to know what red is like.

For the fourth point, it’s important to distinguish KWIL from knowing all the effable facts about what red is like, because I’m a little worried that there might be a conflation here in the minds of some people (probably not Pete). She will know, for example, that red looks a bit like orange, and orange is intermediate between red and yellow. She will know that ripe tomatoes are red. But the question is whether she will have the knowledge that some people doubt is effable. Will she be able to recognise a colour when she sees it for the first time? Will she experience the sensation as entirely novel or as somewhat familiar (because it is as she imagined it)?

Pete might think all these questions require only effable knowledge, but I’m not so sure. Some knowledge is practical rather than propositional. Knowing how to juggle or ride a bike isn’t really a matter of knowing propositions. It’s a matter of training yourself to have the right neural circuitry to be able to do something. You can’t learn them from books. You have to actually go and practice. Recognising and imagining red may be like this. But far from being a problem for physicalism, this is just because physical changes are required in your brain, and these physical changes cannot be achieved by reading and thinking in humans, no matter how smart they are. That’s just not how we’re built. It might be possible for some being. An alien or a robot that can reprogram itself by an act of will. But not for humans.

I really do want to emphasise this point, so at the risk of repeating myself, I want to ask you to try to take physcalism as seriously as you can by thinking of brain as an entirely physical system, not as a non-corporeal mind that only processes propositions. If the brain is a physical system, then everything the brain can do is as a result of physical interactions. Learning is physical. Acting is physical. Some learning is certainly achievable by reading books and thinking hard. Some learning is not. Some learning requires physical activity of other kinds. On physicalism, there is no reason to think that all brain states are reachable only by processing propositions. Knowing what red is like may be one of those brain states that is not. It may be more a case of knowing how to imagine red or recognise red than it is a case of knowing a fact (the Ability Hypothesis).

Given that Pete is a qualia quietist, I find Pete’s opposition to the Ability Hypothesis a little perplexing. The Knowledge Argument depends on the assumption that knowing what red is like is knowing a fact, because the motivating contradiction at the heart of it is that Mary learns a new fact despite already knowing all the facts. But if it is a fact, what sort of fact could it be? It seems to me it must be the fact that some quale <red> is associated with the label “red”. But if we are not to speak of qualia, then I’m not sure that we should be entertaining the notion of facts about qualia. We should deny that there is any such fact. Knowledge of what red is like is instead something else, i.e. knowledge of how to recognise and imagine red. Like Pete, I find the concept of qualia to be unhelpful. That’s precisely why I think the Ability Hypothesis is a better explanation.

The fifth point is subtle. I agree with Pete that nothing is fundamentally mental. It’s all structure. All functional. In humans, this structure is presumably built out of patterns of neuronal connection and firing. But what we consciously grasp of this structure is coarse-grained. I don’t have direct, first-hand knowledge of which neurons are firing when I think of something red. Neither can I by an act of will decide which neurons to fire and when, much less which to wire together. I might be able to put together what I know of shapes and colours to construct the mental image of a Japanese flag, but I cannot perform such acts of composition at the level of individual neurons. Unlike me, Mary knows everything. So, presumably, she can figure out what neurons in her brain would fire when she experiences red or imagines red. But it doesn’t follow that she has fine-grained control of them. She can know what the experiences of imagining or seeing or recognising red are. They are distinct firing patterns. But she cannot necessarily reproduce these patterns in her brain just by thinking. So she cannot necessarily imagine red or recognise red.

But maybe she can. Maybe she is smart enough to find some way to hack her brain into a state where it is imagining red by following some carefully designed sequence of mental activities she does know how to activate. But maybe there is no such way. Maybe the only way to reliably get a human brain into that state requires either invasive physical intervention (surgery) or just having the experience.

Maybe she’s smart enough to figure out distinct downstream recognisable responses she would experience from seeing red as opposed to blue. Maybe she knows that if she were to see blue, she would be inclined to say “Wow!”. Maybe she knows that if she were to see red, she would be inclined to say “Golly!”. So maybe she can figure out if she is seeing red or blue based on monitoring her inclinations to make one exclamation versus another. On the other hand, maybe there is no real noticeable difference in her inclinations either way, so maybe she can’t. And either way, this is quite unlike how an ordinary human recognises red. It’s simply not the same thing.

If physicalism is true, then whether Mary will know what red is like depends on the level of fine-grained control she has of her own brain. Ex hypothesi, she’s arbitrarily smart and knowledgeable. The level of control she has is not specified. So it could go either way.