Credit…Erik Carter

Dear Commons Community,

The novelist, Stephen Marche, had an essay in The New York Times Book Review on Sunday entitled, The Algorithm That Could Take Us Inside Shakespeare’s Mind.   Here is an excerpt:

“Recently, though, Shakespeare’s ghost has been replaced in my mind by an algorithm. Cohere, a start-up company based in Canada, gave me access to its artificial intelligence platform. One of Cohere’s founders was a co-author of a 2017 paper that introduced a machine learning program called the Transformer, which processes language in patterns derived from the position of words in relation to the position of all other words in a given text. To simplify grotesquely, the Transformer converts context into math, and its power has upended the field of natural language processing: It vastly improves translation programs and has taken text generation to an eerie level.

Cohere, a nimble version of this increasingly important technology, is easy to use, meaning that even I — a man with a Ph.D. in English — can do so. It gave me the ability to create algorithms of individual writers’ styles. I started, it goes without saying, with Shakespeare. I’ve always been curious to know what Shakespeare would have written on the subject of Donald Trump’s hair, so I put into the interface a description I found on the internet and asked my Shakespeare-tuned model (generated from his complete works) to continue it. “Behind the undone chignon, the back of Donald Trump is covered in white lank,” the machine wrote. “The wig was tailor-made. It took as long as a full night to knit.” Which is how one of Shakespeare’s young thugs, somebody like Mercutio, might have described Trump’s hair. “Undone chignon” is exactly what Trump’s hair looks like. Even Shakespeare-the-algorithm finds the mot juste.

The entire essay is below.

A good example of how AI and machine learning are taking us into many areas of human endeavor!

Tony

———————————————————————————-

The New York Times

The Algorithm That Could Take Us Inside Shakespeare’s Mind

By Stephen Marche

Nov. 24, 202

I’ve met Shakespeare’s ghost. The encounter was more drudgery than mystery, I’m afraid. Twenty years ago, my doctoral supervisor casually informed me that I would need to read every tragedy written in English up to 1623. He said this as if it were a blessing — at least I wouldn’t have to read all the comedies. The total came to somewhere around 300 plays. I read one in the morning and one in the afternoon, five days a week, for the better part of a year, and they were mostly lousy as hell. But after I read all those lousy plays, I reread all of Shakespeare, and then, when something by Shakespeare was in front of me, his ghost entered the room. By sheer familiarity, by grinding through text and context, I came to know his presence.

Recently, though, Shakespeare’s ghost has been replaced in my mind by an algorithm. Cohere, a start-up company based in Canada, gave me access to its artificial intelligence platform. One of Cohere’s founders was a co-author of a 2017 paper that introduced a machine learning program called the Transformer, which processes language in patterns derived from the position of words in relation to the position of all other words in a given text. To simplify grotesquely, the Transformer converts context into math, and its power has upended the field of natural language processing: It vastly improves translation programs and has taken text generation to an eerie level.

Cohere, a nimble version of this increasingly important technology, is easy to use, meaning that even I — a man with a Ph.D. in English — can do so. It gave me the ability to create algorithms of individual writers’ styles. I started, it goes without saying, with Shakespeare. I’ve always been curious to know what Shakespeare would have written on the subject of Donald Trump’s hair, so I put into the interface a description I found on the internet and asked my Shakespeare-tuned model (generated from his complete works) to continue it. “Behind the undone chignon, the back of Donald Trump is covered in white lank,” the machine wrote. “The wig was tailor-made. It took as long as a full night to knit.” Which is how one of Shakespeare’s young thugs, somebody like Mercutio, might have described Trump’s hair. “Undone chignon” is exactly what Trump’s hair looks like. Even Shakespeare-the-algorithm finds the mot juste.

Text generation is the spectacular aspect of natural language processing — the technology powering the latest iteration of Google Assistant, which in 2018 the company claimed passed the Turing test, convincingly imitating consciousness. But the real power of the technology is probably analytic. Cohere predicts the most likely language to follow from any given set of texts: It turns probabilities into words. When you reverse the process — turn words into probabilities — you have an interpretive tool of astonishing power. This method will eventually be used for more serious business: to evaluate publishers’ back catalogs or unmade film scripts for potential hits (by comparing them to a database of actual hits) — any language where money is at stake. But Shakespeare is a good test of its power, too.

The playwright has always been a contradiction. Despite his palpable presence, he’s fundamentally ungraspable. The historical evidence of his life is negligible: There’s a will that makes him only harder to understand — what kind of man leaves his wife his “second-best bed”? — and a handful of other, equally half-significant, records; we don’t even know the exact date of his birth. The only way to know Shakespeare is through his works, and his works are textual quagmires.

Shakespeare was a working playwright of his period, and, much like screenwriters of our own era, he brought in other writers to help him with his plays and helped out other writers with theirs. The Folio, published in 1623, contains most of the works by Shakespeare that we know of, but not all. During his lifetime, quartos, small hand-held books sold on the street like paperbacks, were published, without his permission or approval, in pirated editions.

The result is permanent confusion. In the case of “Hamlet,” there are three versions of the play: the First Quarto, published in 1603, the Second Quarto, published between 1604 and 1605, and the Folio of 1623. In the First Quarto, sometimes called the “bad quarto,” the famous “To be, or not to be” speech begins this way:

To be, or not to be, ay there’s the point,
To Die, to sleep, is that all? Aye all:
No, to sleep, to dream, aye marry there it goes.

Nobody wants to believe that Shakespeare wrote this crap. It is the Second Quarto and the much later Folio that provide the more familiar “To be or not to be, that is the question” speech. But even between the two more palatable versions, there are significant differences. Should the verse read: “For who would bear the whips and scorns of time, / Th’oppressors wrong, the proud man’s contumely, / The pangs of despised love, the law’s delay” (Second Quarto), or “For who would bear the Whips and Scornes of time, / The Oppressors wrong, the poore mans Contumely, / The pangs of dispriz’d Love, the Lawes delay”? (Folio). There’s a big difference between despised love and disprized love, and between a proud man’s contumely and a poor man’s contumely. This is among the best-known passages in all secular literature, and nobody knows for certain how it should read, what actors should recite, what scholars should study. It’s embarrassing.

Every version of Shakespeare you’ve ever read is the result of centuries of debate, mostly arguments over style or historical context, developed through the grinding close study in which I was initiated. Computational modes of Shakespeare analysis are nearly as old as computing itself. The classic stylometric technique, begun in the late 1980s, was to tabulate the relative frequency of “function words” — words like “by” and “you” and “from” — and then to compare their numbers across manuscripts. The most sophisticated form of stylometric analysis so far has been WAN, or word adjacency networks, which register the frequency and proximity of function words in relation to one another. Both these applications have been controversial but broadly effective. The New Oxford Shakespeare editions attributed “Henry VI” to a collaboration with Christopher Marlowe on the basis of WAN analysis.

Cohere works on an entirely different level. It doesn’t require identifying function words or phrases. It just converts language into logarithmic probabilities. You create a Shakespeare algorithm. You put in each of the three different versions of “To be, or not to be” and out pop numbers: -3.6788540925266906 for the First Quarto, -3.179199017199017 for the Second Quarto, and -3.4799767386091127 for the Folio. The closer the number is to zero, the more likely the model thinks the sequence is. And Cohere’s answers make perfect sense — common sense, anyway. “Contumely” means insolence. Wouldn’t it be more likely to be a proud man acting insultingly?

My application of this technology is crude. I’m not a professor with institutional backing. I’m a freelancer on a Macbook Air with a sticky “up” cursor. Cohere offers a fascinating approach to the question of how we register any writer, or even any person, through his or her language. What’s glorious about Shakespeare, and a source of our fascination, is his consistent inconsistency — as well as our enduring uncertainty about who he was. Cohere reflects this: It doesn’t produce absolute answers, only the best possible answers — a moneyball of language, if you will. Ask any gambler, though: The best bet is not always the winning one. Maybe Shakespeare really did intend to say “poor man’s contumely.” It wouldn’t be the craziest thing he ever wrote.

When a ghost shows up, you might know who it is but you don’t know what it’s going to say. The Shakespeare algorithm is the same. Cohere conjures the playwright as an internal core of cohesion with half-predictable expression. Isn’t that a person? What more is there to anyone?

Stephen Marche’s book “The Next Civil War” will be published in January.