But there’s more. I don’t just build tools, I build environments. And that means support for reading the text, navigating it, searching it, and (most importantly) “working” with it. And I have no idea what that means either. So over the past few weeks I’ve been having hour-long chats with late-stage PhD students from the literature and history departments, and asking them to tell me about how they do research. I asked  all kinds of confusing and mundane questions like, “How do you decide what to underline?” and , “Can you define formalism for me?” and, “You mean you actually copy it out by hand?” and “How do you organize all the quotes you collect?” and, “How do you go about proving that?” and, “So you scanned in everything in those boxes?”

I only did twelve of those interviews, but patterns began to emerge.  So I did a survey. A simple one, with six questions about reading habits. This survey’s purpose was to confirm whether some of the patterns I noticed around reading were general. If you just want the charts summarizing the responses, you can find them here (those numbers include around 20 more responses I got while I was writing this post). For a full analysis in which I extract some general patterns in humanities scholars’ reading processes, read on.

This was the first page of the survey, with two screener questions. Over two days, I got 153 responses from humanities scholars whose primary sources were mostly textual (and 18 responses from others, but I removed those responses for this analysis). 

Working with text

My first question was about copied-out snippets. From what I heard in the interviews, humanities scholars working on a project eventually reach a point where they have some interpretation, some interesting angle they want to take. At this point, they start actively reading and re-reading their primary sources. Snippets begin to collect in their notes in large numbers.

The results are below. An overwhelming 90% of textual humanities scholars surveyed copy out snippets either frequently or occasionally.

What are snippets for?

But why? From what the interviewees said, copied-out snippets seem to be in a different category from other annotations (such as margin notes, underlines, highlights, or circles on the page). In fact, copying out seems reserved for a more important class of items. But what makes these copied-out snippets so special? This was my next question.

The results are below, and the responses confirmed the interviews. Most of the textual humanities scholars surveyed said that snippets were evidence, interesting or thought-provoking passages, or examples of something they were looking for.

In addition to the 4 existing choices, there were 28 “other” responses, which fell into these groups:

1. As a way to store quotations I think I’ll need (5 such responses)
2. Writing it out helps me understand and think about it (2)
3. Writing it out helps me remember it (5)
4. The snippet is a good summary of an author’s point or argument (4)
5. They help me outline a longer argument (3)
6. I need to translate it into another language (3)

It seems that copied-out snippets play a role very much like the evidence a lawyer lays before a jury. They provoke thought, they justify an interpretation, they are examples that support an argument (or indeed counterexamples that need to be explained away). A scholar amasses many such “pieces of argument” and then organizes them to tell a coherent story.

What do snippets look like?

As a tool builder and an information retrievalist I need to know: how long are these snippets, and is there accompanying information that users will want to add?

Snippet length

Are snippets a few words long? A few paragraphs?

The results are below. They revealed that snippets varied greatly in length.

• Many respondents had “some” or “many” snippets between a few words to a paragraph long
• About 25% of had “some” or “many” snippets that were longer than a paragraph.

Accompanying information

The interviews suggested that in addition to the literal text of the snippets themselves, there were often various kinds of notes as well as visual finding aids and citation information.

The results are below. Notes about why the snippet was relevant were very common.

And so were ideas that the scholars got from the copied-out snippet.

By far the most common was citation information: 76% always added it.

Other markers, such as post-it flags, colored highlights, and tags to enable keyword search were less common, but also done:

For a tool designer, the message is clear. If you want humanities scholars to read text using your tool, you must support all of the above activities.

Collecting evidence

When a scholar wonders, “Where else have I seen that before?” or, “Is this an unusual exception?” or, “Is this a pattern?”, my interviewees told me that there are two approaches they can take: they had either thought of this before, and relevant passages were already copied into their notes, or they’d have go back and re-read the relevant texts to search for evidence. To see if this was a general pattern, I created the next two survey question.

The responses are below, and they confirm my interviewees responses. 79% re-read when they had a new idea or interpretation, 65% when they had a new hypothesis, and 67% when they noticed a new pattern.

There were also 13 “other” responses, which fell into the following categories:

1. Just for fun (2 responses)
2. When I teach (5)
3. When I find my notes don’t have everything I need (3)
4. To understand it better (2)
5. To compare with other documents (1)

But what do scholars want to do with the text they re-read? This was my final question. I wanted to compare how note-taking behavior varied between first-time reading and re-reading:

The results are below. Compare how frequently scholars copied out snippets the first time (top) with when they were re-reading (bottom). The copy-out rate is pretty much the same, or perhaps even a little higher while re-reading.

These responses reveal an inefficiency in the “finding evidence” portion of the scholarly process. Reading is a great way to understand, to learn, to remember. But it is a very inefficient way to search for something. First, it’s very slow (and when you speed it up, it starts to lose its reliability). Second, it’s very subject to your state of mind (sometimes things pop out at you, sometimes they don’t). And third, it’s impossible to do thoroughly for very large collections.Yes, you need to revisit, re-find and re-acquaint yourself with the material. But what I object to is that you often have to re-read in order to do it.

A (partial) solution already exists: full-text digitization and search. Search can take you a long way, especially if your examples are associated with particular words. However, there are a great many sophisticated information retrieval technologies that go beyond keyword search. We can calculate and retrieve text passages by similarity, we can allow you to mark relevant passages and return more like those, we can train classifiers on what you’ve marked interesting, and have them automatically classify text, we can use the google translate API to help identify foreign words, and online dictionaries to find synonyms.

Based on the answers to these two questions, it seems to me that the following three “finding a snippet-of-text” problems might really benefit from a little information retrieval and visualization, because finding them by reading is especially hard, and formulating them as search queries can be difficult:

1. Find me more examples like this
2. Find me other other places in the text where this happens
3. Show me all the places in the text where this concept comes up

In conclusion

Together, the survey and interviews helped me understand the mechanics of scholarly reading. As I interpret it, humanities scholars working around textual primary sources follow a process like this (and this is why I’m blogging, so you can all violently disagree with me in the comments):

1. Scholars begin a project with a (sometimes vague) hypothesis, interest, or interpretation in mind
2. They read primary sources to solidify their understanding and find evidence for their arguments
3. They notice or realize things while reading passages from the text
4. They copy out the passages if they are sufficiently thought-provoking, provide evidence, or are relevant to an interpretation.
5. They add information to the passages they copy out:
   1. Why the passage is relevant/ how it fits into their argument
   2. Ideas they got from the passage
   3. Citation information so they can find it and cite it properly later
6. If they haven’t already collected the necessary supporting material, they search, read and re-read other texts to find more support.
7. When they find supporting or relevant passages, they copy them out (see step 5)
8. They curate their collected passages (“pieces of argument”) into a written product representing their argument

Laying out these steps has helped me come up with design requirements. For example, even if no extra visualizations and analysis tools are added, and the interface only supports reading, these steps told me the basic “reading tools” my system must have:

1. Copying out variable-length snippets of text into a note-taking area
2. Preserving the link between the copied-out snippet and its place in the text
3. Taking notes around a snippet
4. Tagging and keyword search
5.  Exporting all of the above into a format compatible with MS Word or similar, so that scholars can integrate it with their other work

Next, if we’re talking about systems with keyword search functionality, steps 6 and 7 make it clear that the ability to save, organize, and annotate search results is key. This is in addition to the ability to switch seamlessly between looking at search results and reading the source text surrounding any particular search result.

If we want to get more sophisticated with information retrieval (which I do), the three “finding snippets of text” problems  I identified above need attention.

But what about when you have visualizations? Coming back to my original question, where do these fit in? I’m still not sure, but I think the steps I’ve found above will give me a good place to start looking for answers.

Theme-finding by searching for words poses a problem. Synonymy and the infinite variance of language mean that the same theme might surface in many different forms using many different words.  Even for scholars with intimate knowledge of the text, a single set of words is not enough. Depending on their mental context, the  words that come to mind might not always be complete and representative.

For example, take the Shakespearean theme of “seeing is believing” — that seeing an event with one’s own eyes is more credible than hearing about it second-hand. A scholar might search for the words “believe”, “speak”, “eyes”, and “see”. That search might be able to capture this example (from The Winter’s Tale 5.2):

Then have you lost a sight, which was to be seen, can not be spoken of.

but not this one (from King Lear 4.6):

I would not take this from report; it is, And my heart breaks at it.

As a solution, we  at WordSeer propose search-by-example. This technology dates back to the 80′s in the field of information retrieval, and so far, it’s been successful in helping find relevant documents. We think it could work for theme-finding too.

With search-by-example, instead of inferring which words represent a theme, and then searching for those words, a scholar can search for sentences that match a set of examples. A scholar marks a set of examples of a theme, and the system returns a list of sentences it thinks are relevant.

This process is a cycle. When the system returns results, the scholar gives it feedback by labeling sentences “relevant” if they match the theme, and “not-relevant” if they don’t. The system gradually builds a model of what the scholar is interested in, and eventually returns results that are mostly relevant.

For example, in under five minutes, I was able to use the examples above to come up with seven more candidates:

Gracious my lord, I should report that which I say I saw, But know not how to do’t. (Macbeth 5.5)

Most noble sir, That which I shall report will bear no credit, Were not the proof so nigh. (Winter’s Tale 5.1)

I would not hear your enemy say so, Nor shall you do mine ear that violence, To
make it truster of your own report Against yourself: I know you are no truant. (Hamlet 1.2)

If in Naples I should report this now, would they believe me? (The Tempest 3.3)

They call him Doricles; and boasts himself To have a worthy feeding: but I have it Upon his own report and I believe it; He looks like sooth. (Winter’s tale 4.4)

It is not so; thou hast misspoke, misheard; Be well advised, tell o’er thy tale again: It can not be thou dost but say’ tis so: I trust I may not trust thee; for thy word Is but the vain breath of a common man: Believe me, I do not believe thee, man; I have a king’s oath to the contrary. (King John 3.1)

I do beseech you, either not believe The envious slanders of her false accusers; Or, if she be accused on true report, Bear with her weakness, which, I think, proceeds From wayward sickness, and no grounded malice. (Richard III 1.3)

Of course, this is all theory until it’s been proven to work. And while I’m not a Shakespeare scholar, I did build this particular system, so it might not be surprising that I can get a few results out of it.

So to find out whether search-by-example works, we’ve designed a five-minute study around three Shakespearean themes. There are three systems: one search, and two different example-based ones. Participants are shown an example of a theme, and asked to use a system to find as many relevant results as they can in five minutes. The systems and theme are randomly assigned.

We’ll find our answer by comparing the quality and quantity of the sentences the participants find on the three systems. Expert scholars will help us judge quality: they will rate the relevance of sentences the different systems produce (without knowing which system produced which sentence). For quantity, there is a time limit — which system produces more high-quality  results in five minutes?

So, does example-based exploration work better than search for theme finding?

If you have five minutes, you can help us find out by participating in the study:

http://wordseer.berkeley.edu/themes/

It’s been guided by the advice of our long-suffering literature-scholar collaborators. And by the tales of frustration and trial-and-error of the students of the Hamlet class who tried to use WordSeer to analyze parts of the play. We also thought hard about the text analysis process as a series of steps. “What might Tanya Clement have been thinking and doing at each stage of her computational analysis of repetition in Gertrude Stein’s The Making of Americans“?  ”What about when we analyzed language use differences in the descriptions of men and women in Shakespeare?” Out of this has come a better (we hope) understanding of the needs of scholars of text in the humanities.

We’ve completely rebuilt WordSeer. Instead of a traditional web application with a different visualization on each page, WordSeer now works more like an environment. Almost like a desktop — with windows and menu bars and persistent, useful, objects.

However, as researchers in Human-Computer Interaction, we know that we need to do user studies. First, we need to check whether we’re on the right track. Do our improvements make for a better experience than the old version? More importantly, we need more observations. To understand the humanities text analysis process, we want to observe more humanities text analysis.

Until now, the closest we’ve come to “user studies”  is an iterative bouncing-around of ideas with just three scholars. They have been more like guides and expert consultants than “users” and they helped us sketch the first lines, and refine our first ideas into something that was actually useful.

We’ve acted upon the knowledge they helped us accumulate, the result of which is the completely redesigned WordSeer. We’re looking for a bigger set of users now, for a formal study. We’re hoping to find a set of around 15 professional literature scholars who will allow us to observe them as they use WordSeer to explore a problem of genuine professional interest to them.

So what text collection could possibly interest 15 different scholars in the digital humanities community enough to want to do a computationally-assisted analysis of it? And allow us to observe them at it?

In a rare moment of epiphany, we realized we could just ask you. So here’s a poll. It’s populated with some examples, but we encourage you to respond in the “other” field. Tell us: what collection, if set up with text analysis and visualization tools,  would make you interested?

[polldaddy poll=6382760]

In this post, I demonstrate the collection and comparison features of WordSeer by using it to compare the usage of the word “love” in Shakespeares comedies and tragedies. You can watch the screencast, or simply read on.

[youtube http://www.youtube.com/watch?v=DPhQQExQjZ4]

Figure 1. Creating a new collection called "tragedies"

The first thing to do is collect the comedies and tragedies into separate lists.  To do this, I created a new collection called “tragedies” using the new “collections” feature.

Figure 2. The list of plays in WordSeer, sorted by title.

Next, I had to collect all of Shakespeare’s tragedies into that collection.  Figure 2 shows WordSeer’s list of plays.  I walked down this list and clicked the checkboxes next to the tragedies, using Wikipedia as an authoritative source of tragedies.

Figure 3. The Add Items button

Once I’d selected all the tragedies, I  clicked  the “Add Items” button to add them to a collection.  I selected the “tragedies” collection and added the plays.

Figure 4. Adding some of the tragedies to the collection

This populated the collection with the plays. I did the same for the comedies, ending up with two collections

Figure 5. The two collections. The "comedies" collection is currently open.

I was now ready to compare my collections. I opened up two windows to the heat map view. One was going to visualize the tragedies, and one the comedies.

Figure 6. Setting up the heat maps. One window visualized the "tragedies" collection, and the other window visualized "comedies".

Finally, I was ready to compare the two. I was interested in the word “love”, and whether there would be any differences in how frequently it was used in the comedies and the tragedies. To that end, I typed in “love” into the comedies window and got the heat map in Figure 7.

FIgure 7. The occurrences of "love" in Shakespeare's comedies. Each column is a play, each highlighted block represents that the word "love" occurred there.

Not surprisingly, “love” is everywhere. But what about the tragedies? In the other window, typing in “love” yielded the results in Figure 8.

Figure 8. The occurrences of "love" in Shakespeare's tragedies.

To my surprise, the tragedies were equally full of “love”. Which, among other things, reveals my poor knowledge of Shakespeare.

Still, the hope is that  our Shakespeare scholar, Michael Ullyot, (@ullyot) will use collections and heat maps to discover something truly interesting.

In this post, I give an example of how our project WordSeer, a text analysis environment for humanities scholars, can be used to overcome this problem. In this example, I’ll using an instance of WordSeer running on the complete works of Shakespeare from the Internet Shakespeare Editions. It’s live, so you can follow along with this example on the web at wordseer.berkeley.edu/shakespeare.

You can read the post after the jump, or just watch this video.

[youtube http://www.youtube.com/watch?v=OXkuOzl9GrI]

I began with a simple question, “What are some things that are ‘beautiful’ in Shakespeare?”. Normally, this would be a challenging question by itself. WordSeer, however, uses grammatical search. This is a search feature that goes beyond keyword search and instead also searches over grammatical relationships between words. These are relationships such as subject-object and modifier-subject. For example, in the sentence “The good God has given every man intellect”, there is a relationship between “good” and “God” — “good” is an adjective that modifies ”God”. There is also a verb-agency relationship between “God” and “given”, and a verb-object relationship between “given” and “man”. For more than a decade, it has been possible to automatically extract such relationships from text using computational linguistics algorithms. WordSeer uses these well-known algorithms to analyze the works of Shakespeare (in this case) and allow users to search over them.

Figure 1: The query "____ described as beautiful", which retrieves all words to which the adjective "beautiful" has been applied.

In the case of my question, “What are some things that are ‘beautiful’ in Shakespeare?” I can use grammatical search to good effect, as shown in Figure 1 above. I leave the left-hand-side box blank to retrieve all matches, select the “described as” grammatical relationship, and put in “beautiful” to create the fill-in-the blanks query “_____ described-as beautiful”. I press “Go” to search, and WordSeer returns all the matching sentences. These are sentences containing the adjective “beautiful”, applied to some other word. I get the results in Figure 2.

Results for the query _____ described-as beautiful. To my alarm, there was only one match!

To my alarm, there was only one match: the sentence “His youngest daughter, beautiful Bianca”, from The Taming of the Shrew. Concerned that my algorithms had malfunctioned, I did a simple search for the word beautiful, without any grammatical relationships. Lo and behold, there were only 16 results (Figure 3).

Figure 3. Search results for the word "beautiful" in Shakespeare. There were only 16 results.

I had encountered the vocabulary problem. Not being a Shakespeare scholar, I couldn’t think of any other words that could have been used instead of “beautiful”, and it seemed preposterous that these were the only results in all of Shakespeare. There must be other words.

Figure 4. Clicking on the "book" icon next to a search result opens a new window with the full text of the play, and automatically scrolls to, and highlights, the matching sentence.

To investigate further, I decided to read some of the context around the word “beautiful”. To do this, I clicked on the “book” icon to the left of my “beautiful Bianca” search result.  This brought up a new window (Figure 4) with the full text of  The Taming of The Shrew, opened up to  the exact line matching the search result.

Figure 5. Right-clicking on "beautiful" (or any other word) brings up synonyms of that word -- computed based on similar usage to the query word.

After convincing myself that “beautiful” did actually mean what I thought it did in Shakespeare, I decided that I needed to see synonyms. WordSeer supports this need. Using the contexts of words, it computes synonyms based on other words that “behave” in the same way — that are used in the same contexts, that have grammatical relationships to similar words, and so on. Right-clicking on a word while reading (Figure 5) brings up synonyms. These are computed based on being used in a similar way to “beautiful” in Shakespeare, and not based on some external measure of similarity, such as a dictionary or thesaurus. Therefore, they reflect the particular idiosyncrasies of just the Shakespeare collection.

Figure 6. Adding some interesting synonyms to the heat map query -- click a word to add it to the heat map query.

This list of synonyms seemed promising. It contained words such as “tractable”, “fair”, and “gentle”, that I would never have thought of including in my initial search.  To investigate whether these were more widespread, I decided to investigate their prevalence in the collection using WordSeer’s heat map tool. I clicked some interesting words and added them to my query (Figure 6). This took me to the heat map view.

Figure 7. The collection-wide occurrence of some of the synonyms of "beautiful". It is easy to see that these are much more prevalent.

WordSeer heat maps can be confusing if you have never seen one before, so I’ll explain them here. If you know what this means, you can skip this section.

Heat Maps

WordSeer uses heat maps to visualize collection-wide occurrence patterns of words and phrases. In this example, I’ll use  the word “fair” to illustrate.

Figure 8. The column corresponding to "Macbeth"

Typing in “fair” creates the pretty picture in Figure 8 above. Each vertical column is a single document — in the picture, I am hovering over the column corresponding to “Macbeth”.  The documents are lined up side by side in long vertical columns.

Figure 9. The column corresponding to "A Midsummer Night's Dream"

All of Shakespeare’s works are here . Figure 9 shows the column corresponding to “A midsummer night’s dream”.  The blue highlights show occurrences of the query. In each vertical colum, blue blocks indicate that the query word, in this case “fair”, has occurred in that location. Blocks higher up in the column mean that the word occurred near the beginning, and blocks lower down in the column mean that the word occurred towards the end of the document. The documents all “appear” the same length, so shorter documents are “stretched” (a few taller blocks) and longer documents are squeezed (many squat blocks).

Figure 10. The blue highlights show where the query word has occurred in each document.

Hovering over a highlighted block brings up a window showing the matched sentence. In this case, I’ve hovered over a line containing “fair” from “The Tempest”. The popup shows the line, with “fair” highlighted in the same color, and a book icon. Clicking the icon opens up a new window, in which I can  more of the text if I wish to.

But back to the the vocabulary of beauty

Figure 11. Heat map showing synonyms of the word "beautiful"

The heat map I got from my synonym query (Figure 11. above) showed that, although “beautiful” was quite a rare word, other synonyms for it seemed much more prevalent. “Fair”, in particular, seemed to be used a lot — the whole map was purple. Hovering over individual instances of “fair” showed that it did seem to be used the way “beautiful” is in today’s english.

The word tree for "fair". This is a visualization of the contexts surrounding the word "fair".

For further verification, I looked at the word tree for beautiful (Figure 12), which was displayed on the same page just below the heat map. It showed that “fair” was used in constructions like “fair and virtuous”, “fair and happy”, “fair and good”.

Figure 13. A revised query: "_______ described-as fair"

All this evidence convinced me that I might be better to use the word “fair” instead of “beautiful” to investigate the concept of beauty in Shakespeare. Returning to the search page, I typed in a new grammatical search query – ”_________ described-as fair”. The results (Figure 14) were much more informative:

Figure 14. Search results for the grammatical search query "_________ described-as fair"

Because there was more than one result this time, WordSeer showed a bar graph summarizing the matches. At a glance, I could see that I was on the right track.  There were a lot of women’s names, interspersed with other words like  ”queen”, “daughter”, and “day”. It seemed that I had successfully overcome  the vocabulary problem, at least this time. I had  a starting answer to my original question, “what are some things that are ‘beautiful’ in Shakespeare?”

The goal of text analysis interfaces like WordSeer is to properly combine powerful language processing algorithms with easy user interfaces. Neither is be enough by itself, but together, they allow users to progress naturally from step to step, assisting them through an analysis.

WordSeer is a new way of searching through text inspired by the way literary scholars work. Literature scholars ask detailed, analytical questions of text, for which it’s important for them to get a sense of how different words are used and in what contexts. For our project, we teamed up with scholars who are exploring language use in a collection of North American slave narratives.

When analyzing text, traditional keyword-based search can only take you so far. Instead of having to read every document hoping to come across relevant passages, you can immediately zoom in on them with a search. But can we do better? When trying to form a hypothesis or get a sense of contents, a long list of search results is still unwieldy  because it’s not really the matching sentences we’re interested in, it’s what they have to say about our topic.

Luckily, we don’t have to stop at matching keywords. Sentences aren’t mysterious bags of words, they follow rules and have structures, which computers have been capable of deciphering with speed and precision for some years now. From these structures, computers can automatically infer relationships between words. For example, in the sentence,

“The good God has given every man intellect”

computers can automatically infer that “God” is described as “good”, and that he is the agent doing the giving.

With WordSeer, we’re going beyond keyword search by using language processing to automatically extract and aggregate the parts of matching sentences relevant to a query. In the first place, we make it easy to express an analytical query in terms of a grammatical relationship. For example, if a scholar wanted to know what the slave narratives collection indicated about the relationship between slaves and God, they could simply ask (live demo link) how God  ”is described” (for which WordSeer finds and displays all the adjectives that are applied to the word God) and what “is done by“ God (for which WordSeer finds and categorizes all instances of verbs in which God is an agent).

Of course, this is only a rough, high-level picture of what the slave narratives say about how God is described and what God does, but a rough idea can often serve to guide intuition and help generate or discredit hypotheses. By making the process of “getting a rough idea” quick and inexpensive, we can speed up the entire research pipeline.

Zoom in to the field of American literature, and further into the realm of studying the (digitized) narratives of escaped former slaves, published by white abolitionists. There are widespread stylistic and thematic similarities among these narratives. How can text mining help literature scholars here? That’s where WordSeer, my latest project, comes in.

The MONK project at CMU, and the Voyeur project at McMaster University share the same cause as WordSeer. But, when it comes to text analysis, they are essentially search interfaces that show simple statistics about word order, type and frequency. The grammatical relationships within text are neglected.

WordSeer

WordSeer is an evolving project, as all digital humanities projects inevitably are. As my friends in the English department and I learn what we can do for each other, it will get steadily more well-defined, but right now, it’s simple: a search interface and a reading interface. The search interface allows queries based on grammatical structure, and the reading interface is for reading narratives, comparing them, and coming up with new queries.

Search

The search screen is shown below. It supports standard keyword-based search, so scholars can look for words or exact matches in the text. More interestingly, there’s grammatical search. Using grammatical relationships extracted through natural language processing, users can ask how things were described, what actions were performed upon them and by them, who possessed certain things, or what was possessed by them.

For example, the figure above (click for larger image in new window) shows the query, ‘give all adjectives that are applied to the words “slave, bondman, negro”‘. The system returns not only a list of occurrences in the narratives, but also automatically-generated graphs, showing the frequencies of the different words. As you can see, “poor” is the most frequent adjective. The results are sortable, and filterable: clicking on bars filters the list to show just results containing those words. Above, I’ve filtered to show just the instances where “valuable” is applied to “slave”.

Reading

Interviews with our literary scholar friends suggested that a search interface alone would not be enough, so WordSeer supports reading narratives individually.

The reading view is shown below. Scholars can select one (or, indeed many) sentences from the search results and be taken to a reading screen, where the narratives are opened up to the correct place. Grammatical search doesn’t end there, however, because the entire text is interactive.

Highlighting a portion of a sentence and clicking the “examine” button (bottom right corner) shows the text pattern, as well as all the grammatical relationships in the highlighted portion. For example, I clicked on a passage about hospitals, and was presented with the pattern-examiner screen (below).

I can select some patterns, either the original passage or some grammatical patterns, and examine them further. I can use them as search queries and be taken back to the original search screen, I can save them for later, or I can view their distributions in the text I’m reading.

Being able to compare the distribution of phrases or patterns across texts can give an idea of how similar the texts are, or of how much their subject matter overlaps. For example, if I wanted to know where plantations were mentioned in these texts, I would highlight the word, “plantation” and click “See in Text”, giving the result below.

The white column represent the length of the entire text, and green bars indicate that the pattern of interest occurred. If I had selected multiple patterns, I would see different colored bars.Clicking on any of the little green bars takes me to an occurrence of the pattern, highlighted in the text.

Language Processing

All of this works because I applied language processing to the text beforehand, and stored the information a database for quick access. I applied part-of-speech tagging, syntactic parsing, and dependency parsing to decompose sentences into their grammatical constituents. For example, the sentence, “The cruel man beat us severely” contains the word “cruel” which is an adjective modfier of the word “man”, which is a noun. There is verb object relation between “beat” and “us”, and a verb subject relation between “man” and “beat”.

If you want to know more about natural language processing, I gave a BootCamp about text mining at THATCamp SF recently, here are the slides [pdf]. I also wrote a blog post introducing the subject for a digital humanities audience.

What next?

Syntactic analysis is just a small part of what natural language processing can do. Right now, I’m working on being able to track named entities through a narrative and see descriptions applied to them, and actions in which they participate.

We present a method for extracting social networks from literature, namely, nineteenth-century British novels and serials. We derive the networks from dialogue interactions, and thus our method depends on the ability to determine when two characters are in conversation. Our approach involves character name chunking, quoted speech attribution and conversation detection given the set of quotes. We extract features from the social networks and examine their correlation with one another, as well as with metadata such as the novel’s setting. Our results provide evidence that the majority of novels in this time period do not fit two characterizations provided by literacy scholars. Instead, our results suggest an alternative explanation for differences in social networks.

The paper advances a new technique for extracting social networks from text, and uses it on 19th century novels to argue that certain aspects of literary theory about novels might be false. In this post, I’ll explain the analysis to the digital humanities audience and discuss some strengths and weaknesses in the argument.

Written at Columbia University by two computer scientists and one English scholar, this paper contains exciting things to both computational linguists and literature researchers. For computational linguists, it proposes the first ever algorithm for extracting speaker-to-speaker networks from free text. This opens up fascinating new areas of study because it is now possible to computationally analyze interactions between people in a text and not just what they say to each other.

For literary scholars, it suggests two hypotheses from literary theory about community and society in 19th century novels might be false, namely:

Literary studies about the nineteenth-century British novel are often concerned with the nature of the community that surrounds the protagonist. Some theorists have suggested a relationship between the size of a community and the amount of dialogue that occurs, positing that “face to face time” diminishes as the number of characters in the novel grows. Others suggest that as the social setting becomes more urbanized, the quality of dialogue also changes, with more interactions occurring in rural communities than urban communities. Such claims have typically been made, however, on the basis of a few novels that are studied in depth. In this paper, we aim to determine whether an automated study of a much larger sample of nineteenth century novels supports these claims.

To make their arguments, the authors frame the statements above in terms of social networks:

• If face-to-face time diminishes as the number of characters grows, then the more characters the novel has, the less dense its extracted social network will be.
• Second, if more interactions occur in rural settings than urban settings, networks from rural novels will be densely connected, but contain fewer characters, but networks from urban settings be large and loosely connected.

Then, they extract social networks from novels using the following steps. First, the Stanford named-entity tagger automatically locates all the names in each novel. Then, a classifier automatically assigns a speaker to every instance of direct speech in the novel using features of the surrounding text. A “conversation” occurs if two characters speak within 300 words each other, and finally, a social network is constructed from the conversations. Nodes are named speakers (that appear 3 times or more – the named-entity tagger is somewhat error prone). Edges appear if there was a conversation between two characters, a heavier edge means more conversations. The end result is a social network like the one shown above, which was extracted from Mansfield Park by Jane Austen.

Using the social networks they extract, the authors show that there is no significant difference in this dimension between urban and rural novels. Instead, they show that the biggest differences seem to be between novels in the third person and novels in the first person – the third-person novels have “dense, talkative” networks, whereas the first-person novels all center around the character “I”:

Our data suggests … that the “urban novel” is not as strongly distinctive a form as has been asserted, and that in fact it can look much like the village fictions of the century, as long as the same method of narration is used

Their claim seems too strong to me. In order to make the problem tractable, they have reduced the concepts of “characters” and “conversation” to simple metrics, but important information isleft out:

• Characters are equated with names that appear more than thrice, but this leaves out
  • Named characters that are mentioned less than 3 times
  • Nameless characters that don’t speak
• They assume all conversations are direct speech – they ignore indirect and reported speech

From an NLP perspective, it’s easy to see why they’ve made these simplifications. In the kinds of text that we are used to dealing with: expository things like news articles, or explanatory things like journal papers, infrequent, nameless entities don’t matter and are rare. We’re used to looking for significant entities, popular topics of discussion, so the “drop the infrequent” approach goes a long way, and eliminates noise.

Nevertheless, when it comes to characterizing the aesthetics of a novel’s depiction of a social network, it’s a different matter, no longer about how “important” a character is or how “significant” some topic is. To me, it seems plausible that the number of infrequently-appearing named characters, and the number of nameless characters who are seen but never heard chan change the quality of a social network one experiences in a novel. Without further investigation into how frequent these infrequent-character, or nameless-character-cases are in this particular corpus, I really don’t think they have enough data to claim to have refuted scholarly intuition.Without further investigation, we have no idea whether the cases they leave out are frequent or infrequent enough in urban novels to sway the analysis, and in which direction the decision would go.

It’s very easy to point out problems, but I can’t think of any ways to fix them: the tools to separate infrequent named entities from junk entities just don’t exist. And the tools to identify nameless, speechless characters haven’t made much headway either – what is the difference, quantitatively, between the words “the woman standing at the station” and “the hansom cab standing at the station”? To computers that rely on statistical, automatically extracted information about language, the difference is currently very difficult to detect.

What do the humanists among you think of this work? Compared to the other literary analysis of the same novels done at Stanford, this approach is more linguistically sophisticated – but do all of these computational attempts seem heavy handed? Or do they spark ideas in your head, inspiring you to apply and improve them on your own problems?

Throughout this post, we’ll try to answer a hypothetical question a scholar in the humanities, perhaps a literary scholar or historian, might be interested in:

“How is the character Mary talked about in this novel or historical text?“

It’s fairly open ended – what does “talked about” mean? How do we translate this into computational terms? In this post, I’ll describe some tools that natural language processing (NLP) has to offer, and show how each can be used to tackle this question along with pointers to sofware and tutorials.

The goal of of NLP is to model the workings of natural language as we speak, read, and write it, so all the tools here are motivated by some kind of language model.

N-Grams

These are strings of consecutive words within a sentence. Take the sentence,

Mary was born on a cold March morning.

The words born and morning are 1-grams. cold March and born on are examples of 2-grams. This might seem like a crude way of modeling a language, but n-grams capture a lot of information because we speak grammatically. We can use them to get a sense of how Mary is talked about, for example, by asking what 4-grams we can find that start with:

• Mary is ____.
• Mary is a ____.
• Mary is an ____.

Toolkits like NLTK and openNLP come with tutorials that explain how to get started on such analyses. We may find that Mary is nice and a darling, but of course we may end up with a fragment like Mary is getting, from the sentence Mary is getting sleepy. which isn’t what we’re looking for.

N-grams capture information about grammar through frequency of use: the less frequent an n-grams is, the less likely it is to be grammatical. But as early as 1957,  Noam Chomsky argued that there is much more to modeling grammar than this in his famous sentence,

“Colorless green ideas sleep furiously.”

which is perfectly grammatical even though it contains no frequent n-grams.

Parts of Speech

Parts of speech (POS) are a more detailed way of modeling of how we use words: verbs refer to actions, adjectives describe properties of things, nouns refer to entities, and so on. NLP algorithms have long been capable of assigning part-of-speech labels to words in sentences with high accuracy. This task is called POS-tagging, and we can use it to refine our analysis of how Mary is talked about by asking, “What are the adjectives that occur within five words of ‘Mary’?”. From a fragment like:

It was a cloudy day, which young Mary found fortunate. “Are we close yet?” her companion asked. “No, actually, it’s quite far,” Mary replied.

Using a part-of-speech tagger such as the Stanford POS tagger, GENIA, or one of the many taggers that come with NLTK, to extract adjectives, we would get a list like:

• cloudy
• young
• far

While this is more precise than an n-gram analysis in that we only see adjectives now, it’s still not perfect because only young refers to Mary. This is because “within five words” is still an approximation for what we really want: adjectives that refer to Mary. We could try this analysis again, with a smaller window 1 or 2 words, but then we might miss many adjectives.

Parsing Phrase Structure

The structure of natural language extends beyond parts of speech, because words have relationships with each other. For example, in English, we say that every sentence has a main verb, which has a subject and, depending on the verb, an object, and an indirect object as well. These constituent parts can be small units like nouns, or bigger units, phrases which have their own constituents. NLP algorithms called parsers analyze sentences and return their internal structure. The Berkeley Parser, for example, parsed the following sentence:

She thinks Mary is nice to animals.

to give:

where the symbols on each branch represent parts of speech and phrase types. For example, ADJP is an adjective phrase, NP a noun phrase, and VP a verb phrase. Here is a description of the standard set of labels.

We can use these concepts to ask very precise questions now. Referring to the tree above, if we’re searching for descriptions like “Mary is ____”, we’re searching for ADJP‘s (adjective phrases)which are part of a VP (verb phrase) containing the word is, and which immediately follows the word Mary.

The easiest parser to use is the Stanford Parser, which parses about 4-5 sentences a second. Using their Tregex software (which is a little harder to use), you can browse the output and search for specific patterns like the one above.

Dependency relations: grammatical structure:

The most precise way to ask which adjectives describe Mary is to look directly at grammatical relationships, and ask which adjectives modify Mary.  Modern parsers can do this accurately. For example, the Stanford Parser could look at the phrase structure in the sentence above (Figure 1) and return the following representations:

• nsubj(She, thinks)
• ccomp(thinks, is)
• cop(nice, is)
• nsubj(nice, Mary)
• xcomp(nice, animals)

The relation nsubj between nice and Mary indicates, for example, that nice is what  Mary is. Parsers that extract these types of relationships are called dependency parsers, they extract these grammatical relationships from phrase structures like the one in Figure 1. The Stanford Parser is one of many that includes this ability, and here [PDF] is a list of all the dependency relationships it can extract.

A Phrase Net of ” ____ and _____ ” relationships in Pride and Prejudice

Using dependency parsers gives us a lot of power: we can ask for all the adjectives that apply to Mary and locate them with high accuracy. We can find the verbs of which Mary is a subject, and those of which she is an object and see if there are any interesting patterns, or we can look at all the conjunctions in which Mary participates. A visualization like the one above, specifically designed for visualizing grammatical relationships (more here), might then make excellent food for thought.

Topic modeling: a statistical approach

With the availability and relative popularity of topic modeling algorithms in machine learning toolkits like Mallet, it would not be appropriate to leave this class of analysis out of my post.  Topic models were originally developed as a way to represent a large collection of documents in a compact way, but are interesting to more people now because the “topics” they produce can sometimes correspond to coherent concepts.

One way of representing a document in a compact way is by representing it as a set of word counts. This bag-of-words contains no information about relative ordering, only information about co-occurrence. Topic modeling is motivated by the idea that there are more words in a language than topics to which they belong, so documents can be represented even more compactly by a set of topics, where a topic itself encodes some distribution of the probability of words. For example, one can imagine that every article in the literature on psychology can be compactly represented by its proportions of a vocabulary of topics such as experiments, personality, drugs, theories, cognition etc.

Below are the most frequent words in the 9 most frequent automatically-extracted topics in the abstracts of the Psychological Review, extracted using this topic modeling toolbox.

‘similarity bias strategies drug systematic biases conditions’
‘order serial search process parallel elements attention’
‘stimulus response stimuli responses color cs increase’
‘ss s change rate normal underlying practice’
‘self individual situations individuals those others consequences’
‘environment general behaviors constraints internal other external’
’2 experiments single results experimental high trial’
‘personality variables measures research consistency issues cross’
‘pattern patterns changes critical false food sequences’

This method can be applied to any text, and can give interesting results when paired with humanistic intuition. For an illustrative example from the digital humanities (much better than any I could make up in involving Mary), read the work of  Cameron Blevins, a history Ph.D. student at Stanford, who has used topic modeling to glean relationships and trends from a text he was studying: Martha Ballard’s diary. Finally, for an excellent, and thorough, introduction aimed at the digital humanities audience, I can’t think of a better piece than Scott Weingart’s guided tour of topic modeling.

The techniques I’ve talked about here are building blocks. Natural language processing algorithms exist for many more complicated (and potentially more useful) purposes: named entity recognition,  semantic similarity calculation, relationship extraction, opinion mining, pronoun resolution, summarization, question answering, translation…

There are many tools, and they’re probably very badly documented, but hopefully I’ve managed to advance the case for considering sophisticated language processing like this part of the natural toolkit of the digital humanities.

You and other data geeks can ask and answer questions on machine learning, natural language processing, artificial intelligence, text analysis, information retrieval, search, data mining, statistical modeling, and data visualization.

Here you can ask and answer questions, comment and vote for the questions of others and their answers. Both questions and answers can be revised and improved. Questions can be tagged with the relevant keywords to simplify future access and organize the accumulated material.

I’ve never been a forum participant, but finally, something I can use! This community was so spread out and disconnected that the best way to get advice about these topics used to be to walk up to one of your colleagues and hope they’d dealt with the same problem before. But, brought together by this forum, we can give each-other informative answers to obscure (but terribly-important-at-the-moment) questions about stuff we work on every day.

(Thanks to Flowing Data for the tip.)