Extensive Vocabulary (Due 12 August 2011)

For this assignment, we would like to study the representative works of two authors and ask ourselves the question "Whose vocabulary is more extensive?"

The authors that we will be looking at are Charles Dickens and Thomas Hardy. And the representative novels that we will analyze are A Tale of Two Cities and The Return of the Native respectively. Here are the following steps in our analysis.

Step I: Go to the Project Gutenberg and download the plain text versions of both books. Call the first book Tale.txt and the second book Return.txt. Open the books in a text editor and delete the preamble in the beginning of the books and the license agreement and the closing blurbs at the end of the books. The first book in Tale.txt should begin and end with these lines: 

A Tale of Two Cities

by Charles Dickens

...

it is a far, far better rest that I go to than I have ever known."
The second book in Return.txt should begin and end with these lines: 
THE RETURN OF THE NATIVE

by Thomas Hardy

...

kindly received, for the story of his life had become generally known.

Step II: The program that you will be writing will be called Books.py. The following is the suggested structure. You do not have to adhere to it. However, we will be looking at good documentation, design, and adherence to the coding convention discussed in class. Use meaningful variable names in your program. 

# Create word dictionary from the comprehensive word list 
word_dict = {}
def create_word_dict ():

# Removes punctuation marks from a string
def parseString (st):

# Returns a dictionary of words and their frequencies
def getWordFreq (file):
  
# Compares the distinct words in two dictionaries
def wordComparison (author1, freq1, author2, freq2):


def main():
  # Create word dictionary from comprehensive word list
  create_word_dict()

  # Enter names of the two books in electronic form
  book1 = raw_input ("Enter name of first book: ")
  book2 = raw_input ("Enter name of second book: ")
  print

  # Enter names of the two authors
  author1 = raw_input ("Enter last name of first author: ")
  author2 = raw_input ("Enter last name of second author: ")
  print 
  
  # Get the frequency of words used by the two authors
  wordFreq1 = getWordFreq (book1)
  wordFreq2 = getWordFreq (book2)

  # Compare the relative frequency of uncommon words used
  # by the two authors
  wordComparison (author1, wordFreq1, author2, wordFreq2)

main()

Step III: Declare a global dictionary variable word_dict. In the function create_word_dict() open the file words.txt and populate the dictionary. Each word in the file will be a key in the dictionary and the value will be 1. Hard code the name of the file, words.txt, in your program. When we test your program we will be using a file with the same name.

Step IV: You will have to get the frequency of words in each text to start with. We have already developed a piece of code to do that. But here are some additional processing that you will have to do in the function getWordFrequency().

You should now have a dictionary of words in lower case and their frequencies. You will have removed all proper names of people and places. You will have added the names of people and places that are common words like Green and Bath. But such words should be few compared to the total number of words that we are dealing with in those novels. You can always write the list of words beginning with a capital letter in a file and examine that file to check if the deletions were properly made.

Step V: In this step you will be working on the function wordComparison(). First you will get some statistics of the two novels separately and then you will compare the two together. For each novel compute and print the following pieces of information:

You will create two sets with the list of keys from the two word frequency dictionaries. Let us call these sets D and H for the two authors respectively. The set difference D - H represents all the words that Dickens used that Hardy did not. The set difference H - D represents all the words that Hardy used that Dickens did not. For each of these set differences print the following pieces of information:

Here are two sample files - dickens.txt and hardy.txt taken from the two novels. Your output for these two sample files should be of the following form: 

Enter name of first book: dickens.txt
Enter name of second book: hardy.txt

Enter last name of first author: Dickens
Enter last name of second author: Hardy

Dickens
Total distinct words = 58
Total words (including duplicates) = 119
Ratio (% of total distinct words to total words) = 48.7394957983

Hardy
Total distinct words = 93
Total words (including duplicates) = 123
Ratio(% of total distinct words to total words) = 75.6097560976

Dickens used 50 words that Hardy did not use.
Relative frequency of words used by Dickens not in common with Hardy = 63.8655462185 

Hardy used 85 words that Dickens did not use.
Relative frequency of words used by Hardy not in common with Dickens = 77.2357723577
Here is the output of the frequencies of the words in the two excerpts - dickens.out.txt and hardy.out.txt. These outputs have now been checked against the comprehensive word list.

The above program will have a header of the following form: 

#  File: Books.py

#  Description:

#  Student Name:

#  Student UT EID:

#  Course Name: CS 303E

#  Unique Number: 

#  Date Created:

#  Date Last Modified:

This is an exercise in Computer Science and not in linguistics. What we are trying to illustrate is how easy it is to perform these types of computation in Python that lead to interesting observations but not necessarily deep insights.

Use the turnin program to submit your Books.py file. The TAs should receive your work by 11 PM on Friday, 12 August 2011. There will be substantial penalties if you do not adhere to the guidelines.