/u/mooney/ir-code/ir/vsr/
. See the Javadoc for this system. Use the main
method for
InvertedIndex to index a set of documents and then process queries.
You can use the web pages in
/u/mooney/ir-code/corpora/curlie-science/
and
/u/mooney/ir-code/corpora/cs-faculty/
as the sets of test
documents. The Curlie Science dataset contains 900 pages, 300 random samples each from the
Curlie indices for biology,
physics, and chemistry. The UTCS dataset
contains 1000 pages from the UT CS website.
See the sample trace of using the system on the Curlie Science dataset and on the UTCS dataset.
For example, in the sample trace for the Curlie Science dataset, for the first query "background radiation", the top retrieved document does not contain the phrase "background radiation". Instead, it has 41 occurrences of the word "radiation" and zero occurrences of the word "background". In fact, none of the top 10 retrieved queries contain the phrase "background radiation".
The next sample query "virtual reality" has similar problems that the top retrieved documents do not contain the phrase. The first document includes 42 occurrences of the word "reality" but no occurrence of "virtual". The second document only contains 3 occurrences of the word "virtual". The document that actually talks about "virtual reality" is only ranked number 8. Not until the eighth result does a relevant page that contains "virtual reality" actually appear.
We can see similar examples in the sample trace for the UTCS dataset. For the first query "academic achievements", the top results contain the words "academic" and "achievements" in separate parts of the web page but do not include relevant information about the academic achievements.
For the second query "real world", the top two results are web pages with nothing related to "real world" but different "real time" projects. The third result is relevant and includes the phrase "real-world".
Here are the sample solution traces produced by my solution to this problem or the Curlie Science dataset and the UTCS dataset. Note that the top documents now contain all the query words, close together and in the correct order.
In addition to the normal cosine-similarity metric, I calculated a specific proximity score for each retrieved document that measured how far apart the query words appeared in the document. The final score was the ratio of the vector score and the proximity score (both components are shown in the trace). The proximity score was computed to be the closest distance in the document (measured in number of words, excluding stop words) that a query word appeared from another query word averaged across all pairs of words in the query and all occurrences of the words in the document. A multiplicative penalty factor was included in the distance metric when a pair of words appeared in the reverse order from that in the query. This is only a sketch of what I did, many details are omitted.
You do not have to adopt this exact approach. Feel free to be creative. However, your solution should be general-purpose (not hacked to the specific test queries), address the fundamental issue of proximity, and produce similarly improved results for the sample queries. Note that you may need to change many of the fundmental classes and methods in the code to extract and store information on the position of tokens in documents. When making changes, try to add new methods and classes rather than changing existing ones. The final system should support both the original approach and the new proximity-enhanced one (e.g. I created a specialization of InvertedIndex called InvertedProxIndex for the new verison). Hint: I found it useful to use the Java Arrays.binarySearch method to efficently find the closest position of a token to the occurence of another token given a sorted array of token positions.
Make sure to include the following in your report:
/u/mooney/ir-code/corpora/curlie-science/
/u/mooney/ir-code/corpora/cs-faculty/
Please submit the following to Gradescope:
code/
.
This should necessarily include the main java class called InvertedProxIndex.java
.
In the autograder, the code will be executed as follows: java ir.vsr.InvertedProxIndex -html <path-to-dataset>
report.pdf
trace/curlie.txt
and trace/faculty.txt
. Don't include trace files of the original inverted index.On submitting to Gradescope, your files should look something like this:
After the deadline, we will evaluate your code on a set of hidden queries and autograde the retrievals based on a set of rubrics. These rubrics are yet to be finalized. But a general guideline might be that the retrievals using proximity scoring need to boost the scoring of the multi-word queries and improve their ranking in the top retrievals.
The grading breakdown for this assignment is: