CS 439: Project 3

Memory Management with Multi-level Page Tables

Due Friday, 11/9/12, 11:59PM

Individual or Team of Two

This project will allow you to gain experience implementing a virtual memory system, including frame (real memory) management, virtual memory management, and multi-level page tables.
You are provided with a Simulation framework (much like the previous labs) which simulates a single CPU with an instruction set that’s designed to allow for loads/stores to virtual addresses. This framework provides complete implementations for most components, but only partial CPU and OS implementations.
You are provided with: project3_cs439.zip
wget http://cs.utexas.edu/~rockhold/CS439/project3/project3_cs439.zip

• A complete Simulator (sources)
• A skeleton implementation of an OS class (os.OS)
• A skeleton implementation of a CPU class (cpu.CPU)
  
  OS.java and CPU.java both contain generous amounts of hints and guidelines. Look through these carefully.
• 6 output files named config_?_v.txt (in the outputs_pt subdirectory) that your output must match.
• 6 additional, more-verbose, output files named config_?_vv.txt that you don't have to match, but which you might find useful in debugging.
• Makefile for building, testing, and turning in your project.
• README file you need to complete prior to turnin.

You need to:

1. Complete the implementations of the CPU and OS classes.
2. Produce exactly the same results (in the 6 config_N_v.txt files) for the provided configurations.

To accomplish this, you’ll need to:

• Complete the implementation of class CPU and its inner classes.
• Complete the implementation class OS and its inner classes.

As in most simulators, many options are configurable (to allow for comparing various options.)
Some of the hardware/software mechanisms are fixed.  The fixed onesinclude:

• The use of a two-level page table.
• A frame replacement policy of true LRU.  So your PageTable management code must maintain the order of references made by the CPU while fetching instructions and loading/storing data. You OS implementation must use this ordering when making frame replacement decisions.
  Hint: Simulator.getMonotonicInt() can be very useful for helping implement true LRU frame replacement.
• All pages are Read/Write/Executable.  There is no notion of a read-only or execute-only page.

There are some hardware/software characteristics that are configurable via a properties file (name specified on the command line).  Please refer to the provided config_N.properties files for a description.  The Simulator captures these configuration options in a SystemInfo instance that it passes to both your OS and CPU class constructors.  These options are represented as the log base 2 of the vales, and include [invariants in braces]:

• The size of a page in bytes (and of a frame) [at least 4 bytes (since instructions are 4 bytes)]
• The maximum number of pages available in each process’ address space [at least 2 – since an instruction can reference memory]
• The number of frames of “real” memory [at least 2 – since an instruction can reference memory.]
• The number of entries in the first-level page table [at least 1, not to exceed the number of pages in each process]

Note that your code does *not* need to sanity check these invariants.

There are some requirements that your code must meet that a “real” OS would not have to meet.  This is because your solution must match the provided output.  These include:

• Free frame list management.
  You must ensure frames are selected in the "right" order (to match the expected output), so:
  
  
  • Initially, frames should be added from lowest frame number, to highest (the very first frame selected should be frame 0, then 1, etc.) This ensures that frame 0 is at the head of the list.
  • When a frame is released, it should be added to the tail of the list.
  • When a process ends, any frames it returns should be returned in lowest-to-highest virtual address order (i.e. the frame [if any] associated with page 0, then page 1, etc.)
  • When a frame is selected from the list of free frames, it should use the frame at the head of the list.
• Dirty pages.
  All resident pages (even instruction pages) should be assumed to have been modified (dirty) since they were last loaded into a frame.  So, if there are more (possibly) modified pages across all the active processes than there are frames, you may (will!) have to "steal" a frame from a still-active process. In which case you'll first have to swap out its contents before reusing the frame (see helper function OSBase.swapOutPage()).
  Likewise, if a process references a page that's non-resident, but which has previously been swapped out, its contents must be swapped in to be placed in a frame. See OSBase.swapInPage().

Notes:

1. Only those pages of a process’s address space that have been explicitly allocated with syscallAlloc() may be successfully referenced by the process.
   When a process is created (after your IOS.initProcess()) method is called), the Simulator (pretending to be the OS Loader), invokes OS.syscallAlloc(...) to allocate all of the process’s instruction pages.
   The instruction set includes an instruction that can request the allocation of some or more data pages. Its execution results in additional calls to OS.syscallAlloc(...).
2. Instructions for a process are mapped contiguously, beginning at virtual address 0.
3. Virtual addresses for process data are allocated contiguously, beginning at the page immediately after the last instruction page.
4. Lab3Process provides an IProcessState object that has getters/setters for each of the CPU state values that need to be saved/restored between dispatches.  Use p.getState() to retrieve the IProcessState instance for process p.
   (The execution state includes an Instruction-Pointer, 2 Registers(0 and 1), and the PTBR.)
5. This project uses a single CPU and expects page-faults to be resolved during instruction execution.  This is *not* reasonable for a real OS, but this reduces the problems associated with multi-threaded programming.
6. CPUBase "manages" a PageTableBaseRegister, that the OS must set each time a different Process is dispatched (see ICPU.setPTBR(IPageTableEntry[][])) – it expects a 2 dimensional array of type IPageTableEntry.  The 1st dimension’s length is the number of level-1 page table entries.  The second dimension’s length is the number of  PTEs  in each 2nd-level page table.   Your CPU.xlate() method will need to use this.getPTBR() in order to retrieve it when searching for a page-to-frame mapping.
7. Your two-level page table does *not* need to be lazily constructed.  Your code can completely populate it for each process when it enters the system.
8. You can build your own programs, using the ISA, defined in Instruction.java.
9. When a frame is first allocated to a page, your code must set its contents to zero. This is required in real OSes for data privacy. It is also required in this project. So when you steal a frame, or allocate a currently-unused one, you'll need to iterate over the bytes in the frame, setting each byte to 0.  The byte array of the entire contests of System-Memory can be accessed via SystemInfo.getSystemMemory(). There's no interface to get only the content of a specific frame.
10. Even though a real OS would allocate its data structures from System-Memory, all of your data structures (Objects) are, of course, allocated from the Java heap (using new).  They do not consume the simulated System-Memory, which is used exclusively to back pages of process memory.
11. You are not allowed to change any files in the simulator package.
    We will test your solution with the original content.
    If your code does not compile with the original version of these files, you will receive a zero.
12. Your implementations of OS.java and CPU.java must not directly depend on each other.
    • Your implementation of OS.java must NOT import OS, e.g. import os.OS;
    • Your implementation of CPU.java must NOT import CPU, e.g. import cpu.CPU;
    If you violate this rule, you will receive a zero.

Approach

For each config file, config_N, (1<=N<=6) there is an associated output file named config_N_v.txt.
This was created by running Simulator with a single –v (verbose) flag against the reference solution.
Your solution must match all 6 of these files to get full credit.

You should begin with config_1 and move forward numerically.

For each config file, config_N, there are actually two associated output files, named config_N_v.txt and another named config_N_vv.txt. The one with a single v is the result of running Simulator with a single –v (verbose) flag against a correctly implement solution. The one with two v's in its name is created by running Simulator with the flag -vv.

Your output only needs match the config_N_v.txt files exactly to get credit.

You will find the 6 config_N_v.txt and the 6 config_N_vv.txt files in the outputs_pt sub-directory.

You should un-zip the provided zip-file, and import the contents into an empty eclipse project.

To run the simulation from Eclipse, create a new “Java Application” Run Configuration, then:

1. Specify simluator.Simulator as the main class (Main tab).
2. Specify “–v  config_1” as the arguments (without the quotation marks) ( (x)= Arguments tab)
3. Specify an output file if you wish to capture the output (Common tab, Standard Input and Output section).

To run the simulator from the command line using config_1.properties:

• $ cd project3_cs439
• $ ls
• src bin configs outputs_pt README src Makefile
• $ make config1
  This will build your Java files (if necessary), run config1, and compare (diff) your output against outputs_pt/output_1_v.txt
  Likewise for targets config2, 3..., config6
• $ make configs # will run all 6 tests and diff their output
• $ make classes # to compile all the classes

Some debugging help

The Simulator supports 3 levels of logging, implemented in the base.Debug class:

1. Debug.log(String message) will always output the specified message.
2. Debug.info(String message) will produce output if Simulator is run with either the -v flag, or the -vv flag.
3. Debug.user(String message) will produce output only if the Simulator is run with the -vv flag.

The reference solution for OS and CPU is sprinkled with a few uses of Debug.user() messages.
For your reference, I've included output files in the outputs_pt directory that were produced running the reference solution with the -vv flag.
These files are named config_?_vv.txt (note the double v's).
So, if you get stuck, you can look these over to get a sense of what the reference solution is doing.
If you want, you can sprinkle your code with calls to Debug.user("...") at what should be the same places where they likely appear in the reference solution.
Likewise, you can sprinkle calls to Debug.user() throughout your code, knowing full well that they won't interfere with getting the correct output when you test you code (we only test your solution with -v, not -vv.)

Files to turn in

1. CPU.java (Your completed implementation)
2. OS.java (Your completed implementation)
3. README (sekelton provided, but you need to complete it)

You need to fill out the README file.
When you're finished with the project, use the make target:

           make turnin

which will create a tar.gz file of your work and invoke turnin for you.
If you'd like to create the tar.gz file and look it over, you can create it without doing the turnin using:

             make turnin_setup