Info on Programming Assignment 2 (MPI):

• MPI Tutorial
• Another One

Champion Info

• Compiling MPI programs
  
  
  • To compile an MPI program in c, use the following command:
    mpcc_r -o prog_name mpi.c
  
  
  • To compile a C++ program, use the following command:
    mpCC_r -cpp -o prog_name mpi.cpp
  
  
• Launching programs using Champion's job scheduler

The job scheduler on the Champion machine is called "LoadLeveler". You need to prepare a "LoadLeveler" job submission script to run jobs on the champion nodes. You can read more about it here . I provide a short summary here to get you started.

A sample job submission script is given below (for a program with 8 MPI tasks):
--------------------------
#@ shell = /bin/bash
#@ initialdir = FIX_YOUR_DIRECTORY_PATH
#@ account_no = A-cs41
#@ job_name = FIX_mpi_2
#@ input = /dev/null
#@ output = $(job_name).o$(jobid)
#@ error = $(job_name).o$(jobid)
#@ job_type = parallel
#@ environment = COPY_ALL; LL_JOB=TRUE;
#@ resources = ConsumableCpus(1) ConsumableMemory(500mb)
#@ network.MPI_LAPI = sn_all, not_shared, US
#@ wall_clock_limit = 01:00:00
#@ node = 1
#@ tasks_per_node = 8
#@ node_usage = not_shared
#@ notify_user = FIX_YOUR_EMAIL_ID
#@ notification = complete
#@ class = normal
#@ queue
poe ./FIX_PROGRAM_NAME ARGUMENTS
------------------------------

All places where you need to modify this script for your use start with the prefix "FIX". The "initialdir" variable denotes your working directory where the executable and the input files are stored.

The "account_no" variable denotes our parallel systems project. PLEASE REMEMBER TO INCLUDE THIS LINE in your script. You can optionally modify the "job_name" variable.

Please set ConsumableCpus ALWAYS to 1. Depending on the number of tasks in your MPI program, you need to modify the values of "node", and "tasks_per_node". If your program has 8 MPI tasks or less, set node to 1 and tasks_per_node to the number of tasks. If your program has more than 8 tasks, for example 16, set node to 2 and tasks_per_node to 8. The maximum allowed values for tasks_per_node is 8 (as each node has 8 processors) and for node is 4. You can adjust these variables to achieve different task sizes.

Adjust ConsumableMemory to your needs. If you set ConsumableMemory to 1000 MB, and tasks_per_node to 4, your job will be reserved 4000 MB of memory. If you run your task on multiple nodes, you should EXPERIMENT with different values of ConsumableMemory to fit your needs. Please include your email id so that the system will notify you when your job completes. (You can change this to notify only there is an error too. Other scenarios are listed in the documentation.) Also, the wall_clock_limit variable sets an upper bound on the running time for your program. Tweak this if necessary.

The last line is your program name and its arguments. The KEYWORD "poe" in the last line is VERY IMPORTANT!!!!

Please use the other options AS IS unless you know what you are doing.

After you prepare a job submission script, use the command "llsubmit job_file_name" to launch your job. (Replace job_file_name with your filename).

Email me if you have any questions or spot any errors!



• Measuring performance


• You can system calls like gettimeofday inside your program to time the relevant portions of the program.


• The MPI library provides some functions like MPI_Wtime and MPI_Wtick for measuring time. Please refer the MPI documentation for more information.


• You can also use performance counters in the Power5 processor through the PAPI library. Info on PAPI is found on the Champion Site under the "Tools" section (on the left navigation menu)