Running Jobs on UTCS Computing Clusters

What is Condor?
Why use Condor?
Availability in UTCS
Road-map for running Jobs with Condor
What is Condor Job Command File?
Some Examples of Condor Job Command Files
General Procedure to run Programs using Condor
Running Cocluster on clusters (an example)
Running JAVA on clusters (an example)
Related URLs

What is Condor?

Condor is a software system that creates a High-Throughput Computing (HTC) environment. It effectively utilizes the computing power of workstations that communicate over a network. Condor can manage a dedicated cluster of workstations. Its power comes from the ability to effectively harness non-dedicated, preexisting resources under distributed ownership.
A user submits the job to Condor. Condor finds an available machine on the network and begins running the job on that machine. Condor has the capability to detect that a machine running a Condor job is no longer available (perhaps because the owner of the machine came back from lunch and started typing on the keyboard). It can checkpoint the job and move (migrate) jobs to a different machine which would otherwise be idle. Condor continues job on the new machine from precisely where it left off.
In those cases where Condor can checkpoint and migrate a job, Condor makes it easy to maximize the number of machines which can run a job. In this case, there is no requirement for machines to share file systems (for example, with NFS or AFS), so that machines across an entire enterprise can run a job, including machines in different administrative domains.
Condor does not require an account (login) on machines where it runs a job. Condor can do this because of its remote system call technology, which traps library calls for such operations as reading or writing from disk files. The calls are transmitted over the network to be performed on the machine where the job was submitted.

From the UTCS local condor documentation:

Instead of running a CPU-intensive job in the background on their own workstation, a user submits their job to Condor.
Condor finds an available machine on the network and begins running the job on that machine.
When Condor detects that a machine running a Condor job is no longer available (perhaps because the owner of the machine came back from lunch and started typing on the keyboard), Condor checkpoints the job and migrates it over the network to a different machine which would otherwise be idle.
Condor restarts the job on the new machine to continue from precisely where it left off.
If no machine on the network is currently available, then the job is stored in a queue on disk until a machine becomes available.

[Back to Top]

Why use Condor?

From the Overview of the Condor Manual:

Condor takes advantage of computing resources that would otherwise be wasted and puts them to good use. Condor streamlines the scientist's tasks by allowing the submission of many jobs at the same time. In this way, tremendous amounts of computation can be done with very little intervention from the user. Moreover, Condor allows users to take advantage of idle machines that they would not otherwise have access to.
Condor provides other important features to its users. Source code does not have to be modified in any way to take advantage of these benefits. Code that can be re-linked with the Condor libraries gains two further abilities: the jobs can produce checkpoints and they can perform remote system calls.
A checkpoint is the complete set of information that comprises a program's state. Given a checkpoint, a program can use the checkpoint to resume execution. For long-running computations, the ability to produce and use checkpoints can save days, or even weeks of accumulated computation time. If a machine crashes, or must be rebooted for an administrative task, a checkpoint preserves computation already completed. Condor makes checkpoints of jobs, doing so periodically, or when the machine on which a job is executing will shortly become unavailable. In this way, the job can be continued on another machine (of the same platform); this is known as process migration.
A user submits a job to Condor. The job is executed on a remote machine within the pool of machines available to Condor. Minimal impact on and the security of the remote machine are preserved by Condor through remote system calls. When the job does a system call, for example to do an input or output function, the data is maintained on the machine where the job was submitted. The data is not on the remote machine, where it could be an imposition.
By linking in a set of Condor libraries, system calls are caught and performed by Condor, instead of by the remote machine's operating system. Condor sends the system call from the remote machine to the machine where the jobs was submitted. The system call's function executes, and Condor sends the result back to the remote machine.
This implementation has the added benefit that a user submitting jobs to Condor does not need an account on the remote machine.

[Back to Top]

Availability in UTCS

UTCS operates two general-purpose High Throughput Computing (HTC) clusters, Mastodon and Scout, which are managed using the Condor job scheduling software.
Condor is currently supported on the Department's Linux, Sparc Solaris, and x86 Solaris machines .
Among five condor universes, only STANDARD and VANILLA universes are currently supported.
All of the Condor programs are in /lusr/condor/bin.
So Add that directory to your PATH environment variable or use that path with the commands described in How To Use Condor.

[Back to Top]

Road-map for running Jobs with Condor

From the UTCS local condor documentation:

Optional: Compile and link the program with Condor's libraries to enable the job to be migrated among the machines using condor_compile.
Submit the job with condor_submit.
The job will enter the queue and be executed. To monitor it, use condor_q.
To check the status of the condor queue, use condor_status.
To remove a job before it has finished executing, use condor_rm.

From the Users' Manual:

Code Preparation
Condor Universe
Submit Description File
Submit Job
Job Completion.

1. Code Preparation.

Job run under Condor must be able to run as a background batch job.
Condor runs the program unattended and in the background.
A program that runs in the background will not be able to do interactive input and output.
Condor can redirect console output (stdout and stderr) and keyboard input (stdin) to and from files for you.
Create any needed files that contain the proper keystrokes needed for program input. Make certain the program will run correctly with the files.

2. What is Condor Universe.

Condor has five runtime environments (called a universe) from which to choose.
Right now only two universes, STANDARD and VANILLA, are supported on UTCS machines.

STANDARD:

Condor provides checkpointing and remote system calls.

To prepare a program as a standard universe job, it must be relinked with condor libraries using condor_compile.

Put condor_compile in front of your usual link command. You do not need to modify the program's source code. Recompilation is not necessary, only relinking.

VANILLA:

The vanilla universe provides a way to run jobs that cannot be successfully relinked.

Shell scripts are another case where the vanilla universe is useful.

It depends on a shared file system for access to input and output files, and there is no way to take a checkpoint or migrate a job executed under the vanilla universe.

PVM:

It allows programs written for the Parallel Virtual Machine interface to be used within the opportunistic Condor environment.

MPI:

It supports the execution of parallel jobs that utilize MPI.

GLOBUS:

It is intended to provide the standard Condor interface to users who wish to start Globus system jobs from Condor.

3. What is Submit Description File?

Controlling the details of a job submission is a submit description file. The file contains information about the job such as what executable to run, the files to use for keyboard and screen data, the platform type required to run the program (for example, memory/hardware requirements, etc.), and where to send e-mail when the job completes. You can also tell Condor how many times to run a program; it is simple to run the same program multiple times with multiple data sets.
Comment lines begin with '#'.
Valid Condor keywords are any initial word (no blanks) followed by an equal (=) sign.

Local Predicates

In order to specifiy requirements which must be met by the machine to be executing the job:

requirements = clusterName

Requirement = InPhylofarm (The Phylofarm cluster) Requirement = InScout (The Scout cluster) Requirement = InMastodon (The Mastodon cluster) Requirement = InPublic (Completely public machines only)

In order to indicate who submits job(s):
+Group = groupName

+Group = "CARTEL" (a part of the CART) +Group = "PROF" (a CS professor) +Group = "GRAD" (a CS graduate student) +Group = "UNDER" (a CS undergraduate) +Group = "UTGRID" (a UTGRID partner) +Group = "GUEST" (a guest account)

In order to define the UTCS research area to which the jobrelates:
+Project = researchArea

+Project = "ARCHITECTURE" +Project = "FORMAL_METHODS" +Project = "AI_ROBOTICS" +Project = "OPERATING_DISTRIBUTED_SYSTEMS" +Project = "NETWORKING_MULTIMEDIA" +Project = "PROGRAMMING_LANGUATES" +Project = "THEORY" +Project = "GRAPHICS_VISUALIZATION" +Project = "COMPONENT_BASED_SOFTWARE" +Project = "SCIENTIFIC_COMPUTING" +Project = "COMPUTATIONAL_BIOLOGY" +Project = "INSTRUCTIONAL" +Project = "UTGRID" +Project = "OTHER"

In order to describe what you are working on:
+ProjectDescription = "This is a free-form text field which cannot be blank."

+ProjectDescription = "Coclustering of gene expression data"

4. Submit Job.

Submit the program to Condor with the condor_submit.
Once submitted, Condor does the rest toward running the job.
Monitor the job's progress with the condor_q and condor_status
You may modify the order in which Condor will run your jobs with the condor_prio. If desired, Condor can even inform you in a log file every time your job is checkpointed and/or migrated to a different machine.

5. Job Completion.

When your program completes, Condor will tell you (by e-mail, if preferred) the exit status of your program and various statistics about its performances, including time used and I/O performed.
If you are using a log file for the job (which is recommended) the exit status will be recorded in the log file.
You can remove a job from the queue prematurely with the condor_rm.

[Back to Top]

What is Condor Job Command File (=Condor Submit Description File)?:

Format of Condor Job Command File:

############################################### # # General Format of Condor Job Command File # (Notice that comment lines begin with #.) # ############################################### +Group = "GRAD" +Project = "SCIENTIFIC_COMPUTING" +ProjectDescription = "Computing..." Requirements = ClassAd Boolean Expression # a boolean, which when evaluated within the context of the machine ClassAd # and a job ClassAd, must evaluate to TRUE before Condor will allow the job # to use this machine. Universe = which Condor Universe to use when running this job (OPTIONAL) # vanilla/standard/pvm/scheduler/globus/mpi Executable = name # the name of the executable file for this job cluster Initialdir = directory-path (OPTIONAL) # the current working directory for the job Arguments = list of the command line arguments to be supplied to the job (OPTIONAL) ... ... ... Input = pathname (OPTIONAL) # stdin of the job is redirected from a file specified here Output = pathname (OPTIONAL) # stdout of the job is redirected to a file specified here Error = pathname (OPTIONAL) # stderr of the job is redirected to a file specified here Log = pathname # a filename where Condor will write a log file (OPTIONAL) Notification = when owners of the job(s) are notified by email (OPTIONAL) # always/complete(DEFAULT)/error Notify_user = the email address to use when Condor sends email about a job (OPTIONAL) Queue ... ... ...

Some Examples of Condor Job Command Files:

For the explanation of the following examples of submit description files, see:

Submitting a Job to Condor
condor_submit manual page

From the Submitting a Job to Condor:

Example 1:
################################################## # # Example 1 # Simple condor job description file # ################################################## Executable = foo Log = foo.log Queue

Example 2:
################################################## # # Example 2: demonstrate use of multiple # directories for data organization. # ################################################## Executable = mathematica Universe = vanilla input = test.data output = loop.out error = loop.error Log = loop.log Initialdir = run_1 Queue Initialdir = run_2 Queue

From the condor_submit:

Example 3:
################################################## # # Example 3: queueing multiple jobs with differing # command line arguments and output files. # ################################################## Executable = foo Arguments = 15 2000 Output = foo.out1 Error = foo.err1 Queue Arguments = 30 2000 Output = foo.out2 Error = foo.err2 Queue Arguments = 45 6000 Output = foo.out3 Error = foo.err3 Queue

Example 4:
################################################## # # Example 4: Show off some fancy features including # use of pre-defined macros and logging. # ################################################## Executable = foo Requirements = Memory >= 32 && OpSys == "IRIX6" && Arch =="SGI" Rank = Memory >= 64 Image_Size = 28 Meg Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = foo.log Queue 150

[Back to Top]

General Procedure to run Programs using Condor

Login to one of public machines:

Compile code if necessary:
/usr/bin/g++ -c filename(i.e. c or c++ code)
(Compile programs for C (or C++), and create object files only)

Create a Condor executable (for an example, Standard-Universe job):
condor_compile   /usr/bin/g++ object_files.o -llibraries
(Links programs for C (or C++), respectively)
Create a Condor job command file: See What is Condor Job Command File? above:
requirements = InMastodon           #Don't forget to add this line to run on Mastodon cluster.
+Group = "GRAD"
+Project = "SCIENTIFIC_COMPUTING"
+ProjectDescription = "Computing...."
(Let say this, "test_run.condor".)

Submit the job using "condor_submit" command:
condor_submit   test_run.condor
View job status in queue using "condor_q" (or "condor_status"), and wait for completion:
condor_q (or condor_status)

[Back to Top]

Running Cocluster on clusters (an example)

Login to one linux machine:
Let's assume you've already logged in "twoflower.cs.utexas.edu" with "user_name" as user ID.

Compile Cocluster using "make":
make condor
Let's call the resulting execution file "condor_cocluster-Linux".

Create a Condor executable:
If Cocluster is not successfully relinked using condor_compile.
So, ignore step 3 and use VANILLA universe.

Create a Condor Job Command File:
Let's name the following Condor Job Command File for Cocluster, "condor_cocluster". Remember that you have to store the condor job description file, "condor_cocluster", in the initial directory. However, notice that you can specify the directory of your executable file. Let's assume you store the two files in ~/CONDOR/.
Here is the contents of "condor_cocluster" (You can give an arbitrary filename as you prefer.):
    It is forced to be run on Mastodon cluster (line 12).
    It queues up one copy of Cocluster program, "cocluster-Linux" for execution by condor (line 14 and 23).
    The command line arguments are given in (line 14).
    It specify the current working directory in which all resulting files will be stored (line 16).
    It will write its standard output to a file, "cocluster.out" (line 17).
    It will write its standard err to a file, "cocluster.err" (line 18).
    It will write its log information to a file, "cocluster.log" (line 19).
    You will be notified whenever the job is checkpointed, and when it completes (line 20).
    The email address is to be used when Condor sends email about the job (line 21).

1 ################################################## 2 # 3 # Example for Cocluster: 4 # condor job description file 5 # 6 ################################################## 7 8 +Group = "GRAD" 9 +Project = "SCIENTIFIC_COMPUTING" 10 +ProjectDescription = "Computing coclusters of gene expression matrix" 11 12 Requirements = InMastodon 13 Executable = /LATEST/condor_cocluster-Linux 14 Arguments = -a e -c 100 -r 2 -I d s /Leukemia/ALLAML.txt 15 16 Initialdir = ./ 17 Output = cocluster.out 18 Err = cocluster.err 19 Log = cocluster.log 20 Notification = always 21 Notify_user = hyukcho@cs.utexas.edu 22 23 Queue

Submit the job using "condor_submit" command:
condor_submit condor_cocluster
Then, you have the following message if it is successfully submitted:

Submitting job(s).
1 job(s) submitted to cluster ##.
,where the integer ##, (for example, 360), means the cluster number to which the job is submitted.
Now suppose you are submitting the job from the directory ~/CONDOR/. if "Initialdir" were not specified as (line 16), condor_submit will automatically insert the user's current working directory at the time condor_submit was run as the value for "initialdir". Then, all resulting files will be stored in your current working directory.

View job status using "condor_q" or remove job from the queue using "condor_rm":
condor_q
Then you have the following message:

-- Submitter: twoflower.cs.utexas.edu:<128.83.120.167:1029>:twoflower.cs.utexas.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD ... ... ... ... .. ... ... ... 360.0 user_name 10/10 15:15 0+00:00:00 R 0 1.1 condor_cocluster-L 10 jobs; 1 idle, 9 running, 0 held
condor_rm 360.0
Then you have the following message:

Job 360.0 has been marked for removal.

Upon completion of the submitted job, you will get an email from condor like below :
If it is completed successfully, you may have some resulting files in the initial directory you specified, depending on your commandline arguments. (Currently, you don't specify arguments for any resulting files from the coclustering algorithm.)
Date: Wed, 29 Sep 2004 17:18:29 -0500 From: Condor account To: hyukcho@cs.utexas.edu Subject: [Condor] Condor Job 360.0 This is an automated email from the Condor system on machine "habanero.cs.utexas.edu". Do not reply. Your Condor job 199.0 .../LATEST/condor_cocluster-Linux -a e -c 100 -r 2 -I d s .../Leukemia/ALLAML.txt has exited normally with status 0. Submitted at: Wed Sep 29 17:18:22 2004 Completed at: Wed Sep 29 17:18:29 2004 Real Time: 0 00:00:07 Virtual Image Size: 500 Kilobytes Statistics from last run: Allocation/Run time: 0 00:00:04 Remote User CPU Time: 0 00:00:00 Remote System CPU Time: 0 00:00:00 Total Remote CPU Time: 0 00:00:00 Statistics totaled from all runs: Allocation/Run time: 0 00:00:04 Network: 0.0 B Run Bytes Received By Job 0.0 B Run Bytes Sent By Job -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Questions about this message or Condor in general? Email address of the local Condor administrator: fool@cs.utexas.edu, yyz@cs.utexas.edu, dkotz@cs.utexas.edu The Official Condor Homepage is http://www.cs.wisc.edu/condor

[Back to Top]

Running JAVA on clusters (an example)

Login to one linux machine:
Let's assume you login in "twoflower.cs.utexas.edu" whith "user_name" as user ID.

Compile your java program(say, "Stirling2.java"), using "javac":
/lusr/java2/java/javac Stirling2.java
Then, it will result in the execution file "Stirling2.class".

Create a Condor executable:
In fact, JAVA is not successfully relinked using condor_compile.
So, ignore step 3 and use VANILLA universe as below.

Create a Condor Job Command File:
Let's name the following Condor Job Command File for the program, "Stirling2", "condor_s2".
Here is the contents of "condor_s2" (You can give an arbitrary filename as you prefer.):
    It is forced to be run on Mastodon cluster (line 12).
    It shows program submission of "Stirling2" as a vanilla universe job (line 15).
    It queues up one copy of the JAVA program, "Stirling2" for execution by condor (line 16 and 25).
    The java script file name, java program name, and command line arguments are shown in (line 16).
    It specify the current working directory in which all resulting files will be stored (line 18).
    Its standard output will be stored in a file, "s2.out" (line 19).
    Its error(s) (if it has error(s)) will be put to a file, "s2.err" (line 20).
    Its history will be recored in a file, "s2.log" (line 21).
    You will be notified whenever the job is checkpointed, and when it completes (line 22).
    "hyukcho@cs.utexas.edu" will get email when Condor sends email about the job (line 23).

1 ################################################## 2 # 3 # Example for JAVA: 4 # condor job description file 5 # 6 ################################################## 7 8 +Group = "GRAD" 9 +Project = "SCIENTIFIC_COMPUTING" 10 +ProjectDescription = "Computing Stirling number of the second kind" 11 12 Requirements = InMastodon 13 14 Executable = /bin/sh 15 Universe = vanilla 16 Arguments = my_java Stirling2 10000 100 log10 only 17 18 Initialdir = /v/filerla/v0q027hyukcho/CONDOR/ 19 Output = s2.out 20 Error = s2.err 21 Log = s2.log 22 Notification = always 23 Notify_user = hyukcho@cs.utexas.edu 24 25 Queue

Create JAVA Script File:
Notice that the following java script needs to be stored "my_java" in order to mach the arguments in line 11 in "condor_s2" shown above.

1 #!/bin/sh 2 exec /lusr/java2/bin/java "$@"

Submit the job using "condor_submit" command:
condor_submit condor_s2

View job status using "condor_q" or remove job from the queue using "condor_rm":
condor_q
condor_rm ID

Upon completion of the submitted job, you will get an email from condor.

[Back to Top]

Related URLs

[Back to TOP]

Running Jobs on UTCS Computing Clusters

Table of Contents

What is Condor?

Why use Condor?

Availability in UTCS

Road-map for running Jobs with Condor

What is Condor Job Command File (=Condor Submit Description File)?:

Format of Condor Job Command File:

Some Examples of Condor Job Command Files:

General Procedure to run Programs using Condor

Running Cocluster on clusters (an example)

Running JAVA on clusters (an example)

Related URLs