Simple network computing
Addjob lets you distribute jobs over a network.
- It's painless to use,
- Start the addjob-server, then submit jobs like
addjob gcc -c foo.c
- Nearly transparent,
- You can still interact with command-line programs, and exit codes are
preserved (caveats)
-
Some interactive programs have special handling of the EOF (e.g., Control+D)
character and are still able to read input after EOF is encountered. We do not
support this. At most one EOF can be sent to an program which is run with
addjob, and it is sent by closing the program's input stream. After this is
done, no further input can be sent.
- Secure,
- Only you (and root) can submit jobs on your behalf (details)
-
When a client connects, the addjob server generates a reasonably large
random number, and saves it into an challenge file. To authenticate, the
client must read this file and send its contents to the server. These files
are created with permissions 0600 so only your user account (and root)
can read them, so only you (and root) can submit jobs on your behalf. Each
challenge is unique to prevent eavesdropping attacks.
- And free.
- GNU General
Public License, version 2 or later
Installation and setup
We make several assumptions about your network:
- Your jobs should be able to run on any machine
I.e., don't try to mix Linux and Solaris object code
- You should have the same username on each
machine
- The machines should have a shared file system such as NFS
Your project files and home directory should be shared
- You should be able to SSH to any machine without entering a password
How do I set this up?
Supporting software
You'll need this installed on all the machines you want to use. I use
the versions listed below, but later versions may also work. It's easiest
to install these to some shared location on your NFS, and add the appropriate
directories to your PATH, etc.
- Ruby 1.9.1-p129 (with pthreads enabled)
- Open4 library for Ruby
This is a available as a "Gem" for Ruby
Typically, you can install it by running gem install open4 after
installing Ruby.
See rubygems.org for more info.
Addjob setup
- Just edit your login scripts and add the
addjob-x.xx directory to your PATH.
You'll finally need to edit addjob-x.xx/Config.rb
and enter the names of the machines you want to use. Some options
are available for controlling how heavily each machine is used --
see the comments in the configuration file for more details.
Tips and notes
You can monitor your machines by pointing your web browser to
http://your-server:33333
Where your-server is the machine you ran the addjob-server program on. Here's a
screenshot of what it looks like.
If your program needs special environment variables like $PATH, you'll need
to put them into your .bashrc
instead of your .bash_profile. Your .bash_profile will not be run by addjob jobs.
It's easiest (and most efficient) to just quote the whole job to do
IO redirection. For example:
addjob "myprogram --foo --bar < infile > outfile"
This way, all the redirection is handled on the remote machine instead of across the
network.
Bugs, patches, etc.
Feel free to email jared@cs.utexas.edu if
you have trouble with it or want to submit a patch.