CITRIS Itanium 2 Cluster

The Center for Information Technology Research in the Interest of Society was established to sponsor collaborative information technology research that will ultimately provide solutions to grand-challenge social and commercial problems affecting the quality of life of all Californians. The set of applications include energy efficiency, transportation, environmental monitoring, seismic safety, education, cultural research and health care.

As part of the CITRIS project, Hewlett-Packard and Intel have agreed to donate a 128 node cluster. This will consist of 256 Itanium-2 processors, up to 6 Gflops per processor, for a peak of around 1 Teraflop, with half a Terabyte RAM and 18 Terabytes of disk.

Introduction to the CITRIS Cluster

The CITRIS Itanium-2 Cluster currently has 96 compute nodes and 3 frontend nodes with the following configurations:

Qty. 22 (c1-c16, c49-c54)

HP rx2600
Dual 900MHz Itanium 2 (McKinley)
level 1 cache -- 32KB
level 2 cache -- 256KB
level 3 cache -- 1.5MB
4GB RAM
2 36GB 10K rpm disks
Linux 2.6.18

Qty. 2 (c65-c66)

HP rx2600
Dual 1.5GHz Itanium 2 (Madison)
level 1 cache -- 32KB
level 2 cache -- 256KB
level 3 cache -- 6MB
8GB RAM
2 73GB 15K rpm disks
Linux 2.6.18

Qty. 42 (c17-c48, c55-c61, lemon, lime, grapefruit)

HP rx2600
Dual 1.3GHz Itanium 2 (Madison)
level 1 cache -- 32KB
level 2 cache -- 256KB
level 3 cache -- 3MB
4GB RAM
2 73GB 15K rpm disks
Linux 2.6.18

Qty. 30 (c67-c96)

HP rx1620
Dual 1.3GHz Itanium 2 (Madison)
4GB RAM
2 73GB 15K rpm disks
Linux 2.6.18

Nodes c17-c48 have Myricom Myrinet LANai XP PCI-X M3F-PCIXD-2 high-speed interconnect cards.
Nodes c1-c16, c49-c61 have Myricom Myrinet 2000 LANai 9 M3S-PCI64B high-speed interconnect cards.
The IBM Fast Storage Cluster provides high-speed shared filesystems to this compute cluster.

Using the Cluster

Scheduling on the CITRIS cluster was previously facilitated by the PBS Pro batch system version 8.0. We have switched to the Torque Resource Manager and Maui Cluster Scheduler.

There are 3 frontend nodes to the CITRIS Cluster:

lemon.millennium.berkeley.edu lime.millennium.berkeley.edu grapefruit.millennium.berkeley.edu

MPI

If you are unfamiliar with MPI, please visit the MPI Tutorial. MPI jobs can be run over gigabit Ethernet using our modified P4 version, or over Myrinet GM (the preferred, faster way).

Filesystems

While you should be able execute jobs from your EECS department home directory, we strongly suggest that you launch all jobs from a /work/$user directory. Before executing a program, copy all binaries and data files into your /work directory, cd into that directory, and execute from there. This avoids putting unnecessary load on EECS department fileservers.

/work has a 30 day deletion policy. Any file not touched for 30 days will be deleted without warning. /work is meant for staging runs on the cluster, not for long-term storage. /work is never backed up.
/scratch is available on every machine in the cluster. /scratch is high-speed RAID0 storage local to each machine and is intended for use for program checkpointing. /scratch has a similar 10 day deletion policy. Data left on compute nodes is never backed up.

Consider adding /usr/mill/bin to your path. /usr/mill hosts several useful software packages, including Java Development Kits and the Intel C & Fortran compilers.

Acknowledgements

Please use the following language in papers that have benefited from the use of the CITRIS cluster:

"The authors wish to acknowledge the contribution from Intel Corporation, Hewlett-Packard Corporation, IBM Corporation, and the National Science Foundation grant EIA-0303575 in making hardware and software available for the CITRIS Cluster which was used in producing these research results."

UC Berkeley Clustered Computing - Last modified on 07-Aug-2007 14:57:48 -0700