Main Page
From GridWiki
Contents |
About this wiki
The core Grid Engine websites are:
This wiki server (and the related http://gridengine.info blog) has been set up by members of the Grid Engine user community as a way of collecting and distributing usage, configuration, tips and HOWTO information in a fast and democratic way. We don't want to reinvent the wheel or waste time so wherever there are better online resources available, this Wiki will simply link out to them.
The best place to get help with Grid Engine related issues is the sge-users mailing list.
Grid Computing & Grid Engine Overview
This is not the place for an academic dissertation or even a management-friendly summary of "grid" or "cluster" computing. If you are just starting out learning about clusters you may want to check out sites like http://www.clustermonkey.net. Some good intro-level information on Grid Engine and it's capabilities can be found here:
- Chapter 1 of the 'N1 Grid Engine Users Guide'
- Introduction to the Cluster Grid - Part 1 (August 2002) (Sun BluePrint PDF)
Grid Engine is software that facilitates "distributed resource management" (DRM) - other similar software packages include Portable Batch System, Torque and Platform LSF. Far more than just simple load-balancing tools or batch scheduling mechanisms, DRM software typically provides the following key features across large sets of distributed resources:
- Policy based allocation of distributed resources (CPU time, software licenses, etc.)
- Batch queuing & scheduling
- Support diverse server hardware, OS and architectures
- Load balancing & remote job execution
- Detailed job accounting statistics
- Fine-grained user specifiable resources
- Suspend/resume/migrate jobs
- Tools for reporting Job/Host/Cluster status
- Job Arrays
- Integration & control of parallel jobs
Is Grid Engine commercial or open source software?
Both. Sun Microsystems sponsors development of the freely available and open source Grid Engine product hosted at http://gridengine.sunsource.net. The software found on the http://gridengine.sunsource.net site is the free "open source" version.
Sun also takes the open source codebase inhouse and offers it for sale as a formally supported enterprise distributed computing product. There is no functional difference between the base open source and commercial versions. The commercial version differs from the free version in the following key ways:
- Additional QC/QA testing done internally by Sun
- Localization to non-English languages
- Additional layered products offered for sale
- Support for Windows execution hosts (SGE 6 only)
- Global support & professional services
63690493182502362827393
Grid Engine 5.x "Standard" vs. "Enterprise" Editions
Prior to the release of Grid Engine 6 in 2004, there were two different "flavors" of Grid Engine offered by Sun Microsystems. The "standard" edition was a Sun software product made available for free to any user or institution. The "Enterprise Edition" was available from Sun at additional cost.
The only difference between the two offerings was the number of scheduling policy types supported. The standard edition generally supported only a "first-in, first-out" FIFO type scheduling policy along with a simple user-sort scheduling mechanism. The Enterprise Edition supported FIFO scheduling in addition to several other more flexible policy based scheduling mechanisms.
Interestingly enough, the codebase and binaries for both flavors are 100% the same. The difference between "standard" and "enterprise edition" is triggered by a simple flag passed via an installation script. During the time that Sun Microsystems was offering two different types of "Sun Grid Engine", the open source site was offering both flavors of "Grid Engine" for free.
"Sun N1 Grid Engine 6" vs. "Grid Engine 6"
When Grid Engine 6 was first released, Chris Dagdigian wrote a simple whitepaper entitled Understanding the differences between Grid Engine 5.3, 6.0 and Sun N1 Grid Engine 6 (N1GE) - while semi dated it provides a good overview of the differences.
The December 2005 Bombshell
Further complicating the interesting relationship between the Sun branded and open source versions of Grid Engine, Sun made a surprise announcement in December 2005 where the company announced that (among many other software products), the full Sun N1 software stack including N1 Grid Engine would now be available for "free".
The Grid Engine community is still digesting this announcement, at a minimum it means that layered products formally sold with N1 Grid Engine 6 are freely available.
More information on this announcement can be found online: http://gridengine.info/articles/2005/12/01/sun-n1-grid-engine-is-now-free 285823810211758374689213 870650651294943836554700
Documentation
See http://gridengine.sunsource.net/documentation.html
HOWTOs
See this link: http://gridengine.sunsource.net/project/gridengine/howto/howto.html
Sun BluePrints
Sun maintains an interesting technical library of "BluePrint Documents". Interesting publications include:
- Sun Grid Engine, Enterprise Edition-Configuration Use Cases and Guidelines (July 2003)
- Global Grid Connectivity Using Globus Toolkit With Solaris Operating System (May 2004)
- Using Host Groups and Cluster Queues in the Sun N1 Grid Engine 6 System (August 2005)
- Scheduler Policies for Job Prioritization in the N1 Grid Engine 6 System (October 2005)
Application Integration
Notes on LAM-MPI Tight PE Integration
FLEXlm License Server Integration
- The new method: http://gridengine.info/wiki/index.php/Olesen-FLEXlm-Integration
- The older "load sensor" approach: http://bioteam.net/dag/sge-flexlm-integration/
MatLab Integration
- The Grid Engine community is looking for integration methods and tips. Please share your ideas under the Matlab-Integration section of this wiki.
Clearcase Integration
Information on integrating Grid Engine with cleacase code repositories should be posted to the Clearcase-Integration section of this wiki.
Grid Engine XML parser project
- Dan has started Grid Engine XML parser project under Passau
Misc.
Simple introduction to Grid Engine Array Jobs (with R example)
Distributed software builds (compile farming)
Grid Engine has a tool called "qmake" which can help distribute large source code building tasks across a cluster of machines. Information on distributed builds has been moved to the Distributed-Compilation wiki page.
Zeroconf support ( aka Apple Bonjour' & 'Rendezvous')
A patch was recently accepted by the developers to make zeroconf a build-time compile option. Specifics are mentioned in this blog post: http://gridengine.info/articles/2006/02/16/zeroconf-enabled-grid-engine
Other links:
Grid Engine wiki projects
Documenting Grid Engine XML output
Chris Dag has started a page GridEngine_XML to document the internals of Grid Engine XML output status data.
Documenting and understanding Grid Engine 'qping' output
Chris Dag has created a placeholder page GridEngine_qping to document troubleshooting methods involving the 'qping' utility.
Development Specifications
Development specifications are kept on GridWiki
Command line tools for GridWiki editing
The web GUI Interface is very time consuming once you know eactly what you want to do. By learning command line methods, we can automate the wiki contribution work and speed up the process.
Andreas: Charming! There are crowds of good contents deserving to become though Grid Engine Wiki. Wouldn't it be possible to achieve this through wget(1) in combination with some scripts to rephrase things into MediaWiki dialect?
Joachim: AFAIK MediaWiki has some sort of API for generating content. It would certainly be interesting to publish generated data to this Wiki. An example could be Gridengine testsuite publishing performance metrics - currently it generates a HTML report.
Grid Engine Packaging Efforts
Packaging documentation and disussion has been moved to its own GE-Packaging wiki page. This is of interest to people working on alternative binary or source installation/distribution methods for Grid Engine.
Success Story
There is a list of DRMAA success stories kept under http://www.drmaa.org/wiki/index.php/DrmaaUsers
Tips
Snippets has a collection of short, but useful scripts.
References
Please see documentation on customizing the interface and the User's Guide for usage and configuration help.
