Posted by chris
Thu, 01 Dec 2005 16:34:03 GMT
Rayson was the first person on the SGE users list to notice this big announcement from Sun Microsystems.
The key bits from the release:
Included at no cost in the new Solaris Enterprise System are:
- The award winning and open sourced Solaris 10 OS, with the recently announced PostgreSQL database;
- The entire Sun Java Enterprise System infrastructure software platform, including the Sun Java Identity Management Suite, Sun Java Integration Suite, Sun Java Communications Suite, Sun Java Application Platform Suite, Sun Java Availability Suite and Sun Java Web Infrastructure Suite;
- The N1 Management Software including the Sun N1 System Manager, the Sun N1 Service Provisioning System, the Sun N1 Grid Engine;
- All tools for C, C++ and Java development, including Sun Studio 11, Sun Java Studio Enterprise 8 and Sun Java Studio Creator;
- SunRay ultra-thin client software;
- Sun Secure Global Desktop Software.
What does this mean for you?
The biggest change is that now people will have access to Grid Engine components that were previously only available to paying Sun N1 Grid customers. The biggest items include:
- The ARCO job and resource usage monitoring and reporting subsystem
- Windows Grid Engine exec host client code
If I had to summarize industry coverage it would be something like this from the Reuters article:
"...Although more of Sun's software will now be free, customers will have to pay for service and support, which is how Sun aims to boost revenue."
Other coverage:
Posted in About Grid Engine, News
Posted by DanT
Tue, 29 Nov 2005 22:29:00 GMT
Whopee! We are finally wrapping up the QA for the next release of Grid Engine, version 6.0u7. This is an interesting release because it focuses on scalability improvements. With this release, we have addressed several issues that crop up when building grids with thousands of nodes. My personal contribution was to rewrite the lexigraphical parser for the qconf utility so that it is more memory efficient and is able to handle cluster queue configurations with thousands of queue instances. I also fixed a big stack of man page issues. Stephan added some serious process improvements to the qmaster and scheduler, which can lead to big performance gains, as well as making our qstat XML schemas quite a bit cleaner.
The 6.0u7 release is not yet publicly available, unless you download the source from the website (Grab the V60s2_BRANCH branch.) and build it yourself. The plan is to have it available around the second week of December. Once it’s out, I’ll post a link to where you can download the courtesy binaries.
Posted in News | Tags engine, grid, release, update | no trackbacks
Posted by DanT
Thu, 17 Nov 2005 19:58:00 GMT
This topic is one which has been under discussion for some time now. The basic idea is that using the JAXB RI from the JWSDP, we could build a set of classes which would parse qstat output, making it trivial for a developer to write an app which keeps tabs on Grid Engine. I believe the final decision was that we would not officially include such classes with Grid Engine for supportability reasons. Instead, I’m going to explain to you how to build the classes yourself. If you’re too lazy to follow these instructions, here’s a tarball of the classes I generated while writing this post, along with the source, etc.
Read more...
Posted in Grid Engine XML | Tags java, jaxb, qstat, xml | no trackbacks
Posted by chris
Thu, 17 Nov 2005 00:43:27 GMT
In a post to the Grid Engine users list, Mark Olesen proposed a novel approach to FlexLM license integration on Grid Engine managed clusters.
He offered to provide code that implemented his ideas with the caveat that he would be completely unable to provide support or end-user assistance. Since I had already written one of the FlexLM integration HOWTO's and have a personal interest in application integration issues I volunteered to try to support Mark's code if it meant that he would be able to release it to the general user community.
I've been playing with versions of his work for several weeks now and it is very cool, very useful and very well written. Within the last week, Mark has finalized his code and selected a Creative Commons license. Over the last two evenings I've been working on a GridWiki entry documenting the basic installation, configuration and installation.
The first draft (along with a downloadable tarball!) is now ready for public review at this URL: http://gridengine.info/wiki/index.php/Olesen-FLEXlm-Integration.
Obviously, Mark Olesen needs to be acknowledged for proposing his ideas and backing them up with well-written code. I'd also like to specifically thank the fine people at Schrödinger for providing access to some of their FLEXlm licensed software. Schrödinger's valuable contribution made it possible for me to test, evaluate and write about FLEXlm on Grid Engine managed systems. Thanks!
Posted in Application Integration, External Tools & Apps
Posted by chris
Wed, 16 Nov 2005 18:51:57 GMT
Andreas reports that the http://gridengine.sunsource.net/howto/troubleshooting.html page has recently updated.
One of the changes reflects the new recommended SGE 6.x method of getting nicely formatted error reporting data out of qacct output:
# qstat -j | grep error
Posted in MailList Bits
Posted by chris
Tue, 15 Nov 2005 18:49:58 GMT
Idle Question: What sorts of interesting things have happened to the
Grid Engine
codebase and
Issue tracking
system in the last 7 days?
Answer: IssueZilla
knows all.
Summary:The following is a summary of recent issues and RFEs opened, updated, closed or resolved within the last 7 days.
Reopened issues:
None.
New issues:
#1874: qconf -mstree aborts
Status:NEW Priority:P1Resolution:
Commentary: Wow! Priority 1 bugs are pretty rare these days. It seems that manipulating the sharetree via -mstree can cause a fatal client error and abort: "critical error: Dechaining element from other list !!!".
#1282: QCONF: qconf -dattr queue slots <val> <queue>@<host> doesn't work
Status:NEW Priority:P3Resolution:
Commentary:
Read more...
Posted in MailList Bits
Posted by chris
Fri, 11 Nov 2005 19:45:23 GMT
Joe Landman from Scalable Informatics posted about his success with the Perl XML::Smart ( CPAN, readme, FAQ, tutorial) module.
Unlike many of the XML handling methods within the Perl universe, this module stands on its own without a huge and complicated chain of external dependencies.
XML::Smart can quickly and cleanly parse XML documents into perl datastructures that can efficiently traversed and sorted. This makes it a great method for simple perl scripts designed to grab bits of data or information that does not get displayed in the human-readble qstat output.
Joe's comments:
Our 6.0u6 perl based parser fits into a single line, after we grab
the data.
$qstat=`/opt/gridengine/bin/lx24-amd64/qstat -xml`;
$xml = XML::Smart->new($qstat);
(no schema/DTD needed)
then for example, iterating over all the jobs ...
foreach ($xml->{job_info}->{queue_info}->{job_list}('@') )
{
...
}
Using some example code (included at the end of this article by permission) kindly provided by Joe, I was able to whip up a little "just playing" script that checks all pending jobs for hard resource requests. When a hard request is found, the script simply prints out a line that lists the Job ID, Job Name and the value of the hard resource request. The script looks like this:
#!/usr/bin/perl -w
use XML::Smart;
my ($xml,$qstat);
$qstat=`/opt/sge6s2u1/bin/lx24-amd64/qstat -xml -r -f`;
$xml = XML::Smart->new($qstat);
foreach ($xml->{job_info}->{job_info}->{job_list}('@') )
{
if($_->{hard_request}) {
print "Job ID $_->{JB_job_number} ($_->{JB_name}) has a hard_request: ";
print "$_->{hard_request}{name}=$_->{hard_request} \n";
}
}
Output looks like this:
[dag@dcore-amd ~]$ ./test.pl
Job ID 47 (impossibleJob) has a hard_request: arch=darwin
[dag@dcore-amd ~]$
Additional pointers and examples from Scalable Informatics are included below ...
Read more...
Posted in Grid Engine XML | Tags perl, xml, xml::smart | no comments
Posted by chris
Wed, 09 Nov 2005 17:55:02 GMT
Idle Question: What sorts of interesting things have happened to the
Grid Engine
codebase and
Issue tracking
system in the last 7 days?
Answer: IssueZilla
knows all.
.
Summary:The following is a summary of recent issues and RFEs opened, updated, closed or resolved within the last 7 days.
Reopened issues:None.
New issues:
#1855
- flush_time=0:0:0 is treated as 0:0:15
Status:NEWPriority:P5Resolution:
Commentary:
If the flush_time reporting parameter in "qconf -mconf global" is set to 0:0:0,
the flushing behavior will behave as though flush_time were set to 0:0:15.
#1857
- No description of RESTRING for Grid Engine complexes configuration
Status:NEWPriority:P3Resolution:
Commentary: RESTRING is documented in the man pages but not in the N1GE Administration Guide. RESTRING is an input format that lets one use regular-expression-like syntax with some SGE arguments. One example would be in a resource request string which now (in SGE 6) can contain syntax such as: -l arch="*x24*|sol*"
Read more...
Posted in MailList Bits
Posted by chris
Mon, 07 Nov 2005 23:39:57 GMT
“One year ago this month …”
NAMD is:
“… a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems. Based on Charm++ parallel objects, NAMD scales to hundreds of processors on high-end parallel platforms and tens of processors on commodity clusters using gigabit ethernet. NAMD uses the popular molecular graphics program VMD for simulation setup and trajectory analysis, but is also file-compatible with AMBER, CHARMM, and X-PLOR.”
An email query today reminded me of NAMD/SGE integration notes I had posted to a wiki back in November 2004. The information still appears to be correct for SGE/N1GE 5.x and 6.x. The document still resides at:
http://www.ks.uiuc.edu/Research/namd/wiki/index.cgi?NamdOnGridEngine.
Posted in Application Integration | Tags NAMD | no comments
Posted by DanT
Fri, 04 Nov 2005 19:33:54 GMT
One of the short-comings of Grid Engine 6.0 (as compared to 5.3) is that a gap exists between when a job finishes and when the job shows up in the accounting tool. The result is that for a few seconds after a job ends, it is as though the job never existed. Many people have complained about it, and now it is fixed. With the next release of Grid Engine, that gap is closable.
A little background. One of the big changes going from 5.3 to 6.0 was making the qmaster multi-threaded. In 5.3, the qmaster was one big loop. With 6.0, the qmaster now runs in about 14 different threads. Among those threads is a timed event thread. The timed event thread is used to do things on a periodic basis. Many tasks that were handled in the main loop in 5.3 wandered into the timed event thread in 6.0. One of the things that moved over was the writing of accounting and reporting data to disk. In 5.3 the data was written as soon as it was available. In 6.0, to improve performance, the data is buffered before being written. The buffer period is controled by the flush_time parameter of the reporting_params in the global host configuration. The minimum value for this setting is 1 second.
The problem comes from the fact that two separate buffers are being written at the same time. One is the accounting information buffer, which is what qacct uses to find historical job data. The other is the reporting information buffer, which is used by ARCo to create a utility computing database. Because the reporting buffer produces massive amounts of information, in a normal system setting flush_time to any small value results in an overloaded qmaster. However, since the buffers are only flushed every flush_time seconds, there’s a gap between when the accounting information for a finished job enters the buffer and when it is written to the accounting file.
Read more...
Posted in Code & Building from source | Tags Accounting | 1 comment | no trackbacks
Posted by chris
Fri, 04 Nov 2005 15:15:45 GMT
Learning something new every day ...
Someone on the users mailing list recently asked how he could create a queue configuration that would only accept parallel jobs. The answer itself is pretty simple but the implementation methods depend on the major version of Grid Engine one is using.
Read more...
Posted in Administration, MailList Bits | Tags parallel
Posted by chris
Thu, 03 Nov 2005 17:57:48 GMT
This is going to be one of those posts that will be completely boring and uninteresting to most (if not all) people reading it. It may, however, someday and somehow, be of use to some poor soul googling for info on what those digits mean in the JAT_state element when dealing with qstat XML output. It also has scary implications for me since I have no idea how to handle bitmask operations inside XSL stylesheets.
A user parsing XML output from "qstat" posted a query to the dev list asking for information on interpreting the various integers such as "128" and "2112" he was seeing as values for the JAT_state XML element. By way of explanation, "JAT" in this scenario means "Job Array Task".
The answer is short, but needs lots of explanation and accompanying data. It turns out that the decimal values seen in JAT_state are "the SUM of all applicable JAT bitmask status codes".
For a listing of JAT-applicable bitmask status values and the stunning conclusion where the real meaning of JAT_state=2112 is finally revealed please read on...
Read more...
Posted in Grid Engine XML | Tags JAT_state, qstat, xml | no comments
Posted by chris
Tue, 01 Nov 2005 21:28:56 GMT
An upcoming release of Grid Engine tagged "SGE 6.0 Scalability Update 2" has been available for testing for some time now.
In a post to the dev mailing list, Mac McCalla provided some nice large cluster performance numbers. In the test scenarios the S2 snapshot delivered better performance (measured in job throughput terms) than the current production SGE 6.0u4 system.
Mac ran streams of 8,000 job submissions, each job with a runtime of 2 minutes and targeted at a set of 100 execution hosts. The theoretical 'fastest' time that this workload could be pushed through the 100 cluster nodes (with 2 job slots each) would be 80 minutes.
The results? The S2 snapshot, running on a qmaster set with 4k pagesize ran the 8,000 jobs in 113.6 minutes. This is significantly faster than the production 6.0u4 system which runs the same workload through in 184 minutes.
Congrats to the SGE developers for their efforts on scaling and performance. Mac promises additional data from a production workflow shortly.
An interesting thing about this post is that the speedup may also have something to do with pagesize settings on the qmaster. Anyone managing a SGE qmaster machine that deals with very large workloads should check out this thread on the Dev mailing list, in particular the first post from Ron that started the conversation. I'd really like to learn more about the real world effects of pagesize tweaks on SGE systems as it seems that the feedback on the dev list was that these techniques would not yield significant gains.
Posted in Administration, MailList Bits | Tags 6.0S2, largepage, pagesize, sge-performance | no comments
Posted by chris
Tue, 01 Nov 2005 15:41:35 GMT
Idle Question: What sorts of interesting things have happened to the gridengine codebase and Issue tracking system in the last 7 days?
Answer: IssueZilla
knows all.
Summary:The following is a summary of recent issues and RFEs opened, updated, closed or resolved within the last 7 days.
Reopened issues:
#1798 - qconf -mattr can crash qmaster
Status:REOPENED
Commentary: Roland found a new failure case and reopened the issue. He reports, " 'qconf -mattr exechost complex_values = global' kills the qmaster in a recent maintrunk & s2 cluster "
New issues:
#1799 - qmaster messages error logging upon subordinated queue is removed
Status:NEW
Commentary:Roland found odd log error messages when a "subordinated queue is removed while a job is running in the superordinated queue...". He provided specific reproduction steps. DanT commented that he was unable to reproduce the error. The messages in question looked like: "|E|cqueue_list_locate_qinstance("sub@durin"):
cqueue == NULL("sub", "durin", 1, 0)" and "|E|got resend signal timer event for unknown
queue"
Read more...
Posted in MailList Bits
Posted by chris
Tue, 01 Nov 2005 00:45:00 GMT
A user recently asked:
... I know it is possible to submit the jobs with nice in each submit
sge script but is it possible to fix a nice value by queue ?
Reuti replied with the quick & simple procedure:
- Use "qconf -mq " to set the cluster queue "priority" parameter to the nice value you wish to use
- Use "qconf -msconf" to make sure "reprioritize_interval=0:0:0"
- Use "qconf -mconf" to make sure "reprioritize=0"
A quick test to verify the commands, setting priority=15 and sure enough the test.sh script was running with an altered nice level:
USER PID TIME UID PPID CPU NI COMMAND
dag 6843 7:12PM 501 6679 0 31 - sge_shepherd-5 -bg
dag 6844 7:12PM 501 6843 0 16 -sh /opt/sge/default/spool/chrisdag/job_scripts/5
Posted in MailList Bits | Tags Snippets
