PoD 3.12

Anar here...

finally after a very long vacation time and other stuff I am very glad to release a new stable version of PoD.
PoD v3.12 is out and ready for consumption.

Highlights of this release

• Worker packages are now added as a payload to the worker script. (more details on this one you can find in my presentation on the CHEP 2012 also here)
• Added a support for BOOST 1.50 and Boost.Filesystem version3.
• PoD installations are relocatable now.
• Updated User's Manual.
• Many minor bugfixes and further improvements were made.

please check the release notes for more details.

PoD Home
Release Notes
PROOF Home

PoD at CERN's LSF

Anar here...

Many PoD user facing a problem running PoD at CERN on LSF due to the fact that on the worker nodes for some reason a gLite environment is setting up. I therefore has just created a know issue topic on how to workaround this issue.

PoD in PROOF connection string

Anar here...

I am very glad to confirm, that PoD is now officially supported as one of the PROOF connection strings.
To connect to a local or remote (managed by new pod-remote) PoD cluster you can just enter:

TProof::Open("pod://");

This feature is already in the trunk of ROOT and will be released with the next production ROOT 5.30.

Video Tutorials

A short news about PoD website update.

PoD's home website got a "Video Tutorials" page. There will be all screencasts about PoD collected.

PoD Screencasts. Part 2.

Part 1.

Anar here...


I have just recorded another screencast. This time it is about how to setup a dynamic PROOF cluster on a resource management system. I used CERN's LSF farm as a playground and PoD 3.4.

Enjoy:

PoD Screencasts

Anar here...

I think you would agree, that watching a live demo is much better than just reading a presentation slides or something ;)
This is why I decided to start recording screencasts about PoD. In the future screencasts you will see how to install PoD, how to use different plug-ins of PoD and so on and so on.

I've just started from a simplest screencast - on how to install PoD (see below).
This is the very first public screencast in my life, so, please, be gentle ;)

A PROOF cluster in 7 seconds

Anar here...

Can anybody install a PROOF cluster (dynamically, from zero, on requested machines) in 7 seconds?
With PoD anybody can do that!


A bit of propaganda... ;)

I have just finished some small improvements of PoD's SSH plug-in and also decided to give it a bit of propaganda push once again.

Among other changes I made the plug-in a bit faster and also now if a user doesn't request a specific number of PROOF workers for some of the worker nodes, then pod-ssh will automatically setup PROOF workers on that worker node according to a number of CPU cores. It means, that the last parameter in the pod-ssh configuration file is not required anymore and can be left empty.

The screenshot below is just a demonstration of how easy and fast pod-ssh works.

In the demo, I have used the following configuration file:

r1, manafov@lxi020.gsi.de , , /misc/manafov/tmp/test,

r2, manafov@lxi021.gsi.de , , /misc/manafov/tmp/test,

r3, manafov@lxb333.gsi.de , , /misc/manafov/tmp/test,

r4, manafov@lxir036.gsi.de, , /misc/manafov/tmp/test,

r5, manafov@lxir037.gsi.de, , /misc/manafov/tmp/test,

r6, manafov@lxir038.gsi.de, , /misc/manafov/tmp/test,

#r_32bit, manafov@lxi009.gsi.de, , /misc/manafov/tmp/test, 

in order to use all described machines as my PROOF slaves.

For the sake of demonstration I wrote a simple script, just to show the timestamp of the execution of the all PoD command required to setup a cluster:

There is nothing special in the script, just the usual PoD command sequence:

pod-server start

pod-ssh (in case of SSH. For other RMS plug-ins it is "pod-submit")

and

pod-info

(refer to PoD user's manual for more information)

The screenshot below shows the output of the script:

It took me exactly 7 seconds to 

1. Start a PoD server (PROOF master).

2. Submit PoD jobs to 6 worker nodes using ssh.

3. Setup 44 PROOF workers out of these 6 PoD workers.

These worker nodes are just interactive machines or desktops standing in my institution somewhere. These boxes have no pre-installed PROOF software and are being configured on the fly by PoD.

At night I can count much more machines, which nobody use. With PoD and the SSH plug-in I can make a dynamic PROOF cluster out of them at any time just in seconds :)

If you want to try the latests PoD features, then use the latest beta.

PoD 3.1.3

Anar here...

While working on a new PoD version I noticed an unpleasant bug, which was introduced with some new stuff in PoD 3.1. I decided to fixe it not only in the PoD master branch, but also back-port fixes to PoD 3.1.

The bug is not critical, but I would strongly recommend all PoD users, who use PoD 3.1 to upgrade to PoD 3.1.3, which has been just released. Otherwise in some cases PoD could leave xproof processes behind and slowly pollute user's environment with unwanted daemons.

Some details about the bug can be found here and here.

I am very sorry for any inconvenience this may have caused.

PoD is being constantly tested in different environments and as soon as a critical bug found a patch will be immediately released.

If you found a bug or have a feature request, please use our new issue tracker to report.

PoD 3.1

Finally a new PoD stable version is ready for consumption.

PoD version 3.1 has now been released.

Those who have been following PoD development know, that a shared installation feature and a Condor plug-in are the main topics of this release.

The Condor plug-in has just joined a long list of supported job managers' plug-ins.

PoD is now shipped with SSH, gLite, LSF, PBSPro/OpenPBS/Torque, Grid Engine and Condor plug-ins.

Highlights of this release

• PoD now supports Condor and is shipped with the Condor plug-in.
• Now PoD supports shared installations.
• PoD server and PoD worker instances clean now precisely only own processes. It makes it possible to start several different PoD workers/servers sessions under the same user ID (this is important for an AliEn integration).
• The PoD server starter script is a factor of 3 faster now.
• Improved idle time-out handling in PoD servers.
• Updated User's Manual.
• Many minor bugfixes and further improvements were made.

PoD Home

Release Notes

PROOF Home

Condor plug-in for PoD

Anar here...

As it was promised, I am very happy to release today an early beta of a Condor plug-in for PoD.

Before saying anything else I would like to show a screenshot-demo :)

In order to test the plug-in I have used a Condor cluster at CERN (special thanks to Wen Guan, who has helped me with that).

Now the demo. 

I will show PoD CLI, since the Condor plug-in doesn't support GUI yet.

First of all, same as always, PoD server needs to be started:

than I submitted 6 PoD jobs, using a Condor job manager (this is done via the plug-in). I submit only 6, because we were in hurry and there are only 2 worker nodes on which my account was enabled.

After a few seconds we can check whether there are already PROOF workers online and how many of them:

That's it! Our dynamic PROOF cluster is ready:

So, from now on PoD supports Condor resource management system as well. Which makes it 6.

Six supported RMS.

PoD supports: SSH, LSF, PBS (PBSPro/OpenPBS/Torque), Grid Engine (OGE/SGE), Condor, and gLite.

All most popular RMS are supported. If you need PROOF for your users, just get one of these RMSs and PoD will handle the rest ;)

No need to administrate, to setup and support any PROOF cluster. Each user can setup its own PROOF cluster on the fly using PoD just on demand.

The stable release of PoD, which will be shipped with the Condor plug-in is scheduled for the first week of January, 2011.

I think there is no reason to be hurry and to do the release before the upcoming Holidays ;) We rather can use this time to do more tests and improve User's manual...

But you shouldn't wait until the stable is out. Try the Condor plug-in right now and help us to improve the product. If you want to check this release out, please use the latest nightly build:

http://pod.gsi.de/releases/pod/nightly/

Contact us in case of any problem: http://pod.gsi.de/support.html

Downloads by Country

Anar here ... with a bit of propaganda right now... :)

I am very glad to see that PoD is being used or at least evaluated in many countries already!

The statistics below shows the downloads of PoD source tarball for the last 6 months. It is only 166 downloads by about 12 countries. 

"It's a very small number" - would you say.  But don't forget, that for such a young and specific product like PoD, which works on a very specific field - it is a great success. Every new user is a success, I think.

Since PoD supports a shared installation feature now, I expect the download counts to dramatically reduce. Due to the fact, that in most cases PoD is and will be downloaded only once for a site and shared between several users on that site.

BTW, I also expect more customers from the USA to join, as soon as a Condor plug-in for PoD is released. The plug-in now in development and expected in an early beta in several days.

Anyway the downloads count doesn't really matter to us. It is just interesting to see how our product slowly gets maturer.

What is really important to us is to create a product, which will help, simplify and populate a PROOF-like-analysis.

New beta of PoD 3.x series

A new PoD development (beta) release of upcoming PoD v3.1 was issued today...

Most of the work in this beta revision was concentrated on the shared installation feature, plug-ins compatibility, and tests on different RMSs (including SSH plug-in).

BTW, today I've also installed the first central PoD at my home institution - GSI. Now users at GSI can either use a centrally installed PoD by using a simple login script or install any PoD version privately in their desired locations.

If you want to check this release out, please use the latest nightly build:

http://pod.gsi.de/releases/pod/nightly/

PoD at CERN's lxplus

Anar here...

Today I've been testing the latest PoD nightly on different resource management systems and decided to test it at CERN as well. I have CERN's lxplus always in my list of thinks to test on, because of AFS - you know it has some restrictions, like pipes and so on. So, it is always worth to play with PoD at CERN.

I was quite surprised that PoD deployed for me 50 PROOF workers within 30 seconds! I was using standard LSF queue, named "1nh". Besides, the LSF looks quite busy these days... and you know why ;)

But think about it, these are just regular LSF machines at CERN and you get your private PROOF cluster just in two clicks (or two commands) in a matter of seconds - dynamically...  ;)

Usually I use PoD command line interface, but just for demonstration's sake I used its GUI:

I mean, look, I get in less than 30 seconds exactly the same number of workers as you can get on CAF (where a standard "static" PROOF cluster is installed). Taking into account, that CAF allows to run more that one worker per core, my PoD/PROOF cluster on LSF will be much faster. So these 50 on LSF (one worker per core), which PoD made for me will process my analysis faster and even so I can use CAF machines as storage elements ;)

Let's make another test: 100 workers. This can't be even compared to CAF, because regular users can't get more than 52 ;)

Starting PoD using its CLI and submitting 100 workers:

 

40 seconds later:

:D

Of course I understand, that the start up time on LSF or any other RMS depends on many factors, but even without pre-configuring RMS, I will be and I am able to get my PROOF cluster fast and on demand.

Additionally, I've advantages by using PoD, which are:

- it is my PROOF cluster (I don't have anyone who administrates it for me),

- LSF (or other RMS) takes care of resource sharing, distributing, logging, accounting and so on...,

- I am free to use this cluster at anytime I want with any ROOT version I want,

- I can't misuse the cluster, since RMS accounts it,

- etc.

In case you want to try the latest PoD at CERN, use the latest nightly or stable release.  For users' convenience I also provide some environment settings for lxplus, check the "PoD at CERN's AFS" section.

I would of course recommend trying the latest nightly, even so it still has some bugs, but you can help to evaluate it ;)

News on PoD Server

Anar here...

I want to continue to keep you informed on ongoing development of the upcoming stable release of PoD - v3.1.

This time I'm going to cover a new stuff in PoD server implementation, which were done recently.

First of all, the server now starts 3 times faster. A logic of start/stop/restart functions was changed and improved.
I have also improved a server processes discovery algorithm. This algorithms is responsible to find out which of server processes run( uid, pid, and tcp port it listens on). PoD server is a lot more smarter now when handling its processes, even third-party ones, like xproofd and its company. For example, the server implementation got a two-stage-shutdown machinery, which first tries to gracefully shutdown all its processes and only if a process ignores it, the server will send a non-ignorable kill signal to it.
The server now also smartly kills the children of xproofd (like proofserv). It kills not all of children, just by name and user id, but rather it kills only children processes produced by the xproofd, which were started by exactly this PoD server. This brings us one step further in PoD and AliEn integration as well, since it is now possible to start several PoD servers under the same user id - which is a requirement for AliEn Grid. On AliEn all processes run under the same user id.

What is also new in the latest releases is that there is a better support for shared installations. Many minor bugs were fixed. The xproofd configuration, which PoD creates on the fly, is now handled a bit differently. Namely, it stored in user's working directory and renewed every time user restart pod server. This helps to use several PoD versions with the same user's configuration.

If you want to check the new PoD server out, please use the latest nightly build:
http://pod.gsi.de/releases/pod/nightly/

PoD Shared Installations

Anar here...

Some time ago users start to ask for a, so called, central installation or a shared installation feature.

PoD now supports both types - Private and Shared installations.

• A Private Installation - it is when a user installs PoD for individual use to his/her local folder. Any Private Installation can be used by other users as well. It's just a matter of file privileges.
• A Shared Installation - it is when a site administrator installs PoD in some central location, so it can be shared by many users. It makes it easeir for users, since they don't need to install PoD by their own. In case of a shared Installation you need to execute one additional step. All the rest is the same as with Private Installations.

Be advised, that in both cases PoD acts identically and always provides private clusters, one for each user. In case of a shared installation, users share only binaries and configurations, but each user get's its own PoD instance and can't disturb other users. 

Each user can tune PoD by using its own PoD user defaults configuration in $HOME/.PoD/PoD.cfg.

This feature is still in beta and if you like to test it or just play around, than try the latest PoD nightly: http://pod.gsi.de/releases/pod/nightly/

I am going soon to release a new version of PoD, which will include this feature, but before the release I want to finish updating User's Manual to reflect all changes and to process some more tests.

If you can test the latest PoD nightly and give some feedback, it would be great help and can speed up the release ;)

ROOT supports out-of-source builds

Yesterday Fons has committed a very handy feature to ROOT, see revision #36659

Now ROOT supports out-of-source builds. 

What is an "out-of-source" build?

When you build a ROOT source, the build process generates files and they have to go somewhere. In in-source builds they were in your ROOT's source tree. The ROOT out-of-source build puts them in a completely separate directory, so that your source tree is unchanged.

I personally prefer only out-of-source builds. Even in PoD User's Manual I recommend this as a default installation procedure. This way builds are more elegant and clean. I can produce several different configurations in different build directories for a single source tree.

So, now ROOT also supports it and it is great offer to users! That was also not easy to implement in ROOT, see the number of changed files in the revision. Great job ROOT Team!

Author: rdm

Date: Mon Nov 15 11:07:02 2010

New Revision: 36659

URL: http://root.cern.ch/viewvc?rev=36659&root=root&view=rev

Log:

add support for out-of-source building of binaries. Normal build is unaffected.

To make an out of source build do, assuming the source is in ~/root:

  mkdir ~/root-x8664

  cd ~/root-x8664

  ~/root/configure

  make

This is convenient to build e.g. 32 and 64-bit version from one source, but

also needed for cross-compilation, where it is now possible to build in

a special directory only the compile time tools, like rootcint for the

host architecture while the binary is build for the remote architecture

(like iOS).

PoD 2.4

I am very glad to announce the release of PoD 2.4.

This version releases many important changes and among others, there is the main event, which is the first stable release of an Oracle Grid Engine (OGE) plug-in for PoD.

OGE is fully supported now in both PoD GUI and PoD CLI. OGE is a great RMS with a lot of sites using it. I think this PoD plug-in will help to convince many potential PoD customers to give our product a try ;)


The OGE plug-in has just joined a list of supported job manager's plug-ins.
PoD is now shipped with gLite, LSF, PBSPro/OpenPBS/Torque, SSH, and OGE plug-ins.

Highlights of this release

• Support of Oracle Grid Engine. PoD is now shipped with the OGE plug-in.
• A new schema of WNs' pre-compiled binaries handling.
• PoD now a bit more smarter and more efficient, when a shared home file system is present on UIs and WNs.
• The startup time of PoD jobs is notably improved.
• Several security issues were fixed.
• This version implements a better logging machinery for PoD workers.
• The Job Manager plug-in system is slightly improved. It now makes it much easier to extend with new plug-ins and to support old ones.
• Updated User's Manual.

PoD Home

Release Notes

PROOF Home

PoD supports OGE/SGE

Some more updates on the upcoming PoD v2.4 ;)

I finally finished the implementation for GUI of the Oracle Grid Engine (OGE) plug-in. Since a month OGE was supported by PoD, but only CLI. Now, starting from PoD v2.3.100.gac84 the OGE is fully supported in both, PoD GUI and PoD CLI.

As you can see, the interface is very simple and easy to use:

Using PoD GUI you can submit PoD jobs to an OGE cluster, check status of jobs, and kill jobs :)

PoD also supports an OGE option file. If you want to provide some additional Grid Engine options to your PoD jobs submitted to OGE cluster, to select some specific resource or something like that, than PoD gives you this possibility via an OGE option file. Just create a file $POD_LOCATION/etc/Job.oge.option and write valid OGE options in it. If the file exists, than PoD will automatically use it while submitting OGE jobs.

See qsub man page of OGE for more information on the option file (search for the "-@" option in the man page).

Pre-compiled binaries on worker nodes

Today I continue to give an overview of upcoming PoD 2.4 ;)

In this post I will cover a new behavior of PoD in terms of how it handles its workers now.

What we had before is that each PoD worker during the startup was checking environment of the machine it runs on and according to OS and CPU architecture tried to download a pre-compiled binaries of the needed PoD modules.

The binaries were downloaded from the central PoD repo. The binaries are needed, since PoD could run on different type of workers with different architectures. Talking about binaries, these are  only two modules needed so-far, but they are very important ones. Namely, pod-agent (core module of PoD) and pod-user-dsefaults (handles PoD settings). 

As it was pointed out by one of PoD users, this schema has many disadvantages and security issues. Indeed, having each worker (in each PoD session)  to download "some" binaries from the outside of a site or an institution is very inconvenient and insecure. This schema also makes it very difficult for users to provide their own binaries, if, for example, some type of architecture is missing from PoD central repo, but user want's to support it.

This is actually very old schema and was introduced at the time when PoD was called gLitePROOF (designed for gLite Grid MW).

Now, since PoD 2.3.84, I've changed this behavior and introduced a new schema. 

PoD workers don't download pre-compiled bins from the PoD central repository anymore. Now only PoD UI downloads them if they are missing from the UI. When a user starts PoD server ("pod-server start") a package with pre-compiled  binaries will be downloaded. That actually means, it will only be downloaded once to one machine and users could also provide own binaries, instead of using the PoD central repository. If a user will provide binaries, than PoD will not download anything from the central repository.

Additionally, PoD workers now try to find out whether a shared home file system is present and whether they can share the binaries with PoD UI/Server. If a shared FS is present, than workers will try to use UI's bins directly and if bins fail to start, workers will use pre-compiled bins from the worker package.  The new schema has resolved security issues and made PoD much more flexible. BTW, the start up time of the workers is also slightly reduced.

I'll keep you updated, so stay tuned!

TORQUE/PBS plug-in

Anar here...

I've just finished the first stable version of a TORQUE/PBS plug-in for PoD.

One more plug-in which will be shipped by default with PoD distributions, additionally to gLite and LSF.

Worth to mention, that the LSF plug-in is already in production for ALICE users at GSI.

From now on PoD user can also make use of PBS facilities and setup private PROOF clusters on the fly using PoD.

The plug-in will be released with PoD v2.1.4 and currently it is in a beta.

I am looking for test users, who has access to PBS clusters and want help and try/test PoD. Any feedback is appreciated. ;)

If you want to test PoD (with pbs), just drop me an email on A.Manafov@gsi.de.

You can also just download the latest nightly and give it a try. More information here.

BTW, v2.1.4 will also come with a new command line interface. So now finally we have both, GUI and CLI ;)

The next plug-in is already in development - it is SSH plug-in. With help of it, any machine on which you have ssh access can become your PROOF worker on demand. ;)

See you soon...