PoD in PROOF connection string

Anar here...

I am very glad to confirm, that PoD is now officially supported as one of the PROOF connection strings.
To connect to a local or remote (managed by new pod-remote) PoD cluster you can just enter:

TProof::Open("pod://");

This feature is already in the trunk of ROOT and will be released with the next production ROOT 5.30.

PoD at CERN's lxplus. Part 2.

Anar here...

PoD at CERN's lxplus. Part 1.

Today I've been testing a new beta (v3.2.61.geeb0) of PoD and same as always I have been playing with it at CERN's lxplus (LSF) just to check whether this version properly behaves on AFS.

It is really surprising to see how effective CERN's LSF works. At GSI we have a special queue for PoD - preemptive queue, in order to provide as much as possible interactivity. I and other PoD users always get PoD very fast at GSI. But at CERN I used a standard "1nh" queue which was full of pending jobs and since my share should be good at CERN's LSF (I almost don't use the cluster, only doing my tests time to time) I got my PoD jobs through just within 40 seconds - almost a record :)

So, simply to say I got a dynamic PROOF cluster of 37 workers just in 40 seconds ;)

Ufff... Obviously CERN's LSF foreshore works perfectly correct.

If I used LSF at CERN more intensively, then I would get my worker up and running a bit later, I guess. Depends on how intensive I used it. There is no magic - fare share. This is why, if you want to provide the maximum interactivity for PoD user on your cluster, you need to tune a bit your cluster or make a dedicated queues. This is a good trade for a fast dynamic PROOF cluster, which will give resources back to batch users as soon as nobody use it.
Anyway, as we can see, even without any pre-configuration of resource management clusters, PoD is very fast and more than usable ;)

I would be very grateful if somebody at CERN, who use LSF intensively than I do, would test PoD and report back how fast he/she gets PoD works online.

BTW, since PoD now in a redesign stage. I wouldn't recommend PoD users to use PoD CLI instead of PoD GUI. I am currently working on a new GUI which will also allow to work with remote PoD servers and will reflect the latest development of PoD.

Some screenshots of my CERN tests of today:

1. start PoD server:

2. submit PoD jobs to CERN's LSF:

Just ~40! seconds later:

3. check how many workers we got already and which workers are they:

4. we have our dynamic PROOF cluster and now we can process a PROOF analysis as usual:

Here is another test I did, just for the sake of demonstration I used the "date" command to show the current time. Of course I also have PoD logs as a evidence :D

1. at 11:41:14 requested 40 PoD workers:

2. at 11:41:50 I got my first 18 workers.

3. at 11:42:10 I got all requested workers:

So, in the second test it took me ~36 seconds to get the first half of the workers and in ~ 1 min I got my last requested worker online.

Actually I could start my analysis as soon as I got some reasonable amount of workers, for example more than 30 workers and it was just about 40 seconds of waiting since I requested them. The rest of the workers will be connected to PoD automatically as soon as they online and ready. Also if I want, at any time, I can submit more workers and they also will be connected to my cluster automatically. ;) PoD is very flexible.

Meet us on http://pod.gsi.de

Downloads by Country

Anar here ... with a bit of propaganda right now... :)

I am very glad to see that PoD is being used or at least evaluated in many countries already!

The statistics below shows the downloads of PoD source tarball for the last 6 months. It is only 166 downloads by about 12 countries. 

"It's a very small number" - would you say.  But don't forget, that for such a young and specific product like PoD, which works on a very specific field - it is a great success. Every new user is a success, I think.

Since PoD supports a shared installation feature now, I expect the download counts to dramatically reduce. Due to the fact, that in most cases PoD is and will be downloaded only once for a site and shared between several users on that site.

BTW, I also expect more customers from the USA to join, as soon as a Condor plug-in for PoD is released. The plug-in now in development and expected in an early beta in several days.

Anyway the downloads count doesn't really matter to us. It is just interesting to see how our product slowly gets maturer.

What is really important to us is to create a product, which will help, simplify and populate a PROOF-like-analysis.

PoD at CERN's lxplus

Anar here...

Today I've been testing the latest PoD nightly on different resource management systems and decided to test it at CERN as well. I have CERN's lxplus always in my list of thinks to test on, because of AFS - you know it has some restrictions, like pipes and so on. So, it is always worth to play with PoD at CERN.

I was quite surprised that PoD deployed for me 50 PROOF workers within 30 seconds! I was using standard LSF queue, named "1nh". Besides, the LSF looks quite busy these days... and you know why ;)

But think about it, these are just regular LSF machines at CERN and you get your private PROOF cluster just in two clicks (or two commands) in a matter of seconds - dynamically...  ;)

Usually I use PoD command line interface, but just for demonstration's sake I used its GUI:

I mean, look, I get in less than 30 seconds exactly the same number of workers as you can get on CAF (where a standard "static" PROOF cluster is installed). Taking into account, that CAF allows to run more that one worker per core, my PoD/PROOF cluster on LSF will be much faster. So these 50 on LSF (one worker per core), which PoD made for me will process my analysis faster and even so I can use CAF machines as storage elements ;)

Let's make another test: 100 workers. This can't be even compared to CAF, because regular users can't get more than 52 ;)

Starting PoD using its CLI and submitting 100 workers:

 

40 seconds later:

:D

Of course I understand, that the start up time on LSF or any other RMS depends on many factors, but even without pre-configuring RMS, I will be and I am able to get my PROOF cluster fast and on demand.

Additionally, I've advantages by using PoD, which are:

- it is my PROOF cluster (I don't have anyone who administrates it for me),

- LSF (or other RMS) takes care of resource sharing, distributing, logging, accounting and so on...,

- I am free to use this cluster at anytime I want with any ROOT version I want,

- I can't misuse the cluster, since RMS accounts it,

- etc.

In case you want to try the latest PoD at CERN, use the latest nightly or stable release.  For users' convenience I also provide some environment settings for lxplus, check the "PoD at CERN's AFS" section.

I would of course recommend trying the latest nightly, even so it still has some bugs, but you can help to evaluate it ;)

PROOF progress dialog window

Yesterday Gerri has implemented a useful patch in PROOF. From the now on it is possible to switch the PROOF progress dialog off.

Add method TProof::SetProgressDialog(Bool_t on = kTRUE) to disable the progress dialog
window while preserving graphics from the final actions in Terminate. This may be useful
when X over network is slow. To disable the dialog the call

proof->SetProgressDialog(0)

has to be placed before Process(...).

Many users, who does PROOF remotely will find this as an extremely handy feature. ;)

The patch was committed as a revision 33562.