pod-remote

Anar here...

It has been a long time since my last post... :) 

I am very excited right now, because in this post I am going to introduce to you a new PoD development. Another big step forward for PoD!

This is a fresh stuff, which is not completely finished yet, but already in a good and usable shape. ;)

So, it is pod-remote. A new command of PoD.

I've been dreaming for a long time about having a possibility to control PoD Servers from a remote machine. I mean, I have all my work, development, analysis scripts on a laptop and I want to use some remote machines as my PoD/PROOF servers, some with LSF, some with PBS, some with SSH plug-ins.

But my work stays on my laptop (or other dedicated desktop machine) and hasn't to be copied if I want to use different PoD/PROOF servers.

  

PoD UI and PROOF UI are lightweight entities and they don't do much. I even might want to have my UI on an iOS device ;)

With pod-remote my dream is coming to reality and starting to have a shape...

Using pod-remote it is possible to start/restart/stop remote PoD servers. It is possible to submit jobs from remote PoD servers. Most importantly, pod-remote automatically creates and handles SSH tunnels for remote PoD servers, so that these servers can be used only via SSH connection - outside of a firewall. Another important feature of pod-remote is its integration into PoD.

There is no yet a documentation for pod-remote, but I can give you a simple use case, so that you get the picture.

Let us say, I have two machines A and B.

A (host: demac012) - is my laptop.

B (host: lxg0527) - is a machine standing somewhere at my work place. From this machine I can submit jobs to my batch system (LSF) or use it as a server for PoD SSH plug-in.

On both machines I have PoD installed.

All the following I do from my laptop (machine A, "demac012").

1. Start the remote PoD server:  
   pod-remote --start --remote manafov@lxg0527.gsi.de:/misc/manafov/PoD/3.5.75.gbecd4 --env ../GSI_env_5_27.sh
   
   --start - start the server
   manafov@lxg0527.gsi.de:/misc/manafov/PoD/3.5.75.gbecd4 - a connection string to the server, including a full path to a PoD installation
   --env - an environment file, which will be executed on the remote-end before PoD starts the server. In my case I just source a proper ROOT version in it. You can set some more env. variables if needed.
   
   pod-remote will remember your settings and next time you want to stop/start/restart this server you can just write: pod-remote --start(stop/restart)
   without --remote and --env. The command will always use the latest given settings.
   If you want to change the server, just provide new arguments values.
   
   if everything is OK and remote server is started, pod-remote will create and manage special SSH tunnels from machine A to B. So, the whole PoD communication and PROOF requests will go via these tunnels.
   Tunnels stay alive until remote server is alive or you restart/stop pod-remote. The pod-remote command creates a background daemon, which regularly checks the status of the remote PoD server and manages tunnels.  
   
   
2. Submit PoD jobs from the remote Server (in case of RMS plug-ins):
   pod-remote --command "pod-submit -r lsf -n 50 -q my_lsf_queue"
   or in case of ssh plug-in
   pod-remote --command "pod-ssh -c --submit"
   
   Using --command, you can execute any command vie SSH on the remote server.
   I am going to keep it that way for a timebeing, and see whether it is usfull to wrap "submit commands" into sub-options or keep it as generic as it is. I like to have it that way and let users have the flexibility to execute any remote command and wrap commands for their special needs/environments.
   
3. Now, you can just use pod-info as usual, as if everything would run locally:
   pod-info -s
   pod-info -c
   pod-info -n
   ...
   The pod-info automatically detects that there is a pod-remote-managed server and will gather the information directly from it via the SSH tunnels.
   It means, of course, that to connect from your local machine (a laptop) to your remote PoD/PROOF cluster you need just to use:
   TProof::Open(gSystem->GetFromPipe("pod-info -c"));
   
   That's it!

The second time I want to use the remote server I have to just execute the following on my local machine:  

pod-remote --start (note: no other arguments, pod-remote remembers my settings)

pod-info -n (note: no need to use --remote here. The pod-info command handles pod-remote automatically)

root[] TProof::Open(gSystem->GetFromPipe("pod-info -c"));

to stop the server: pod-remote --stop

or it will use standard PoD idling timeouts...

You can use your laptop, your development machine as a command center.
Don't need to copy analysis code or to have different machines for different PROOF clusters.
Don't need to handle SSH tunnels by hands and so on and so on...

Please give it a try and send me your feedback. The pod-remote command is available in PoD starting from PoD v3.5.75.

The pod-remote command will be a part of upcoming stable release PoD v3.6.

If you want to try this build, please download the latest Beta: http://pod.gsi.de/download.html

In case of Problems: http://pod.gsi.de/support.htm

P.S. As soon as PoD v3.6 is out I am going to record a screen-cast demonstrating pod-remote use cases ;)

LoadLeveler plug-in

Anar here...

I've just released a new PoD beta. In this version we have a new job manager plug-in implemented.
Now PoD supports IBM Tivoli Workload Scheduler LoadLeveler!

A special thanks to Simon Heisterkamp (Copenhagen, Denmark), who helped a lot in development of the plug-in.


If you want to try this build, please download the latest Beta: http://pod.gsi.de/download.html
In case of Problems: http://pod.gsi.de/support.html

New PoD WN

Anar here...

I've just released a new beta build of PoD. This version got three major improvements:

1. PoD WN has now been officially ported to Mac OS X. That means PoD now fully supports Linux and Mac. You can run PoD Server, PoD UI or PoD WN on Mac OS X (and/or Linux). While testing this feature today, I set up a dynamic PROOF cluster of 46 WNs using the SSH plug-in, containing different Linux versions (Debian Etch, Debian Lenny) and two Mac OS X workers. :D Thanks to PROOF and PoD you now are able to combine different OSs and architectures in your dynamic PROOF clusters.
2. Each PoD WN starts its own xproofd daemon now. This feature makes it easer to handle PoD workers individually. The feature is also important for PoD and AliEn integration.
3. The start up time of PoD WNs is significantly improved in this version of PoD.

If you want to try this build, please download the latest Beta: http://pod.gsi.de/download.html

In case of Problems: http://pod.gsi.de/support.html

PoD at CERN's lxplus. Part 2.

Anar here...

PoD at CERN's lxplus. Part 1.

Today I've been testing a new beta (v3.2.61.geeb0) of PoD and same as always I have been playing with it at CERN's lxplus (LSF) just to check whether this version properly behaves on AFS.

It is really surprising to see how effective CERN's LSF works. At GSI we have a special queue for PoD - preemptive queue, in order to provide as much as possible interactivity. I and other PoD users always get PoD very fast at GSI. But at CERN I used a standard "1nh" queue which was full of pending jobs and since my share should be good at CERN's LSF (I almost don't use the cluster, only doing my tests time to time) I got my PoD jobs through just within 40 seconds - almost a record :)

So, simply to say I got a dynamic PROOF cluster of 37 workers just in 40 seconds ;)

Ufff... Obviously CERN's LSF foreshore works perfectly correct.

If I used LSF at CERN more intensively, then I would get my worker up and running a bit later, I guess. Depends on how intensive I used it. There is no magic - fare share. This is why, if you want to provide the maximum interactivity for PoD user on your cluster, you need to tune a bit your cluster or make a dedicated queues. This is a good trade for a fast dynamic PROOF cluster, which will give resources back to batch users as soon as nobody use it.
Anyway, as we can see, even without any pre-configuration of resource management clusters, PoD is very fast and more than usable ;)

I would be very grateful if somebody at CERN, who use LSF intensively than I do, would test PoD and report back how fast he/she gets PoD works online.

BTW, since PoD now in a redesign stage. I wouldn't recommend PoD users to use PoD CLI instead of PoD GUI. I am currently working on a new GUI which will also allow to work with remote PoD servers and will reflect the latest development of PoD.

Some screenshots of my CERN tests of today:

1. start PoD server:

2. submit PoD jobs to CERN's LSF:

Just ~40! seconds later:

3. check how many workers we got already and which workers are they:

4. we have our dynamic PROOF cluster and now we can process a PROOF analysis as usual:

Here is another test I did, just for the sake of demonstration I used the "date" command to show the current time. Of course I also have PoD logs as a evidence :D

1. at 11:41:14 requested 40 PoD workers:

2. at 11:41:50 I got my first 18 workers.

3. at 11:42:10 I got all requested workers:

So, in the second test it took me ~36 seconds to get the first half of the workers and in ~ 1 min I got my last requested worker online.

Actually I could start my analysis as soon as I got some reasonable amount of workers, for example more than 30 workers and it was just about 40 seconds of waiting since I requested them. The rest of the workers will be connected to PoD automatically as soon as they online and ready. Also if I want, at any time, I can submit more workers and they also will be connected to my cluster automatically. ;) PoD is very flexible.

Meet us on http://pod.gsi.de

Condor plug-in for PoD

Anar here...

As it was promised, I am very happy to release today an early beta of a Condor plug-in for PoD.

Before saying anything else I would like to show a screenshot-demo :)

In order to test the plug-in I have used a Condor cluster at CERN (special thanks to Wen Guan, who has helped me with that).

Now the demo. 

I will show PoD CLI, since the Condor plug-in doesn't support GUI yet.

First of all, same as always, PoD server needs to be started:

than I submitted 6 PoD jobs, using a Condor job manager (this is done via the plug-in). I submit only 6, because we were in hurry and there are only 2 worker nodes on which my account was enabled.

After a few seconds we can check whether there are already PROOF workers online and how many of them:

That's it! Our dynamic PROOF cluster is ready:

So, from now on PoD supports Condor resource management system as well. Which makes it 6.

Six supported RMS.

PoD supports: SSH, LSF, PBS (PBSPro/OpenPBS/Torque), Grid Engine (OGE/SGE), Condor, and gLite.

All most popular RMS are supported. If you need PROOF for your users, just get one of these RMSs and PoD will handle the rest ;)

No need to administrate, to setup and support any PROOF cluster. Each user can setup its own PROOF cluster on the fly using PoD just on demand.

The stable release of PoD, which will be shipped with the Condor plug-in is scheduled for the first week of January, 2011.

I think there is no reason to be hurry and to do the release before the upcoming Holidays ;) We rather can use this time to do more tests and improve User's manual...

But you shouldn't wait until the stable is out. Try the Condor plug-in right now and help us to improve the product. If you want to check this release out, please use the latest nightly build:

http://pod.gsi.de/releases/pod/nightly/

Contact us in case of any problem: http://pod.gsi.de/support.html

New beta of PoD 3.x series

A new PoD development (beta) release of upcoming PoD v3.1 was issued today...

Most of the work in this beta revision was concentrated on the shared installation feature, plug-ins compatibility, and tests on different RMSs (including SSH plug-in).

BTW, today I've also installed the first central PoD at my home institution - GSI. Now users at GSI can either use a centrally installed PoD by using a simple login script or install any PoD version privately in their desired locations.

If you want to check this release out, please use the latest nightly build:

http://pod.gsi.de/releases/pod/nightly/