PoD Shared Installations

Anar here...

Some time ago users start to ask for a, so called, central installation or a shared installation feature.

PoD now supports both types - Private and Shared installations.

• A Private Installation - it is when a user installs PoD for individual use to his/her local folder. Any Private Installation can be used by other users as well. It's just a matter of file privileges.
• A Shared Installation - it is when a site administrator installs PoD in some central location, so it can be shared by many users. It makes it easeir for users, since they don't need to install PoD by their own. In case of a shared Installation you need to execute one additional step. All the rest is the same as with Private Installations.

Be advised, that in both cases PoD acts identically and always provides private clusters, one for each user. In case of a shared installation, users share only binaries and configurations, but each user get's its own PoD instance and can't disturb other users. 

Each user can tune PoD by using its own PoD user defaults configuration in $HOME/.PoD/PoD.cfg.

This feature is still in beta and if you like to test it or just play around, than try the latest PoD nightly: http://pod.gsi.de/releases/pod/nightly/

I am going soon to release a new version of PoD, which will include this feature, but before the release I want to finish updating User's Manual to reflect all changes and to process some more tests.

If you can test the latest PoD nightly and give some feedback, it would be great help and can speed up the release ;)

ROOT's make to list all targets

Apparently there is one more remarkable commit (revision: 36664) to ROOT I want to mention today, since I was waiting for it.

The r36664 is actually implements a possibility to list targets via the make command (the feature, which cmake user can enjoy since long time already).

Now, ROOT users could issue the following command to find out all available targets of their ROOT source tree:

make help

Try it with your ROOT source tree. Very nice, isn't it?

I think it's a very user-friendly improvement in ROOT's build system.

ROOT supports out-of-source builds

Yesterday Fons has committed a very handy feature to ROOT, see revision #36659

Now ROOT supports out-of-source builds. 

What is an "out-of-source" build?

When you build a ROOT source, the build process generates files and they have to go somewhere. In in-source builds they were in your ROOT's source tree. The ROOT out-of-source build puts them in a completely separate directory, so that your source tree is unchanged.

I personally prefer only out-of-source builds. Even in PoD User's Manual I recommend this as a default installation procedure. This way builds are more elegant and clean. I can produce several different configurations in different build directories for a single source tree.

So, now ROOT also supports it and it is great offer to users! That was also not easy to implement in ROOT, see the number of changed files in the revision. Great job ROOT Team!

Author: rdm

Date: Mon Nov 15 11:07:02 2010

New Revision: 36659

URL: http://root.cern.ch/viewvc?rev=36659&root=root&view=rev

Log:

add support for out-of-source building of binaries. Normal build is unaffected.

To make an out of source build do, assuming the source is in ~/root:

  mkdir ~/root-x8664

  cd ~/root-x8664

  ~/root/configure

  make

This is convenient to build e.g. 32 and 64-bit version from one source, but

also needed for cross-compilation, where it is now possible to build in

a special directory only the compile time tools, like rootcint for the

host architecture while the binary is build for the remote architecture

(like iOS).

PoD 2.4

I am very glad to announce the release of PoD 2.4.

This version releases many important changes and among others, there is the main event, which is the first stable release of an Oracle Grid Engine (OGE) plug-in for PoD.

OGE is fully supported now in both PoD GUI and PoD CLI. OGE is a great RMS with a lot of sites using it. I think this PoD plug-in will help to convince many potential PoD customers to give our product a try ;)


The OGE plug-in has just joined a list of supported job manager's plug-ins.
PoD is now shipped with gLite, LSF, PBSPro/OpenPBS/Torque, SSH, and OGE plug-ins.

Highlights of this release

• Support of Oracle Grid Engine. PoD is now shipped with the OGE plug-in.
• A new schema of WNs' pre-compiled binaries handling.
• PoD now a bit more smarter and more efficient, when a shared home file system is present on UIs and WNs.
• The startup time of PoD jobs is notably improved.
• Several security issues were fixed.
• This version implements a better logging machinery for PoD workers.
• The Job Manager plug-in system is slightly improved. It now makes it much easier to extend with new plug-ins and to support old ones.
• Updated User's Manual.

PoD Home

Release Notes

PROOF Home

PoD supports OGE/SGE

Some more updates on the upcoming PoD v2.4 ;)

I finally finished the implementation for GUI of the Oracle Grid Engine (OGE) plug-in. Since a month OGE was supported by PoD, but only CLI. Now, starting from PoD v2.3.100.gac84 the OGE is fully supported in both, PoD GUI and PoD CLI.

As you can see, the interface is very simple and easy to use:

Using PoD GUI you can submit PoD jobs to an OGE cluster, check status of jobs, and kill jobs :)

PoD also supports an OGE option file. If you want to provide some additional Grid Engine options to your PoD jobs submitted to OGE cluster, to select some specific resource or something like that, than PoD gives you this possibility via an OGE option file. Just create a file $POD_LOCATION/etc/Job.oge.option and write valid OGE options in it. If the file exists, than PoD will automatically use it while submitting OGE jobs.

See qsub man page of OGE for more information on the option file (search for the "-@" option in the man page).

Pre-compiled binaries on worker nodes

Today I continue to give an overview of upcoming PoD 2.4 ;)

In this post I will cover a new behavior of PoD in terms of how it handles its workers now.

What we had before is that each PoD worker during the startup was checking environment of the machine it runs on and according to OS and CPU architecture tried to download a pre-compiled binaries of the needed PoD modules.

The binaries were downloaded from the central PoD repo. The binaries are needed, since PoD could run on different type of workers with different architectures. Talking about binaries, these are  only two modules needed so-far, but they are very important ones. Namely, pod-agent (core module of PoD) and pod-user-dsefaults (handles PoD settings). 

As it was pointed out by one of PoD users, this schema has many disadvantages and security issues. Indeed, having each worker (in each PoD session)  to download "some" binaries from the outside of a site or an institution is very inconvenient and insecure. This schema also makes it very difficult for users to provide their own binaries, if, for example, some type of architecture is missing from PoD central repo, but user want's to support it.

This is actually very old schema and was introduced at the time when PoD was called gLitePROOF (designed for gLite Grid MW).

Now, since PoD 2.3.84, I've changed this behavior and introduced a new schema. 

PoD workers don't download pre-compiled bins from the PoD central repository anymore. Now only PoD UI downloads them if they are missing from the UI. When a user starts PoD server ("pod-server start") a package with pre-compiled  binaries will be downloaded. That actually means, it will only be downloaded once to one machine and users could also provide own binaries, instead of using the PoD central repository. If a user will provide binaries, than PoD will not download anything from the central repository.

Additionally, PoD workers now try to find out whether a shared home file system is present and whether they can share the binaries with PoD UI/Server. If a shared FS is present, than workers will try to use UI's bins directly and if bins fail to start, workers will use pre-compiled bins from the worker package.  The new schema has resolved security issues and made PoD much more flexible. BTW, the start up time of the workers is also slightly reduced.

I'll keep you updated, so stay tuned!

PoD UI/Server on Mac OS X

Anar here...

As I promised, I am going to make several posts dedicated to new features of upcoming PoD v2.4.

Today's new feature or better to say possibility is "PoD UI/Server on Mac OS X".

Currently PoD UI and PoD Server are always on the same machine and can't be separated. A more preferable use case would be having UI on the laptop and a possibility to command to have PoD server somewhere else, maybe on a remote cluster, possibly close to PoD workers. This use case gives the maximum performance and flexibility for users.

We have a plan and we are working on disentangling of PoD UI and Server. So that user will be able to run PoD UI and PoD Server on different machines.

As a small step towards accomplishing the goal of disentangling, I have just finished porting PoD UI/Server to Mac OS X. So far only Command Line Interface and only the SSH plug-in are working and have been tested. But since main core modules have been just ported to support Mac, the other plug-ins will be easily ported as well.

You can already try running PoD on Mac using any build equal or higher than PoD-2.3.83. Check PoD nightly builds up: http://pod.gsi.de/releases/pod/nightly/

Download statistics

Anar here...

It has been a long time since my last post ;) You know, I was busy with PoD development and stuff.

There are many new features implemented in PoD and I am doing my best to get them all tested and available in the next release of PoD, which I am planning to make in a week or so. In my next posts on this blog I am going to highlight some features. So, please, subscribe for updates or follow me on this blog (on the right you see the followers bar)...

Anyhow this post is not exactly about PoD, but about a PoD web site update. There are some cosmetic changes, and among others I have implemented an automatic publishing of PoD download statistics. It is time, I think, to start to get some statistics (it's better to be early than late). You can check it out here.

So far we have around 130 Downloads since summer 2010. I think it is very good number, since PoD is a specific product for a specific field. 

As you may know or have noticed, pod.gsi.de is a docbook based website. Since long time I am in love with docbook (PoD User's manual http://pod.gsi.de/documentation.html is made in docbook) and even the web site of PoD is also a docbook based. I am using docbook Website schemas and very happy with it. For a docbook Website concept I will dedicate a separate post some time later.

Back to the subject. The stats are generated automatically every night. Since I don't have a direct access to our Web Server and its logs, it was quite tricky to implement.

The whole process is completely automated.

I am getting the log via a script. The stats are parsed by a simple C++ program, which outputs a docbook page. The page is than committed to PoDWebsite Git repo. As soon as the Git repo updated with a new commit, buildbot will automatically start a build of PoDWebsite and if parsing and conversion to html are ok, the site will be published. WoW :) So, simple is that. I love automations!

PoD Download Statistics