Download statistics. Part 2.

Anar here...

Download statistics. Part 1.

I got some free time on the holidays and slightly updated the download statistics page of PoD, in order to make it a bit more convenient.

A little bit of effort can make a beautiful combination of a docbook Website technology and Google Visualization API. I learned how to introduce custom docbook elements to docbook Website schemas and this helped to properly marry it with Google charts.

Stats are updated automatically and, thanks to docbook technology, the page is generated automatically as well. All is going via a Git repository. The whole process, including publishing of the site, is completely automated.

I am so much in love with DOCBOOK!!!

Now, concerning the stats. Since June, 2010, PoD was downloaded 179 times from 13 countries.

56% of downloads are made from different locations in Germany. Not a surprise, since PoD was started in terms of the D-Grid project and was developed in Germany and the first stable version was used in GSI (Germany), where, for the first time in the world ;), a production "static" PROOF cluster was staged out in favor of PROOF via PoD.

I will be not tired to repeat, that these are very good results for such a specific product like PoD is.

Check the stats page out yourself ;)

Here you have some screenshots:

Bugzilla

Anar here...


Due to a growing number of users I set up a PoD Bugzilla service, which will help to track bugs and new features.

From now on, all bug reports and support/feature requests must go via PoD Bugzilla. These stuff I will not accept via emails.
Anyway Feel Free to express yourself and show what a wonderful product PoD is ;) All the rest via Bugzilla.

direct link
or
on PoD Website: http://pod.gsi.de/support.html  ---> Bug Reports and Feature Requests.

Condor plug-in for PoD

Anar here...

As it was promised, I am very happy to release today an early beta of a Condor plug-in for PoD.

Before saying anything else I would like to show a screenshot-demo :)

In order to test the plug-in I have used a Condor cluster at CERN (special thanks to Wen Guan, who has helped me with that).

Now the demo. 

I will show PoD CLI, since the Condor plug-in doesn't support GUI yet.

First of all, same as always, PoD server needs to be started:

than I submitted 6 PoD jobs, using a Condor job manager (this is done via the plug-in). I submit only 6, because we were in hurry and there are only 2 worker nodes on which my account was enabled.

After a few seconds we can check whether there are already PROOF workers online and how many of them:

That's it! Our dynamic PROOF cluster is ready:

So, from now on PoD supports Condor resource management system as well. Which makes it 6.

Six supported RMS.

PoD supports: SSH, LSF, PBS (PBSPro/OpenPBS/Torque), Grid Engine (OGE/SGE), Condor, and gLite.

All most popular RMS are supported. If you need PROOF for your users, just get one of these RMSs and PoD will handle the rest ;)

No need to administrate, to setup and support any PROOF cluster. Each user can setup its own PROOF cluster on the fly using PoD just on demand.

The stable release of PoD, which will be shipped with the Condor plug-in is scheduled for the first week of January, 2011.

I think there is no reason to be hurry and to do the release before the upcoming Holidays ;) We rather can use this time to do more tests and improve User's manual...

But you shouldn't wait until the stable is out. Try the Condor plug-in right now and help us to improve the product. If you want to check this release out, please use the latest nightly build:

http://pod.gsi.de/releases/pod/nightly/

Contact us in case of any problem: http://pod.gsi.de/support.html

Downloads by Country

Anar here ... with a bit of propaganda right now... :)

I am very glad to see that PoD is being used or at least evaluated in many countries already!

The statistics below shows the downloads of PoD source tarball for the last 6 months. It is only 166 downloads by about 12 countries. 

"It's a very small number" - would you say.  But don't forget, that for such a young and specific product like PoD, which works on a very specific field - it is a great success. Every new user is a success, I think.

Since PoD supports a shared installation feature now, I expect the download counts to dramatically reduce. Due to the fact, that in most cases PoD is and will be downloaded only once for a site and shared between several users on that site.

BTW, I also expect more customers from the USA to join, as soon as a Condor plug-in for PoD is released. The plug-in now in development and expected in an early beta in several days.

Anyway the downloads count doesn't really matter to us. It is just interesting to see how our product slowly gets maturer.

What is really important to us is to create a product, which will help, simplify and populate a PROOF-like-analysis.

New beta of PoD 3.x series

A new PoD development (beta) release of upcoming PoD v3.1 was issued today...

Most of the work in this beta revision was concentrated on the shared installation feature, plug-ins compatibility, and tests on different RMSs (including SSH plug-in).

BTW, today I've also installed the first central PoD at my home institution - GSI. Now users at GSI can either use a centrally installed PoD by using a simple login script or install any PoD version privately in their desired locations.

If you want to check this release out, please use the latest nightly build:

http://pod.gsi.de/releases/pod/nightly/

PoD at CERN's lxplus

Anar here...

Today I've been testing the latest PoD nightly on different resource management systems and decided to test it at CERN as well. I have CERN's lxplus always in my list of thinks to test on, because of AFS - you know it has some restrictions, like pipes and so on. So, it is always worth to play with PoD at CERN.

I was quite surprised that PoD deployed for me 50 PROOF workers within 30 seconds! I was using standard LSF queue, named "1nh". Besides, the LSF looks quite busy these days... and you know why ;)

But think about it, these are just regular LSF machines at CERN and you get your private PROOF cluster just in two clicks (or two commands) in a matter of seconds - dynamically...  ;)

Usually I use PoD command line interface, but just for demonstration's sake I used its GUI:

I mean, look, I get in less than 30 seconds exactly the same number of workers as you can get on CAF (where a standard "static" PROOF cluster is installed). Taking into account, that CAF allows to run more that one worker per core, my PoD/PROOF cluster on LSF will be much faster. So these 50 on LSF (one worker per core), which PoD made for me will process my analysis faster and even so I can use CAF machines as storage elements ;)

Let's make another test: 100 workers. This can't be even compared to CAF, because regular users can't get more than 52 ;)

Starting PoD using its CLI and submitting 100 workers:

 

40 seconds later:

:D

Of course I understand, that the start up time on LSF or any other RMS depends on many factors, but even without pre-configuring RMS, I will be and I am able to get my PROOF cluster fast and on demand.

Additionally, I've advantages by using PoD, which are:

- it is my PROOF cluster (I don't have anyone who administrates it for me),

- LSF (or other RMS) takes care of resource sharing, distributing, logging, accounting and so on...,

- I am free to use this cluster at anytime I want with any ROOT version I want,

- I can't misuse the cluster, since RMS accounts it,

- etc.

In case you want to try the latest PoD at CERN, use the latest nightly or stable release.  For users' convenience I also provide some environment settings for lxplus, check the "PoD at CERN's AFS" section.

I would of course recommend trying the latest nightly, even so it still has some bugs, but you can help to evaluate it ;)

News on PoD Server

Anar here...

I want to continue to keep you informed on ongoing development of the upcoming stable release of PoD - v3.1.

This time I'm going to cover a new stuff in PoD server implementation, which were done recently.

First of all, the server now starts 3 times faster. A logic of start/stop/restart functions was changed and improved.
I have also improved a server processes discovery algorithm. This algorithms is responsible to find out which of server processes run( uid, pid, and tcp port it listens on). PoD server is a lot more smarter now when handling its processes, even third-party ones, like xproofd and its company. For example, the server implementation got a two-stage-shutdown machinery, which first tries to gracefully shutdown all its processes and only if a process ignores it, the server will send a non-ignorable kill signal to it.
The server now also smartly kills the children of xproofd (like proofserv). It kills not all of children, just by name and user id, but rather it kills only children processes produced by the xproofd, which were started by exactly this PoD server. This brings us one step further in PoD and AliEn integration as well, since it is now possible to start several PoD servers under the same user id - which is a requirement for AliEn Grid. On AliEn all processes run under the same user id.

What is also new in the latest releases is that there is a better support for shared installations. Many minor bugs were fixed. The xproofd configuration, which PoD creates on the fly, is now handled a bit differently. Namely, it stored in user's working directory and renewed every time user restart pod server. This helps to use several PoD versions with the same user's configuration.

If you want to check the new PoD server out, please use the latest nightly build:
http://pod.gsi.de/releases/pod/nightly/

PoD Shared Installations

Anar here...

Some time ago users start to ask for a, so called, central installation or a shared installation feature.

PoD now supports both types - Private and Shared installations.

• A Private Installation - it is when a user installs PoD for individual use to his/her local folder. Any Private Installation can be used by other users as well. It's just a matter of file privileges.
• A Shared Installation - it is when a site administrator installs PoD in some central location, so it can be shared by many users. It makes it easeir for users, since they don't need to install PoD by their own. In case of a shared Installation you need to execute one additional step. All the rest is the same as with Private Installations.

Be advised, that in both cases PoD acts identically and always provides private clusters, one for each user. In case of a shared installation, users share only binaries and configurations, but each user get's its own PoD instance and can't disturb other users. 

Each user can tune PoD by using its own PoD user defaults configuration in $HOME/.PoD/PoD.cfg.

This feature is still in beta and if you like to test it or just play around, than try the latest PoD nightly: http://pod.gsi.de/releases/pod/nightly/

I am going soon to release a new version of PoD, which will include this feature, but before the release I want to finish updating User's Manual to reflect all changes and to process some more tests.

If you can test the latest PoD nightly and give some feedback, it would be great help and can speed up the release ;)

ROOT's make to list all targets

Apparently there is one more remarkable commit (revision: 36664) to ROOT I want to mention today, since I was waiting for it.

The r36664 is actually implements a possibility to list targets via the make command (the feature, which cmake user can enjoy since long time already).

Now, ROOT users could issue the following command to find out all available targets of their ROOT source tree:

make help

Try it with your ROOT source tree. Very nice, isn't it?

I think it's a very user-friendly improvement in ROOT's build system.

ROOT supports out-of-source builds

Yesterday Fons has committed a very handy feature to ROOT, see revision #36659

Now ROOT supports out-of-source builds. 

What is an "out-of-source" build?

When you build a ROOT source, the build process generates files and they have to go somewhere. In in-source builds they were in your ROOT's source tree. The ROOT out-of-source build puts them in a completely separate directory, so that your source tree is unchanged.

I personally prefer only out-of-source builds. Even in PoD User's Manual I recommend this as a default installation procedure. This way builds are more elegant and clean. I can produce several different configurations in different build directories for a single source tree.

So, now ROOT also supports it and it is great offer to users! That was also not easy to implement in ROOT, see the number of changed files in the revision. Great job ROOT Team!

Author: rdm

Date: Mon Nov 15 11:07:02 2010

New Revision: 36659

URL: http://root.cern.ch/viewvc?rev=36659&root=root&view=rev

Log:

add support for out-of-source building of binaries. Normal build is unaffected.

To make an out of source build do, assuming the source is in ~/root:

  mkdir ~/root-x8664

  cd ~/root-x8664

  ~/root/configure

  make

This is convenient to build e.g. 32 and 64-bit version from one source, but

also needed for cross-compilation, where it is now possible to build in

a special directory only the compile time tools, like rootcint for the

host architecture while the binary is build for the remote architecture

(like iOS).

PoD 2.4

I am very glad to announce the release of PoD 2.4.

This version releases many important changes and among others, there is the main event, which is the first stable release of an Oracle Grid Engine (OGE) plug-in for PoD.

OGE is fully supported now in both PoD GUI and PoD CLI. OGE is a great RMS with a lot of sites using it. I think this PoD plug-in will help to convince many potential PoD customers to give our product a try ;)


The OGE plug-in has just joined a list of supported job manager's plug-ins.
PoD is now shipped with gLite, LSF, PBSPro/OpenPBS/Torque, SSH, and OGE plug-ins.

Highlights of this release

• Support of Oracle Grid Engine. PoD is now shipped with the OGE plug-in.
• A new schema of WNs' pre-compiled binaries handling.
• PoD now a bit more smarter and more efficient, when a shared home file system is present on UIs and WNs.
• The startup time of PoD jobs is notably improved.
• Several security issues were fixed.
• This version implements a better logging machinery for PoD workers.
• The Job Manager plug-in system is slightly improved. It now makes it much easier to extend with new plug-ins and to support old ones.
• Updated User's Manual.

PoD Home

Release Notes

PROOF Home

PoD supports OGE/SGE

Some more updates on the upcoming PoD v2.4 ;)

I finally finished the implementation for GUI of the Oracle Grid Engine (OGE) plug-in. Since a month OGE was supported by PoD, but only CLI. Now, starting from PoD v2.3.100.gac84 the OGE is fully supported in both, PoD GUI and PoD CLI.

As you can see, the interface is very simple and easy to use:

Using PoD GUI you can submit PoD jobs to an OGE cluster, check status of jobs, and kill jobs :)

PoD also supports an OGE option file. If you want to provide some additional Grid Engine options to your PoD jobs submitted to OGE cluster, to select some specific resource or something like that, than PoD gives you this possibility via an OGE option file. Just create a file $POD_LOCATION/etc/Job.oge.option and write valid OGE options in it. If the file exists, than PoD will automatically use it while submitting OGE jobs.

See qsub man page of OGE for more information on the option file (search for the "-@" option in the man page).

Pre-compiled binaries on worker nodes

Today I continue to give an overview of upcoming PoD 2.4 ;)

In this post I will cover a new behavior of PoD in terms of how it handles its workers now.

What we had before is that each PoD worker during the startup was checking environment of the machine it runs on and according to OS and CPU architecture tried to download a pre-compiled binaries of the needed PoD modules.

The binaries were downloaded from the central PoD repo. The binaries are needed, since PoD could run on different type of workers with different architectures. Talking about binaries, these are  only two modules needed so-far, but they are very important ones. Namely, pod-agent (core module of PoD) and pod-user-dsefaults (handles PoD settings). 

As it was pointed out by one of PoD users, this schema has many disadvantages and security issues. Indeed, having each worker (in each PoD session)  to download "some" binaries from the outside of a site or an institution is very inconvenient and insecure. This schema also makes it very difficult for users to provide their own binaries, if, for example, some type of architecture is missing from PoD central repo, but user want's to support it.

This is actually very old schema and was introduced at the time when PoD was called gLitePROOF (designed for gLite Grid MW).

Now, since PoD 2.3.84, I've changed this behavior and introduced a new schema. 

PoD workers don't download pre-compiled bins from the PoD central repository anymore. Now only PoD UI downloads them if they are missing from the UI. When a user starts PoD server ("pod-server start") a package with pre-compiled  binaries will be downloaded. That actually means, it will only be downloaded once to one machine and users could also provide own binaries, instead of using the PoD central repository. If a user will provide binaries, than PoD will not download anything from the central repository.

Additionally, PoD workers now try to find out whether a shared home file system is present and whether they can share the binaries with PoD UI/Server. If a shared FS is present, than workers will try to use UI's bins directly and if bins fail to start, workers will use pre-compiled bins from the worker package.  The new schema has resolved security issues and made PoD much more flexible. BTW, the start up time of the workers is also slightly reduced.

I'll keep you updated, so stay tuned!

PoD UI/Server on Mac OS X

Anar here...

As I promised, I am going to make several posts dedicated to new features of upcoming PoD v2.4.

Today's new feature or better to say possibility is "PoD UI/Server on Mac OS X".

Currently PoD UI and PoD Server are always on the same machine and can't be separated. A more preferable use case would be having UI on the laptop and a possibility to command to have PoD server somewhere else, maybe on a remote cluster, possibly close to PoD workers. This use case gives the maximum performance and flexibility for users.

We have a plan and we are working on disentangling of PoD UI and Server. So that user will be able to run PoD UI and PoD Server on different machines.

As a small step towards accomplishing the goal of disentangling, I have just finished porting PoD UI/Server to Mac OS X. So far only Command Line Interface and only the SSH plug-in are working and have been tested. But since main core modules have been just ported to support Mac, the other plug-ins will be easily ported as well.

You can already try running PoD on Mac using any build equal or higher than PoD-2.3.83. Check PoD nightly builds up: http://pod.gsi.de/releases/pod/nightly/

Download statistics

Anar here...

It has been a long time since my last post ;) You know, I was busy with PoD development and stuff.

There are many new features implemented in PoD and I am doing my best to get them all tested and available in the next release of PoD, which I am planning to make in a week or so. In my next posts on this blog I am going to highlight some features. So, please, subscribe for updates or follow me on this blog (on the right you see the followers bar)...

Anyhow this post is not exactly about PoD, but about a PoD web site update. There are some cosmetic changes, and among others I have implemented an automatic publishing of PoD download statistics. It is time, I think, to start to get some statistics (it's better to be early than late). You can check it out here.

So far we have around 130 Downloads since summer 2010. I think it is very good number, since PoD is a specific product for a specific field. 

As you may know or have noticed, pod.gsi.de is a docbook based website. Since long time I am in love with docbook (PoD User's manual http://pod.gsi.de/documentation.html is made in docbook) and even the web site of PoD is also a docbook based. I am using docbook Website schemas and very happy with it. For a docbook Website concept I will dedicate a separate post some time later.

Back to the subject. The stats are generated automatically every night. Since I don't have a direct access to our Web Server and its logs, it was quite tricky to implement.

The whole process is completely automated.

I am getting the log via a script. The stats are parsed by a simple C++ program, which outputs a docbook page. The page is than committed to PoDWebsite Git repo. As soon as the Git repo updated with a new commit, buildbot will automatically start a build of PoDWebsite and if parsing and conversion to html are ok, the site will be published. WoW :) So, simple is that. I love automations!

PoD Download Statistics

PoD 2.2

Anar here...

After several months of development :) I am pleased to announce the release of PoD 2.2!

Highlights of this release

• A major highlight of this release is the first implementation of the PoD with SSH plug-in (see User's Manual, Chapter 11. SSH plug-in).
• The PoD distribution package is even more smaller now, it is only 3 MB :)
• A heavily revised build system, should help users to build and to install PoD from the source code conveniently.
• PoD doesn't use xrootd anymore. Instead xproofd is used. Starting with ROOT version 5.24/00 an executable named xproofd is also available under $ROOTSYS/bin. The xproofd executable is built out of the same main program used to build the xrootd executable. However, it loads by default the XrdProofdProtocol instead of the data-serving protocol XrdXrootdProtocol. The purpose of this daemon is to simplify the configuration when data-serving is not needed or it is handled differently.
• Implemented an automatic execution of a custom (user specified) environment script on workers before PoD processes start. (see User's Manual, section 5.2.)
• Revised PoD workers package and algorithms helped to reduce requirements and decrease start up time.

PoD Home

Release Notes

PROOF Home

PoD 2.1.4

Anar here...

It is always a pleasure to announce a new release of PoD. Yesterday night I have released PoD 2.1.4.

Some Highlights of this release

• A new build system has been introduced. The source code of sub-modules of PoD is now shipped along with the PoD distribution. Now PoD is much more easer to install and to maintain. Please see a how-to-install page in user's manual.
• Now PoD is shipped with a new plug-in. Starting from this release we implemented and will support a PBS/Torque plug-in. This opens a possibility to our users to submit PoD jobs to PBS resource management system. We support PBS (torque) and PBSPro, with both shared and not shared home file systems.
• PoD got a new command line interface (CLI). Using PoD's CLI users will be able to start, stop PoD server, retrieve its status, submit PoD jobs to a different resource management systems, check status and a number of available PoD/PROOF workers and more... Please see PoD user's manual for more details.
• PoD gLite plug-in and glite-api-wrapper library is now supporting gLite UI 3.2.

PoD Home

Release Notes

PROOF Home

PROOF progress dialog window

Yesterday Gerri has implemented a useful patch in PROOF. From the now on it is possible to switch the PROOF progress dialog off.

Add method TProof::SetProgressDialog(Bool_t on = kTRUE) to disable the progress dialog
window while preserving graphics from the final actions in Terminate. This may be useful
when X over network is slow. To disable the dialog the call

proof->SetProgressDialog(0)

has to be placed before Process(...).

Many users, who does PROOF remotely will find this as an extremely handy feature. ;)

The patch was committed as a revision 33562.

TORQUE/PBS plug-in

Anar here...

I've just finished the first stable version of a TORQUE/PBS plug-in for PoD.

One more plug-in which will be shipped by default with PoD distributions, additionally to gLite and LSF.

Worth to mention, that the LSF plug-in is already in production for ALICE users at GSI.

From now on PoD user can also make use of PBS facilities and setup private PROOF clusters on the fly using PoD.

The plug-in will be released with PoD v2.1.4 and currently it is in a beta.

I am looking for test users, who has access to PBS clusters and want help and try/test PoD. Any feedback is appreciated. ;)

If you want to test PoD (with pbs), just drop me an email on A.Manafov@gsi.de.

You can also just download the latest nightly and give it a try. More information here.

BTW, v2.1.4 will also come with a new command line interface. So now finally we have both, GUI and CLI ;)

The next plug-in is already in development - it is SSH plug-in. With help of it, any machine on which you have ssh access can become your PROOF worker on demand. ;)

See you soon...

PoD 2.1.3

I am very glad to announce the release of PoD v2.1.3.

Some Highlights of this release

• This version makes it possible to setup a native PROOF client/server connection in PoD environment, rather than to have it packet forwarded. if your workers are behind a firewall than PoD will automatically use its packet forwarding algorithms to maintain the PROOF traffic between server and workers. But if a direct connection is possible, than PoD will setup a native PROOF connection for such a workers. That means, by using PoD you can now setup a dynamic PROOF cluster and get a FULL functionality of PROOF on the native PROOF speed. For example you could now retrieve PROOF logs from a ROOT session or use PROOF's new submerger functionality to speed up the merging phase + all other nice and useful PROOF features.

Minimally required version of ROOT is 5.25.04 or higher. Otherwise PoD will not be able to detect idle status when a native PROOF connection is used and will consider that your session is always idle and will shutdown it after a defined timeout.

PoD Home

Release Notes

PROOF Home