AMACAA Web Application for an AFS Cell monitoring

A. Rocchi, G. Bracco, S. Migliori, S. Podda, A. Santoro, C. Sciò

OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Summary Motivations Operating scenery

ENEA-GRID infrastructure

AMACA Software core modules

Main features

Conclusions

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Motivations

OpenAFS monitoring in a complex environment can be tricky

No visual tools for analysis

It is impossible to track state modifications of core components through time.

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Why not a NAGIOS plugin? Some OpenAFS control components have already

been written to be integrated in NAGIOS.

We choose not to pursue this strategy:

Nagios is designed to be a “quasi-realtime” general purpose monitor

AMACA, instead, aims to be a “periodic photographer” of OpenAFS, with sophisticated features of analysis and comparison among snapshots taken in different moments.

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Operating scenery ENEA (Italian National Agency for New Technologies,

Energy and the Sustainable Economic Development)‏

12 research sites, 6 computer centers

Computational resources are integrated in a global grid infrastructure (ENEA-GRID)‏

AFS is the lower layer, providing information distribution across multiple sites

One cell (enea.it), multiple served locations

See G. Bracco presentation (AFS in a grid context)‏

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AFS Memorize And CheckApplication

• Initial development in collaborationwith CASPUR

Two cooperating modules

• CRAWLER ↦ scans state ofOpenAFS core components, using the N. Gruener’s Perl API

• EXPLORER ↦ Provides a convenient visualizationinterface to the crawling results

Access to application modules isvalidated by Unified Access APIs

AMACA - Architecture

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA – Database splitting The DB backend is splitted in two

• A production portion is used to store results of nightly indexing. Users can’t modify its content.

• A scratch portion, filled up by custom invocations of the crawler. Only users belonging to specific (customizable) pts groups

can write on it

For example, convenient for admins who want to see the effect produced by management tasks!

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Crawler stores unstructured data in a MySQL backend

• fine-grained data retrieval operations

Differentiation among subsequent crawler invocations is done by identifying every resultset with a unique ID (snapshot)‏

Explorer implements business logic of data aggregation and display

• Historical analysis on file system variations

• Parameterized search queries

AMACA - Details

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Why?

• OpenAFS + K5 integration: strong authentication over the entire GRID realm.

One account, many services: It would be good to keep only one set of credentials to access anywhere.

In particular, UAA is meant to be used to protect web resources (i.e.: webapps and static HTML pages)

• However, we need to be sure that only selected users will use some applications or software features

allowed users don't have to be system administrators!

Unified Access API (1)‏

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

How?

• Server-side Inter Process Communication with PAG shell

We need to be sure that every race condition on tokenacquisitions is avoided

• Requests for login and PTS membership are sent, and results are compared with user-defined entries.

• If access is granted, an encrypted cookie is generatedin order to store information

AMACA – UAA (2)‏

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA – Explorer The Crawler can be ran in a stand-alone way (by command line invocation)‏

Explorer provides a convenient way to dig into indexed results.

• Web application, PHP/AJAX

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA – GUI explained The explorer module exposes 4 groups of

information and controls:

• Global system state (sync site, volumes and partitions occupation stats, max Volume ID, …)‏

• Customization components

Admins can restrict the whole view to a specific cell subset (i.e.: to a single site)‏

• Alarms Highlight of (more or less) critical situations

• Navigation menu

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA – Main page

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA - Alarms Alarms are a way to highlight potentially critical

situations.• Locked and unattacheable volumes• Servers send/receive ratio threshold (via rxdebug)‏• Overbooked partitions (i.e.: sum of volumes reserved

quotas is greater than actual partition size)‏• Volumes mapped only in Volume servers OR in file

servers• Volumes which don’t respect the numbering schema

(RO=RW+1, BK=RW+2)‏• …

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA – Alarms (2) Alarms (as everything else in AMACA) are

discrete-time evaluated It could happen that criticality is really low

E.g.: a server, offline at the moment when snapshot has been taken, can become up and running at the very next moment

In any case, admins have to verify if there is some issue waiting to be solved.

For that reason, AMACA allows “power users” to run crawling directly from the Explorer

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Using AMACA – Login form

Using AMACA – Online crawling

Press the camel-shaped icon, wait

for the

crawler to finish, and see if there’s

some alarm still alive

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Using AMACA – Online crawlingCrawler invocation:

an asyncronouspopup appears. Waitfor the run to finish.

After a while (time dependson network conditions and cell size), results are shownin form of text output.Close the window and explore results.

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

AMACA – Analysis features The upper part of the navigation menu exposes 4 voices

• “List Differences” is a true diff between 2 snapshots Given two referring instants, AMACA searches what is changed

between them.

• “Volume or server search” Finds volume or server in every snapshot, using wildcards and size constraints

• “Time search” For a given entity, shows how its state has changed through time.

• “Volume history” Volume details (similar to vos ex) with respect to a single snapshot.

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Conclusions AMACA has been in production within our

infrastructure for 2.5 years.

• Downloadable tarball (source code + documentation) will be soon available.

Beta-tester/contributor anyone? :-)‏

• It’d be interesting to run AMACA on bigger cells than enea.it

• Performance improvement and features extension

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome

Thank you!!!

Any Questions?

A. Rocchi et al. - OpenAFS European Meeting – September 2009 – Uniroma3, Rome