Upload
bethany-king
View
26
Download
0
Embed Size (px)
DESCRIPTION
R-GMA. Producers and Archiver Paul Mealor. gsiwuftp. Overview. edg-netmon-archiver. edg-gridftplog2rgma. Log. Archiver API. P. Producer API. MySQL DB. Here be servlets. edg-netmon-rgma-info. P. Producer API. P. A. C. DBP. edg-pinger-timeping. edg-netmon2rgma. Producer API. - PowerPoint PPT Presentation
Citation preview
R-GMA
Producers and Archiver
Paul Mealor
edg-pinger-timeping
edg-netmon-rgma-info
Producer API
edg-gridftplog2rgma
Producer APILog Here be servlets
Overview
AC DBP
P
P
P
edg-netmon2rgma
Producer API
edg-netmon-archiver
Archiver API
MySQLDB
gsiwuftp
FIFO
edg-netmon2rgma
• Maintains producers for the tools– Tools’ scripts are rerun for each measurement batch– Program running APIs must be persistent
• Files– /opt/edg/sbin/edg-netmon2rgma.pl– /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo– /etc/rc.d/init.d/edg-netmon2rgmad
• Configuration– RGMA client configuration
Tool/netmon2rgma interaction
•addmetric pinger rtt 10 NMIdSource=pc18.hep.ucl.ac.uk tool=pinger
PingER Normallog
edg-netmon2rgma
•CircularBufferProducer_new(“NetworkRTT”,“WHERE NMIdSource=‘pc18.hep.ucl.ac.uk’ AND tool=‘pinger’”);
•CircularBufferProducer_setRemoteBufferSize($producer,10*2);
Web interface
If producer doesn’t exists
If bufferSize < count * 4 / 3
Tool/netmon2rgma interaction
•measurement pinger rtt pc18.hep.ucl.ac.uk edgnm.gridpp.rl.ac.uk pinger 100 57
PingER Normallog
edg-netmon2rgma
•CircularBufferProducer_insert($producer,”INSERT INTO NetworkRTT(NMIdSource, NMIdDestination, tool, packetSize, value)VALUES(pc18.hep.ucl.ac.uk, edgnm.gridpp.rl.ac.uk, pinger, 100, 57)”);
Web interface
edg-netmon2rgma
• Possibly should rewrite in Java– Problem in C (Perl) API results in memory
leak and heavy performance hit– Current solution is to restart the daemon with
a cron job• /etc/cron.d/edg-netmon2rgma-restart
– But java lacks the text-handling of Perl– Would have to have java on the NM
Tools modifications
• Generally 2 new functions• initialise_rgma
– Writes initialisation directives to the FIFO– i.e. fixed columns, metric names, expected number of
entries• Netmon2rgma creates producers, changes buffer sizes when
necessary
• produce_it– Writes the results of the measurements
• No extra configuration– Assumes /opt/edg/var/edg-netmon2rgma/edg-
netmon2rgma-fifo
edg-netmon-rgma-info
• Publishes NetworkSE, NetworkCE tables– Republishes every 12 hours (i.e. keepalive)
• Files– /opt/edg/sbin/edg-netmon-rgma-info.pl– /opt/edg/etc/edg-netmon-rgma-info.conf
• Local NM, associated SEs and CEs
– /etc/rc.d/init.d/edg-netmon-rgma-infod
• RGMA client configuration
edg-ftlog2rgma
• Publishes file-transfer logs (i.e. GridFTP)
• Files– /opt/edg/sbin/edg-gridftplog2rgma.pl– /etc/rc.d/init.d/edg-gridftplog2rgmad
• Reads from GridFTP log– /var/log/gsiwuftpd.log– Publishes new entries when they are added
• RGMA client configuration
edg-netmon-archiver
• Program– /opt/edg/share/edg-netmon-archiver.jar– /opt/edg/sbin/edg-netmon-archiver
• runs the Java code
– /opt/edg/bin/edg-netmon-archiver-setup• Configures MySQL database; requires mysql root password• What to do in LCFG?
– /opt/edg/etc/edg-netmon-archiver.conf• List of tables to archiver plus predicates for each
– /etc/rc.d/init.d/edg-netmon-archiverd– Also RGMA client configuration
edg-netmon-archiver
• Ironically this might be the best package to write in C or Perl– Archiver API is all but unaffected by C API problem– But might run on R-GMA servlet machine
• Interaction with edg-netmon-info– Archives every row that edg-netmon-info produces– Perhaps need to automatically remove duplicates (via
mysql)
Package progress
• Tools, edg-netmon2rgma, edg-netmon-rgma-info– 99%– Latest tweaks still need to be uploaded
• edg-ftlog2rgma– 75%– rc script, autobuild process
• edg-netmon-archiver– 95%– need to upload, tag, check build process (ant)
Testing
• Tools on pc18.hep.ucl.ac.uk• edg-netmon2rgmad on pc18.hep.ucl.ac.uk• “User servlets” on pc23.hep.ucl.ac.uk• Registry and Schema servlets on
pc23.hep.ucl.ac.uk– Something always goes wrong when I try to connect
to the WP3 Registry at infocat.gridpp.rl.ac.uk• >1200 RTT/loss measurements archived by
Tuesday• >130 TCP throughput measurements to
localhost :)
GridFTP logging in MDS
Paul Mealor
Status
• Information provider– Will be in EDG 2 on SE
• Publishes the most recent information for each host pair
• MDS being phased out in EDG 2– GIIS structure replaced by R-GMA– Will be all R-GMA later
Details
• Publishes achieved throughput– Filesize, time etc as parameters
• Publishes most recent transfer between each host pair
• Specifies source and host pair, plus, if known, source and host NM– NM names taken from config file as with other netmon
info providers
• Uses same schema as other netmon– But doesn’t use the full tree structure
Schemadn: NMMeasureId=$source/$dest,$opt{rootdn}objectClass: NetworkMeasurementNMMeasureId: $source/$destSourceHost: $sourceSourceNE: $sourceNEDestHost: $destDestNE: $destNEMetricName: gridftpthroughputMetricValue: $throughputMetricUnit: bit/sMonitorTool: GridFTPMetricTime: $timestampParameter: time: $etimeParameter: size: $sizeParameter: buffersize: $bufferParameter: streams: $streamsParameter: stripes: $stripescreateTimestamp: $timestampmodifyTimestamp: $timestamp
END