May 22, 2003VOLDEMORT-S. Timm--LSCCW 1 VOLDEMORT VOLatile Distribution of Electronic Media Over Rsync Transport May 22, 2003 LSCCW Steven Timm

May 22, 2003 VOLDEMORT-S. Timm--LSCCW1

VOLDEMORT

VOLatile Distribution of Electronic Media Over Rsync Transport

May 22, 2003

LSCCW

Steven Timm

May 22, 2003VOLDEMORT-S. Timm--LSCCW2

Introduction:

Disclaimer--Any resemblance to characters in Harry Potter books of J.K. Rowling is pure coincidence.

Rsync is open-source package which allows to keep directories on remote machines synchronized with each other

Common method at many installations of distributing volatile local files on machines that are already installed.

Needs a set of supporting scripts to make it a useful tool.


VOLDEMORT overview

Currently deployed on over 700 machines at Fermilab Works on RH Linux 6, 7, 9, Advanced Server, Sun, SGI Written in shell scripts and perl Two major uses:

– Keeping production computing farms installations up to date without reinstalling

– Partitioning of US/CMS computing dynamically


Major Goals of Voldemort

Replace NIS with system to put passwd files locally on each node

Have a unified structure to push new files out to existing nodes and install them on new nodes

Have a single place where each volatile file is modified. Keep current capability to have special files for a single

farm, subcluster, or node. Production Farms plus US/CMS have at least 13

different hardware configurations


Components of Voldemort 0.6

Voldemort-push, includes rsync_push binary and scripts to clone slave servers. Installed on all servers.

Voldemort, installed on all clients, includes pullrsync and a number of auxiliary scripts that are called by pusher and puller.

Tree file structure, set up in $VOLDEMORT_DIR/clusters

Databases to describe the file structure Available as RPM or in Fermi ups/upd format.


Features

Pullrsync included in 7.3 Farms workgroup post-install of Fermi Linux

Scripts tested on all flavors of Linux Includes option to sync out changes in Fermi

Linux comps file But—not tied to Fermi Linux..can and does

work with other installation systems such as Rocks or System Imager


Why replace NIS?

NIS was stable on CDF farms-169 nodes, no timeouts for months—BUT

We had to have at least 64 NIS slave servers to accomplish this

Pushing to all those slave servers is a network load in itself

Yppush doesn’t gracefully handle when a node is down Initial configuration of ypinit –s is error prone and can’t

be automated during install.


Why replace NIS cont’d

Malformed map on one slave server can mess up several nodes

NIS is small amount of network traffic but is very sensitive to bigger network flows and is disrupted by them.

On our farms, we don’t store any real passwords in NIS, accounts change rarely. Ideal situation to distribute files


Installer vs on-line changes

Whenever we made a change to the farm, we had to change in two places…on the nodes and in the installer.

Often this has been forgotten Method of making installer changes is not

straightforward Need to make a system where any file that goes on the

system is only changed in ONE place.


Down nodes problem:

Right now, if we put extra files on the system, we have to go back and fix nodes that were down later, manually.

Need a system that will remember which nodes were down, and keep retrying until it gets them all.


Design goals of Voldemort

Do not put any node-specific info into the Fermi Linux workgroup—we don’t want our whole structure available to world via anonymous FTP. (or our account names and groups or nfs servers).

Replace /export/linux/Workgroups/Farms/nodes with a new structure that is used both by online activities and the installer.

Keep our current capacity to have node specific, farm-specific, and subcluster-specific files


$VOLDEMORT_DIR/clusters/common/db/nodes.conf

Nodes.conf—database of nodes. Read by both rsync_push and pullrsync fncdf75:cdffarm1:Linux+2.4.18:i-acd:38400::N:2518 Reader

Fields:– Node name APIC used in install– Cluster name Node specific– Flavor Subclusters– Disk arrangement – Baud Rate


$VOLDEMORT_DIR/clusters/common/db/files.conf

Files.conf: Not fully populated yet. Three fields:

– Full pathname to the file– (example: fnsfo/files/Linux+2.4.18/etc/passwd)– Files it depends on– common/templates/Linux+2.4.18/etc/passwd– fnsfo/templates/NULL/yppasswd– Command used to make it– (cat the above two files together).


$VOLDEMORT_DIR/clusters/fnpce/

Prescripts—scripts that have to be executed before a rpm or file can be installed

RPMS Files—single files that are pushed out to

worker nodes Scripts—usually run only by the installer Tarballs—Mainly for pushing out /local/ups

directory to worker nodes


$VOLDEMORT_DIR/clusters/fnpce/files

Under each category, space for more than one flavor. Right now:

Linux+2.4.18 (731) Linux+2.4 (711) Linux+2.2 (612) IRIX+6.5 Can also define arbitrary flavor “foo” as long as

database matches.


$VOLDEMORT_DIR/clusters/fnpce/files/Linux+2.4.18

Each subdirectory of files directory gets pushed out independently—governed by .pushdir files

Four subdirectories (typ) /etc, /root, /usr/local, /var/adm Three types of files:

– Passwd, group, netgroup, auto.*, .k5login– Non-standard config files for RPMS in redhat base– Hardware-specific or farm-specific files


$VOLDEMORT_DIR/fnpce/tarballs/Linux+2.4.18

Currently only one tarball Structure same as files (.pushdir governs) /local/ups/localups.tar Tarball should be created to be untarred in the

directory it’s pushed into. Had to add this option because pushing a

ups/upd tree of 19K files (180 Mb) was too slow.


$VOLDEMORT_DIR/clusters/fnpce/RPMS/Linux+2.4.18

RPMS that go here are either farm-specific or hardware-specific.

Anything for whole farm should go into farms workgroup.


$VOLDEMORT_DIR/clusters/fnpce/[pre]scripts/Linux+2.4.18

Scripts and prescripts are mainly executed during the install

Installer calls /sbin/pullrsync –I which forces running of all scripts

Scripts should be smart enough to detect if the action has already been done


Subclusters

Subclusters can exist in any of five categories, files, tarballs, RPMS, scripts, prescripts

Subcluster membership determined by the database Convention: All hardware specific files (ethernet,

lm_sensors) go into a subcluster named after the motherboard type

Node can be in more than one subcluster For files and tarballs, a .pushdir at the top level.


Node-specific files

Can also have files specific to a single node Enabled by having field in database be “Y”

instead of the default “N”


Rsync_push

Rsync_push reads through the database and pushes to every node that matches the command-line options it was called with.

*IMPORTANT* Default is to push to everything! There is an are-you-sure option now that warns you what you are pushing.

Rsync_push –r allows you to retry nodes that didn’t push successfully the first time.

Default transport is kerberized rsh. Others can be used as well.

To push to a node, host principal of the server must be in /root/.k5login of the client node.


Rsync_push options 1

-c Push for a given cluster -f Push for a given flavor -b Push for a list of nodes -B Push for a range of nodes -l <filename> push for all the nodes in <filename> If more than one is specified, we take the AND Example: rsync_push –c cdffarm –f Linux+2.4.18 –B fncdf “171

173-176”



-R don’t push the RPMS -F don’t push the files -S don’t push the scripts -P don’t push the prescripts -T don’t push the tarballs -L don’t push the Linux /etc/workgroup Default is to push everything



-w specify the workgroup you are pushing (default is Farms) -e use an alternative rsh command besides /usr/krb5/bin/rsh -q quiet—minimum or no output -v verbose—the more v’s, the more verbose -i Install mode—run new scripts and prescripts when they

are pushed out -I Install mode—run all scripts and prescripts when they are

pushed out -C Clear out the RPMS, scripts, and prescripts directory on

the worker nodes.


pullrsync

Determines node ID and type either from local config file or from database read

Runs only if machine wasn’t shut down clean, and during the install

Options –h help –H <hostname> –c <cluster> -f <flavor> –M <module> -t <target server> -R –S –P –F –T –L –w –I –i –q –v (as in rsync_push)


Future plans

Version v0_6 current, no known bugs right now. Next version needs better and faster database Also need ability to automatically distribute the push

across slave servers Big task, integrating more closely with ROCKS and RH

9.0. http://www-oss.fnal.gov/scs/public/farms/doc/

voldemort.html

Documents

May 22, 2003VOLDEMORT-S. Timm--LSCCW 1 VOLDEMORT VOLatile Distribution of Electronic Media Over Rsync Transport May 22, 2003 LSCCW Steven Timm