Herding Cats

Managing a mobile Unix platform in the enterprise

Who are we?

• IT department of Cisco TAC

• Sun Solaris servers/desktops, HA environment

• 5600 active accounts, 1200 workstations, 5 major sites globally

• 18 people

• Mail us: maartenwout

@cisco.com@cisco.com

This presentation

• Desktop replacement plan

• Desktop analysis and source code available

• http://radmind.org/contrib/LISA05

1. Choose Platform & Tools

2. ???

3. Profit !!!

Choose platform

• Find out what’s important to you, attach weights

• For every platform, give a score for each attribute

• sum(), total()

• Important in our case: Unix networking and engineering tools Only one platform Supportability Laptop experience: hotplugging, battery life, … Security

Herding Cats

• Managing laptops is like herding cats

• Always know where your cats are They’re off the network before you know it Perform hourly client-side update checks with

fast tools Use asset tracking to know IP address and

current system state

This is a personal laptop!

Herding Cats

• Keep your cats healthy & happy Feed them regular software updates Stroke their ego by giving them control over

the update process File system checking makes sure there are no

invading organisms With backups, your users can have nine lives

too!

Tools and tricks

These apply not just to OS X !

How

• Manage file system: rpm, tripwire?

• Asset database: ethers?

• Uniquely identify systems: SSL/PGP cert?

• Naming service: LDAP?

• Secure data: FileVault?

• Business continuity: Backups?

Managing the file system

• Our rules: Keep full control over the client systems

• No root access for the user One image to rule them all

• Identical problems/solutions• Fast system replacement• No need to back up system image

Enforce known-good state on client• Install what we want there• Remove what we don’t want there• Do it without bothering the sysadmins

Which toolkit?

• Apple’s OS X package system is lacking No consistent uninstall No dependency tracking

• Most package tools can’t manage OS X They assume they’re alone on the system Resource forks / extended attributes File system is case insensitive

So… radmind

• We chose RAdminD

• Large installed base of radmind on OS X

• http://radmind.org

• Thanks, UMich!

Why … radmind?

• Can repair broken systems without re-imaging Bit flips, rootkits, user with sudo rights⇒ repair automatically!

Can handle OS upgrades

• Server can run on any platform re-use existing infrastructure

• KISS, easily extensible, Unix philosophy

How radmind works

System state

• Command file determines the overloads that make up the system

• Positive transcripts add files, negative ignore files

p base-system-10.3.4.Tn os-negative.T

p omnigraffle.T

d /Applications/OmniGraffle.app 0755 0 80f /Applications/OmniGraffle.app/.DS_Store 0755 0 0 1096985459 6148 WCf1IuqHcXrNGZDUiX+Buucs83Q=d /Applications/OmniGraffle.app/Contents 0755 0 80…

d /Users 0755 0 0d /private/tmp 1777 0 0

os-negative.Tcommand.K

omnigraffle.T

User experience

• Updates run while user is working

• User gets prompted before downloading

Radmind conclusion

• 1 mechanism

• Fixes all problems

• Introduces new ones Solved by client side trigger scripts

• Normally used in lab setting at reboot time

• Normally not used while user is working But it works really well for us

Our radmind setup

• Global DNS service points to nearest, available radmind server Produces a scalable, highly available setup SSL certs should contain DNS alias

• 3 ports for 3 trees: stable, testing, staging Reduces operator error Shared file tree for disk space optimization

• Master host to maintain trees Push changes downstream using rsync Script checks correctness and dependencies before

pushing (dist-it)

Distributed setup

Multi-release setup

• The file storage is shared between releases

• Stable, testing and staging are the source dirs for 3 radmind daemons on 3 ports

• Symlinks allow fast switching

AssetInterface

• Asset tracking software tracks: Owner of machine (set on first login) IP addresses Logins Etc

• Saves data locally until machine can reach the server

• Precompiled SQL• Once system leaves us, we don’t see it again

until it breaks ⇒ Margaritas @ the beach!

RegServ

• Radmind differentiates systems based on IP or certificate name We encode system info in the certificate name e.g. ppc7450.PowerBook5,4.W842219KQW3.mthibaut Wildcards allow matching any of the parts

• RegServ uses generic client certificate installed on base image to securely provide a machine specific certificate Secure as long as client base image is secure

Based on radmind code

How - LDAP

• We used our existing LDAP servers• OS X can cache the credentials• Lots of policy enforcement possible

provide default or forced custom settings for any Cocoa application, lots more

• Sounds simple, but we had quite a bit of trouble “MCX” keywords are undocumented Trial and error Final solution: use same LDAP layout as OS X server in a

subtree, allows using Apple’s GUI tools

• Overall it works, could be better

How - FileVault

• Secures user data using AES-128 encryption Data is stored in a resizing disk image Master certificate allows password recovery by admin

• Deemed mandatory in our organization We had to hack things a bit A script runs at login time and verifies existence Creates FileVault if not there Works reasonably well, fast (2-3 MiB/s)

Backups

• KISS• We already back

up our home directories on Solaris

• So we wrote a GUI for rsync over SSH

• Works fine, even though resource forks not copied

Wrapping up

Mac OS X vs Solaris

In our experience: Better application availability/installation Little need to manually compile tools (e.g. fink) GUI/usability is not an afterthought

• User mountable filesystems• User installable programs

It Just Works• We can concentrate on our users

Giving back

• All our programs, utilities and scripts for this project have been published

• http://radmind.org/contrib/LISA05/

• We apologize for the inconvenience Hard coded paths No installation scripts Little documentation Choose your license, but don’t call Cisco TAC about

these!

131313© 2001, Cisco Systems, Inc. All rights reserved.

Questions?

Backup Slides

Because we didn’t tell you everything :)

Desktop replacement

Why

• Laptops are becoming standard Telecommuting Lab work Customer visits One system for everything One computer per employee

• Our users need/want Unix

Why

• Needed a supportable platform

• Experiments with Linux failed

• OS X was the only viable Unix for portable systems It Just Works

Why … not Linux?

No vendor hardware support No drivers for recent hardware Missing UI for common laptop tasks

• Networking setup• Mounting network file systems• Hotplugging disks, audio, video, …

Why … choose OS X?

• Unix! X-Windows capability

• Well designed, e.g. fixing long-time Unix issues (launchd, directory services, …)

• Everything is integrated Applescript User preference system Server side preference setting overrides Naming services Etc (very extensive)

Why … choose OS X?

• Security Admin access hardly needed

• User installs, network setup, USB drives, audio devices, …

FileVault Secure screen lock

• Good software support from Apple• Reasonable software availability for

platform

Why … choose Apple?

• One vendor for hardware and software

• Perfect platform support for OS X

• Quality hardware

• Supportability: Firewire target mode for disk rescue One disk image for all system types

• Except for brand-new systems

Why … not OS X?

• Politics (MS centric world)

• Apple not geared towards enterprise HW release schedule forces use of buggy OS

releases Undocumentation & secrecy No on-site or phone support as with Sun Support costs $$ per admin rather than $$ per

machine

Why … not OS X?

• LDAP caching System always contacts LDAP server when

cached data expires and name lookup occurs Where can we change expiry timers? Can’t this be done asynchronously?

• Have to learn “the Apple Way” Sometimes there is no other way

• Using X-Windows is a pain

FileVault architecture

• Sparse disk image

• Encrypted by long key stored with disk image

• Key is encrypted with 2 keys, each can unlock it: User password Master FileVault certificate

• Master FileVault certificate is encrypted with an admin password

What is radmind?

• Command files + overloads Describe system state, files + checksums

• ktcheck Download wanted system state

• fsdiff Compare file system with wanted state

• lapply Repair changes found by fsdiff

• All done using encryption and authentication

Our overload layout