Lots of Stuff

Lots of Stuff

Gordon D. BrownScience & Technology Facilities Council

WLCG Service Reliability WorkshopCERN 26-30 November 2007

WLCG Service Reliability Workshop 2

• Database Security• Passwords• Database Security Checklist• Host Housekeeping• Grid Control• Backups Overview

• Interactive• Any questions

Overview


Database Security


Database Security

• Authenticating DBA logins on host

– Operating system (o/s) authentication

– A password file

• For nonsecure remote connections

– Must be authenticated by a password file

• Connecting to database as a privileged user over a local connection or a secure remote connection in two ways

– If have password file and have been granted the SYSDBA or SYSOPER system privilege

– If no password file, then operating system authentication, username of the database administrator in a special group

• OSDBA:. Users in that group are granted SYSDBA privileges• OSOPER: Users in that group are granted SYSOPER privileges


Database Security

• Operating System Authentication– OSDBA and OSOPER– The groups are created and assigned specific names as

part of the database installation process.

– If you are not a member of either of these operating system groups and you attempt to connect as SYSDBA or SYSOPER, the CONNECT command fails


Database Security

• Using Password File Authentication– To enable authentication of an administrative user using

password file authentication you must do the following:– If not already created, create the password file using the

ORAPWD utility:• ORAPWD FILE=filename PASSWORD=password ENTRIES=max_users

• For RAC the environment variable for each instance should point to the same password file

• Protect password file!


Database Security

• Operating system authentication takes precedence over password file authentication.

• Specifically, if you are a member of the OSDBA or OSOPER group for the operating system, and you connect as SYSDBA or SYSOPER, you will be connected with associated administrative privileges regardless of the username/password that you specify. i.e.

sqlplus / as sysdba

• If you are not in the OSDBA or OSOPER groups, and you are not in the password file, then attempting to connect as SYSDBA or as SYSOPER fails.


Database Security

• Setting REMOTE_LOGIN_PASSWORDFILE– With password file, you must set the initialization parameter

REMOTE_LOGIN_PASSWORDFILE. Values:• NONE: Database behaves as if password file does not exist.

– No privileged connections are allowed over nonsecure connections.

• EXCLUSIVE: (default) Used with only one instance of one database.– Only an EXCLUSIVE file can be modified– Enables you to add, modify, and delete users. It also enables you to

change the SYS password with the ALTER USER command.

• SHARED: Used by multiple databases running on the same server, or multiple instances of a (RAC) database.

– Password file cannot be modified.– This option is useful if you are administering multiple databases or a RAC

database.


• Login overview

Database Security


• Viewing Password File Members– V$PWFILE_USERS view to see the users who have been

granted SYSDBA or SYSOPER• USERNAME: User that is recognized by the password file • SYSDBA: If TRUE, then the user can log on with SYSDBA • SYSOPER: if TRUE, then the user can log on with SYSOPER

Database Security


• Ensure that internal networks are inside a firewall

• Oracle Advanced Security– Add on– Enables data encryption and integrity checking,

enhanced authentication, single sign-on, and support for DCE

– Combines network encryption, database encryption and strong authentication together to help customers address privacy and compliance requirements:

Database Security


• Transparent Data Encryption– Protection of stored data by transparently encrypting data

(using 3DES or AES with up to 256 bits) at the column or tablespace level.

• Network encryption and data integrity– Supports:

• RC4 (40, 56, 128, and 256 bits) • DES (40 and 56 bits) • 3DES (2 and 3 keys) • AES (128, 192, and 256 bits)

Database Security


• Strong authentication– Two-factor (or "strong") authentication is based on

something the user has (a smart card, token, etc.) and a PIN or passcode. Supports:

• Kerberos • RADIUS (Remote Authentication Dial-In User Service) • Secure Sockets Layer (with digital certificates) • PKI

Database Security


• Use strong passwords• Try against a password checker• Passwords stored as hashes in database• Cleartext passwords can be typically but not necessarily found at the

following places– Server – Shell History files – Unix Scripts – Log Files – Dump Files – Trace Files – Application Server

Database Passwords

– JDBC-Config-Files – Trace Files – DBA Client PC – Desktop-Shortcut – Batch-Files – Tool Configuration files (connections.ini) – Trace Files


• Do not email passwords around

• Where are yours written down?

• Do your machine room staff have them? In emergency?

• Who actually knows the passwords? A list of people?

• Who has access to the host (as root? oracle?)

• Credentials often in Grid Control

Database Passwords


• We use “pwsafe”– Use SSH key to access– Password on pwsafe to use– File replicated across two sites– Usernames and passwords are stored using a unique name

to identify them, normally of the form group.name. However the group name is optional.

• o/s <machine name>.<o/s user>• database users: <database name>.<user name>• application express: <apex instance>.<workspace>

Database Passwords


• pwsafe– Showing A Password:

• # pwsafe up <unique name>

– Adding A Password:• # pwsafe a <unique name>

– Deleting A Password:• # pwsafe delete <unique name>

• Can add notes too

Database Passwords


• Check the Oracle default port list

• Changing the default ports can help to stop simple attacks but not real portscans.

• In Oracle it is very often not possible to change the default port because the port is hardcoded.

• At least for the Oracle database (except iasdb) it's is recommended to change the TNS listener port from 1521/1526 to something else.

• Options include running a local firewall then using something called SQLNet Proxy to manage the port. Otherwise we could use Connection Manager.

Machine Ports


Database Security Checklist


• Install only what is required

• Oracle Database Installation contains a host of options and products in addition to the database server.

• Install additional products and options only as necessary.

• Use the custom installation option to avoid installing unnecessary products or, perform a typical installation, and then deinstall unrequired options and products.

• It is best practise to know what the database will be used for so that you can apply only what Oracle products are needed.

Database Security Checklist – Step 1


• Common components that you can install with each database are: – Oracle Data Mining – Oracle Text – Oracle Olap – Oracle Spatial – Oracle Ultra Search – Oracle Label Security – Sample Schemas – Enterprise Manager Repository– Oracle database extensions for .NET




• None of these are needed to get a database up and running. They are "extras" to be added for more functionality.

• Check which of these have been installed by running DBCA

• ACTION: Use the instructions below to deinstall any options on PRODUCTION databases that are not actually used– Dropping Sample Schemas – Sample Schemas script directories are located in

$ORACLE_HOME/demo/schema. – The script xx_drop.sql, where xx is the schema abbreviation, removes

all objects from a particular schema. Run this for HR/OE/PM/IX/SH/BI schemas.



• Dropping schemas (cont)# cd $ORACLE_HOME/demo/schemas

# sqlplus hr/pass

SQL> @human_resources/hr_drop

SQL> exit

# sqlplus oe/pass

SQL> @order_entry/oe_drop

SQL> exit

# sqlplus pm/pass

SQL> @product_media/pm_drop

SQL> exit



• Dropping schemas (cont)# sqlplus ix/pass

SQL> @info_exchange/dix_v3.sql

-- ignore any errors

SQL> exit

# sqlplus sh/pass

SQL> @sales_history/sh_drop.sql

-- ignore any errors

SQL> exit



• Dropping schemas (cont)• no script to run for BI• Once this has been done for all the schemas, drop the users

HR/OE/PM/IX/SH/BI. SQL> drop user <user> cascade;

• Manual de-install of Spatial • Spatial is installed by default when using DBCA. To de-

install manually follow Metalink Note:179472.1

• Also drop the MDDATA user (the notes miss this one)


• Lock and Expire Default User Accounts • Oracle installs a number of default database server user

accounts.• On DBCA installation

– most default database user accounts automtically locked and expireed .

• On MANUAL installation– no accounts are locked

• The following two slides show a list of Oracle account and their status that is given AFTER the database is created using DBCA

• ACTION: Check these accounts are still expired and locked if not being used.







• Review all database users every 6 months (or more often)• Contact users and check schemas are still required

– If not export and drop them

• If any further options or Oracle products e.g. Apex, Oracle Warehouse Builder, HTTP server have been installed, check they are still required– If not deinstall them

• Run the Oracle Installer in $ORACLE_HOME/oui/bin/runInstaller to check if there are any other Oracle Homes with products in them and deinstall from there.

• Other products like Oracle Warehouse Builder may have an alternative de-install process. Check the documentation for your product.



• Change default User Passwords • Unlock ONLY those accounts that need to be accessed on a

regular basis and assign a strong password to each of these accounts.– Minimum password of 8 alphanumeric characters.– Make sure the passwords are different between the users– Make sure they are also different for any counterpart account

on a production/development/test account.

• This is mandatory for all DBA accounts on the database– sys, system, sysman, dbsnmp



• Change passwords with$ sqlplus / as sysdba

SQL> alter user system identified by ********;

User altered.

SQL>

• Basic password management rules (such as length, history, complexity etc...) as provided by the database be applied to all user passwords and that these passwords are changed periodically.

• ACTION: ensure all passwords are at least 8 random alphanumeric characters. If change of password is required to satisfy this, make sure you advise users/arrange appropriate downtime as an application may need to change it's settings.



• ACTION: Change SYS, SYSTEM, SYSMAN (on Grid Control DB), DBSNMP passwords every 6 months. Remember to change preferred credentials in Grid Control, agent config and pwsafe (or your wherever you store passwords).

• Changing DBSNMP password – Change DBSNMP password as above. To tell the agent that the

password has changed, edit the file $AGENT_HOME/<machine>/sysman/emd/targets.xml and edit the database target section as follows: Property NAME="UserName" VALUE="dbsnmp";

Property NAME="password" VALUE="newpass" ENCRYPTED="FALSE";

• Then stop and start the agent. The password will be encrypted again when the agent restarts.



• Enable Data Dictionary Protection• To prevent regular users that have ANY system privileges e.g. DROP

ANY TABLE being able to use these on the data dictionary make sure the parameter is O7_DICTIONARY_ACCESSIBILITY is set to FALSE.

• Action FALSE is the default in 10g but check it anyway $ sqlplus / as sysdba

SQL> show parameter O7_DICTIONARY_ACCESSIBILITY

NAME TYPE VALUE

------------------------------- ------------ -----

07_DICTIONAIRY_ACCESSIBILITY boolean FALSE



• If a user does need to access a dictionary view for non malicious information, they can be granted the "SELECT ANY DICTIONARY" privilege.



• Practise the principle of least privilege• Do not provide users with any more priviliges than are strictly

necessary• Restrict the following as much as possible:

– The number of SYSTEM and OBJECT privileges granted to database users

– 2) The number of people who are allowed to make SYS-privileged connections to the database.

• ACTION: Review the SYSTEM privs (Note most regular users should only have CREATE SESSION privilege or CONNECT role assigned (although check CONNECT role only contains "CREATE SESSION" privilege in DBA_SYS_PRIVS).



• ACTION: The RESOURCE role (allows creation of tables etc) should only be assigned on creation of the schema and for schema changes and then dropped again on production databases). Drop RESOURCE role from regular users (see DBA_SYS_PRIVS). Review the contents of the RESOURCE role in DBA_ROLE_PRIVS.

• ACTION: Also "ANY" privileges should not be assigned as they allow this user access to objects in all other schemas. Change ANY system privileges to object privileges e.g. drop SELECT ANY TABLE and assign ....



• To check currently assigned system privsSELECT * from DBA_SYS_PRIVS

WHERE GRANTEE NOT IN ('DBA', 'SYS', 'SYSTEM', 'OEM_MONITOR', 'OEM_ADVISOR', 'IMP_FULL_DATABASE', 'EXP_FULL_DATABASE', 'ORDSYS', 'ORDPLUGINS', 'JAVADEBUGPRIV', 'OUTLN', 'FLOWS_020200', 'FLOWS_030000', 'AQ_ADMINISTRATOR_ROLE', 'CTXSYS', 'EXFSYS', 'XDB', 'APEX_PUBLIC_USER', 'CONNECT', 'RESOURCE', 'DIP',

'SCHEDULER_ADMIN', 'DBSNMP', 'ANONYMOUS', 'WMSYS', 'RECOVERY_CATALOG_OWNER')

ORDER BY GRANTEE, PRIVILEGE;



• Review and revoke unneccessary privs– GRANT CONNECT TO <user>;– REVOKE CREATE PROCEDURE FROM <user>;– REVOKE RESOURCE FROM <user>;

• To see roles assigned to users– SELECT * FROM dba_role_privs WHERE USER = 'ADAM';

• ACTION: Review object privs for each regular user and revoke as necessary – select * from dba_tab_privs where grantee ='SSO';

• ACTION: Check that only SYS has DBA role



• Additionally, review the privileges grants to the PUBLIC role as you may wish to restrict what access this will give to all users.

• Oracle provides execution rights to four packages on the PUBLIC role that should be removed after installation:

• ACTION: Revoke the execution privilege on PUBLIC to the packages:– UTL_SMTP [EMAIL]– UTL_TCP [Network Connections]– UTL_HTTP [request/retrieve information from HTTP server]– UTL_FILE [access to files outside the DB]



• Remove with# sqlplus / as sysdba

SQL> REVOKE EXECUTE ON sys.utl_smtp FROM "PUBLIC";

Revoke succeeded.

SQL> REVOKE EXECUTE ON sys.utl_tcp FROM "PUBLIC";

Revoke succeeded.

SQL> REVOKE EXECUTE ON sys.utl_http FROM "PUBLIC";

Revoke succeeded.

SQL> REVOKE EXECUTE ON sys.utl_file FROM "PUBLIC";

Revoke succeeded.



• Enforce access controls effectively and authenticate clients stringently

• Setting the initialization parameter REMOTE_OS_AUTHENT = FALSE, creates a more secure configuration that enforces proper, server-based authentication of clients connecting to an Oracle database.

• You should not alter the default setting of the REMOTE_OS_AUTHENT initialization parameter, which is FALSE. Setting this parameter to FALSE does not mean that users cannot connect remotely. It simply means that the database will not trust that the client has been already authenticated, and will therefore apply its standard authentication processes.



• ACTION: Check parameter REMOTE_OS_AUTHENT is FALSE on each database instance (all on RAC): $ sqlplus / as sysdba

SQL> show parameter remote_os_authent

NAME TYPE VALUE

---------------------- --------------------- -----

remote_os_authent boolean FALSE



• Restrict Operating System Access• Limit the number of operating system users.• Limit the privileges of the operating system accounts

(administrative, root-privileged or DBA) on the host (physical machine) to the least privileges needed for the user's tasks.

• ACTION: Check with System Administrator that only default users have been created and any unused default ones are locked. Ask them to remove any old users.

• ACTION: Check with System Administrator that the privileges of OS accounts have the least privileges necessary.



• ACTION: Check that any database developers have their own O/S id and are in the osinstall group but not the dba group. This allows them to access sqlplus but not to log on as SYSDBA.

• Restricting the ability to modify the default file and directory permissions for the Oracle Database home (installation) directory or its contents. Even privileged operating system users and the Oracle owner should not modify these permissions, unless instructed otherwise by Oracle.



• ACTION: Check with System Administrator that no O/S users can change file/directory permissions of the Oracle Home

• Restricting symbolic links. Ensure that when providing a path or file to the database, neither the file nor any part of the path is modifiable by an untrusted user.

• The file and all components of the path should be owned by the DBA or some trusted account, such as root. This recommendation applies to all types of files: data files, log files, trace files, external tables, bfiles, and so on.

• ACTION: Check no symbolic links against data files, log files, trace files, external tables, bfiles etc



• Restrict Network Access

• Use a firewall

• Oracle Listener (9i)– An Oracle Listener running without an established password may be

probed for critical details about the databases on which it is listening such as trace and logging information, banner information and database descriptors and service names.

– Restrict the privileges of the listener, so that it cannot read or write files in the database or the Oracle server address space. This restriction prevents external procedure agents spawned by the listener (or procedures executed by such an agent) from inheriting the ability to do such reads or writes.



• Oracle Listener– Prevent online administration by requiring the administrator to have

write privileges on the LISTENER.ORA file– Use the parameter ADMIN_RESTRICTIONS_listener_name to restrict

runtime administration of the listener. The parameter is useful if the listener is not password-protected.

– Setting ADMIN_RESTRICTIONS_listener_name=on disables the runtime modification of parameters in listener.ora. That is, the listener will refuse to accept SET commands that alter its parameters.

– To change any of the parameters in listener.ora, including ADMIN_RESTRICTIONS_listener_name itself, modify the listener.ora file manually and reload its parameters (with the RELOAD command) for the new changes to take effect without explicitly stopping and restarting the listener.



• ACTION: Add or alter this line in the LISTENER.ORA file – ADMIN_RESTRICTIONS_LISTENER=ON

Then RELOAD the configuration.

• If you are administering the listener remotely over an insecure network and require maximum security, configure the listener with a secure protocol address that uses the TCP/IP with SSL protocol. If the listener has multiple protocol addresses, ensure that the TCP/IP with SSL protocol address is listed first in the listener.ora file.



• ACTION: Use SSL when administering the listener, by making the TCPS protocol the first entry in the address list as follows: LISTENER=

(DESCRIPTION=

(ADDRESS_LIST=

(ADDRESS=

(PROTOCOL=tcps)

(HOST = ed-pdsun1.us.oracle.com)

(PORT = 8281)))



• For Oracle Database 10g Release 1 and higher the default authentication mode is local O/S authentication which requires administrator to be a member of the local dba group.

• Setting a password for the TNS listener in Oracle Database 10g Release 1 and higher simplifies administration.

• However, setting a password requires good password management to prevent unauthorized users from guessing the password and potentially gaining access to privileged listener operations.

• Customers may wish to consider not setting a password for the TNS listener starting with Oracle Database 10g Release 1.

• ACTION: No need to set a listener password for release 10.1 or higher but ensure that only the oracle O/S user (or equivalent) is a member of the dba O/S group (or equivalent)



• ACTION: Remove the external procedure configuration from the listener.ora file if you do not intend to use such procedures. Remove the following

EXTPROC_LISTENER=

(DESCRIPTION=

(ADDRESS=

(PROTOCOL=ipc)(KEY=extproc)))



• Monitor listener activity

• Authenticating client computers over the Internet is problematic. Do user authentication instead, which avoids client system issues that include falsified IP addresses, hacked operating systems or applications, and falsified or stolen client system identities.

• Configure the connection to use SSL. Using SSL (Secure Sockets Layer) communication makes eavesdropping unfruitful and enables the use of certificates for user and server authentication. (requires OAS??)



• Consider setting up certificate authentication for clients and servers such that: – The organization is identified by unit and certificate issuer and the

user is identified by distinguished name and certificate issuer.– Applications test for expired certificates.– Certificate revocation lists are audited



• ACTION: Check network IP addresses– Use the Oracle Net valid node checking security feature to allow or deny

access to Oracle server processes from network clients with specified IP addresses. To use this feature, set the following protocol.ora (Oracle Net configuration file) parameters:

tcp.validnode_checking = YES

tcp.excluded_nodes = {list of IP addresses}

tcp.invited_nodes = {list of IP addresses}

– The first parameter turns on the feature whereas the latter parameters respectively deny and allow specific client IP addresses from making connections to the Oracle listener (This helps in preventing potential Denial of Service attacks).



• Encrypt network traffic– If possible, use Oracle Advanced Security to encrypt network traffic

between clients, databases, and application servers.

• Harden the operating system – disabling all unnecessary operating system services.– Both UNIX and Windows platforms provide a variety of operating system

services, most of which are not necessary for most deployments. Such services include FTP, TFTP, TELNET, and so forth.

– Be sure to close both the UDP and TCP ports for each service that is being disabled. Disabling one type of port and not the other does not make the operating system more secure.

– See your friendly local system administrator



• Apply all security patches

• Always apply all relevant and current security patches for both the operating system on which Oracle Database resides and Oracle Database itself, and for all installed Oracle Database options and components.

• Periodically check the security site on Oracle Technology Network for details on security alerts released by Oracle Corporation at http://www.oracle.com/technology/deploy/security/alerts.htm



• Also check Oracle Worldwide Support Service site, Metalink, for details on available and upcoming security-related patches at http://metalink.oracle.com

• ACTION: Check with System Administrator re outstanding OS security patches and apply

• ACTION: Install Oracle Critical Patch updates as soon as possible after release



• Contact Oracle Security products if you come across a vulnerability in Oracle Database

• If you believe that you have found a security vulnerability in Oracle Database, then submit a Service Request to Oracle Worldwide Support Services using Metalink, or e-mail a complete description of the problem, including product version and platform, together with any exploit scripts and examples to the following address: [email protected]



• Policies in Enterprise Manager – Each target should be checked to make sure it complies with

all security polices for host, database and cluster instances, listener and other targets where appropriate.

• ACTION: Go to Grid Control, click on Compliance | Policy Violations. Correct any security violations for your system including Host, Cluster Database, Database Instance and Listener targets.

Database Security Checklist – Other Steps


• Personal Data (from HBI audit) – If the database schema contains Personal data it must comply

to the Data Protection Act. – Define Personal Data - what are the rules of compliance?

• Sensitive Data (from HBI audit) – If the database schema contains Sensitive data it must comply

to the Data Protection Act. – Define Sensitive Data - what are the rules of compliance?

Database Security Checklist – Other Steps


• Audit any object

NAME VALUE

---------------- ------

audit_file_dest /opt/oracle/app/oracle/admin/ngsdb/adump

audit_sys_operations FALSE

audit_trail DB

• See dba_audit_* tables

• There is an AUDIT privilege

• Logins/logouts stored in dba_audit_session

Database Auditing


• Check logins

SET LINESIZE 120;COL username FORMAT a15COL terminal FORMAT a15COL logon_time FORMAT a25COL logoff_time FORMAT a25COL action_name FORMAT a20COL returncode FORMAT 9999SELECT username, terminal, action_name, TO_CHAR(timestamp,'DD-MON-YYYY HH24:MI:SS') logon_time, TO_CHAR(logoff_time,'DD-MON-YYYY HH24:MI:SS') logoff_time, returncodeFROM dba_audit_sessionWHERE ROWNUM < 16ORDER BY logoff_time DESC;

Database Auditing


• Check logins

Database Auditing


• O/S logins monitored in ora_<pid>.aud files

Audit file /opt/oracle/app/oracle/admin/<host>/adump/ora_22155.audOracle Database 10g Enterprise Edition Release 10.2.0.3.0 - ProductionWith the Partitioning, Real Application Clusters, OLAP and Data Mining optionsORACLE_HOME = /opt/oracle/app/oracle/product/10.2.0System name: LinuxNode name: <hostname>Release: 2.6.9-55.0.2.ELsmpVersion: #1 SMP Tue Jun 12 17:59:08 EDT 2007Machine: i686Instance name: <instance name>Redo thread mounted by this instance: 1Oracle process number: 76Unix process pid: 22155, image: oracle@<hostname> (TNS V1-V3)

Wed Nov 28 20:54:00 2007ACTION : 'CONNECT'DATABASE USER: '/'PRIVILEGE : SYSDBACLIENT USER: oracleCLIENT TERMINAL: pts/0STATUS: 0

Database Auditing


• Use ssh key pairs– Update /etc/ssh/sshd_config where you can say that you do

not want to allow password authenticationPasswordAuthentication yesto

PasswordAuthentication nothen restart the ssh daemon/etc/init.d/sshd restart

– After that change you need to have your public ssh key in the ~/.ssh/authorized_keys file for any user that you would like to log in as.

Host Security


• Do not leave host names or database names in presentations etc

• Google your hosts (or will that then go in their cache?)

• Lock your computer when you’re away

• Don’t leave putty or “screen” sessions open

• Speak to your security officer if in any doubt

• Arrange a security audit

Other Security


Host HousekeepingManaging Your Environment


• Log files can grow to large sizes and be difficult to view or take up too much space

• We have developed script to purge them– Runs through Grid Control

• Make sure files are backed up first if you need them

• Balance between history and space

Host Housekeeping


function rmfiles { DIR=$1 DESC=$2 DAYS=$3 if [ -d $DIR ] then echo ` ` echo "Checking ${DESC}: ${DIR}" echo "Removing files/directories older than: $DAYS days" # The * stops find deleting the directory itself echo `find ${DIR}/* -mtime +$DAYS | wc -l` files or directories # This lists the files/dirs to delete find ${DIR}/* -mtime +$DAYS # then deletes them find ${DIR}/* -mtime +$DAYS -exec rm -rf {} \; else echo "Directory $DIR does not exist on this server" fi}

Host Housekeeping


echo "Housekeeping files"

date +\%d\%m\%y_\%H\%M\%S

echo `date`

OBASE=/opt/oracle/ora01/app/oracle

OHOME=$OBASE/OracleHomes/10.2.0/db_1

AGHOME=/opt/oracle/ora01/app/oracle/OracleHomes/oms10g/agent10g

# For RAC

CRSHOME=/opt/oracle/crs/oracle/product/10/app

# For HTTP server installations

HTTPHOME=/opt/oracle/ora01/app/oracle/OracleHomes/oms10g/oms10g

# For Grid Control installations

OMSHOME=/opt/oracle/ora01/app/oracle/OracleHomes/oms10g/oms10g

Host Housekeeping


rmfiles "$OBASE/admin/$ORACLE_SID/bdump" "BACKGROUND dump destination" 120

rmfiles "$OBASE/admin/$ORACLE_SID/cdump" "CORE dump destination" 31

rmfiles "$OBASE/admin/$ORACLE_SID/dpdump" "DATA PUMP destination" 31

rmfiles "$OBASE/admin/$ORACLE_SID/udump" "USER dump destination" 31

rmfiles "$OBASE/admin/$ORACLE_SID/adump" "AUDIT destination" 31

rmfiles "$OBASE/admin/$ORACLE_SID/hdump" "HIGH AVAILABILITY tracefile destination" 31

rmfiles "$OHOME/admin/+ASM/bdump" "ASM BACKGROUND dump destination" 31

rmfiles "$CRSHOME/log/`hostname -s`/client" "CRS client logs" 31

rmfiles "$AGHOME/`hostname`/sysman/log" "AGENT logs" 31

rmfiles "$AGHOME/sysman/log" "AGENT logs" 31

rmfiles "$HTTPHOME/Apache/Apache/logs" "HTTP SERVER logs" 31

rmfiles "$OMSHOME/opmn/logs" "OMS opmn logs" 31

rmfiles "$OMSHOME/sysman/log" "OMS sysman logs" 31

Host Housekeeping


Housekeeping files 291107_074653 Thu Nov 29 07:46:53 GMT 2007

Checking BACKGROUND dump destination:/opt/oracle/ora01/app/oracle/admin/mercury/bdumpRemoving files/directories older than: 120 days1 files or directories/opt/oracle/ora01/app/oracle/admin/mercury/bdump/alert_mercury.log.20070731

Checking CORE dump destination:/opt/oracle/ora01/app/oracle/admin/mercury/cdumpRemoving files/directories older than: 31 daysfind: /opt/oracle/ora01/app/oracle/admin/mercury/cdump/*: No such file or directory 0 files or directories

.

.

Completed Thu Nov 29 07:46:55 GMT 2007

Host Housekeeping








• Script keeps alert log to a managable size• Allows file to be emailed daily

#!/bin/bashBDUMP=/opt/oracle/app/oracle/admin/ogma/bdump

# This script will move and rename the current alert.log by adding the date as the extension

# GDB 08-OCT-2004

# Email the logcat $BDUMP/alert_ogma1.log | mail -s "Oracle: ogma1 alert log for `date`"

db_admins

# Rename the logCURRDATE=`date '+%Y%m%d'`cp $BDUMP/alert_ogma1.log $BDUMP/alert_ogma1.log.$CURRDATEcat /dev/null > $BDUMP/alert_ogma1.log

Alert Log Maintenance


• Run daily from cron0 8 * * * /home/oracle/mercury/alert.sh >/dev/null 2>&1

• Emailed to group– Details in /home/oracle/.mailrc file

alias db_admins [email protected]



• Keeps alert logs small and easy to find

-rw-r----- 1 oinstall 35347 Nov 20 08:00 alert_ogma1.log.20071120









-rw-r----- 1 oinstall 0 Nov 28 08:00 alert_ogma1.log



• Directory Structure – /opt/oracle/product/10.2.0/db_1 # oracle database home– /opt/oracle/product/10.2.0/client_1 # oracle client home – /opt/oracle/product/10.2.0/agent_1 # oracle agent home– /opt/oracle/product/10.2.0/crs # oracle clusterware home– /opt/oracle/oraInventory # oracle inventory– /opt/oracle/archive/<sid> # archives if not using ASM– /opt/oracle/backup/<sid> # backups– /opt/oracle/oradata/<sid> # data if not using ASM– /opt/oracle/admin/<sid> # logs and alerts– /opt/oracle/flash_recovery_area/<sid> # flash recovery area if

used

Host Housekeeping


• All database, environment and schema scripts should be kept under /home/oracle

• The /home/oracle directory should be the home directory for the unix user oracle

• In /home/oracle the following scripts should be available: – ora10<database name>.sh # Script to set database SID– ora10asm.sh # Script to set ASM SID– ora10common.sh # Script to set common environment variables

• Scripts should use LOGIN_PATH and LOGIN_LD_LIBRARY_PATH environment variables so that ora10 scripts can be run more than once in a session and the variables PATH and LD_LIBRARY_PATH to not grow to big by adding themselves to their new definition

Host Housekeeping


• Summary of environment files[oracle@lcgdb01 ~]$ l ora*

-rwxrwxrwx 1 oinstall 43 May 18 2006 ora10asm.sh

-rwxrwxrwx 1 oinstall 302 May 22 2006 ora10agent_cern.sh

-rwxrwxrwx 1 oinstall 5114 Mar 29 2007 orahousekeep.sh

-rwxrwxrwx 1 oinstall 552 May 22 2007 ora10common.sh

-rwxrwxrwx 1 oinstall 55 Aug 8 10:00 ora10ogma.sh

-rwxrwxrwx 1 oinstall 298 Sep 12 15:32 ora10agent_cclrc.sh

Host Housekeeping


• ora10common.sh# clusterwareexport CRS_HOME=/opt/oracle/crs/oracle/product/10/app

# oracleexport ORACLE_BASE=/opt/oracle/app/oracleexport ORACLE_HOME=$ORACLE_BASE/product/10.2.0export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/jdk/bin:

$CRS_HOME/binexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

# sqlplusexport EDITOR=viexport ORA_NLS10=$ORACLE_HOME/nls/dataexport NLS_LANG="english_united kingdom.AL32UTF8"export NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"export RMAN="$ORACLE_HOME/bin/rman target / catalog rman/<password>@rcat"

Host Housekeeping


• ora10ogma.sh/home/oracle/ora10common.sh

export ORACLE_SID=ogma1

Host Housekeeping


• ora10asm.sh/home/oracle/ora10common.sh

export ORACLE_SID=+ASM1

Host Housekeeping


• Agent environment script: ora10agent_cclrc.shexport ORACLE_BASE=/opt/oracle/app/oracle

export ORACLE_HOME=$ORACLE_BASE/agent10g

export AGENT_HOME=$ORACLE_BASE/agent10g

unset ORACLE_SID

export PATH=$PATH:$ORACLE_HOME/lcgdb01.gridpp.rl.ac.uk/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/jdk/bin

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

Host Housekeeping


• Agent environment script: ora10agent_cern.shexport ORACLE_BASE=/opt/oracle/app/oracle

export ORACLE_HOME=$ORACLE_BASE/agent10g_CERN/agent10g

export AGENT_HOME=$ORACLE_BASE/agent10g_CERN/agent10g

unset ORACLE_SID

export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/jdk/bin

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

Host Housekeeping


• Agent environment scripts– Check you are using the right emctl– which emctl

• Check status with emctl status agent

• Check files are uploading

• Automatically start on host boot

Host Housekeeping


• System documentation– versions, hosts, databases, users, patch versions

• wiki– Who can access?

• Disaster Recovery

• Contacts

• Standard Operating Procedures– startups, shutdown, user alerts

• Do you know who to ring/email in an emergency?

• Do you know what to without power/telephones?

Documentation


Grid Control



• Secure agent upload

• The agents as to be secured against the OMS before they can be upgrade via grid control. – ./emctl secure agent

• To secure the agent you need a registration password from the OMS. Which you can obtain or add one to use for a one -off registration

• Go to Grid Vontrol setup page and then click-on registration password link.

Grid Control


target properties

Always set Target Properties


Target Properties found in Target Home


There is workaround for missing button!


Check alerts - Critical


Check alerts - Warning


Check alerts - Errors


Set Credentials


email alerts

Setup Email Alerts


Setup Templates


Setup Metric Thresholds


Backups


• Backup details stored centrally• Common scripts• One connection from all machines• Version has to be newest• Setup at RAL:

– Dedicated server– Cold backups

Backups


• Jobs in crontab on each database

# RMAN Schedules follow here

30 1 * * 1 /home/oracle/rmanfulback.sh mars rman <password> rcat >/dev/null 2>&1

30 1 * * 2-7 /home/oracle/rmanincback.sh mars rman <password> rcat >/dev/null 2>&1

30 3 * * * /home/oracle/rmanvalidate.sh mars rman <password> rcat >/dev/null 2>&1

Backups – RMAN Catalog


#!/bin/ksh####################################################################### THE INPUT PARAMETERS ARE :## $1 - Target DB# $2 - RMAN owner# $3 - RMAN password# $4 - RMAN DB## Script to backup archived redo log files####################################################################### Check arguments#if [[ $# -ne 4 ]]then echo "$0 Error on $HOST: \nUsage: $0 TARGET_DB RMAN_OWNER RMAN_PWD RMAN_DB" exit 1fi

Backups – Full Backup Script (1)


###################

# RMAN variables #

###################

#

export SID=$1

export RMAN_OWNER=$2

export RMAN_PWD=$3

export RMAN_DB=$4

export LOGIN_PATH=$PATH

export LOGIN_LD_LIBRARY_PATH=$LD_LIBRARY_PATH

. /home/oracle/ora10$SID.sh

export HBlog=/home/oracle/$SID/rmanlogs/fulback${SID}.`date +%d%m%y%H%M%S`.log

echo $HBlog

export emailsub=$HOSTNAME.$ORACLE_SID..Backup.Failure



rman target / catalog ${RMAN_OWNER}/${RMAN_PWD}@${RMAN_DB} log \"${HBlog}\" << SQLshow all;run {backup incremental level 0 database plus archivelog delete input;}report obsolete;delete noprompt obsolete;list backup;ExitSQL

oraerror=`cat $HBlog | grep "ORA-"`rmanerror=`cat $HBlog | grep "RMAN-"`echo $oraerrorecho $rmanerrorif [ -z "$oraerror" -a -z "$rmanerror" ]then echo "No Errors"else echo "Errors Found !!!!" cat $HBlog |mailx -s $emailsub [email protected]



show all;

run {

backup incremental level 1 database plus archivelog delete input;

}

report obsolete;

delete noprompt obsolete;

list backup;

exit

Backups – Level 1 Backup Script


show all;

run {

backup archivelog all delete input;

}

report obsolete;


list backup;

exit

Backups – Archive Logs Backup Script


• Run through OEM

show all;

run {

resync catalog;

}

list backup;

exit

Backups – RMAN Sync Script


show all;

run {

restore validate database archivelog all;

}

list backup;

exit

Backups – Restore Validate Script


Backup Job in Grid Control


Latest Database Backup Times

Database DBID Latest Backup Bytes Processed Seconds Taken Backup Type

------------------------ ---------- ----------------------------- --------------- ------------- ---------------------

ATLASDLF 4091293823 26-NOV-2007 23:44:39 6021971968 3850 DB INCR

ATLSTAGE 819435209 27-NOV-2007 00:54:13 85770240 56 DB INCR

CASTORP 2263675954 27-NOV-2007 00:31:40 12722176 26 DB INCR

CASTORT 2415163970 16-NOV-2007 00:31:39 23625728 28 DB INCR

CMSDLF 1227029095 26-NOV-2007 21:11:42 3530555392 2098 DB INCR

.

.

MINERVA 402814294 26-NOV-2007 22:24:11 6494879744 4843 DB INCR

STAGERDB 1089287619 26-NOV-2007 22:31:57 164298752 91 DB INCR

VENUS 2579470848 26-NOV-2007 22:17:14 3631218688 2679 DB INCR

WARHORSE 780645177 26-NOV-2007 18:47:06 127975424 82 DB INCR

WARHORSE 792067278 27-NOV-2007 00:46:17 7471104 9 DB INCR

23 rows selected.

---------------------------------------------------------------------------------------------------------------------

The Following Databases have not been backed up in the last 24 hours

DB_NAME

------------------------

CASTORT

Backups – Backup Report by Email (1)


Latest Restore Validate Runs

Database DBID Latest Restore Validate

------------------------ ---------- -----------------------------

VENUS 2579470848 27-NOV-2007 01:11:22

ATLASDLF 4091293823 27-NOV-2007 04:38:24

LHCBSTAG 3505332007 27-NOV-2007 02:50:41

MERCURY 344759533 27-NOV-2007 06:42:30

.

.

WARHORSE 792067278 27-NOV-2007 02:46:54

SETH 2382003523 27-NOV-2007 02:07:17

CASTORP 2263675954 27-NOV-2007 03:32:05

HEKATE 3791789259 27-NOV-2007 00:04:33

CMSDLF 1227029095 27-NOV-2007 00:19:53

CASTORT 2418831922 27-NOV-2007 02:34:26

MARS 1068316094 27-NOV-2007 04:48:05

LHCBDLF 2704015590 27-NOV-2007 03:06:59

24 rows selected.

--------------------------------------------------------------------------------------------------------

The Following Databases have not had a Validate in the last 24 hours

no rows selected

Backups – Backup Report by Email (2)


###################################################################### This script Produces Backup Reports to be sent to the DBAs## Amendments :# AJW - 15/05/2007 initial construction#####################################################################

# set the rcat environment

. /home/oracle/ora10rcat.sh

sqlplus / as sysdba <<-!!@/home/oracle/rmanrep1.sqlexit!!

export emailsub=Backup.Report.`date +%d%m%y%H%M%S`echo $emailsub

cat /home/oracle/rmanrep1.txt |mailx -s $emailsub [email protected]

Backups – Email Script


spool /home/oracle/rmanrep1.txt;set line 120set pagesize 999

/* get list of latest database backup times */

select a.db_name "Database", db.dbid "DBID", a.end_time "Latest Backup", a.output_bytes "Bytes Processed", (end_time - start_time) * 60 * 60 * 24 "Seconds Taken" from rman.rc_rman_status a, rman.rc_database dbwhere object_type in ('DB FULL','DB INCR') and status = 'COMPLETED' and operation = 'BACKUP'and end_time = (select max(end_time) from rman.rc_rman_status b where b.db_name = a.db_name and b.db_key = a.db_key and object_type in ('DB FULL','DB INCR') and status = 'COMPLETED' and operation = 'BACKUP')and db.db_key = a.db_keyorder by end_time ;

Backups – Email SQL


${ORACLE_HOME}/bin/rman target / nocatalog log \"${HBlog}\" << SQL

show all;

shutdown immediate

startup mount;

backup database;

startup;

report obsolete;


list backup;

exit

Backups – Catalog Backup Script


RMAN> show all;

RMAN configuration parameters are:CONFIGURE RETENTION POLICY TO REDUNDANCY 2;CONFIGURE BACKUP OPTIMIZATION ON;CONFIGURE DEFAULT DEVICE TYPE TO DISK;CONFIGURE CONTROLFILE AUTOBACKUP ON;CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/opt/oracle/backup/mars/%F.bak';CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1;CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 100 M FORMAT '/opt/oracle/backup/mars/mars_

%U.bak';CONFIGURE MAXSETSIZE TO UNLIMITED;CONFIGURE ENCRYPTION FOR DATABASE OFF;CONFIGURE ENCRYPTION ALGORITHM 'AES128';CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/opt/oracle/app/oracle/product/10.2.0/dbs/snapcf_mars1.f';

Backups – RMAN Configuration


• We run backup jobs through crontab

• Resync and Email job through OEM

• Archive Logs backup job could be run automatically as oem job when area 70% full (for example)

• Keep it simple, keep it documented

Backups - Summary


Questions & (hopefully) Answers

[email protected]

Documents

Lots of Stuff