gozonian.orggozonian.org/.../DALECLARKE_KMA_PRACNIQUES_UNI… · Web viewUnix Praqniques (Practical Techniques)by Dale C ClarkePage 1

PRAQNIQUES

=================================================================================================================Index for this document.... Search on the sequence number in case there is more full documentation (eg. find 18.)8E08B46DCB99DA4278C860F37A

Daily use (20120517-1012)=================================================================================================================01. Convert dos2unix in place - perl -i -pe 's/\r//g' <fully qualified path/name> Convert with nawk 'sub("$", "\r")' LOGS/20111107_073010_stageing_content.txt > LOGS/20111107_073010_stageing_content.txt3 Convert with sed in perl - system(" sed -e 's/\$/\r/' ${xcontfile} > ${xattachfile} ");01a. Convert unix2dos - perl -pi -e 's/\n/\r\n/' Convert unix2dos - sed s/$/$'\r'/ < input.txt > output.txt

02. List dirs indented - ls -R | grep ':$' | sed -e 's/:$//' -e 's/[^-][^\/]*\//|--/g' -e 's/^/ /'03. List the netgroups on a server - cat /etc/passwd |grep +@ |awk -F: '{print $1}' |awk -F\@ '{print $2}' Must be on a Solaris box for this command. eg. pcredit01EXAMPLE:login as: as EE pwh-sas17lp2g8udoc@pwh-sas17lp2's password:% sudo -H -u badftpp sudo-sh; export TMOUT=0; cd /export/appl/badops/tools/dseg8udoc's Password:% export TMOUT=0; cd /export/appl/badops/tools/dse% /export/appl/badops/tools/dse/WRK/getnetg.ksh dcredit-sas03lp2

04. nmon -h (help) -t for total CPU - wait for the screen to refresh as it is many CPUs05. Display or list only files Type the following command: ls -l | grep -v '^d'06. EMM reports: http://alsysadm-epmw05b/ssmview/filesys/index.html 07. Turn off sudo timeout once you are in sudo: export TMOUT=0 08. MUST BE ON SOLARIS psysadm50% ldaplist -l passwd m5prod |grep userPassword userPassword: {crypt}*NOLOGIN*09. Reboot schedules found at: http://techops/cgi-bin/autosys/reboot_jobs.pl?format=1 and

10. IP address - /usr/sbin/ifconfig -a11. direct login - m5prod - (must be solaris) dcredit01% ldaplist -l passwd m5prod |grep userPassword (KEY IS ctl-f/ldapdcc)

userPassword: {crypt}*NOLOGIN*12. LUN/SAN - > lsvg (result rootvg oscopyvg1 DG01_T1 DG01_T0 DG01_T3 DG01_T2 oscopyvg2 , > lsvg DG01_T2 | grep FREE

Unix Praqniques (Practical Techniques)by Dale C Clarke Page 1

PRAQNIQUES

13. Usage investigation - du -khsa /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/*14. Usage investigation (AIX) - du -sg /export/appl/lw_data_d/prod/lwprod/lfm/*15. Numeric userid (solaris only - dcredit01) ldaplist -l passwd m5prod |grep idNumber gidNumber: 8209 uidNumber: 3363916. perl create a dir if it does not exist - unless(-d $dir){mkdir $dir or die;}17. perl code for MD5 perl for files hash18. Here are the command lines to test SAS - Nishit19. Evaluate FS sizes with df - Eval this from John on 8/2620. Creation of a lockdown doc and ticket xxx21. ASD install issues 4/22/1122. ps -ef |grep rsync |grep badops - check for hung async (John Simon)23. date; pwd; ls -laR|grep -v "\-\-\- "|grep -v "total "|grep -v "vbadf" <--- get universal read/execute24. date; uname -a ; df -gs /export/appl/lw_data_p3 <---AIX df25. Outlook logon correction faq26. Link to good advanced vi (Visual editer cmds) http://staff.washington.edu/rells/R110/27. Perl output to screen AND logfile - LoadAll.sh 2>&1 | tee -a Load.log28. Delete a file with no name or special char in the name. First find out file inode number with any one of the following command: stat file-name OR ls -il file-name If you cannot type the file name, just use * instead of file-name. Use find command as follows to find and remove the specified file: find . -inum [inode-number] -exec rm -i {} \;29. find and list all directorys in the current directory

find . | xargs perl -e '@ARGV = grep( -d $_ , @ARGV); print "@ARGV \n"'

30. Link to Cygwin - \\fanniemae.com\CORP\DC\SHARED\DAC\CYGWIN-15X

31. Change the -mdate (last modified date) - touch -t 200012311800 employees.txt

32. rsync --dry-run -razvv /export/appl/lw_archive_p/* /export/appl/bad_archive1_p (From John Simon) then if all looks good nohup rsync -razvv /export/appl/lw_archive_p/* /export/appl/bad_archive1_p &

Note: remove the -z turn of the real-time verify to speed a factor of 2x Runs ~11GB/hr on same server. Remove z to transfer raw to speed up and use cksum against directories to verify.

==================================================


PRAQNIQUES

EXAMPLE WITH OUTOUT NAMED bad_data1_w [and nohup] rsync --dry-run -ravv /export/appl/lw_data_p2/* /export/appl/bad_data1_w then if all looks good nohup rsync -ravv /export/appl/lw_data_p2/* /export/appl/bad_data1_w &

To check the structure of the output: ls -laR /export/appl/bad_data1_w/* | grep ":" | grep -v "drw" | tail -5 To watch it grow: cd /export/appl/badops/tools/arc/ WRK/archive/watcher51.bsh bad_data1_w

33. To play with - alias jobs=` ps -e -a -o user,pid,etime,pcpu,args | grep /export/apps/sas | grep -v /utilities/bin | grep -v grep | sort` From 17lp2 lwprod sudo alias

34. grep examples (LOTS)

35. How to get a server or dns name for and IP address - by Kate Dougherty % nslookup <IP Address>example:

35a. $ cp filename{,.bak} --> In this one-liner filename{,.bak} gets copied filename.bak

36. Using cut to Print out unique lines in a file - Print out unique lines in a file cat LOGS/CMDSRUN.txt| sort -n | uniq |head- Print out unique lines in a file for csv selecting 2nd field cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -nr | uniq |head

cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename

37, Get "ALL" the ldap information for userid g8udoc - by Kate Dougherty

- % ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep gid

38. =================================================================================================================


PRAQNIQUES

Global find and/or replacefind ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print <--- find in the directory files that contain paul_obrecht and print themfind ./VAL/RTI.d20120913DCCbackup/ | xargs perl -pi -e 's/paul_obrecht/ /g' <--- find in the directory files that contain paul_obrecht and replace them with a blank

39. =================================================================================================================GET A NAME FROM A USERID ON UNIX===================================== Check OS: uname -aSolaris: ldaplist -l passwd g8udoc|grep cn:AIX: getable -a passwd g8udocLinux: ldapsearch???

Linux:

40. #############################################################################################################Get a nice clean recursive chmod without going outside the directory (like following links) (John Simon)===================================== chmod -R -h 755 /export/appl/ain_archive2_p/LPpop

41. . =============================================================================================================Quickies:

df -g |grep add ß Check the EPF env is on the server (addprod= prod, addacpt=acpt)EPF Schuff:Qtr: (extention [zip or tar] MUST be the same for bert as used in fred&ernie); -t is for trace only for checking – remove for live executionssh -o "StrictHostKeyChecking no" g8udoc@astate-db03 ls -ltr /export/appl/addacpt/controlfile|grep G7190 |grep 20122 ß ACPTFred: % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -f -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Ernie % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -e -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Bert: % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -b -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t

Mon: (MON has 6 digit date, below is month 12 || QTR is 5 digits [q-1-4], above is 2nd)ssh -o "StrictHostKeyChecking no" g8udoc@pstate-db03 ls -ltr /export/appl/addprod/controlfile|grep G7190 |grep 201212 ß PRODFred: % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -f -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t


PRAQNIQUES

Ernie % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -e -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Bert: % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -b -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t

42. =================================================================================================================TRANSFER A FILE WITH SCP THAT HAS A SAS GDG (INVALID # CHARACTER IN THE FILE NAME) On the sourceserver:% ls -il lasdbkp*% find . -inum 13% find . -inum 13 -exec echo {} \;% find . -inum 13 -exec chmod 777 {} \;% find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \;On the target server:% cd /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/% find . -inum 77921 -exec mv {} dstrs_clm_draft.sas7bdat% ls -ltr% uname -a; pwd;cksum dstrs_clm_draft.sas7bdat

43. How 2 add an acl on a linux box========================================================================================1) Sudo to t2runp on plgrid-ugc04 -dcc2) cd /appl/t2prod -dcc 3) mkdir -p /appl/t2prod/mfcw/stage -dcc4) setfacl -m u:a0prod:rwx /appl/t2prod/mfcw/stage –dcc

44. =================================================================================================================Find if a server name is in the name serversudo-sh[badftpp@pwh-sas17lp2]$ nslookup pwh-sas17lp2Server: 158.137.218.75Address: 158.137.218.75#53

Name: pwh-sas17lp2.fanniemae.comAddress: 10.204.64.27

sudo-sh[badftpp@pwh-sas17lp2]$

45. Add one of yours! =================================================================================================================


PRAQNIQUES

=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

=================================================================================================================BELOW ARE CHRONOLOGICAL REFERENCES - (search in line number (LIKE: 38.) to see if there is more explanation) =================================================================================================================

//////////////////////////////////////////sudo-sh[lwftpp@pwarehouse-rsas17lp1]$ date; pwd; ls -R | grep ':$' | sed -e 's/:$//' -e 's/[^-][^\/]*\//|--/g' -e 's/^/ /'Tue Jun 7 13:09:17 EDT 2011/export/appl/lw_data_d/prod |--lwprod |--|--lfm |--|--|--lfm_201103 |--|--|--|--basedata |--|--|--|--inputs |--|--|--|--outdata |--|--|--lfm_201104 |--|--|--|--basedata |--|--|--|--inputs |--|--|--|--logs |--|--|--|--|--saslog

=================================================================================================================6/24/11 From Karuna Nidhi Chelmella(show dir sizes/usages)

sudo-sh[badftpp@tcredit01]$ pwd/appl/fa_data_t/cw_0/input/interestRatesudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/* 12G 2.2.CW/grp1


PRAQNIQUES

14G 2.2.CW/grp2 9.8G 2.2.CW/grp3 48G 2.2.CW/grp4 76G 2.2.CW/natsudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/nat/*|grep 201106 3.1G 2.2.CW/nat/201106 0K 2.2.CW/nat/201106IS 0K 2.2.CW/nat/201106IS_binary_suse 0K 2.2.CW/nat/201106IS_binary_unixsudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/nat/*|grep 201105 5.7G 2.2.CW/nat/201105 746M 2.2.CW/nat/201105IS 670M 2.2.CW/nat/201105IS_binary_suse 670M 2.2.CW/nat/201105IS_binary_unixsudo-sh[badftpp@tcredit01]$ du -khsa /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/* 12G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp1 14G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp2 9.8G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp3 48G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp4 76G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/natsudo-sh[badftpp@tcredit01]$

=================================================================================================================09. 6/10/11 From John Simon...

Clarke, Dale C [5:12 PM]: John is there a central log for weekend reboots that are public?Simon, John V [5:13 PM]: Autosys: http://techops/cgi-bin/autosys/reboot_jobs.pl?format=1Clarke, Dale C [5:13 PM]: thxSimon, John V [5:13 PM]: VCS: http://uxweb/Policies/FailoverSchedules/VCS-Failover-Schedule.htmClarke, Dale C [5:15 PM]:

=================================================================================================================6/5/11 From John Simon...


PRAQNIQUES

How to see if it has direct login - m5prod

psysadm50% ldaplist -l passwd m5prod |grep userPassword userPassword: {crypt}*NOLOGIN*//////////See if a user id has dierect native login.

ldaplist -l passwd <userid> |grep userPassword

For example, if direct login is disabled it will look like this:

psysadm04% ldaplist -l passwd vnothp |grep userPassword userPassword: {crypt}*NOLOGIN*

If direct login is NOT disabled there will be an encrypted string:

psysadm04% ldaplist -l passwd g2ujas |grep userPassword userPassword: {crypt}CMvrgwerav49sj//////////=================================================================================================================6/5/11 From John Simon...

Quick question... Do u use putty? If so is there a NO TMOUT type command like the sudo Answer: sudo -H -u m5prod sudo-sh export TMOUT=0 command=================================================================================================================6/3/11 From John Simon...

See if a user id has dierect native login.

ldaplist -l passwd <userid> |grep userPassword

For example, if direct login is disabled it will look like this:


PRAQNIQUES

psysadm04% ldaplist -l passwd vnothp |grep userPassword userPassword: {crypt}*NOLOGIN*

If direct login is NOT disabled there will be an encrypted string:

psysadm04% ldaplist -l passwd g2ujas |grep userPassword userPassword: {crypt}CMvrgwerav49sj

5/30/11=================================================================================================================Dale,

These answers are pretty straight forward. 1. Q: What is the AD access that should requested here?ANSWER: since you have to write to the folder, you will need, Read/Write (-RW, .FC, etc) 2.1. Q: From the listing Karuna sent how would I know the access needed; ANSWER: Again pretty easy to figure out: The security group aligns EXACTLY up with the AD PATH - Since the AD PAth is: .... fanniemae.com\corp\DC\Shared\Finance\SFFinanceBAD\HPF This means the associated security group would be: SG-DC-Shared-Finance-SFFinanceBAD* NOTE: the SG in front means "Security Group"While in my prifile I requested to add access to this path and pulled up the following: - SG-DC-SHARED-FINANCE-SFFINANCEBAD-FC2.2 Q: i.e. what alias?ANSWER: The alias that is required is already in the HPFA Phase 2 runbook: > cd SFFINANCEBAD/HPFThis means that you can infer the alias is: SFFINANCEBAD and translates to: \\fanniemae.com\corp\DC\Shared\Finance\SFFinanceBAD3. Q: Should we all request it?ANSWER: Yes, at least ALL Engineers

5/24/11=================================================================================================================From: Simon, John V Sent: Wednesday, May 18, 2011 11:04 AMTo: Chen, Jennifer; Hadavi, Behrooz; Elsinger, James A; Milner, Christopher (Contractor)Cc: Mahensaria, Udit (Contractor); Karayi, Sanjith K


PRAQNIQUES

Subject: RE: HPFA Production Security with SFTP

Here is an example of how to generate, convert and run this. In this case it was UNIX to UNIX. The only difference for AD would be to take the converted public key and have it added to the user you are logging into on the AD side.:

$ pwd/tmp/g2ujas$ ssh-keygenGenerating public/private rsa key pair.Enter file in which to save the key (/home/g2ujas/.ssh/id_rsa): /tmp/g2ujas/id_rsaEnter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /tmp/g2ujas/id_rsa.Your public key has been saved in /tmp/g2ujas/id_rsa.pub.The key fingerprint is:85:ef:52:b1:5e:72:bc:bd:01:93:7f:65:8b:e1:c6:ed g2ujas@pstate-etl04$ ls -latotal 64drwxr-xr-x 2 g2ujas admstaff 247 May 18 10:52 .drwxrwxrwt 7 root sys 1912 May 18 10:51 ..-rw------- 1 g2ujas admstaff 1675 May 18 10:52 id_rsa-rw-r--r-- 1 g2ujas admstaff 401 May 18 10:52 id_rsa.pub$ ssh-keygen -e -f id_rsa.pub > fsecure.pub$ cat fsecure.pub---- BEGIN SSH2 PUBLIC KEY ----Comment: "2048-bit RSA, converted from OpenSSH by g2ujas@pstate-etl04"AAAAB3NzaC1yc2EAAAABIwAAAQEA0JntIWU2eC4WhoP6ES/hzpe/Js5lHpilh0JH3EDvt9YKxP5c3oXalzGZyxgG1o7cQlWfPtKB4hcNOEnwCvrkm0ZuceLbDpvDO2MakCvtVcmIeLa9xFhZ0A/apTc0wSPiJ5fJUdZj+c6lmI9OQWxcpOUYFDdjtPRXlNnPjt3xVlDyxm0RWz0PQHbx3ff1hhHXrKMhH+YE2O0OdvTzsMZ0FKpExGubtK0Wx3wAkB/S0Ns4KpVxq3wtUvqpg7U01OYuyVN25bzkEgpAyRhFNO1woaLymD7HKfVrhuVx/cwt3PbOsnJsbV8yZSlu2ZiEiDztcXUoqoWZow1mfmfVNYOZmw==---- END SSH2 PUBLIC KEY ----$ cat id_rsa.pub


PRAQNIQUES

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0JntIWU2eC4WhoP6ES/hzpe/Js5lHpilh0JH3EDvt9YKxP5c3oXalzGZyxgG1o7cQlWfPtKB4hcNOEnwCvrkm0ZuceLbDpvDO2MakCvtVcmIeLa9xFhZ0A/apTc0wSPiJ5fJUdZj+c6lmI9OQWxcpOUYFDdjtPRXlNnPjt3xVlDyxm0RWz0PQHbx3ff1hhHXrKMhH+YE2O0OdvTzsMZ0FKpExGubtK0Wx3wAkB/S0Ns4KpVxq3wtUvqpg7U01OYuyVN25bzkEgpAyRhFNO1woaLymD7HKfVrhuVx/cwt3PbOsnJsbV8yZSlu2ZiEiDztcXUoqoWZow1mfmfVNYOZmw== g2ujas@pstate-etl04$ ls -latotal 96drwxr-xr-x 2 g2ujas admstaff 449 May 18 10:53 .drwxrwxrwt 7 root sys 1912 May 18 10:53 ..-rw-r--r-- 1 g2ujas admstaff 0 May 18 10:52 commercial.pub-rw-r--r-- 1 g2ujas admstaff 75 May 18 10:56 config-rw-r--r-- 1 g2ujas admstaff 511 May 18 10:52 fsecure.pub-rw------- 1 g2ujas admstaff 1675 May 18 10:52 id_rsa-rw-r--r-- 1 g2ujas admstaff 401 May 18 10:52 id_rsa.pub$ cat configIdentityFile /tmp/g2ujas/id_rsaUserKnownHostsFile /tmp/g2ujas/known_hosts$ sftp -F /tmp/g2ujas/config g2ujas@dsysadm15Connecting to dsysadm15...Could not create directory '/home/g2ujas/.ssh'.The authenticity of host 'dsysadm15 (10.210.68.15)' can't be established.RSA key fingerprint is f2:ec:6d:6b:13:2a:3f:ec:9a:85:54:34:2a:d2:cb:79.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'dsysadm15,10.210.68.15' (RSA) to the list of known hosts.sftp> ls -ladrwxrwxr-x 23 g2ujas admstaff 2048 Apr 18 16:54 .dr-xr-xr-x 31 root root 31 May 18 10:55 ..drwxr-xr-x 3 g2ujas admstaff 96 Jan 14 2008 .NetscapeCalendar-rw------- 1 g2ujas admstaff 594 Jun 21 2008 .TTauthority-rw------- 1 g2ujas admstaff 55 Aug 4 2010 .Xauthority-rw-r--r-- 1 g2ujas admstaff 451 Sep 2 2004 .Xdefaultsdrwx------ 2 g2ujas admstaff 96 Oct 25 2004 .acrobat-rw------- 1 g2ujas admstaff 237 Oct 25 2004 .acrosrchdrwx------ 2 g2ujas admstaff 96 Oct 25 2004 .adobe-rw------- 1 root other 10479 Apr 7 17:03 .bash_history-rw-r--r-- 1 g2ujas admstaff 859 Mar 29 2010 .cshrc-rw-r--r-- 1 g2ujas admstaff 1731 Nov 24 2004 .desksetdefaultsdrwxr-xr-x 14 g2ujas admstaff 1024 Jun 25 2008 .dt


PRAQNIQUES

5/12/11=================================================================================================================_____________________________________________From: Perera, Anthony H Sent: Thursday, May 12, 2011 4:38 PMTo: Chelmella, Karuna Nidhi; Clarke, Dale CCc: Elsinger, James A; BA-D OperationsSubject: How to check diskspace on a server

<k2uahp@pcredit-sas01lp2> 21: lsvgrootvgDG01_T1DG01_T2DG01_T3oscopyvg1oscopyvg2<k2uahp@pcredit-sas01lp2> 22:

<k2uahp@ccredit-sas01lp2> 25: lsvg DG01_T2| grep FREEMAX LVs: 4096 FREE PPs: 780 (49920 megabytes)<k2uahp@ccredit-sas01lp2> 26:

21. ASD install issues 4/22/11=================================================================================================================From: Beutler, Joshua (Contractor) Sent: Friday, April 22, 2011 11:20 AMTo: Clarke, Dale CSubject: ASD install issues

Dale,

If you have any other install issues relating to drive space, try this fix and see if it helps.Go to Start > Settings > Control Panel > System Management > Advanced Tab.Increase the size of the temporary program download folder.If you have any other questions please feel free to contact the 7s.


PRAQNIQUES

Thanks!

JoshCSC

4/21/11 ROM Flash for 8440p?=================================================================================================================HP Notebook System BIOS Update (ROM Family 68CCU)From: Teates, Mike B (Contractor) Sent: Thursday, April 21, 2011 2:16 PMTo: Teates, Mike B (Contractor); Clarke, Dale CSubject: Conversation with Teates, Mike B (Contractor), Clarke, Dale C

Teates, Mike B (Contractor) [2:09 PM]: http://bizsupport1.austin.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=321957&prodSeriesId=4095872&swItem=ob-85486-1&prodNameId=4095873&swEnvOID=4060&swLang=13&taskId=135&mode=4&idx=3

20. 3/16/11 Creation of a lockdown doc and ticketxxx=================================================================================================================From: Elsinger, James A Sent: Wednesday, March 16, 2011 2:02 AMTo: Madduri, Anuradha X (Contractor); Jones, Annette XCc: Rogensues, Gary (Contractor); Perera, Anthony H; Clarke, Dale CSubject: Plan for RPS Production environment setup and LOCKDOWNImportance: High

Anu/Annette,

Can one of you please submit an OPERATIONAL, LOW Risk Change ticket to support a Production Lockdown of the RPS production environment. Please plan the lockdown for 3/17 0800 -7/231700. Attached is the DRAFT version of the full lockdown instructions

The full path of the RPS buildout in is the Draft Runbook. However, below is the full list:


PRAQNIQUES

Server: pcredit-sas01lp2/sastemp_rps1_p SAS Temp* 1 250 rpsprod rpsprod 1750/export/appl/rpsprod Code/Scripts 2 10 rpsprod rpsprod 755/export/appl/rpsprod/bin Code/Scripts 2 10 vbadfp rpsprod 750/export/appl/rps_data1_p Data Space 2 500 rpsprod rpsprod 750/export/appl/rps_data2_p Data Space 2 250 rpsprod rpsprod 750/export/appl/rps_archive1_p Archive Space 3 500 rpsprod rpsprod 750

Long window Justification: Additional time is needed to support a pre-production shakeout of this new EUC.Low Risk Justification: There is currently no executions occurring in this new RPS production area.

The details of the ticket should be the information noted in the DRAFT Production Runbook that related to the EUC setup; mkdirs, transferring ANY staged data, the setting up of Oracle interface as well as review of FS and Directory Ownership and Permissions to align with BADOps standard practices.

Gary,

Please verify the rpsprod SUDO is working properly on the production server.

Elsinger, James ABA&D Operations Manager Analytics, Finance & Controls(202) 520-1889

vvvvvvvv=========================================================================================================

Operational Remedy Change Ticket - Detailed Description:===========================================

Purpose: This Operational Change ticket is to prepare the <EUC> environment for controlled process by managing ownership and permissions. This ticket is also to verify and ensure the input data for <EUC> can be successfully read or transferred from interfaces servers in associated SLA’s and/or the server list as noted below for current month data or setup of historical data.

LONG WINDOW JUSTIFICATION: This lockdown process is scheduled to run into a weekend and could therefore take 5-6 days.

LOW RISK JUSTIFICATION: This change does not require downtime or reboot. No user sessions or applications are impacted.


PRAQNIQUES

Hosts affected:================pcredit-sas01lp2,<Additional Servers see Final Draft Runbook>

Please verify you can sudo to both the OPS Transfer ID as well as the EUC Prod Execution ID <

SETUP INSTRUCTIONS===============================================

- As necessary, make sure the Execution Engineer (EE) is logged into pcredit-sas01lp2 and sudo-ed as necessary to RPS Prod Execution ID: rpsprod

Alternate: sudo-ed to OPS ID <badftpp> to support file transfers…

Verification:

NON PROD: > sftp <EE ID>@dcredit-sas01lp1 PROD: > sftp <EE ID>@ pcredit-sas01lp2

1.) Build structure

Create the following sub-folder in <EUC> AD structure: Example: \\fanniemae.com\corp\dc\shared\RE07\PROD\GAGOValSingleFam\HPFA\2009Q3_200912\Data

Create the FS and Directory Structure (SEE Runbook line ??

Example:

As necessary, make sure that the Execution Engineer is logged into pwh-sas17lp2 and sudo-ed to

> login /usr/SYSADM/bin/ssh to pwh-sas17lp2<using Execution Engineers own ID>> sudo -H -u badftpp sudo-sh > mkdir /export/appl/<APPLCODE>_data_p/input> mkdir /export/appl/<APPLCODE>_data_p/output


PRAQNIQUES

> mkdir /export/appl/<APPLCODE>_data_p/input/> mkdir /export/appl/<APPLCODE>_data_p/output/200912> quit

2.) Transfer all “One Time” Staged data (SEE Runbook line ?? <If Applicable>3.) Adjust / Configure permissions and ownership for both Data and Code areas to ensure structure and data files are in lockdown.

- Ensure the permissions and ownership are correct as they are noted in the Infrastructure REQUIREMENTS document

LOCKDOWN INSTRUCTIONS===================================

Parameter: <APPLCODE> = “rps”

***3a.) CODE

- In General: Code area Settings: ID = vbadfp, Group - <APPLCODE>prod

Instructions====================

Log in to pwh-sas17lp2 using CM ID (vbadfp) and cd /export/appl/<APPLCODE>prod

Ensure CM ID owns all code (vbadfp)

- Code transfers should be completed with CM ID

NOTE: Please change the group ownership of code from faptcm (IF present) to EUC UNIX group to "<APPLCODE>prod" on Production and <APPLCODE>acpt on Acceptance of the code areas to be in line with standard practices. <--CONFIRMED (listing in the Activity tab)

- chgroup as necessary to <APPLCODE>prod permissions - 750 (ID = RW, Group should be Read Only, No universal

- NOTE: If there was any permission or ownership issues, pass to SCM Team to review the deployment script and make these adjustments as well.

Code Area Verification:


PRAQNIQUES

- Once complete with vbadfp owner ls -laR /export/appl/<APPLCODE>prod/<code area>o Example: /export/appl/rpsprod/bin

* The code directory should only be 750 Recursive and NOT contain Universal Read/Write or Execute.

- cd to path /export/appl/<APPLCODE>prod

- all sub directory permissions should be 750 unless otherwise noted.

***3b.) DATA

- In General: o Input data file Settings: ID = OPS Transfer ID (badftpp), chgroup should be <APPLCODE>prod or <APPLCODE>acpt in acceptance permissions – 640o Output and result data file Settings: ID = OPS EUC Execution ID (rpsprod), chgroup should be <APPLCODE>prod or <APPLCODE>acpt in acceptance permissions – 640

Instructions====================

Log in to pwh-sas17lp2 using rpsprod and cd /export/appl/<APPLCODE>_data_p or <APPLCODE>_data_a in acceptance

Data Area Verification:

- Ownership of the data input directory should be the OPS ID (badftpp). If OPS ID is not the owner of the input directory, sudo to the ID that owns the directory and make a ".d" copy of the current one (ex. d20090928) then exit out of sudo and sudo into the OPS ID. Once sudoed with badftpp, mkdir a new inputs directory and chgrp to <APPLCODE>prod NOTE Ensure you copy all previous files in the backups directory back to the new inputs directory.

- Once complete ls -ltr (ls -laR?) while in path /export/appl/<APPLCODE>_data_p/input/200909 - Permissions should be 640 for all files, same should be for output as well- cd to path /export/appl/<APPLCODE>_data_p - FS permissions on ALL data directories should be 750 - Ownership of the data input directory should be the OPS ID (badftpp) with group-<APPLCODE>prod. - Owner of output directory should be the Appl ID (<APPLCODE>prod) with group = <APPLCODE>prod


PRAQNIQUES

4.) Confirm accessibility of the stages input files in both rps_curr and prev_mth

BADOps will confirm that all of the required input files can be transferred:

*** Ensure that the files can be read Permissions 644 or group readable and available for transfer during transfers.

4a.) As necessary confirm data can be written to the locked down input directory by verifying the availability and successful transfer of the Input Data. Please consult BADOps Manager for step 3a and 3b

Unix to Unix <See pre-execution Runbook steps>

NOTE EE will drop a foo test file as necessary on UNIX file destination path

Unix to AD < If applicable> <See pre-execution Runbook steps>

NOTE Alias for the AD path will be required and needs to be verified. EE will drop a foo test file as necessary

4b). if necessary: Save a copy of the current input file

3c) Assure access and that the Unix and AD accesses are correct through all lines of the Runbook. Due to the aggressive schedule this may not be possible until the execution. Mitigation should then be recorded in the Errata column of the Runbook.

The validating of the transfer lines and will take 4-6 hours or more

4) Create/verify the execution code for all the transfers to be done and add them to the final draft runbook. This is VERY resource intensive. Due to the aggressive schedule this may not be possible until the execution. Code development should then be recorded in the Errata column of the runbook

*** Ensure that the files can be read (Permissions 640 or group readable) and available for transfer during executions.

5.) Notification:


PRAQNIQUES

Once all validation steps are complete, BADOps Rep will notify CreditSupp Manager (Jim Elsinger) and BADOps Mailbox. - OPS to coordinate with RPS as well as input source systems and Downstream systems like FAVM Business

NOTE: Keep this change ticket open until consumer has verified the data and structure is correct.

Shakeout:========ls -la <filename>

Backout:=======rm <filename>

Test / Verification Result:============All verification results and adjustments will be noted in the worklog. Also all adjust to Runbook steps will be noted in the Runbook Errata Column.

^^^^^^==================================================================================================================================================================================================================================

Please note, CORELOGICCLIENT CONTACT LOGIN NAME: P455W0RD

Fannie Mae James Elsinger jelsinger u85EGJ9a


PRAQNIQUES

Fannie Mae Dale C Clarke dclarke HJKcXH96

Fannie Mae Anthony H Perera aperera 85qq9WAe

Fannie Mae Gary Rogensues grogensues bL5SqtSk

From Anthony 6/17/11=================================================================================================================

Get a LUN or SAN free space

<s8ujsm@pcredit-sas01lp3> 1: lsvgrootvgoscopyvg1DG01_T1DG01_T0DG01_T3DG01_T2oscopyvg2<s8ujsm@pcredit-sas01lp3> 2: lsvg DG01_T2 | grep FREEMAX LVs: 4096 FREE PPs: 1010 (64640 megabytes)

^^^^^^=================================================================================================================From Dale Password ldap check password must be on solaris xcredit01Key is ldapdcc

login as: g8udoc


PRAQNIQUES

g8udoc@dcredit01's password:Last login: Fri Jul 8 12:33:21 2011 from 10.136.197.253FANNIE MAE MANAGEMENT APPROVED PURPOSES ONLY**** SYSTEM ACTIVITIES MAY BE MONITORED ****

<g8udoc@dcredit01> 2: uname -aSunOS dcredit01 5.10 Generic_142900-03 sun4u sparc SUNW,Netra-T12<g8udoc@dcredit01> 3: bashbash-3.00$ ldaplist -l passwd auqprod |grep userPassword userPassword: {crypt}*NOLOGININITIAL*bash-3.00$ ldaplist -l passwd badftpp |grep userPassword userPassword: {crypt}YM2U7.ddJ8R5gbash-3.00$=================================================================================================================17. MD5 perl for files hash

sudo-sh[badftpp@pwarehouse-rsas17lp1]$ csum -h MD5 ./md5sum.tst00ba45f9cbd9dba978eea6ce97dfd82a ./md5sum.tstsudo-sh[badftpp@pwarehouse-rsas17lp1]$ ./md5sum.pl ./md5sum.tst00ba45f9cbd9dba978eea6ce97dfd82a ./md5sum.tstsudo-sh[badftpp@pwarehouse-rsas17lp1]$ cat ./md5sum.tstsomethingsomethingsudo-sh[badftpp@pwarehouse-rsas17lp1]$ cat ./md5sum.pl#! /usr/bin/perl

use Digest::MD5; use IO::File;

my $chk = Digest::MD5->new();

foreach my $file (@ARGV) { $chk->addfile(IO::File->new($file));

print "",$chk->hexdigest," $file \n"; }


PRAQNIQUES

sudo-sh[badftpp@pwarehouse-rsas17lp1]$=================================================================================================================18. Here are the command lines to test SAS -

1. Running with auqprod UID

> login /usr/bin/ssh to cwh-sas17lp2> sudo -H -u auqprod sudo-sh> cd /export/appl/auqprod> /export/apps/sas_9.1/sas -noterminal -is 'proc setinit; run ;' /dev/null -log /tmp/test.log> cat /tmp/test.log> rm -f /tmp/test.log

2. Running with any other UID

> login /usr/bin/ssh to cwh-sas17lp2> /export/apps/sas_9.1/sas -noterminal -work /tmp -sasuser /tmp -is 'proc setinit; run ;' /dev/null -log /tmp/test.log> cat /tmp/test.log> rm -f /tmp/test.log

a) If you see following error, SAS does not have execute permission

The file access permissions do not allow the specified action..

b) Change SAS command as below if we agree to use contingency SAS installation as it is (not changing to match production) -

/export/apps/sas_9.1.3_sp4/sas

=================================================================================================================19. Eval this from John on 8/26=================================================================================================================sudo-sh[root@pcredit-sas01lp2]# ps -ef |egrep '450720|598016|610452|725028|761996|958636|1126516|1159178|1581270'


PRAQNIQUES

root 450720 508016 0 Aug 14 - 8:56 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_481272/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 483346 610452 0 Aug 19 - 1:05 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 528608 1581270 0 Aug 21 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 573674 450720 0 Aug 14 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 598016 1163346 0 16:46:46 pts/4 0:00 sudo-sh root 610452 1941684 0 Aug 19 - 8:50 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_622246/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 655576 761996 0 Aug 15 - 1:05 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 680144 958636 0 06:00:28 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 725028 442586 0 Aug 23 - 9:05 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1150717/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 761996 393404 0 Aug 15 - 8:39 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_769173/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 958636 1691750 0 06:00:28 - 8:47 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1966601/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 971004 1126516 0 Aug 17 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 1126516 892972 0 Aug 17 - 8:55 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1128165/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 1143014 725028 0 Aug 23 - 1:09 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" badftpp 1159178 1872024 0 15:04:10 pts/3 0:00 sudo-sh root 1581270 995518 0 Aug 21 - 8:46 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete --rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1596486/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 1736720 598016 0 16:54:14 pts/4 0:00 egrep 450720|598016|610452|725028|761996|958636|1126516|1159178|1581270 root 1822966 598016 7 16:54:14 pts/4 0:00 ps -efsudo-sh[root@pcredit-sas01lp2]# kill 450720 483346 528608 573674 610452 655576 680144 725028 761996 958636 971004 1126516 1143014 1581270sudo-sh[root@pcredit-sas01lp2]# ps -ef |egrep '450720|598016|610452|725028|761996|958636|1126516|1159178|1581270' root 442590 598016 1 16:55:20 pts/4 0:00 egrep 450720|598016|610452|725028|761996|958636|1126516|1159178|1581270 root 598016 1163346 1 16:46:46 pts/4 0:00 sudo-sh root 725046 598016 7 16:55:20 pts/4 0:00 ps -ef


PRAQNIQUES

badftpp 1159178 1872024 0 15:04:10 pts/3 0:00 sudo-shsudo-sh[root@pcredit-sas01lp2]# df -k |grep badop/dev/badops 31457280 31450428 1% 100 1% /export/appl/badopssudo-sh[root@pcredit-sas01lp2]#==================================================================================================================================================================================================================================24. sudo-sh[lwftpp@pwh-sas17lp2]$sudo-sh[lwftpp@pwh-sas17lp2]$ date; uname -a ; df -gs /export/appl/lw_data_p3Mon Oct 3 16:33:09 EDT 2011AIX pwh-sas17lp2 3 5 00C371A04C00Filesystem GB blocks Free* %Used Iused %Iused Mounted on/dev/lw_data_p3 500.00 148.06 71% 3641 1% /export/appl/lw_data_p3sudo-sh[lwftpp@pwh-sas17lp2]$=================================================================================================================25. When you get a dialog box when connecting to Outlook the has a User prompt looking like EXSW01NS/<your userID> requesting a password (which it then rejects) try the fillowing.

Change the EXSW01NS/<your userID> to fanniemae/<your userID>, enter your correct password. It should then connect. This works when opening Communicator as well.

This happens most often when your connection has been lost or when changing from wired to wireless but may have other causes.

=================================================================================================================TOOLS:sudo-sh[badftpp@pwarehouse-rsas17lp1]$ pwd; ls -ltr *.ksh/export/appl/badops/tools/devl-rwx------ 1 badftpp badftpp 2706 Oct 12 18:17 ckperlver.ksh-rwx------ 1 badftpp badftpp 2195 Oct 12 18:20 cktoolsize.kshsudo-sh[badftpp@pwarehouse-rsas17lp1]$=================================================================================================================34. grep examples:

SPECIFIC ONE LINERS: ######## Find if there are ANY files in a directory, recursively that contain data: Where file/dir, - not owned or grouped by lwftpp or lwprod - not an laR total line, colon title, or blank ls -laR /export/appl/badops/export|grep -v '[lwftpp|lwprod]'|grep -v total|grep -v ":"|grep -v '^$'|wc


PRAQNIQUES

% Tells if there are ANY non-zero size directories: du /export/appl/badops/export|grep -v '^0'|wc

######## An easy, one liner command to replace a test string in multiple files in a sub-directory using recursion. % grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<chgword>/g' ***CAUTION BACK UP YOUR DIRECTORY BEFORE MAKING THE ERROR YOU CERTAINLY WILL ((-8 OR Use the perl oneliner that provides the option of creating a backup of each file changed replacing the sed: % perl -p -i.bak -e ’s/<oldword>/<chgword>/g’ * Note: In both cases some characters require negation with the backslash (experiment)

GENERAL: % grep -v '^$' - grep OUT blank lines % Search vivek in any case (i.e. case insensitive search): grep -i -w vivek /etc/passwd % Search vivek or raj in any case: grep -E -i -w 'vivek|raj' /etc/passwd % Match Vivek or vivek:grep '[vV]ivek' filename OR grep '[vV][iI][Vv][Ee][kK]' filename for any upper/lower in all positions % You can also match digits (i.e match vivek1 or Vivek2 etc): grep -w '[vV]ivek[0-9]' filename OR You can match two numeric digits (i.e. match foo11, foo12 etc): grep 'foo[0-9][0-9]' filename % You are not limited to digits, you can match at least one letter: grep '[A-Za-z]' filename % Display all the lines containing either a "w" or "n" character: grep [wn] filename % Display any lines starting with a dot and digit: grep '^\.[0-9]' filename % Match a character "v" two times in a line: egrep "v{2}" filename % Match both "col" and "cool": egrep 'co{1,2}l' filename % Match any row of at least three letters 'c': egrep 'c{3,}' filename % Search directories recursively: grep -r 'hello' /home/gigi % The -l option print will only the filename (e.g., /etc/crontab, /etc/group) if the file contains a hit: grep -l root /etc/* Can be used to create an array for a command to process.

CLASS brackets: Within a bracket expression, the name of a character class enclosed in "[:" and ":]" stands for the list of all characters belonging to that class. Standard character class names are: [:alnum:] - Alphanumeric characters. [:alpha:] - Alphabetic characters [:blank:] - Blank characters: space and tab. [:digit:] - Digits: '0 1 2 3 4 5 6 7 8 9'. [:lower:] - Lower-case letters: 'a b c d e f g h i j k l m n o p q r s t u v w x y z'. [:space:] - Space characters: tab, newline, vertical tab, form feed, carriage return, and space. [:upper:] - Upper-case letters: 'A B C D E F G H I J K L M N O P Q R S T U V W X Y Z'.


PRAQNIQUES

In this example match all upper case letters: grep '[:upper:]' filename

WILDCARDS (Dot): You can use the "." for a single character match. In this example match all 3 character word starting with "b" and ending in "t": % grep '\<b.t\>' filename Where, The Dot matches ANY character \< Match the empty string at the beginning of word \> Match the empty string at the end of word. % Print all lines with exactly two characters: grep '^..$' filename

ESCAPING (backslash before a character): % In an IP search all three dots need to be escaped: grep '192\.168\.1\.254' /etc/hosts % The following example will only match an IP address: egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' filename % The following will match word Linux or UNIX in any case: egrep -i '^(linux|unix)' filename % Search for all lines matching '--test--' using -e option. Without -e, grep would attempt to parse '--test--' as a list of options: grep -e '--test--' filename

Anchors: % grep -v '^$' - grep OUT blank lines % Match line only containing foo: grep '^foo$' filename % Find lines ending with word foo: grep 'foo$' filename % Display only lines starting with the word vivek only i.e. do not display vivekgite, vivekg etc: grep -w ^vivek /etc/passwd % %

ls -laR /export/appl/lw_archive_p2|grep -v lwftpp|grep -v lwprod|head grep -v total|grep -v ":"|wc=================================================================================================================35. How to get a server or dns name for and IP address - Dougherty, Kate [11:46 PM]: just run nslookup <IP Address>Example:bash-2.03$ nslookup 10.204.64.21Server: psysadm-dn07.fanniemae.comAddress: 158.137.218.75

Name: pwarehouse-rsas17lp1.fanniemae.com


PRAQNIQUES

Address: 10.204.64.21

bash-2.03$

=================================================================================================================35a. $ cp filename{,.bak} --> In this one-liner filename{,.bak} gets copied filename.bakThis one-liner copies the file named filename to a file named filename.bak. Here is how it works. It uses brace expansion to construct a list of arguments for the cp command. Brace expansion is a mechanism by which arbitrary strings may be generated. In this one-liner filename{,.bak} gets brace expanded to filename filename.bak and puts in place of the brace expression. The command becomes cp filename filename.bak and file gets copied.

=================================================================================================================36. Print out unique lines in a file - Print out unique lines in a file cat LOGS/CMDSRUN.txt| sort -n | uniq |head- Print out unique lines in a file for csv selecting 2nd field cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -nr | uniq |head cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -n | uniq |headSimplified: cut -d',' -f 2 <file> | sort -nu

MORE:bash$ cat list-1 list-2 list-3 | sort | uniq > final.list# Concatenates the list files,# sorts them,# removes duplicate lines,# and finally writes the result to an output file.

bash$ cat testfileThis line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times.

bash$ uniq -c testfile


PRAQNIQUES

1 This line occurs only once. 2 This line occurs twice. 3 This line occurs three times.

bash$ sort testfile | uniq -c | sort -nr 3 This line occurs three times. 2 This line occurs twice. 1 This line occurs only once.

SCRIPT:#!/bin/bash# wf.sh: Crude word frequency analysis on a text file.# This is a more efficient version of the "wf2.sh" script.

# Check for input file on command-line.ARGS=1E_BADARGS=85E_NOFILE=86

if [ $# -ne "$ARGS" ] # Correct number of arguments passed to script?then echo "Usage: `basename $0` filename" exit $E_BADARGSfi

if [ ! -f "$1" ] # Check if file exists.then echo "File \"$1\" does not exist." exit $E_NOFILEfi

######################################################### main ()


PRAQNIQUES

sed -e 's/\.//g' -e 's/\,//g' -e 's/ /\/g' "$1" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr# =========================# Frequency of occurrence

# Filter out periods and commas, and#+ change space between words to linefeed,#+ then shift characters to lowercase, and#+ finally prefix occurrence count and sort numerically.

# Arun Giridhar suggests modifying the above to:# . . . | sort | uniq -c | sort +1 [-f] | sort +0 -nr# This adds a secondary sort key, so instances of#+ equal occurrence are sorted alphabetically.# As he explains it:# "This is effectively a radix sort, first on the#+ least significant column#+ (word or string, optionally case-insensitive)#+ and last on the most significant column (frequency)."## As Frank Wang explains, the above is equivalent to#+ . . . | sort | uniq -c | sort +0 -nr#+ and the following also works:#+ . . . | sort | uniq -c | sort -k1nr -k########################################################

exit 0

bash$ cat testfileThis line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times.


PRAQNIQUES

bash$ ./wf.sh testfile 6 this 6 occurs 6 line 3 times 3 three 2 twice 1 only 1 once

1. cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename

2. It is even possible to specify a linefeed as a delimiter if apropos. The trick is to actually embed a linefeed (RETURN) in the command sequence.

bash$ cut -d' ' -f3,7,19 testfileThis is line 3 of testfile. This is line 7 of testfile. This is line 19 of testfile.

37. =================================================================================================================bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udocCould not create directory '/home/g8udoc/.ssh'.dn: uid=g8udoc,ou=people,ou=corporate,dc=fanniemae,dc=com shadowLastChange: 15555 fmpasswordresetdate: 20120803005933Z userPassword: {crypt}d5Q9tVme3qYb2 fmDepartment: Enterprise Technology Svcs fmdivision: Technology fmbusinessunit: Ent Data Technology fmcostcenter: 485 fmProfileOwner: 1 roomNumber: 2450B


PRAQNIQUES

physicalDeliveryOfficeName: 2h-2n/14 fmManagerLevel: Associates fmservicedate: 02/28/2011 12:00:00 AM EST fmjoblevel: 5 title: Applctn Support Analyst III fmlegalname: Dale C Clarke fmemployeestatus: A fmPrivilegedID: No fmSharedID: No employeeNumber: 07887 employeeType: Employee fmofficelocation: 4000 Wisconsin Avenue fmfloor: Second telephoneNumber: 202-752-3465 shadowMin: -1 shadowFlag: 0 shadowExpire: -1 shadowInactive: -1 shadowMax: -1 shadowWarning: 7 homeDirectory: /home/g8udoc uidNumber: 46496 gidNumber: 5069 gecos: Dale C Clarke loginShell: /bin/csh mailAlternateAddress: [email protected] mailRoutingAddress: [email protected] mail: [email protected] mailHost: exchange.fanniemae.com cn: Dale C Clarke fmmarketroom: No fmpeoplemanager: 0 fmContractorType: Regular fmbusinesscritical: No fmkiosk: No fmmanageruid: g8ujxe fmstatus: Active


PRAQNIQUES

manager: James Elsinger initials: C fmisterminated: No fmregion: dc uid: g8udoc givenName: Dale objectClass: inetorgperson objectClass: organizationalPerson objectClass: person objectClass: top objectClass: fannieMaePerson objectClass: mailRecipient objectClass: posixAccount objectClass: shadowAccount objectClass: account sn: Clarkebash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep UIDCould not create directory '/home/g8udoc/.ssh'.bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep uidCould not create directory '/home/g8udoc/.ssh'.dn: uid=g8udoc,ou=people,ou=corporate,dc=fanniemae,dc=com uidNumber: 46496 fmmanageruid: g8ujxe uid: g8udocbash-2.05b$

38. =================================================================================================================

Find in a directory all occurrences of "paul_obrecht“ and print; then replace all occurrences, then print again to show there are none found.

//////////////////////////////// example of the first file in the print below before it is editedsudo-sh[badftpp@pwh-sas17lp2]$ cat ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES#|gnum|G9999|#|dsesrc|RTI|#|dsesrcsrvr|pwh-sas17lp2|#|dsedest|"M&A RESEARCH"|#|dsedestsrvr|pcredit-sas01lp3|


PRAQNIQUES

#|dsedistro|"paul_obrecht daniel_dausman dean_m_henderson af-c_operations ba-d_operations"|#|dsesubject|"TRANSFERS COMPLETED SUCCESSFULLY"|#|dseerrorsubject|"TRANSFERS FAILED"|#|dsebody|"PLEASE SEE THE ATTACHED FILE FOR COMPLETE TRANSFER DETAILS."|#|dseendnote|"Please contact [email protected], if you have any questions or concerns regarding this transfer."|#|dseattachmentT|TRANSFER.log|#|dseattachmentE|ERROR.log|sudo-sh[badftpp@pwh-sas17lp2]$

//////////////////////////////////////////sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print ß find - found 5 occurrences#|dsedistro|"paul_obrecht daniel_dausman dean_m_henderson af-c_operations ba-d_operations"|./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|./VAL/RTI.d20120913DCCbackup/rtit2300.1_g6941_rti-ldm.listN#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|./VAL/RTI.d20120913DCCbackup/rtit2300.2_g6941_rti-ldm.listN#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|./VAL/RTI.d20120913DCCbackup/rtit2300.3_g6941_rti-ldm.listN#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|./VAL/RTI.d20120913DCCbackup/rtit2300.4_g6941_rti-ldm.listNsudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ | xargs perl -pi -e 's/paul_obrecht/ /g' ßfind and change in place perl commandCan't do inplace edit: ./VAL/RTI.d20120913DCCbackup/ is not a regular file.sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print ß find – found nonesudo-sh[badftpp@pwh-sas17lp2]$ ß None found

//////////////////////////////// example of the first file after editedsudo-sh[badftpp@pwh-sas17lp2]$ cat ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES#|gnum|G9999|#|dsesrc|RTI|#|dsesrcsrvr|pwh-sas17lp2|#|dsedest|"M&A RESEARCH"|#|dsedestsrvr|pcredit-sas01lp3|#|dsedistro|" daniel_dausman dean_m_henderson af-c_operations ba-d_operations"| ß Notice the extra space – could be a replacement name#|dsesubject|"TRANSFERS COMPLETED SUCCESSFULLY"|#|dseerrorsubject|"TRANSFERS FAILED"|#|dsebody|"PLEASE SEE THE ATTACHED FILE FOR COMPLETE TRANSFER DETAILS."|


PRAQNIQUES

#|dseendnote|"Please contact [email protected], if you have any questions or concerns regarding this transfer."|#|dseattachmentT|TRANSFER.log|#|dseattachmentE|ERROR.log|sudo-sh[badftpp@pwh-sas17lp2]$

39. =================================================================================================================GET A NAME FROM A USERID ON UNIX=====================================Solaris:bash-3.00$ ldaplist -l passwd <userID> |grep cn: cn: Dale C Clarkebash-3.00$

AIX:bash-2.05b$ getable -a passwd <userID>g8udoc:*NP*:46496:5069:Dale C Clarke:/home/g8udoc:/bin/csh

Name...........: Dale C ClarkeTitle..........:Employee Type..: EmployeeTelephone No...: 202-752-3465Manager........: James Elsinger

bash-2.05b$

Linux: Not yet found

42. =================================================================================================================50. TRANSFER A FILE WITH SCP THAT HAS A SAS GDG ////////////////////////sudo-sh[ainprod@pwh-sas17lp2]$ cd /export/appl/ain_archive_p/LASD/sudo-sh[ainprod@pwh-sas17lp2]$ ls -iltotal 34331200 18 -rwxr-xr-x 1 ainprod ainprod 1539776512 Jul 02 2010 dstrs_clm_draft.sas7bdat_bad100702 10 -rw-r--r-- 1 ainprod ainprod 2587697152 Aug 03 11:05 lasdbkp#048.sas7bdat 11 -rw-r--r-- 1 ainprod ainprod 2622562304 Sep 03 09:46 lasdbkp#049.sas7bdat 12 -rw-r--r-- 1 ainprod ainprod 2658934784 Oct 03 09:47 lasdbkp#050.sas7bdat


PRAQNIQUES

13 -rw-r--r-- 1 ainprod ainprod 2690850816 Nov 03 09:58 lasdbkp#051.sas7bdat 15 -rw-r--r-- 1 ainprod ainprod 2723553280 Dec 03 09:56 lasdbkp#052.sas7bdat 17 -rw-r--r-- 1 ainprod ainprod 2754224128 Jan 03 08:35 lasdbkp.sas7bdatsudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec cksum {} \;1009243769 2690850816./lasdbkp#051.sas7bdatsudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \;badftpp@dcredit-sas02lp1's password:

sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \;badftpp@dcredit-sas02lp1's password:Could not chdir to home directory /appl/badftpp: A file or directory in the path name does not exist.lasdbkp#051.sas7bdat 100% 2597MB 40.6MB/s 01:04sudo-sh[ainprod@pwh-sas17lp2]$

RENAME ON THE TARGET SERVER:bash-3.2$ cd /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/bash-3.2$ ls -iltotal 525556877921 -rw-r--r-- 1 badftpp badftpp 2690850816 Jan 11 13:24 lasdbkp#051.sas7bdatbash-3.2$ uname -a; pwd;find . -inum 77921 -exec cksum {} \;AIX dcredit-sas02lp1 1 6 00F62A394C00/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD1009243769 2690850816 ./lasdbkp#051.sas7bdatbash-3.2$ find . -inum 77921 -exec mv {} dstrs_clm_draft.sas7bdat bash-3.2$ ls -ltrtotal 5255568-rw-r--r-- 1 badftpp badftpp 2690850816 Jan 11 13:24 dstrs_clm_draft.sas7bdatbash-3.2$ uname -a; pwd;cksum dstrs_clm_draft.sas7bdat \; AIX dcredit-sas02lp1 1 6 00F62A394C00/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD1009243769 2690850816 dstrs_clm_draft.sas7bdatbash-3.2$

43. =================================================================================================================HOW-2 find man pages

sudo-sh[ainprod@pwh-sas17lp2]$ apropos acl


PRAQNIQUES

aclconvert(1) - Converts the access control information of a file system object from onetype to another.acledit(1) - Edits the access control information of a file.aclget(1) - Displays the access control information of a file.aclgettypes(1) - Gets ACL types supported by a file system path.aclput(1) - Sets the access control information of a file.ctaclfck(1) - Verifies the contents of a clustersecurity services ACL file.sudo-sh[ainprod@pwh-sas17lp2]$ apropos aclputaclput(1) - Sets the access control information of a file.sudo-sh[ainprod@pwh-sas17lp2]$ man aclput==================================================================================================================================================================================================================================


Documents

gozonian.orggozonian.org/.../DALECLARKE_KMA_PRACNIQUES_UNI… · Web viewUnix Praqniques (Practical Techniques)by Dale C ClarkePage 1