MasteringModernLinuxSecondEdition

MasteringModernLinuxSecondEdition

PaulS.WangKentStateUniversity,Kent,Ohio

CRCPressTaylor&FrancisGroup6000BrokenSoundParkwayNW,Suite300BocaRaton,FL33487-2742

©2018byTaylor&FrancisGroup,LLCCRCPressisanimprintofTaylor&FrancisGroup,anInformabusiness

NoclaimtooriginalU.S.Governmentworks

Printedonacid-freepaperVersionDate:20180420

InternationalStandardBookNumber-13:978-0-8153-8098-6(Paperback)InternationalStandardBookNumber-13:978-0-8153-8111-2(Hardback)

Thisbookcontains informationobtained fromauthentic andhighly regarded sources.Reasonable effortshave been made to publish reliable data and information, but the author and publisher cannot assumeresponsibilityforthevalidityofallmaterialsortheconsequencesoftheiruse.Theauthorsandpublishershaveattemptedtotracethecopyrightholdersofallmaterialreproducedinthispublicationandapologizetocopyrightholdersifpermissiontopublishinthisformhasnotbeenobtained.Ifanycopyrightmaterialhasnotbeenacknowledgedpleasewriteandletusknowsowemayrectifyinanyfuturereprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,transmitted,orutilizedinanyformbyanyelectronic,mechanical,orothermeans,nowknownorhereafterinvented, includingphotocopying,microfilming,andrecording,or inany informationstorageor retrievalsystem,withoutwrittenpermissionfromthepublishers.

For permission to photocopy or use material electronically from this work, please accesswww.copyright.com(http://www.copyright.com/) or contact theCopyrightClearanceCenter, Inc. (CCC),222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization thatprovides licenses and registration for a variety of users. For organizations that have been granted aphotocopylicensebytheCCC,aseparatesystemofpaymenthasbeenarranged.

TrademarkNotice:Productorcorporatenamesmaybetrademarksorregisteredtrademarks,andareusedonlyforidentificationandexplanationwithoutintenttoinfringe.

LibraryofCongressCataloging-in-PublicationDataNames:Wang,PaulS.,author.Title:MasteringmodernLinux/PaulS.Wang.Othertitles:MasteringLinuxDescription:Secondedition.|BocaRaton:Taylor&Francis,CRCPress,2018.|Revisededitionof:MasteringLinux/PaulS.Wang.2011.|Includesbibliographicalreferencesandindex.Identifiers:LCCN2018008944|ISBN9780815380986(pbk.:alk.paper)|ISBN9780815381112(hardback)Subjects:LCSH:Linux.|Operatingsystems(Computers)Classification:LCCQA76.774.L46W362018|DDC005.4/46--dc23LCrecordavailableathttps://lccn.loc.gov/2018008944

VisittheTaylor&FrancisWebsiteat

http://www.taylorandfrancis.com

andtheCRCPressWebsiteathttp://www.crcpress.com

CHAPTER1■1.11.2

1.3

1.41.5

1.6

1.71.81.91.10

1.111.121.13

1.141.15

ContentsPreface

Introduction

ALinuxPrimerWhatIsanOperatingSystem?GettingStarted:LoginandLogout

DesktopLoginStartingaTerminalWindowRemoteLogin

UnderstandingtheShellEnteringCommandsTryingaFewCommandsCorrectingTypingMistakesAbortingaCommand

ExerciseAUsingFilesandDirectories

CurrentWorkingDirectoryandFilenamesHandlingFilesandDirectoriesStandardPersonalDirectories

ProtectingFiles:AccessControlTheSuperUserExaminingthePermissionSettingsSettingPermissions

ExerciseBTextEditingGettingHard/SavedCopiesCommunicatingwithOthers

Who’sWhoontheSystem:fingerEmail

BrowsingtheWebExerciseCCreatingandRunningYourOwnProgram

CompilingExerciseDConsultingLinuxDocumentation

1.161.171.18

CHAPTER2■2.12.22.32.42.5

2.6

2.7

2.8

2.92.102.11

CHAPTER3■3.13.23.3

ExerciseERoundingUpUsefulCommandsSummary

TheDesktopEnvironmentDesktopOverviewDesktopComponentsTheGNOME3DesktopUnderstandingGUIWorkingwiththeDesktop

SessionControlandSystemSettingsLaunchingApplicationProgramsManagingFilesMultipleWorkspacesDesktopAppearance

WindowsTheXWindowSystemWindowManagerWindowInformation

TheFileBrowserNavigatingtheFileTreeOpeningaFileorFolderFindingFilesManagingFilesandFoldersAccessControlforFilesandFoldersWritingCDsorDVDsChangingYourDefaultFileBrowser

TerminalWindowStartingaGNOMETerminalTerminalWindowandtheShellSelect,Copy,andPasteWebandEmailLinks

AccessingHelpandDocumentationSummaryExercises

InteractingwiththeBASHShellBashInteractingwithBashCommand-LineEditingandCommandCompletion

3.43.5

3.63.7

3.83.93.10

3.11

3.123.133.14

3.153.163.173.18

CHAPTER4■4.1

BashCommandExecutionBashInput/OutputRedirection

StandardInputandOutputI/ORedirectionPipes

BashJobControlBashShellExpansions

HistoryExpansionAliasExpansionBraceandTildeExpansionsVariableExpansionCommandExpansionProcessExpansionFilenameExpansion

BashBuilt-inCommandsShellVariablesEnvironmentofaProgram

CommandExecutionEnvironmentExamplesofBashUsage

CustomizedPromptRemovingFilesSafelyCopy,Paste,andI/ORedirectionDisplayingManualPagesSettingUpYourPersonalWebFolder

DefaultFilePermissionsShellStartupandInitializationShellSpecialCharactersandQuoting

QuotinginBashSimpleFunctionsForMoreInformationSummaryExercises

PuttingCommandsandApplicationstoUseUsefulGUIApps

WordProcessingDocumentFormattingandTypesettingDrawingandDiagrammingRasterGraphicsandImageProcessing

4.2

4.34.4

4.54.64.7

4.84.94.10

CHAPTER5■5.15.25.35.45.55.65.7

5.85.95.105.115.125.13

FileUploadandDownloadPasswordManagerCloudStorage3DModelingand3DPrintingMathematicalCalculations

CommandsandFiltersLeadingandTrailingLines:headandtailCharacterTranslation:trTabExpansionFoldingTextLinesCalendarRemindersbyEmailSortingTextLines

ThegrepCommandRegularExpressions

QuotinginSearchPatternsPatternsforgrepAStreamEditor:sedBuildingPipelines

AddressProcessingForMoreInformationSummaryExercises

WritingBASHScriptsInvokingShellScriptsAFirstShellScriptShellScriptExecutionPositionalParametersTheforCommandTheifCommandTestExpressionsandExitStatus

ExitStatusTestExpressions

TheshiftCommandThecaseCommandThewhileanduntilCommandsNumericalExpressionsThebreakandcontinueCommandsFileQueries

5.145.155.165.175.18

5.195.20

5.215.22

5.235.245.255.26

CHAPTER6■6.16.2

6.3

6.4

VariablesArraysVariableModifiersTheHereDocumentMoreonFunctions

FunctionArgumentsReturnValueofaFunction

RedefiningBashBuilt-inFunctionsExampleBashScripts

Example:RemovingUnwantedFilesExample:ConditionalCopyExample:TotalFileSizesExample:SecureFileTransferExample:ResizingPictures

DebuggingShellScriptsErrorandInterruptHandling

InterruptHandlingThePerlandPHPAlternativesForMoreInformationSummaryExercises

TheFileSystemAFileLocationRoadMapFileTypes

OrdinaryFilesDirectoriesSpecialFilesLinksSymbolicLinks

MoreonFileAccessControlMeaningofPermissionsforaDirectory

FileStatusFileModeFileUseridandGroupidAccessControlEnforcementSetuidandSetgidModesEstablishingaGroupDACandMAC

6.5

6.66.76.86.96.106.11

6.126.136.146.156.16

CHAPTER7■7.17.2

7.37.4

7.57.6

7.77.87.97.10

7.11

7.12

FileSystemImplementationFilesystemOrganizationMountedFilesystemsFilesystemSuperBlockandBlockGroups

TheFilesystemTableCreatingSpecialFilesNetworkFilesystemSearchingtheFileTree:findThelocateCommandSaving,Compressing,andDistributingFiles

PackingFileswithsharFileSharingwithSambaMoreFile-RelatedCommandsForMoreInformationSummaryExercises

Networking,Internet,andtheWebNetworkingProtocolsTheInternet

NetworkAddressesPacketSwitchingClientandServer

TheDomainNameSystemNetworkinginNautilus

AccessingSambaSharedFilesNetworkingCommandsSSHwithX11Forwarding

NoPasswordssh,sftp,andscpRemoteFileSynchronization

CryptographySymmetricCryptosystemsPublic-KeyCryptographyandDigitalSignatureGNUPrivacyGuard

SettingUpGnuPGKeysncryption/DecryptionwithGnuPG

SecureEmailSecureEmailwithThunderbird

MessageDigests

7.13

7.147.157.167.177.18

7.19

7.20

7.217.227.237.247.25

CHAPTER8■8.1

8.2

8.38.48.58.6

8.7

8.8

SoftwareandMessageSigningTheWeb

HypertextMarkupLanguageURLsAccessingInformationontheWeb

HandlingDifferentContentTypesPuttingInformationontheWebWhatIsHTML?WebHostingDomainRegistration

AccessingDomainRegistrationDataTheDNS

DNSServersDNSResolvers

DynamicGenerationofWebPagesDynamicServerPages

HTTPBrieflyARealHTTPExperienceForMoreInformationSummaryExercises

BasicSystemAdministrationManagingUsers

SudoPackageManagement

SoftwareManagementTasksManagingProcessesNetworkConfigurationFirewallSettingsManagingFilesystemsandDisks

DiskPartitionsManagingFilesystemsAddingaNewDiskLVMFileStorageQuotas

FileandSystemBackupBackupwithDéjàDup

SystemBooting

8.9

8.108.118.12

CHAPTER9■9.19.29.39.49.59.6

9.7

9.8

9.9

9.109.11

9.129.139.149.159.16

9.17

SELinuxSELinuxStatusandEnforcingModesSecurityContextsMaintainingandManagingFileContexts

ForMoreInformationSummaryExercises

WebHosting:Apache,MySQL,andPHPWhatIsaWebServer?URLandURIRequestProcessingResponseandContentTypesTheApacheWebServerApacheonLinux

InstallingApachewithPackageManagementRunningtheApacheWebServerControllingtheApacheServer

ApacheRun-TimeConfigurationApacheConfigurationFileBasicsAboutConfigurationDirectivesLoadingModulesGlobalDirectivesContainerDirectives

AccessControlunderApacheWhatIsAccessControl?AccessControlbyHost

RequiringPasswordsSettingUpUserLoginunderApache

HowHTTPBasicAuthenticationWorksHowHTTPDigestAuthenticationWorks

Basicvs.DigestAuthenticationPasswordEncryptionAutomaticFileDeflationHTTPSandSSL/TLSHTTPSSupportManualInstallationofApache

ConfigureandCompileWhatIsPHP?

9.18

9.199.209.219.229.23

9.249.259.269.279.28

CHAPTER10■10.1

10.210.3

10.4

10.510.6

10.7

ThePHPModuleforApacheInstallingthePHPModule

TestingPHPPHPConfigurationPHPCommandLineScriptingDatabaseSupportfortheWebMySQL

Initializing,Starting,andStoppingMySQLMySQLRun-TimeConfigurationAdministeringMySQLResettingtheMySQLRootPassword

InstallingphpMyAdminInstallingMySQL/MariaDBForMoreInformationSummaryExercises

CProgramminginLinuxCommand-LineArguments

ExitStatusCompileandExecute

LinuxCommandArgumentConventionsTheGCCCompiler

ThegccCommandOptionsforgccTheCPreprocessorPreventingMultipleLoadingofHeaderFilesCompilationAssemblyLinkingandLoading

TheCLibraryI/OtoFilesFileUpdatingI/ORedirection

CreatingLibrariesandArchivesErrorHandlinginCPrograms

ErrorsfromSystemandLibraryCallsErrorIndicationsfromMathematicalFunctions

ErrorRecovery

10.8

10.910.1010.1110.12

CHAPTER11■11.111.211.3

11.4

11.5

11.6

11.711.811.9

11.10

11.1111.12

11.1311.1411.15

11.16

DebuggingwithGDBInteractiveDebuggingBasicgdbCommandsASampleDebuggingSessionwithgdb

ExaminingCoreDumpsForMoreInformationSummaryExercises

I/OandProcessControlSystemCallsSystem-LevelI/OI/ODescriptorsReadingandWritingI/ODescriptors

MovingtheCurrentPositionOperationsonFiles

CreatingandDeletingaFileLinkingandRenamingFilesAccessingFileStatusDeterminingAllowableFileAccess

OperationsonDirectoriesCreatingandRemovingaDirectory

DirectoryAccessCurrentWorkingDirectory

AnExample:ccpShell-LevelCommandsfromCProgramsProcessControl

VirtualAddressSpaceProcessLifeCycle

TheProcessTableThepsCommand

ProcessCreation:forkProgramExecution:execRoutines

Example:ASimpleShellSynchronizationofParentandChildProcessesProcessTerminationTheUserEnvironmentofaProcess

Example:CommandSearchInterruptsandSignals

BasicConcepts

11.1711.1811.19

CHAPTER12■12.112.2

12.312.412.5

12.6

12.712.812.9

12.10

12.1112.12

12.1312.1412.1512.1612.17

SendingSignalsSignalDeliveryandProcessingSignalTrapping

ForMoreInformationSummaryExercises

Inter-processandNetworkCommunicationOpeningaProcessforI/OIPCwithpipe

PipebetweenTwoCommandsConnectingaFileDescriptortoaFileStreamTwo-WayPipeConnectionsNetworkCommunication

ClientandServerSockets

CreatingSocketsSocketAddressLocalandInternetSocketAddresses

ATCPEchoClientUsingDatagramSocketsSocketI/OSystemCalls

ShuttingDownSocketsTCP-BasedServers

AcceptingaConnectionAnExampleTCP/IPServer

NetworkLibraryRoutinesDaemonProcesses

ProgrammingaDaemonInput/OutputMultiplexingTCPOut-of-BandDataForMoreInformationSummaryExercises

AppendicesOnline

WebsiteandExampleCodePackage

References

Index

Preface

Linux,agreatsuccessstoryofopen-source,community-developedsoftware, isincreasingly used in Web and application servers, software developmentplatforms,personalworkstations,andresearchmachines.Inthepastfewyears,Linux has improved its user interface, addedmany useful and powerful apps,andgreatlyexpandeditshomeandbusinessuserbase.Incomputerscienceandengineeringdepartments,you’ll findLinuxsystems

inclassrooms,programminglabs,andcomputercenters—notjustbecauseLinuxisfreebutalsobecauseitoffersarichcomputingenvironmentforteachingandlearning.Fromitsbeginningin1991,andwithhelpfromtheGNUProject,Linuxhas

evolved quickly and has brought new powers and conveniences to users.CompetencyonLinuxwillbeimportantforanyseriouscomputerprofessional.ThisbookisarevisededitionofMasteringLinux(late2010),whichwasvery

well received and had the most wonderful review from ACM ComputingReviews:

“This authoritative and exceptionally well-constructed book has myhighest recommendation. It will repay careful and recursive study.—ComputingReviews,August2011”

Theneweditionhasanewtitle,MasteringModernLinux,yetretainedmuchofthegoodmaterialswhileupdatingthem,addingnewtopicsandremovingoldones.Thisbookprovidesacomprehensiveandup-to-dateguidetoLinuxconcepts,

usage, and programming. This text will help you master Linux with a well-selectedsetoftopics.Hands-onpracticeisencouraged;itistheonlywaytogainfamiliarity with an operating system. A primer gets you started quickly. Thechaptersleadyoufromuserinterfaces,commandsandfilters,Shellscripting,thefile system, networking, basic system administration, andWeb hosting, to C-levelprogrammingandkernelsystemcalls.Therearemanyexamplesandcompleteprogramsreadytodownloadandrun.

A summary and exercises of varyingdegrees of difficulty canbe found at the

end of each chapter. A companion website provides appendices, informationupdates,anexamplecodepackage,andotherresourcesforinstructorsaswellasstudents.Seepage353fordetails.

UserFriendlyandComprehensiveThereisbothbreadthanddepthinthisbook’spresentation.Chapter1containsaLinux primer to get the new user started as quickly as possible, withoutawkwardnessorconfusion.Beingable toplayandexperimentwith thesystemadds to the user’s interest andmotivation to learnmore.Once introduced andcomfortable,theuserisguidedthroughawell-selectedsetoftopicscoveringthetypeofdetailedmaterialappropriateforaone-semestercourseat theadvancedundergraduateorbeginninggraduatelevel.ThefirstpartofthetextbookcoversinteractiveuseofLinuxviatheGraphical

User Interface (GUI) and the Command-Line Interface (CLI), includingcomprehensive treatment of the Gnome desktop and the Bash Shell. Usingdifferentapps,commandsandfilters,buildingpipelines,andmatchingpatternswithregularexpressionsaremajorfocuses.Next come Bash scripting, file system structure, organization, and usage,

whichbringustoaboutthemiddleofthebook.The next chapters present networking, the Internet and the Web, data

encryption,andbasicsystemadmin,aswellasWebhosting.TheLinuxApacheMySQL/MariaDBPHP(LAMP)Webhostingcombinationispresentedindepth.SuchpracticalknowledgecanbevaluableformanyLinuxprogrammers.InChapters–12,attentionisthenturnedtoC-levelprogramming.Becausethe

LinuxkernelandmostofitsapplicationsareimplementedinC,itisconsideredthe native language of Linux. In-depth knowledge of Linux requiresunderstanding the Standard C Libraries and the system calls which form theinterface to the Linux kernel. Topics covered include the C compiler,preprocessor, debugger, I/O, file manipulation, process control, inter-processcommunication,andnetworking.Manycompleteexampleprograms,written inthestandardISOC99,areprovided.Appendicesarekeptonthebook’swebsite(mml.sofpower.com).Theysupply

usefulsupplementalinformationforstudents,includingtexteditingandhowtosetupLinuxlearningenvironmentsontheirownWindows®orMac®computers.

FlexibleUsageThisbookisforpeoplewhowishtolearnLinuxandtobecomegoodatusingitandwritingprogramsinit.ThebookdoesnotassumepriorknowledgeofLinux

orUNIX,buthasthedepthtosatisfyeventhosewithLinuxexperience.ComparedtootherLinuxbooks, this text isnota thickvolume.However, it

presentsmanytopicscomprehensivelyandindepth.Manyexamplesaregiventoillustrate concepts and usage. It is well-suited for a one-semester course. Aninstructor can cover all the chapters in sequence or choose among them,dependingontheclassbeingtaught.ForanIntroductiontoLinuxcourse,thechaptersonC-levelprogrammingand

perhapsonWebhostingcanbeskipped.Forasystemprogramming-orientedcourse,theLinuxprimer,interactiveuse

ofBash,andtheGNUdesktopmaterialcanbeomittedorassignedforreadingatthebeginningoftheclass.Thiswillprovidemoretimeforthehardcoretopicsonprogramming.ForanIntroductiontoOperatingSystemPrinciplescourse,thisbookisagood

supplement. Discussion of Linux subjects—the Shell, file system structure,concurrent process management, I/O structure, signals/interrupts, and inter-processcommunication—providesconcreteexamplesandadds to thestudents’understandingoftheabstractoperatingsystemprinciplesbeingstudied.Foraserver-sideWebprogrammingcourse,thecoverageofBash,filesystem,

InternetandtheWeb,andWebhostingcanmakethisbookagreatsupplementaltext.Forcoursesonnetworkprogramming,graphics,Cprogramming,distributed

computing,etc.,thebookcanbeavaluablesupplementaswell.ForthosewhouseLinuxinschooloratwork,thisbookenablesyoutoapply

the system’s capabilities more effectively, resulting in much increasedproductivity.Ready-to-useexamplesprovidemanyimmediatepracticalapplications.Goingbeyond, you can learnhow towrite programs at theShell and theC

levels.This ability enables you to build new capabilities and custom tools forapplicationsorR&D.

ExampleCodePackageThroughout the book, concepts and usages are thoroughly explained withexamples. Insteadofusingcontrivedexamples,however,everyefforthasbeenmade to give examples with practical value and to present them as completeprogramsreadytorunonyourLinuxsystem.Theseprogramsarecollectedinanexamplecodepackagereadytodownload

fromthecompanionwebsite(mml.sofpower.com).Seepage353forinstructionsondownloadingandunpackingtheexamplecodepackage.Thedescriptionfor

eachexampleprogramiscross-referencedtoitsfilelocationwithanotationsuchas(Ex:ex05/argCheck.sh).

EasyReferenceYou’ll find a smooth, readable style uncharacteristic of a book of this type.Nevertheless,itisunderstoodthatsuchbooksareusedasmuchforreferenceasfor concentrated study, especially once the reader gets going on the system.Therefore, information isorganizedandpresented inawaythatalsofacilitatesquickandeasyreference.Thereareampleresourcelistingsandappendices(onthewebsite)andathoroughandcomprehensiveindex.Thein-textexamplesarealsocross-referencedwithfilesintheexamplecodepackage.Thisbookwillbeavaluableaidforanyonewhousestools,accessestheInternet,orwritesprogramsunderLinux,evenoccasionally.

AcknowledgmentsIwould like to thank the editorsRandiCohen,PaulBoyd, andothers atCRCpressfor theirhelpandguidancethroughout thewritingandproductionof thissecondedition.Theirdedicationisindeedmuchappreciated.During the planning and writing of this book, several reviews have been

conducted.SpecialthanksgotoProfessorCharlesBorder,RochesterInstituteofTechnology,NY,whomadeexcellentsuggestionsforimprovingthemanuscriptandmakingthebookeasiertouseforstudentsandteachers,includingtheideatoprovidestudentswithinformationtoeasilysetupLinuxlearningenvironmentsontheirowncomputers.Finally, Iwant to expressmy sincere gratitude tomywife, Jennifer,whose

supportandencouragementhavebeensoimportanttomethroughtheyears.

PaulS.Wangwww.cs.kent.edu/pwang

Introduction

Eversinceitsintroductionintheearly1990s,Linuxhasevolved,improved,andsignificantly expanded its user base. It has become an important factor inmoderncomputing.Linux is a free and open-source operating system that works, in many

respects, just like UNIX. Linux became popular as a widely preferred serverplatformforWebhosting,cloudcomputing,andotherpurposes.However,withthe introductionof theGNOMEandKDEdesktopuser interfaceenvironments(and othermore recent ones), plusmany improvements and new applications,Linux has been gaining ground as a home/office system, as well as a moredominantforceintheservermarket.Becauseitisfreeandopensource,1Linuxisaveryattractiveteachingtoolfor

computer science and engineering departments. Also, because it is fast andreliable,businesses,suchasAmazon,GoogleandFacebook,oftenchooseLinuxto run their Web and application servers. Companies and developercommunities, in the United States and worldwide, contribute to kerneldevelopment,newproducts,personneltraining,andtechnicalsupportforLinux,whiletheoperatingsystemitselfremainsfree.Let’s takeabrief lookat thehistoryofLinux, itsversionsandfeatures,and

thetopicsinvolvedinlearninghowtouseLinux.

ABriefHistoryofLinuxThe beginning of Linux can be traced back to 1991 when Linus Torvalds, ayoung student at the University of Helsinki, Finland, began to create a newPOSIX2compliantkernelandanoperatingsystemmorepowerfulthanMINIX(MIni-uNIX).3Threeyearslater,version1.0oftheLinuxkernel,thecentralpartofthenewUNIX-likesystem,wasreleased.The GNU open-source software movement would also later make many

contributionstoLinux,asremarkeduponbyRichardStallman:

“When you are talking about Linux as aOS, you should refer to it asGNU/Linux.Linux is just thekernel.All the tools thatmakeLinuxanOShave been contributed by GNU movement and hence the name

GNU/Linux."

Linuxhascomealongwaysinceitsearlydays.Today,itisaprimeexampleofthesuccessofopen-source,community-developedsoftware.Linuxisusedonservers,desktopcomputers,laptops,andnetbooks.ThehighlysuccessfulmobilesystemAndroid(version1.02008)deriveditskerneldirectlyfromLinux.ThearticleFedoraproject leaderMatthewMillertalksworlddominationon

Linux’s25thbirthday(PCWorld08/25/2016)says:

“In many ways, we’ve actually reached the fabled ‘world domination’everyone joked about 20 years ago," says Miller. “Linux is the defaultoperatingsystemformostthings...AndroidputsLinuxat theheartof themost common consumer operating system in the world. Open source, tosomedegreeoranother,isnowthedefaultlicensingmodel."

LinuxVersionsUnlike proprietary operating systems, Linux is a combination of open-sourceprograms, including the Linux kernel, GNU tools, desktop managers, andinstallationandpackagemanagementsystems,plusmanyothersystem-,server-,anduser-level applications.Anyone can create different combinations of thesecomponents, perhaps also change or improve them, and create a Linuxdistribution with unique characteristics. Thus, it is not surprising that manycompanies and groups all over the world have been distributing somewhatdifferentversionsofLinuxreadytoinstallonyourcomputer.Linux systems are widely used by individuals, academic institutions,

corporations, and serviceproviders suchasWebhosts,datacenters, andcloudservers.WidelyusedLinuxversionsinclude

Ubuntu—“Ubuntu”means“humanity” inZulu.UbuntuLinuxstartedasaversionof thepopularDebianGNU/Linux.AllversionsofUbuntuLinuxarefree,andthereisnochargeformailingaCDtoyou.UbuntusupportstheGNOMEDesktop environment,while another version,Kubuntu, usestheKDEDesktop.Ubuntu iseasy to install andveryuser friendly,whichhasquicklymadeitthemostpopularversionofLinux.Ubuntuissponsoredby theU.K.-based Canonical Ltd., owned by SouthAfrican entrepreneurMarkShuttleworth.RedHatEnterpriseLinux—TheoriginalRedHatLinuxstartedin1994andwasdiscontinuedbyRedHat Inc. in2004.Thecompanynowfocuseson

Red Hat Enterprise Linux (RHEL) for business environments and onFedoraasacommunity-supportedsoftwareprojectforhome,personal,andeducationaluse.CentOS—RHELlargelyconsistsoffreeandopen-sourcesoftware,buttheexecutables are made available only to paying subscribers. CentOS(CommunityENTerpriseOperatingSystem)isacompletelyfreeversionofRHEL(minustheRedHatlogos)madeavailabletousersasnewversionsofRHELarereleased.Fedora—Fedora is a leading-edge Linux distribution where new featuresand improvements are tested before being included in RHEL/CentOS.Fedoramakes frequent softwareupdatesand is tightly integratedwith theGNOMEuserenvironment.openSUSE—ThisisamajorretailLinuxdistributionsupportedworldwidebyNovell (nowpartofMicroFocus).Novellacquired theSuSELinux(aGerman translation of the original Slackware Linux) in 2004. In thefollowingyear,NovelldecidedtomaketheSUSEProfessionalseriesmoreopen as a community-developed, open-source software and to rename itopenSUSE.Debian—DebianLinuxconsistsentirelyoffreeandopen-sourcesoftware.Its primary form,DebianGNU/Linux, is a popular and influential Linuxdistribution.Debianisknownforanabundanceofoptions.Recentreleasesincludeover26,000softwarepackagesforallmajorcomputerarchitectures.UbuntuisaderivativeofDebian.Mint—Linux Mint, a newcomer, is a reliable and popular desktopdistribution. It adopts a conservative approach to software updates and isbasedonDebianandUbuntu.Raspbian—Based on Debian, Raspbian is Linux optimized for theRaspberry Pi, a credit-card-sized computer for education as well aspracticaluses.

TherearemanykindsofLinuxdistributions fordesktopcomputers, servers,and mobile devices, as well as embedded systems. The Linux Standard Base(LSB)isaneffort,throughtheLinuxFoundation,tostandardizemanyaspectsofLinux distributions, and is a superset of the POSIX standards. Major Linuxdistributions follow LSB. This textbook addresses features common to mostLinuxsystemsandindicatesimportantdifferenceswhereappropriate.

TheUNIX/LinuxPhilosophy:SmallIsBeautifulThe UNIX philosophy influenced not just the original operating system

developed by Ken Thompson at Bell Labs (1969), but also the many UNIXclonesandUNIX-likesystemscreatedafterward.Taken together, theseUNIX-likesystemsaresomeoftheverybestoperatingsystemsdevelopedtodate.The generally agreed-upon central tenants of the UNIX Philosophy can be

listedas

Keepprogramssmall—Writeaprogramtodoonewell-definedtask;doitefficiently,anddoitwell.Avoid verbosity—Perform no unessential output from any programs; useshortnamesforcommandsandcommandoptions.Make programs modular—Build small, independent, and self-sufficientprogramparts,witheachservingaspecific function.Theseprogrampartscan be flexibly combined to form larger programs. This principle isreflectedinthesmallkernel(coreoftheoperatingsystem)cooperatingwithalargesetofsmallcommandswhichworkwelltogether.Compose programs through interfaces—Write programs that are easy tointerfacewithotherprograms.ThefamousUNIXpipe,whichinterfacestheoutput of a program to the input of another, is a primary exampleof thisphilosophy.

Keepingprograminput/output,configuration,anddocumentationinplaintext(characterstrings)asmuchaspossiblemakeselementsoftheoperatingsystemeasytointerface,read,understand,andimprove.LinuxsystemshavegenerallyadheredtotheseprinciplesofUNIX,buthave

alsointroducedrefinementsandimprovements.

LinuxFeaturesLinux incorporates all the outstandingUNIX core features and adds graphicaluser interface (GUI), software update and package management, securityimprovements, and many useful applications. Important features of Linuxinclude

Multi-user and multi-processing—The ability to allow multiple users tologinatthesametimeandtheabilitytorunmanyprogramsconcurrently.Graphicaluser interface—Offeringadesktop environmentwithwindows,icons, panels, and menus, making it easy to use point-and-click foroperations.Most Linux systems use theXWindow system and allow theuser to choose between two popular desktop environments,GNOME andKDE.

Package management—A systematic way to find, install, upgrade,configure, and remove themany software packages available.A packagecontains theexecutableprogramandmetadataspecifyingits title,version,purpose, author/vendor, dependencies (on other packages), etc. Packagesaremadeavailableinrepositoriesfordownloading.TheFedoraandtheRedHat familyLinux systems use thednf (dandifiedyum) tool and the rpmpackage format, while the Debian varieties use the apt (AdvancedPackagingTool)andthedebformat.Shells—AShellisacommand-lineinterface(CLI)totheoperatingsystem.It provides interactive processing and execution of user commands. Thestandard (default) Shell for Linux isBash (born-again Sh), but youmayeasilychoosetouseadifferentShell.Hierarchical file system—The entire file system is tree structured and isanchored at a singledirectory called the root.The rootdirectorycontainsfiles andotherdirectories that, in turn, containmore files anddirectories.Eachuserhasahomedirectoryforhis/herownfiles.Thefilesystemtreeisdivided into volumes, which can bemounted ordismounted by attachingthemtoanodeinthefiletree.Aphysicalstoragedevicecancontainoneorseveralfilesystemvolumes.Fileaccesscontrol—Eachfileinthefilesystemisprotectedbyasequenceofbitswhosevalueisspecifiedbytheownerofthefile.Accesstofilesiscontrolled by the operating system. System-wide access is granted to so-calledsuperusers,usually thesystemadministrators.To improvefileandsystem security, the Security-Enhanced Linux (SELinux) (kernel moduleavailableonmostmodernLinuxdistributions)canalsobeenabled.Compatiblefile,device,andinter-processI/O—I/OtophysicaldevicesandI/O to a file look the same to a user program. A user can redirect aprogram’sI/Osothatwithoutchangingtheprogramitself, inputoroutputcanbedirectedtoaterminalwindow,file,oreventoanotherprogram’sI/O.The ability to combine and connect existing programs in this pipelinefashionprovidesgreatpowerandflexibility.Concurrent processes—Following UNIX, Linux provides a set of ShellcommandsandC-languageAPIs to initiateandmanipulateasynchronous,concurrent processes. This allows a user tomaintain several jobs at onceand to switch between them. It is also critical for pipelining severalcommands(processes)together.ServingtheInternet—AsUNIX,Linuxsystemsprovidelocalandwideareanetworking through sockets that support the Internet Protocol (IPv4 andIPv6) and provides efficient network services. System admin can easily

manage,configureandstartorstopdifferentnetworkservices.Linuxalsoworks well with the Apache Web Server to provide Web hosting. As aresult,Linuxisverypopularasanetworkserverplatform.Utilities—The Linux architecture encourages building self-containedprograms to add new facilities. Linux systems come with many utilityprograms including text editors, document processors, email servers andclients,Webbrowsers,rasterandvectorimageeditors,scriptinglanguages,language compilers, file manipulation tools, databases, multimedia tools,GUI design and programming tools, software engineering tools, andnetworkingandothersystemfacilities.Theseutilitiesusuallycomein theformofaLinuxpackagewhichcanbedownloaded,installed,andmanagedeasilywithapackagemanager.

TheLinuxkernel, thecentralpartof theoperating systemwhichprovidesaprogramminginterfaceto thehardware, isrobustandhighlyefficient.Figure1showsanoverviewoftheLinuxsystemorganization.Whenstudyingthevarioustopicsinthistextbook,thisorganizationaldiagramhelpstotiethemtogether.

Figure1LinuxOrganization

TheLinuxEnvironmentLinuxisamulti-user, time-sharingsystemthatoffersbothaGUI(desktopandapplicationwindows)aswellasaCLI(theShells).Thedesktopisthefirstthingyouseeafter login.Thedesktopdisplaysoneormorepanelsat the topand/orbottom of your screen. A panel provides menus, launchers, and workspaceswitchers which perform various tasks. Icons on the screen provide access to

Computer,Services,FileSystem,andsoon.Application programs fall into two broad categories: GUI applications and

CLI applications. A GUI application displays its own graphical window withwhichtheusermayinteractviathemouseandthekeyboard.Incontrast,aCLIapplicationmustruninsidea terminalwindowand interactswith theuseronlythroughthekeyboard.Launchers in panels, in menus, or on the desktop make starting programs

easy. However, any program can be invoked by typing a command inside aterminalwindow.Youcancontrolandswitchamongmultiplewindowson thescreen.AcommandShellhelpsyoucontrolandmanagemultiplejobsinsideanysingleterminalwindow.Thefilesystemcontainspublicfilesandprogramsforallusers.Eachuseralso

hasapersonalfiledirectoryknownastheuser’shomedirectory.Accesstofilesanddirectoriesiscontrolledbythefileownerandgroupdesignations.Linuxallowsahighdegreeofcustomizationonaper-userbasis.TheShell,as

wellasimportantutilitiessuchastheXWindowSystemandtexteditors,refersto initialization andconfiguration files.You can tailor these files tomake theutilities run according to your needs and preferences. You can even chooseamongdifferentShells to serveasyourCLI.Documentation forLinuxand itsutilitiescanbeconvenientlyaccessedlocallyonyourcomputer,aswellasontheWeb.

LearningLinuxLinuxsystemsareusedwidelyincolleges,universities,andcompanies,bothasserversandasworkstationsincomputerlabs.ManyusershaveLinux/Windowsdualbootontheirpersonalmachines.KnowledgeofLinuxisimportantforbothlearningandemployment.Tomakeexperimentationeasier,astudentcansetupaprivateLinuxlearning

environmentonhis/herownWindows®orMac®computer.SpecificinstructionscanbefoundintheAppendix.This book covers a set of carefully selected topics that enable you to

understandoperatingsystemconcepts,touseLinuxeffectively,andtotakefulladvantageofyourLinuxcomputer.Thechaptersaresequencedinadrill-downprogressionstartingwithaprimer

to get you started quickly on Linux with hands-on learning and meaningfultasks.Next,wepresenttheLinuxGUIandthestandardLinuxCLI(theBashShell).

Thenwediscussusefulapps,commands, filters tobuildpipelines,andregular

123

4

expressions for pattern matching. All this paves the way for writing BashprogramscalledShellscripts.Diggingdeeper,wediscusshowtocontrol filesandfolders,andhowLinux

organizesandmanipulatesfilesinasetoffilesystemsthatisanimportantpartoftheLinuxkernel.Computers are rarely used in isolation, and, like other modern operating

systems,Linuxreliesheavilyonnetworkingformanyoperations.Withagoodfoundation from the earlier chapters, we discuss networking, Web, Internet,public-keyencryptionanddigitalsignature.Linuxsystemadministrationbecomesimportantafterausergetsfamiliarwith

the operating system. For people serious about a Linux-related career, systemadminknowledgeiscritical.WecoverthebasicsofLinuxsystemmanagementinChapter8.Attention then turns toC-levelprogramming,kernelsystemcalls,processes,

and inter-process communication. These topics shed light on the internals ofLinux and provide a deeper understanding of concepts and topics covered inearlier chapters. The material should prove especially important for CS/CEmajors.Thus, youwill find traditional aswell as contemporary topics important for

the modern Linux environment. The material in this book applies to mostpopular Linux systems. The knowledge gained will enable you to use anyversionofLinuxwithease.MajordifferencesamongLinuxversionsarenotedwhereappropriate.BecauseLinuxisbestlearnedthroughfrequentexperimentationandpractice,

webeginwithaprimerthatgetsthenewuserstartedquickly.Weofferexamplesand practical ways to use Linux throughout the book. Many examples areprovidedtoillustrateconceptsandtodemonstrateprogrammingtechniques.Thistextbook also contains an example code package 4 which provides completeprograms ready to download and run on your computer. The material ispresented for smooth reading as a textbook, but also for convenient referencelateron.

LinuxisdistributedundertheGNUGeneralPublicLicense.EEEComputerSocietystandardsforPortableOperatingSystemInterface.MINIX is the first open-source clone of UNIX for the IBM PCwritten by Professor Andrew S.

Tanenbaumin1987.Seepage353fordownloadinginstructions.

1.1

Chapter1

ALinuxPrimer

If you are serious about computing, safeguarding security, and understandinghowanoperatingsystemworks,Linuxis thesystemofchoice.TolearnLinuxyoumustuse it, and,of course, touse it youmust learn it.Suchaparadox israther common—you probably learned to drive a car this way. You just needsome basic help and pointers to get started. Here we present an overview ofbasics.Onceyouunderstandthematerialinthischapter,youwillbeabletousetheoperatingsystemtolearnmoreineachsuccessivechapter.Atfirst,youneedalearner’spermittodriveacar.Considerthischapteryourlearner’spermitforLinux;withalittlepracticeyouwillbeusingLinuxalmostrightaway.LearningLinuxinvolvesunderstandinghowtouseitfromtheuserlevel,how

toapplyitspowersandappseffectively,featuresmakingitsuchagoodserver,andhowtoprogramitfromthesystemlevel.Withthistextbook,you’llbegintomasterLinux.Thisprimerprovidesbasic informationandaselectionof topicsdesigned to

getyoustartedusingLinuxquickly.Asyou read thischapter, try thedifferentcommands and features as you come to them. In each case, we will provideenoughinformationtogetyouonthesystemandlearning.

WHATISANOPERATINGSYSTEM?Theoperatingsystemcontrolsacomputerandmakesitusable.Itbringslifetothe innate electronic hardware components and orchestrates all activities on acomputer. The same hardware under a different operating system is literally adifferentcomputer.The operating system provides service and control functions to users,

programs,files,operators,displaymonitors,printers,networkconnections,andeverythingelseonacomputersystem.Acomputeroperatingisoneofthemost

1.2

complicatedandsophisticatedobjectshumanseverbuilt.AmodernoperatingsystemlikeLinuxconsistsofthreemainparts:akernel,

interfacesforusers,programs,devicesandnetworks,andasetofcommandsandapps. The kernel deals with central functions, including concurrent programexecution, memory management, input/output (I/O), file services, networkingandsecurity.Commandsandappssupplyotheroperationssuchasfilemanagers,text editors, email processors, Web browsers, software package managers,audio/video and image processing tools, language compilers, and so on.Interfacesdefineandsupportcommunicationsamongallthecomponents.Forusers,Linuxprovideseasy-to-useGraphicalUserInterfaces(GUIs)inthe

form of desktop environments. Linux also provides efficient and effectiveCommand-LineInterfaces(CLIs)intheformofShells.

GETTINGSTARTED:LOGINANDLOGOUTTo access your Linux system, you must have a user account, identified by auserid and a password, that have been created by a system administrator. Atmost installations, youruseridwill beyour last nameor your first initials andlastname(oftenalllowercase).Your password is a safeguard against unauthorized use of your computer

account.Youneed to choose a passwordof at least eight or twelve characters(your localsystemmayenforceotherconventionsaswell, suchasaminimumlengthorthattherebeatleastonenumeralorsymbol).Passwordsmustbehardto guess. Correctly spelled words or names of relatives are bad choices. Asequence containing upper and lower case characters, digits, and symbols isusually better. Longer passwords can be a phrase. Since you are the only onewho knows your password, you must be careful with it. Forgetting yourpasswordmeansthesystemadministratormustcreateanewoneforyou.Givingyour password to thewrong person could have evenmore dire consequences;youcouldbeblamedforwhateverdamageiscaused,intentionallyorotherwise,bytheotherperson.Donottellorshowanybodyyourpassword.Keepitwrittendownsomewheresafe.Onceyouhaveauseridandpassword,youcanbeginyourLinuxsession.The

firststepistheloginprocedure.Loginprotectsthesystemagainstunauthorizeduse and authenticates the identity of the user. You can use Linux from theconsoleoracrossanetwork.

Figure1.1LinuxLoginScreen

DesktopLoginFindacomputerdisplayingtheLinuxdesktoploginscreen(Figure1.1).Thiscanbetheconsolewherethekeyboard,mouse,anddisplayaredirectlyconnectedtothe computer hardware running the Linux system. Or it can be a differentcomputer on the LAN (Local Area Network). Colleges, universities, andcompanies often run computer labs with Windows or Mac stations that canaccessLinuxserversanddisplaytheirdesktopscreens.Inanycase,enteryourcorrectpasswordcarefullyandprivately.Ifyouarea

newuserand,afterseveralcarefultries,youareunabletologin,itmaybethatthe system administrator has not yet established your userid on the computer.Wait a reasonable length of time and try again. If you still have a problem,contactyoursystemadministrator.Afterlogin,you’llseeyourdesktopdisplayed.Thedesktopenablestheuseof

full-GUI (Graphical User Interface) applications that allow point-and-clickoperationswiththemouse,touchpadortouchscreen.From the desktop, you can press thewindows or super key or click on the

starticontoshow/hideastartmenuorapplicationsmenudisplayingmanytasksyoucando.ThestarticonisusuallyaLinuxdistributionlogolocatedontheleftendofyourdesktopPanel(normallyahorizontalbaracrossthetoporbottomofyourscreen). InGNOME3 forexample, simplymoving themousequickly totheupperleftcornershows/hidestheActivityScreen(Figure1.2).

Figure1.2AGNOME3ActivityScreen

TologoutfromLinuxlookforalogouticononthedesktopPanel.Morewillbesaidaboutdesktopsin2.

StartingaTerminalWindowFrom the desktop, you can conveniently initiate many operations includingstartingaterminalwindow(Figure1.3)thatrunsaShell(Section1.3).TheShellprovides you with a command-line interface (CLI) where you can entercommandstoperformalmostanytaskonLinuxquicklyandefficiently.

Figure1.3ATerminalEmulationWindow

Tostartaterminalwindow,gotothestartmenuandclickontheSystemtools-> Terminal option or the Accessories- > terminal option, depending on yourLinuxanddesktop.Theterminalmaybegnome-terminal,konsole,oranotherdepending on your Desktop environment. The GNOME 3 Applications menu(Figure1.4) includesagnome terminal icon toconveniently launcha terminal

window.Aterminalwindowemulatesacharacter-basedcomputer terminalandallowsyoutouseLinuxthroughacommandinterpretercalledtheShell(Section1.3).Theterminalwindowallowsyoutochangeitsappearanceandfonttoyourownliking.Asitstarts,theShellalsopositionsyouatyourhomedirectory(Section1.5),

the file folder reserved for you as a user on Linux. The Shell indicates itsreadiness to takeyourcommandsbydisplayingapromptat thebeginningofaline.

Figure1.4GNOME3ApplicationsMenu

Whenyouarefinishedwithaterminalwindow,youmaycloseitbyexit(exitsfromShellandclosestheterminalwindow)logout(sameasexit)Thecharacterctrl+d(thecharacterd typedwhileholdingdownthectrlkey)

typedaloneonacommandlineoftencanbeusedinplaceoftheexitcommand.Exitwithctrl+disconvenientbutdangerous,becauseonetypingerrorcancloseyourterminalwindow.SeeChapter3forhowtodisableexitviactrl+d.Bytheway,weshallusethenotationctrl+Xto denote a control character, where X is some character. Note also that

althoughtheconvention is toshowanuppercasecharacter,youdonotneed toholddownshiftwhentypingacontrolcharacter.

RemoteLoginUniversities and other institutions often run large Linux servers for users toaccessthroughaLANoreventheInternet.YoucanuseTELNET,ormorelikelySSH(SecureShell),toaccessaLinuxsystemfromanothercomputer,whichcan

be a PC, anotherLinux system, or anyother platform.Figure1.5 shows SSHaccess, via the Putty tool (free and recommended), to a Linux hosttiger.cs.kent.edufromMSWindows®.OnLinux,theShell-levelcommandsshprovidesSSHandisusedtoaccessa

remoteLinuxserverfromaLinuxsystem.Forexample,sshpwang@tiger.cs.kent.eduorssh-Xpwang@tiger.cs.kent.edunetworkstothecomputertiger.cs.kent.edu(thedomainnameofthecomputer)

andattempts to log inwith theuseridpwang.Remote loginnormallysupportsonlyCLIaccess.The-X(capitalX)optionallowstheremotecomputertoopenthegraphicaldisplayonthelocalLinuxandthereforeenablesyoutoalsolaunchremote applications that require a GUI. Running GUI programs remotelyinvolvesmuchheaviernetworktrafficandcanbeslow.Without the -Xoptionyou’llbeable to runonlycommand-lineapplications

ontheremotecomputerwhichisusually theefficientandsensible thingtodo.WewillreturntoSSHinChapter7(Section7.6)wherenetworkingisdiscussed.Download,installation,andusageinformationforSSH/SFTPcanbefoundintheappendicesonthecompanionwebsite(mml.sofpower.com).SuccessfulremoteloginviaSSHresultsinyourSSHwindowbeingconnected

toaloginShellrunningontheremoteLinux.

Figure1.5SSHLoginviaPutty

After login,Linuxwill recordyour login inasystemlog,displayamessageshowingthetimeandplaceforyourlastlogin,andinitiateaShelltotakeyourcommands.Whenyousee theprompt,youare ready tobegincomputing.Afteryouare

1.3

done,youwillneedtologoutfromtheremoteLinux.Tologout,firstcloseanyprograms that youhavebeen running and then issue theShell-level commandexitorlogout.Itisagoodpracticetofirstcloseallrunningprogramsmanuallyinsteadofrelyingonthelogoutprocesstoclosethemforyou.

UNDERSTANDINGTHESHELLTheShell displays a prompt to signal that it is ready for your next command,which it then interprets and executes. On completion, the Shell re-signalsreadinessbydisplayinganotherprompt.ThereareseveralavailableShells:theoriginalShellwrittenbyS.R.Bourne

known as the Bourne Shell or Sh, theC-Shell orCsh developed at UCB byWilliamJoy,andanenhancedCshnamedTcshwhichhasreplacedCshfor themostpart.TheDashshellisabare-bonesandPOSIX-compliantimplementationofShusuallyusedonlyatsystemboottime.ThestandardShellforLinuxistheBash(Bourne-AgainSh),whichisapowerfulandmuchimprovedversionofSh.ThedefaultShellonmostLinuxdistributionsisBash.AttheShellprompt,enterthecommandecho$0+todisplaythenameoftheShellyouareusing.Hereechodisplaysthevalueof

theShellvariable$0.Don’tworry,3explainshowthisworks.YoucanchangethedefaultShellwiththechsh(changeShell)command.Forsecurityreasons,usuallyonlyapprovedShellscanbeused.InthistextwewillassumetheBashShell,althoughbasicfeaturesofallShells

areverysimilar.

EnteringCommandsIn Linux, you can give commands to the Shell to start application programs,managefilesandfolders,controlmultiplejobs(tasksthatarerunning),redirectI/O of programs from/to files, connect one program to another, and performmany other tasks. Virtually anything you want done in Linux can beaccomplishedbyissuingacommandtotheShell.Manydifferentcommandsareavailable,butsomegeneralrulesapplytoallof

them.Onesetofrulesrelatestocommandsyntax—thewaytheShellexpectstosee your commands.A command consists of one ormorewords separated byblanks.Ablankconsistsofoneormorespacesand/ortabs.Thefirstwordisthecommandname(inthisbookthenameofacommandwillappearinboldface);the remaining words of a command line are arguments to the command. A

command line is terminated by pressing the return (or enter) key. This keygenerates a newline character, the actual character that terminates a commandline.Multiplecommandscanbetypedonthesamelineiftheyareseparatedbyasemicolon(;).Forexample,thecommandlsfolderliststhenamesoffilesinafolder(directory)specifiedbytheargumentfolder.

Ifadirectoryisnotgiven,lsliststhecurrentworkingdirectory(Section1.5).Sometimesoneormoreoptionsisgivenbetweenthecommandnameandthe

arguments.Forexample,ls-Ffolderaddsthe-F(filetype)optiontolstellinglstodisplaythenameofeachfile,or

eachfilename,withanextracharacterattheendtoindicateitsfiletype:/forafolder,*foranexecutable,andsoon.AttheShelllevel,thegeneralformforacommandlookslikecommand-name[options]...[arg]...The brackets are used to indicate elective parts of a command that can be

given or omitted. The ellipses (… ) are used to indicate possible repetition.These conventions are followed throughout the text. The brackets or ellipsesthemselvesarenottobeenteredwhenyougivethecommand.Commandoptionsareusuallygivenasasingleletterafterasinglehyphen(-).

Forexample,thelonglistingoptionforthelscommandis-l.Suchsingle-letteroptions can sometimes be hard to remember and recognize. Many Linuxcommands also offer full-word options givenwith twohyphens. For example,the–helpoptiongivenaftermostcommandswilldisplayaconcisedescriptionofhowtousethatparticularcommand.Tryls–helptoseeasampledisplay.After receiving a command line, theShell processes the command line as a

character string, transforming it in various ways. Then, the transformedcommand line isexecuted.Afterexecution is finished, theShellwilldisplayaprompttoletyouknowthatitisreadytoreceivethenextcommand.Figure1.6illustratestheShellcommandinterpretationloop.Typeahead isallowed,whichmeansyoucantypeyournextcommandwithoutwaitingfortheprompt,andthatcommandwillbetherewhentheShellisreadytoreceiveit.

TryingaFewCommandsWhenyouseetheShellprompt,youareattheShelllevel.NowtypeechoHelloLinux

Figure1.6CommandInterpretationLoop

You’llseethattheechocommanddisplayswhatyoutype.Next,enterecho-n"HelloLinux";echouserThis command line contains two commands separated by the ; command

separator. (Ifyoumakeamistake typing thesecommands,glanceahead to thenext subheading on correcting typingmistakes.)The option -n causes echo toomitanewlinecharacterattheendofitsoutput,sotheworduserappearsonthesame line asHelloLinux.Note also the use of quotationmarks for the stringHelloLinuxwhichhasatrailingspace.Oneuseofecho istoexaminethevalueofaShellvariable.Forexample, if

youtypeecho$HOMEyou’llseethevalueoftheShellvariableHOMEwhichisthelocationofyour

home directory in the file system. Note that the value of a Shell variable isobtainedbyprefixing the variable namewith a dollar sign ($).MoreonShellvariablescanbefoundin3.A computer on a network is known as ahost and is usually identified by a

hostname.TofindoutyourLinuxsystem’shostname,givethecommandhostnameToidentifytheoperatingsystemversionrunningonyourcomputer,enterthe

commanduname–allAnothercommandiswho.Typewhoto list currentusers signed inon the system.Thisgivesyouan ideaofhow

manypeoplearesharingthecomputingfacility.Thelscommandwillnotlisthiddenfiles,anyfilewhosenamebeginswiththe

period(.)character,unlessthe-aoptionisgiven.ls-alists thenamesof allyour files, including thehiddenones.Hidden files are

usuallystandardoperatingsystemorapplicationfilesforconfigurationorother

prescribedpurposesandoughtnotbemixedwithotherfilescreatedbytheuser.For the Bash Shell, one standard file is .bash_profile in a user’s home

directory.YoucanplaceinthisfileyourpersonalinitializationtobeusedwhenbashstartsasaloginShell.Ifyouarecuriousaboutwhat’sinthefilebash_profile.,typethecommandmore.bashprofiletodisplay its contents.Press space tocontinue to thenextpageorq toquit

fromthemoredisplay.Don’tbediscouragedbywhatyoudon’tunderstand inthis file. When you have progressed further in this book, the contents willbecomeclear.TheLinux systemkeeps trackof the time anddate precisely, as youwould

expectanycomputertodo.Thecommanddatedisplays the current date and time as given by the following typical output

showingEasternDaylightTimeThuDec416:37:07EST2018TheLinuxsystemhasadictionaryofwordsforspellcheckingpurposes.The

commandspellfilewilldisplaysuspectedmisspellingsforyou.Oryoucanuseaspell-cfiletointeractivelyspellcheckthegivenfile.Tolookforwords,youcanuselookprefixonmostLinuxsystems,andallwordsinthedictionarywiththegivenprefix

aredisplayed.Anotherusefulcommandispasswd.Typepasswdtochangeyourpassword.ThiscommandwillpromptasfollowsChangingpasswordforyouruseridOldpassword:Newpassword:Retypenewpassword:pausingaftereachprompttowaitforinput.ManyLinuxinstallationsgiveout

newuseridswithastandardpassword,andthenewuser isexpectedtouse thepasswdcommandtochangetoapersonalpasswordassoonaspossible.Thecommandmanconsultsthemanualpagesformostcommands.Thus,mancommandwilldisplaythedocumentationforthegivencommand.Trymanpasswd

justtoseewhatyouget.LearnaboutmanwithmanmanDetailsonthemancommandcanbefoundinSection1.15.Themancommanddocumentsregularcommands(applicationprograms),but

normally not commands built in to Shells or other application programs. ForBashyoucanusehelpbuiltin_commandto see a summary of any particular Bash built-in command. Many Linux

systemsaddaBash_BuiltinsmanpagesothemancommandwillworkforBashbuilt-incommandsaswell.

CorrectingTypingMistakesAs you entered the preceding commands, you may have made at least onekeystroke error, or you may wish to reissue a command you have enteredpreviously. Linux Shells provide easy ways to correct typos and to reusepreviouscommands.Basically,youcanusethearrowkeystomovethecharactercursorleftandrightonacommandlineanduptoapreviouscommandordowntothenextcommand.The delete (backspace) key deletes the character under (before) the cursor.

Theenter(ret)keyissuesthecommandnomatterwherethecursorisontheline.The Bash Shell has great support for editing the command line. It actually

allows you to pick a text editor to help do the job.We will return to this inChapter3,Section3.3.

AbortingaCommandApartfromcorrectingtypingmistakes,youcanalsoexerciseothercontrolsoveryourinteractionwithLinux.Forinstance,youmayabortacommandbeforeitisfinished, or youmaywish to halt, resume, and discard output to the terminalwindow.Sometimes,youmayissueacommandandthenrealizethatyouhavemadea

mistake.Perhapsyougiveacommandandnothinghappensoritdisplayslotsofunwantedinformation.Theseareoccasionswhenyouwanttoabortexecutionofthecommand.To abort, simply type the interrupt character, which is usually ctrl+c. This

interrupts (terminates) execution and returns you to the Shell level. Try thefollowing

1. Typepartofacommand.2. Beforeyouterminatethecommand,pressctrl+c.

1.4

1.5

Itcancelsthecommandandgivesyouanewprompt.

EXERCISEA1. Howdoyoustartaterminalwindow?2. Whatcommandandoptionshouldbeusedtolistallthefilesinyourhome

directory?3. Setupctrl+alt+Tasthekeyboardshortcutforrunningaterminalwindow.4. What command is used to change your password? Can you change it to

something like 123 Why Make up a longer password and change yourpasswordtoit.Whydidyouhavetotypeyourpasswordtwicethistime?

5. TryinputeditingwiththearrowkeysunderBash.Afterdoingacommandls -l,pressup-arrowonceand left-arrow twice.Where is thecursornow?Now, pressRIGHT-ARROW once and the cursor should be over the letter lwhich is the last character on the command line. Can you press RIGHT-ARROWagaintomovethecursorbeyondlIfnot,canyoufindaway?(Hint:Limityourselftousingonlythearrowkeys.)

6. What is theLinuxdistributionyouare running?What is thehostnameofyourLinuxcomputer?Howdoyouobtainthisinformation?

Figure1.7ASampleFileTree

USINGFILESANDDIRECTORIESLikeothermodernoperatingsystems,Linuxstoresfilesforusers,applications,andtheoperatingsystemitselfonharddisksforreadyaccess.Thestructureusedto store and manage such files is called a file system. Files under Linux areorganizedintoatreestructurewitharootnamedbythesinglecharacter/.Aregularfilestoresaprogramordata.Adirectoryorfoldercontainsfilesand

possibly other directories. Internal nodes on the Linux file tree representdirectories; leafnodesrepresent regular files.Thishierarchical filestructure is

widelyusedbydifferentoperatingsystems.AsampleLinuxfiletreeisshowninFigure1.7.Avisualfilebrowser(Figure1.8)utilityallowsyoutonavigatethefilesystem

and perform operations on files and folders. Two popular file browsers areKonquerorandNautilus.Forexample,thecommandnautilusfolderlaunchesNautilusandpositionsitatthegivenfolder.Whilethefilebrowsermakesmovingaboutthefilesystemmorevisual,many

LinuxusersstillfinddealingwithfilesandfoldersviatheShellcommandlinemoreefficient.

CurrentWorkingDirectoryandFilenamesWhen you get a userid and account on your Linux system, you are given apersonalfiledirectoryknownasyourhomedirectory.Yourhomedirectorywillhaveyouruseridasitsname,anditwillusuallybeachildofadirectorycalledhome.Yourfilesandfoldersarekeptinyourhomedirectory.

Figure1.8LinuxFileBrowser

Toaccess a fileor directory in the file system from the command line, youmustcall itupbyitsname,andthereareseveralmethodstodothis.Themostgeneral,andalso themostcumbersome,waytospecifya filename is to listallthenodesinthepathfromtheroottothenodeofthefileordirectoryyouwant.This path, which is specified as a character string, is known as the bsolutepathname,or fullpathname, of the file.After the initial /, all components inapathname are separated by the character /. For example, the file note.txt inFigure1.7hastheabsolutepathname/home/pwang/note.txt

The full pathname is the complete name of a file. As you can imagine,however, this name often can be lengthy. Fortunately, a filename also can bespecified relative to thecurrentworkingdirectory (alsoknownas theworkingdirectory or current directory). Thus, for the file /home/pwang/note.txt, if thecurrent working directory is /home, then the name pwang/note.txt suffices. Arelative pathname gives the path on the file tree leading from the workingdirectorytothedesiredfile.Thethirdandsimplestwaytoaccessafilecanbeusedwhentheworkingdirectoryisthesameasthedirectoryinwhichthefileisstored. In this case, you simplyuse the filename.Thus, aLinux file has threenames

Afullpathname(forexample,/home/pwang/note.txt)Arelativepathname(forexample,pwang/note.txt)A(simple)name(forexample,note.txt)

The ability to use relative pathnames and simple filenames depends on theabilitytochangeyourcurrentworkingdirectory.If,forexample,yourworkingdirectory is /tmpandyouwish toaccess the filenote.txt,youmayspecify theabsolutepathname/home/pwang/note.txtoryoucouldchangeyourworkingdirectorytopwangandsimplyrefertothe

filebyname,note.txt.Whenyoulogin,yourworkingdirectoryisautomaticallysettoyourhomedirectory.Thecommandpwd(printworkingdirectory)displays the absolute pathname of your current working directory. The

command(Bashcommand)cddirectory(changeworkingdirectory)changesyourworkingdirectorytothespecifieddirectory(givenbyasimple

name,anabsolutepathname,orarelativepathname).Twoirregularfilesarekeptineverydirectory,andtheyserveaspointers

File . isapointertothedirectory(directoryselfpointer)inwhichthisfileresides.

File .. isapointertotheparentdirectory(parentdirectory)ofthedirectoryinwhichthisfileresides.Thesepointersprovideastandardabbreviationforthecurrentdirectoryandits

parentdirectory,nomatterwhereyouareinthefiletree.Youalsocanusethesepointersasashorthandwhenyouwanttorefertoadirectorywithouthavingtouse,orevenknow,itsname.Forexample,thecommandcd.

hasnoeffect,andthecommandcd..changestotheparentdirectoryofthecurrentdirectory.Forexample,ifyour

workingdirectory is jdoe, andyouwant to access the file sort.c in thepwangdirectory,youmayuse../pwang/sort.c.Whydoesthiswork?Yourhomedirectoryalreadycomeswithaname,youruserid.However,you

nameyourfilesandsubdirectorieswhenyoucreatethem.Linuxislenientwhenitcomestorestrictionsonfilenames.InLinuxyoumaynameyourfilewithanystring of characters except the character /. But, it is advisable to avoid whitespacecharactersandanyleadinghyphen(-).

HandlingFilesandDirectoriesGenerally,therearetwokindsofregularfiles:textandbinary.ALinuxtextfilestorescharacters inASCIIorUNICODEandmarks theendofa linewith thenewline character. 1 A binary file stores a sequence of bytes. Files may becopied, renamed, moved, and removed; similar operations are provided fordirectories.ThecommandcpwillcopyafileandhastheformcpsourcedestinationThe file source is copied to a file named destination. If the destination file

does not exist, it will be created; if it already exists, its contents will beoverwritten.Themv(move)commandmvoldnamenewnameisusedtochangethefileoldnametonewname.Nocopyingofthefilecontent

is involved. The new namemay be in a different directory—hence the name“move.”Ifnewnamealreadyexists,itsoriginalcontentislost.Once a file or subdirectory has outlived its usefulness, you will want to

removeitfromyourfiles.Linuxprovidesthermcommandforfilesandrmdirfordirectoriesrmfilenamelfilename2rmdirdirectoryname1directoryname2...Theargumentofrmisalistofoneormorefilenamestoberemoved.rmdir

takesasitsargumentalistofoneormoredirectorynames;butnote,rmdironlywilldeleteanemptydirectory.Generally, toremoveadirectory,youmustfirstcleanitoutusingrm.To create a new directory, use the mkdir command, which takes as its

argumentthenameofthedirectorytobecreatedmkdirnameWhenspecifyingafileordirectorynameasanargumentforacommand,you

1.6

mayuseanyoftheformsoutlined.Thatis,youmayuseeitherthefullpathname,therelativepathname,orthesimplenameofafile,whicheveryouprefer.

StandardPersonalDirectoriesItiseasytochangetoahomedirectory,justdocd (goestoyourhomedirectory)cd (goestothehomedirectoryofuserid)In Linux, there are a number of standard folders under each user’s home

directory,usuallyincluding

Desktop—Files in this folder appear as icons on your graphical desktopdisplay, including regular files and application launchers (with filenamesuffix.desktop)Documents—TextualdocumentssuchasPDFfilesandthosecreatedusingtoolssuchasApacheOpenOfficeandLibreOffice.Download—FilesdownloadedfromthenetworkMusic—SoundandmusicfilesPictures—Picturesfromdigitalcameraspublic_html—FilesunderthisfolderaremadeavailabletotheWebviaanHTTPserveronyourLinuxsystemVideos—Filesfromvideocamerasandrecorders

In addition to these, you may consider setting up a bin/ for your ownexecutables, a tmp/ for temporary files, a templates/ for reusable files, ahomework/foryourclasses,andsoon.

PROTECTINGFILES:ACCESSCONTROLEvery file has an owner and a group designation. Linux uses a 9-bit code tocontrol access to each file. These bits, called protection bits, specify accesspermission toa file for threeclassesofusers.Ausermaybeasuperuser, theownerofafile,amemberinthefile’sgroup,ornoneoftheabove.Thereisnorestrictiononsuperuseraccesstofiles.u (Theownerorcreatorofthefile)g (Membersinthefile’sgroup)o (Others)Thefirst threeprotectionbitspertain touaccess, thenext threepertain tog

access, and the final three pertain to o access. The g type of user will be

discussedfurtherin6.Each of the three bits specifying access for a user class has a different

meaning.Possibleaccesspermissionsforafilearer (Readpermission,firstbitset)w (Writepermission,secondbitset)x (Executepermission,thirdbitset)

TheSuperUserRoot refers toa classof superusers towhomno file access restrictionsapply.The root status is gained by logging in under the userid root (or some otherdesignatedrootuserid)orthroughthesucommand.Asuperuserhasreadandwrite permissionon all files in the system regardlessof theprotectionbits. Inaddition, the superuserhasexecutepermissiononall files forwhichanybodyhas execute permission.Typically, only system administrators and a fewotherselected users (“gurus” as they’re sometimes called) have access to the superuserpassword,which,forobviousreasons,isconsideredtopsecret.

ExaminingthePermissionSettingsThenineprotectionbitscanberepresentedbya3-digitoctalnumber,whichisreferredtoastheprotectionmodeofafile.Onlytheownerofafileorasuperusercansetorchangeafile’sprotectionmode;however,anyonecanseeit.Thels-llistingofafiledisplaysthefiletypeandaccesspermissions.Forexample,-rw-rw-rw-1smith127Jan201:24primer-rw-r–r–1smith58Jan243:04updateisoutputfromls-lforthetwofilesprimerandupdate.Theownerofprimeris

smith,followedbythedate(January20)andtime(1:24A.M.)ofthelastchangetothefile.Thenumber127isthenumberofcharacterscontainedinthefile.Thefiletype,accesspermissions,andnumberoflinksprecedethefileowner’suserid(Figure 1.9). The protection setting of the file primer gives read and writepermission tou,g,ando.Thefileupdateallowsreadandwrite tou,butonlyreadtogando.Neitherfilegivesexecutionpermissions.Therearetenpositionsin theprecedingmodedisplay(of ls).The firstpositionspecifies the file type;the next three positions specify the r, w, and x permissions of u; and so on(Figure 1.9). Try viewing the access permissions for some real files on yoursystem.Issuethecommandls-l/bintoseelistingsforfilesinthedirectory/bin.

1.7

Figure1.9FileAttributes

SettingPermissionsA user can specify different kinds of access not just to files, but also todirectories.Auserneedsthexpermissiontoenteradirectory,therpermissiontolistfilenamesinthedirectory,andthewpermissiontocreate/deletefilesinthedirectory.Usually,afileiscreatedwiththedefaultprotection-rw——-soonlythefileownercanread/writethefile.Tochangetheprotectionmode

onafile,usethecommandchmodmodefilenamewheremodecanbeanoctal(base8)number(forexample,644forrw-r–r–)to

set all 9 bits specifically or can specify modifications to the file’s existingpermissions,inwhichcasemodeisgivenintheformwhooppermissionop2permission2...Who represents the user class(es) affected by the change; it may be a

combinationofthelettersu,g,ando,oritmaybetheletteraforallthree.Op(operation)representsthechangetobemade;itcanbe+toaddpermission,-totake away permission, and = to reset permission. Permission represents thetype(s)ofpermissionbeingassignedorremoved;itcanbeanycombinationofthelettersr,w,andx.Forexample,chmodo-wfilenamechmoda+xfilenamechmodu-w+xfilenamechmoda=rwfilenameThe first example denies write permission to others. The second example

makesthefileexecutablebyall.Thethirdexampletakesawaywriteandgrantsexecute permission for the owner. The fourth example gives read and writepermission(butnoexecutepermission)forallclassesofuser(regardlessofwhatpermissionshadbeenassignedbefore).AdetaileddiscussionontheLinuxfilesystemcanbefoundin6.

EXERCISEB

1.8

1. Gotoyourhomedirectoryandlistallfiles(hiddenonesincluded)togetherwiththepermissionsettings.

2. Usingthelscommand,listyourfilesintimeorder(mostrecentfirst).3. List the permission settings of your home directory. Use the chmod

commandtomakesuretoforbidreadandwritefromgando.4. Createa folderpublic_htmldirectlyunderyourhomedirectoryandmake

sureyouopenreadandexecutepermissionsonthisfolder.5. Connect your digital camera to your Linux box and download pictures.

Where are the pictures placed? Can you find them under your Picturesfolder?

Figure1.10Gedit

TEXTEDITINGCreatingandeditingtextfilesisbasictomanytasksonthecomputer.TherearemanytexteditorsforLinux,includinggedit,nano,vim/gvim/vi,andemacs.Theeditorgedit (Figure1.10) comeswith theGNOMEdesktop. It requires

almost no instructions to use. Start it from the Start menu Text Editor or thecommandgeditfile&Aneditorwindowwilldisplay.Thenyoucantypeinput;movethecursorwith

thearrowkeysormouse;selecttextwiththemouse;removetextwiththedeleteorbackspacekey;andfind,cut,copy,andpastetextwiththebuttonsprovidedorwiththeeditmenuoptions.Itisveryintuitive.Thegedit isaGUIapplication.Ifyouwanta terminal-window–basededitor

then consider nano, which is very easy to learn but is less powerful orconvenient thanvimoremacs.Guides tovim andemacscan be found in theappendicesonthecompanionwebsite(mml.sofpower.com).Editingpoweraside,thereissomethingtobesaidaboutaneditorthatiseasy

andintuitiveforsimpletasks,especiallyifyouareabeginner.Inanycase,pickatexteditorandlearnitwell.ItcanmakelifeonLinuxsomucheasier.

Figure1.11Nano

Toinvoketheeditornanoforeditingfile,typefromtheShelllevelnanofile(startsnano)nano-wfile(startsnanowithoutlinewrapping)If the file exists,nano displays it for editing.Otherwise, you are creating a

newfilebythatname.Asyouentertext,nanowillstartanewlineautomaticallywhen the text line gets close to the right edgeof your editorwindow.The -woptionasksfornosuchautomatic linewrapping. It isalsoadvisable toalwaysuse the -zoptionwhichallowsyou tosuspendnanoandgetback to theShelllevel.Once insidenano, youareworking in a text-editingenvironment controlled

bynano, andyou can create text,make changes,move text about, and so on.Common operations are indicated by the nano editor window (Figure 1.11).Hereisalisttogetyoustarted.

Tosavethefile,typectrl+o.Toquitandterminatenano,typectrl+x.Youcanthenelectwhethertosavethebufferorcanceltochangeyourmindaboutquitting.Tomovethecursor,usethearrowkeys.Tocutandpastewholelines,ctrl+kcutsonelineatatimeandctrl+upastesthelinescut.To cut and paste selected text, type ctrl+6, move the cursor to highlightselectedtext,andthenusectrl+kandctrl+u.Tolookfortextintheeditingbuffer,typectrl+w(where),thetexttofind,andenterorreturn.Togethelponoperations,typectrl+g.

1.9

1.10

GETTINGHARD/SAVEDCOPIESTogetaprintedcopyofafileuselpr[options]filenameThiscommandsendsfilenametoaprinter.Yourprintingrequestjoinsaqueue

ofsuchrequeststhatareprocessedinorder.Notethatonlysupportedfiles(plaintext,postscript,orpdf)canbeprintedthisway.Donotsendunsupportedfiles,suchasacompiledprogram(.ofile),oracompressedfiletoaprinterthisway.The print option on the filemenu of application programs, such as yourWebbrowser,PDF(PortableDocumentFormat)reader,ordocumenteditor,canalsobeused.Often,youcanavoidwastingpaperbyusingtheprinttofileoption.Youcan

easilysavetheresultingfile(mostlyinPDF)andsharewithothersbyemailorSFTP(SecureFileTransferProtocol,5,Section5.20).

COMMUNICATINGWITHOTHERSAssoonasyoulogin,youcanpotentiallyinteractwithothers,whethertheyareuserson thesameLinuxcomputeroronotherhosts (computers)connectedbynetworking. Commands such aswho (who is logged in) and finger help toidentifymembersofyourusercommunity;emailapplicationsallowthesendingand receiving of messages and files; and instant messaging (IM) programsenableimmediateinteractionamongon-lineusersanywhereontheInternet.

Who’sWhoontheSystem:fingerIf you are a new user, you may not know many people on the system, andalthoughtheinformationprovidedbywhoandwisuseful,youdon’tknowwhothese users are. You only know their userids, which may not resemble theiractualnamesevenfaintly.Thecommandfingerwillgiveyousuchdataasfullname, office, address, and phone number for each user; this is sometimesreferredtoasthefingerdatabase,becausefingerisusedtolookupinformationfromthisdatabase.Thegeneralformisfingername...This commandwill display all entries in the finger database that contain a

useridandfirst,middle,orlastnamematchinganyofthegivenarguments.Forexample, either finger smith or finger clydewill result in the entry shown inFigure1.12.

Figure1.12ASamplefingerOutput

This multiline output includes a project field, which is the first line in the.projectfileintheuser’shomedirectory.Theplanfielddisplaystheuser’s.planfile. These two files supply additional information about a user for the fingerdatabase.Thenoplanlineintheexampleindicatesthatcsmithhasno.planfile.Onsomesystems,fingergivesonlyaveryshortsummaryunlessthe-loptionisgiven.Usedwithanargument,fingerwillaccessinformationonanyuserknownto

thesystem,whether thatuser is loggedonornot. If finger isusedwithoutanargument,anabbreviatedfingerentryisdisplayedforeachusercurrentlyloggedin.Thefcommandissometimesavailableasashorthandforfinger.

Figure1.13ThunderbirdEmailProgram

EmailElectronicmailgivesyoutheability tosendandreceivemessages instantly.Amessage sent via email is delivered immediately and held in a user-specificmailboxforeachrecipient.YoucansendemailtousersonthesamecomputeroronothercomputersontheInternet.Many utilities are available on Linux for email, including the popular

thunderbird (Figure 1.13), Evolution, and Kmail. These full-GUI email

1.11

programs are nice when you are at a Linux console. Command-line emailprogramssuchaselmandmuttareusefulfromaterminalwindow.Let’sexplainhowtousemutt.muttuserid@host-address(Internetmail)muttuserid(localmail)Thenjustfollowinstructionsandenterthemessagesubjectandtype/edityour

message.Mutt lets you edit yourmessage using your favorite text editor. Formuttandmanyotherapplicationsthatneedtextediting,setyourfavoriteeditorbygivingavaluetotheenvironmentvariableEDITOR(3,Section3.10).EDITOR=vim+orEDITOR=emacsexportEDITORWhenyoufinisheditingyourmessage,itwillbesentoutautomatically.mutt–help|moredisplaysmore informationonmutt usage.Here, the output is piped via the

+|+notation(Chapter3,Section3.5) to themorepaginatorwhichdisplays theinformationonepageatatime.To receive email (to checkyourmailbox), typemuttwith no argument and

follow instructions. Try to send yourself some email to get familiar with theusage.Emailisfast,butnotinstantorinteractive.OnLinux,youcandoIM.Skype

andGoogleHangoutareverypopular.BothworkwellonallkindsofsystemsincludingLinux.ForIMonLinux,youcanalsochoosepidginorempathy.

BROWSINGTHEWEBOneofthemostimportanttoolsonanycomputersystemistheWebbrowser.OnLinux you have a choice of differentWeb browsers. Popular choices includeGoogle Chrome and Firefox. The command firefox comes with most Linuxdistributions.InFigure1.14weseethehomepageofourtextbookwebsite.

1.12

1.13

Figure1.14AccessTextbookSite

You can enter aURL (UniformResource Locator) in the browser Locationwindow to visit a specific Web address. A local file URL, taking the formfile://full_pathnamecanbeusedtovisityourlocalfilesystem.Normally,Webbrowsersarefull-GUIprogramsusedinteractively,butLinux

also offers a command-lineWeb browser called lynx, a text-only browser thatdoesnotdisplayimages.However,lynxcanbeusedinsideaterminalwindowtointeractivelybrowsetheWebusingthearrowkeysortodownloadfilesfromtheWeb.

EXERCISEC1. Trythemuttemailprogram.Useit tosendanemailandattachafile.Do

thesameusingThunderbird.2. Createatextfileusingnano.3. Try the vim or emacs editor. Read the related appendix on the book’s

website.4. If possible, set up Thunderbird as your email program and Firefox or

ChromeasyourWebbrowser.5. DownloadafileusinglynxfromtheWeb.

CREATINGANDRUNNINGYOUROWNPROGRAM

Skip this section if you have no immediate interest inwriting a program in ageneral-purposeprogramminglanguagesuchasC.Youcanalwaysreturntothissection later.TheLinuxsystemoffersmany languages:C,C++,Java,Fortran

77/95, Python, Ruby, and Perl, just to name a few. You can also write Shellscripts(5)toautomatefrequentlyusedoperations.FileNameSuffixesLinuxfollowsasetofconventionsfornamingdifferentkindsoffiles.Table

1.1illustratessomecommonlyusedfilenamesuffixes.Asourcecodefilecannotbeexecuteddirectly.Theprogramusuallymustbecompiledintomachinecodebeforeexecutioncan takeplace.Analternative tocompilation is to interpret ahigh-levellanguageprogramdirectlyusinganinterpreter.

Table1.1PersonIn-ChargeAttheOrganizationRespondents

We shall follow an example of creating and running a simple C-languageprogram. Use your favorite text editor and create a C source file try.c (Ex:ex01/try.c)withthecode

#include>stdio.h<intmain(){printf("runningmyC

program\n");return0;}

This is a simple source program inC that displays the line “runningmyCprogram.”ThenotationnstandsfortheNEWLINEcharacter.

CompilingBefore try.c can be run, it must be compiled. Compiling is the process oftranslatingaprogramwritteninahigh-levellanguagesuchasCorPascalintoalow-levellanguageforexecutiononaparticularcomputer.Onmanysystemsthecompilerwilloutputafileofobjectcode,whichmustbeinked(combinedwithroutines suppliedby the system library)bya separateprogramcalleda linker.Once linkage is complete, the file is considered executable and is ready to beloadedintomemoryandexecuted.Linux-based compilers will handle both the compiling and the linking of a

program unless you specifically tell them not to, and their output will be anexecutablefile.Availablecompilersincludeproduceobjectfiles(.o).gcc GNUCcompilerg++ GNUC++compiler

javac Javacompilergfortran GNUFortran77/95compilerLet’strycompilingthesampleprograminthefiletry.cgcctry.cThiswillproduceanexecutablefile+a.out+whichcanbeinvokedsimplyby

typingitasacommand(Chapter3Section3.5).a.outNotethatinLinuxthecommandtorunaprogramissimplythepathnameof

theexecutablefile.Thus,./a.out(runstheexecutable)At some point, you probably will want to name your executable file

something other than a.out, especially since a.outwill be overwritten the nexttimeyouinvokeacompilerinthisworkingdirectory.Wealreadyknowthatthemvcommandcanbeusedtorenameafile,butthe-ooptiontogccorg++canbeusedtoprovideanametouseinsteadofthedefaulta.out.Forexample,gcc-omytrytry.cproducestheexecutable+mytry+.Nomatterwhichlanguageprogramyourun,youprobablywillwantarecord

ofyourresults(tosubmitasahomework,forexample).Onewaytodothisistouseoutputredirection.Forexample,./a.out>results+The symbol > tells the Shell to redirect output of a.out from the terminal

screenintoanewfilenamedresults.Thus,youwillseenooutputintheterminalwindowaftertheprecedingcommand.Instead,you’llfindtheoutputinthefileresult.AfullaccountofShellI/Oredirectioncanbefoundin3,Section3.5.Anotherwaytodothisistousethescriptcommandscriptrecord_fileto recordyour terminal session intoa record_file.Whilescript is active, all

I/Otoandfromyourterminalwindowiswrittentothefileyouspecified(ortoafilenamedtypescriptifyouenteredscriptwithoutanargument).Therecordingcontinuesuntilyoutypectrl+datthebeginningofacommandline.Thefilethencanbeviewed laterwitha texteditororemailed to someone.Forexample, torunaCprogramwithscript,thefollowingsequenceofcommandsmaybeusedscriptdisplay_record?ccmyprogram.c?a.outctrl+dThescriptcommandrequests thatall subsequent I/Oberecorded in thefile

1.14

1.15

displayrecord.Thectrl+donthelastlinestopstherecordingandgetsyououtofscriptandbacktotheShelllevel.An advantage of using script over simply redirecting output is that the file

producedbyscriptwillcontainbothinputtoandoutputfromtheprogram.Thefilecreatedbyredirectingoutputwillcontainonlyoutput.

EXERCISED1. Typeinasimpleprogram(inyourfavoriteprogramminglanguage)totype

outthemessage:Linuxisniceonceyouknowit.Compileitandrunit.Usescripttogetadisplayrecordoftheprogram’sexecution.

2. Usemoreornanotoviewthefileproducedbyscriptandthensendthefiletosomeonebyemail.

CONSULTINGLINUXDOCUMENTATIONThe command yelp (regular command) provides a GUI for browsing Linuxdocumentationandwhengivenwithnoarguments launches theGNOMEHelputilitywhichisagoodguideforbeginners(Figure1.15).

Figure1.15GNOMEHelp

The Linux Documentation Project website http://tldp.org providescomprehensivedocumentationforalmostallaspectsofLinux.You’llfindFAQs,topic-specific step-by-step instructions called HOWTOs, guides, and manualpagesforcommands.Asearchfeaturemakesiteasytofindwhatyouneed.You can also find documentation provided by your own Linux. The Help

1.16

1.17

1.18

menuonthetoolbarofGUIapplications,suchastheFileBrowser,theTerminalEmulationWindow,andPidgin,providestool-specificdocumentation.Command-line programs often provide brief and concise usage information

withthe–helpcommandoption.Forexample,tryls–helpTheman command displays manual pages set in standard UNIX-defined

formats. See Section 3, Section 3.11 for more information on Linux manualpages.

EXERCISEE1. Howdoyouaskacommandtohelpyouuseit?2. Accessthemanpageforlsandreaditfrombeginningtoend.3. Explain how to display the introduction section to the user commands

chapteroftheLinuxmanpages.

ROUNDINGUPUSEFULCOMMANDSIn this chapter, we have run into only a small number of themany availableLinux commands. The richness and variety of Linux commands are majorstrengths of the system. It is doubtful, however, that many users know all ofthem; you learn the commands that accomplish what you need. This sectioncollectsthecommandsthatshouldbeinanewuser’sbasicrepertoire.InLinux,bothuppercaseandlowercasecharactersareused,andtheyarenot

interchangeable.Allsystem-definedLinuxcommandsareenteredinlowercase.Also, there are two kinds of commands: (1) built-in Shell commands that aresubroutines in the Shell and (2) regular commands that are initiated as jobscontrolledbytheShell.Theimportanceofthisdistinctionwillbecomeclear.Inthefollowinglistingofcommands,user-suppliedargumentsareshowninitalics.Optional arguments are enclosed in square brackets [ ]. Possibly repeatedarguments are indicated by ellipses (...). These conventions will be followedthroughout the book. Only the most common usages of these commands aregiven.The informationhere is intended togetyou startedand isbynomeanscomplete.Detailsareprovidedinlaterchapters,andyoushouldconsulttheon-linemanualforfulldescriptionsofeachofthesecommands.

SUMMARY

1

Linux provides both full-GUI applications and command-line programs. TheGUIisvisualandmoreintuitivetouse,butmanybasicLinuxutilitiesaremoreconvenientonthecommandline.AShell(typicallyBash)runninginaterminalwindowprovidesaCLI to enter andexecute commands.Learning touseboththeGUIandtheCLIeffectivelywillmakelifemucheasieronLinux.TheCLIisespeciallyimportantforremoteaccessofLinuxusingSSH.Thedesktopmainmenuleadstomanyusefuloperations.2presentstheLinux

desktop.AShellexecutescommandsyouinputfromthekeyboardanddisplaysresults

inyourterminalwindow.Typingerrorscanbecorrectedthroughinputediting.Both the systemand theusers storedata in filesmanagedby theLinux file

system, which has a tree structure. Each file can be referred to by a full(absolute)pathname,arelativepathname,orasimplefilename.Eachuserhasahome directory in which personal files and directories can be kept. Files anddirectoriescanbecreated,moved,copied,listed,anddestroyed.Read,write,andexecute permissions are used to control file access by u (owner), g (groupmember),ando(others).Theownercansetandchangetheaccesspermissionsofafile.Youcancommunicatedirectlywithotherusersbyusingtalktochatdirectly,

byemail,andbyinstantmessaging(Skype,GoogleHangout,Pidgin).Linuxoffersseveral texteditors.Thefull-GUIgedit isagoodchoice.Fora

terminal window, the simple and easy nano is good for beginners and lightediting tasks. Serious editing is more efficient with an editor such as vim.Editing,compiling,andrunningofasimpleCprogramhavebeenpresented.Linux offers many facilities and a complete set of manuals. The man

command can be used to consult the manual pages, and the LinuxDocumentation Project website provides a variety of comprehensive Linuxdocumentations.RefertoSection1.17foralistofusefulcommandsforLinuxbeginners.

OnWindowsorDOSsystems,endoflineisindicatedbyreturnfollowedbynewline.

2.1

Chapter2

TheDesktopEnvironment

Itisimportantforanoperatingsystemtoprovideaconvenientinterfaceforuserstoperformandmanagetasks.Inthebeginning,Linux/UNIXsystemswereusedexclusively from the keyboard via the command-line interface (CLI) (3). Byaddinggraphicaluserinterfaces(GUIs),LinuxsystemsaremademoreintuitivetouseaswellaseasiertolearnforpeoplefamiliarwithMicrosoftWindows®orMacOS®systems.Agraphicaluserinterface(GUI)employsapixel-basedgraphicaldisplayand

addsapointingdevice,suchasamouse, touchpad,or touchscreen, to interactwiththeuser.ThefirsteffectiveGUIonanaffordablepersonalcomputer1wasintroducedbytheAppleLisaintheearly1980s.ADesktopEnvironment (or simplyDesktop) supports theGUIbyproviding

workspaces,windows,panels, icons, andmenus, aswell as clicking, scrolling,copy-and-paste, and drag-and-drop operations. Today, it is hard to imaginecomputerusersdoingwithoutadesktopGUI.Nevertheless,whenyoubecomemore of a Linux expert, you may find the CLI more convenient in manysituations.TherightapproachistocombineGUIandCLItogetthebestofbothworlds.

DESKTOPOVERVIEWAfter login at aworkstation, the first thingyou see is thedesktop fromwhichyou can launch applications, manage files, control your Linux system, andperform many other tasks. A desktop makes operating your computer moreintuitive through a desktop metaphor by simulating physical objects.Overlappingwindowscanbemovedandshuffledlikepiecesofpaper.Buttons(icons)canbepushed(clicked)toinitiateactions.Unlike MS Windows® or the Mac OS®, Linux offers a good number of

2.2

alternativedesktopswithahighdegreeofusercustomization.PopulardesktopsincludeKDEPlasma,GNOME3,Cinnamon,MATE,Unity,Xfce,andothers.Adesktop is often built on top of awindowing system, such as theX WindowsSystemorthenewerWayland.InadditiontoofferingacompletedesktopGUI,aLinuxdistributionoftenalsosuppliesasetofessentialapplicationsincludingaclock/calendar, sound volume control, email client, Web/file browser, instantmessenger, image displayer, media player, address book, PDF reader, photomanager,preference/configurationeditor,andmore.Agoodunderstandingofthedesktopandhowtouseiteffectivelycanmake

life onLinuxmuch easier. It is perhaps safe to assume that you already havegood working experience with MSWindows® or Mac OS®. Our discussionhereoftenfeaturestheGNOMEDesktop,oneofthemostpopulardesktopsforLinuxdistributions.OtherLinuxDesktopsfunctioninsimilarways.

Figure2.1LinuxMintMenuandPanel

DESKTOPCOMPONENTSAdesktopusuallydisplaysthefollowingcomponents:

RootWindow—After login, the entire graphical display screen is coveredby the rootwindow of yourDesktop. It is the spacewhere all otherGUIobjects (desktop components and application windows) are placed andmanipulated.DesktopPanel—ADesktopnormallydisplaysaPanelintheformofataskbaralongthetoporthebottomedgeoftherootwindow.ThePaneldisplaysimportant controls and the start icon which leads to the start menu foralmost all tasks, including checking system status and customization. APanelmayalsodisplayapplicationsthatarestillrunningsoyoucaneasilygo back to them. A Panel can display launchers to invoke specificapplications aswell aswidgets (small applications) such as a clockor anaudiovolumecontrol.SeeFigureforaMintPanelandstartmenu.YoumayalsobeabletoaddobjectstothePanelbyrightclickinganyemptyspaceinit.StartMenu—TheStartmenuisexposedbyclickingthestarticon(oftenintheformofalogoforGNOME,RedHat,Fedora,orUbuntudependingonyourLinuxversion)placedattheendofthePanel.ThekeyboardshortcutfortheStartmenuisusuallyALT+F1.FromtheStartmenu,youcanperformalmost all operations and can access files, the network, installedapplications, commands, andpreferenceoptions.Theremaybe additionalmenusonthePanel.NotificationArea—UsuallypartofthePanel,thenotificationareadisplaysstatusiconsforcertainapplications;forexample,aspeakervolumeicon,anetwork status icon, a system update available icon, an incoming emailicon, and so on. Mouse-over or click a notification icon to see moreinformation and available actions. The notification area is usually at therightendofthePanel.Application Program Windows—Each GUI application works within itsown window. Such windows are displayed in the root window as childwindows.Multipleapplicationwindowscanoverlap.Youcanswitchfromone app to another by changing the input focus (Section 2.4) from onewindow to another, as well as move, resize, maximize, minimize,unmaximize,orcloseeachwindowasyoulike.WindowsList—Alist of buttonsdisplayedon thePanel (Figure2.1)eachrepresenting an application wiondow in the root window. Clicking on awindow button minimizes or restores the window. On newer GNOMEDesktopspressingtheSUPERkey2revealstheactivityoverviewscreenwithlargericonsforallactivewindowsforeasyswitching(Section2.3).Moving

2.3

themousequickly to theupper-left cornerof theDesktopcanalso revealtheactivityoverviewscreen.Workspace Switcher—A workspace switcher enables you to work withmultiple workspaces and to switch from one workspace to another. Aworkspaceisessentiallyaduplicaterootwindowtoprovidemorespaceforplacingadditionalapplicationwindows.Withseveralworkspaces,youcanspreadoutyourapplicationwindowsforeasieruse.AworkspaceswitchercanbefoundonthePanelorbygoingtotheactivityoverviewscreen.Yourrootwindowcandisplayoneworkspaceatatime.Theworkspacevisibleisyour current workspace. Click on the switcher to change the currentworkspace.Desktop Objects—In the vast space left open by the Panel on the rootwindow, you often can place objects (icons) such as files, folders, andapplicationlaunchersforeasyaccess.Desktopobjectsusuallyinclude(1)aComputerorPlacesicontoaccessfilesandremovablemedia,(2)aHomeiconforyourhomefolder,and(3)aTrashiconforrecoveringdeletedfiles.Thesedesktopiconsrepresentfiles in theuser’s$Desktopfolder. Inotherwords,placingafileinyourDesktopfoldercancreateacorrespondingiconontheDesktop.

THEGNOME3DESKTOPRadicallydifferentfrompreviousdesigns,theGNOME3(version3.22released09/2016)DesktoptookaholisticapproachinGUIdesign.Asaresult,GNOME3works somewhat differently frommany traditional desktop environments. ItintroducedthegraphicalGNOMEShellastheoneplaceforuserstocontroltheGNOME 3 desktop, providing easy and efficient operations such as appfinding/launching,window/workspaceswitching,andfavoriteappaccessing.

Figure2.2GNOME3PanelwithShellExtensions

Underthetoppanel(Figure2.2),theGNOMEShellusestwofull-sizedisplayareas:

App Windows Display—This is the root window which now showsexclusivelyoverlapingwindowsofcurrentlyactiveapps.Beforeanyappislaunched,bydefault,you’llseeonlyemptyspacebelowthepanel.

2.4

Activities Overview—Showing a left-side Dash for launching favoriteapps, an app search box, iconified active app windows, and a right-sideworkspaceswitcher(Figure2.3).

Figure2.3GNOME3ActivityOverview

Movingthemousequicklytotheupper-leftcornerofthescreen(orpressingtheSUPERorWINDOWSkey)switchesbetweentheabovetwodisplays.TheGNOMEShellcanbeconfiguredandcustomizedinmanyways.Youcan

use the gnome-tweak-tool to configure settings and add Shell Extensions(Figure2.2). For example, you can enableDesktop icons to show files in theyourDesktopfolder.InGNOME3 theversatileShellExtensionsmechanism replaces applets for

addingwidgets andmenuson thePanel.For example, youhave extensions toprovideanApplicationsmenu,aPlacesmenu,aWeatherindicatorandsoontothetoppanel.A preferences widget is shown on the Panel for managing extensions. The

commandgnome-shell-extension-prefsservesthesamepurpose.Distributions that come with integrated GNOME 3 include openSUSE,

Fedora,UbuntuGnome,Debian,andLinuxMintCinnamon.

UNDERSTANDINGGUIA graphical user interface allows user control via mouse movements, buttonpressing, touchpad (touch screen) gestures, as well as key strokes from thekeyboard.Theseareknownasuserinputevents.TheGUIworksbymonitoringsuchevents,relatingthemtothegraphicalelementsdisplayedonthescreen,andprocessingeacheventquicklytoachievethedesiredeffects.

2.5

When launched, a GUI application, such as aWeb or file browser, usuallymakes a nice graphical display in a new window then does nothing exceptwaitingforuserinput.Whensuchaninputeventoccurs,theGUIappreactstotheevent,handles itquickly,andgoesback todoingnothing—beingreadyforthenextevent.Whenaninputeventtakesplace,theGUIenvironmentcreatesaneventobject

(a data structure) that contains the event attributes (for example, event type,mouse position, button id, or keycode) and delivers the event object to theprogramthatshouldreceiveandhandletheevent.Whenworkingwithmultiplewindows, one of them always has input focus

andisknownasthecurrentwindow.Allinputeventsaretobedeliveredtothecurrent window and the app running within it. The desktop allows a user toeasilychangethecurrentwindowbyclickingorselectingit.Acertainpartwithin thecurrentwindowmayactuallygain input focus.For

example,clickingononeofthreedisplayedbuttonssendstheeventtothebuttonbeing clicked and not the other buttons. In filling a form, key strokes aredeliveredtotheinputfield(suchasname,email,oraddress)thathasinputfocus.Usually clicking themousemoves input focus to the locationof the click andpressingtheTABkeymovesinputfocustothenextelementinsideawindow.Whenithappensthatthereisnoresponsetoyourinput,itisusuallybecause

yourinputissenttoaplacethatdoesnotunderstandit.Inotherwords,theinputfocusisatthewrongplace.Correctyourinputfocustofixtheproblem.

WORKINGWITHTHEDESKTOPOneof thebestways toget familiarwithadesktop is to learnhowtoachievespecifictaskswithit.HerearesometasksimportantforanewLinuxuser.

SessionControlandSystemSettingsYoucancontrolyourloginsessionbyclickingonthepowerswitchicononthePanel (usually toward the right side) allowing you to manage networkconnectedness,currentuser,soundvolume, lockscreen,session logout, restart,andsystemshutdown.Figure2.4showsLinuxMintsessioncontrol.

Figure2.4SystemControl

You can also controlmany other system settings such asmouse, keyboard,display, and devices (printers, scanners, webcam ...). Click on the time/datedisplaytosetit.Figure2.5showsasystemsettingsdisplay.

Figure2.5SystemSettings

LaunchingApplicationPrograms

Perhaps thesinglemost importantpurposeofadesktop is tohelpyourunandmanage application programs. Linux offers a large number of applications.You’ll findmany of them from the Startmenu organized into several generalgroupssuchasaccessories,office,graphics,Internet,programming,multimedia,games and so on. In addition, there aremany commands that can be invokedfromthecommandlineinaterminalwindow.Infact,youhavemultiplewaystostartapplications:

Using an app search to find and launch available apps. If an app is notinstalledonyoursystem,downloadandinstallitfromyourLinuxappstore.Singleclickingalaunchericon(Youmayaddalauncherforanyapplicationyoulike.)Single or double clicking an executable object/file, depending on yourpreferencesetting,displayedonthedesktoporafilebrowser.SelectinganapplicationfromtheStartmenuorasubmenuthereof.Issuing a command in a terminal window or from the command pop-updialogue(tryALT-F2).

Ifyouissueacommandforanappnotyetinstalled,yourLinuxmayevenoffertoautomaticallydownloadandinstallitforyou.Toinitiateagraphicalapplication,saygedit,fromthecommandlinewithout

the Shell waiting for it to finish or the Shell job control mechanism gettinginvolved,use(geditfilename&)This way, a subshell puts the graphical application in the background,

disassociates it from the terminalwindow, andgives youbackyour commandprompt.

ManagingFilesManagingfilesisimportantforanyuser.ClickonPlaces->Hometobrowsetheuser’s files/folders. Click on Places->Computer to browse files/folders for thewholesystem,includinginserteddrives(Figure).Eachuser’shomedirectoryoftencontainsthesestandardfolders:Documents,

Downloads,Music,Pictures,Videos,andTrash. IfyourLinuxserves theWeb,eachusermayalsohaveafolderpublic_htmlwhereper-userWebpagesreside.OtherfilesandfolderscanbesetupasyouworkonyourLinuxcomputer.FilesandfolderscanbemovedtoTrashandtheneventuallydiscardedwhen

youemptytrash(fromFilemenuofTrash).ItisalsopossibletoretrieveitemsfromTrash.TheTrashfolderiskeptwithinauser’shomedirectory,sometimes

underahiddenfolder(.local/share/Trashforexample).Right click a desktop object to rename it, move it to Trash, set its access

permissions,orotherwisemanageit.

Figure2.6APlacesMenu

MultipleWorkspacesThe desktop workspace can get crowded with multiple application windowsquicklywhenyouworkonyourLinux.Fortunately,youcansetupmore thanoneworkspaceandswitch fromone toanother toworkondifferent tasks.Forexample,youmayhaveoneworkspaceforWebbrowsing,anotherforemailandinstantmessaging,yetanotherfortexteditingandwordprocessing,andsoon.ForGNOME3basedsystems,aworkspaceswitchercanbefoundontheright

side of the activity overview. Use it to add, switch, and delete workspaces.Keyboardshortcuts,usuallyALT-UPandALT-DOWN,canbeusedtoquicklyswitchworkspaces.Asyougotoadifferentworkspace,theactivitydisplaychangestoshowthe

windowsinthenewcurrentworkspace.Inotherdesktopsystems,youmayaddaWorkspaceSwitchertoyourPanel.

Thiswill give you fourworkspaces by default.Right click on the switcher tocustomizeittoyourliking.

DesktopAppearanceYoucancustomizethelookandfeelofyourdesktopdisplay.Rightclickonanempty spot on your desktop working area and choose change desktop

2.6

backgroundtofindabackgroundyoulike.FortheGNOMEShell,youcanusetheGNOMETweaktool(gnome-tweak-

tool)tocustomizemanyaspectsoftheGNOMEDesktopappearanceincludingpickingfrommanywell-designedthemes.

WINDOWSTheXWindowSystemIn Linux/Unix, graphical applications use the X Window System (originallydeveloped at the Massachusetts Institute of Technology) to create GUIs.Windowing software suchasXenablespixel-basedgraphicaldisplays and theuse ofwindows,menus, and themouse to interactwith application programs.TheXWindowSystemworksasaGUIserver(theXserver)thatenablesclientprograms(Xclients)tointeractwithusersthroughGUIs.XclientscanconnecttothelocalXserverrunningonthesamehostoraremoteXserverrunningonahostacrossthenetwork.Furthermore,theXservercanhandlemultiplestations,eachpotentiallywithmultipledisplays.(Twoorthree20-inchLCDdisplaysforyourLinuxdesktopcomputer,anyone?)ForLinux,theXserverisbasicandisstartedwithinthebootsequence.Ifthe

X server is not running, no GUI programs will work. Figure shows the XWindowSystemarchitecture.

Figure2.7XWindowSystemArchitecture

When anX client starts, it needs to connect to anX server running on thelocalhostoraremotecomputer.TheXserverisalwaysspecifiedbythedisplayoption.Forexample,xclock-displayhostname:s.m

saysthexclockdisplaywillberenderedonhostname,stationnumbers,andmonitornumberm.Astationonacomputerconsistsofakeyboard,apointingdevice (mouse), andoneormoregraphicaldisplaymonitors.Acomputermayhaveoneormorestations,eachwithoneormoremonitors.IftheXserverislocal(onthesamehostastheclient),thehostnamepartcan

beomitted.Forasingle-displaycomputer,themonitor-stationdesignationwouldbe:0.0andcanusuallybeomittedalso.TheShell environmentvariableDISPLAYspecifies thedefaultX server for

anyclientprogramstartedwithoutanexplicit-displayoption.Trythecommandecho$DISPLAYtoseethevalue.Mostlikely,itwillbethestring:0.0.X11 has been in use for many years and experience told us that it can be

improvedinmanyrespects.Wayland,oneofthenewwindowingsystemsbeingdeveloped, aims to be compact, efficient, and better connected to hardware.Fedora is a Linux distribution that is moving to support Wayland as analternative toX11.However,at thispoint,X11 is still thestandardwindowingsystemformostLinuxdistributions.

WindowManagerYou control windows displayed on your desktop through awindowmanager.Thewindowmanageristhepieceofsoftwarethatcontrolsthedisplaywindowsin theXWindowSystem environment.The opening, closing, size, placement,borders,anddecorationsofanywindowaremanagedbythewindowmanager.TheXWindowSystemcallsforthewindowmanagertobeaclientratherthanabuilt-inprogram.InthiswayXcanworkwithmanydifferentkindsofwindowmanagers.Oneoftheoriginalwindowmanagersistwm.Your desktop environment works with the window manager to display

applicationwindowsonyourscreen(Figure).MutteristhedefaultwindowmanagerforGNOME3replacingMetacity.The

GNOME Shell (Section 1.3), implemented as a plugin forMutter, is speciallydesigned to work on large-screen workstations. Examples of the many otherwindow managers include i3, XMonad, AwesomeWM , Enlightenment andFluxbox.Youcaneasilyminimize,maximize/unmaximize,orcloseawindowusingthe

usualbuttonsonthetitlebar.Ifyoucloseallthewindowsofanapplication,thenthe application will be closed. Right clicking a window’s title bar usuallydisplaystheWindowMenufromwhichyoucancontrolthewindow.Awindowcanbemovedbyholdingontothetitlebaranddraggingitandcanberesizedby

2.7

holdingontoanddraggingasideoracorneroftheframe.Awindowisintheworkspacewhereitiscreatedbutcanbemovedtoanother

workspaceanytimeyouwish.Windowsinaworkspacemayoverlap.Clickingonawindoworitsactivitiesimageshiftsinputfocustothatwindowandbringsittothetop.Inadditiontoregularwindows,anapplicationwillsometimesdisplayadialog

window. Such popup windows are used to display alerts, to solicit userconfirmation,or toobtainuser input.Forexample,anapplicationmayaskifauser really wants to quit before closing. A dialog window can be modal ortransient.Amodaldialogwillnotallowyoutointeractwiththemainapplicationwindowuntilthedialogisclosed.Thus,youmustdealwithamodaldialogandcloseitbeforeyoucanresumeworkingwiththeapplication.

WindowInformationUnder XWindows, all windows form a containment hierarchy with the rootwindow sitting at the top. Each window has a unique window ID. The rootwindow’sIDisroot.ThecommandxwininfodisplaysthewindowIDandmanyotheritemsofinformationaboutanywindow.Runxwininfofirst,thenclickonanytargetwindowtogettheinformationdisplay.HereisasampleontheFirefoxwindow(Ex:ex02/xwininfo).

xwininfo:Windowid:0x1800010Absoluteupper-leftX:470Absolute

upper-leftY:64Relativeupper-leftX:10Relativeupper-leftY:

45Width:1448Height:984Depth:24Visual:0x304VisualClass:

TrueColorBorderwidth:0Class:InputOutput...Corners:+470+64

-2+64-2-32+470-32-geometry1448x984--8+19

NotethatthewindowIDisahexnumber0x1800010.Nowlet’stakealookatsomeusefulGUIapplicationsonLinux.

THEFILEBROWSERAnimportantaspectofanyoperatingsystemistheabilitytostoreandmanagefiles.TheLinuxfilesystemhasahierarchicalstructure.Eitheraregularfile,adirectory(folder),orahardwaredevice(special file) isconsidereda fileintheLinux file system. A directory (folder) is a file that records the names andattributesoffilesitcontains.Anyofthecontainedfilescan,inturn,befoldersaswell.TheLinuxfilesystemisallencompassingbecausealmosteverythinginLinux

hasarepresentationinthefilesystem.Chapter6discussestheLinuxfilesystem

indetail.Afilebrowser,orfilemanager,providesaGUItomanagefiles.Ithelpsyou

navigatetheLinuxfilesystemandmanageyourfilesvisually.Nautilus(Figure)isthefilebrowserthatcomeswithGNOMEShell.

Figure2.8NautilusFileManager

UseacommandsuchasnautilusfolderLocationto browser any desired folder. Special URIs (resource identifiers) such as

thesealsowork:nautiluscomputer:///nautilustrash:///nautilusnetwork:///nautilussmb:///OtherfilemanagersincludeKonqueror,Dolphin,GNUMidnightCommander,

andPCManFM. A file browser enables you to interactively navigate the filesystem,manage files and folders, access special places onyour computer, useopticaldrives,andreachavailablenetworkingplaces.

NavigatingtheFileTreeYoubrowsethefilesystembyfollowingthefolder-subfolderpathuntilyoufindyourtargetfile(s).Thus,atanygiventimeyouarelocatedatacurrentdirectory.Thecontentsofthecurrentdirectorycanbedisplayedinalistvieworaniconview,andyoucanswitchbetweenthemeasily.Theiconviewisrecommendedforeasiervisualinteractions.Double click a folder to open it and see the files in it, and click on the up

button tomoveup to theparent folderof thecurrent folder.TheLocationboxshows thepathname leading from the root directory / to the current directory.Normally,anyfilewhosenamebeginswithaperiod(.)ishidden.SelectView->ShowHiddenFilestorevealthem.

OpeningaFileorFolderDouble click a folder to open it and display its contents. Double click anordinaryfiletoopenitwiththedefaultapplication,forexample,PDFfileswithevince , .txt fileswithgedit, or .html files with your preferredWeb browser.Rightclickanordinaryfiletoopenitwithanyapplicationyouchooseandthatapplicationwillberememberedasapossibilitytorunthatparticulartypeoffile.Byrightclicking,youcanalsoelecttoremoveafile/foldertoTrash,toopenitwith a desired application, or to change its properties, including accesspermissions(seeSection1.7.5).

FindingFilesClickontheSearchbuttontogetaSearchbox.Typeastringofcharactersinthename or contents of the file(s) youwish to find and press ENTER. The searchresultswillbedisplayed.Iftoomanyfilesarefound,youcannarrowyoursearchbyfiletypeortimeofcreation(Figure).

Figure2.9NautilusFileSearch

ManagingFilesandFoldersFromthefiledisplayarea,youselectoneormorefilestomanage.Click(orCTRLclick)afileorfoldertoselectit.UseCTRLclicktoselectadditionalitems.Theselecteditemsarehighlighted.Iniconviewyoumaydragarectanglearoundagroupoficonstoselectthem.Makinganewselectioncancelsthepreviousselection.IfyouCTRLclickona

highlighteditemorclickonanemptyspotinthefiledisplayarea,theselectionisalsocanceled.Aftermakingaselection,youcanperformoperationsontheselectedfiles.

Draganddrop the selected items into a folder (or thedesktopwhich is afolderanyway)tomovethemthere.Grab the selection, then hold down ALT, and drag to a new folder andreleasethemouse.Thenselecttheoperationyouwish,includingmovehere,copyhere, or linkhere.A link is a shortcut or a pointer to an actual file(Section).Rightclickonyourselectiontoseetheavailableoperations.Theseincludemovingtotrash,openwith,copying,send to,andchangingfileproperties(name, permissions, list of applications to open files of this type, and soon).

*********************

AccessControlforFilesandFoldersOnLinuxafileisownedbytheuserwhocreatedit.Thefileownercansetthepermissionsforthefiletocontrolwhichuserswillhavewhataccessestoit.Users are alsoorganized intogroups.Auser canbelong tomultiplegroups.

Thefile/etc/groupslistsgroupsandgroupmembers.Thefileownercanalsosetthegroupattributetoanygrouptowhichtheownerbelongs.As a file owner, you can set the read (r), write (w), and execute (x)

permissionsforthreedisjointsetofusers: thefileowner(u),otherusersinthefile group (g), and all others (o). Each permission is independent of otherpermissions.Forexample,afilemayhavenopermissionsforu,butrandwforo.Itmaynotmakemuchpracticalsense,butitdoesdrivehomethepointoftheu,g,andopermissionsbeingindependent.Themeaningofeachpermissionisasfollows:

r—Permissiontoopenafileortoinspectcontentsofafolderw—Permissiontomodifyafileortocreateordeletefilesinafolderx—Permissiontorunafileasaprogramortoenterafoldertoreachfilescontainedinit

Youset/changepermissionsandmanageapplicationsassociatedwiththefilebyrightclickingthefileandselectingthepropertiesdialog(Figure2.10).

Figure2.10FileProperties

Therootusersaresystemmanagersandhaveallpermissionstoallfilesinthefilesystem,regardlessofanyfilepermissionsettings.

WritingCDsorDVDsTocreateadataCD,clicktheCD/DVDCreatoroptiononthegomenuorsimplyinsertablankdiscintotheopticaldrive.Thisleadstoaspecialfolder(burn:///).Dragfilesandfoldersyouwishtowrite to thedisc into theburnfolder.Whenready,clickontheWritetoDiscbutton.TocopyaCD/DVD,simplyinsertthediscintheopticaldrive,rightclickon

theresultingdesktopicon(orfinditfromtheComputericon),andchooseCopyDisc.Or you can launch aCD/DVD tool such asBrasero to perform the desired

operations.

ChangingYourDefaultFileBrowserThedefaultfilebrowserisusedautomaticallywhenyouclick/double-clickafile,a folder, Places, orComputer. If you like to use a file browser other than thecurrent default, you can change your default file browser by following thesesteps.

1. Downloadandinstallyourfavoritefilebrowser.Themostconvenientwayistouseapackagemanager(Section8.2).Ifyoudon’thaverootprivilegeasksystemadmintodoitforyou.

2. Find the installed file browser in /usr/share/applications. For example, ls

2.8

/usr/share/applications

mayshowtwoavailablefilebrowsersorg.gnome.Nautilus.desktoporg.kde.dolphin.desktopTochangeyourdefaultfilebrowsertodolphin,issuethiscommandxdg-mime default org.kde.dolphin.desktop inode/directory application/x-

gnome-saved-searchwhich sets Dolphin as the default application to open the file type

inode/directoryofanyfolder.

TERMINALWINDOWBecause Linux is based on UNIX and offers many programs that run undercharacter-based terminals, character terminal emulation programs are providedto run such applications. Popular terminal emulators includeGnomeTerminal,konsole,Terminator,Sakura,GnomeGuake,andthemorerecentTilix.Onsomesystems,thekeyboardshortcutCTRL+ALT+Tcanlaunchthedefaultterminal.Linux commands requiring a terminal window include bash (the default

Linux Shell), vim (text editor), ssh (secure remote login), sftp (secure filetransfer), and many other command-line applications such as ls (directorylisting),rm,mv,more,man(displayingmanualpages),andsoon.TheclassicxtermterminalemulatorisavailableonmostLinuxdistributions,

butusersnormallyprefertouseanalternativesuchasgnome-terminal.We will discuss the GNOME Terminal here. Other terminals work in very

similarways.

Figure2.11StartingaTerminal

StartingaGNOMETerminalAGNOMETerminal emulates a character-based computer terminal and allowsyou to run a Shell or command-line applications. Here is how it works. AGNOME Terminal emulates the xterm from the X Consortium which can, inturn, emulate the various DEC (Digital Equipment Corp.) terminals such asVT220andVT320.From the Startmenu, theApplicationsmenu or theActivity Dash you can

easilylaunchaGNOMEterminalwhichwillrunBashbydefault(Figure2.11).Withoutcustomization,theterminalattributesarespecifiedbyadefaultprofile.You can customize the size, font, colors, and other attributes for the terminalwindowfromtheEdit->CurrentProfilemenuoption.By creatingdifferent terminalwindowprofiles andgiving themnames, you

canusethemfordifferentinstancesofGNOMETerminalsyourun.Let’sassumethatyouhavesavedyourfavoritewindowprofileunderthenamemain.Becauseeasyaccesstoaterminalcanbesuchaconvenience,werecommend

that you add a terminal launcher to start your customized terminal window.Followthesestepstocreateanewapplicationlauncher:

1. Fromthecommandlinecd$HOME/.local/share/applications+2. Tocreateanewlaunchervimy-new-launcher.desktop3. Putinthefiletheselines

[DesktopEntry]Type=ApplicationEncoding=UTF-8Name=My-Gnome-

TerminalComment=definedmanuallybyPaulWangExec=gnome-terminal--

window-with-profile=monkey--title=Main-WindowIcon=utilities-

terminalTerminal=falseStartupNotify=true

The value given to the Exec attribute is the command invoked when thelaunchericonisclicked.Foradditionalexamplesoflaunchers,checkoutthe.desktopfilesinthefolder

/usr/share/applications.

TerminalWindowandtheShellWhenyoustartaterminalwindow,yourdesignatedShell(bashbydefault)willbetheapplicationrunninginthewindow.TheShellcanruninthewindowasaregularShellora loginShell.TheGNOMETerminalallowsyou tomake thischoiceaspartofthewindowprofilesetting.ThedifferenceisthataregularShell

readsonly theShell initialization file,whereas a loginShellwill also read theShelllogininitializationfile(Section3.13).Insomesituations,youmaywanttostartaterminalwindowtorunsomething

otherthanaShell.Forexample,gnome-terminal-e"ssh-Xpwang@tiger.cs.kent.edu"givesansshcommandtorunintheterminalwindowinsteadoftheShell.The

result is a GNOME Terminal connected to the remote host pwangtiger.cs.kent.edufortheuserpwangtologin.The terminalwindow closeswhen the application,whether a Shell or some

otherprogram,terminates.

Select,Copy,andPasteIt is generally a good idea to usewhat is alreadydisplayedon the screen andavoidtypingtheinformationagaintosavetimeandpreserveaccuracy.WiththeGNOMETerminal, andother text-displayingwindows such as aWebbrowserwindoworatexteditorwindow,youcanselect,copy,andpastetextwithease.

Select—Presstheleftmousebutton,clickacharacter,doubleclickaword,or triple click a line and then drag the mouse over a section of text tohighlightandselectthedesiredtext.Copy—Simply selecting the text copies it into a clipboard.You can alsoright click the highlighted text (or use theEdit->Copymenu option) toexplicitly copy the text into a copy buffer. Any previous content in theclipboardorcopybufferislost.Paste—Go to the target application, position the input cursor where youwish,andthenclickthemiddlemousebuttontopastefromtheclipboard.Oruse theEdit->Paste option of the target applicationwindow to pastefromthecopybuffer.

AGNOMETerminal remembersdisplayed text lines (500bydefault).Use thescrollbartogobackandforthonthetext.

WebandEmailLinksThe GNOME Terminal recognizesWeb and email addresses. For example, itrecognizeshttp://www.kent.eduandpwang@cs.kent.edu.Move yourmouse cursor over such an address and itwill be automatically

underlined, signaling that the GNOME Terminal has recognized the address.Rightclickandselect theOpenLink (SendEmailTo)option to launchaWebbrowser(anemailapplication)directly.Thisfeatureisveryconvenient.

2.9

2.10

2.11

The application launched in response to such usage is defined by yourPreferredApplicationssettingundertheStartmenu.

ACCESSINGHELPANDDOCUMENTATIONThe command gnome-help, or equivalently yelp, gives you easy access toinformation for the GNOME Desktop. A Web search will lead you todocumentationforyourparticularLinuxdistribution.Ingeneral, theHelpbuttononthemenubarofanyapplicationprogramwill

leadtodocumentationanduserguidesforthatparticularapplication.

SUMMARYLinuxGUI support is provided via theDesktopwhich has been evolving andimprovingsteadily.ModernLinuxdistributionsreceivewideracceptancedue,innosmallpart,togoodGUIandusefulGUIapps.GNOME and KDE are the two most widely used GUI environments for

Linux.Theybothrelyonanunderlyingwindowingsystemforgraphicaldisplayandwindowingsupport.KnowledgeandskillfuluseoftheGUIcanmakelifeonLinuxeasierandyoumoreproductive.GNOME3isthemostup-to-datedesktopthatispowerfulandpopular.Major desktop components are the Panel, Start button, apps menu, system

preferencesmenu,rootwindow,activeappwindows,theDash,appsearchbox,andtheworkspaceswitcher.The gnome-terminal can be customized and your settings can be saved in

profilesforreuse.AcomfortableterminalwindowcanmakelifeonLinuxmucheasier.TheGNOMENautilusfilebrowserprovidesavisualenvironmenttonavigate

the file tree and tomanage files and folders as well as their attributes. Otheruseful GUI applications include image processing, documentpreparation/viewing, audio-video playing, and creating CDs andDVDs.MoreGUIappsarediscussedinChapter4.

EXERCISES1. HowdoyoumoveyourPaneltothetoporbottomoftherootwindow?Is

thispossiblewithGNOME3?2. HowdoyoumakeyourPanelspantheentirewidthoftherootwindowor

12

becenteredatthetop/bottom?3. Findthelogouticonon/inthePanel.Isthereawaytoswitchuser?4. Findouthowtolockscreenandhowtoputyoursystemtosleep.5. Doyouhaveaworkspaceswitcherforyourdesktop?Ifnot,describehow

tocreateone.6. Isitpossibletoadd/removeworkspaces?How?7. Describehowtofindandlaunchappsonyourdesktop.Doyouhaveaccess

toan“appsearch"function?Howdoyoufindappsthatcanbeinstalledonyourdistribution?

8. Howdoesoneplaceananalogueclockonthedesktop?9. Describehowtoplaceapowermanagementlauncheronthedesktop.10. Whatiseog?Placealauncherforitonthedesktop.11. Whatisevince?Placealauncherforitonthedesktop.12. Settingupa terminalwindowcorrectlycanmakeabigdifference inhow

comfortableyouwillbeusingtheLinuxcommand-lineinterface.Considerthe command gnome-terminal –geometry=80x30+350+150? –window-with-profile=mainandexplainitsmeaning.Createanamedprofileofcolor,font,andotherpreferencesforyourself.Makeyourselfapanellaunchertostartyourcustomizedgnome-terminal.

13. On Linux, which GUI application can be used for MSWord and Exceldocuments?Installitonyourcomputerandplaceanicononthedesktop.

14. BurnadataCDanddescribeyourprocedure.Costabout$10Kin1983.AlsoknownastheWINDOWSkey.

Chapter3

InteractingwiththeBASHShell

Aswehavestated,inadditiontoGUIs,Linuxalsooffersefficientandpowerfulcommand-line interfaces (CLIs). In thebeginning,Unix/Linux systemshadnoGUI and only the CLI. Increasingly, many commands also have GUIcounterpartssotheyareeasiertouse.TheCLI,providedbyaprogramcalledaShell,remainsanimportantinterface

forLinuxusers.AShellnormallyrunsinsideaterminalwindowsuchasgnome-terminal or konsole (Chapter 1, Section 1.2). It takes input from the user(keyboard) and serves as a command interpreter to start applications and toperformallotheravailableoperations.ManyLinuxapplicationprogramscamefromUNIXandwerewrittenbefore

thegraphicaldisplaybecamestandard.OtherschosenottouseanyGUI.Thesecommand-line applications tend to be more efficient (less taxing on thecomputer,easiertocombinewithotherapplications,andsimpletoaccessacrossanetwork),butcanbeharder fornoviceusers.GUIapplicationsaregenerallymoreintuitivetolearnanduseinteractively,buttheycanbehardertocontrolorrunwithinotherprograms.WhenaccessingaLinuxsystemfromanotherhost,suchasaPC(Windowsor

Mac) orLinuxbox, through a remote login program such asSSH (Chapter 1,Section1.2)orTelnet,thefull-GUIofadesktop(Chapter2)ishardtoachieve,andtheShellisusuallytheonlyfeasibleuserinterfacechoice.WealreadyknowthatLinuxoffersanumberofdifferentShellsincludingSh

(theoriginalBourneShell),Ksh (theKornShell),Csh (theBerkeleyCShell),Tcsh(TCShell,animprovedCShell),andBash(theBourne-AgainSh).AusercanpickwhichShelltouse.AlthoughtheseShellsarecomparable,BashisthestandardandpreferredShellonLinuxsystems.WewillpresentinteractiveuseofBashinthischapter.ProgramminginBash

ispresentedinChapter5.

3.1

3.2

BASHDevelopedin1987fortheGNUProject(FreeSoftwareFoundation),BashisafreelyavailableShellbasedupontheoriginalSh(BourneShell,1978).TheBashShellincorporatesfeaturesfromSh,Ksh,Csh,andTcsh;addsnewfeaturessuchasShell-definedfunctions;andconformstotheIEEEPOSIX(pronouncedpahz-icksforPortableOperatingSystemInterface)specification.Today,BashisthemostpopularShellonLinuxsystems.Improvedversionsof

Bashhavebeenreleasedregularly.Normally,yourdefaultShell is /bin/bash.Ifnot,youcanalwayssetyourdefaultShellto/bin/bash(recommended)withthecommandchsh-s/bin/bashInaBashShell,thecommandecho$BASH_VERSIONdisplaysitsversioninformation.ItisagoodideatohaveyourBashreadyfor

experimentationwhenreadingthischapter.

INTERACTINGWITHBASHInside a terminal emulatorwindow, Bash serves as your command interpreterandcontinuallyexecutesthecommandinterpretationcycle:

1. Displaysaprompt2. Enablestheusertotype,edit,andenterthenextcommandline3. Breaks thecommand line into tokens (wordsandoperators)andperforms

well-definedShellexpansions,transformingthecommandline4. Carries out (by calling Shell-level functions) or initiates (by starting

externalprograms)therequestedoperations5. Waitsforinitiatedoperationstofinish6. Goesbacktostep1

The default prompt for Bash is $, but it can be customized to become moreuseful(Section3.9).Acommand lineconsistsofoneormorewords separatedbywhitespace or

blanks (spaces and/or tabs). Pressing the ENTER (RETURN) key completes inputtypingandsendstheShelltothenextstep.TheENTERkey(generatingaNEWLINEcharacter)completesacommandlineunlessprecededbyabackslashcharacter(),inwhichcasetheENTERisescaped(Section3.14)andbecomesablank.Thefirstwordinacommandisthecommandnameandindicatestheprogramtobeexecuted;theotherwordsareargumentstothecommand.Therearetwotypesof

commands: Shell built-in commands and regular commands. A built-incommandinvokeseitheraroutinethatispartoftheShell(cd,forexample)orafunctionoraliasdefinedbytheuser.Toexecuteabuilt-incommand, theShellsimplycallsup theappropriate subroutinewithin itself.A regular command isany other executable program in Linux that is not built into the Shell. Theseinclude system commands such as ls, rm, and cp, as well as your ownexecutableprogramssuchasa.out.Eachexecutingprogramisknownasaprocesscontrolledandmanagedbythe

operating system. Your interactive Shell is a process. The Shell spawns(initiates) a separate child process, known as a subshell, to execute a regularcommand. The distinction between built-in and regular commands is animportantone,asyouwilldiscover.A simple command is just the command name followed by its arguments.

Severalcommandscanbegivenonasinglecommandlineiftheyareseparatedbysemicolons(;).TheShellwillexecutethecommandssequentially,fromlefttoright.Twocommandsseparatedbyaverticalbar(|)formapipe(Section3.5).Theoroperator(||)andtheandoperator(&&)specifyconditionalexecutionofcommands:cmd1||cmd2(executescmd2onlyifcmd1fails)cmd1&&cmd2(executescmd2onlyifcmd1succeeds)Theseareexamplesofcompoundcommandswhereseveralsimplecommands

aregroupedtogethertoformasinglecommand.InLinux,acommandreturnsanexitstatusofzerowhenitsucceedsandnon-

zerootherwise.If you enclose one or more commands inside a pair of parentheses (), the

commandswillbeexecutedasagroupbyasubshell.After issuing a command, it is not necessary to wait for a prompt before

typinginadditionalinput.Thisfeatureisknownas typeahead.Whatyou typeaheadwillbetherefortheShellwhenitisreadytoreceivethenextcommand.YoualsocaninstructtheShellnottowaitforacommandtofinishbytyping

an AMPERSAND (&) at the end of the command. In this case, the Shellimmediately returns to process your next command, while the previouscommand continues to run detached from the Shell. Such detached processescontinuetoexecuteandaresaidtoberunninginthebackground.Forexample,firefox&will start the browser and return you to the Shell levelwithoutwaiting for

firefox to finish, which is not likely to be any time soon. Basically, theAMPERSAND instructs the Shell to skip step 5 in the command interpretationcycle.Abackgroundprocessalsogivesupreadaccesstothekeyboard,allowing

3.3

youtocontinueinteractingwiththeShell.Abackgroundprocesscanbereattachedto theShell—that is,brought to the

foreground—bythecommandfgjobidPleaserefertoSection3.6forjobIDsandjobcontrol.A foreground program receives input from the keyboard. If we bring the

firefoxjobtotheforeground,wecantypeaCTRL+Ctoabortit,forexample.Therecanbeonlyonerunningforegroundprogramatanygiventime.

COMMAND-LINEEDITINGANDCOMMANDCOMPLETION

Let’slookattypinginputtoBash.WehaveseeninChapter1(Section1.3)howthearrowkeystogetherwithDELETEandBACKSPACEcanbeusedtocorrectinputerrorsandtoreusepreviouscommands.Theseandothercommand-lineeditingfeaturesareprovidedbythereadlinelibrary.You, in fact, have a choice of usingvi oremacs (see the appendices at the

companionwebsite)foreditingthecommandlinewithset-oviset-oemacsIncaseofvimode,youwouldtypeESCtogetintothevicommandmodeand

then use vi commands to do any editing. When you are done editing thecommandline,pressRETURN(orENTER)toissuethecommand.While entering a command line, Bash helps you complete your typing in

varioususefulways.Basically, you engage the completion feature bypressingtheTABkey.Ifthereisauniquecompletion,itwillbedone.Iftherearemultiplewaystocompleteyourtyping,asecondTABwillrevealthechoices.Forexample,ifyouenterunfollowedbytwoTABs,alistofchoices

unaliasuniqunlinkunstrunameuniqleafunopkg...

willbedisplayed.Thetechniquenotonlysavestyping,butalsoshowsyoualltheBashbuilt-inandregularcommandswithagivenprefix,whichcanbeveryhandyifyouforgottheexactcommandnametouse.Some users prefer getting the choices listed directly with the first TAB by

puttingsetshow-all-if-ambiguousoninthereadlineinitfile/.inputrcStandardcompletionsperformedare

3.4

Commandnamecompletion—CompletingShellbuilt-incommands,aliases,functions,aswellasregularcommands;performedonthefirsttokenofthecommandlineFilename completion—Completing names for files; performed onargumentstoacommandUser name completion—Completing userids for all users on your Linuxsystem,performedonanywordstartingwithaHostnamecompletion—Completingdomainnames;performedonanywordstartingwith@Variablenamecompletion—CompletingnamesforexistingShellvariables;performedonanywordstaringwith$

Further, thebash-completionpackage (includedwithmostLinuxdistributions)enablesyoutoTAB-completecommonargumentstooften-usedcommands.Forexample,theargumenttothesshcommandsshpwang@mTABTABdisplayspwang@magicalmoments.infopwang@mapleglassblock.compwang@monkey.cs.kent.edupwang@mathedit.orgpwang@monkey.zodiac.cs.kent.eduOntopofthesecompletions,youcandefineyourownwiththeBashbuilt-in

completecommandwhichimplementsaprogrammablecompletionAPI.Seethecompletedocumentationfordetails.ThereadlineescapecharacterCTRL+Visusedtoquotethenextcharacterand

preventitfrombeinginterpretedbyreadline.Thus,togetaTABintoyourinputinsteadofinvokingthecompletionfunction,youwouldtypeCTRL+VfollowedbyTAB.Forexample,youcandefinetheCTRL+Laliaswiththefollowing:aliasCTRL+VCTRL+L=clear

BASHCOMMANDEXECUTIONThe first word of a command line is the command name. It can invoke aprocedurewithintheShell(inorder):analias(Section3.7),afunction(Section3.15),orabuilt-incommand.Ifnot,thenthecommandnameinvokesaregularcommandimplementedbyaprogramindependentoftheShell.In a regular command, the command name indicates an executable file and

canbe inoneof two forms. It canbe theabsoluteor relativepathnameof theexecutable file, or if the executable file is on the command search path, thesimple filename itselfwill suffice.Theprocedurebywhich theShell finds the

executablefileisasfollows:

1. Ifthecommandnameisanabsoluteorrelativepathname,thenthenameoftheexecutablefilehasbeengivenexplicitlyandnosearchisnecessary.

2. Ifthecommandnameisasimplefilename(containingno/character),theexecutable file is found by searching through an ordered sequence ofdirectoriesspecifiedbythecommandsearchpath.Thefirstfilefoundalongthissearchpathisused.

Iftheexecutablefilecannotbefound,orifitisfoundbuttheexecutepermissiononthefile isnotset, thenanappropriateerrormessage isdisplayed.Theerrormessagemostlikelywillbefilenotfoundorpermissiondenied.The Shell environment variable PATH (Section 3.10) defines the command

search path, a list of directories containing executable commands. The Shelllooks sequentially through thesedirectories for any commandyougiveon thecommand line. The PATH usually includes the system folders /bin, /sbin,/usr/bin,and/usr/sbin,wheremostsystem-suppliedexecutableprogramscanbefound. The search path can bemodified to include additional directories. Forexample,exportPATH=$PATH:/usr/local/bin:$HOME/binaddstwodirectoriesat theendofPATH:a/local/binwhereyouinstallextra

applicationstoyourLinuxandabininyourhomedirectory.1Now,youcanusea simple filename to run a program whose executable file resides in the$HOME/bindirectory.Bashusesahashtabletospeedupcommandsearchandonlyneedstosearch

through$PATH(andupdatethetable)whenacommandisnotfoundinthetable.Thebuilt-inhashcommandallowsyoutodisplayandmanipulatethistable(seehelphash).Thespecialperiodsymbol(.)isoftenplacedattheendofthesearchpathto

enable you to invoke any command in the current directory with a simplefilename.exportPATH=$PATH:.The built-in export command tells the Shell to transmit this value to the

execution environment (Section 3.10) that will be inherited by subsequentregularcommands.Becauseofaliasing(Section3.7),user-defined functions (Section3.15), and

commandsearch,thecommandactuallyexecutedmaynotbeexactlywhatyouintended.Tobesure,youcancheckbyissuingwhichcommand_nameto display the alias/function or the full pathname of the executable file

3.5

invokedbythecommand_name.Forexample,whichgnome-terminaldisplays/usr/bin/gnome-terminalOnceanexecutable filehasbeen found, theShell spawnsachildprocess to

runtheprogramtakingthesethreesteps:

1. Anew(child)processiscreatedthatisacopyoftheShell.2. Thechildprocess isoverlaidwith theexecutable file.Then thecommand

nametogetherwithanyargumentsarepassedtoit.3. The interactive Shell waits for the child process to terminate before

returningtoreceivethenextcommand,unlessthecommandhasbeengivenwith a trailing ampersand (&). If the command ends with &, the Shellreturnswithoutwaiting,andthecommandisruninthebackground.

BASHINPUT/OUTPUTREDIRECTIONUntilnow,ouruseofLinuxhasbeenlimitedtoissuingcommandsandobservingtheir output. However, you certainly will want results in amore useful form,either as hard copy or stored in a file. Furthermore,many instanceswill arisewhenyouwantinputtocomefromsomewhereotherthanthekeyboard,suchasafile,orperhapsevenfromanothercommandorprogramrunningconcurrently.Linuxprovidesanelegantsolution:input/outputredirection.When processing a command line, the Shell arranges any I/O redirections

beforeexecutingcommandscontainedinthecommandline.

StandardInputandOutputAs an operating system, Linux provides input and output (I/O) services forprocesses.Foreachprocess,asetoffiledescriptorsnumbered0,1,2,andsoonisusedforI/Otransactionsbetweentheprocessandtheoperatingsystem.Whena process opens a file or a device for I/O, a file descriptor is assigned to theprocess to identify the I/O channel between the process and the open file ordevice. When a new process is created, its first three file descriptors areautomaticallyassigneddefaultI/Ochannels.

File descriptor 0, the standard input or simply stdin, is connected to thekeyboardforinput.Filedescriptor1,thestandardoutputorsimplystdout,isconnectedtotheterminalwindowforoutput.

File descriptor 2, the standarderror or simply stderr, is connected to theterminalwindowforerroroutput.

Most CLI commands receive input from standard input, produce output tostandardoutput,andsenderrormessages tostandarderror.TheShell-providedI/OredirectioncanbeusedtoreroutethestandardI/Ochannels.

I/ORedirectionThespecialcharacters>,<,and|areusedbytheShelltoredirectthestandardI/OchannelsofanycommandinvokedthroughtheShell(Figure3.1).

Figure3.1I/ORedirection

Let’slookatasimpleexample.Thecommandlinels>filelistcreatesinyourcurrentdirectoryafilenamedfilelistcontainingtheoutputof

the ls command. The symbol > instructs the Shell to redirect the stdout of lsaway from the terminal screen to the file filelist. If a file by the same namealready exists, itwill bewiped out and replaced by a new filewith the samename,unlessyousetthenoclobberoptionwiththeBashbuilt-insetcommand

set-onoclobber(turnsonthenoclobberoption)set+onoclobber

(turnsoffthenoclobberoption)set-o(displaysalloptions)

Whenthenoclobberoptionison,redirectingoutputwith>toanexistingfilewill result in an error. This feature protects against accidental loss of a filethroughoutputredirection.Ifyoudomeantowipeoutthefile,addaverticalbar(|)afterthe>.Forexample,ls>|filelistMany users set the noclobber variable in their Bash initialization file

.bash_profile(seeSection3.13).Oneexceptionisthat/dev/nullisaspecialdatasink. Output redirected to it disappearswithout a trace. It is usefulwhen you

wishtodiscardoutputfromacommand.Thesymbol>>operatesmuchthesameas>,butitappendstotheendofa

fileinsteadofoverwritingit.Forinstance,catfile1>>file2appendsfile1totheendoffile2.Iffile2doesnotexist,itwillbecreated.So far, we have only talked about redirecting the standard output. But

redirectingthestandarderrorfollowsthesamerules,exceptyouneedtouse2>and2>> instead to explicitly indicate the file descriptor being redirected.Toredirectbothstandardoutputandstandarderror,use

someCommand>file2>&1(stderrjoinsstdoutintofile)someCommand

>file12>file2(sendstodifferentfiles)

Let’slookatanotherexample.cat>fileAfter giving this command, what you type on the keyboard (or copy and

paste) is put into file. Keyboard input is terminated by CTRL+D given at thebeginningofaline.Next,let’sconsiderredirectionofstdin.Usingtheoperator<,acommandthat

takesinteractiveinputfromthekeyboardcanbeinstructedtotakeinputfromafileinstead.Forexample,vitextfile<cmd-filewhere cmd-file contains commands to the vi text editor. Let’s say cmd-file

containsddZZthenthefirstlineoftextfilewillbedeleted.ManyLinuxcommandstakeinput

fromafileifthefileisgivenasanargument(sortfile,forexample);theusagesort<fileiscorrectbutunnecessary.

PipesInadditiontobeingabletoredirectI/Otoandfromfiles,youalsocanredirecttheoutputofoneprogramasinputtoanotherprogram.Theverticalbarsymbol(|)isusedtoestablishapipe,whichconnectstheoutputofthefirstcommandtotheinputofthesecond(Figure3.2).

3.6

Figure3.2APipe

Thus,ls-lt|morepipesthestandardoutputofls-lttothestandardinputofmore.Theresulting

commandiscalledapipeline.Sometimes,fornewusers,itishardtounderstandthedifferencebetween|and>.Justrememberthatthereceivingendofapipe|isalwaysanotherprogramandthereceivingendofa>or>>isalwaysa file.You can pipe the standard error together with the standard output using |&instead of |.More elaborate examples of pipelines are described inChapter 4,Section4.7.BashI/ORedirection

Table3.1summarizesBashI/Oredirection.Optionalpartsinthenotationareenclosedinsquarebrackets.

BASHJOBCONTROLOnthedesktop,weknowwecanrunmultipleapplications,each inadifferentwindow,andwecanswitchinputfocusfromonewindowtoanother.Within a single terminal window, the Shell also allows you to initiate and

controlmultiplecommands(calledjobs).Atanytimethereisonejobthatisinthe foreground and connected to the keyboard input for the terminal window.Other jobs are in the background. We already mentioned that if you add atrailing & to a Shell-level command, the Shell will run the job in thebackground.Hereisanotherexample.xclock(runsxclockinthebackground)Thenyoumaystartanotherjob,say,fortextediting,bythecommandnano-znotes.txtThis job is in the foreground, enabling you to control nano and perform

editing functions using the keyboard. At any time, you can type CTRL+Z to

suspendtheforegroundjobandgetbacktotheShelllevel.Ifyoudothat,thenyou’llsee2[2]+Stoppednano-znotes.txtand a new Shell prompt will appear in your terminal window to provide

confirmation that the current job has been suspended and will be in thebackground waiting to be resumed. Now you can issue any Shell-levelcommand, including one to start another job (which may itself be suspendedwithCTRL+Zinthesameway).Let’ssaythatyouthenstartathirdjob,gimppicture.jpgtodoimageprocessingonapictureandthensuspenditalso.Inthisway,itis

possibletostartthensuspendorputinthebackgroundquiteafewjobs,anditiseasytoseehowthiscanbecomeunmanageablequickly.Fortunately,ifyouissuetheShellbuilt-incommandjobsyou’llseeallyourjobsdisplayed[1]13519Runningxclock&[2]-12656Stoppednano-znotes.txt[3]+13520Stoppedgimppicture.jpgInthiscase,therearetwosuspendedjobswithjobnumbers2and3,andone

jobrunninginthebackgroundwithjobnumber1.TheShellalsoallowsyoutoresumeasuspendedjob,pullabackgroundjobintotheforeground,orkillajobentirely.To identifya job,a jobid isused,whichcanbegiven inanumberofways:

%job-number,%name-prefix,%+,and%-.Forexample,thejobids%3,%+,and%g all refer to same job in the preceding example.The job%+ is always themost recently suspended (the current job), and %- is always the next mostrecentlysuspended(thepreviousjob).The%-isusefulwhenyouaregoingbackand forth between two jobs.When using the name-prefix form, you need justenough prefix of the command name to disambiguate it from other jobs. Forexample,%vim,%vi,or%vallrefertojob2.Ajobcanberesumed(broughttotheforeground)bytheShell-levelcommandfgjobidYoucanabbreviatethecommandtojustjobid.Forexample,%1willbringjob

1totheforeground,%+(orsimplyfgbyitself)resumesthecurrentjob,and%-resumes theprevious job. Ifno jobid isspecified, themost recentlysuspendedjobwillbeactivatedandruninthebackground.If a background job produces output to stdout, it will be displayed in the

terminalwindowandinterferewithoutputfromanyforegroundjob.Further,if

thebackgroundjobrequiresinputfromtheterminal,itwillstopitselfandwaittobebroughttotheforegroundtoreceivetheinputitneeds.Thus,forjobstorunefficiently in the background, redirecting standard I/O to files usually isessential.Whenabackgroundjobterminates,theShelldisplaysamessagetonotifythe

user:[jobnumber]DonecommandasgivenThemessage isdisplayedafternormalcompletionof abackgroundprocess.

The following message is displayed when a background process terminatesabnormally:[jobnumber]Exit1commandasgivenToswitchasuspendedjobtoruninthebackground,usethecommandbgjobidSuspendingajobusingCTRL+Zisnotthesameasexitingorterminatingit.Itis

goodpracticetoexitalljobsproperlyandcloseallwindowsbeforeyoulogout.Each job provides its ownway for exiting (quitting); for example,CTRL+X fornano,:q!orZZforvim,qformutt,andexitfortheShell.Sometimes youmay need to force a program running in the foreground to

terminate.Thiscanbedoneby typing the interruptcharacter, usuallyCTRL+C,whichabortstheexecutingjobandreturnsyoutotheShelllevel.Iftheinterruptcharacterdoesnotstopyourprogramforsomereason,yourlastresortisthekillcommand.UseCTRL+ZtosuspendthejobandgettotheShelllevel,thentypekill-9jobidtoterminatethejob.Theoptionalargument-9instructskilltosendaspecific

signaltotheprocess,whichforcesittoterminate.SignalsaredescribedfurtherinChapter11,Section11.6.Inadditiontojobids,killcanalsotakeprocessnumbers.Thecommandjobs-lgivestheprocessnumbersforalljobs.Thekill commanddiscussedhere is built intoBash.There is also a regular

command,/bin/kill, thatcanbeused.Amongotherdifferences,/bin/killworksonlyonprocessnumbers.Table3.2listsusefuljobcontrolcommands.JobControlCommands

3.7

To sumup, a jobmay be in one of three states: running in the foreground,running in thebackground,orstopped(suspended).Nomore thanone jobcanrun in the foreground at any time, butmany jobs can run concurrently in thebackground. Many also may be stopped. To see the states of the jobs undercontrol of your Shell, use the command jobs. Use fg alongwith the jobid tobring a particular job from suspension or from the background into theforeground.Usethesuspendcharacter(usuallyCTRL+Z)tosuspendaforegroundjob.Usetheinterruptcharacter(usuallyCTRL+C)tokillaforegroundjob.Ifajobisstoppedorrunninginthebackground,itcanbekilledbyissuingthecommandkill[-9]jobid.Ifyougivetheexit(logout)commandwhiletherestillareunfinishedjobs,the

Shellwillremindyouofthefact.Itisbesttoterminateallunfinishedjobsbeforeexiting the Shell. However, if you insist by issuing an immediate second exitcommand, the Shell will abort all your unfinished jobs, and your terminalwindowwillclose.

BASHSHELLEXPANSIONSEachcommandlineundergoesanumberoftransformationsbeforeitisexecutedbytheShell.ThesetransformationsarecalledShellexpansionsandaredesignedtoprovidepowerandconveniencetotheuser.Forexample,youcanusels-l*htmlto see a listing of all files with a name that ends with html. This works

becauseofFilenameExpansion.Let’sseehowtheseexpansionswork.

Figure3.3BashExpansions

Bashtransformseachcommandbyapplyingthefollowingexpansions(Figure

3.3)insequence:

1. Historyexpansion—Allowsreuseofpartsofpreviouscommands2. Aliasexpansion—Replacescommandaliasesbytheirdefinitions3. Braceexpansion—Treatsexpressionswithincurlybraces4. Tildeexpansion—Expandsaprefixedwordtoacertaindirectoryname5. Variableexpansion—Replacesvariablesbytheirvalues6. String expansion—Interprets standard escape characters, such as n

(NEWLINE), r (RETURN), and t (TAB), in strings of the form $ ’xyz’; forexample,$’NametAgern’

7. Commandexpansion—Insertstheoutputofacommandintothecommandline

8. Arithmetic expansion—Includes results of arithmetic expressions in acommand(thisfeatureismostlyusedinShellscriptsandwillbecoveredinChapter5,Section5.11)

9. Processexpansion—Specifiesoutputproducedbyacommandtobecomeafilenameargumentforanothercommand

10. Filename expansion—Adds filenames to the command line by patternmatching

After all transformations, the resulting command line gets executed. You areencouraged to experiment with the expansions as you read their descriptions.The built-in command echo which displays the after-expansion state of itsargumentscanbeveryuseful.Byputtingtheecho infrontofacommandline,theeffectsofallbutaliasexpansioncanbeexamined.

HistoryExpansionTheBashhistorymechanism recordspreviouscommands foreasy reuse.Eachcommandlineissuedbyyou,whethersuccessfulornotandwhetherconsistingofoneormorecommands, iskeptasan item in thehistory list,whichcanbedisplayed using the built-in commandhistory. Each item in the history list isknownasahistoryevent,andeacheventisidentifiedbyasequencenumber.Thetotalnumberofeventskeptonthehistorylisthasalimit(defaultsto500)whichissetbyCommonHistoryExpansions

HISTSIZE=numberNormally, keeping 50 events or so is quite enough. Entering your own

HISTSIZEsettinginthe.bash_profilefile(Section3.13)makesgoodsense.WealreadyknowfromChapter1thatyoucanusetheupanddownarrowkeystogoback and forth on the history list and reuse previous commands.Furthermore,history expansion enables you to substitute history events into the currentcommandlinewithjustafewkeystrokes.Italsomakesmodifyingandreissuingpastcommands,orpartsofthem,easytodo.Historyexpansioniskeyed(activated)bytheexclamationpointcharacter(!),

anditworksbyrecallingitemsfromthehistorylist.Itemsthatcanberecalledfromthelistandsubstitutedintothecurrentcommandincludeanyhistoryevent,anywordorwordsofanyevent,andpartsofcertainwords.Theseitemsalsocanbemodifiedbefore their inclusion into the current command.Table3.3 showssome common history expansions. Table 3.4 contains some applications ofhistoryexpansionincommands.HistoryExamples

Eachexampleisdescribedhere,andthenumberscorrespondtothenumbersinTable3.4.

1. Reusethenamefile3.2. Nameturnsouttobeadirectory.3. Mistypedthecommandnamesort.4. Thedesiredfileisnotinthecurrentdirectorybutinthedirectorydir.

5. Thedirisnotinthecurrentdirectorybutinthehomedirectory.6. Notethatblanksareallowedinthestringreplacement.

Havingseenanumberofexamples,youarereadytoproceedtothegeneralformofahistoryexpansion:event[:worddesignator][:modifier…]Theeventisgiveninoneofthefollowingways:

Eventnumber !12givesevent12onthehistorylist.

Relativeposition

!-2givesthesecondmostrecentevent.Aspecialcaseis!!,whichreferstothelastevent.

Commandprefix !nanogivesthemostrecenteventprefixnano.

Matchingstring

!?string?givesthemostrecenteventcontainingstringanywherewithintheevent.

str1str2 Repeatsthelastcommand,butwithstr1replacedbystr2.Followingtheeventaretheoptionalworddesignators.Thepurposeofaword

designator is to choose certain words from the history event. If no worddesignators are used, the entire event will be selected. The following worddesignatorscanbeused:Anoptionalsequenceofmodifiersalsocanbeused.Onefrequentusageisevent:s/xx/yy/tosubstitute thestringxxbyyy inevent. If aword isa longpathname, it is

sometimes convenient to use a modifier to extract a portion of it, but mostmodifiers are seldomly used interactively. Writing programs in the Shelllanguage(Shellprocedures)isdiscussedinChapter5,andatthatpointyouwillbe able to seewhymodifiers are needed.A number ofmodifiers are listed inTable3.5;refertotheBashmanualforacompletelist.Onceacommandlinehasgone through history expansion, it too becomes part of the history list as themostrecentevent.HistoryModifiers

TheBashbuilt-incommand fc (fix command)puts a rangeof history items

intoyour favorite text editor, allowsyou tomodifyanyparts atwill, and thenexecutestheresultingcommandsautomaticallywhenyouexittheeditor.fcfirst_eventlast_eventFinally, when you are finished interacting with it and exit, Bash saves the

command history to the history file specified by the environment variable$HISTFILE,whichdefaultsto.bash_historyinyourhomefolder.NexttimeyoustartBash,thesavedhistorywillberestoredfromthehistoryfile.ThehistoryfilefeaturecanbedisabledbyexportHISTFILE=

AliasExpansionThe alias feature allows you to define shorthands for often-used commands,making themeasier toenter.Tocreateanalias (anysingleword)andgive itavalue(acharacterstring),usetheBashbuilt-incommandalias.Thenotationaliasname=value…defines the given name as an alias for the specified string value. Multiple

name-valuedefinitionsareallowed.ThevaluepartoftenrequiresquotesaroundittopreventunwantedShellexpansions(seeSection3.14forwhenandhowtousequotes).Herearesomesimplebutusefulaliasdefinitions.aliasdir="ls-l"back=’cd$OLDPWD’aliasmonkey="ssh-lpwangmonkey.cs.kent.edu"aliasappend2end="cat>>"Withthesealiasesdefined,thecommanddirworksbecauseitexpandstols-l.

Thealiasbackworks itsmagicbecause theBashvariable$OLDPWDalwaysholdsontothepreviousworkingdirectory.Aliasexpansionmeansthatifthefirstwordofasimplecommandisanalias,

Bash will replace that first word with the alias value. The first word of thereplacementtextisagaintestedforaliases,butawordthatisidenticaltoanaliasbeing expanded is not expanded a second time. This means the following iscorrectanddoesnotresultinaninfiniteloop.aliasls=’ls-F’Thus, the ls command always is given with the -F option, which causes,

among other things, directory names to bemarkedwith a trailing /, symboliclinks to bemarkedwith a trailing@, and executable files (fileswith executepermission;seeChapter1,Section1.6)tobemarkedwithatrailing*.Todisplayexistingaliases,use

alias(displaysallaliases)aliasname(displaysthealias)

Toremovealiasdefinitions,useunaliasname...

BraceandTildeExpansionsBraceexpansionprovidesashorthandforsimilarwordson thecommand line.Withbraceexpansion,thecommandlinenanomemoSep,Oct,Nov2018.txtbecomesnanomemoSep2018.txtmemoOct2018.txtmemoNov2018.txtandlprchap2..5.pdfbecomeslprchap2.pdfchap3.pdfchap4.pdfchap5.pdfThesequencenotation(..)worksfornumbersandsingleletters,forexample,

a..z.ThecharacterTILDE ( )expands to theuser’sownhomedirectory, useridto

thehomefolderofsomeotheruser,+tothecurrentfolder,and-tothepreviousfolder.Thus,thealiasbackearliercanalsobedefinedasaliasback="cd-"

VariableExpansionTheShell allows the use of variables, also known as parameters.A variable’svalue is a character string. Some variables are reserved for Shell use. Forexample, USER, HOME, PATH, and HISTSIZE are Shell variables havingprescribedmeaninginBash(seeSection3.9).Inaddition,youcanalsosetanduseyourownuser-definedvariables.Generallyspeaking,avariableidentifiercanbeanywordwhosefirstcharacter

isaletterandtherestconsistsofletters,digits,andunderscorecharacters.Usevar=value(setsvariablevalue)to assign a value to a variable. The value can be a singleword ormultiple

wordsinquotes,andnowhitespaceisallowedimmediatelybeforeoraftertheequalsign(=).Afterbeingset,avariablecanbeusedinsubsequentcommands.Forexample,ldir=/usr/localgivesthevariableldirastringvalue/usr/local.Withthisvariableset,youcan

inputcd$ldirwhich is a command with a variable in it. After variable expansion, this

commandbecomes

cd/usr/localAsyoucansee,variableexpansioniskeyedbythecharacter$.Thatis,aword

thatbeginswitha$ isavariable.If$ isfollowedbyablankorprecededbyabackslash(),thenitstandsforitself.Theechocommandcanbeusedtodisplaythevalueofavariable.Forexample,echo$ldirdisplays/usr/local.Useunsetvartoremoveanyvariablevar.The extent of a variable name can be delineated by braces ( and ). For

example,x=abcecho$xdedisplaysthestringabcde,whereasecho$xdedisplaysanemptylinebecausethevariable$xdehasnovalue.Variables often have string values. However, they may also have integer

values.Inside$((...)),youmayperformintegerarithmeticoperations(including+-*/+%**++–)onvariablesusingC-languagesyntax.Forexample,

count=7echo$((3*count))(displays21)echo$((count%5))

(displays2)echo$((count++))(displays7,setscountto8)

You can display variables (Shell built in and user defined) and functiondefinitions(Section3.15)with

set(displaysallvariablesandfunctions)declare(displaysall

variablesandfunctions)declare-f(displaysallfunctions)

CommandExpansionCommandexpansionmakesitpossibletousethestandardoutputofacommandas a stringofwords in another command.Either $(command) or ‘command’(notetheBACKQUOTE)canbeusedforcommandexpansion.Forexample,

dir1=$(pwd)(ordir1=`pwd`)

assignstheoutputofthepwd(printworkingdirectory)commandtotheuservariabledir1.Anotherexample,files=$(ls)assignstofileswordsproducedbythelscommand,namely,thefilenamesin

the current directory. The substitute string of a command expansion also canformpartofasingleword,asinfile1=$(pwd)/test.c

The substitute string isnormallybroken into separatewords atblanks, tabs,andNEWLINEs,withnullwordsbeingdiscarded.

ProcessExpansionBash extends the ideas of I/O redirection one step further by allowing thenotation

<(commandargs...)

tobeusedwhereafilenameargumentisexpectedforacommand.Thus,thenotation < (...) produces a temporary file, with the output produced by thecommandinside,whichcanbegiventoanothercommand.Forexample,

nano<(ls-l-F)

opensnanotoview/edittheresultsproducedbythegivenlscommand.Thisabilitycanbehandysometimes.Itispossibletosupplymultiplefilesinthisway.Forexample,

diff-u<(ls-F/usr/bin)<(ls-F/usr/bin.old)

displaysthedifferencesbetweenthetwodirectorylistings.

FilenameExpansionBecause command arguments often refer to files, the Shell provides filenameexpansion tomake it easier to specify files.When a filename pattern orglobpattern is used in a command line, the pattern is expanded to become all thefilenamesmatching the pattern.A patternmaymatch simple filenames, in thecurrentworkingdirectory,aswellasfullorrelativepathnames.Ifapatterndoesnotmatchanyfile,thenitstandsforitselfandisnotexpanded.Globpatternsarespecifiedusingthespecialcharacters*,?,and[].The*matchesanysequenceofzeroormorecharacters.Forexample,

ls-l*.c

producesa listingofall fileswithanameending in .c.The*.c isapattern,and it expands tomatch all filenames in the currentworking directory endingwith.c.Thecommand

ls-l../*.c

doesthesameforfilesintheparentfolder.Thecommand

ls~/Pictures/2018*/*.jpg

conveniently displays a listing of all pictures, ending in .jpg, under folderswithanameprefix2018,inthe/Picturesdirectory.Filenamepatternsarematchedagainstexistingfilenames.Rulesforfilename

patternsareasfollows:

*Matchesanycharacterstringoflengthzeroormore(the

“wildcard”).?Matchesanysinglecharacter.[…]Matchesanyoneof

thecharacterscontainedbetween[and](arangepattern).For

instance,a[rxz]bmatchesarb,axb,orazb.Thepatternchapter[0-9]

matcheschapter0,chapter1,andsoon.[^…]Matchesanycharacter

notin[and].The!charactercanbeusedinsteadof^.[[:class:]]

Matchesanyinaclassofcharacters.Theclasscanbealnum

(alphanumeric),alpha,digit,lower,orupper.

Forexample,inthecommand

ls[[:digit:]]*

thepatternmatchesallfileswhosenamestartswithadigit.Filename expansion is also known asglobbing. Filename expansion can be

deactivatedwiththeBashbuilt-incommand

set-f(or-onoglob,filenameexpansionoff)set+f(or+onoglob,

filenameexpansionon)

FilenameexpansionshouldnormallybeonwhenusingtheShellinteractively.The character . at the beginning of a filename must be matched explicitly

unlessthedotgloboptionisset.

shopt-sdotglob(enablesmatchingleadingdot)shopt-udotglob

(disablesmatchingleadingdot)shopt(listsBashoptions)

Hence,thecommandls*normallydoesnotlistanyfileswhosenamebeginswithadot.Additionally,thecharacter/inafilenamemustbematchedexplicitly.Afilenamepatterncancontainmore thanonepatterncharacter.Whenmore

thanone filename ismatched, the pattern is expanded into a sorted list of thematched filenames. Matching is case sensitive unless you do shopt -snocaseglob. If a pattern matches no filenames (match failure), then it is notexpanded(staysunchangedinthecommandline)unless

3.8

3.9

shopt-sfailglob(matchfailurecausesanerror)shopt-snullglob

(matchfailureexpandstoemptystring)

BASHBUILT-INCOMMANDSWehaveseenanumberofBashbuilt-incommands.Afewmoreareintroducedinthissection.ToseealistofallBashbuilt-incommands,youcanusethebuilt-inhelp.

help(listsallbuilt-incommands)helpcommandName(describesthe

givencommand)helphelp(tellsyouhowtousehelp)

Bashmaintainsadirectorystackthat,bydefault,containsthecurrentworkingdirectory.The built-inpushddir changes to the given directory and pushes itontothestack.Thebuilt-inpopdchangestothetopdirectoryonthestackafterpoppingitoffthestack.Thus,thesequencepushddirpopdbringsyoubacktowhereyouwerewithoutchangingthedirectorystack.The

built-indirsliststhefoldersonthestack.While interactive inputusuallycomesfromthekeyboard, it isconvenient to

save and edit commands in a file and then ask the Shell to execute thosecommandsfromthatfile.TheBashbuilt-incommandsource(orsimplyadot.)canreadafileofBashcommandsandprocessthemonebyone.AfileofShellcommandsisknownasaShellscript.Thus,eitherofsourcescript.scriptcauses your interactive Bash to read commands from the given script as

though they were entered from the keyboard individually. Since source is abuilt-incommand,thescriptisnotreadbyasubshell(Section3.2).

SHELLVARIABLESBashusesanumberofspecialvariables,withalluppercasenames,forspecificpurposes.SettingspecialvariablescontrolsthewaycertainBashoperationsarecarriedout.Forexample,settingtheCDPATHtoalistofoften-useddirectoriesenables you to use simple folder names with the cd command (cdsimpleFolderName).BashwillthensearchforthetargetfolderunderdirectoriesontheCDPATH.Besuretoincludethe.ontheCDPATH.SomevariablesthataffectinteractiveuseoftheShellarelistedhere.Otherspecialvariablesaffecting

3.10

theprocessingofShellscriptsarediscussedinChapter5.

ENVIRONMENTOFAPROGRAMThe exact manner in which a program works depends on the executionenvironment withinwhich it is supposed to do the job. For example, the texteditornanoorvimneedstoknowthecapabilitiesoftheterminalemulatoritisdealingwith,andsodoesthecommandmore.Thecurrentworkingdirectoryissomethingalmostallprogramswillwanttoknowwhentheyrun.Forfileaccesspermissionpurposes,anyprogramthataccessesfilesneedstoknowtheuseridoftheuserwhoinvokedit.Theexecutionenvironmentofeveryprocessconsistsoftwo parts: user defined and system defined. The userid, current workingdirectory,openfiles,etc.aredeterminedbythesystemandpassedonfromyourShelltoanyinvokedapplication;whereasquantitiessuchasthehomedirectory,thecommandsearchpath,andthedefaulteditoraredefinedbytheuser.Theseare known as environment variables. Many applications use certain specificenvironment variables of their own; for example, DISPLAY for any GUIapplication, CLASSPATH, and JAVA_HOME for the Java compiler,MOZILLA_HOMEforFirefox,andEDITORformutt.

CommandExecutionEnvironmentA principal task of a Shell is to launch applications by interpreting usercommands. When Bash launches an application, it creates a child process(anotherrunningprogram)andtransmitstoitanexecutionenvironment(Figure3.4)thatincludesthefollowingattributes:

StandardI/OandotheropenfilesCurrentworkingdirectoryFilecreationmask(Section3.12)Environment variables already in the Shell’s own execution environmentandadditionalonesdefinedbytheuser

A child process (an application, for example) is said to inherit its initialenvironmentfromitsparentprocess(theShell,forexample).Anychangesintheenvironmentofthechildprocessdoesnotaffectthatoftheparent.

3.11

Figure3.4ExecutionEnvironmentofaProcess

LetXYZbeanyvariable.YoucanmakeitpartoftheShell’senvironmentbyexportXYZthereforemakingitavailabletoanychildprocesstheShellinitiateslater.Ifa

variable is unset, then it, of course, is also removed from the Shell’senvironment.Insteadofexportingandthenunsettingavariable,youcanaddvariablestothe

environment on a per-command basis.When you issue any regular command,you can set variables in front of the command name to add them to theenvironment passed to the command without affecting the environment of theShellitself.Forexample,ifwestartasubshellwithYEAR=2018bashThesubshellwillhaveanenvironmentvariableYEARsettothevalue2018

whileyourShellremainsunchanged.TheenvironmentvariableTERMrecordstheterminaltype.ForLinuxusers,

TERMismostlikelysettoxterm(XTerminal)byaterminal-windowprogramsuchasgnome-terminal(Chapter3,Section2.8).Thecommandsearchpathisanother environmental parameterwhose value is contained in the environmentvariablePATH.Also,XWindowsclientprogramsusethesettingofthevariableDISPLAY (Chapter 2, Section 2.6). The Bash built-in commandprintenv (orenv)displaysallcurrentlysetenvironmentvariablesandtheirvalues.Hereareafewmorecommonenvironmentvariables.TERM TypeofterminalEDITOR DefaulttexteditorDISPLAY XserverandphysicaldisplaydevicedesignationMANPATH SearchpathforthecommandmanRemember,inBashanyvariablecanbecomeanenvironmentvariablebythe

export command.However, it isgoodpractice touse alluppercasenames forenvironmentvariables.

EXAMPLESOFBASHUSAGE

By studying examples, you can gain a deeper understanding of how the Shellworksandhowthevariousexpansionscanbeused.Almostallexamplesgivenhereareofpracticalvalue,andyoumayconsideradoptinganyorallofthemforyourownuse.

CustomizedPromptTheShelldisplaysapromptwhenitisreadyforyournextcommand.ForGNULinux, thedefaultBashprompt isPS1=’ s- v$ ’,meaning -Shell_base_name-version$.Forexample,-bash-3.2$Thetrailing$isautomaticallyreplacedby#iftheuserisroot.Manyuserschoose tocustomize theprompt todisplaymore information.A

goodexampleisPS1=’u@h:W$’which specifies userid@hostname:current_folder history_number$ and

produces,forexample,thepromptpwang@acerwang:ch03361$You may also set the special variable PROMPT_COMMAND to any

command to be executed before displaying each prompt. See the Bashdocumentationformoreinformationonsettingtheprompt.

RemovingFilesSafelyDeleting files accidentally is not unusual. This is especially true with thepowerful and terse notation of the Shell. It is entirely possible tomistype thecommand

rm*.o(deletesallfileswiththe.osuffix)asrm*.o(deletesall

filesandthefile.o)

byaccidentallytypinganextraSPACEinfrontofthe.o.Itisrecommendedthatyoudefineanaliasaliasrm="rm-i"The -i option requires interactive confirmation before deleting any file.

Considerplacingthisaliasinyour.bash_profile(Section3.13).Someuserspreferanevensaferalternative,movingunwantedfilestoatrash

folderratherthanactuallydeletingthem.YoushouldalreadyhaveaTrashfolderinyourhomedirectoryoryoucancreateonewithmkdir/TrashNow, define a function rm (Section 3.15) that usesmv tomove any given

filesto/Trash(seeExercise20).

Copy,Paste,andI/ORedirectionYoucancombinecopy-and-paste(usingthemouse,seeChapter2,Section2.8)with I/O redirection to make certain operations easier. For example, you canmarkandcopydisplaytextcontaininginformationyouwishtosaveandenteritdirectlyintoafile.Justtypecat>notes.txtandpastethemarkedlinefollowedbyCTRL+Donanewline(tosignalendof

inputtocat).Tomailsomescreenoutputtoanotheruser,simplydocat|mailuserid-ssubjectandthenpastethematerial.

DisplayingManualPagesEach manual page provides a concise description of a Linux command. Themain body of the manual pages is divided into chapters. Although the exactorganization of the chapters may vary with the particular Linux system, thefollowingisatypicalorganization

1. User-levelcommands2. LinuxsystemcallsintheClanguage3. SystemlibraryfunctionsforC,Fortran,networking,andotherpurposes4. Specialfiles,relateddevicedriverfunctions,andnetworkingsupport5. Formatsforexecutableandsystemdatabasefiles6. Miscellaneoususefulinformation7. Linuxmaintenance,operation,andmanagement

Youcanusethecommandmanmanto see the organization of yourmanual pages.To display an introduction to

chaptern,typemannintroTodisplaythemanualforcommand_name,typeman[n]command_namewherethechapternumbernisoptional.A typical manual page contains the following information: NAME (and

principal purpose), usage SYNOPSIS, DESCRIPTION, OPTIONS, relatedFILES,andSEEALSO(relatedcommands).Ifthemanualpageistoolargetofitononescreen,theprogramwilldisplay

onepageatatimeuntiltheentireentryhasbeenshown.YoucantypeqtoquitmanandreturntotheShellprompt.Thisisespeciallyusefulifthemanpageislargeandyoudon’twanttoseeitall.TheSYNOPSISpartofthemanualpagegives a concise description of the command syntax. In the synopsis, certaincharactersaretobetypedliterallywhenusingthecommand;othercharactersorstringsaretobereplacedbyappropriatewordsorfilenamessuppliedbytheuser.Portionsenclosed inbracketsareoptional,and thebrackets themselvesarenotpartofthecommand.Ellipses(…)areusedinthesynopsistoindicatepossiblerepetitions.MostLinuxcommandsreceiveoptions thatmodify thebehaviorofthe command. As mentioned earlier, an option is usually given as a singlecharacterprecededbyadash(-),butmoreverboseoptionsarealsopossible.TheFILESsectionof themanualpagegives the locationsof files related to

the particular command.TheSEEALSO section gives related commands thatmay be of interest. The BUGS section lists some known problems with thecommand.Thecommandmanalsocanperformakeywordsearchthroughthenameand

purpose part of themanual pages, displaying each line containing any of thegivenkeywords.Thisisdonebyusingthe-koptionman-kkeyword...Thisfeatureisusefulwhenyouwanttodosomething,butcan’trememberthe

appropriatecommand.Forexample,tofigureouthowtocopyafile,youcouldtryman-kcopy.Thekeywordcanbeanycharactersequence.Soyoucanfindacommandifyourememberonlyapartofitsnameordescription.There are also Web page versions of the Linux man pages (for example,

linuxmanpages.com)thatcanbemucheasiertouseasareference.AlsoyoumayusetheYelpdocumentbrowseryelp’man:name_of_command’to conveniently view anymanual page. For example, Figure 3.5 shows the

displayofyelpman:chmod.

3.12

Figure3.5BrowsingManpage

SettingUpYourPersonalWebFolderOften,theLinuxsystematschoolortheofficewillalsoservetheWeb.Ifso,theLinuxsystemoftenalsosupportsper-userWebpages.Thismeansyoucansetupapublic_htmlfolderinyourhomedirectoryinthefollowngway:

cd(goestohomedirectory)chmoda+x.(allowsWebserver

access)mkdirpublic_html(createsnewfolder)chmoda+xpublic_html

(allowsWebserveraccess)

Now you may create Web pages (filename.html) in your public_html andmakeeachoneWebreadable:chmoda+rpublic_html/filename.htmlYoucanthenaccessthemovertheWebwiththeWebaddresshttp://hostname/your_userid/filename.html

DEFAULTFILEPERMISSIONSFileprotectionwasdescribedinChapter1,Section1.6.Whenyoucreateanewfile, Linux gives the file a default protectionmode.Often, this default settingdenieswritepermissiontogandoandgrantsallotherpermissions.Thedefaultfileprotection setting iskept in a systemquantityknownasumask.TheShellbuilt-in command umask displays the umask value as an octal number. Theumask bit pattern specifies which access permissions to deny (Chapter 11,Section11.4).The positions of the 1 bits indicate the denied permissions. Forexample, theumaskvalue0022(octal022)hasabitpattern000010010,anditspecifies denial ofwrite permissions for g ando.TheShell built-in command

3.13

umaskalsosetstheumaskvalue.Forexample,umask0077setstheumasktodenyallpermissionsforgando.Ifyoufindyourselfusing

chmodgo-rwxalot(Chapter2,Section2.7),youmightwanttoconsiderputtingumask0077intoyour.bash_profileand.bashrcfiles(Section3.13).

SHELLSTARTUPANDINITIALIZATIONAsmentioned,theShellitselfisauserprogram.Thetermuserprogramreferstoprogramsnotbuilt into theLinuxoperatingsystemkernel.Examplesofkernelroutines are file system routines, memory management programs, processmanagementprograms,andnetworkingsupport.Thecommandsls,nano,mail,andcat,aswellasShellsbash,csh, andsoon,areuserprograms. In fact, allLinuxcommandsareuserprograms.TheloginShellisselectableonaper-userbasisandisspecifiedintheuser’s

passwordfileentryinthepasswordfile/etc/passwd.Thisfilecontainsaone-lineentryforeachauthorizeduseronthesystem.Eachpasswdentryconsistsofthefollowingfields:

Loginname(containsnouppercaseletters)EncryptedpasswordorxNumericaluseridNumericalgroupidUser’srealname,office,extension,andhomephoneUser’shomedirectoryProgramtouseastheShell

The fields are separated by colons (:). For example, a passwd entrymay looklikethefollowing:pwang:x:500:500::/home/pwang:/bin/bashThexpasswordindicatesthatashadowpasswordfileisusedtobetterprotect

andmanageuserpasswords.The/bin/bashat theendspecifies theuser’s loginShell.Immediately after a login window starts, the user’s login Shell is invoked

(Chapter2,Section2.8).The loginShell specified in thepasswdentry canbechangedusingthecommandchsh(changeShell).Forexample,chsh-s/bin/bashwillchangeyourloginShellto/bin/bash.AttheShelllevel,thecommandecho$0

displaysthenameofyourcurrentShell.When a Shell starts, it first executes commands in Shell initialization files,

allowingaLinuxinstallationandindividualuserstocustomizetheShelltosuittheirpurposes.ExactlywhichinitializationfileBashloadsdependsonhowitisinvoked.

LoginBash—Ifbashisinvokedviaaloginwindoworgiventheoption-lor –login, then it is a login Shell. As a login Shell, Bash first loads thesystem-wide initialization file /etc/profile which defines environmentvariablessuchasPATH,USER,HOSTNAME,andTERM.Thenitloadsaper-user initialization file which is the first of .bash_profile, .bash_login,and .profilefoundin theuser’shomedirectory.Theper-userclean-upfile.bash_logoutisexecutedwhenaloginBashexits.Non-logininteractiveBash—WhenBashisrunfromthecommandline,itisaninteractiveShell(withstandardI/Oconnectedtotheterminalwindow)butnota loginShell.SuchaBash loads thesystem-wide /etc/bash.bashrcfirstandthenloadstheper-user/.bashrc.Non-interactiveBash—Bashstartedtorunacommand(bash-ccmd)orascript (Chapter 5) is non-interactive. Such a Bash does not load any initfiles by default. It will load a file specified by the environment variableBASH_ENV.

There are some differences among Linux distributions on Shell initializationfiles. For example, CentOS/Fedora/Red Hat also provides the system-wide/etc/bashrcfileforuserstoloadifdesiredwithaconditionalexpression:

if[-f/etc/bashrc];then./etc/bashrcfi

Notethatthe.commandisthesameassource.WritingBashprogramsisthetopicofChapter5.Amongotherthings,the/etc/bashrcusuallysetstheumasktoadefaultvalue

(Section3.12). It is agood idea to include /etc/bashrc ifyour systemprovidesone.Hereisasample.bashrcfile.

#Sourcesystemdefinitionsif[-f/etc/bashrc];then.

/etc/bashrcfiset-onoclobberumask0007

The.bashrcisusuallyincludedinthe.bash_profile,whichaddsothersettingsimportant for interactive use of the Shell. Figure 3.6 shows a sample.bash_profile(Ex:ex03/bash_profile).

3.14

Figure3.6ASample.bash_profile

A non-interactive Bash is a subshell, and the execution of any Bash script(Chapter 5) involves a subshell Bash. Therefore, the setting for aliases,functions, and PATH used for Shell procedures ought to be placed in .bashrcinsteadofin.bash_profile.

SHELLSPECIALCHARACTERSANDQUOTING

TheShell usesmany special characters in establishing the command languagesyntaxandaskeysforthevariousexpansionsprovided.Someoften-seenspecialcharactersarelistedinTable3.6.BashSpecialCharacters

SpecialcharactershelpachievemanyShellfunctionalities.However,becausetheShell interpretsaspecialcharacterdifferentlyfromaregularcharacter, it isimpossible for a special character to stand for itself unless additional

arrangementsaremade.Forexample,ifthereisafilenamedf&g.c,howcanyourefertoitinaShellcommand?Thesolutiontothisproblemistheuseofmorespecialcharacters,knownasquotecharacters.IfyouaregettingtheimpressionthattherearemanyspecialcharactersinLinux,youareabsolutelyright.Infact,any character on the keyboard that is not alphabetic or numeric is probablyspecial in someway.Notable exceptions are the period (.) and the underscore(_).

QuotinginBashBash provides the backslash ( ) escape character, single quotes (‘...’), doublequotes(“...”),andANSI-Cquotes($‘...’).Thecharacterquotesorescapesthenextcharacter.Forexample,nanof&g.candgrepUS$report.*Thecharacters&and$losetheirspecialmeaningwhenprecededbyİnstead,

theystandfortheliteralcharactersthemselves.Ifaspaceortabisprecededbyathenitbecomespartofaword(thatis,itlosesitsspecialmeaningtodelineatewords). If theNEWLINE character isprecededbya it isequivalent toablank.Thus,usingaattheendofalinecontinuestheShellcommandtothenextline.Togetthecharacterwithoutescapingthenextcharacter,use.Whereas theescapes thenextcharacter,apairofsinglequotationmarks(’)

quotestheentirestringofcharactersenclosed.

echo´a+b>=c*d´

When enclosed by single quotation marks, all characters are escaped. Thequotedstringformsallorpartofaword.Intheprecedingexample,thequotedstringformsonewordwiththespacesincluded.Thecommand

cat/user/pwang/´my>=.c´

is used to type out a C program in the file /user/pwang/my > =.c. In thisexample, thequotedstringformspartofaword.To includeasinglequotationmarkinastring,theisused,asinechoIt’sagooddayThefollowingrulessummarizequotationwithsinglequotationmarks:

1. All quoted characters, including are taken literally. Thus, escaping thesinglequotewithbackslashwithinasingle-quotedstringdoesnotwork.

2. Thequotedstringformspartorallofoneword.

3.15

Sometimes it is desirable to allow certain expansions within a quoted string.Quotingwithdoublequotationmarks (") serves thispurpose.Apairofdoublequotation marks functions the same as a pair of single quotation marks withthreedifferences:

First, variable and history expansions are performed within doublequotationmarks;thatis,variableexpansionkeyedbythe$signandhistoryexpansions keyed by the ! signworkwithin double quotationmarks. Forexample, echo "your host name is $HOST" echo "Last command is !-1"workasexpected.Second, command expansions are allowed inside double quotationmarksand are treated slightly differently from normal command expansions.Normally, the output of a command expansion, via $(...) or ‘...‘ (Section3.7),isbrokenintoseparatewordsatblanks,tabs,andNEWLINEs,withnullwords being discarded; this text then replaces the original backquotedstring. However, when command expansion is within double quotationmarks,onlyNEWLINEsforcenewwords;blanksandtabsarepreserved.Thesingle,finalNEWLINEincommandexpansiondoesnotforceanewwordinany situation. For example, date=‘date‘ and datestring="‘date‘" aredifferentinthat$dateincludesmultiplewords,but$datestringisoneword.Third, escaping " with backslash within a double-quoted string works.Actually,withinadouble-quotedstring,thebackslash()escapesonly$,‘,", orNEWLINE.Within a double-quoted string, the combination escapeshistory expansion, but the backslash is not removed from the resultingstring.

Now,westillneedaneasywaytoincludehard-to-keyboardcharactersinstrings.ThisiswheretheANSI-Cquotesareuseful.Astringintheform$’str’allowsyou to use ANSI-C escape characters in str. For example, you can useBACKSPACE),f(FORMFEED),n(NEWLINE),andsoon.Forexample,alias$’f’=cleardefines a convenient alias, allowing you to clear your terminal screen by

typingCTRL+Lasacommand.

SIMPLEFUNCTIONSYoucantakeahard-to-entercommandorasequenceofcommandsforacertaintask and build a function tomake the task easy.To define a function, use thesyntax

functionfnName(){command1;command2;...commandn;}

A command in a function can be a Shell built-in command, a regularcommand, or a call to another function.Aliases don’twork inside a function.Eachcommandinthefunctiondefinitionmustbeterminatedbyasemicolon.Somewhitespacebetweentheandcommand1isnecessary.Oncedefined,youcanuse the functionnameas a commandnameandalso

passthefunctionarguments.Forexample,

functionoffice(){/usr/bin/libreoffice$1;}

defines the function office.You can then invoke libreoffice on a documentwiththecommandofficenote.docThespecialvariable$1inthefunctiondefinitionreferstothefirstargumentin

the function call. In general, the positional parameters $1, $2, ... are used toaccessargumentspassedinafunctioncall.Infact,thekeywordfunctionisnotnecessaryif()aregiven.Forexample,

dir(){ls-lF--color=auto--color=always"$@"|less-r;}

givesyouaDOS-likedircommand.3Thespecialvariable$@referstoalltheargumentsinthefunctioncall(Chapter3,Section5.3).AfunctionisnormallynotinheritedbychildShellsunlessitisexportedwith

export-ffunctionName.Youcanremoveafunctionwithunset-ffunctionNameanddisplayallfunctionswithdeclare-fThereisnobuilt-incommandtodisplayaspecificfunction,butthefollowing

functionwilldothejob

functionwhich(){(alias;declare-f)|\/usr/bin/which--tty-only

-i\--read-functions$@;}

Thepairofparenthesesaround(alias;declare-f)groupscommandsjustlike,except it calls for a subshell to execute the commands. The stdout of thatsubshellisfedtothe/usr/bin/whichcommand.Withthisfunctiondefined,thecommandwhich fnamewillnowdisplayany

aliasor functiondefinitionfor thegiven fname. If there isnosuch functionoralias,itwillalsolookforfnameon$PATH.Thespecialvariable$@evaluatestoall the arguments passed to the function. Also note we used /usr/bin/which

3.16

3.17

instead of just which because it is not our intention to call the functionrecursively.4Hereisthedisplayproducedbywhichwhich.

which(){(alias;declare-f)|/usr/bin/which--tty-only-i\--

read-functions--show-tilde--show-dot$@;}

MorewillbesaidaboutfunctionsinChapter5,Section5.18.

FORMOREINFORMATIONYoucanusetheBashcommandhelp|moretogetalistingofbuilt-incommandsandhowtogetmoredetailsonthem.TheBashmanpagemanbashisagoodreferenceontheBourne-AgainShShell.TheBashManualfromGNUcanbefoundatwww.gnu.org/software/bash/manual.

SUMMARYRunning in a terminal window, the Bash Shell provides a CLI to your Linuxsystem.You interact with the Shell via the input-processing-execution-promptcycle.Thecommandlinegoesthroughawell-definedsetofexpansionsbeforegettingexecuted.AShellbuilt-incommandiscarriedoutbytheShellitself.Anon-built-inorregularcommandinvolveslocatinganexecutableprograminthefile system, running it in a child process, and passing to it any command-lineargumentsandanyenvironmentvalues, includingexportedShellvariablesandfunctions.Acommandnamecanbeeitherasimplenameorapathname.Intheformer

case,thecommandmayinvokeaShellaliasorfunctionifoneexists.Otherwise,thecommandisfoundbysearchingthroughthecommandsearchpath—alistofdirectoriesgivenbytheenvironmentvariablePATH.I/Oredirectionenablesyoutodirectthestdin,stdout,andstderrofcommands

to/fromfilesandothercommands(formingpipes).Jobcontrolmakesitpossibletostartmultipletasks,suspendthem,puttheminthebackground,orbringanytotheforegroundtoreassertterminalcontrol.Entering of input is helped by input editing, TAB-completion, history

3.18

substitution,andfilenameexpansion.Bash loads initialization files at start-up time. It is important to keep your

favoritesettingsintheappropriateinitfiles.bashrcand.bash_profile.TheShellusesmanyspecialcharacters,suchas*,=,(),[],blanks, ;,andso

on. Quoting with single and double quotes and character escaping with arenecessarytocountertheeffectsofsuchcharacters.Thisisespeciallyimportanttorememberwhenissuingcommandsthatrequiretheuseofsuchcharacters.Bash also supports function definition. A function becomes a new built-in

command. A function can take arguments and access them as positionalparameters.IfyoulikeShellaliases,you’lllovefunctions.MoreaboutfunctionscanbefoundinChapter5.

EXERCISES1. ThecommandcdisbuiltintotheShell.Whycan’titbeimplementedasa

regularcommand?2. Findanddescribeawaytodoakey-wordsearchoftheLinuxmanpages.3. WherecanyoufinddocumentationforacommandbuiltintoBash?4. Consider thespecialdirectorysymbol .and its inclusionon thecommand

search path ($PATH).What difference does it make if you do or do notinclude .? If youdo include .,where should it be placed relative to otherdirectorynamesonthesearchpath?Why?

5. You have written a program that takes input from stdin and writes it tostdout.Howcouldyourunthisprogramifyouwantedinputtocomefromafilenamedinandoutputtobestoredattheendofafilenamedoutandanyerrortostderrberecordedinafilenamederrlog?

6. Whatifyouwishtohavestdoutandstderrsenttothesamefile?7. Johnwantedtoappendthefilefbtotheendofthefilefa,sohetypedcatfa

fb>|faWhatreallyhappenedhere?Howwouldyoudoit?8. Johnthenwantedtosendaline-numberedlistingoffilefatotheprinter.He

typedcat-nfa>lprbutnoprintoutappeared.Why?Whathappenedhere?9. Johnmadeatyposrotfile1file2SpecifytwowaysusingtheShellhistory

mechanismtocorrectsrottosortandreissuethecommand.10. HowdoesonesettheeditorusedinBashcommand-lineediting?Showthe

code.11. NameatleasttwocommandsthatarebuiltintoBashbutalsoareregular

Linuxcommands.12. Give a command to edit, using nano, every file in the current directory

1234

whosefilenameendsin .txt thatcontainsthestringLinux.(Hint:considerthe-loptionofgrep.)

13. What isa foreground job,background job,andsuspended job?Howdoesonedisplayalistofalljobs,orswitchfromonejobtoanother?

14. HowdoyouexitfromyourinteractiveShell?Specifyatleastthreeways.15. WhathappensifyouexitfromyourShellandthereareunfinishedjobs?16. Explain the difference between these two commands: ls chap[0-9] ls

chap0..917. What is command expansion in Bash? Give the two notations used for

commandexpansion.18. WhatisstringexpansioninBash?Explainandgivetwoexamples.19. Try

country="usa";echo${country^^}

FindoutmoreaboutBashcasemodificationinvariableexpansion.Explainandgivetwoexamples.

20. Consider the twoBash initializationfiles: .bashrcand .bash_profile.Whatinitializationcommandsshouldbekeptinwhich?Why?

21. What is the syntax for function definition in Bash? After defining afunction,canyouundefineit?How?

22. InBash,whatarepositionalparametersofafunction?Howdoyouexportafunctionintotheenvironment?Whatgooddoesitdo?

23. WriteaBashfunctionrm tomoveitsargumentfilestothe/Trashfolder.(Hint:Usemv-i.)

24. Explainthecodeman()yelp"man:$@";andgiveusageexamples.25. Find the Linux version running on your computer. (Hint: The uname

command.)ThevalueoftheShellvariable$HOMEisthefilenameofyourhomefolder.NotethatnanoignoresCTRL+Zunlessgiventhe-zoption.NotethatLinuxalreadyhasaregularcommanddirforlistingdirectories.IfyourBASHShellcomeswithanaliasforwhich,unaliasitsoyoucanreachthefunction.

Chapter4

PuttingCommandsandApplicationsto

Use

Linuxoffersarichsetofcommandsandapplications,todoalmostanythingyoudesire.EffectiveuseofLinux involvesknowingwhatappsandcommandsareavailable.Through software searchandpackagemanagement, apps, aswell assystemsoftware,canbeeasilyfound,installed,updated,andremoved.Furthermore, existing commands can be combined easily to form new ones

eitheronthecommandlineorinShellscripts(Chapter5).WewilldiscussasetofoftenusefulGUIappsandCLIcommands.We’llalso

show you how to combine commands into new commands, and selecting therightcommandstoapply.Throughoutthisbook,wewillintroducemanyusefulLinuxcommandsanddemonstratehowtheycanbeputtogooduseindividuallyandincombination.Many commands are filters. A filter usually performs a simple and well-

defined transformation of its input and follows certain conventions tomake iteasy to connect to other programs. Filters can be strung together using pipes(Chapter3,Section3.5)tobecomepipelinesthatcanperformcomplexfunctionsondata.Manyusefulfiltersarepresentedinthischapter.Examplesshowhowtobuildpipelinesinpractice.For instance, the command sort is a filter that orders its input lines. The

commandtrtranslatesspecificcharactersintheinputintoothercharacters.Youcan combine these two filters with others to create and maintain a simpledatabaseofaddresses.Utilizingandprocessinghumanreadabletextualdatahavebeenanemphasis

ofLinux.Withintextualdata,weoftenneedtoidentifytheexactplaceswheretransformationsormanipulationsmust takeplace.Regularexpressions provide

4.1

standard ways to specify patterns in textual data. It is important to becomefamiliarwithregularexpressionsbecausetheyoccurfrequentlyandarebasictoprogramming. We explain the regular expression notations and how they areusedinapplicationssuchasgrepandsed/vi.

USEFULGUIAPPSLinuxoffersalargenumberofapps.Usersjustneedtoknowwhat’savailable,makesureanappisinstalled,andthenlearntouseit.GUI tools for finding, installing and managing apps, such as GNOME

Software (gnome-software) and Ubuntu Software Center (software-center)whereappsaregroupedintosearchablecategories,makeappmanagementeasy(Figure 4.1). In addition, there are also the command-line oriented DNF andAPTpackagemanagers(Chapter8,Section8.2).

Figure4.1GNOMESoftware

Wewilltalkaboutafewappsthatcanoftenbehandy.

WordProcessingForschoolorwork,wordprocessingisafrequenttask.OnLinuxwedon’thavethe expensive Microsoft Office but we do have Apache OpenOffice andLibreOfficethatarefree.Thelibreofficecommand(Figure4.2)comesinstalledonmanyLinuxdistributions.

Figure4.2LibreOffice

Use the free software forwordprocessing, spreadsheets, slidepresentations,drawings, editing PDF files, import and export documents from/to differentformatsincludingtext,PDFandMicrosoftformats.To view PDF on Linux you can also use Okular, Evince, or Foxit Reader.

Often evince is pre-installed and the default viewer for PDF. To select, split,mergeanddeletepagesofaPDFfile,considerPDFmod.Also,qpdfcanselectand combine pages from one or more PDF files as well as add passwordprotectiontoPDFfiles.

DocumentFormattingandTypesettingTo prepare larger documents such as technical papers and even books, theLaTeX system is often helpful and may even be required by scientificconferences and publishers. LaTeX is a high-quality typesetting system and amarkuplanguagethatallowsyoutouseplaintext tomarkupdocument layout,font,color,andmore.Forexample,thefilemydoc.texEx:04/mydoc)

\documentclass{article}\title{ASampleLaTeXDocument}\author{Paul

S.Wang}\date{2018-09-

01}\begin{document}\maketitle\section{Introduction}Moretexthere

...\section{Background}Moretexthere...\end{document}

canbeprocessedbythecommand1pdflatexmydoc.textoproducemydoc.pdfasdisplayedinFigure4.3.

Figure4.3ALaTeXProducedDocument

In addition to all the usually expected document formatting capabilities,LaTeXexcelsinfeaturessuchas

ExperttypesettingofmathematicalformulasAutomaticnumberingofchapters,sections,pages,tablesandfiguresAutomaticcrossreferencingGenerationoftableofcontentsandindices

InfactthisverytextbookhasbeensetinLaTeXusingatemplateprovidedbythepublisher.HereishowtoinstallLaTeX(chooseeithertexliveortexstudio):dnfinstalltexlive-scheme-fullapt-getinstalltexlive-scheme-fulldnfinstalltexstudioapt-getinstalltexstudio

DrawingandDiagrammingTools for drawing points, lines, arrows, rectangles, circles/ellipses, and othercurves to form diagrams often use vector graphics where coordinates, angles,anddistancesareusedtoformdrawings.Unlikerastergraphicswherepixelsareusedtoformshapes,vectorgraphicscanscaledrawingsupordown(zoominorout)withoutlosingclarity(becomingpixelated).TheLibreOfficeDrawdoesaverygood jobasadrawinganddiagramming

tool. The dia (Figure 4.4) is a Microsoft Visio like program very good formakingmanykindsofdiagrams.

Figure4.4DiagrammingwithDia

Xfig is another simple and efficient tool for diagram drawing. Inkscape(Figure4.5)isapowerfultool,notunlikeAdobeIllustratororCorelDraw,givingyoutheabilitytocreatebeautifuldrawingsbasedonvectorgraphics.

Figure4.5Inkscape

ThesK1isacapablevectorgraphicstooltopreparehighqualitydrawingsforprofessionalpublishing.Flowisanotherappforflowcharts,networkgraphs,andsoon.Asymptote isavectorgraphics languageandsystemallowingyou tospecify

drawings using plain textual instructions and producing graphics in files ofdifferent formats (Postscript,PDF,SVG,3DPRC)aswell as forviewing.Forexample,thetextfileline.asy(Ex:ex04/line.asy)

draw((0,0)–(50,50));says todrawa straight line from theorigin to (50,50).The file can thenbe

processedbythecommandasy:asy-Vline(producesline.epsanddisplaystheresult)asy-fpdfline(producesPDFfileline.pdf)

RasterGraphicsandImageProcessingOnLinux you can use eog (Eye of Gnu) for viewing photos and shotwell tomanagethem.Perhaps the best known and most widely used raster image creation and

processing application on Linux is GIMP (the GNU Image ManipulationProgram). Comparable to Adobe Photoshop, GIMP is feature rich and verycapable. Additionally, GIMP is designed to be augmented with plug-ins andextensions.Figure4.6 shows thegimp commandbeingused todesign the topbannerforthisbook’swebsite.GIMP can also directly create an image to process by taking a screenshot.

Alternatively you can use the PRINTSCREEN key (for the entire screen),ALT+PRINTSCREEN(forthecurrentwindow)andSHIFT+PRINTSCREEN(foracursor-selectedarea).The result image isplaced inyourPictures folder.Thegnome-screenshotcommandgivesyoumorecontrolovermakingscreenshots.Alsotheimportcommandcanbehandyforscreencaptures.

Figure4.6ApplyingGIMP

ForscanningyoumayusetheeasySimpleScan(simple-scan)orthefancierXSane(xsane).

FileUploadandDownloadForfileuploadanddownloadfromthecommand linewecanuse ftpandsftp(Section5.20).

Wealreadyknowthatnautilussupportsbothlocalandremoteaccessoffiles(Section2.7).FileZillaisaGUItoolforFTPandSFTPwhichcanbeeasiertouseforbeginnersandoccasionalusers.InstallFileZillawithdnffilezillaapt-getfilezilla

Figure4.7FTPToolFileZilla

andinvokeitwithfilezilla(Figure4.7).

PasswordManagerWeallhave toomanyaccountsanddevicesrequiring login toaccess.Keepingall those userids and passwords safe and easy to access becomes a problemneedingasolution.You can use a Web browser’s auto-login feature, where your browser

remembersyouruseridsandpasswordsfordifferentwebsites.But,youmustfirstset a browser master password to protect the saved login information fromotherswhomay gain access to your browser. Select your browser’s advancedsecurityoptiontosetitsmasterpassword.There are a number of password manager programs designed to store and

retrievepasswordssecurely. IfyouareusingGnome, theSeahorse tool,whichcansavenotonlypasswordsbutalsoencryptionkeys(Chapter7,Section7.10),isusuallyalready installed.Look forApplications>Utilities>Passwords andKeysorgivethecommand

seahorse

Figure4.8TheSeahorseTool

to launchSeahorse(Figure4.8).SelectLoginandclick the sign toaddnewentries.Click the lock icon to lock/unlockstored information.Makesureall islocked before you quit Seahorse. To unlock something in Seahorse, use yourLinuxloginpassword.Anotherablepasswordmanageriskeepassx2.Toinstalldodnfinstallkeepassxapt-getinstallkeepassx

CloudStorageStoringfilesandfoldersinthecloud(onserversovertheInternet)cangiveyouadditionalstoragespace,makesharingfileswithotherseasy,andprovideawaytobackupimportantfilesforsafety.ForLinux,oneparticularcloudstorageservicestandsout,namelyDropBox.A

freeaccountgivesyou2GBofstorage that isaccessibleon theWebandfromyourlocalfolder$HOME/Dropbox.InstallingDropBoxissimple:dnfinstalldropboxapt-getinstallnautilus-dropboxOnceinstalledyoucanlaunchDropBoxwiththecommanddropboxstart-iwhichwill leadyou to install theDropBoxdaemon (dropboxd) and to link

yourLinuxwithyourDropBoxaccount.Adaemonisaprogramrunninginthebackground ready to instantly provide a specific service. A Linux systemnormallyhasmanydaemonstosupportmanyusefulservices.

Now you can use your $HOME/Dropbox folder which is automaticallymirrored on your DropBox cloud storage. Read the Getting Started guideincludedinthesamefolderforusageinformation.Manyothercloudstorageoptionsareavailable forLinux includingAmazon

S3,GoogleCloudStorage,SpiderOak,andSeaFile.

3DModelingand3DPrintingFree apps for 3Dmodeling and 3D printing are also available on Linux. Thepowerful Blender supports animation, modeling, game development, 3DPrinting,andmore.Installitandyou’llhavetheblendercommand.While Blender is more for artistic projects, freeCAD helps you make

parametric 3Ddesigns in engineeringprojects.FreeCADcan input andoutputmanystandardfileformatsandcanbeprogrammedusingaPythoninterface.ItisagoodchoiceforbeginningandexperiencedCAD(ComputerAidedDesign)engineers.Figure4.9showsascreenshotfromtheFreeCADwebsite.

Figure4.9FreeCAD

MathematicalCalculationsThe expr command can perform simple calculations and comparisons withintegers.Seemanexprfordetails.

4.2

Figure4.10GnomeCalculator

For a desktop calculatorgnome-calculator is handy for interactive use andprovides several different input modes (Figure 4.10). The gcalccmd is acommand-lineversion.For awonderfully powerfulmathematics tool, considerMAXIMA, a freely

availablesymboliccomputation system derived fromMIT’sMacsyma (ProjectMAC’s SYmbolicMAnipulator) which is a general purpose computer systemdesigned to perform exact aswell as approximatemathematical computations(Figure4.11).MAXIMA offers an impressive collection of mathematics capabilities that

rivals well-trained engineers and mathematicians. Part of the author’s Ph.D.work contributed to the development of Macsyma including polynomialfactorization and GCD, complex numbers, limits, definite integration, linearalgebra,FortranandLaTeXcodegeneration.Seetheonlinedemosatthebook’swebsiteforacloserlook.

Figure4.11MAXIMA

ItissimpletoinstallMAXIMA.dnfinstallwxmaximaapt-getinstallwxmaximaThen use wxmaxima to invoke the GUI version ormaxima for the CLI

version.Now let’s turn our attention to command-line applications and how to put

themtouse.

COMMANDSANDFILTERS

Simply put, a filter is any command that produces output by transforming itsinput by following a set of well-defined conventions. The conventions makefilterseasytocombinewithotherprogramsinapipeline(Figure4.2).

Figure4.12AFilter

A filter is distinguished from other commands by the followingcharacteristics:

1. Afiltertakesinputfromthestandardinput(stdin).Thus,whenweinvokeafilter,itdoesnotneedafileargument.

2. Afiltersendsitsresultstothestandardoutput (stdout).Therefore, itdoesnotneedanoutputfileargument.

3. Afilterperformsawell-defined transformationon the inputandproducestheoutputwithnoheader,trailer,label,orotherformatting.

4. Afilterdoesnotattempttointerpretitsinputdatainanyway.Thus,itnevertreatsitsinputasinstructionsorcommands.

5. Withfewexceptions,a filterdoesnot interactwith theuser foradditionalparametersotherthanthosesuppliedonthecommandline.

6. Anyerrorordiagnosticoutputproducedbyafilter issent tothestandarderroroutput (stderr).Hence, errormessages are nevermixedwith resultsproduced.

Thesecharacteristicsmakeafiltereasytofitintoapipeline.Theoverallpurposeis tomake a programproduce output that can be fed into another program asinputandthatcanbeprocesseddirectly.Typically,suchinputcontains linesoftextwithnodecorativelabels,comments,orextraformatting.Aseparatelineisused for each data entry. For example, if the data entries arewords, then theinput should be one word per line. For more complicated data entries (forexample,thoseproducedbyls-l),thelinemayconsistofseveralfieldsseparatedbyspaces,tabs,orcolons(forexample,/etc/passwd).ManyLinuxcommandsarefiltersthatcanalsoworkonfiles.Theconvention

is If filenames are supplied as arguments, a command can use them forinput/output.Otherwise,ifnofilesaregiven,thecommandactsasafilter.The process expansion (Chapter 3, Section 3.7) feature of Bash makes it

possibletotreatoutputfromfiltersasinputfilestoothercommands.

Let’slookatsomefiltersandthenshowhowtobuildpipelineswiththem.

LeadingandTrailingLines:headandtailThecommandsheadandtailareavailablefordisplayingtheleadingandtrailinglinesofafile,respectively.Thecommandhead[-k][file...]outputsthefirstk(default10)linesofeachgivenfiletothestandardoutput.If

no file argument isgiven, the standard input isused.Thehead command is aquickway to examine the first few lines of a file, which are often all that isneeded.The command tail is the opposite, displaying the last part of a file on the

screen:tail[starting-point][file...]outputsthelastpart(fromstarting-pointtotheendor,bydefault,thelast10

lines)ofeachgiven file. Ifno file is specified, the standard input isused.Thestartingpointisspecifiedas+k(linekfromthebeginning)-k>(linekfromtheend)If the integer k is followed immediately by the characters b or c, tail will

countblocksorcharacters,respectively,insteadoflines.The-foptioninstructstail to continue, even after the end of the file has been displayed, repeatedlyprobingthefileincasemorelinesareappended.Thisoptionprovidesawayofmonitoringafileasitisbeingwrittenbyanotherprogram.Inpipelines,headandtailareusefulforselectingsomelinesfromtheinput

and excluding others. Themore (less) command can be used at the end of apipelinetomanagelongoutput.

CharacterTranslation:trThe command tr copies standard input to standard output, substituting ordeletingspecifiedcharacters.Forexample,

trA-Za-z<file1>file2

creates file2 as a copy of file1, with all uppercase letters translated to thecorrespondinglowercaseones.Anotherexampleis

tr'tab'%<file1>file2

where TAB must be escaped by CTRL+V when typing this command. ThismethodallowsyoutoseeeachTABinfile1asa%character in file2 (assuming

file1doesnotcontainany%characters).Generally,trstring1string2translates string1 characters to the corresponding % string2 characters,

assumingthetwostringsareofthesamelength.Ifstring2isshorter,itistreatedas if itwerepaddedwithenoughrepetitionsof its lastcharacter tomakeit thesamelengthasstring1.Arangeofcharacterscanbegiven,asinx-y.AcharacteralsocanbegivenbyitsASCIIcodeinoctal(forexample,040for%SPACE,011forTAB,and012forNEWLINE).Forexample,toreplaceastringofblankswithaNEWLINE,usetr-s’040011’’012’The-s(squeeze)optionshortensallstringsofconsecutiverepeatedcharacters

instring1 to justonecharacter.The -c (complement)option isused to specifystring1bynamingcharactersnotinit.Thus,tr-cs0-9A-Za-z’012’createsalistofallwords(oneperline)intheinput.Inthisexample,string1is

allcharactersexceptnumeralsandletters.Whentheoption-d(delete)isgiven,charactersinstring1aredeletedfromthe

output,andthereisnoneedforstring2.Forexample, toridtheinputofallCRcharacters,wecanuse

tr-d"\015"<file

TabExpansionTabs often need to be expanded into an equivalent number of spaces or viceversa. However, this transformation is not performed by tr because each TABmustbereplacedbyjustenoughspaces tomovetheoutputcolumnpositiontothenexttabstop.Tabexpansionanditsinversetransformationareslightlymorecomplicatedthansimplecharacter-for-characterreplacement.Thefiltersexpand(substitutesspacesfortabs)unexpand(substitutestabsforspaces)areusedforthesepurposes.Forexample,expand-t6<filereplaceseachTABinfilebyspaces,assumingthatTABstopsare6(default8)

spacesapart.

FoldingTextLinesIt issometimesnecessarytomakesure linesof textarewithinacertain lengthfor easy display, viewing, or printing. The fold filter breaks up long lines byinsertingaNEWLINEcharacterwherenecessary.

fold<fileThedefaultistolimitlinestoalengthof80characters.Usefuloptionsinclude-wn(setswidthtoncolumns)-s(breakslinesonlyatspaces)Forexample,fold-w72-sreport>new_reportcreatesnew_report as a version of report with all lines folded at spaces to

within72characters.

CalendarRemindersbyEmailOnLinuxyouhavefancyGUItoolssuchasEvolution,California,andLightningforThunderbird.However,usingasimplepipeline,thereisalsoanotherwaytogetcalendar remindersandhave themsent toyoubyemail thedaybefore thetargetevent.The calendar command is a filter that reads a calendar file (./calendar or

$HOME/.calendar/calendar)andwritesallevents for todayand tomorrow.ThecalendarfilelistsoneeventperlineintheformdateTABanytextdescriptionForexample,

4/15paytax1/21Reminder:ChineseNewYear1/28/201709/01

reminder:Sister'sbirthday(9/08)Saturdayweeklyfilebackup

Thepipelinecalendar|mail-sMyReminderemailAddresssendsthedesiredreminderbyemail.Seethecalendarmanpageforpossible

dateformats.

SortingTextLinesDataoften are sorted in somekindoforder for easyaccess andmanipulation.Youmay want to alphabetize a list of names and addresses, combine severalsuch lists into one, look an entry up in a list, or compare two lists already inorder.Thesortcommandtakesinputlinesandwritesthemtothestandardoutputin

sortedorder.Theunitsbeingsortedareentirelines.Eachlinemaycontainoneormore fields,which are separated by one ormore blanks (spaces or tabs). Forexample, a file called students (Ex: ex04/students)may contain the followinglines:

F.Smith213.75PhysicsJ.Wang232.00AccountingR.Baker203.20

ChemicalEngineeringS.Doe243.20BusinessP.Wang224.00Computer

Science

The first line contains five fields (separated by white space); the third linecontains six fields. The sort command allows you to use field positions tospecifysortkeysfororderingthelines.Asortkeyisdefinedbyastartingandanendingfieldpositionina line.Thesortkeys indifferent linesarecomparedtoorderthelines.Thus,ifyouspecifythesortkeyasthesecondfieldtosort thefilestudents,

thenthelineswillbeorderedbylastnameusing,bydefault,theASCIIcollatingsequence. In the absence of any specification, the sort key is the entire line.Multiplesortkeysaregiveninorderofimportance.Incomparinganytwolines,sortusesthenextsortkeyonlyifallprevioussortkeysarefoundtobeequal.Thecommandhasthegeneralformsort[options][–key=key...][file...]Alllinesinthegivenfilesaresortedtogether.Afilenamed“-”isthestandard

input. Ifnofile isgiven,sortuses thestandard input. Itwrites to thestandardoutputbydefault.Keysaregiveninorderofsignificance.Akeyisgivenbytwofieldpositions:begin[,end]which specify a sort key consisting of all characters between thebegin and

endpositions(fieldseparatorsexcluded).Whenomitted,endbecomestheendofline.Eachpositionhastheformf[.c]where f is a field number, and the optional c is a character number. For

example, the position 2.3 indicates the third character of the second field. Ifomitted,cis1.Thus,theposition3isthesameas3.1.Table4.1providessomeexamplesofsortkeyspecifications.SortKeys

Therefore,thecommandsort–key=2,3.0studentssortsthefilestudentsbylastname.Inthisandmanyothercases,theending

fieldcanbeomittedwithoutaffectingthesearch.SortkeysarecomparedusingASCIIordering,unlessoneofseveraloptionsis

4.3

used.Afewimportantoptionsarelistedhere:f Treatsalluppercaselettersaslowercaseletters

nSortsbyincreasingmagnitudeusingaleadingnumericalstringinthesortkeywherethenumericalstringmayhaveleadingblanksand/orasignfollowedbyzeroormoredigits,withanoptionaldecimalpoint

r ReversesthesenseofcomparisonsandsortsthelinesinreverseorderThese option characters can be given globally, affecting all sort keys, or

immediately after a key specification to affect only that sort key. Note someexamples:

ls-l|sort-n--key=5,6.0(sortbyincreasingbytecount)ls-l|

sort--key=5,6.0nr(sortbydecreasingbytecount)

Formultiplesortkeys,considersort–key=4,4.4nr–key=5studentswhich sorts by grade point average (4th field), highest first, and break ties

withthesecondkey,thedepartmentname(field5toendofline).Seemansortformoreinformation.

THEGREPCOMMANDThe grep command is a filter,fgrep, that provides the ability to search andidentify files containing specific text patternsor to find all lines ingiven filesthatcontainacertainpattern.Thecommandhasmanypossibleapplications.Youmay search for a name, a subject, or a phrase.Youmay search for somethingcontained ina filewhose filenameyouhave forgotten,oryoucanextract textlinesfromfilesthatpertaintoaparticularsubject.Thegrepfilterisoftenusefulinpipelines.Forexample,lookmen|grepgitisisacutewaytofindtheword“meningitis.”Thenamegrepcomesfromgeneralizedregularexpressionswhichareexactly

what grep uses to specify search patterns. The general form of the grepcommandisgrep[options][patterns][files]It searches for the given regular expression patterns (Section 4.4), using a

fairly efficientmatchingalgorithm, in thegiven files andoutputs to stdout thematchinglinesand/orfilenames.Makingitflexible,manyoptionscontrolhowexactlygrepworks.OptionsofthegrepCommand

Agrepcommandsearches thespecified filesor standard input for lines thatmatchthegivenpatterns.Alinematchesapatternifitcontainsthepattern.Eachmatched line is copied to the standardoutputunless specifiedotherwisebyanoption(Table4.2).Theoutputlinesareprefixedwithafilenameifmultiplefilesaregivenasarguments.Generallyspeaking,thegrepcommandisusedeithertoobtainlinescontainingaspecificpatternortoobtainthenamesoffileswithsuchlines.Forexample,let’ssayyouhaveafileofphonenumbersandaddresses.Each

lineinthefilecontainsthenameoftheperson,aphonenumber,andanaddress.Let’snamethisfilecontacts(Ex:ex04/contacts).Afewtypicalentriesfollow:

(330)555-1242BobSmithC.S.Dept.UnionCollege.StowOH

44224(415)555-7865JohnGoldsmithP.O.Box21951PaloAltoCA

94303(415)555-3217BertLin248HedgeRdMenloParkCA94025(617)

555-4326IraGoodman77Mass.Ave.CambridgeMA02139

Considerthecommandgrep-FstringcontactsorequivalentlyfgrepstringcontactsIfstring is aname, thenany linecontaining thegivenname isdisplayed. If

stringisanareacode,thenallentrieswiththesameareacodearedisplayed.Ifstringisazipcode,thenalllineswiththesamezipcodearedisplayed.Also,fgrep-vMAcontactsdisplaysalladdressesexceptthoseinMA.Here is an application dealing with multiple files. Let’s say you have a

directorynamedlettersthatyouusetofileawayelectronicmailforsafekeepingandlater reference.Supposeyouneed tofinda letter in thisdirectory,butyoudon’tremembertheletter’sfilename.Allyourecall isthattheletterdealswith

4.4

thesubject“salary”.Tofindtheletter,usecdlettersfgrep-i-lsalary*Thecommandsearchesall (non-hidden) filesunder thecurrentdirectory for

lines containing the string salary (ignoring case differences) and displays onlythenameofanyfilewithmatchinglines.TheShellvariable$?recordstheexitstatusofacommand(Chapter5,Section5.7).Thegrepcommandreturnsexitstatus0ifanymatchesarefound,1ifnone,and2iferror.

REGULAREXPRESSIONSInthegrepcommandandmanyothertextprocessingsituations,theneedtofindastringofcharactersmatchingaparticularpatternarises.Forexample,testingifafilenameendsin.pdf,checkingifaparticularuserinputrepresentsanumberwithanoptional leadingsign,ormakingsure thata lineof texthasnotrailingwhitespaces.Inordertodefinepatternstomatch,weneedanotationtospecifypatterns for programs. A regular expression is a pattern matching notationwidelyusedandunderstoodbyprogrammersandprograms.Thesimplest regularexpression isafixedstringsuchasUbuntuorCentOS.

Such a regular expression matches a fixed character string. However, regularexpressions are much more flexible and allow you to match strings withoutknowingtheirexactspelling.InLinux, the applicationsgrep,vi/vim,sed, egrep, andawk/gawk, among

others,use largely thesameregularexpressions.Table4.3gives thebasics forregularexpressionnotationsthatmostprogramsunderstand.Thegrepcommandaccepts many additional pattern notations (see Section 4.5 and the grep manpage).BasicRegularExpressions

Consider editing, with vim, a recipe that contains many steps labeledsequentiallybyStep1,Step2,andsoon.Inrevisingtherecipe,youneedtoadda few steps and renumber the labels.A search pattern can be specified by theregularexpressionStep[1-9]wherethenotation[1-9]matchesanysinglecharacter1-9.Inthevimeditor(seeappendices),youcansearchwiththecommand/Step[1-9]and make the appropriate modification to the number. After that, you can

repeat the search using the vim search repeat command n, change anothernumber,search,andsoonuntilallthechangeshavebeenmade.Let’s put the regular expression notations to use and look at some specific

patterns.In a regular expression, the * character indicates an occurrence of zero or

more times of the previous character/pattern. In Table 4.3, we see regularexpressionspecialcharacters:[,],*,,and$,eachhavingaprescribedmeaningasapatternspecifier.

QuotinginSearchPatternsThe use of special characters in any searching scheme inevitably leads to thequestionofhowtosearchforapatternthatcontainsaspecialcharacter.Let’ssaythat you are editing a report and you want to search for [9], which is abibliographicalreferenceusedinthereport.Becausetheregularexpression[9]matches the single character 9, you need to%quote the [ and ] so that theyrepresentthemselvesratherthanpatternspecifiers.Thesolution,ironically,istointroduce yet another special character, the backslash ( ), to serve as a quotecharacter that prevents the immediate next character from being treated as a

4.5

patternspecifierandforcingittostandforitself.Thus,thepattern

9

matches[9],andthepattern

[1-9]

matches the strings [1] through [9]. To match any such bibliographicalreference,usethepattern

[1-9][0-9]∗

.Herearesomemorepatternexamples:

... (matches...,namelythreedots)/* (matches/*) (matches)[0-9A-z] (matchesanyoftheindicatedcharacters)Quotingacharacterthatdoesnotneedquotingusuallycausesnoharm.

PATTERNSFORGREPMost of the basic regular expression patterns listed in Table 4.3 work inprogramsacceptingregularexpressionpatterns.Thegrepcommandalsoacceptsextended regular expressions available via the -E option or through the egrepcommand.ExtendedregularexpressionsaddnotationsdescribedinTable4.4tothebasicregularexpressions.ExtendedRegularExpressions

4.6

InTable4.4redenotesany regularexpression.Theprecedenceofoperatorsusedforextendedregularexpressionsis(),[],",+,?,concatenation,and|.Careshould be taken when entering patterns on the command line because manypatterncharactersarealsospecialShellcharacters.Itissafesttoalwaysenclosethe entire pattern in a pair of single quotation marks. Here are some moreexamples:

grep´\-s´(matches-s;the\prevents-sfrombecominga

commandoption)grep-i´^linux´(matcheslinuxatthefrontofa

line,ignoringcase)grep´ch[0-9]*´(matcheschfollowedbyany

numberofdigits)egrep\.html?\<(matchesawordendingin.htmor

.html)egrep´\>\w+\.docx?´(matchesanywordfollowedby.docor

.docx)

Thegrepcommandsareoftenusedinapipelinewithothercommandstofilteroutput.MoreexamplesofgrepwithinpipelinesarediscussedinSection4.7.Informationonregularexpressionspresentedhereformsabasisfor learning

moreelaborateregularexpressionsinlanguagessuchasPerl,Ruby,Javascript,andJava.

ASTREAMEDITOR:SEDThesedprogramisa filter thatuses line-editingcommands to transforminputlines, fromstdinora file, andproduces thedesiredoutput lines (Figure4.13).Sedisanon-interactive,line-orientededitor.Itappliesprescribededitingactionstolinesmatchinggivenbasicregularexpressionpatterns.

Figure4.13TheStreamEditorsed

In practice, sed is used for such chores as deleting particular lines, doublespacingaprogramlisting,andmodifyingalloccurrencesofsomepatterninoneormoretextfiles.Infact,sedandgrepcanperformmanyofthesamefunctions.However,sed

ismorepowerfulbecauseitsupplies texteditingcapabilities.Thesedprogrambuffersoneinputlineatatime,repeatingthefollowingstepsuntiltherearenomoreinputlines.Figure4.14showsthesedprocessingcycle.

1. If therearenomore input lines, terminate.Otherwise, read thenext inputlineintothebuffer,replacingitsoldcontent,andincrementthelinecount(initially0)by1.

2. Applyallgiveneditingactionstothebuffer.3. Writethebufferouttothestandardoutput.4. Gotostep1.

Figure4.14TheEditingCycleofsed

Eacheditingactionmaybeapplicabletoalllinesortojustafew.Therefore,itis possible for some lines to pass through sed unchanged; at the same time,otherscanbemodifiedordeletedentirely.Frequently,sedisusedinthesimpleformsedscript[file]…wherescript specifies one ormore editing actions separated by semicolons.

Forexample,sed’s/Website/website/’chapter1sed’s/Website/website/g’chapter1Thefirstcommandreadstheinputfilechapter1,substitutes(thesaction)any

firstoccurrenceofWebsiteineachlinewiththestringwebsite,2andoutputsalllines, changed or not, to the standard output. If any line contains multipleinstances ofWeb site, only the first instance in the line will be replaced. Toreplacealloccurrences,usethesecondcommandwheretheg(globalmodifier)doesthetrick.Ifnofile isspecified,sededits lines fromstdin.Thesinglequotationmarks

aroundscript prevent the Shell from interpreting any special characters in thescript.Thecommandsed’s/RedHat/Fedora/g;s/ubuntu/Ubuntu/g’chapter1appliestwostringreplacementactions,sequentially,toeachlineofchapter1.

Theoption-fscriptfile_fileindicatesafilecontainingthedesirededitingscript.Ifascriptfiledoublecontainsthetwolines

s/$/\/

thensed-fdoublefileaddsanemptylineaftereachline in file,producingadouble-spacedoutput.

Asingrep,thepattern$meanstheendofaline.Eacheditingactioncanalsobespecifiedtoactonarangeoflines.Hereisthe

generalform:[address1][,address2]action[args]wheretheaddressesspecifytherangeofinputlinestoapplythegivenaction.

Anaddresscanbealinenumberorapattern.

Noaddress—Thegivenactionappliestoeveryline.Oneaddress—Theactionappliestoeverylinematchingthataddress.Two addresses—Theaction is applied repeatedly to the next set of linesbeginningwitha line thatmatchesaddress1, up to and including the firstlinethatmatchesaddress2(butnotaddress1).

Forexample,sed’/$/d’fileappliestheactiond(deleteline)toeachlinematchingthesingleaddress/$/,

anaddressobtainedbysearchingforthenextemptyline.Theoutputwillbefilewithall empty linesdeleted.Anotherversionof thisexampledeletesallblanklinessed’/[⊘▹]*$/d’fileWe use the symbols⊘ and▹ to stand for a SPACE and a TAB, respectively.

Remember toescape theTABwithCTRL+V oryoucanuse instead.Theaddressmatchesalinecontainingzeroormorespacesandtabsandnothingelse.Let’s look at an example involving a two-address action. Say that in your

HTMLfilestablesaresandwichedbetweentwolines

<table...>

and

</table>

Supposeyouwish to removeall tables fromagivenHTMLdocument (Ex:ex04/remove_table).Youmayuse

sed´/<table.*>/,/<\/table>/d´try.html>notables.html

4.7

Thedeletelineactiondisappliedtoalltablelines.A useful sed option is -n, which skips step 3 of the sed cycle. Hence, the

commandsed-n’/pattern/p’withtheoutput-lineactionp,isequivalenttogrep’pattern’,andsed-n’12,20p’fileoutputsonlylines12–20ofthegivenfile.Hence, if youwish to extract all the tables from a givenHTML document

(Ex:ex04/extract_table),youmayuse

sed-n´/<table.*>/,/<\/table>/p´try.html>tables

tooutput linesbetween thebeginningand theendof each tableusing thepactionandthe-noption.Alternatively,youcanuse

sed´/<table.*>/,/<\/table>/!d´try.html>tables

The exclamation point (!) reverses the sense of the specified addresses; itapplies the specified action to every line except the lines matching the givenaddress.Also,theyactiony/string1/string2/when given two equal-length character strings, performs character

translations.Thus,sed’y/abc/ABC/’filefunctionsthesameastrabcABCfileSimplescriptsareeasy togiveon thesedcommand line.Morecomplicated

scriptsshouldbeplacedinfilesandappliedwiththe-foption.Storingscriptsinfilesmakesthemeasilyreusable.Thesedcommandoffersanumberofotherfeaturesandoptions.Pleaserefer

tothesedmanpagesforadditionalinformation.

BUILDINGPIPELINESWehave discussed a good number of filters and seen some pipelines already.Let’snowseeafewmoreexamples.Hereisapipelinetolookupthecorrectspellingsofwords:lookprefix|fgrepstringAll words in the dictionary /usr/dict/words with the specified prefix are

producedbylookandfedtofgrep,whichselectsonlythosewordsthatcontainthegivenstring.Forexample,lookdis|fgrepsiongivesthefollowingoutput:discussiondispersiondissensionAnother example is a pipeline that saves to a file those commands that you

havegiventoyourShell.TheBashcommandhistorydisplaysanumberedlistofyourmostrecentcommands.Toenterthelasteightcommandsintoafile,youcanusethefollowingpipeline:history|tail-8|sed’s/*[0-9]**//’>filewherethesedcommandremovestheleadingsequencenumbers.A third example collects a list of directorynames from the currentworking

directory:

ls-l|grep^d|sed´s/^d.*⊘//´

Herethesededitingcommanddeletesamaximal(longest)stringstartingwiththeletterdatthebeginningofalineandendingwithaspace(⊘)foreachline.Anotherwaytoaccomplishthesametaskis

ls-F|grep´/$´|sed´s/\/$//´

Afinalexamplehastodowithmaintaininganaddresslist.Let’sassumeyouhaveafileofaddresses,myaddr,inhuman-readableform.Itsentriesaremulti-line addresses, and a single empty line follows each entry. A typical addressentrywouldlooklikethefollowing(Ex:ex04/myaddr):Dr.JohnF.DoeGreatEasternCo.40NorthRd.Cambridge,MA02139This form is easy for a user to read, but hard to maintain using filters.

However, you can transform this address filewith the followingpipeline (Ex:ex04/toaddr):

sed´s/^$/@/´myaddr|tr´\012@´´:\012´\|sed´s/^://;s/:$//´|

sort-u-t:--key=1,2>|addr

Thefirstsedsubstitutesthecharacter@foreachemptyline.ThetrcommandtranslateseveryNEWLINEcharacterintoacolonandevery@intoaNEWLINE.At

4.8

this point, each address entry is on a separate linewith a colon separating thefieldswithineachaddress.Thesecondsedremovesanycolonatthebeginningortheendofaline.Thefinalsortcommandorderstheaddressentriesusingthefirstfieldandremovesanyduplicateentries.

AddressProcessingNowyour address file addr is sorted and contains one address per line in thefollowingform:Dr.JohnF.Doe:EasternCo.:40NorthRd.:Cambridge,MA02139Youcanextractanaddressbyusing(Ex:ex04/useaddr)grep’JohnF.Doe’addr|tr’:’’012’Youcandeleteanyaddressbyusingsed’/JohnF.Doe/d’addr>temp.filemvtemp.fileaddrYoucaninsertoneormoreaddressesbyusingsort-u-t:-key=1,2addr->temp.filewhich allows you to type in the entries from the standard input. You may

insertanotheraddressfile,addr2,byusingsort-mu-t:–key=1,2addraddr2>temp.filemvtemp.fileaddrIn the preceding example, the first field contains the title and name of a

person.Thesortedaddressfileisnotinalphabeticalorderwithrespecttonames,unless everyone has the same title. To avoid this problem, you may want tomodifytherecordformatto(Ex:ex04/newaddr)Doe:John:F:Dr.:EasternCo.:40NorthRd.:Cambridge,MA02139andsorttheaddressfileusingthefirst,second,andthirdfieldsaskeys.Then

thefollowingcanbeusedtodisplayanentry(Ex:ex04/usenewaddr):look’Doe’newaddr|gawk-F:’print$4,$2,$3".",$1;print$5;print$6;print$7’For largefiles, the lookcommand,whichusesabinarysearchmethodfora

line prefix, is much faster than the fgrep command. The gawk is a GNUimplementationofawk.

FORMOREINFORMATIONSeetheappsearchforyourLinuxandsearchontheWebforgoodappstouse.Seethemanpagesfor thecommandsandfilterscoveredinthischapter.For

filtersacceptingregularexpressions, theirmanpageswillspecifyexactlywhat

4.9

4.10

patternsarerecognized.AdetaileddescriptionofawkcanbefoundinAppendix:PatternProcessing

withawkatthebook’swebsite.

SUMMARYLinuxhas an abundanceof softwarepackagesmostly free.Agoodnumberofuseful GUI-based apps have been discussed. Find what’s available on yourLinuxdistributionanduse themto takefulladvantageofyoursystem.Systemadmins can easily install missing apps with the Linux package managementsystem.Filtersproduceoutputbyperformingasimple,well-definedtransformationon

their input and follow a set of well-defined conventions so they can becomestagesinpipelinesthatcombinethemtoperformmanyvariedtasks.FiltersandpipelinesareconcreteartifactsoftheUNIX/Linuxphilosophy.Linux filters range from simple character substitutions (tr and expand) to

findingstringpatternsintextlines(thegrepcommands),toorderingtextlines,and to complicated stream editing (sed). How these commands workindividually and in pipelines for realistic applications, such as creating,maintaining,andaccessinganaddressdatabase,havebeendiscussed.Regularexpressionsarewell-establishednotationsforcharacterstringpattern

matching.Theyareused,inverysimilarways,inmanydifferentprogramssuchas grep, egrep, sed/vim. In Chapter 5, you’ll see that the Bash Shell alsounderstandsregularexpressions.Itisimportanttobecomefamiliarwithregularexpressionconcepts.CommandsSummary

Table4.5summarizesthecommandsdescribedinthischapter.

EXERCISES1. FindoutifyourLinuxhasthegnome-softwareoranotherappcentertool.

CanyouuseitorthepackagemanagertoinstallMAXIMA?

2. Find the best media (image, audio, video) playing apps for your Linuxdistribution.

3. FindoutabouttheKeepassxtool.Whatisthemostrecentversion?Howisitinstalledandused?

4. Findthebestmediaformatconversion(image,audio,video)appsforyourLinuxdistribution.

5. Findouthowtousexfig.6. WhatwebcamtoolsareavailableonyourLinuxdistribution?Iscamorama

available?7. Findouthowtousethecommandsgimpanddisplay.8. Findouthow to installandusean Internet speed test tool foryourLinux

distribution.9. Consider how expand works. Write an algorithm for figuring out how

manyspacesshouldbegeneratedforaTAB.10. Writeapipeline,using lsandhead, to list the tenmost recent files in the

currentdirectory.11. Howcanyouusegrep to locate all lines in a file thatdonot contain the

pattern-option?12. WhatisaGlobpattern?Whatisaregularexpressionpattern?Whatisthe

difference?Giveapatternineachcasetomatchastringendingin.html.13. Specifyaregularexpressiontomatch(a)anywordendingin.html;(b)any

imagenameending in .jpg, .png,or .gif; (c)anyempty line (linewithnocharacters in it whatsoever); (d) any blank line (line that is empty orcontainsonlywhitespacecharacters);and(d)anynumber.

14. Explain the following regular expressions: (a) a+$, (b) http[s]*: and (c)[@]+@gmailċom.

15. Considerthefollowingsedcommand:sed-n’/begin/,/end/p’fileDiscussitseffectiffilecontainsmanylineswithbeginandorendinthem.

16. Considerbuildingpipelinestomanageanaddressfile.Supposeyouwishtohaveanaddress,anemail,andaphonenubmeroneachaddressline.Howwouldyoudesign the record format?Writeapipeline toextractadesiredemailorphonenumberfromtheaddressfile.

17. Following the previous exercise, write a pipeline to add/change a phonenumberoremailtoanexistingaddressentry.

18. Specifyansedcommandtoreplaceanysetofconsecutiveemptylinesinafilewith justoneempty line.Anempty line isonewithnothing in it,notevenblankcharacters.

19. Rot13isamethodtoencodeASCIItextfiles:eachletterinthealphabetAthroughzisreplacedbyanother13positionsaway(AbyNandnbyA,for

12

example).Writeatrcommandtoperformthisencoding/decoding.20. Theyfunctionofsed canperformmostof the same translationsas tr. Is

thereanythingtrcandothatsedcannot?Ifso,discuss.See/usr/binforversionsoftex/latexcommands.TheAP(AssociatedPress)stylebookrecentlymadethechange.

5.1

Chapter5

WritingBASHScripts

TheShellismorethanjustaninteractivecommandinterpreter.Italsodefinesasimpleprogramminglanguage.AprogramwritteninthislanguageisknownasaShellprocedureorShellscript,which,initssimplestform,isjustasequenceofcommands in a file. The file, when executed, performs the tasks as if eachcommandinthescripthadbeenenteredandexecutedindividually,butwithoutall the typing. Shell scripts can save you a lot of time if you find yourselfrepeating a sequence of commands over and over. The Shell language alsoprovides variables, control structures such as if-then-else, looping, functiondefinition,andmeansfor inputandoutput. Ifaparticular taskcanbeachievedbycombiningexistingcommands,thenconsiderwritingaShellscripttodothejob.AswithotherLinuxcommands, aShell script canbe invoked throughyour

interactive Shell and can receive arguments supplied on the command line.Sometimes,scriptswrittenbyindividualusersalsocanbeofgeneraluse.Suchscriptscanbeinstalledinasystemdirectoryaccessibletoallusers.This chapter covers Shell script writing and techniques for effective Shell-

levelprogramming.WewillfocusonBashscriptsbecauseBashiscurrentlythemostwidely used andmost advanced Shell. Csh, Tcsh,Dash and Sh 1 scriptsfollowmanysimilarrules.Thepresentationsinthischapterareorientedtowardscriptwriting.However,

most constructs discussed here can be used interactively aswell. Some topics(forexample,commandgrouping)areasrelevanttointeractiveuseastoscriptwriting.

INVOKINGSHELLSCRIPTSAsmentioned, a Shell script is a programwritten in the Shell language. The

5.2

program consists of variables, control-flow constructs, commands, andcomments.TheShellscriptiskeptinatextfilewhosefilenameissaidtobethenameofthescript.TherearetwowaystoinvokeaShellscript:byexplicitinterpretationandby

implicitinterpretation.Inexplicitinterpretation,thecommand

bashfile[arg…](forBashscript)tcshfile[arg…](forTcsh

script)shfile[arg…](forShscript)

invokesaspecificShelltointerpretthescriptcontainedinfile,passingtothescriptanyargumentsspecified.In implicit interpretation, the script file containing the script is first made

readable andexecutablewith the chmodcommand to turn on the appropriateprotectionbits (Chapter1,Section1.6).Then the script canbe invoked in thesameway as anyother command: bygiving the script nameon the commandlinefollowedbyanyarguments.In either explicit or implicit interpretation of a Shell script, two Shells are

involved:(1)theinteractiveShell(usuallytheloginShell)thatinteractswiththeuserandprocessestheuser’scommandsand(2)theinvokedShellthatactuallyinterprets thescript.The invokedShell isaprocessspawnedby the interactiveShell.Since thespawnedprocess isalsoaShell, it is referred toasasubshell.Theeffectofthiscanbeillustratedbythefollowingexperiment.Firstcreateafilenamedtrythatcontainsthesimplescriptcd/usr/libpwdTorunthisscript,typebashtryThescriptcalledtrydisplaysthestring/usr/lib,whichistheoutputofthepwd

contained in the script.However, once it is finished, if you typepwd in yourinteractive Shell, your old working directory will appear. Obviously, the cdcommandexecutedinthescripthasnotaffectedthecurrentworkingdirectoryofyourinteractiveShell.Thisisbecausecdisexecutedbyasubshell.ToexecutethecommandsintrywiththeinteractiveShell,useinsteadsourcetry

AFIRSTSHELLSCRIPTNowlet’sconsiderasimpleBashscript.Thepurposeofthisscriptistoconsultalist of email addresses that are kept in a file named myContactList (Ex:

ex05/myContactList) in a user’s home directory. Each line in the contact listgives the name of a person, email address, phone number, and perhaps someotherinformation.Thescript(Ex:ex05/contact_one.sh)is

#!/bin/bash##consultsmyContactListgrep-i"$1"~/myContactList

Wewillusethesuffix.shforBashscriptsasanamingconvention.Thefirstline is special. InLinux, the properway to begin anexecutable text file is #!,followedbythefullpathnameofanexecutablefiletogetherwithanyargumentsto it.Thisspecifies thecommand to invokean interpreter for the remainderofthe script.Make sure #! are the very first two characters in the file, with noemptyline,whitespace,oranyothercharacterbeforethem.The first line of contact.sh indicates a Bash script. Similarly, the line

#!/bin/cshbeginsaCshscript,andtheline#!/bin/shbeginsanShscript.Thesecondlineisacomment.InShellscripts, thepartofanylinefromthe

first#totheendoflineisignoredbytheShell.Thesymbol$1 iscalledapositionalparameter.Thevalueof thepositional

parameter$n is thenth command-line argument.Thus, if the first argument issmith,then$1hasthatvalue,andthescriptisequivalenttogrep-ismith/myContactListRecall that expands to your home directory. Now you should issue the

commandchmod+rxcontact.shtomakecontact.shreadableandexecutable.Nowthecommandcontact.shsmithrunsthecontact.shscript(inthecurrentdirectory).Theprecedingcommand

assumesthatthespecialperiodsymbol(.)isincludedinyourcommandsearchpath(Section3.4).Otherwise,youneedtouse./contact.shsmithIfthecontact.shscript isput inadirectorywhosenameisonthecommand

searchpath,thencontact.shsmithwillworknomatterwhatyourcurrentdirectoryis,withouthavingtospecify

thecontact.shcommandwithapathname.Usually,youwouldcreateadirectorybinorcmd inyourhomedirectory to

holdallscriptsandotherexecutablecommandswrittenorobtainedbyyou.Byincludingtheline

PATH=$PATH:$HOME/cmd:.

5.3

5.4

in your .bash_profile, you can invoke executable files in your own cmddirectoryjustlikeotherLinuxcommands.Ifyouinclude.onPATH,makesureitisat theveryend.Otherwise,youmayexecuteunexpected,orevenmalicious,codeinthecurrentfolder.

SHELLSCRIPTEXECUTIONA Shell script consists of a sequence of Shell built-in commands and regularLinuxcommandsseparatedbyNEWLINEorsemicolon(;)characters.Commentsare introduced by #, as previously mentioned. Commands in a script areexecuted insequence. If theexecutionofacommandresults inanerror,scriptexecution will be aborted if the offending command is a Shell built-in.Otherwise, for a regular command, the default action is to skip the offendingcommandandcontinuewiththenextcommandinthescript(Figure5.1).

Figure5.1BashShellScriptExecution

IndescribingtheShelllanguages,thetermcommandlistmeansasequenceofzeroormorecommandsseparatedbyNEWLINEorsemicolon(;)characters.Thetermwordlistreferstozeroormoreblankseparatedwords.

POSITIONALPARAMETERSIn Shell scripts, the variables $0, $1, $2, and so on are known as positionalparameters.Thevariable$0referstothefirsttokenofthecommandlinewhichinvoked the script. Thus, $0 may have the value contact.sh or ./contact.shdependingonthecommandgiven.Thevariables$1,$2,andsoonrefer to thecommand-linearguments.WhenaBashscriptisinvoked,thespecialvariable$0issettothecommand

name. The positional parameters $1, $2, etc. are set to the command-linearguments (use $n for n bigger than 9); $* (and $@) is set to the list ofargumentsgivenonthecommandline;and$#issettothenumberofpositional

5.5

parameters.TheBashscript(Ex:ex05/myecho.sh)

echo'$0='$0echo'$1='$1echo'$2='$2echo'$3='$3echo

'$#='$#echo'$*='$*echo'$@='$@

displaystheseparametervalues.Forexample,thecommandmyecho.shABCDproducestheoutput

$0=./myecho.sh$1=A$2=B$3=C$#=4$*=ABCD$@=ABCD

Tryityourself.

THEFORCOMMANDTheforcommandisusedtoexecuteasetofcommandsrepeatedly.Thegeneralformis

forvarinwordlistdocommandlistdone

Thelinebreaksareneededunlessyouusethe;commandseparatorasinforvarinwordlist;docommandlist;doneThe commandlist is executed once for each word inwordlist as, each time

through, the control variable var takes the next word for its value. As anexample, let’s rewrite the contact_one.sh script given in Section 5.2 as (Ex:ex05/contacts.sh):

#!/bin/bash##consultmycontactsforargsgivenforxin"$@"##

(0)dogrep-i"$x"~/myContactListdone

Bashhastwobuilt-invariables,$*and$@,referringtotheargumentsgivenonthecommandline.Eachisalistofwordsfromthecommand-linearguments.Consideracommandwiththreearguments:somecmdab"cd"The$*and$@inthiscasewillbothbethelistabcdwithfourwords.The

quotation "$*"makes it oneword,whereas thequotation "$@"makes it threewordsaband"cd".Itisimportanttounderstandthisdifference.Itturnsoutalsothatline0canbewrittensimplyasforx,whichmeansxwilltakeonsuccessivecommand-linearguments.Nowcontact.shcanbeusedononeormorenames,asincontact.sh"JohnSmith""PaulWang"Theforcommandcanbeusedtogothrougheachfileinadirectory.Trythe

5.6

followingscript:

#!/bin/bash##exampletogothroughallfilesinthecurrent

directoryforfilein*doecho$filedone

Execute this script, and you’ll see the filenames in the current directorydisplayed.Sincethefilenameexpansiondoesnotmatchanyfilenamethatbeginswithaperiod(.),thosefilenameswillnotbedisplayed.Togetallfiles,use

forfilein.**doecho$filedone

Bash supports another form of for loop that is similar to that of the Clanguage.

#!/bin/bashfor((i=0;i<9;i++))doecho$idone

The iteration control involves numerical expressions (Section 5.11). Suchloops areuseful for indexing througharrays (Section5.14) and, of course, fornumericalcomputations.

THEIFCOMMANDThe if construct provides for conditional execution of commands. The simpleformofifis

iftest-exprthencommandlist1elsecommandlist2fi

If the given test-expr is true, then commandlist 1 is executed; otherwise,commandlist2isexecuted.Theelsepartmaybeomitted.Forexample,thetestexpression[[-ffile]],knownasanextendedconditional

(Section 5.7), tests if file exists and is a regular file. We can improve thecontact.shasfollows(Ex:ex05/contact_check.sh).

#!/bin/bash##consultmycontactsforargsgivenif[[-f

~/myContactList]]##(A)thenforxdogrep-i$x

~/myContactListdoneelseecho"File~/myContactListnotfound."fi

Inatestexpression,theSPACEafter[[andtheSPACEbefore]]arepartof theconditionalnotation(lineA).Withintheifstatement,theelifconstructcanbeused.Thegeneralformis

iftest-expr1thencommandlist1eliftest-

expr2thencommandlist2elsecommandlist3fi

Iftest-expr1istrue,commandlist1isexecuted.Iftest-expr1isnottrue,andiftest-expr2istrue,thencommandlist2isexecuted.Therecanbeanynumberofelifconstructs.Ifalltestexpressionsarefalse,thentheelsepartisexecuted.Often,itisimportantforanyprogramtochecktheargumentsitreceives,and

aShell script isnoexception.Here is some typical argumentcheckcode (Ex:ex05/argCheck.sh).

#!/bin/bash##checkandsetcommand-lineargumentsif[[$#<2||

$#>1]]##(1)thenechousage:"$0[from-file]to-file"##

(2)exit1;##(3)elif[[$#==2]]##(4)thenfrom="$1"to="$2"else

##(5)to="$1"fi

Theexpression$#>2checks if thenumberof arguments isgreater than2.The || is logical or, whereas < is less than. This script expects one or twoarguments.Ifthenumberofargumentsisincorrect,itdisplaysanerrormessage(line2)and terminates thescriptwithanabnormalexitstatus1 (line3). Ifwehavetwoarguments(line4),wecansetthevariablesfromandto.Otherwise,wehaveonlyoneargumentand itbecomes thevalueof to.Argumentchecking iscriticalatthebeginningofeveryprogram.

Figure5.2ThecmdsearchScript

Now let’s look at a complete script using for and if constructs. The script(Figure 5.2) locates a command on the command search path ($PATH) anddisplays its full pathname (Ex: ex05/cmdsearch.sh). The first (and lone)argument is the target commandname (line a).On line b, each : in $PATH isreplaced by a SPACE with the tr command (Chapter 4, Section 4.2), and theresultingmultiwordstringisassignedtoavariablepathviacommandexpansion

5.7

(Chapter3,Section3.7).Foreach$diron$path(linec),weseeif$cmdisfound(line d). The conditional expression [[ -x file ]] is true if file exists and isexecutable (see Section 5.13 for more on file queries). If the program everreacheslinee,thenthetargetcommandisnotfound.Herearesomesampleusesofcmdsearch.cmdsearchgnome-terminalcmdsearchvimcmdsearchgcc

TESTEXPRESSIONSANDEXITSTATUSExitStatusInLinux,acommandindicateswhetherithassucceededbyprovidinganintegerexitstatustoitsinvokingenvironment.Azeroexitstatusmeansokay,andnon-zeromeanserror.TheShell, being a command interpreter, is a primary invoking environment

for commands. After executing a command, the exit status is available in thespecialShellvariable$?.InaShellscript,usethebuilt-incommandexitn toterminateexecutionand

returnnastheexitstatus.

TestExpressionsTestexpressionsareusedinifaswellasotherconstructs(while,until,etc.) toproducetrue/falsevaluesbytestinggivenconditions.ThetruthvalueofaBashtestexpressionisreallydeterminedbyitsexitstatus.

Atestexpressionistrueifitreturnsazeroexitstatus;otherwise,itisfalse.Nowlet’stakealookatthedifferentformsoftestexpressions.Atest-exprcanbealistofoneormoreoftheseexpressions:

Aregularorbuilt-incommand(Section5.7)Anextendedconditionalexpression[[]]A numerical expression (( )), with 0 being false and non-zero being true(Section5.11)(test-expr),using()forprecedencegrouping!test-expr“logicalnot”oftest-exprtest-expr1&&test-expr2“logicaland”ofthetwoexpressionstest-expr1||test-expr2“logicalor”ofthetwoexpressions

5.8

Hereisanexamplethatusesgrepasatestexpression(Ex:ex05/condEdit.sh).

#!/bin/bashforfilein*##foreachfileincurrentfolderdoif

grep-q"$1"$file##ifpattern$1isin$filethennano$file##

invokenanoon$filefidone

Anextendedconditionalisenclosedby[[SPACEontheleftandSPACE]]ontheright.2Table5.1liststestexpressionsforstrings.Withinthe[[conditional,Globpatterns(Chapter3,Section3.7)areallowedontheright-handsidesof==and!=.Furthermore,extendedregularexpressions(Chapter4,Section4.5)following=aresupported.BashStringConditions

Theextendedconditionalsalsosupportnumericaltest[[arg1roparg2]]tocomparetwointegersarg1andarg2witharelationaloperatorropwhich

canbe==,!=,<,>,-le,or-ge.3Often,programmersprefer tousenumericaltestsprovidedby(())(Section5.11)instead.Inside[[]]youcanalsousethelogicaloperators!(not),||(or),and&&(and)

on test expressions.Please refer toSection5.11 for numerical test expressionsandtoSection5.13forfile-relatedtestexpressions.

THESHIFTCOMMANDTheBashbuilt-incommandshiftleftshifts2to1,3to2,etc.Ingeneral,shiftnshiftsnto1,n+1to2,etc.The shift command is often useful after the first few positional parameters

havebeenprocessedandyouwanttousealoopsuchas(Ex:ex05/shift.sh)

forvardoecho$vardone

5.9

togoovertherestofthepositionalparameters.

THECASECOMMANDWhile the if-elif-else command enables logical branching in general, the casecommand provides branching based on simple pattern matching. The generalformofcaseiscase(str)in

case(str)inpattern1)commandlist1;;pattern2)commandlist2;;...esac

The given expression str is successivelymatched against the case patterns.EachcasepatterncanbeoneoralistofGlobpatterns(Chapter2,Section3.7)separatedby|andterminatedby).Onlythelistofcommandsforthefirstmatchwillbeexecuted.Nothingisexecutedifthereisnomatch.Forexample,thestringab.cmatchesthecasepattern*.corthepatterna*c.

Figure5.3TheappendScript

Asanexample,ascriptforappendingeitherthestandardinputorafiletotheendofanotherfileisshowninFigure5.3(Ex:ex05/append.sh).Thecommandappend.shfile1file2appendsfile1totheendoffile2.Thecommandappend.shfilefirstlinesecondlinethirdlineDappends the three lines to theendof file.Note thecatch-allpattern*as the

lastcaseclausetoprocessanyunmatchedcases.

5.10

5.11

THEWHILEANDUNTILCOMMANDSInadditiontotheforcommand,thewhileanduntilcommandscontroliterationwithanarbitrarycondition.Thegeneralformofthewhilecommandiswhiletest-exprdocommandlistdoneThetest-exprisevaluated.Ifitistrue,thencommandlistisexecuted,andtest-

expr is retested. The iteration continues until the test-expr tests false. For aninfiniteloop,usetheBashbuilt-incommand:(yes,thecharacterCOLON)asthetest-expr.The : command does nothing other than expand any arguments andgivea0exitstatus.In the following script (Ex: ex05/myfortune.sh), we continue to display a

fortunemessageuntiltheuserwishestostop.

#!/bin/bash##displaysfortuneuntiltheuserquitsgo="yes"while[[

"$go"=="yes"]]##(i)do/usr/bin/fortune##(ii)echo-n"****

Morefortune?[yes/no]:"##(iii)readgo##(iv)done

Thewhile condition is checked (line i). If true, the fortune command 4 isinvoked(lineii),andapromptisdisplayed(lineiii)toseeiftheuserwishestocontinue. The -n option tells echo not to output the usual line break after themessage. The user input is read into the variable go (line iv),whose value istestedagain.Ifwereplacethewhiletestexpressionwiththepatterncondition

[["$go"==y*]]

thentheusermayenteranythingstartingwithytocontinue.Theuntilloopisthesameasthewhileloop,excepttheiterationstopsassoon

astheuntilconditionismet.

NUMERICALEXPRESSIONSSinceShellvariablesarestring-valued,weneedtousethearithmeticexpansionnotation$((arith-expr))to perform integer arithmetic computations.TheShell built-in command let

canalsobeusedtoperformarithmeticoperations.letarith-expr1arith-expr2...

5.12

5.13

Herearesomeexamples(Ex:ex05/arith.sh).

#!/bin/basha=2echo$((a+3))##displays5letb=2*++aecho$b##

displays6echo$((a*b))##displays18letc=-8echo$((c<0?c

:-c))##displays8

Tocomparenumbersinnumericalconditionalsuse,forexample,if((a>b))(ifaisgreaterthanb)TheBashcommandhelpletdisplaysafulllistofoperatorsavailableforthenumericalexpressionsforlet

orinside(()).Hereisaloopthatdisplaysthecommand-lineargumentsinreverseorder(Ex:

ex05/echoback.sh).

#!/bin/bashoutput=""until(($#==0))dooutput="$1

$output"shiftdoneecho$output

THEBREAKANDCONTINUECOMMANDSThebreak command is used inside the iteration control structures for,while,anduntil.Whenbreakisexecuted,controlshiftstothefirstlineaftertheendofthenearestenclosingiteration.Thiscommandprovidesameansto“breakout”ofaniterationloopbeforeitsnormalcompletion.Thecontinuecommandisusedinthesamemanneras thebreakcommand,

except it transfers control to the beginning of the next iteration instead ofbreakingoutoftheloopentirely.Theexamplescriptclean.sh(seeSection5.20)involvessometypicalapplicationsofbreakandcontinue.Withinnestedloops,breakorcontinuecantakeanoptionalintegerargument

(1,2,3,etc.)tobreakorcontinueoutofthenthlevelofnestedloops.

FILEQUERIESTomakefileanddirectoryaccessandmanipulationeasy,Bashalsoprovidesasetofconditionstoquerystatusinformationforfiles.Filequeriesareintheform-xfile,wherexisasinglecharacter.CommonfilequeriesarelistedinTable5.2.Togetacompletelistingusehelptest.If the file does not exist or if it is inaccessible, all queries return false. For

example,thefollowingcodefragmentisvalid:

if[[-e$file&&-f$file&&-w$file]]thencat$1<<

5.14

5.15

$fileelseecho"accessproblemfor$file"fi

BashFileQueries

Inthefilesystem,anordinaryfileisonethatstoresapplicationdataandnotone that serves filesystem functions such as a directory (folder) or link(shortcut).SeeChapter6,Section6.2formoreinformationonLinuxfiletypes.

VARIABLESTherearedifferentkindsofvariables:

1. Positionalparameters($1,$2,...)andspecialvariables($0,$#,...).2. EnvironmentvariablessuchasDISPLAYandSHELL3. Ordinaryvariablesandarraysofyourownchoosing

Toassignvaluetoavariablevar=valueShell expansions and evaluations are performed on value, and the result is

assigned to thegivenvariable. Ifvalue is omitted, then the variable has valuenull.Variableattributescanbedeclared:

declare-ivar1var2...(holdingintegervalues)declare-rvar1

var2...(read-only)declare-aarr1arr2...(arrays)declare-x

var1var2...(exportedtotheenvironment)

To remove a variable use unset var. The special operator += performsadditiononintegervariablesandconcatenationonstringvariables.Forexample(Ex:ex05/varusage.sh),

#!/bin/bashdeclare-iab;a=10;b=5b+=$a;##bis15declare-r

b;b=0##error,bisread-onlyunsetb##error,bisread-

onlyname="John";last="Doe"echo${#name}##lengthof$nameis

4name+=$last##nameisJohnDoe

ARRAYSTodeclareanarrayvariableuse

5.16

declare-avarHowever,itisnotnecessarytofirstmakesuchadeclaration.Forexample,to

createanarrayfruits,youcanusetheassignment

fruits=("redapple""goldenbanana")

orequivalently

fruits[0]="redapple"fruits[1]="goldenbanana"

Thus,Bashindexedarraysarevariableswithzero-basedindexing;thatis,thefirstelementofanarrayhasindex0($fruits[0]forexample),thesecondelementhasindex1,andsoon.However,theindicesdonothavetobeconsecutive.Thefollowingexamplesillustratearrayusage(Ex:ex05/arrusage.sh).

#!/bin/bashbr=()#emptyarrayfruits=("redapple""golden

banana")fruits+=("navelorange")#arrayconcatenation(1)echo

${fruits[1]}#valuegoldenbananaecho${#fruits[*]}or

${#fruits[@]}#lengthofarray(2)fruits[2]="greenpear"#element

assignmentfruits[6]="seedlesswatermelon"#gapinindex

allowedbr+=("${fruits[@]}")#brnowsameasfruits(3)

Note#(line2)forthelengthofanarrayandthe+=operator(line1and3)forarrayconcatenation.Togothroughelementsinanarraywithaloop,youmayuse

forelin"${br[@]}"do##use$elforsometaskDone

or,ifindexingisconsecutive,

for((i=0;i>${#br[@]};i++))do##dosomethingwith

${br[$i]}Doneecho-n"Pleaseinputanarray:"read-aarr

Thereadbuilt-incanalsoreceivewordsinputbytheuserintoanarray.echo-n"Pleaseinputanarray:"read-aarrIftheuserentersggffkkbb,then$arrgetsfourelements.

VARIABLEMODIFIERSBashprovidesnotationstomakevariableusageevenmoreflexibleforadvancedscripting. The value obtained from a variable can be modified before it isintroducedintoacommandorexpression.Youcan

1. Specifythevaluereturnedincaseavariableisunset(doesnotexist)ornull(Table5.3).

2. Returnasubstringofthevariablevalue(Table5.4).

VariableTestingModifiers

Forexample,ascriptrequiringonecommand-lineargumentmayuse

file=${1:?"Usage:$0filename"}

Notethatthe:inTable5.3canbeomittedfromthenotationsinTable5.3,anditmeansthetestisonlyfortheexistenceofthevariableandnotforitbeingnull.Bash alsomakes it easy to obtain a substring from the value of a variable

(Table5.4).VariableSubstringModifiers

Let’slookatsomeexamplesofsubstringmodifiers(Ex:ex05/strModifier.sh).

file=/tmp/logo.jpg${file:3}##isp/logo.jpg${file:3:5}##is

p/log${file#*/}##istmp/logo.jpg${file##*/}##islogo.jpg

(tail)${file%/*}##is/tmp(dirnameorhead)${file%.jpg}or

${file%\.*}##is/tmp/logo(root)${file##*\.}##isjpg

(extension)

Whenappliedtothepositionalparameters($*and$@)ortoarrays($array[*]and$array[@]), thefirstmodifier inTable5.4producesa listofwordsfromasubarray.Whereas,theothermodifiersinthetableeachproducesalistofwordsby acting on each value in the given array. Here is how it works (Ex:ex05/arraymod.sh).

pictures=(a.jpgb.jpgc.jpgd.jpg)echo${pictures[*]:2}##c.jpg

5.17

d.jpgecho${pictures[*]%.jpg}##abcdnames=(${pictures[*]%.jpg}

)##isarray(abcd)

Asanotherexampleofvariablemodifiers,considerthefunction

functionlatex(){/usr/bin/pdflatex${1%.tex}.tex&&

\/usr/bin/acroread${1%.tex}.pdf}

Themodifier $1%.texmakes it possible to use either of the following twowaystoinvokethelatexfunctionlatexpathname.texlatexpathnametocreateandviewthepdffilecreatedfromthegivenLaTeXfile.

THEHEREDOCUMENTItispossibletoincludeinascriptinputthatisnormallyenteredinteractively.InShell script terminology, this type of input is known as ahere document. Forexample, you may create a script (Ex: ex05/newyear.sh) that contains thefollowing:

mutt-s´HappyNewYear´>>ABCTodayis`date`andhowtimeflies.May

IwishyouaveryhappyandprosperousNEWYEAR.signed...ABC

Thepurposeofthisfileistoinvokethemuttcommand(foremail)andsendamessagetoeachnameonthealiaslistcalledfriends.Theheredocumentconsistsofall textbetween the firstABCand the secondABCona linewithoutothercharactersorwhitespace.Havingsetupthisfile,youthencanissueat0010aJan1happynewyeartoschedulethegreetingtobesentoutat12:10A.M.onNewYear’sDay.Theheredocumentisactuallyaformofinputredirection.Afterthe<<isan

arbitrary word (in this case, EOF) followed by a NEWLINE that delimits thebeginningandendoftheheredocument.Thegeneralformofaheredocumentiscommand<<wordzeroormorelinesofinputtextincludedherewordThedelimiterwordisnotvariable,filename,orcommandsubstituted.Thelast

linemustcontainonlythesamewordandnoothercharacters.Theinterveninglines are variable and command substituted, but SPACE, TAB, and NEWLINE

5.18

charactersarepreserved.Theresultingtext,uptobutnotincludingthelinewiththeenddelimiter,issuppliedasstandardinputtothecommand.Anexampleisthetimestampscript(Figure5.4).

Figure5.4ThetimestampScript

The here document contains a variable substitution and two commandsubstitutions.Thehostnamecommanddisplaysthenameofthehostcomputer.Thedatecommanddisplaysthedateandtime(Ex:ex05/timestamp.sh).Substitutionscanbesuppressedwithinaheredocumentbyquotingallorpart

ofthestartingdelimiterwordwith",‘,or’,forexample,

\EOF´here´a"b"`a`b

Notethatacorrespondingenddelimiterdoesnotneedanyquotes.If<<-isusedinsteadof<<fortheheredocument,thenanyleadingTABsin

front of the input lines and the delimiter line will be stripped away, makingindentingthescriptsourcecodeforeasierreadingpossible.Also,iftheheredocumentisasinglestringwithnowhitespace,youmayuse

instead(Ex:ex05/herestr.sh)

>>>any_string

MOREONFUNCTIONSWehavealreadyseenBashfunctionsinChapter3,Section3.15.Eachfunctiongives rise to a new Shell-level command that can be used just like any othercommand—interactivelyon thecommand lineoraspartofaShell script. InaShell script, you may call functions defined earlier in that script as well asfunctionsmadeavailablefromtheinvokingShell.IftheinvokingShelldefinesa

functionxyz, thenit ismadeavailableforShellscriptswithexport-fxyz. It isrecommended that you avoid this feature and make each Shell script self-sufficientbyincludingdefinitionsofallthefunctionsitneeds.UnlikefunctionsingeneralprogramminglanguagessuchasCorC++,Bash

functionshavetheirownwayofpassingargumentsandreturningvalues,aswewillexplain.

FunctionArgumentsABashfunctionisdefinedwithoutanynamedparameter.Thus,thefollowingisimpossible:

functioncompare(str1,str2)##wrong,noparametersallowed{...}

Instead,anyargumentspassedinafunctioncallareaccessedfromwithinthatfunctionusingthepositionalparameters$1,$2,andsoon.Thus,compare(Ex:ex05/strcompare.sh)canbecodedasfollows:

functioncompare(){localstr1="$1";##1stargumentlocal

str2="$2";##2ndargumentif[[$str1==$str2]]thenecho0;elif

[[$str1<$str2]]thenecho1;elseecho-1;fi}

The keyword local declares variables local to the function (not accessiblefromoutsidethefunction).Hereisasamplecall:compare"apple""orange";Arrayscanalsobepassed in functioncalls.The followingfunctiondisplays

anyarraythatispassedtoit(Ex:ex05/arrusage.sh).

functiondisplayArray(){echo-n"(";forel##iteratesover

positionalparameters(a)doecho-n"\"$el\""doneecho")";}

Saythatwehaveanarrayprime=(2357111317),thenwecanpassallthearrayelementsinacall

displayArray"${prime[@]}"

resultinginthedisplay

("2""3""5""7""11""13""17")

The function displArray works by iterating over the positional parameterspassed(linea).Normally, arguments are passedbyvaluewhen a copy of the value of each

argument is passed to the called function.However, it is also possible to pass

arguments by reference when the variable itself (a reference to its value) ispassedinsteadofitsvalue.Toillustratepassbyreference,considerthefunction

functionaddOne(){let$1+=1;}

HereisacalltoaddOnewithareferenceargumentn(insteadof$n).

n=12;addOnen;##functioncallwithreferenceargumentecho$n##

13

Whenweusen,insteadof$n,inthecalltoaddOne,the$1insidethefunctionevaluates to the symbol n. Thus, the code let $1+=1 is the same as let n+=1whichexplainshownbecomes13after thefunctioncall. Ifwewish toaccessthe value of n inside addOne, we can use the indirect reference evaluationnotation

${!1}##meanseval\$$1or$n

Hence,wemightimprovethefunctionasfollows(Ex:ex05/addOne.sh):

functionaddOne(){echo${!1};##displays$nlet$1+=1;##let

n+=1echo${!1};##displays$nagain}

Ingeneral,wehave

x=y;y="abc"echo$x##displaysyecho${!x}##displaysabc

Passingbyreferencecanbeusefulinpractice.Forexample,wecandefineafunctionsetenv tomakesettingofenvironmentalvariables(Chapter3,Section3.10)easier(Ex:ex05/setenv.sh).

functionsetenv(){eval$1=\$2;export$1;}

Withthisfunction,youcansetthecommandsearchpath(Chapter3,Section3.4)withonecall:setenvPATHyourdesiredpathstringThe indirect reference evaluation also allows you to pass an array by

reference,aswewillseeinthenextsubsection.

ReturnValueofaFunctionLet’swriteafunctionsumthataddstogetherallnumbersinanygivenarrayandreturnsthetotal(Ex:ex05/sum.sh).

functionsum(){localtotal=0;##localvariableforidolet

total+=$i##or((total+=$i))doneecho$total##returnvalue

(I)}s=$(sum${prime[@]})##callingsumandgetvalue(II)echo$s

##58

Noteherethatwereturnavaluebyechoingit(lineI)andcapturethereturnedvaluewithcommandsubstitution(lineII).Alternatively,wecanpassthetotalbackinareferenceparametermyTotal.To

dothat,werevisethefunctionsumtonewSum.Whileweareatit,wealsopasstheprimearrayintothefunctionbyreference(Ex:ex05/newSum.sh).

functionnewSum(){localp="$1[@]";##$pis"prime[@]"foriin

"${!p}"##evaluates${prime[@]}dolet$2+=$i##$2isthesymbol

myTotaldone}myTotal=0newSumprimemyTotal##passingtworef

parametersecho$myTotal

ThethreelinesinnewSumwithcommentsdeserveclosestudy.Apredicate is a function that tests for a condition and returns true or false.

Hereisapredicatethattestsifafileismorerecentlymodifiedthananotherfile(Ex:ex05/newer.sh).

functionnewer(){if[[$1-nt$2]]##iffile$1isnewerthan

file$2thenreturn0##exitstatus0meanstrueelsereturn1##

falsefi}

Thereturnstatementinafunctionreturnsanexitstatus (asmall integer lessthan256).Thevalueoftheexitstatusisavailableinthespecialvariable$?rightafterthefunctioncall.Ifafunctiondoesnotcallreturn,thenitsexitstatusisthatof the last statementexecutedbefore the functionended.Apredicate function,suchasnewer,canbeuseddirectlyinconditionalexpressions.Hereisacalltonewer.

ifnewerfile1file2then...fi

However,asyoumayhaverealized,thepredicatefunctioncanbesimplifiedto

functionnewer(){[[$1-nt$2]]##availablealsois-otfor

olderthan}

Finally, it ispossibleforafunctiontoreturnavaluebyassigningit tosomeglobal variable. Because there is always the danger of some other codeusing/setting the same global variable for some other purpose, we do notrecommendthisapproach.

5.19

5.20

REDEFININGBASHBUILT-INFUNCTIONSIfyoudefineafunctionwhosenamecoincideswithaBashbuilt-incommandoraregularcommand, then thatnameinvokes thefunction instead.However, thecommandsarestillaccessible:

builtincommandNameargs(invokesthebuilt-incommandName)command

commandNameargs(invokestheregularcommandName)

Hereisasimpleexamplethatredefinescdtodoadirectorylistingeachtimeitiscalled.

functioncd(){builtincd"$1"/bin/ls-l}

Often,ShellscriptscanbewrittenasShellfunctionswithlittlechangeandnoimpactonhowtheyarecalled.Byimplementingascriptasafunction,youcanplaceitinyourShellinitializationfileandmakeitpartofyourShell.

EXAMPLEBASHSCRIPTSNow let’s consider some more substantial Bash scripts. You can find thesescriptsintheexamplecodepackage.Totestthemyourself,placethescriptsinafolder,$HOME/bin,forexample,andopenuptheirexecutionpermissions.Also,makesurethatthefolderisonthecommandsearchPATH.

Example:RemovingUnwantedFilesModernoperatingsystemssuchasLinuxmakeiteasytocreate,download,copy,andotherwisemanipulatefiles.However,mostusersarehesitantaboutremovingfiles, and the clutter of obsolete files canbe anuisance let alonewastingdiskstorage.Onereasonisthesheertediumoflookingthroughfilesanddiscardingthose that are no longer needed. Thus, although disk storage is decreasing incost, new supplies of additional disk space never seem to quite match thedemand. The clean.sh script provides some help (Ex: ex05/clean.sh). Thecommandclean.shdirectorydisplaysfilenamesinthegivendirectory,oneatatime,andallowstheuserto

decideinteractivelywhetherornottokeepordeletethefile.Thisscriptislongerand will be explained in sections. The clean.sh script begins with argumentchecking:

#!/bin/bash##bashscriptclean.sh##usage:clean.shdir##helpsto

rmunwantedfilesfromadirectoryif(($#!=1))##numberof

argsisnot1thenechousage:$0directoryexit1fidir="$1"if![[-

d"$dir"&&-w"$dir"]]##notadirornotwritablethenecho$dir

notawritabledirectoryechousage:$0directory;exit1ficd

"$dir";

After checking for correct input, the script changes the current workingdirectorytothegivendirectory.Aforloopisusedtotreateachfile(*)inthecurrentdirectory(line1).Onany

giveniteration,if$fileisnotanordinaryfileornotreadable,thenitisskippedviacontinue(line2).Foraregularfile,aninfiniteloop(line3)isusedtohandleitsprocessing.Wemustbreakfromthisinnerwhilelooptogettothenextfile.Foreachfile,thefilenameisclearlylistedwithls,andtheuserisprompted

with*****Deletefilenameornot??[y,n,e,m,t,!orq]:indicatingsevenpossible(single-character)responses(terminatedbyRETURN).

Userinputisreceivedviaread(line4)andtreatedwithacaseconstruct(line5).

forfilein*##(1)doif![[-f"$file"&&-r"$file"]]then

continue##(2)fiecho""##ablankline/bin/ls-l"$file"while:

##(3)doecho-n"***Delete$fileornot??"echo-n"[y,n,e,m,

t,!orq]:"readc##(4)case$cin##(5)y)if[[!-w"$file"

]]thenecho$filewrite-protectedelse/bin/rm"$file"if[[-e

"$file"]]thenechocannotdelete$fileelseecho"+++++$file

deleted"fifibreak;;##tohandlenextfilen)echo"-----$filenot

deleted"break;;e)${EDITOR:-/bin/vi}"$file";continue;;

Thecasesfory,nareclear.Notetheuseofbreaktoleavethewhileloopandprocessthenextfileunderfor.Theecaseinvokestheuser’sfavoritetexteditor(setbytheenvironmentvariableEDITOR)orvi.The choices m and t offer the user a chance to examine the file before

decidingonitsdisposal.Notetheuseofcontinuetogobacktothewhileloop.

m)/bin/more"$file";continue;;t)/bin/tail"$file";continue

;;!)echo-n"command:"readcmdeval$cmd;;##(6)q)break2;;##

break2levels*)##helpforuserechocleancommands:followedby

RETURNecho"yyesdeletefile"echo"ndon'tdeletefile,skipto

nextfile"echo"eedit/viewfilewith${EDITOR:-/bin/vi}"echo"m

displayfilewithmore"echo"tdisplaytailoffile"echo"!Shell

escape"echo"qquit,exitfromclean";;esacdonedone

Inaddition tocallingonmoreandtail, theusermayexecuteanycommand(with!)tohelpmakethedecision.Inthiscase,thescriptreadsacommandstringfrom the user and executes it as a Shell command using eval (line 6), whichexecutesthestringasalineofcommand.Notethatthevariable$filecanbeused

inthiscommandstringandthatthereisnorestrictionastowhatcommandcanbeused.Somecommandstringstheusermayenterare

head$filecp$file...mv$file...

The q case quits from the script. For all other cases,we display amenu ofsingle-lettercommandsforclean.shandproceedtoanotheriterationofwhileforthe same $file. If the user mistypes and enters a character other than thoseexpectedby the script, thewhile loop is restarted.Also,note that the clean.shscriptprovidesfeedback,tellingtheusereachactionithastaken.

Example:ConditionalCopyTheccp.sh(conditionalcopy)scriptcreatesacommandthatcopiesfilesfromasourcedirectorytoadestinationdirectoryusingthefollowingconditionsonanyordinaryfiletobecopied:

1. Iffileisnotindestination,copy.2. Iffileisindestinationbutnotasrecentasthatinsource,copy.3. Otherwise,donotcopy.

Thescript(Ex:ex05/cpp.sh)beginswithargumentchecking:

#!/bin/bash##bashscriptccp.sh##usage:ccp.shfromDirtoDir[

file...](($#&#x003C;=2))&&[[-d"$1"&&-d"$2"]]\||{

echousage:$0fromDirtoDir[file...];exit1;}##(A)from=$1;

to=$2if(($#&#x003C;2))##filessuppliedthenfilenames=${@:3}

##(B)else##allfilesinfromDirpushd$fromfilenames=(*)##

(C)popdfi

Unlesswehaveatleasttwoarguments,thefirsttwobeingdirectories,wewillerrorout(lineA).Thisworksbecausethenextoperandof||(logicalor)willbeevaluatedonlyifthepreviousoperandisfalse.Given the correct arguments, the script proceeds to record the from and to

directoriesandtostorethefilestobeprocessedinthefilenamesarray.Ifthefilestobecopiedaregivenonthecommandline,theyarepickedup(lineB)withavariablemodifier (Section 5.16).Otherwise, all files in the from directory areincluded(lineC).Nowthestageissettoprocesseachfiletobecopiedconditionally.Aforloop

isusedtogothrougheachelementinthearrayfilenames(lineD).

forfilein"${filenames[@]}"##(D)doecho$file;if[[!-f

"$from/$file"]]##notaregularfilethencontinue##skipfiif[[

-f"$to/$file"]]##$fileinfolder$tothenif[["$from/$file"-

nt"$to/$file"]]##(E)thenecho/bin/cp\"$from/$file\"

\"$to\"/bin/cp"$from/$file""$to"fielse##$filenotinfolder

$toecho/bin/cp\"$from/$file\"\"$to\"/bin/cp"$from/$file"

"$to"fidone

If$fileispresentin$to,thenwechecktoseeiftheversionin$fromisnewer(lineE).Anyfilecopyingactionisdisplayedtoinformtheuser.Notetheuseofdoublequotes(")throughoutthescripttoguardagainstmultiwordfilenames.

Example:TotalFileSizesInthisexample(Ex:ex05/total.sh)weusearecursiveShellfunctiontocomputethetotalnumberofbytescontainedinallfilesinacertainfilehierarchy.Theducommand only provides a rough accounting in kilobytes. The script total.shrecursively descends through a directory hierarchy and sums the file sizes byextractinginformationprovidedbythelscommandoneachfileinthehierarchy.

#!/bin/bash##bashscript:total.sh##computetotalbytesin

files##underanygivendirectoryhierarchy[[$#==1&&-d"$1"]]

\||{echousage:$0directory;exit1;}

After thecheckingcommand-lineargument,weproceedtodefineafunctiontotal which sums up the file sizes for all files in the current directory andrecursivelydescendsthedirectoryhierarchy.

functiontotal(){localcount=0##bytesusedinsideworkingdirfor

filein.**##allfilesincludinghiddenonesdoif[[-f"$file"

]]thenfl=($(/bin/ls-ld"$file"))##(a)letcount+=${fl[4]}##

(b)continuefiif[["$file"==*\.||"$file"==*\.\.]]##

(c)thencontinuefi

Foraregularfile, the ls -loutput iscaptured in thearrayfl (linea),and thebytesizeisaddedtothetotalbytecount(lineb).Thespecialfiles.and..areexcluded(linec).For a subdirectory,we temporarily change to thatdirectory (lined), include

the sum obtained by a recursive call to total (line e), and then change thedirectoryback(linef).

if[[-d"$file"]]thenpushd"$file"&#x003C;/dev/null##(d)y=$(

total)##(e)letcount+=$ypopd&#x003C;/dev/null##(f)elseecho

\"$file\"notincludedinthetotal&#x003C;&2fidoneecho$count##

(g)}

Notethatweredirectedechooutputtostderrandoutput by pushd and popd to the data sink /dev/null. The only output to

stdout allowed is the total count (line g). This is the way the function totalreturnsavaluethatispickedupinacallwithcommandsubstitution(lineseandh).

dir="$1"cd$direcho"Totalfor$dir="$(total)Bytes##(h)

Example:SecureFileTransferTheneedoftenarisestotransferfilesbetweencomputers.Thesftpcommandiscommonlyused for thispurpose.WewillwriteaBashscript (Ex:ex05/mput)thathelpsfileuploadanddownloadwithsftp.Thescriptwillworksmootherifyouhavealreadysetuppassword-lessSSHandSFTPbetweenyour local andremotehosts(Chapter7,Section7.6).The ideanow is to setupa specialdirectory foruploadanddownloadona

remotecomputer(say,atworkorschool)andusethemputormgetcommandtoinvokethescripttotransferfilestoandfromit.Hereisthescript.

#!/bin/bash##uploadanddownloadfilesusingsftp##Usage:mput

"*.jpg"ormget"*.pdf"####begincustomizable:user,host,

rdiruser=pwang##(1)host=monkey.cs.kent.edu##(2)rdir=tmp##

(3)####endcustomizableif[[$0==*mget*]]##(4)then

action=mgetelseaction=mputfi/usr/bin/sftp$user@$host

&#x003E;&#x003E;HEREcd$rdir$action"$@"HERE

Customizable parameters are user (user ID), host (remote host), and rdir(remotefolder)(lines1-3).Thescriptisnamedmputwithahardlink(Chapter6,Section6.2)mgettoit.lnmputmgetSo the script canbe invokedaseithermputormget.The sftp action is set

accordingtothevalueof$0(line4).These values being set, the sftp can be invoked with a here document to

performthedesireduploading/downloading.Forexample,

mgetmemo.pdf(downloadsmemo.pdf)mget´*.pdf´(downloadsallpdf

files)mput´*.jpg´(uploadsalljpgfiles)

Example:ResizingPicturesConnect your digital camera to your Linux computer and download a set ofpictures toyourPicturesfolder.Often,youneedtoscaledownthepicturesforemailingorpostingontheWeb.Here isascript thatmakes the taskeasy(Ex:ex05/resize).resize’75

5.21

willreduceeach.jpgfileby75%underthenewnamestrip001.jpgetc.Theresizescriptfirstprocessesthecommand-linearguments.

#!/bin/bash##resizeasetofpictures##Usage:$0size-factor

newNamepic1.jpgpic2.jpg...##scalesallpicsbysize-factor

into##newname1.jpg,newname2.jpg...(($#>3))\||{echo

usage:"$0\"50%\"newNamepic.jpg...";exit1;}sz=$1;name="$2";

declare-ik=1

Then,weresizeeachpicture (line i)andsave itundersequentialnumberingafter the given new name using three-digit numbers (lines ii and iii). Thenotation"$@:3"producesalistofallnamesonthecommandlinestartingfromthefourthword(Section5.16).

forpicin"${@:3}"##(i)doif(($k>10))##(ii)then

n="00$k"elif(($k>100))##(iii)thenn="0$k"fiecho"convert-

resize$sz\"$pic\"\"$name$n.jpg\""convert-resize$sz"$pic"

"$name$n.jpg"letk++done

The convert command is part of the ImageMagick tool that is commonlyfoundonLinuxsystems.Seemanconvertformoredetailsonitsusage.

DEBUGGINGSHELLSCRIPTSWhenaShellscriptfailsbecauseofsyntaxproblems,thesyntaxerrorwillcausea display of some unexpected token. You usually will get an error messagestating the token and the line number containing the token. This means yoursyntaxproblemisonorbeforethatline.Takeacloselookatyourcode,andyouusuallycanfindtheproblemortypoandfixit.Youcanalsoplaceecho commands at appropriate places to showvalues of

variablestohelpyoucatchthebug.Suchechocommandscanberemovedafterthedebuggingisdone.Oryoumayuseaconditionalecho,

functiondbecho(){[[${DEBUG:-off}==off]]||echo"$*"<&2}

We see that the function dbecho produces output to the stderr unless thevariable DEBUG is null, not set, or set to off. Thus, you would placeDEBUG=on at the beginning of your script to enable dbecho output andcommenttheDEBUG=onouttodisableit.Ifyoustillcannotfindtheproblem,thenplacingthecommandset-x(turnsontracing)set+x(turnsofftracing)

5.22

in your script will turn tracing on/off from selected places. Tracing willdisplayeachcommandbeforeitisexecuted.Moretracinginformationcanbedisplaywithbash-xscript.shto run the script with trace turned on within Bash. This will show all

commandsexecuted,includinganyinitfiles.

ERRORANDINTERRUPTHANDLINGAnerrormayoccurduring theprocessingof aShell script at severaldifferentstages.Asyntaxorsubstitutionerrorwillresultintheprematureterminationofthe script. However, if a regular command invoked by the script runs into anerror, the interpretation of the script continues with the next command in thescript.Errormessagesareproducedandsenttothestandarderroroutput.Tohelpdebugging,thestderrcanberedirected(Chapter3,Section3.5)toafile.InLinux,whenaprogramterminates(becauseofeithercompletionorerror),

an exit status is set to a small integer value to provide an indication of thecircumstances underwhich executionwas terminated.By convention, the exitstatusis0ifterminationisnormalandgreaterthan0ifterminationisabnormal.AShellbuilt-incommandgivesanexitstatusof0whensuccessfulandanexitstatus of 1when unsuccessful. The special Shell variable $? is set to the exitstatus after the execution of each command. The value of $? is 0 if the lastcommandwassuccessfulandgreaterthanzero(usually1)ifitfailed.Totestwhetheracommandhasfailed,thefollowingconstructoftenisused:

ifcommandthencommandstoexecuteifcommandsucceedselsecommands

toexecuteifcommandfailsfi

InterruptHandlingAn interrupt is an asynchronous signal sent to a running program by anotherprocess or through the keyboard by the user. The user can send an interruptsignaltoaShellrunningascriptbytypingintheinterruptcharacter,normallyCor DELETE. There are various system-defined signals that can be sent to anexecutingprogramusingthekillcommand.SignalswillbediscussedinChapter11,Section11.16.Fornow,itissufficienttostatethatkill-2pidsends the interrupt signal2 to theprocesspid,whichcauses it to terminate.

Theprocesspid canbegiven either as a jobidor as a process number. If thisdoesnotterminatetheprocess,use

5.23

kill-9pidwhichsendssignal9,unconditionallyterminatingpid.ThedefaultresponseofaShellexecutingascriptistoterminateifitreceives

an interruptsignal,but thiscanbemodified.TheBashbuilt-incommand trapcontrols the action of theShellwhen specific interrupt signals are received orwhenspecificeventstakeplace.trapcommandsigThegivencommand (givenasastring inquotes)willbeexecutedwhen the

Shellreceivestheindicatedsignalorevent.Thesigisasignalnumberorsignalname(seeman7signal).IfsigisDEBUG,thenthecommandisexecutedaftereachcommandinthescript. Ifsig isEXIT, thecommand isexecutedafter theShell script is done. Without any arguments, trap displays a list of trappedsignals.Often,aShell scriptwillcreatea temporary file thatwillbe removedat the

endofthescript.Forexample,

...spell$file<|/tmp/badwords$$......##atendofscript/bin/rm-

f/tmp/badwords$$

The value of the special variable $$ is the process number of the runningscript, and its use heremakes the temporary file name unique to the process.However,thatfilecanbeleftunremovedifthescriptterminatesduetoasignalinstead of completing all commands. To fix that problem simply add (Ex:ex05/trap.sh)

trap"/bin/rm-f/tmp/badwords$$"EXIT

beforecreatingthetemporaryfile.Theactionplacesthegivenrmcommandas something to execute upon normal or error exit of the script. As aconsequence,thermcommandattheendofthescriptisnolongernecessary.

THEPERLANDPHPALTERNATIVESShellscriptingisnot theonlywaytowritescripts toautomatetasks.Formorecomplicatedtasksorforproblemsinvolvingstructureddatafiles,manyprefertouse Perl, the Practical Extraction and Report Language, over simple Shellscripts.ThePerllanguageisoutsideofthescopeofthistext,andtherearemanybooksdedicatedtoPerl.Wewillgiveonlyabriefintroductionhere.Perl is a portable, command-line–driven, interpreted programming/scripting

language. Written properly, the same Perl code will run identically on

5.24

Linux/UNIX,Windows,andMacoperatingsystems.Mostlikely,you’llfindthelatestversionofPerlpre-installedonyourLinuxsystem.ThePerlscriptinglanguageisusuallyusedinthefollowingapplicationareas:

DOS,Linux/UNIXcommandscriptsWebCGIprogramming(Chapter7,Section7.20)TextinputparsingReportgenerationTextfiletransformationsandconversions

Perl 1.0 was released December 18, 1987, by Larry Hall with the followingdescription:

Perl is an interpreted languageoptimized for scanningarbitrary text files,extracting information from those text files, andprinting reportsbasedonthat information. It’s also a good language formany systemmanagementtasks. The language is intended to be practical (easy to use, efficient,complete)ratherthanbeautiful(tiny,elegant,minimal).Itcombines(intheauthor’sopinion,anyway)someofthebestfeaturesofC,sed,awk,andsh,sopeoplefamiliarwiththoselanguagesshouldhavelittledifficultywithit.(Languagehistorianswillalsonotesomevestigesofcsh,Pascal,andevenBASIC—PLUS.) Expression syntax corresponds quite closely to Cexpressionsyntax.Ifyouhaveaproblemthatwouldordinarilyusesedorawkor sh, but it exceeds their capabilitiesormust run a little faster, andyoudon’twanttowritethesillythinginC,thenPerlmaybeforyou.⋯

Perl5.0,acompleterewriteofPerladdingobjectsandamodularorganization,was released in 1994. The modular structure makes it easy for everyone todevelop Perl modules to extend the functionalities of Perl. In late 2017, thenewestversionwasPerl5.27.The Comprehensive Perl Archive Network (CPAN; www.cpan.org) was

establishedtostoreanddistributePerlandPerl-relatedsoftware.InadditiontoPerl,PHP(Chapter9,Section9.17)isanothergreatchoicefor

writingscripts.

FORMOREINFORMATIONAt the book’s companion website (http://ml2.sofpower.com), you’ll find acomplete example code package containing ready-to-run code files for the

5.25

examplesinthisbook.TheShellscriptexamplesinthischapterare,ofcourse,partofthispackage.YoucangetaquickreferenceforBashbymanbashandyou’llseemanydetailsincludingalistofbuilt-infunctions.OntheGNUBashhomepage(www.gnu.org/software/bash/)youcanfindthe

BashManual which is a complete reference for Bash. You’ll also be able todownloadthelatestreleaseofBash.POSIX defines standards for utilities, the Shell programming language, the

Shellcommandinterface,andaccesstoenvironmentvariables.ScriptsfollowingthePOSIXstandardcanbemuchmoreportable.Foradditionalinformation,seePortable Operating System Interface (POSIX) – Part 2: Shell and Utilities,publishedbyIEEE(IEEEStd1003.2-1992).

SUMMARYBashprovidesmany features forwriting scripts to automate tasks for yourselfand others. Proficiency in script writing can make you more efficient andeffectiveonLinux.AShellscriptisanexecutabletextfilewhosefirstlinemustfollowaspecial

convention.Suchafilecanbeinvokedviaexplicitorimplicitinterpretationandis executed by a subshell of the invokingShell.Command-line arguments arepassedintoaShellscriptandareavailableinthescriptaspositionalparameters.Other values can be transmitted to the script by environment variables. Upontermination,aShellscriptreturnsanexitstatustotheinvokingShellwhichcanaccess this value via the special variable $?. A zero exit status indicatessuccessfulcompletionofthescript.Bashprovidesagoodnumberofconstructsforscriptwriting.

Loopingconstructs:for,while,anduntilDecisionmakingconstructs:case...esac,if...then...else...fiTestexpressions:[[...]],((...)),andanycommandexitstatusLogicaloperators:&&,||,!Arithmeticexpressions:let,((...))Globpatternmatching:==,!=,andcaseRegularexpressionpatternmatching:=ArraysandfunctionsVariablemodifications:with:,%,#,

5.26

Functions are invoked just like commands. A function takes positionalparametersandproducesanexitstatus.Argumentscanbepassedbyvalueorbyreference.Avaluecanbereturnedbyechoingittostdout,settingareturn-valuereferenceparameter,orsettingtheexitstatus.Manypracticalscriptshavebeengivenasexamplesandtheready-to-runcode

ismadeavailableintheexamplecodepackageatthebook’swebsite.DebuggingtechniquesaswellaserrortrappingforShellscriptshavebeendiscussed.In addition to using BASH, we can write more complicated scripts using

languagessuchasPerlandPHP.

EXERCISES1. Whatisthedifferencebetweenthesetwowaysofinvokingascriptabc.sh:

bashabc.shabc.sh2. Bash allows the use of $0, $1, $2, and so on to refer to positional

parameters.Isitpossibletouse$10,$15,andsoon?Explain.3. Thecharacter*isaspecialcharacterinBash.

1. Explainhowitisusedforfilenameexpansion.2. ListatleasttwosituationsinBashsyntaxwherethecharacter*isnot

quoted, but does not serve the function of filename expansion orglobbing.

4. UsingthecmdsearchexampleinSection5.6asaguide,writeaBashscriptcmdfind. cmdfind pattern The script takes a regular expression patternargumentandfindsallcommandsonPATHthatmatchthegivenpattern.

5. Thecharacter@isaspecialcharacterinBash.

1. Explainthemeaningof$*,$@,"$*",and"$@".2. Howabout$arr[*],$arr[@],"$arr[*]",and"$arr[@]"?

6. Explainhowthecharacter#isusedinBashscripts:asacommentcharacter,as the number of positional parameters, and as the number of arrayelements.

7. Refertothesectiononvariablemodifiers(Section5.16)andseeifitgivesawaytochange thecaseofcharacters inavariable. Ifnot, findoutwhatBashparameterexpansionnotationsdothat.

8. Bash also supports conditional expressions using [ ... ]. Explain thedifferencebetweenthatandthe[[...]]conditionals.Whataboutthe((...))conditionals?

9. Canyousuggestwaystoimproveclean?Whataboutcleaningoutonlyold

1234

files? Is an undo or undelete feature desirable? What about recursivelycleaning out subdirectories as an option?Howwould you implement theimprovements?

10. WriteaShellscripttochangethenamesofallfilesoftheform*.JPGinadirectory(suppliedasargument1)sothattheyhavethesamerootasbeforebutnowendin.jpg.Generalizethisscriptsothatanytwoextensionscouldbeused.

11. WriteaShellscriptdeletethatmimicsthewayrmoperates,butratherthanerasing any files, it would put them in a user’s .Trash folder. Write anadditional Shell script undelete to make these files reappear where theyweredeleted.

12. ReimplementthedeletescriptofthepreviousexerciseasaBashfunction.DiscusstheprosandconsofShellscriptsvs.functions.

13. WriteaShellpredicatefunctionevenp that takesan integerargumentandtestsifitisanevennumberornot.

14. WriteaShell functionfindfile so that findfilenamedir1dir2 ... searchesthenamedfileinthedirectoriesspecified.Ifthefileisfoundinoneofthedirectories, the current directory is changed to it. Why do we need toimplementitasafunctionintheinteractiveShellratherthanaregularShellscript?

15. Improvethemget/mputscriptsothatitcanalsobeinvokedasrvandwillallowyoutoviewaremotePDF(.pdf)orMSWord(.doc)filelocally.Nocopyoftheremotefilewillbeleftonthelocalfilesystem.

16. Designandwrite aBash scriptSend-Mail that is convenient to use fromthe command line. The script sends an email by invoking thunderbirdfromwithinthescript.

17. Bash also supports associative arrays. Find out how it works andexperimentwith(Ex:ex05/asso.sh).

18. Findout aboutDash.Find themajordifferences amongTcsh,Csh,Dash,Bash,andSh.OnsomedistributionsShisasymboliclinktoBash.TheearlierBashconstruct[]canstillbeusedbutissupersededbythe[[]].Unfortunately,inside[[]]theusual<=and>=arenotrecognized.IfyourLinuxdoesnot include thefortunecommand,youcanget itby installing thefortune-mod

package(Chapter8,Section8.2).

Chapter6

TheFileSystem

Storingdataasfiles thatcanbeaccessedimmediatelybyprogramsisessentialformodern operating systems. Files are identified by their filenames andmaycontainmanykindsofdata.Forexample,afilemaycontainaletter,areport,aprogram written in a high-level language, a compiled program, an organizeddatabase,alibraryofmathematicalroutines,apicture,oranaudio/videoclip.Theoperatingsystemprovidesaconsistentsetoffacilitiesallowingtheuser

to create, store, retrieve, modify, delete, and otherwise manipulate files. Thephysical storagemedia (usuallyhigh-speedmagneticorsolid-statediskdrives)aredividedintomanyblocksoflogicalstorageareas.Afileusesoneormoreoftheseblocks,dependingon theamountofdata in thefile.Blocksareusedandfreedasfilesarecreatedanddeleted.Theprogramthatcreates,stores,retrieves,protects, and manages files is the file storage system (or simply file system)whichispartofthekernelofanymodernoperatingsystem.Historically, theUNIXoperating systemevolved froma project to design a

new computer data storage system at the then Bell Laboratories. ThishierarchicalfilestoragesystemisahallmarkofUNIX.AsUNIXevolved,sodidtheimplementationofitsfilestoragesystem.LinuxbasicallyadoptedthesameUNIX file storage system implementation. The file system usually consists ofoneormoreself-containedfilemanagementunits,eachisknownasafilesystem.Also, the Linux file hierarchy usually follows the File System Standard(FSSTND), allowing users to find important system files at the same filelocationsonanycompliantLinuxsystem.The file system affects almost every aspect of the operating system. In this

chapter,thefilesystemisdiscussedindetail,includingsuchtopicsastypeandstatusoffiles,accessprotection,filesystemstructure,implementation,extendedattributes(xattr),specialfiles,andnetworkedfilesystems.AclearunderstandingofhowLinuxtreatsfileswillbehelpfulforanyLinuxuser.

6.1 AFILELOCATIONROADMAPThefilesysteminLinuxismuchmorethanaplacetostoreuserfiles.Itcontainsthe operating system itself, application programs, compilers, network servers,shared libraries, documentation, system configuration and administration datafiles,mediamountpoints,logfiles,temporaryscratchareas,andsoon.Inotherwords, almost every bit of data and programming that is needed to boot thecomputerandkeepitworkingmustbesavedinthefilesystem.Linux systems generally follow the FSSTND in organizing the file system

hierarchy.ThismakesiteasyforLinuxuserstofindtheirwayondifferentLinuxsystems.TheRootDirectory:/

Table6.1 shows a typical organizationof the root folder (/) of the file tree.From your desktop, clicking on the Computer icon then selecting the FileSystemlinkbringsyoutotherootdirectory.Onthecommandline,cd/willdo.Wealreadyknowthatfilesandfoldersformatreehierarchyrootedat/.Eachfileon this file tree is uniquely identified by its full pathname, as we alreadymentionedinChapter1,Section1.5.Inside each user’s home directory, you’ll often find these standard folders:

Documents, Downloads, Music, Pictures, Videos, Desktop, and the hidden.Trash.Whenfilesandfoldersaccumulate,itcanbecomehardertolocateafilethat

youneed.SeeSection6.9andSection6.10forhelpfulcommands.

6.2 FILETYPESThefiletreecontainsdifferenttypesoffiles.

1. Anordinaryfilethatcontainstext,programs,orotherdata2. Adirectorythatcontainsnamesandaddressesofotherfiles3. A special file that represents an I/O device, disk drive, or a filesystem

partition4. Asymboliclinkthatisapointertoanotherfile5. Asocket(ordomainsocket)thatisusedforinter-processcommunication6. Anamed pipe that is away for inter-process communicationwithout the

socketsemantics

Thefirstcharacterinanls-llistingofafileisafiletypesymbol.Table6.2liststhedifferentfiletypesymbols.FileTypeSymbols

Symbol Meaning Symbol Meaning- Regularfile d Directoryl Symboliclink c Characterspecialfileb Blockspecialfile s Socketp Namedpipe Now,let’sdescribefiveof thefile typesinturn.Thesocketandnamedpipe

willbediscussedlaterinChapter12,Section12.6.

OrdinaryFilesAn ordinary file stores data of various content types. The entire file storagesystem is designed to store, retrieve, and manage ordinary files. Your homedirectoryisnormallywhereyoustoreyourownfiles.Filenames are character strings (it is best not to use any white space).

AlthoughLinux filenamesdonot require them, filesofdifferent content typesoften use different extensions. For example, a picture might use the .jpgextension.The Multipurpose Internet Mail Extensions (MIME) provides a standard

classification and designation for file content types. Files of different contenttypes often use well-known filename extensions for easy recognition andprocessing. There are hundreds of content types in use today. Many populartypes are associated with standard file extensions. Table 6.3 gives someexamples.ContentTypesandFileSuffixes

For a more complete list of content types and file suffixes, see the/etc/mime.typesfileonyourLinuxsystem.

DirectoriesFilesarestoredindirectories,andthatiswhytheyarealsoknownasfilefolders.A directory is a file whose content consists of directory entries for the filesplacedinthedirectory.Thereisonedirectoryentryforeachfile.Eachdirectoryentrycontainsthefilenameandthelocationofitsfileinformationnode(i-node).A filename is a sequence of characters not containing /. The maximum

sequencelengthisdependentontheversionoftheLinuxsystem.Itcanbeupto255charactersonmostsystems,butcanbenomorethan14charactersonsomeolderversions.Thei-nodelocationisanintegerindex,calledthei-number,toatable known as the i-list. Each entry in the i-list is an i-node, which containsstatusandaddressinformationaboutafileorpointstofreeblocksyettobeused.Theentirefilesystemmayinvolveseveralindependentandself-containedparts,eachknownasafilesystem.Eachindividualfilesystemhasitsowni-list.

SpecialFilesBy representing physical and logical I/O devices such as graphical displays,terminalemulators,printers,CD/DVDdrives,andharddrivesasspecialfilesinthefilesystem,LinuxachievescompatiblefileI/OanddeviceI/O.Thismeansthat an application program can treat file and device I/O in the same way,providinggreatsimplicityandflexibility.UnderFSSTND,allLinuxspecialfilesare under the directory /dev. There are two kinds of special files: a characterspecial file and ablock special file.A character special file represents a byte-orientedI/Odevicesuchasadisplayoraprinter.Ablockspecialfilerepresentsahigh-speedI/Odevicethattransfersdatainblocks(manybytes),suchasaharddrive.Typicalblocksizesare1024bytesand2048bytes.Specialfilesusuallyareownedbythesuperuser(root).Theownershipofa

terminalemulatorspecialfile(under/dev/pts/)issettotheuseroftheterminalforthedurationoftheterminalsession.

Links

Linuxallowsadirectoryentrytobeapointertoanotherfile.Suchafilepointeriscalledalink.Therearetwokindsoflinks:ahardlinkandasymboliclink.Aregularfileisanentryinadirectorywithanameandani-number.Ahardlink,orsimplyalink, isanentryinadirectorywithanameandsomeotherfile’si-number.Thus,ahardlinkisnotdistinguishablefromtheoriginalfile. Inotherwords,afterahardlinkismadetoafile,youcannottellthefilefromthelink.The net result is that you have two different directory entries referring to thesamei-node.Afilemayhaveseverallinkstoit.Ahardlinkcannotbemadetoadirectoryortoafileonanotherfilesystem.Thus,hardlinksallowyoutogivedifferentnamestothesamefilewithinthe

samefilesystem.Forexample,youmayhaveafilecalledreportandyouenterlnreportreport.txtthenthereportisalsounderthefilenamereport.txt.Theregularcommand ln isused tomake links.Thegeneral formsof the ln

commandareasfollows:lnfile makesalinktofileinthecurrentfolderlnfilelinkname establisheslinknameasalinktoexistingfilelnfile1...dir makeslinksindirtothegivenfile(s)Bydefaultlnformshardlinks.Itispermittedtoestablishalinktoafileeven

ifyouarenottheownerofthefile.Whendeletingafile(withthermcommand),thedirectoryentryofthefileisdeleted.Forrmfile tosucceed,youneedwritepermission to the parent directory of file, not the file itself. A file is onlyphysicallydeletedfromthefilesystemwhenthelastlinkofitisrmed.Thetotalnumberofhardlinkstoafileiskeptaspartofthefilestatus(Section6.4).

SymbolicLinksAsymbolic linkisadirectoryentry thatcontains thepathnameofanotherfile.Thus,asymbolic linkisafile thatservesasanindirectpointer toanotherfile.Formostcommands,ifasymboliclinkisgivenasanargument,thefilepointedto is tobeused.Forexample, if the fileabc isasymbolic link to thefilexyz,thencatabcdisplaysthecontentsofxyz.Therearesomeexceptions:rmabcremovesthedirectoryentryabc(evenifitisasymboliclink).Aswell,ls-labcdisplaysstatusinformationforabc(notxyz).Ifyougivethecommandrmxyz

6.3

then the symbolic link abc points to a non-existent file. If abcwere a hardlink,thissituationcouldnotoccur.Asymboliclinkisdistinguishablefromthefileitself,maypointtoadirectory,

andcanspanfilesystems.The-soptioncauseslntocreatesymboliclinks:ln-sfilenamelinknameUnlikeahardlink,herefilenamedoesnotevenhavetobeanexistingfile.The command ls -F displays a symbolic link with a trailing @. The ls -l

commanddisplaysasymboliclinkintheformLet’slookatanapplicationofsymboliclinks.Supposeyouhavetheclean.sh

Shellscriptinyourownhomedirectory,andyouwishtomakeitavailabletoallothers on your Linux system.Oneway to achieve this is tomake a link in asystem directory to your program. For example, you can issue the followingcommand:ln-s$HOME/cmd/clean.sh/usr/local/bin/cleanThisestablishesthecommandcleanasasymboliclinkinthesystemdirectory

/usr/local/bintoyourclean.sh.Assumingthedirectory/usr/local/binisonusers’commandsearchpath,thenoncethislinkisinplace,anewcommandcleanismade available to all users. Note that because of file protection, systemdirectoriessuchas/usr/local/binareusuallywritableonlybyasuperuser.

MOREONFILEACCESSCONTROLFromChapter1,weknow that fileshaveaccess control, and the file typeandaccesspermissionscanbedisplayedeitherbytheFileBrowsertoolor,byusingthe ls -l command. Also, you can change permissions of your own files andfoldersusingthechmodcommand(Chapter1,Section1.6andFigure1.9)ortheFileBrowser(Chapter2,Section2.7).Inthefollowingsamplelsdisplay-rw-r—–1pwangfaculty464332018-03-0615:35reportthefour filemode parts (- rw- r–—) show regular file type, read andwrite

permission to u (the file owner), read permission for g (anyone in the facultygroup),andnoaccessforo(allothers).Therearetenpositionsinthefilemode:Position1 filetype:seeTable6.2

Positions2-4

r(read),w(write),andx(execute)permissionfortheowner(u),a-isnopermission;thelettersisusedinsteadofxforanexecutablefilewithaset-useridbitthatison(Section6.4)

Positions r,w,andxpermissionforg,a-isnopermission;thelettersisused

6.4

5-7 insteadofxforanexecutablefilewithaset-groupidbitthatison(Section6.4)

Positions8-10 r,w,andxpermissionforo,a-isnopermission

Asdiscussed inChapter3,Section3.12,youcan set/display thedefault filepermissionswiththeShellcommandumask.

MeaningofPermissionsforaDirectoryThemeaningofread,write,andexecutepermissionsisobviousforaregularfile.Foradirectory, theirmeaningsaredifferent.Toaccessadirectory, theexecutepermission is essential.No execute permission for a directorymeans that youcannotevenperformpwdorcdonthedirectory.Italsomeansthatyouhavenoaccess to any file contained in the file hierarchy rooted at that directory,independent of thepermission settingof that file.The reason is that youneedexecutepermissiononadirectorytoaccessthefilenamesandaddressesstoredinthedirectory.Since a file is locatedby followingdirectorieson thepathname,youneedexecutepermissionsonalldirectoriesonthepathnametolocateafile.Afterlocatingafile,thenthefile’sownaccessmodegovernswhetheraspecificaccessispermitted.Toaccessadirectory,younormallyneedbothreadandexecutepermissions.

Noreadpermissiontoadirectorysimplymeansthatyoucannotreadthecontentof the directory file. Consequently, ls, for example, will fail, and you cannotexamine the filenames contained in the directory. Any filename expansionattemptalsowillfailforthesamereason.However,filesinsuchadirectorystillcanbeaccessedusingexplicitnames.Thewritepermissiontoadirectoryisneededforcreatingordeletingfilesin

thedirectory.Thispermissionisrequiredbecauseafileiscreatedorremovedbyenteringorerasingadirectoryentry.Thus,writepermissiononthefileitselfisnotsufficientfordeletingafile.Infact,youdon’tneedwritepermissiononafiletodeleteitfromthedirectory!Ontheotherhand,ifyouhavewritepermissiononafile,butnowritepermissionforitsdirectory,thenyoucanmodifythefileorevenmakeitintoanemptyfile,butyoucannotdeletethefile.

FILESTATUSForeachfileintheLinuxfilesystem,asetoffilestatus itemsiskept in the i-nodeofthefileandismaintainedbytheoperatingsystem.Thei-nodeofafileisa data structure that records filemeta information (information about the file)

thatisusedbyLinuxtoaccessandmanipulatethefile.FilestatusitemsincludeThecommandls-lfiledisplaysmanystatusitemsofagivenfile.Thesystemcallstat (Chapter11)

canbeusedinaCprogramtoaccessfilestatusinformation.ModernLinuxsystemsimplementext4(evolvedfromext2,ext3)whichuses

extents instead of fixed-size blocks, better supports large-capacity disks, andimprovesperformance.Ext4filesystemsalsosupportextendedattributes(xattr),metadatastoredasname-valuepairs.Suchattributescanbeattachedtofilesbyusers or by the operating system. The name part can belong to different xattrnamespaces. For example, SELinux security contexts (Chapter 8, Section 8.9)are attached to files as extended attributes under the name security.selinux(securityisthenamespace).Thecommandattrgetsandsetsextendedattributesforfiles,inthenamespaceuserbydefault.Youmayalsousecommandsgetfattrandsetfattr(Ex:ex06/getfattr).

FileModeThefilemodeconsistsof16bits.Thefourhighbits(C-FinFigure6.1)of thefilemodespecifythefiletype.Thenextthreebitsdefinethemannerinwhichanexecutable file is run. The lowest nine bits of the filemode specify the read,write,andexecutionpermissionsfortheowner,group,andother.Thefiletypeisfixedwhenafileiscreated.Therunandaccess

Figure6.1FileModeBits

bits are settable by the file owner. You already know how to set the nineaccessbitswiththechmodcommand.Therunbitscanbesettogetherwiththeaccessbitsbythechmodcommandusinganumericalmodesetting,asinchmodmodefileSettableFileModes

Thenumericalmodeisanoctalnumberthatisthelogical-orofanynumberof

the settable filemodes (Table 6.4). For set-id-on-execution, the symbolic u+sandg-smodesarealsoavailable.Onlytheownerofafileorasuperusermaychangethemodeofafile.OnmostLinuxsystems,the-Roptioncauseschmodtoperformtherequestedmodesettingonallfilesunderthegivenfiledirectories.

FileUseridandGroupidInLinux,eachfilehasauseridandagroupid.Thefileuseridistheuseridoftheowner who created the file. Each user may belong to one or more (up to areasonable limit, say, eight) groups of users. Each group has a name. Thepassword file (/etc/passwd) entry of each user contains a group affiliation.Bydefault,anewuserbelongstoagroupwithagroupidthesameastheuserid.Ifauserbelongs tomore thanonegroup, then theadditionalgroupaffiliationsarespecifiedinthefile/etc/group.Thegroupidofafilecanbesettoanygrouptowhichthefileownerbelongs.

The group permissions control access to the file bymembers of the specifiedgroup. When a file is first created, it is given by default the groupid of thedirectorythatcontainsit.Thecommandchgrpgroupidfilename...is used to assign a specified groupid to the named files. For example, if

researchisagroupname,thenchgrpresearch*willchangethegroupidofeachfile inthecurrentdirectorytoresearch.The

useridofafilecanbechangedonlybythesuperuser.Thecommandchownowneridfilename...is used to change the ownership of the named files. For example, the

commandchown-Rpwang.changestheownershipofallfilesinthehierarchy(rootedat.)topwang.Both

chgrpandchowntakethe-Roptiontoprocessfilesandfoldersrecursively.Bashprovidesasetofqueriestodeterminethefiletype,accesspermissions,

and so on of a file (Chapter 5, Section 5.13). In addition, the regular Linuxcommandtestcanbeusedtoobtain informationabout the typeandmodeofafile.The test command is a general conditional command often used in Shellscripts(especiallyinShscripts).

AccessControlEnforcementA file always is accessed through a process, for instance, ls,cat, rm, vim, oryourShell(tocd,forexample).Toenforceaccesscontrol,Linuxusestheuserid

andgroupidofaprocesstograntordenyaccesstoafileaccordingtothefile’saccessmode.Theuserid andgroupid of a process are usually that of the userwho invoked the process.Ausermaybelong tomore than one group; thus, aprocessalsokeepsasupplementarygroupidlist.Specifically, if theuseridof theprocessis thesameas theuseridof thefile,

then the access permissions for u apply. Otherwise, if the groupid of the filematchesagroupidoftheprocess,thenthegpermissionsapply.Otherwise,theosettingsapply(Figure6.2).

Figure6.2AccessPermissionsEnforcement

SetuidandSetgidModesIfthesetuid(setgid)bitisturnedonforanexecutablefile,thenitrunsundertheuserid(groupid)oftheexecutablefileratherthanthatoftheinvokingprocess.The setuid and setgid bits are often used to grant temporarily elevated

privileges for certain tasks involving access to files/programs normallyunaccessibletoregularusers.Consider password changing, for example. The command passwd has its

setuidbitturnedon-rwsr-xr-x.1rootroot35480Jul162018/usr/bin/passwdThus, when you call passwd, it takes on an effective uid of root and can

modifythestoredpassworddatafile(usually/etc/shadow).Settingthesetgidbitforadirectorycausesnewfilesandfolderscreatedunder

ittotakeonthedirectory’sgid,ratherthantheprimarygidofthecreatinguser.This is useful for a shared directory used by members working on the sameproject.Thesetuidbitondirectoriesusuallyhasnoeffect.Thestickybit,usedonoldersystemstomakecertainprogramsloadfaster,is

largelyobsolete.InsomeLinuxsystems,thisbitbecomestherestricteddeletionflagfordirectories.Whenset,itpreventsaunprivilegeduserfromremovingorrenamingafileinthedirectoryunlesstheuseris theownerofthedirectoryorthefile.Inanlslisting,at(T)inthe10thpermissionpositionmeansthesticky

bitisonandxforoison(off).

EstablishingaGroupAs an example application of the file access control facilities, let’s considerestablishingagroupwhosememberscancollaborateonaprojectbyaccessingselectedfilesofoneanother.Toestablishthegroup,youfirstdecideonaname.In this example, the groupid is projectx. Next, you must decide who will bemembersof thegroup.In thisexample, thegroupmembersarepwang,rsmith,jdoe, sldog, and yourself. Now ask your system administrator to create groupprojectx. A system administrator can either edit /etc/group directly or use acommandsuchasgroupaddorsystem-config-userstosetupanewgroupandadd themembers.Assoonas this isdone,projectxexistsonyoursystemasavalid group. Once projectx is established, members can assign desired accesspermissionstoselectedfilestoallowsharingwithinthegroup.Onesimplewayforyoutodothisisasfollows:

1. Establish a directory, alpha, say, under your home directory. All files inalphaaretobesharedwithothersinprojectx.

2. Changethegroupidofalphatoprojectxbychgrpprojectxalpha3. Nowsetthegroupaccesspermissionsforthealphadirectory.Dependingon

theaccessyouwishtogive,useoneofthefollowing:chmodg+rwxsalpha(or simply g+rws) chmod g=rx alpha chmod g=x alpha The differencebetweenthesepermissionsisdescribedearlierinSection6.4.

4. Optionally, use chmod +t alpha to set the restricted deletion flag for thealphafolder.

5. Youmustmake sure that each file in alpha carries the groupid projectx,especiallyfilesestablishedthroughcpormv.Asmentioned,thegroupidofafileisdisplayedwithls-gl.Dependingonthenatureofafile,youshouldassignappropriategrouppermissions.Givethegroupwritepermissiononlyifyouallowothersinprojectxtomodifyafile.

DACandMACThe file accesscontrol scheme,using filemodeanduser/group IDs,describedhere provides users and administrators a mechanism to make Linux systemssecure.Butitisstilluptothepeopleinvolvedtosetthepermissionbitsanduserand group IDs at their own discretion. Thus, the scheme is known asDiscretionaryAccessControl(DAC).LinuxsecuritycanbefurtherenhancedbyaddingMandatoryAccessControl

(MAC)whereaLinuxsystemwouldcomewithitsownsetofsecurityrulesthat

6.5

willbeappliedautomatically.SELinux1 (SecurityEnhancedLinux),developedbytheNSA(USNationalSecurityAgency)andtheLinuxcommunity,isawayofachievingMAC.Modern Linux distributions all have SELinux either built-in or available to

install. SELinux can help strengthen security and be especially important forLinux servers. SELinux rules are applied after the DAC rules and can denyaccessevenifDACallowsit.Ifyouexperienced“accessdenied"butfoundnoDACreasons,inalllikelihood,yourLinuxwasenforcingSELinux.WewilldiscussSELinuxinChapter8,Section8.9.

FILESYSTEMIMPLEMENTATIONAsstatedearlierinthischapter,afilesystemisalogicalorganizationimposedon physical data storage media (usually hard disks) by the operating system.Thisorganization, togetherwith the routinessuppliedby theoperatingsystem,allowsforsystematicstorage,retrieval,andmodificationoffiles.

FilesystemOrganizationTypically for Linux, the entire storage system consists of one or morefilesystems.Eachfilesystemisaself-containedunitconsistingofagroupofdatablocks in a particular storage partition (Chapter 8, Section 8.6).A file can beviewedasaone-dimensionalarrayofbytes.Thesebytesarestoredinanumberofdatablocksfromagivenfilesystem.Moderndiskdrivesoffersizablestoragefordata.Typicaldatablocksizesare

1024,2048,and4096bytes.Afilesystemcangainspeedbyemployingalargerblocksize.Theblocksizeisdeterminedatfilesystemcreationtime.For each filesystem, the addresses (locations) of the data blocks, the status,

andperhapsalsotheattributeinformationofafilearestoredinadatastructureknownasthei-node(indexnode).Allthei-nodesofafilesystemarestoredinalinearlistcalledthei-list(ori-table),whichisstoredataknownaddressonthephysicalstoragemedium.I-nodeandi-listwerementionedinSection6.4.Thei-node(Figure6.3)storesmetainformationforafileincludingfilelength

(inbytes),device,owner,andgroupIDs,filemode,andtimestamps.Thei-nodealsocontainspointers(addresses)tothefile’sdatablocks.Forexample,anext2filesystemallows12directpointers,a single-indirectpointer,adouble-indirectpointer,anda triple-indirectpointer.Andext3andext4filesystemsallowevenlargerindividualfilesandoverallfilesystemsize.

Figure6.3Thei-Node

Adirectpointer is theaddressofablockstoringthecontentdataof thefile.Anindirectpointerpointstoablockofdirectpointers.Adoubleindirectpointerpointstoablockofindirectpointers.Atripleindirectpointerpointstoablockofdouble indirect pointers. With this arrangement, very large files can beaccommodated.The i-node contains all the vital meta information of a file. Therefore, the

implementation of a filesystem centers around access to the i-node. The i-numberinadirectoryentryisusedtoindexthei-listandaccessthei-nodeofthefile.Thus,afilepathnameleads,throughasequenceofi-nodes,tothei-nodeofthefile.Figure6.4showshowthepathname/bin/lsleadsfromtherootdirectory/tothefilelsthroughasequenceofi-nodesanddirectoryentries.

Figure6.4FileAddressMapping

Ahardlinktoafilecanbeseenassimplyanotherdirectoryentrycontainingthesame i-number.Once the i-nodeofa file is located, it is read intoprimarymemoryandkeptontheactivei-nodetableuntilaccesstothefileisclosed.Thei-listalsocontainsfreei-nodesthatareusedtocreatenewfiles.

Thecommandlsobtainsanddisplaysfilestatusinformationfromi-nodes.SeeChapter11,Section11.4fordirectaccesstoi-nodedatafromCprograms.

MountedFilesystemsInLinux,afilesystemreferstothelogicalstoragedevicerepresentedbyasinglei-list.ThecompleteLinuxfilesystemmaycontainoneormorefilesystems.Oneoftheseistherootfilesystem;theothersaremountedfilesystems.Thelocationofthe i-list of the root filesystem is always known to the operating system. Amountedfilesystemisattached(mounted)totherootfilesystematanydirectoryintherootfilesystem.Amountedfilesystemcanberemovedbyunmountingitwiththeumountcommand.Asuperusermayusethecommandmount[-r]devfiledirectorytomount the filesystem stored on the block special filedevfile at the given

directory, which is usually an empty directory created for this purpose. Thisdirectoryiscalledtherootdirectoryofthemountedfilesystem.Iftheoption-risgiven,thefilesystemismountedasread-only.Themountcommandwithoutanyargumentsdisplaysthenamesofallmountedfilesystemsandthepointsonthefile treewhere they aremounted.The commanddf displays file system spaceusageandthefreediskspacesonallthefilesystems.Hereisatypicaldfdisplay.df-h

Filesystem Size Used Avail Use%Mountedon/dev/sda6 140G 18G 115G 14%//dev/sda3 99M 20M 75M 21%/boottmpfs 376M 68K 376M 1%/dev/shm/dev/sda2 146G 32G 115G 22%/media/ACERshowingaLinux/MSWindows®dual-bootcomputerwithfourfilesystems.The/media/ACERisthemountpointofanNTFS(NTFilesystem)fortheMS

Window®side.MostLinuxsystemshavebuilt-insupportforNTFSsofilesandfolders in anNTFS partition are usable from either Linux orMSWindows®.Thiscanbeveryconvenient.Doaman-kntfstoseeLinuxsupportforNTFSonyoursystem.

FilesystemSuperBlockandBlockGroupsALinuxext2(ext3,ext4)filesystemconsistsofanumberofblockgroups.Eachblock group also contains a duplicate copy of crucial filesystem controlinformation(superblockandgroupdescriptors)inadditiontotheblockgroup’s

6.6

6.7

ownblockbitmap,i-nodebitmap,i-list,and,ofcourse,datablocks.Thesuper block defines a filesystem. It records vital information about the

configuration,organization,andoperationsofafilesystem:

ThefilesystemtypeandablockdevicereferenceTheoverallsizeandblocksizeofthefilesystemThelengthofthei-nodelistFreeblocksandfreei-nodesRead,write,andothermethodsfori-nodes

Thegroupdescriptorstoresthelocationoftheblockbitmap,i-nodebitmapandthestartofthei-nodetableforeveryblockgroup;andthese,inturn,arestoredinagroupdescriptor table.Thesuperblockand thegroupdescriptor tablearecriticalforafilesystem,andtheyarestoredatthebeginningofeachblockgrouptoprovideredundancy.

THEFILESYSTEMTABLEEach different filesystem on Linux has its own block-type special file. Thenames of these special files, together with other information for control andmanagementoftheentirefilesystem,arekeptinthefilesystemtable (typically,/etc/fstab).This file contains one line for each filesystem specifying the blockspecial filename, thedirectorynamewheremounted, the filesytem type (local,NFS,2orformemoryswapping),mount/swapoptions,andotherinformation.Ofallthefilesystemscontainedinthefilesystemtable,allorasubsetmaybe

mounted at any given time. The mount table (/etc/mtab) contains a list ofcurrentlymounted filesystems.Themount table ismodifiedby the commandsmountandumount.

CREATINGSPECIALFILESAs previously mentioned, the Linux system uses special files to representphysicalandlogicalI/Odevices,andachievesuniformfileI/OanddeviceI/O.Special files normally are created exclusivelyunder the systemdirectory /dev.Thecommandmknodfilename[borc]majorminorisusedtoestablishaspecialfilebythegivenfilename.Thecharacterbisused

if the device is a block I/O device (hard disk). The character c is used for acharacterI/Odevicesuchasaterminalemulatororaprinter.EachphysicalI/O

6.8

6.9

device on Linux is assigned a major device number according to the type ofdevice it is and aminor device number indicating the unit numberwithin thesametypeofdevices.Thesenumbersareintegers.Forexample,thetwoprinterslp0andlp1crw-rw—-1rootlp6,02018-03-0611:48lp0crw-rw—-1rootlp6,12018-03-0611:48lp1havemajordevicenumber6andminordevicenumbers0and1,respectively.

Onlyasuperusercancreatespecialfiles.

NETWORKFILESYSTEMManyLinuxsystemsallowfileoperationsnotonlyon local filesystemsstoredonthehostcomputer,butalsoonremotefilesystems storedonothercomputersconnected by a network. The Network Filesystem (NFS) allows transparentaccess to remote files. In other words, there is no difference between userrequestsforoperationsonremoteandlocalfiles.NFSbringsmanyadvantagesto file organization for businesses and organizations. For example, duplicatestorageofthesamefilesondifferenthostscanbeavoidedbycentralizingthemonfileservermachinesaccessibleviaNFS.Tomakethingsevenmoreconvenient,NFScanworkwithdifferenthardware

andoperatingsystems.Afilesystemonalocalhostismaderemotelyaccessibleby exporting it. The file /etc/exports specifies local filesystems that can beexportedandanyrestrictionsoneachfilesystem.Thecommandexportfsmustberunaftermodifying/etc/exports.Thefile/var/lib/nfs/etab(orxtab)liststhefilesytemscurrentlybeingexported.

Afilesystemcanbeexportedtoalistofallowedclientsortoallandcanallowread-onlyorread-writeaccess.Aclienthostmakesaremotefilesystemaccessiblebythemountcommandmountremote-filesystemlocal-directorywhichmounts a remote filesystem, specified byhost:directory, onto a local

directoryofchoice.On most Linux systems, even the mounting and unmounting of remote

filesystemsareautomatedthroughtheautofsmechanismassisteddirectlybytheLinux kernel. The kernel calls the automount program to mount a remotefilesystemwhen an actual file access to itsmount point occurs.Automountedfilesystemsaredismountedafteratimeperiodwithnoaccess.

SEARCHINGTHEFILETREE:FIND

WeknowtheLinuxfilesystemisorganizedintoatreestructure.Itissometimesnecessarytosearchapartofthistreeandvisitallnodesinasubtree.Thismeansvisiting all files in a given directory and, recursively, all files contained insubdirectories of the given directory. The find command provides just such atreesearchingmechanism.Thefindcommandvisitsallfilesinasubtreeandselectsfilesbasedongiven

Booleanexpressions.Theselectionfeatureallowsustofindthedesiredfilesandapplyoperations to them.Any file in the subtree forwhich thegivenBooleanexpressionsevaluatetotruewillbeselected.The find command can be used to locate (display the pathname of) files

whosenamesmatchagivenpatterninthesubtree.Forexample,find.-name.c-printInthisexample,thefindcommandisgiventwoBooleanexpressions,-name

.cand-print.Thecommandsearchesthesubtreerootedatthecurrentdirectoryvisiting each file. The file that currently is being visited is referred to as thecurrentfile.Ifthenameofthecurrentfilematchesthepattern*.c(thefilenameendsin.c),thenthenextexpression(-print)isevaluated.The-printexpressionsimplydisplaysthepathnameofthecurrentfileonthestandardoutput.Thus,theeffect of the preceding example is to find allC source files under the currentdirectoryanddisplaytheirpathnames.Thegeneralformofthefindcommandisfindfilename…expression…The command name is followed by one or more filenames, each either an

ordinaryfileoradirectory,andthenbyoneormoreexpressions.Thetreesearchisconductedoneachfileanddirectorygiven.Eachexpressionisapredicateonthecurrentfileandalwaysproducesatrue/falsevalue,althoughtheexpressionalsomay have other effects. An expression is evaluated only if all precedingexpressions are true. In otherwords, expression evaluation for the current fileterminateson thefirst falseexpression,and thesearchprocess thengoeson tothenextfileinthesubtree.The expressions used in find are primary expressions or a Boolean

combination of primary expressions. Some important primary expressions areexplained here. The effect and the Boolean value of each also is described.(Sincesomeexpressionsmayinvolveconceptsandfeatureswehavenotcoveredyet,youmayskipthoseexpressionsfornowifyouwish.)Inthedescriptions,theargumentn is used as a decimal integer that can be specified in one of threeways: an integer, an integer preceded by + , or an integer preceded by - .Specifying+nmeansmorethann,-nmeanslessthann,andnmeansexactlyn.ThefollowingBooleanoperations(inorderofdecreasingprecedence)canbe

6.10

6.11

usedtocombineanyvalidexpressionse1ande2.Here are some additional examples. To remove all files, under your home

directory,namedeithera.outor*.othathavenotbeenaccessedforatleastfourweeks,typein(Ex:ex06/findrm)find(-namea.out-o-name’*.o’)-atime+28-execrm”Youcanavoidthelinecontinuationbyenteringeverythingononecommand

line.Consider another example. To display the names of all files not owned by

smithunderthecurrentdirectory,typeinfind.-usersmith-printNotethatmanycharactersusedintheseexampleshavebeenquotedtoavoid

Shellinterpretation.Now,forathirdexample(Ex:ex06/findstr),supposeyouhaveseveralHTML

files under your personal Web space ($HOME/public_html) that contain thewordLinux,butyouarenotsureexactlywhichfiles.YoucanusefindtoapplyfgreptoeachHTMLfile.findpublic_html-name‘*.html’-execfgrepLinux{}-print

THELOCATECOMMANDWhilefind isniceandpowerful, the locatecommandcanbeeasier touseandfaster.YougivelocateaGlobpatternoraregularexpressionanditcandisplayall pathnames, in the file tree, that contain a node whose namematches. Forexample,

locategnome(pathnamecontaininggnome)locate-b\gnome(base

filenameexactlygnome)locate--regex\.html$(filenameendingin

.html)

The locate command runs faster because it searches a database of files andfolders on your system called an updatedb which is regularly updatedautomaticallydaily.

SAVING,COMPRESSING,ANDDISTRIBUTINGFILES

Sometimes the need arises to pack a number of files into a neat package andsendthemtoanothercomputer.Thetarcommandisusedtocollectasetoffiles

ontoasinglefile,calleda tarfile (thenamecamefrom tapearchive).The tarcommandcopiesentiredirectoryhierarchies.Adirectoryhierarchyreferstoallfilesanddirectoriescontainedinasubtreeof thefile tree.Itworksbypackingmultiplefilesintoasinglefileinthetarformatwhichcanlaterbeunpackedbytarpreservingtheoriginalfileandfolderstructure.Thetarfilecanbesavedasabackup or transferred easily by email or ftp (Chapter 7,Section7.6). The tarcommandisoftenusedtogetherwithcommonfilecompressionschemessuchasgzip(GNUZip),bzip2andxz.Thelattergenerallyprovidesbettercompression.Let’sfirstlookatthesimplestusesoftar.tarcvftarfile.tarname1name2...(A)tarzcvftarfile.tgzname1name2...(B)tarjcvftarfile.tbzname1name2...(C)tarJcvftarfile.txzname1name2...(D)savesthenamedfilehierarchiestothegiventarfilewithnocompression(A),

gzip compression (B), bzip2 compression (C), or xz compression (D). Theoptionsarec(create tarfile),v(verbose), f (tarfilenamefollows),z(usegzip),andj(usebzip2).Thecorrespondingcommandstarxvftarfile.tartarzxvftarfile.tgztarjxvftarfile.tbztarJxvftarfile.txzextract the files contained in tarfile. If you wish to preserve the file

permissionsandotherattributes,usethepoptionwhenpackingandunpackingwith tar.Many software packages in tar format are available for download toyourLinuxsystem.The ZIP utility commonly used onWindows platforms is also available on

Linux.Thezip andunzip commandsmake it convenient to exchange archivefileswithotherplatforms.zip-rarchive.zipname1name2…packsfilesandfoldersintothegivenarchive,whileunzipunpacks.When providing an archive file for downloading, it is good practice to also

provide a finger print file to check the integrity of the download.Creating anMD5(Message-Digestalgorithm5) fingerprintforyourarchivefile issimple.Thecommandmd5sumarchivefile>archivefile.md5placesthenameofthearchivefileanditsMD5fingerprintinthefingerprint

filearchivefile.md5.More secure alternatives tomd5sum use theUSNational SecurityAgency

6.12

published SHA-2 algorithms. On Linux, these include the commandssha224sum,sha256sum,sha384sum,andsha512sumthatproducefingerprintsofdifferentlengths.

PackingFileswithsharThetaristheregularcommandforsavingandretrievingfilesbecauseitrestoresall file attributes such as ownership and access protection modes. The sharcommand isanotherway topackmultiple files intoonewhichdoesnotworryaboutretainingfileattributeinformation,anditcanbeeasiertouse.Basically,sharpacksthefilesintoasinglefileofshcommands.Thepacked

fileisunpackedbylettingshprocessthefile.Thecommandsharfile1file2...>outfile.shpacks the named files (including directories) into one file and sends that to

standardoutput.Theresultingoutfile.shfilecanbesentbyemailoruploadedtoanotherLinux/UNIXcomputer.Tounpacksimplydosh<outfile.sh

FILESHARINGWITHSAMBASambaisasuiteofprogramsforLinux/UnixtosharefilesandprinterswithMSWindows®systems.MostLinuxdistributionscomewithSamba.

Figure6.5FileSharingwithSAMBA

To access shared files simply go to Network > SAMBA or Network >Workgroupinyourfilebrowserorusethesecommands(Figure6.5):nautilussmb:///dolphinsmb:///This is also an easyway to access a shared disk attached viaUSB to your

6.13

6.14

6.15

homerouter.

MOREFILE-RELATEDCOMMANDSSomeadditional commands that areuseful indealingwith files andmanagingthefilesystemarelistedhere.Thefunctionofeachcommandisindicated,butnofullexplanationsaregiven.Formoredetailedinformationandoptionsonthesecommands,refertotherespectivemanualpages.

basenameremovesprefixesandsuffixesfromafilename.chrootchangestemporarilytherootdirectory(/)fortestingaprogram.cmpcomparestwofilestoseeiftheyareidentical.commselectsorrejectslinescommontotwosortedfiles.dfdisplaysdiskspacefreeonallfilesystems.diffcomparestwofilesordirectoriesandoutputsthedifferences.dudisplaysallfilesizesinkilobytesinadirectoryhierarchy.sizedisplaysthesizeofanobjectfile.splitsplitsafileintopieces.touch updates the lastmodified time of a file; if a file does not exist, itcreatesanemptyone.uniqreportsrepeatedlinesinafile.wccountsthenumberofwords,linesingivenfiles.

FORMOREINFORMATIONFortheFileSystemStandard(FSSTD),seetheLinuxJournalarticlebyDanielQuinlanavailableontheWebfromACM:portal.acm.org/citation.cfm?id=324517For complete information on the Linux file hierarchy, see the Linux

DocumentationProjectonlinearticle:tldp.org/LDP/Linux-Filesystem-Hierarchy/htmlFormoredetailsonfilesysteminternalsandimplementations,refertoDesign

and Implementation of the Second Extended Filesystem and to Linux NFS-HOWTOatSourceForgeSourceForge.net.See also Linux Filesystems Explained – EXT2/3/4, XFS, Btrfs, ZFS at

fossbytes.comr.

SUMMARY

6.16

ThefilesystemiscentraltoanyoperatingsystemandispartoftheLinuxkernel.TheLinuxfilesystemhierarchycontainsfilesanddirectoriesarrangedinatreestructure that grows down from the root directory /. The Linux file hierarchylargelyfollowstheFSSTND.Different file types are directories, special files, links, regular files, sockets,

andnamedpipes.Therearetwokindsoflinks:hardlinksandsymboliclinks.Asymbolic linkcan link toadirectoryandcan span filesystems.Access to filesanddirectoriesisgovernedbyrwxpermissionsforthefileowner(u),forusersinthefilegroup(g),andforothers(o).Theset-useridbitforexecutablefilesisanimportantconcept.Whenaprocess

executesaset-useridfile,itseffectiveuseridbecomesthatofthefileowner.The entire file system consists of a root filesystem and possibly additional

mountable filesystems. Linux supports different filesystem implementations,includingext2anditsextensions(currentlyext4)thatsupportextendedattributes(xattr)intheformofname-valuepairs.Each filesystem is organized by an i-list, which is a list of i-nodes that

contains status and address information for each file and all free space in thefilesystem. File status information includes userid, access groupid, mode,timestamps, and disk addresses. Part of the file mode specifies file accesspermissions.TheseattributesareusedinenforcingDiscretionaryAccessControl(DAC). SELinux (Chapter 8) can further strengthen system security byprovidingMandatoryAccessControl(MAC).The NFS allows transparent access to remote (NFS) and local filesystems,

makingiteasytosharefilesacrossanetwork.SambamakesitpossibletosharefileswithWindowssystemsonyournetwork.Todoasystematicsearchthroughafilehierarchy,usethefindcommand.To

quicklylocatefiles/foldersbasedontheirnames,usethelocatecommand.Usethesimplesharcommandorthemoreefficienttarcommand(withgzip,bzip2,orxz filecompression) topackandcompressmultiplefiles intoanarchiveforeasy transport. Use zip to manage archive files across different computersystems.

EXERCISES1. Trytheumaskcommand.Whatdoesittellyouaboutthefilesyoucreate?

Try setting the umask value and then creating some files. Look at theirprotectionbits.

2. Ifyouhavenotdoneityet,downloadthemostrecentHTMLversionofthe

Linux man pages from www.tldp.org/manpages/man-html/ to yourcomputer.UnpackitsothatyoucanuseitwithyourWebbrowser.

3. The term filesystem is different from the phrase “file system.” Can youclearlyspecifytheirmeaning?

4. Why is a hard link indistinguishable from the original file itself? Whathappensifyourmahardlink?Whyisitnotpossibletohaveahardlinktoafileinadifferentfilesystem?

5. Clearly state the meaning of the rwx permissions for a directory. Whatwould happen if you perform ls dir with read permission to dir but noexecutepermission?Why?

6. Write a Shell script forwebwhich takes the name of a folder fname andmakesallfileso+randallfolderso+rxinthefilehierarchyrootedatfname.

7. What is an xattr? Forwhat purpose?Where and inwhat form are xattrsstored?

8. What command is used for a user to get, set, list, and remove extendedattributes?Pleaseshowexamples.

9. Whatcommanddisplaysthei-numberofafile/directory?10. Itisclearhowcommandsrmandlsworkonordinaryfiles.Describehow

theyworkonsymboliclinks.Mustasymboliclinkpointtoanexistingfile?Whathappensiftheactualfileofasymboliclinkisdeleted?Isitpossibleforasymboliclinktopointtoanothersymboliclink?

11. Consider the . and .. special files. Is it correct to say that these files aresystem-createdhardlinkstodirectories?

12. ConsidertheBashscriptclean.sh(Chapter5,Section5.20).Doesthescriptstillworkcorrectlyiftherearesymboliclinksinthedirectoryitistryingtoclean?Ifthereisaproblem,howwouldyoufixit?

13. Trytormafiletowhichyouhavenowritepermission.Whatmessagedoesrmgive?Howdidyourespond?Wereyouabletodeletethefile?Why?

14. Whenanexecutable file is invoked,does thenewprocessalwaysassumetheuseridoftheuserwhoinvokedit?Explain.

15. You are looking for a file somewhere under your home directory thatcontainsthestringzipcodeinit.Describehowyoucanlocatethefileifyoudo/don’tknowwhichdirectorycontainsthefile.Whatifthefilemaybeahiddenfilewhosenamebeginswithadot?

16. Howexactlydoesonecreatea.tgzfile?Howdoesoneextractfroma.tgzfile?Whatabout.tbzandtxzfiles?

17. FindouthowtosetupaSamba-sharedfolderinyourLinuxhomedirectory.Explaineachstep.

12

PronouncedSELinux.SeeSection6.8.

Chapter7

Networking,Internet,andtheWeb

Early packet-switched computer networking, involving a few researchinstitutionsandgovernmentagencies,startedinthelate1960sandearly1970s.Today, it ishard to tellwhere the computer ends and thenetworkbegins.Theview “The Network is the Computer” is more valid than ever. Most peoplecannottolerateevenafewminutesofInternetconnectionoutage.A computer network is a high-speed communications medium connecting

many, possibly dissimilar, computers orhosts. A network is a combination ofcomputer and telecommunication hardware and software. The purpose is toprovidefastandreliableinformationexchangeamongthehosts.Typicalservicesmadepossiblebyanetworkinclude

ElectronicmailOn-linechattingandInternetphonecallsFiletransferRemoteloginDistributeddatabasesNetworkedfilesystemsAudioandvideostreamingVoiceandtelephoneoveranetworkWorldWideWeb,E-business,E-commerce,andsocialnetworksRemoteprocedureandobjectaccess

In addition to host computers, the network itself may involve dedicatedcomputersthatperformnetworkfunctions:hubs,switches,bridges,routers,andgateways. A network extends greatly the powers of the connected hosts. Anddedicatedserverhosts(Chapter9)greatlyenhancethepowerandusefulnessofthenetworkbyprovidingefficientandeffectiveservices.Agoodunderstandingofbasicnetworkingconcepts,commands,information

7.1

security, and how the Web works will be important for any Linux user,programmerandservermanager.

NETWORKINGPROTOCOLSFor programs and computers fromdifferent vendors, under different operatingsystems, tocommunicateonanetwork,adetailedsetofrulesandconventionsmustbeestablishedforallpartiestofollow.Suchrulesareknownasnetworkingprotocols.Weusedifferentnetworkingservicesfordifferentpurposes;therefore,eachnetworkservicefollowsitsownspecificprotocols.Protocolsgovernsuchdetailsas

AddressformatofhostsandprocessesDataformatMannerofdatatransmissionSequencingandaddressingofmessagesInitiatingandterminatingconnectionsEstablishingservicesAccessingservicesDataintegrity,privacy,andsecurity

Thus, for a process on one host to communicate with another process on adifferenthost,bothprocessesmustfollowthesameprotocol.TheOpenSystemInterconnect (OSI)ReferenceModel (Figure 7.1) provides a standard layeredview of networking protocols and their interdependence. The correspondinglayers on different hosts, and inside the network infrastructure, performcomplementary tasks to make the connection between the communicatingprocesses(P1andP2inFigure7.1).

Figure7.1NetworkingLayers

7.2

Amongcommonnetworkingprotocols,theInternetProtocolSuiteisthemostwidelyused.ThebasicIP(InternetProtocol ) isanetwork layerprotocol.TheTCP(TransportControlProtocol )andUDP(UserDatagramProtocol )areatthetransportlayer.TheWebisaservicethatusesanapplicationlayerprotocolknownasHTTP(theHypertextTransferProtocol).Networkingprotocolsarenomystery.Thinkabouttheprotocolformakinga

telephonecall.You(aclientprocess)mustpickupthephone,listenforthedialtone, dial a valid telephone number, and wait for the other side (the serverprocess)topickupthephone.Thenyoumustsay“hello,”identifyyourself,andsoon.Thisisaprotocolfromwhichyoucannotdeviateifyouwantthecalltobemade successfully through the telephone network, and it is clear why such aprotocolisneeded.Thesameistrueofacomputerprogramattemptingtotalktoanothercomputerprogramthroughacomputernetwork.Thedesignofefficientandeffectivenetworkingprotocolsfordifferentnetworkservicesisanimportantareaincomputerscience.ChancesareyourLinuxsystemisonaLocalAreaNetwork(LAN)whichis

connected to the Internet. This means you have the ability to reach, almostinstantaneously, across great distances to obtain information, exchangemessages,upload/downloadfiles,interactwithothers,doliteraturesearches,andmuch more without leaving the seat in front of your workstation. If yourcomputer is not directly connected to a network but has a telephone or cablemodem, then you can reach the Internet through an Internet service provider(ISP).

THEINTERNETThe Internet is a global network that connects computer networks using theInternet Protocol (IP). The linking of computer networks is calledinternetworking, hence the name Internet. The Internet links all kinds oforganizations around theworld: universities, government offices, corporations,libraries, supercomputer centers, research labs, and individual homes. ThenumberofconnectionsontheInternetislargeandgrowingrapidly.The Internet evolved from theARPANET, 1 a U.S. Department of Defense

Advanced Research Projects Agency (DARPA) sponsored network thatdeveloped the IP as well as the higher level Transmission Control Protocol(TCP) and User Datagram Protocol (UDP) networking protocols. Thearchitecture and protocol were designed to support a reliable and flexiblenetworkthatcouldendurewartimeattacks.

The transition ofARPANET to the Internet took place in the late 1980s asNSFnet, the U.S. National Science Foundation’s network of universities andsupercomputing centers, helped create an explosive number of IP-based localandregionalnetworksandconnections.TheInternetissodominantnowthatithasvirtuallyeliminatedallhistoricalrivalssuchasBITNETandDECnet.The Internet Corporation for Assigned Names and Numbers (ICANN;

www.icann.org) is a nonprofit organization responsible for IP address spaceallocation, protocol parameter assignment, domain name systemmanagement,andmaintainingrootserversystemfunctions.

NetworkAddressesAnaddresstoahostcomputerislikeaphonenumbertoatelephone.Everyhoston the Internet has its own network address that identifies the host forcommunication purposes. The addressing technique is an important part of anetworkanditsprotocol.AnInternetaddress(IPv4address)isrepresentedby4bytes in a 32-bit quantity. For example, tiger, a host atKent State, has the IPaddress131.123.41.83(Figure7.2).Thisdotnotation (orquad notation) givesthedecimalvalue(0to255)ofeachbyte.Toaccommodatetheexplosivegrowthofthenumberofconnecteddevices,theInternethasbeenmovingtoIPv6,whichsupports128-bit addresses.The IPaddress is similar toa telephonenumber inanotherway:theleadingdigitsarelikeareacodes,andthetrailingdigitsarelikelocalnumbers.

Figure7.2IPv4Address

Becauseof their numerical nature, thedotnotation is easyonmachinesbuthardonusers.Therefore,eachhostmayalsohaveadomainnamecomposedofwords, rather like apostal address.For example, thedomainname for tiger istiger.zodiac.cs.kent.edu (at the Department of Computer Science, Kent StateUniversity).TheLinuxcommandhostdisplaystheIPanddomainnameofanygivenhost.Forexample,hosttiger.zodiac.cs.kent.edudisplaystiger.zodiac.cs.kent.eduhasaddress131.123.41.83With domain names, the entire Internet name space for hosts is recursively

dividedintodisjointdomainsinahierarchicaltree(Figure7.3).Theaddressfortigerputsitinthecslocaldomain,withinthezodiacsubdomain,thenwithinthekent subdomain, which is under the edu top-level domain (TLD) for U.S.educational institutions.OtherTLDs includeorg(nonprofitorganizations),gov(U.S. government offices), mil (U.S. military installations), com (commercialoutfits),net(networkserviceproviders),uk(UnitedKingdom),cn(China),andso forth. Within a local domain (for example, cs.kent.edu), you can refer tomachinesbytheirhostnamealone(forexample,monkey,dragon,tiger),butthefulladdressmustbeusedformachinesoutside.FurtherinformationonInternetdomainnamescanbefoundinSection7.19.

Figure7.3TheDomainNameHierarchy

TheICANNaccreditsdomainnameregistrars,whichregisterdomainnamesforclientsso theystaydistinct.AllnetworkapplicationsacceptahostaddressgiveneitherasadomainnameorasanIPaddress.Infact,adomainnameisfirsttranslatedtoanumericalIPaddressbeforebeingused.

PacketSwitchingData on the Internet are sent and received in packets. A packet envelopstransmitted data with address information so the data can be routed throughintermediatecomputersonthenetwork.Becausetherearemultipleroutesfromthe source to thedestinationhost, the Internet is very reliable andcanoperateevenifpartsofthenetworkaredown.

ClientandServerMost commonly, a network application involves a server and a client (Figure7.4).

7.3

Aserverprocessprovidesaspecificserviceonahostmachine thatofferssuch a service. Example services are email (SMTP), secure remote hostaccess (SSH), secure file transfer (SFTP), and the World Wide Web(HTTP). Each Internet standard service has its own unique port numberthat is identical on all hosts. The port number together with the Internetaddress of a host identifies a particular server program (Figure 7.4)anywhereon thenetwork.For example,SFTPhasportnumber115,SSHhas22,andHTTPhas80.OnyourLinuxsystem,thefile/etc/servicesliststhestandardandadditionalnetworkservices,indicatingtheirprotocolsandportnumbers.Aclientprocessonahostconnectswithaserveronanotherhosttoobtainits service.Thus,aclientprogramis theagent throughwhichaparticularnetworkservicecanbeobtained.Differentagentsareusually required fordifferentservices.

AWebbrowsersuchasFirefoxisanHTTPclient.ItrunsonyourcomputertoaccessWebserversonanyInternethosts.TheLinuxwgetcommandisanotheruseful client that can download files from the Internet using theHTTP or theFTPprotocol.

Figure7.4ClientandServer

THEDOMAINNAMESYSTEMAsstatedinSection7.2,everyhostontheInternethasauniqueIPaddressandadomainname.Thenetworknamespace, thesetofalldomainnameswiththeirassociatedIPaddresses,changesdynamicallywithtimeduetotheadditionanddeletionofhosts,regroupingoflocalworkgroups,reconfigurationofsubpartsofthe network, maintenance of systems and networks, and so on. Thus, newdomain names, new IP addresses, and new domain-to-IP associations can beintroduced in thename space at any timewithout central control.Thedomainname system (DNS) is a network service that supports dynamic update andretrievalofinformationcontainedinthedistributednamespace(Figure7.5).Anetworkclientprogram(forexample,theFirefoxbrowser)willnormallyusethe

7.4

DNS to obtain IP address information for a target host beforemaking contactwith a server. The dynamic DNS also supplies a general mechanism forretrievingmanykindsofinformationabouthostsandindividualusers.

Figure7.5DomaintoIP

HerearepointstonoteabouttheDNSnamespace:

TheDNSorganizestheentireInternetnamespaceintoabigtreestructure.Each node of the tree represents a domain and has a label and a list ofresources.Labelsarecharacterstrings(currentlynotcasesensitive),andsiblinglabelsmust be distinct. The root is labeled by the empty string. ImmediatelybelowtherootaretheTLDs:edu,com,gov,net,org,info,andsoon.TLDsalso include country names such as at (Austria), ca (Canada), and cn(China).Underedu,forexample,therearesubdomainsberkeley,kent,mit,uiuc,andsoon(Figure7.3).Afulldomainnameofanodeisadot-separatedlistoflabelsleadingfromthenodetotheroot(forexample,cs.kent.edu.).Arelativedomainnameisaprefixofafulldomainname,indicatinganoderelativetoadomainoforigin.Thus,cs.kent.eduisactuallyanamerelativetotheroot.A label is the formal or canonical name of a domain.Alternative names,calledaliases,arealsoallowed.Forexample,themainWebserverhostinfohas the aliaswww, so it is also known aswww.cs.kent.edu.Tomove theWebservertoadifferenthost,alocalsystemmanagerreassignsthealiastoanotherhost.

SeeSection7.19formoreinformationontheDNSandnameservers.

NETWORKINGINNAUTILUSWefirstintroducedtheGNOMENautilusfilemanagerinChapter2,Section2.7.LaunchNautilusandgoto+OtherLocations(Figure7.6);youcanbringupalistofallsystemsonyourlocalandremotenetworkandaccessfilesonthem.

Figure7.6NetworkinginNautilus

Linux systems are listed individually. Systems running other operatingsystemsaregroupedunderdifferent iconssuchas theWindowsNetwork icon.Ofcourse,youcanbrowseonlymachineswithpermission.Normally,loginwillberequiredunlessyouhavearrangedano-passwordlogin(seeSection7.6).Youcanalsoconnecttonewserverstowhichyouhaveaccess.Herearesome

sampleconnections:

sftp://pwang@tiger.zodiac.cs.kent.edu—Secure FTP, home directory ofpwangontiger.zodiac.cs.kent.edussh://pwang@tiger.zodiac.cs.kent.edu—Secureshell,sameasabovesftp://pwang@tiger.zodiac.cs.kent.edu/Pictures—Secure FTP, Picturesfolderofpwang(Figure7.7)ftp://pwang@tiger.zodiac.cs.kent.edu—RegularFTP

Figure7.7SFTPviaNautilus

AccessingSambaSharedFilesUsually, you’ll find Linux andMSWindows® systems on the same in-housenetwork.NautilusmakesiteasytoaccesssharedfilesfromMSWindows®.JustentertheLocation

7.5

smb://host/share_folderto reach the target shared folder via the Common Internet File System

protocol, the successor of Server Message Block (SMB). Linux systems useSaMBa,afree,open-sourceimplementationoftheCIFSfilesharingprotocol,toactasserverandclienttoMSWindows®systems.UseanIPforthehosttobesure.HerearesomeLocationexamplesonahomenetwork.

smb://192.168.2.102/SharedDocssmb://192.168.2.107/Public

NETWORKINGCOMMANDSLinux offers many networking commands. Some common ones are describedhere to get you started. In earlier chapters, we mentioned briefly severalnetworkingcommands.Forexample,weknowthathostnamedisplays the domain name of the computer you are using. If given an

argument, thiscommandcanalsoset thedomainname(whenrunasroot),butthedomainnameisusuallyonlysetatsystemboottime.TogettheIPaddressandotherkeyinformationfromtheDNSaboutyourcomputeroranotherhost,youcanuse

host$(hostname)(foryourcomputer)hosttargetHost(fortarget

host)

Forexample,hostgoogle.comproduces

google.comhasaddress74.125.45.100google.comhasaddress

74.125.67.100google.comhasaddress209.85.171.100google.commail

ishandledby10smtp4.google.com.google.commailishandledby10

smtp1.google.com.google.commailishandledby10

smtp2.google.com.google.commailishandledby10smtp3.google.com.

Foranygivenhost,itsDNSdataprovideIPaddress,canonicaldomainname,alias domain names, DNS server hosts, and email handling hosts. Othercommands that helpyouaccess theDNSdata from the command line includenslookupanddig(DNSInformationGroper).Forexample,digtiger.zodiac.cs.kent.edugives

;>><<DiG9.10.5-P2-RedHat-9.10.5-2.P2.fc25>><<

tiger.zodiac.cs.kent.edu;;globaloptions:+cmd;;Gotanswer:;;-

<<HEADER>>-opcode:QUERY,status:NOERROR,id:60868;;flags:qr

rdra;QUERY:1,ANSWER:1,AUTHORITY:0,ADDITIONAL:1;;OPT

7.6

PSEUDOSECTION:;EDNS:version:0,flags:;udp:8192;;QUESTION

SECTION:;tiger.zodiac.cs.kent.edu.INA;;ANSWER

SECTION:tiger.zodiac.cs.kent.edu.43200INA131.123.41.89;;Query

time:41msec;;SERVER:209.18.47.62#53(209.18.47.62);;WHEN:Wed

Sep2011:14:17EDT2017;;MSGSIZErcvd:69

Thedesiredinformation(ANSWERsection)togetherwiththeidentityofthenameserver(SERVER)thatprovidedthedataisdisplayed.The command dig is very handy for verifying the existence of hosts and

findingtheIPaddressordomainnamealiasesofhosts.Oncethenameofahostisknown,youcanalsotestifthehostisupandrunning,asfarasnetworkingisconcerned,withthepingcommand.pinghostThissendsamessagetothegivenremotehostrequestingittorespondwithan

echoifitisaliveandwell.Toseeifanyremotehostisupandrunning,youcanuseping,whichsendsan

echoInternetcontrolmessagetotheremotehost.Iftheechocomesback,you’llknowthatthehostisupandconnectedtotheInternet.You’llalsogetround-triptimesandpacketlossstatistics.Whensuccessful,thepingcommandscontinuestosendechopackets.TypeCTRL+Ctoquit.

SSHWITHX11FORWARDINGNetworking allows you to conveniently access Linux systems remotely.MostLinuxdistributionscomewithOpenSSHinstalled.Asmentioned inChapter1,Section1.2,youcanssh toa remoteLinuxanduse it fromthecommand line.Furthermore,youcanssh-Xuserid@remoteHostnameto log in to the given remote host with X11 forwarding/tunneling, which

allowsyouto

Figure7.8DigitalSignature

startanyXapplications,suchasgeditorgnome-terminal,ontheremotehostandhavethegraphicaldisplayappearonyourlocaldesktop.This works if your local host is a Linux/UNIX/MacOS system. It can also

workfromMSWindows®.Followthesesteps:

1. Obtain and install anX11 server onWindows, such as theXming or theheavierdutyCygwin.

2. AssumingyouhavedownloadedandinstalledXming,clicktheXmingiconto launch theX11 server. TheX11 server displays an icon on your startpanelsoyouknowitisrunning.

3. SetupSSHorPuttyonyourMSWindows®system:

Putty Settings—Go to Connection- > SSH- > X11 and check theEnable X11 forwarding box. Also set X display location to127.0.0.1:0.0.SSHSettings—ChecktheTunneling->TunnelX11Connectionsbox.AlsochecktheAuthentication->EnableSSH2connectionsbox.

4. UseeitherPuttyorSSHtoconnecttoaremoteLinux/Unixcomputer.Makesure your remote account login script, such as .bash_profile, does not settheDISPLAYenvironmentvariable.Itwillbesetforyoutosomethinglikelocalhost:10.0automaticallywhenyouconnectviaSSH.

5. MakesureyourX11server(Xming,forexample)isrunning.Now,ifyoustart an X application on the remote Linux system, that graphicalapplicationwillthenSSHtunneltoyourcomputerandusetheX11serveronyourcomputertodisplayagraphicaluserinterface(GUI).Forexample,youcanstartgedit,nautilus–no-desktop,orevenfirefox.

Note,usinganapplicationwitharemoteGUIcanbeslowduetomuchheaviernetworkingloadascomparedaremoteCLI.

NoPasswordssh,sftp,andscpThe commands ssh,sftp, and scp are for remote login, secure ftp, and secureremotecp,respectively.Whenusinganyoftheseyouusuallyneedtoenterthepassword for the remote system interactively.Whenyouneed toperformsuchtasksfrequently,thiscanbeabother.Fortunately,youcaneasilyavoidhavingtoenterthepassword.Justfollowthesesteps.MostLinuxsystemscomewithOpenSSHinstalled.Thismeansyoualready

have the SSH suite of commands.These enable you to securely communicatefromonecomputer(asuser1onhost1)toanother(asuser2onhost2).Wewillassumeyouareloggedinasuser1onhost1(thisisyourlocalhost),andyouwishtoarrangesecurecommunicationwithyouraccountuser2onhost2,whichwewillrefertoastheremotehost.SSHcanusepublic-keyencryption fordata securityanduserauthentication

(Section 7.9). If you have not done it yet, the first step in arranging forpassword-lessloginistogenerateyourownSSHkeys.Issuethecommandssh-keygenYou’llbeaskedforafoldertosavethekeysandapassphrasetoaccessthem.

In this case, don’t provide any input in response to these questions from ssh-keygen.SimplypresstheENTERkeyinresponsetoeachquestion.Keygenerationtakesalittletime.Thenyou’llseeamessagetellingyouthat

your identity (private key) is id_rsa and your public key is id_rsa.pub savedunderthestandardfolder.ssh.inyourhomedirectory.The second step is to copy your id_rsa.pub to your account on the desired

remote-host.Issuethecommandssh-copy-id-i/.ssh/id_rsa.pubyour_userid@remote-hosttoappendyourpublicSSHkeytothefileuserid/.ssh/authorized_keyonthe

remote-host.Now you are all set. You can log in to remote-host without entering a

password.sshuserid@remote-hostThesamesetupavoidsapasswordwhenyouusesftporscp.

RemoteFileSynchronizationThersynccommandmakesiteasytokeepfilesinsyncbetweentwohosts.Itisvery efficient because it uses a remote-update protocol to transfer just thedifferencesbetweentwosetsoffilesacrossthenetworkconnection.Noupdatingisperformedforfileswithnodifference.Withthecommands

rsync-azuserid@host:sourcedestDir(remotetolocalsync)rsync

-azsourceuserid@host:destDir(localtoremotesync)

thegivensource file/folder isused toupdate the sameunder thedestinationfolderdestDir.Whensource is a folder,givenwithout the trailing /, theentirehierarchyrootedatthefolderwillbeupdated.Usetheformsource/tosyncallfilesinsidethesourcefoldertothedestinationfolder.The -az option indicates the commonly used archivemode to preserve file

7.7

types andmodes andgzip (Chapter 6,Section6.11) data compression to savenetworking bandwidth. The rsync tool normally uses ssh (Section 7.6) forsecure data transfer and does not require a password if you have set uppassword-lessSSHbetweenthetwohosts(Section7.6).Forexample,eitheroneofthesetwocommandsrsync-azpwang@tiger.zodiac.cs.kent.edu:/linux_book/projectsrsync-azpwang@tiger.zodiac.cs.kent.edu:/linux_book//projects/linux_bookupdates the local folder /projects/linux_book based on the remote folder

/linux_bookbylogginginaspwangontheremotehosttiger.zodiac.cs.kent.edu.Seethersyncmanpageforcompletedocumentation.

CRYPTOGRAPHYCryptosystems keep communication safe by encryption, a technique inventedlong before the Internet or digital computers. The concept is simple, theplaintext, the original message, is encrypted into ciphertext beforecommunication.Onlythereceiverknowshowtodecrypttheciphertextbackintoplaintext.For example, rot13 (Figure 7.8) is a simple letter substitution cipherwhere

each letter in the plaintext is replaced by a letter 13 places after it, assumingthereareonly26lettersandthelastletterisfollowedbythefirstletterinacycle.Arot13ciphertextcanbedecryptedbyapplyingtheencryptiononitagain.

Figure7.9ARot13Wheel

Asanotherexample,astationery-paper-sizetemplatewithholescutinitcanbeusedtosendsecretcorrespondences.Asenderwouldwritetheplaintextontocommonstationerypaper through theholesof the template.Thesenderwouldthencomposeaninnocentsoundingletterwithwordsoftheplaintextembeddedinthemanyotherwordsoftheletter.Thereceiverwoulduseacopyofthesametemplatetoeasilyrecovertheplaintext.Or, senders and receivers can agree on a book to use. Ciphertext would

containpagenumber, linenumber andwordnumber to identify anyparticular

7.8

wordfromthebook.Onlypeoplewhoknowwhichbookandwhatthenumbersmeancandecryptthemessage.Further,oneofthemanynumbersmayindicateaparticularbookamongseveralpossibleonestouse.Before and duringWorldWar II, the Germans made heavy use of various

electromechanicalrotorciphermachinesknownasEnigma.These are examples of symmetric cryptosystems (Section 7.8) that use the

samekeytoencipheranddeciphermessages.Communicatingpartiesmustknowthekeybeforehand.Andthekeymustbekeptsecrettoothers.Obviously,rot13,the paper template, the book plus numbering scheme, the Enigma machinesettingsarethekeysintheaboveexamples.Public-keycryptosystems,however,areasymmetricandusenotonebutapair

of keys—one to encrypt and the other to decrypt. The decryption key is keptsecret,whiletheencryptionkeycanbesharedopenly(Section7.9).

SYMMETRICCRYPTOSYSTEMSModernelectronicsymmetricencryptionsystems(Figure7.9)need toworkondigitaldata.Most,ifnotall,ofthemuseanencryption/decryptionalgorithmthatisopenandakeythatiskeptsecret.

Encryption/decryption algorithm: The algorithm performs varioussubstitutionsandpermutationsonchunks,typically128-or256-bitblocks,oftheplaintextorciphertext.Secret key: The plaintext (ciphertext) and the secret key are input to theencryption (decryption) algorithm. The exact transformations performeddependonthekeyused.Thealgorithmproducesdifferentoutputdependingon the key given. Using the same key on the ciphertext, the decryptionalgorithmproducestheoriginalplaintext.

Figure7.10SymmetricCryptosystems

Asymmetriccryptosystemusuallyhasthesetwocharacteristics:

1. Openalgorithm:Theencryption/decryptionalgorithmcanbedescribed intheopen.Itisimpracticaltodecodeanyciphertextknowingthealgorithmandnotthesecretkey.

2. Secretkey:Sendersand receiversmusthaveobtained thekey securely inadvanceandmustallkeepthekeysecret.

The secret key is usually a bit pattern of sufficient length. The quality of thesecretkeyisimportant.Itshouldberandomlygeneratedand256-bitorlongertomakebrute-forceattacks,tryingallpossiblekeys,impractical.Whenapassword(orpassphrase)isusedasakey,itisusuallyputthrougha

key derivation function, which compresses or expands it to the key lengthdesired.Often,arandomlygeneratedpieceofdata,calledasalt,isalsoaddedtothepasswordorpassphrasebeforetransformingittotheactualkey.TheAdvancedEncryption Standard (AES) is a symmetric cryptosystem for

electronic data established by the US National Institute of Standards andTechnology(NIST)in2001.AEShasbeenadoptedbytheUSgovernmentandis now used worldwide. It supersedes the Data Encryption Standard (DES),whichwaspublishedin1977.Thereareothersymmetricciphers,suchasRC4andBlowfish,butAES-256seemstobethebest.Let’s take a closer look atAES-256,whichuses a 256-bit key and encodes

data by encrypting one 256-bit block at a time. The following is an over-simplifiedviewofhowitworks:

1. Arrangesthedatablocktobeencoded/decodedintoa4by4arrayofbytes2. Generatesroundkeysusingthegivenkey3. Transformseachbytebybitwisexorwitharoundkey4. Scramblesandtransformsthe4by4array,inmultiplerounds,byshifting

rows,mixingcolumns,andsubstitutingbytesfromalook-uptablederivedfromthecurrentroundkey

7.9

Figure7.11SampleAESEncryption

Figure7.10showsaplaintextinvolvingthetitleofthisbookandtheAES-256producedciphertextusingthekey“Modern-is-key.”Theencryptedbinaryresult,asequenceofbytes,isdisplayedasastringofcharactersusingbase64encoding.Base64encodingiswidelyusedtoencodeemailattachments.SixtyfourASCIIcharacters are used to represent each 6-bit piece of the binary data tomake ittextual,fordisplay,printing,oremail.

PUBLIC-KEYCRYPTOGRAPHYANDDIGITALSIGNATURE

Security is a big concern when it comes to networking. From the user’sviewpoint,itisimportanttokeepdataandnetworktransportsecureandprivate.Public-key cryptography is an essential part of the modern network securityinfrastructuretoprovideprivacyandsecurityformanynetworkingapplications.Beforetheinventionofpublic-keycryptography,thesamesecretkeyhadtobeused for both encryption and decryption of a message (symmetric-keycryptography).Symmetric-key is fineandefficient,and remains inwidespreaduse today.However, a secret key is hard to arrange among strangers never incommunication before; for example, parties on the Internet. The public-keycryptographybreakthroughsolvesthiskeydistributionproblemelegantly.Inacriticaldeparturefromsymmetriccryptographywhichusesthesamekey

for encryption and decryption, public-key cryptography uses a pair ofmathematicallyrelatedkeys—onekey,thepublickey,toencryptandtheother,theprivatekey,todecrypt.Thepairofkeysareintegerssatisfyingwell-definedmathematicalpropertiesandusuallyproducedbyakeygenerationprogram.Foreachpublickey,thereisonlyonecorrespondingprivatekey,andviceversa.Thepublickeyismadeavailableforanyonewhowishestosendanencrypted

7.10

messagetoarecipientwhousestheprivatekeytodecryptthemessage.Thepublickeyusuallybecomespartofadigitalcertificate,whichverifiably

associates thekey to itsowner.Or, theownerof thekeypairmaypublish thepublic key in online key repositories open to the public. Thus, anyone whowishes to send secure messages will use the public key for encryption. Theownerthenusestheprivatekeytodecrypt(Figure7.11).

Figure7.12Public-KeyCryptography

GNUPRIVACYGUARDGnuPG (GNU Privacy Guard) supports both symmetric cryptography andpublic-keycryptographyandiscompliantwiththeIETFOpenPGPstandard.TheLinux command for GnuPG is gpg or the largely equivalent gpg2. See themanpageforhowtousegpg.Withgpg, you can generate a public key that you share with others and a

privatekeyyoukeep secret.Youandotherscanuse thepublickey toencryptfilesandmessageswhichonlyyoucandecryptusingtheprivatekey.You can digitally sign amessage by encrypting itwith your private key.A

receiver of the signedmessage can decrypt it with the sender’s public key torecovertheoriginalmessageandauthenticatethatitisindeedsent/signedbythesender(Figure7.12).

Figure7.13DigitalSignature

Todoall that,makesureyoufirstsetupGnuPGandyourpersonalkeys. IfyourLinuxdistributiondoesnotalreadyprovidegpg,youcaneasilyinstallthegnupgpackage(Chapter8,Section8.2)witheitherofthefollowingcommands:

dnfinstallgnupg(CentOS/Fedora)apt-getinstallgnupg

(Ubuntu/Debian)

Ifyoulike touseaGUIforgpg, installalso thegpapackage.However, thecommand-lineinterfaceisentirelyadequate.

SettingUpGnuPGKeysTousegpg,youfirstneedtogenerateyourpublic-privatekeypair.gpg–gen-keyYou’ll be prompted to enter your choices for keytype (pick the default),

keysize(pick2048),andapassphrase(picksomethingyouwon’tforget,butwillbe very hard for anyone to guess). The passphrase is required each time youaccessyourprivatekey,thuspreventingothersfromusingyourprivatekey.You’llgetakeyiddisplayedwhenyourkeypairisgenerated.Yourkeysand

otherinfoarestoredbydefaultinthefolder$HOME/.gnupg.Usegpg–list-public-keystodisplayyourpublickeys.Forexample,pub1024D/FCF2F84D2018-07-25uidPaulWang(monkeykia)<pwang@cs.kent.edu>sub1024g/B02C4B402018-07-25Thepub linesays thatPaul’spublicmasterkey (for signature) isa1024-bit

DSAkeywithidFCF2F84Dandthathispublicsubkey(fordataencryption)isa1024-bitElGamakey.

Thekeyuidis"PaulWang",monkeykia,orthelistedemailaddress.To enable others to encrypt information to be delivered for your eyes only,

youshouldsendyourpublickeystoapublickeyserver.Thecommandgpg–send-keysyour_keyidsendsyourpublickeytoadefaultgpgkeyserver,suchashkp://subkeys.pgp.netAlso, you can send your public keys to anyone by sending them anASCII

armoredfilegeneratedbygpg–armor–exportyour_keyid>mykey.ascThe.ascsuffixsimplyindicatesthatafileisanASCIItextfile.Themykey.asc

containsyourkeyencodedusingbase64,awaytouse64ASCIIcharacters(A–Z, a–z, 0–9 and +/) to encode non-ASCII files for easy communication overnetworks, especially via email. The Linux base64 command performs thisencoding/decodingonanyfile.Seemanbase64formoreinformation.Such ASCII armored key files can be emailed to others or sent to another

computerandimportedtoanotherGnuPGkeyringwithacommandsuchasgpg–importmykey.ascAlso,edityour$HOME/gnupg/gpg.conffileandappendthelinedefault-keyyour_keyid

ncryption/DecryptionwithGnuPGToencryptafileusingapublickeyofkey_uid,gpg–encrypt-rkey_uidfilenameresultinginanencryptedfile filename.gpgthatcanbesent to the targetuser

whoistheonlyonethatcandecryptit.Evenifyouarenotgoingtosendafiletoanyone,youcanstillkeepsecretsin

that fileofyoursprotected incase someonegainsunauthorizedaccess toyourcomputeraccount.Youcangpg–encrypt-r"your_key_uid"filenamermfilenamegenerating the encrypted filename.gpg and removing the original filename.

Youcaneasilyviewtheencryptedversionwithgpg–decryptfilename.gpg|moreTomakemaintaininganencryptedfileeveneasier,youmayconfigurevi/vim

toworktransparentlywithgpg,allowingyoutousevim toviewandeditclearaswellasgpgencryptedfiles.TheVIMextensiontGpg(yetanotherplug-inforencryptingfileswithgpg)isagoodchoiceforthispurpose.The Seahorse tool (Chapter 4, Section 4.1) is convenient to manage your

7.11

encryptionkeys.

SECUREEMAILModern email clients, such as Microsoft Outlook and the open sourceThunderbird, support secure email specified by the S/MIME (SecureMultipurposeInternetMailExtensions)standard.Oncesetup,youcansendandreceiveencryptedemail,aswellassignedmessages.Whenanemailmessageisencrypted,emailcontentsandattachmentsareturnedintociphertext.Ofcourse,theemailsubjectorotheremailheadersarenotencrypted.Whenanemailissigned,nothingisencrypted,exceptasignedmessagedigest

is attached.Normally,wedonotwant to signour email.But, it is possible tobothencryptandsignanemail.Theprerequisite forS/MIMEis thateachcorrespondentmusthaveanemail

certificateinstalledinasecureemailclient.Commercial personal email (S/MIME) certificates are easily available from

CAs. You may even find CAs that offer free email certificates. But thecertification application and verification process may be complicated andbothersometomanyusers.A good alternative to S/MIME is PGP/MIME (PrettyGood Privacy)which

doesnotrequireaCA-issuedcertificate.ThefreeGnuPG(GNUPrivacyGuard;GPG) is an implementation of the OpenPGP standard. Using GPG, you cangenerateanddistributeyourownkeypairforsecureemail.Let’stakealookathowtosetupThunderbirdforsecureemailwithGnuPG.

Other email clients, such as evolution and mutt, can work with GnuPGsimilarly.

SecureEmailwithThunderbirdFirst make sure you have Thunderbird and GnuPG installed on your Linuxdistribution. Then follow this simple procedure to enable Thunderbird secureemail(PGP/MIME).

1. OpenThunderbird and use the tools->Add-ons option to search for andinstalltheEnigmailadd-on.

2. FollowtheEnigmailset-upwizardtosetupyourkeypair(useyourcorrectname and email address). The email address should correspond to yourThunderbird email identity.Youmay choose for the key to never expire.Alsoit isrecommendedthatyouchoosethe4096RSA/RSAkeyfromtheAdvancedoptions.

3. Optionally,youmaychoosetoassociateaJPGimagewiththekey.Thiscanbedonelateralso.

Figure7.14ThunderbirdwithEnigmailAdded

Now,yourThunderbirdissetup(Figure7.13).But,beforeyoucansendandreceiveencryptedemail,youneedto(A) Send your public key to people you know so they can send encrypted

emailtoyou.(B)Receive/installtheirpublickeyssoyoucanencryptemailtothem.For(A)dothis:

1. From the ThunderbirdMenu (click the three-bars icon on the right-handsideoftheThunderbirdmenubar),selectOpenPGP->KeyManagementtopopuptheKeyManagementdialog(Figure7.14).

2. IntheKeyManagementdialog,checktheoptionboxDisplayAllKeysbyDefault.Youshouldseeyourkeylisted.

3. Right click the key youwant, and select the option Send PublicKey byEmail.ThesameoptionisalsoavailablefromtheFilemenu.

4. An email compositionwindow openswith the key file (with .asc suffix)alreadyattached.Justsendthisemailnormally.Thatisit.

Figure7.15SendingPublicKey

7.12

For(B),dothis:

1. Open incoming email from your friend containing his/her public key asattachment.

2. Open (double click) the key file attachment (with .asc suffix), andThunderbirdwillinstallthepublickeyreceivedautomatically.

Nowyouare truly ready for secureemail.Aftercomposingyourmessageandaddingattachments,selecttheoptionsOpenPGP->encrypt(toencrypttheemailmessage) and/orOpenPGP- > sign (to add a signature) before sending. If theemail has one or more attachments, be sure to also select the OpenPGP- >PGP/MIME option. When you open an encrypted email in your inbox,Thunderbirdasksyouforthepassphraseofyourprivatekeyandthendecryptsitforyouautomatically.

MESSAGEDIGESTSAmessagedigestisadigitalfingerprintofamessageorfile.Variousalgorithmshavebeendevisedtotakeamessage(file)ofanylengthandreduceittoashortfixed-length hash known as the digest of the original message or file (Figure7.15).

Figure7.16MD5MessageDigest

Thesealgorithmsaredesignedtoproduceadifferentdigestifanypartofthemessageisaltered.Itisalmostimpossibletodeducetheoriginalmessagefromknowledge of the digest. However, because there are an infinite number ofpossiblemessagesbutonlyafinitenumberofdifferentdigests,vastlydifferentmessagesmayproducethesamedigest.Messagedigestsarethereforeusefulinverifyingtheintegrity (unalteredness)

of files. When software is distributed online, a good practice is to display afingerprintforthefile,allowingyoutochecktheintegrityofthedownloadandtoavoidanyTrojanhorsecode.MD5isapopularalgorithmproducing128-bitmessagedigests.AnMD5hash

isusuallydisplayedasasequenceof32hexadecimaldigits.OnLinux,youcanproduceanMD5digestwiththemd5sumcommandmd5sumfilename>digestFileYou’ll get adigestFile file containingonly thehash and thename filename.

AfterdownloadingbothfilenameanddigestFile,ausercancheckfile integritywithmd5sum-cdigestFileOther digest algorithms in wide use include SHA-1 and others. The Linux

commandsha1sumisanalternativetomd5sum.

SoftwareandMessageSigningTo digitally sign a particularmessage (or file) without having to encrypt thatentiremessage isoftendesirable.Todo this, adigestof themessageor file iscreatedfirst,usingasuitablemessagedigesthashfunction.

Figure7.17DigitalSignature

Todigitallysignaparticularmessageorapieceofsoftware,adigestorhashis created first. The digest is then encrypted using the signer’s private key toproduce a digital signature. Any receiver of a signed message/software cangenerate amessage digest from the receivedmessage and check it against thedigestobtainedbydecryptingthedigitalsignaturewiththesigner’spublickey.Amatchverifiestheintegrityandtheauthenticityofthereceivedmessage(Figure7.16,source:Wikipedia).

7.13

Hereishowtousegpgfordigitalsignature.

gpg--signfile(producessignedbinaryfile.gpg)gpg--clearsign

file(producessignedASCIIfile.asc)

The–decryptoptionautomaticallyverifiesanyattachedsignature.

THEWEBOut of all the networking applications, the Web is perhaps one of the mostimportantanddeservesourspecialattention.There is no central control or administration of the Web. Anyone can

potentiallyputmaterialon theWeband retrieve information from it.TheWebconsists of a vast collection of documents that are located on computersthroughout theworld.Thesedocuments are createdbyacademic,professional,government, and commercial organizations, as well as by individuals. Thedocuments areprepared in special formats anddelivered throughWeb servers,programsthatreturndocumentsinresponsetoincomingrequests.LinuxsystemsareoftenusedtorunWebservers.AnintroductiontotheWebisprovidedinthischapter.Chapter9discussesservingtheWeb.Primarily, Web documents are written in Hypertext Markup Language

(HTML,Section7.16). EachHTMLdocument can contain (potentiallymany)links to other documents served by different servers in other locations andthereforebecomepartofawebthatspanstheentireglobe.NewmaterialsareputontheWebcontinuously,andinstantaccesstothiscollectionofinformationcanbeenormouslyadvantageous.AstheWebgrew,MIT(MassachusettsInstituteofTechnology, Cambridge, MA) and INRIA (the French National Institute forResearchinComputerScienceandControl)agreedtobecomejointhostsoftheW3Consortium,astandardsbodyfortheWebcommunity.AWebbrowser isaprogramthathelpsusersobtainanddisplay information

fromtheWeb.Giventhelocationofatargetdocument,abrowserconnectstothecorrectWeb server and retrieves and displays the desired document. You canclicklinks inadocument toobtainotherdocuments.Usingabrowser,youcanretrieveinformationprovidedbyWebserversanywhereontheInternet.Typically,aWebbrowser,suchasFirefox,supportsthedisplayofHTMLfiles

andimages instandardformats.Helperapplicationsorplug-inscanaugmentabrowsertotreatpageswithmultimediacontentsuchasaudio,video,animation,andmathematicalformulas.

HypertextMarkupLanguage

AWebbrowsercommunicateswithaWebserver throughanefficientprotocol(HTTP)which is designed toworkwithhypertext andhypermediadocumentsthatmaycontain regular text, images,audio,andvideo.NativeWebpagesarewrittenintheHTML(Section7.16)andusuallysavedinfileswiththe.html(or.htm)suffix.HTMLorganizesWebpagecontent(text,graphics,andothermediadata)and

allows hyperlinks to other pages anywhere on theWeb. Clicking such a linkcauses your Web browser to follow it and retrieve another page. The Webemploys an open addressing scheme that allows links to objects and servicesprovidedbyWeb,email,filetransfer,audio/video,andnewsgroupservers.Thus,theWebspaceisasupersetofmanypopularInternetservices.Consequently,aWeb browser provides the ability to access awide variety of information andservicesontheInternet.

URLsTheWebusesUniformResourceLocators(URLs)toidentify(locate)resources(filesandservices)availableontheInternet.AURLmayidentifyahost,aserverport, and the target file stored on that host. URLs are used, for example, bybrowserstoretrieveinformationandbyHTMLtolinktootherresources.AfullURLusuallyhastheformscheme://server:port/pathnameThe scheme, part indicates the information service type and therefore the

protocol touse.Commonschemes includehttp(Webservice), ftp(file transferservice), mailto (email service), file (local file system), https (secure Webservice),andsftp(securefiletransferservice).Forexample,sftp://pwang@tiger.zodiac.cs.kent.edu/users/cs/faculty/pwanggetsyou thedirectory listof /users/cs/faculty/pwang.ThisworksonFirefox

and on the Linux file browser nautilus, assuming that you have set up yourSSH/SFTP (Section 7.6). Many other schemes can be found atwww.w3.org/addressing/schemes.ForURLs ingeneral, theserver identifiesahostandaserverprogram.The

optionalportnumber isneededonly if theserverdoesnotuse thedefaultport(forexample,21forFTPand80forHTTP).Theremainderof theURL,whengiven,isafilepathname.Ifthispathnamehasatrailing/character,itrepresentsadirectoryratherthanadatafile.Thesuffix(.html,.txt,.jpg,etc.)ofadatafileindicatesthefiletype.ThepathnamecanalsoleadtoanexecutableprogramthatdynamicallyproducesanHTMLorothervalidfiletoreturn.WithinanHTMLdocument,youcanlinktoanotherdocumentservedbythe

7.14

sameWebserverbygivingonlythepathnamepartoftheURL.SuchURLsarepartiallyspecified.ApartialURLwith a / prefix (for example, /file_xyz.html)refers to a file under the serverroot, the top-level directory controlled by theWeb server. A partial URLwithout a leading / points to a file relative to thelocation of the document that contains the URL in question. Thus, a simplefile_abc.html refers to that file in the samedirectory as the current document.Whenbuildingawebsite,itisadvisabletouseaURLrelativetothecurrentpageasmuchaspossible,makingiteasytomovetheentirewebsitefoldertoanotherlocationonthelocalfilesystemortoadifferentserverhost.

AccessingInformationontheWebYoucandirectly access anyWebdocument, directory, or servicebygiving itsURL in the Location box of a browser. When given a URL that specifies adirectory, aWeb serverusually returns an index file (typically, index.html) forthatdirectory.Otherwise,itmayreturnalistofthefilenamesinthatdirectory.YoucanuseasearchenginesuchasGoogle toquickly lookfor information

ontheWeb.

HANDLINGDIFFERENTCONTENTTYPESOntheWeb,filesofdifferentmediatypescanbeplacedandretrieved.TheWebserver andWebbrowseruse standardcontent type designations to indicate themediatypeoffilesinordertoprocessthemcorrectly.The Web borrowed the content type designations from the Internet email

system and uses the same MIME (Multipurpose Internet Mail Extensions)definedcontent types.Therearehundredsofcontent types inuse today.Manypopulartypesareassociatedwithstandardfileextensions.Chapter6,Table6.3givessomeexamples.When a Web server returns a document to a browser, the content type is

indicated. The content type information allows browsers to decide how toprocesstheincomingcontent.Normally,HTML,text,andimagesarehandledbythebrowserdirectly.Othertypessuchasaudioandvideoareusuallyhandledbyplug-insorexternalhelperprograms.

7.15

Figure7.18WebServerFunction

PUTTINGINFORMATIONONTHEWEBNow let’s turn our attention to how information is supplied on theWeb. TheunderstandingshedsmorelightonhowtheWebworksandwhatittakestoserveupinformation.TheWebputsthepowerofpublishinginthehandsofanyonewithacomputer

connectedtotheInternet.Allyouneedis torunaWebserveronthismachineandestablishfilesforittoservice.Major computer vendors offer commercialWeb serverswith their computer

systems.Apacheisawidelyusedopen-sourceWebserverthatisfreelyavailablefromtheApacheSoftwareFoundation(www.apache.org).Linux systems are especially popular as Web hosting computers because

Linux is free, robust, and secure. Also, there are many useful Web-relatedapplicationssuchasApache,PHP(activeWebpage),MySQL(databaseserver),andmoreavailablefreeofcharge.OnceaWebserverisupandrunningonyourmachine,alltypesoffilescanbe

served (Figure7.17).On a typical Linux system, follow these simple steps tomakeyourpersonalWebpage.

1. Make a file directory in your home directory ( userid /public_html) tocontainyourfilesfortheWeb.ThisisyourpersonalWebdirectory.Makethisdirectorypubliclyaccessible:chmoda+xuserid/public_htmlWhenindoubt, ask your system managers about the exact name to use for yourpersonalWebdirectory.

2. In your Web directory, establish a home page, usually index.html, inHTML.Thehomepageusuallyfunctionsasanannotatedtableofcontents.Makethisfilepubliclyreadable:chmoda+ruserid/public_html/index.html

7.16

3. Placefilesanddirectoriescontainingdesired informationinyourpersonalWeb directory. Make each directory and each file accessible as before.Refertothesefileswithlinksinthehomepageandotherpages.

4. Let people know the URL of your home page, which is typicallyhttp://your-severyour-userid

In a Web page, you can refer to another file of yours with a simple linkcontaining a relative URL ( < a href="filename" > ), where filename can beeitherasimplenameorapathnamerelativetothecurrentdocument.AmongtheWebfileformats,hypertextiscriticalbecauseitprovidesameans

foradocumenttolinktootherdocuments.

WHATISHTML?HTML (theHypertextMarkup Language) is used tomarkup the content of aWeb page to provide page structure for easy handling byWeb clients on thereceiving end. Since HTML 4.0, the language has become standardized.XHTML (XML compatible HTML) is the current stable version. However, anewstandardHTML5isfastapproaching.AdocumentwritteninHTMLcontainsordinarytextinterspersedwithmarkup

tagsandusesthe.htmlfilenameextension.Thetagsmarkportionsofthetextastitle,sectionheader,paragraph,referencetootherdocuments,andsoon.Thus,anHTMLfileconsistsoftwokindsofinformation:contentsandHTMLtags.AbrowserfollowstheHTMLtagstolayoutthepagecontentfordisplay.Becauseof this, line breaks and extra white space between words in the content aremostly ignored. Inaddition to structuringand formattingcontents,HTMLtagscan also reference graphics images, link to other documents, mark referencepoints,generate formsorquestionnaires, and invokecertainprograms.VariousvisualeditorsorpagemakersareavailablethatprovideaGUIforcreatinganddesigningHTMLdocuments.Forsubstantialwebsitecreationprojects,itwillbehelpful to use integrated development environments such as MacromediaDreamweaver.Ifyoudon’thavereadyaccesstosuchtools,aregulartexteditorcancreateoreditWebpages.AnHTMLtagtakestheform<tag>.Abegintagsuchas<h1>(level-onesectionheader)ispairedwithanendtag,</h1> inthiscase,tomarkcontentinbetween.Table7.1listssomefrequentlyusedtags.SomeHTMLTags

7.17

ThefollowingisasampleHTMLpage(Ex:ex07/Fruits):

>html<>head<>title<ABasicWebPage>/title<>/head<>body<>h1<Big

onFruits>/h1<>p<Fruitsaregoodtastingandgoodforyou

...>/p<>p<Therearemanyvarieties,...andhereisashortlist:

>/p<>ol<>li<Apples>/li<>li<Bananas>/li<>li<Cherries

>/li<>/ol<>/body<>/html<

Figure7.18showstheBigonFruitspagedisplayedbyFirefox.

WEBHOSTINGWebhostingisaservicetostoreandserveready-madefilesandprogramssothattheyareaccessibleontheWeb.Hence,publishingontheWebinvolves

1. Designing and constructing the pages and writing the programs for awebsite

2. Placingthecompletedsitewithahostingservice

Colleges and universities host personal and educational sites for students andfacultywithoutcharge.Webhostingcompaniesprovidetheserviceforafee.

Figure7.19ASampleWebPage

CommercialWebhostingcanprovidesecuredatacenters(buildings),fastandreliable Internet connections, specially tuned Web hosting computers (mostlyLinuxboxes),serverprogramsandutilities,networkandsystemsecurity,dailybackup,andtechnicalsupport.Eachhostingaccountprovidesanamountofdiskspace, a monthly network traffic allowance, email accounts, Web-based sitemanagement and maintenance tools, and other access such as FTP andSSH/SFTP.

7.18

7.19

Tohost a site under a givendomainname, a hosting service associates thatdomain name to an IP number assigned to the hosted site. The domain-to-IPassociation is made through DNS servers and Web server configurationsmanagedbythehostingservice.

DOMAINREGISTRATIONTo obtain a domain name, you need the service of a domain name registrar.Mostwillbehappytoregisteryournewdomainnameforaverymodestyearlyfee.Onceregistered,thedomainnameispropertythatbelongstotheregistrant.Nooneelsecanregisterforthatparticulardomainnameaslongasthecurrentregistrantkeepstheregistrationingoodorder.ICANN accredits commercial registrars for commonTLDs, including .com,

.net, .org, and .info. Additional TLDs include .biz, .pro, .aero, .name, and

.museum.Restricteddomains(forexample, .edu, .gov,and .us)arehandledbyspecial registries (forexample,net.educause.edu,nic.govandnic.us).Country-codeTLDsarenormallyhandledbyregistriesintheirrespectivecountries.

AccessingDomainRegistrationDataThe registration record of a domain name is often publicly available. ThestandardInternetwhoisserviceallowseasyaccesstothisinformation.OnLinuxsystems,easyaccesstowhoisisprovidedbythewhoiscommandwhoisdomain_namewhichliststhedomainregistrationrecordkeptataregistrar.Forexample,whoiskent.eduproducesthefollowinginformation

DomainName:KENT.EDURegistrant:KentStateUniversity500E.Main

St.Kent,OH44242UNITEDSTATESAdministrativeContact:PhilipL

ThomasNetwork&TelecommKentStateUniversitySTHKent,OH44242

UNITEDSTATES(330)672-0387pki-admin@kent.eduTechnical

Contact:NetworkOperationsCenterKentStateUniversity120Library

BldgKent,OH44242UNITEDSTATES(330)672-3282noc@kent.eduName

Servers:NS.NET.KENT.EDU131.123.1.1DHCP.NET.KENT.EDU

131.123.252.2Domainrecordactivated:19-Feb-1987Domainrecordlast

updated:17-Feb-2016Domainexpires:31-Jul-2018

OnLinuxsystems,thewhoiscommandisusuallyalinktojwhois.

THEDNS

DNS provides the ever-changing domain-to-IP mapping information on theInternet.Wementioned that DNS provides a distributed database service thatsupports dynamic retrieval of information contained in the name space. Webbrowsers and other Internet client applications will normally use the DNS toobtain the IP of a target host before making contact with a server over theInternet.TherearethreeelementstotheDNS:theDNSnamespace(Section7.2),the

DNSservers,andtheDNSresolvers.

DNSServersInformation in the distributed DNS is divided into zones, and each zone issupported by one ormore name servers running on different hosts.A zone isassociatedwithanodeonthedomaintreeandcoversallorpartofthesubtreeatthatnode.Anameserverthathascompleteinformationforaparticularzoneissaidtobeanauthorityforthatzone.Authoritativeinformationisautomaticallydistributed to other name servers that provide redundant service for the samezone. A server relies on lower level servers for other information within itssubdomainandonexternalserversforotherzonesinthedomaintree.Aserverassociatedwiththerootnodeofthedomaintreeisarootserverandcanleadtoinformation anywhere in the DNS. An authoritative server uses local files tostore information, to locate key serverswithin andwithout its domain, and tocache query results from other servers. A boot file, usually /etc/named.boot,configuresanameserveranditsdatafiles.Themanagementofeachzoneisalsofreetodesignatethehoststhatrunthe

nameserversandtomakechangesinitsauthoritativedatabase.Forexample,thehostns.cs.kent.edumayrunanameserverforthedomaincs.kent.edu.Anameserveranswersqueries fromresolversandprovideseitherdefinitive

answersorreferralstoothernameservers.TheDNSdatabaseissetuptohandlenetworkaddress,mailexchange,hostconfiguration,andothertypesofqueries,withsometobeimplementedinthefuture.The ICANNandothersmaintainrootnameservers associatedwith the root

node of theDNS tree. In fact, theVeriSign host a.root-servers.net runs a rootnameserver.Actually, the letterarangesuptomfora totalof13rootserverscurrently.Domainnameregistrars,corporations,organizations,Webhostingcompanies,

andotherInternetserviceproviders(ISPs)runnameserverstoassociateIPstodomain names in their particular zones. All name servers on the Internetcooperatetoperformdomain-to-IPmappingsonthefly.

7.20

DNSResolversADNS resolver is a program that sends queries to name servers and obtainsreplies fromthem.OnLinuxsystems,a resolverusually takes theformofaClibrary function. A resolver can access at least one name server and use thatnameserver’sinformationtoansweraquerydirectlyorpursuethequeryusingreferralstoothernameservers.Resolvers, in the form of networking library routines, are used to translate

domain names into actual IP addresses. These library routines, in turn, askprescribednameserverstoresolvethedomainnames.Thenameserverstousefor any particular host are normally specified in the file /etc/resolv.conf or/usr/etc/resolv.conf.CommonDNSRecord/RequestTypes

The DNS service provides not just the IP address and domain nameinformationforhostsontheInternet.Itcanprovideotherusefulinformationaswell.Table7.2showscommonDNSrecordandrequesttypes.

DYNAMICGENERATIONOFWEBPAGESDocuments available on theWeb are usually prepared and set in advance tosupplysomefixedcontent,eitherinHTMLorinsomeotherformatsuchasplaintext, PDF, or JPEG.These fixed documents are static.AWeb server can alsogeneratedocumentsontheflythatbringtheseandotheradvantages:

Customizing a document depending on when, where, who, and whatprogramisretrievingitCollecting user input (withHTML forms) and providing responses to theincominginformationEnforcingcertainpoliciesforoutgoingdocumentsSupplying contents such as game scores and stock quotes, which arechangingbynature

DynamicWeb pages are not magic. Instead of retrieving a fixed file, aWebserver calls another program to compute thedocument to be returned.Asyoumay have guessed, not every program can be used by a Web server in thismanner.Therearetwowaystoaddserver-sideprogramming:

LoadprogramsdirectlyintotheWebservertobeusedwhenevertheneedarises.Callanexternalprogramfromtheserver,passingarguments to it (via theprogram’s stdin and environment variables) and receiving the results (viatheprogram’sstdout)thusgenerated.SuchaprogrammustconformtotheCommonGateway Interface (CGI) specificationsgoverninghow theWebserverandtheexternalprograminteract(Figure7.19).

Figure7.20CommonGatewayInterface

DynamicServerPagesThedynamicgenerationofpagesismadesimplerandmoreintegratedwithWebpagedesignandconstructionbyallowingaWebpagetocontainactivepartsthataretreatedbytheWebserverandtransformedintodesiredcontentontheflyasthepageisretrievedandreturnedtoaclientbrowser.Theactivepartsinapagearewritteninsomekindofnotationtodistinguish

themfromthestaticpartsofapage.TheASP(ActiveServerPages),JSP(JavaServerPages),andthepopularPHP(HypertextPreprocessor;Chapter9,Section9.17)areexamples.BecauseactivepagesaretreatedbymodulesloadedintotheWebserver, the

processingisfasterandmoreefficientcomparedtoCGIprograms.Activepagetechnologies such as PHP also provide form processing, HTTP sessions, andeasyaccesstodatabases.Therefore,theyoffercompleteserver-sidesupportfordynamicWebpages.BothCGIandserverpagescanbeusedtosupportHTMLforms,thefamiliar

fill-outformsyouoftenseeontheWeb.

7.21 HTTPBRIEFLYOn the Web, browser-server communication follows HTTP. A basicunderstanding of HTTP is important for Linux programmers because LinuxsystemsareverypopularWebserverhosts.ThestartofHTTPtracesbacktothebeginningoftheWebintheearly1990s.

HTTP/1.0wasstandardizedearlyin1996.ImprovementsandnewfeatureshavebeenintroducedandHTTP/1.1isnowthestableversion.HereisanoverviewofanHTTPtransaction:

1. Connection—Abrowser(client)opensaconnectiontoaserver.2. Query—Theclientrequestsaresourcecontrolledbytheserver.3. Processing—Theserverreceivesandprocessestherequest.4. Response—Theserversendstherequestedresourcebacktotheclient.5. Termination —The transaction is finished, and the connection is closed

unlessanother transaction takesplace immediatelybetween theclientandserver.

HTTPgovernstheformatofthequeryandresponsemessages(Figure7.20).Theheaderpart is textual,andeachlineintheheadershouldendinRETURN

andNEWLINE,butitmayendinjustNEWLINE.Theinitiallineidentifiesthemessageasaqueryoraresponse.

Aquery linehas threepartsseparatedbyspaces:aquerymethod name, alocal path of the requested resource, and an HTTP version number. Forexample,GET /path/to/file/index.htmlHTTP/1.1 or POST /path/script.cgiHTTP/1.1 TheGETmethod requests the specified resource and does notallowamessagebody.AGETmethodcaninvokeaserver-sideprogrambyspecifyingtheCGIoractive-pagepath,aquestionmark,andthenaquerystring: GET /cgi-bin/newaddr.cgi?name=value1&email=value2 HTTP/1.1Host: tiger.zodiac.cs.kent.edu Unlike GET, the POST method allows amessage body and is designed to work with HTML forms for collectinginputfromWebusers.A response (or status) line also has three parts separated by spaces: anHTTPversionnumber,astatuscode,andatextualdescriptionofthestatus.Typical status lines are HTTP/1.1 200 OK for a successful query orHTTP/1.1404NotFoundwhentherequestedresourcecannotbefound.TheHTTPresponsesendstherequestedfiletogetherwithitscontenttype(Section7.14)andlength(optional)sotheclientwillknowhowtoprocessit.

7.22

7.23

AREALHTTPEXPERIENCELet’smanuallysendanHTTPrequestandgetanHTTPresponse.Todothatwewill use the nc command. The command ncat provides command-line (andscripting) access to the basic TCP and UDP (Chapter 12, Section 12.6) andtherefore allowsyou tomake anyTCPconnectionsor send anyUDPpackets.Suchabilitiesareusuallyreserved toprogramsat theC-language level thatsetupsockets(Chapter12,Section12.6)fornetworking.Forexample,thesimpleBashpipeline(Ex:ex07/poorbr.sh)echo$’GET/WEB/test.htmlHTTP/1.0n’|ncattiger.zodiac.cs.kent.edu80retrieves the Web page tiger.zodiac.cs.kent.edu/WEB/test.html. In this

example,weappliedtheBashstringexpansion(Chapter3,Section3.7).Note the HTTP get request asks for the file /WEB/test.html under the

document root folder managed by the Web server on tiger. The request isterminatedbyanemptyline,asrequiredbytheHTTPprotocol.Trythisandyou’llseetheresultdisplay.

HTTP/1.1200OKDate:Tue,20Mar201819:45:03GMTServer:

Apache/2.4.27(Fedora)X-Powered-By:PHP/7.0.23Cache-Control:max-

age=86400Expires:Wed,21Mar201819:45:03GMTVary:Accept-

EncodingContent-Length:360Connection:closeContent-Type:

text/html;charset=UTF-8>!DOCTYPEHTML<>html

xmlns="http://www.w3.org/1999/xhtml"lang="en"xml:lang="en"<AND

THERESTOFTHEHTMLPAGE>/html<

AsyoucanseefromtheHTTPresponse, theWebserverontiger isApacheversion2runningunderFedora,aLinuxsystem.FordownloadingfromtheWeb,youdon’tneedtorelyonourlittlepipeline.

The wget command takes care of that need nicely. Wget supports HTTP,HTTPS, and FTP protocols and can download single files or follow links inHTMLfilesand recursivelydownloadentirewebsites forofflineviewing.Thewget command can continue towork after you log out so you can downloadlargeamountsofdatawithoutwaiting.

FORMOREINFORMATIONIPv6 is the next-generation Internet protocol. See www.ipv6.org/ for anoverview.The official website for Gnu Privacy Guard is www.gnupg.org, and for

7.24

OpenSSH,iswww.openssh.com.Public-Key Cryptography Standards (PKCS) can be found at RSALaboratories(www.rsa.com/rsalabs).HTML5isthenewandcomingstandardforHTML.SeethespecificationatW3C.TheDNSisbasictokeepingservicesontheInternetandWebrunning.FindourmoreaboutDNSatwww.dns.net/dnsrd/docs/.HTTPisbasictotheWeb.SeeRFC1945forHTTP1.0andRFC2068forHTTP1.1.

SUMMARYInthemoderncomputingenvironment,computersandnetworksareinseparable.Networking is an important aspect of any operating system, especially Linuxbecause the Internet has its origins in UNIX/Linux, and Linux systems areexcellentserverhosts.OntheInternet,eachhostcomputerisidentifiedbyitsIPaddressaswellasby

its domain name. TheTCP/IP andUDP/IP protocols are basic to the Internet.Network-based services often follow the client-and-servermodel,where clientprograms(suchasWebbrowsers)communicatewithserverprograms(suchasWebservers)usingwell-definedprotocols (suchasHTTP).Aparticular serverprogramrunningonaspecifichostisidentifiedbythehost’sIPordomainnametogetherwiththeserverprogram’sportnumber(suchas80forWebservers).The ICANN manages the IP address space and the DNS. The distributed

Domain Name Service is a fundamental networking service because itdynamicallymaps domain names to IP addresses and also provides importantinformation for sending/receiving email. The commands host,nslookup, anddigcanbeusedtoobtainDNSdatafortargethosts.Withnetworkingyoucanupload/downloadfileswith ftpandsftp; log in to

remotecomputerswithtelnetandssh;copyandsynchfileswithrcp,scp,andrsync;checkifaremotesystemisalive/connectedwithping;testprotocolswithnc; access the Web; send and receive emails, and perform many otheroperations.When it comes to networking, security and privacy are important concerns.

Increasingly, computer systems require SSH, SFTP, and SCP for betterprotection. Automatic file sync can also use SSH for data transfer. The GnuPrivacyGuard(GnuPG)supportssymmetricandpublic-keycryptography. It isuseful for data/file encryption, secure email, and digital signature. Message

7.25

digest algorithms such as MD5 can produce digital fingerprints fordata/programstoguardtheirintegrity.LinuxsystemsareoftenusedtorunWebserversandtoprovideWebhosting

for individuals and organizations. BasicWeb documents are coded inHTML.Hyper references use URLs to link to other documents.MIME content typesindicatethemediatypeservedontheWeb.ThestatelessHTTPisarequest-responseprotocolwhosemessagesmayhave

anumberofheadersandanoptionalmessagebody.

EXERCISES1. What isacomputernetwork?Name themajorcomponents inacomputer

network.2. What is a networking client? What is a networking server? What is a

networkingprotocol?3. WhataddressingschemedoestheInternetuse?WhatistheformatofanIP

address?Whatisthequadnotation?4. Consider theIPaddress123.234.345.456Is thereanythingwrongwith it?

Pleaseexplain.5. RefertoSection7.6andsetupyourownpassword-lessSSHandSFTP.6. You can schedule commands to be executed automatically by Linux at

regular intervals. Find out about the crontab and the crontab command.Thensetupyourcrontabtorsyncsomeimportantfolderfromonesystemtoanother.Showyourcrontabcodeinfullandexplain.

7. RefertoSection7.9andsetupyourGnuPGkeys.8. RefertoSection7.11andsetupsecureemailwithThunderbird.9. Writea script thatwillencrypt/decryptwithgpg a file and leave it in the

sameplaceasbefore(withthesamefilename).10. WhatisDNS?Whydoweneedit?11. Whatdonameserversdo?Whydoweneedthem?12. WhatistherelationbetweentheWebandtheInternet?Whatistherelation

betweenHTTPandTCP/IP?13. WhatarethemajorcomponentsoftheWeb?WhyisHTMLcentraltothe

Web?14. What is the difference between aWeb server and aWeb browser? Is the

Webserverapieceofhardwareorsoftware?Explain.15. HowdoesaWebpagegetfromwhereitistothecomputerscreenofauser?16. WhatisaURL?WhatisthegeneralformofaURL?Explainthedifferent

1

URLschemes.17. Whatarecontenttypes?Howaretheyuseful?18. What is the difference between a staticWeb page and a generatedWeb

page?19. WhatisanHTTPtransaction?WhatisanHTTPquery?WhatisanHTTP

response?20. Take the domain name sofpower.com and write the full URL that will

accessitsWebserver.Add/linuxtotheendofthatURL.Wheredoesthatlead?

21. Take the domain name sofpower.comand find its IP address.Use this IPaddressinsteadofthedomainnametovisitthesite.WritethebitpatternforthisIPaddress.

22. SearchontheWebforICANN.Visit thesiteanddiscoveritsmissionandservices.

23. Findthedomainrecordforsofpower.com.Whoistheownerofthisdomainname?Whoaretheadministrativeandtechnicalcontacts?

24. FindtheDNSrecordforsofpower.com.25. Find out and describe in your own words what the special domain in-

addr.arpais.26. Refer to Section 7.22. Explain the notation $’GET /WEB/test.html

HTTP/1.0n’27. Refer to Section 7.22. Use the nc command to write a poor man’s Web

browserscriptpoorman.sh.poorman.shpathhostretrievesthepagehttp://host/path.The ARPANET was started in the late 1960s as an experimental facility for reliable military

networking.

8.1

Chapter8

BasicSystemAdministration

StudentsofLinuxnaturallywillfirstfocusonunderstandingLinuxandhowtouseiteffectively.Systemadministrationissomethingsuperusershandle.Yet,itisanimportantaspectofLinuxandfamiliaritywith,oratleastsomedegreeofunderstandingof,systemadminwillbebeneficialtohomesystemusersaswellas want-to-be administrators. A rewarding system admin job is a distinctpossibilityforwell-trainedLinuxpersonnel.Linux system admin is a vast topic ranging from hardware configuration,

managing multiple systems company wide, server farm operations, to cloudcomputing.Mostofthisisoutsidethescopeofthistextbook.Ourpresentationherefocusesonsystemadminbasicssuchasmanaginghome

Linux systemsand small-scalenetworked systems.Wewill coverusual admintasks including user accounts, software installation, process and servicemanagement,networkconfiguration,diskand filesystem tasks, systembackup,booting,andsecuritywithSELinux.

MANAGINGUSERSLinuxisamulti-usersystembynatureandoneofthemostbasicadmintasksisusermanagement.TherearethreetypesofusersonLinux,regularusers,adminusers,androot.The root user (userid root) is built-in to Linux and need not be created

manually.The root canperformalloperationsandaccess all fileswithout anyrestrictions. An admin user runs as a regular user but can perform rootoperationsviathesudo(seenextsubsection)orthesucommand.Itisadvisabletoavoidloginasrootandtoperformadmintasksasanadmin

userasmuchaspossible.Thereislittlepracticaldifferencebetweenrootandanadminuser except using sudo and having to enter the admin password. Some

appssuchasgoogle-chrome(Webbrowser)andvlc(videoplayer)won’trunasrootbutwillasadmin.SomeLinuxdistributionssuchasUbuntudisablerootbydefault.Userandothersystemmanagementcommandsarefoundmostlyin/usr/sbin

and/usr/bin.Andtheirusagecanbefoundinsection8oftheLinuxmanpages.Toadd, remove,modifyuser accountsuseuseradd,userdel, andusermod.

To add, remove,modifygroupsusegroupadd,groupdel, andgrouprmod. AGUI tool, usually found in the system settingsmenu, canmake creating newuserseasier(Figure8.1).

Figure8.1UserAccountsTool

Whenauserxyziscreated,usuallyahomedirectory/home/xyziscreatedwithanumberof standard folder in it (Chapter6,Section6.1).Anewgroupxyz isalsocreatedand listed in /etc/group.Anentry isplaced in the /etc/passwd filewhere all users of the system are listed. The entry for user xyz indicates itsnumericalUSERID, andgroup affiliations.Thepassword is indicated by an xandtheactualhashedpasswordislistedin/etc/shadow,afileoff-limitstoallbutroot.You’ll notice that /etc/passwd (/etc/group) also lists many standard users

(standardgroups)forparticularprocesseswhentheyexecute.Ofcourse,theuserandgroupsettingsareusedinDiscretionaryAccessControl(DAC)asdescribedinChapter6.Togivexyzadminstatus,makeitamemberofaspecialsystemadmingroup.

Depending on the Linux distribution the groupmay be admin,wheel (Fedoraand Redhat), staff, sudo (Ubuntu), or sudoers. Another way to make xyz anadministoaddthisentryxyzALL=(ALL)ALL

tothe/etc/sudoersconfigurationfilewiththecommandvisudo.Whenauser logs in, thepasswordgiven is hashed and checked against the

passwordhash in /etc/shadowtoauthenticate theuser. Inaschoolorcorporateenvironment,auserusuallyisabletousethesameloginformultiplecomputerson the organization’s network. Linux uses PAM (Pluggable AuthenticationModules) to satisfy login and authentication needs for many apps includinglogin.Per-appPAMconfigurationcanbefoundin/etc/pam.d/.Usernamesandpasswordsmaybestoredin localfilesoronacentralserver

accessed either directly viaLDAP (LightweightDirectoryAccess Protocol) orindirectly using SSSD (System Security Services Daemon). Commandsluseradd,lgroupadd,andsoonaretobeusedinsuchcases.Not infrequently,usersmayhaveaccess tomultipleLinuxcomputerswithin

thesameorganization.Thisiswhenlogin/fileserversarerequiredtoenableeachusertouseasingleuserid/passwordtoaccessauthorizedcomputers.Inaddition,userhomedirectoriescanbecentrallylocatedonfileserversandaccessedfromindividualLinuxboxesviaNFS(Chapter6,Section6.8).For auser, havingasingle home directory ismuchmoremanageable than a separate one on eachLinuxbox.

SudoLinuxadministrationtaskssuchassettingupnewuseraccounts, installingandupdatingsystem-widesoftware,andmanagingnetworkservicesmustusuallybeperformedbyprivilegeduserssuchasroot.Thisissecurebutnotveryflexible.Sudoisamethodtoallowregularuserstoperformcertaintaskstemporarily

asrootorassomeothermoreprivilegeduser.Thecommandnamesudocomesfromthecommandsu (substituteuser)whichallowsauser tobecomeanotheruser.Putting sudo in front of a complete commandyouwish to execute says:“allow me enough privilege to execute the following command.” If the givencommand is allowed, sudo sets the real and effective uid and gid (Chapter 6,Section 6.4) to those of a specific privileged user for the duration of theexecutionofthegivencommand.Allsudocommandsareloggedforsecurity.Thefile/etc/sudoerscontainsdatagoverningwhocangainwhatprivilegesto

executewhichcommandsonexactlywhathostcomputersandwhethertheuser’spasswordisrequiredornot.Thus,thesamesudoersfilecanbesharedbymanyhosts within the same organization. The file can only be modified via theprivilegedcommandvisudo.Youcanreadaboutsudoersanditssyntaxrulesbyman5sudoersThegeneralformofauserentryinsudoersis

8.2

r_userhosts=(s_user)commandsmeaningr_usercanexecutethegivencommandsass_useronthelistedhosts.

The(s_user)partcanbeomittedifs_userisroot.Herearesomeexamplesudoersentries.

pwanglocalhost=/sbin/shutdown-hnowpwanglocalhost=

/user/bin/systemctlstarthttpd,\/user/bin/systemctlstophttpd,

\/user/bin/systemctlrestarthttpdpwanglocalhost=/usr/bin/dnf

groupinstall"WebServer",\/usr/bin/dnfgroupupgrade"Web

Server"rootALL=(ALL)ALL%wheelALL=(ALL)ALL

Eachentrymustbegivenononeline.Therootentryisalwaystheretogiveroot the ability to sudo all commands on hosts as any user. Thewheel is thesystemadmingroup.Evenifyoulogin(orsu)asroot,youmayprefertousesudosoastoleave

logentriesforthetasksperformed.

PACKAGEMANAGEMENTInstalling, updating, and removing software are of course part of systemadministration.IntheworldofLinux,wedon’thaveappstoresbecausealmostallthepiecesofsoftwarearefreeandnotforsale.Linux software aremade available aspackages in repositories andpackage

management systems are used to manage software packages. A packagemanagement system supports system as well as application software. EvenupdatingtheLinuxkernelandmovingtothenextLinuxreleaseareincluded.For Linux we have two major package systems: the DEB-based Advanced

Packaging Tool (APT) for the Debian family and the RPM-basedYellow dogUpdater,Modified (YUM)for theRedHat family.ThenewerDNF(DandifiedYUM)isintheprocessofreplacingYUM.With a package management tool you can search, install/remove,

update/upgrade and otherwise manage software packages designated for yourversion of Linux stored in on-line repositories. The checking of softwaredependenciesandplacement/replacementoffilesandcommandsareperformedautomatically.MorerecentprojectssuchasGNOMESoftware(Figure4.1)arecreatingapp-

store-like tools to make software management more intuitive. These user-friendly software tools are increasingly being integrated into newer Linuxdistributions.Let’sseehowcommand-linepackagemanagementisdone.

SoftwareManagementTasksOn CentOS/Fedora, the dnf command is used for package management. OnUbuntu/Debian,usetheaptcommandforpackagemanagement.See themanualpages fordnf (man dnf) andapt-get (man apt-get) for full

documentationoftheircommandsandoptions.Toshowhowtheycanbeusedtoperform common softwaremanagement tasks, let’s give some examples.Notethatmostofthesecommandsrequireadminuserstatus.

To search for packages with name or description matching the givenkeywords:dnf search "keywords" (dnf search "media player")apt-cachesearch"keywords"(apt-cachesearch"mediaplayer")Toviewupdates available foryour systemwithout installing them:dnf–refreshcheck-updateapt-get-uupgradeTo install the given (or all) packages on your system: dnf upgrade[packages] (dnf upgrade firefox) apt-get upgrade [packages] (apt-getupgradefirefox)Toinstallnewpackagesalongwithanyrequireddependencies:dnf installpackages (dnf install thunderbird) dnf groupinstall group-name ... (dnfgroupinstall LibreOffice) apt-get install [packages] (apt-get installthunderbird)Tolistallinstalledpackages:dnflistinstalleddnfgrouplistinstalledapt–installedlistTo remove installed packages: dnf remove packages (dnf installthunderbird) dnf group remove group-name ... (dnf group removeLibreOffice)apt-getremove[packages](apt-getremovethunderbird)

DNFkeepsallactions inahistory listandnumbers themsequentially.Use thednf history op to display the history list, to undo/redo, rollback, or list allpackagesaddedtothesystembytheuser(userinstalled).

8.3

Figure8.2GUIforDNF

The command yumex-dnf provides aGUI (Figure 8.2) fordnf and can beeasiertouse.Togetitsimplydodnfinstallyumex-dnfSomeLinuxdistributions,Fedoraforexample,canuseDNFtoautomatically

installpackageswhentheuserissuesacommandthat ismissing,possiblyalsoinstallappsyoufindintheappstore.Theaptitudecommandisaninteractivefrontendforapt-get.

MANAGINGPROCESSESA program under execution is called a process. At any given moment, thereusuallyareagoodnumberofprocessesindifferentstagesofexecution.Becausetheyaremakingprogressatthesametime,weoftenrefertothemasconcurrentprocesses(Chapter11,Section11.9).Forexample,differentusersmaybedoingvariedtaskssuchaseditingfiles,

reading/sending email, surfing the Web, instant messaging, video chatting,listening tomusic etc. In addition to these concurrent user processes there arealsoservicedaemons, systemprocesses running in thebackgroundready todotheirduties.Todisplayandcontrolprocesses,youcanuse theGUI toolgnome-system-

monitor(Figure8.3).Ausercanmanagehis/herownprocessesandadmincanmanage all processes. Often a run-away process or a process that is notrespondingtouserinputcanbeterminatedusingthistool.

Figure8.3GnomeSystemMonitor

Alternatively,thecommandkill-9process_idcanbeused to immediately terminate thegivenprocess (Chapter5, Section

5.22).Tofindtheprocessidyoucanusepidofname_of_appThus,thefollowingcommandworkswellindeed(Chapter3,Section3.7).kill-9‘pidoffirefox’You can use the ps command to display more detailed information on

processes (seeChapter11,Section11.10 formore information).Otherprocessmanagement commands include top (displays processes and their systemresourceusage),pstree(displaysprocessesintreeform),pgrep(findsprocessesbypatternmatching),pkill/killall(terminatesprocessesbyname).In Figure 8.3 you also see several instances of the HTTP daemon (httpd)

whichistheApacheWebserverprogramthatstandsreadytoserveupwebpagesonthishostcomputer.Toseetheavailableservicedaemonprograms,statusofeach,andcontroltheir

execution,usetheGUItoolsystem-config-services(Figure8.4)whereyoucanenable/disable,start/stop,restartanychosenservice.

8.4

Figure8.4ServicesConfigurationTool

InFigure8.4youseethathttpdisenabledandrunning.Alsoweseethenote“httpdismanagedbysystemd.ThisisbecauseLinuxusesthesystemdaemonsystemdasacontrollerofallsystemprocesses.ThesystemdhasPID1andallotherprocessesareitsdescendants.Therefore,asystemadmincan,alternatively,usethefollowingcommandtoenable/disable,start/stop/restartagivenservice:systemctlopserviceForexample,systemctlstarthttpdsystemctlstophttpdsystemctlrestarthttpdsystemctlstatushttpdNotethatenablearrangestohaveaserviceautomaticallystartedonbootwhile

disableistheopposite.Thestart/stopoperationstarts/terminatesagivenserviceimmediately,enabledornot.Thestatusoperationdisplaysthecurrentstatusofagivenservice.

NETWORKCONFIGURATIONNetworking is an important function of the Linux kernel which performsoperationsthroughNetworkInterfaceCards(NICs).Forexample,in-househostsmay be connected to a router on an Ethernet LAN. The router in turn isconnected to the Internet through a cable or an ISDNmodemprovided by anInternetServiceProvider (ISP).ThesamehostmayalsohaveadditionalNICswiredorwireless.Thesamehostconnectedtotwodifferentnetworkscanserveasagatewaythatrelaystrafficbetweenthetwonetworks.Thus,therouterinourexampleisagatewaybetweenthein-houseLANandtheISP’snetwork.

Most likely, your home Linux workstation obtains its IP address via theDynamic Host Configuration Protocol (DHCP) from a DHCP server on yourLAN. On a home LAN, a wireless router is also the DHCP server. Whilebooting,aDHCPclientbroadcastsarequestwithitsownMediaAccessControl(MAC)addresstoobtainanIPaddressandasubnetmaskfromtheDHCPserverlocatedonthesamebroadcastsubnet.TheDHCPserverassignsanavailableorpreassigned IP address and usually also provides default gateway and DNSserver(Chapter7,Section7.3)addresses.Forahost,therearefourimportantpiecesofnetworkconfigurationdata:

ItsIPaddress—Forexample,192.168.1.42Itssubnetmask—Forexample,255.255.255.0meaningall192.168.1.xareonitssubnetIts default gateway—For example, 192.168.1.1 identifying the routerconnectingthesubnettotheoutsideIPaddressesofDNSservers

Note that the subnet mask is a bit mask whose leading sequence of 1 bitsindicates the fixedpartof subnet IPaddresses.Theumask for filepermissions(Chapter3,Section3.12)isanotherbitmaskwehaveseen.TheInternettransmitsdatainpackets.HencetheLinuxkernelmustdealwith

IP packets when performing networking. Figure 8.5 shows the structure of apacket.

Figure8.5AnIPv4Packet

Ifthepacketdestinationisonthehost’ssubnet,thekernelcansendthepacketdirectly.Ifnot,thepacketissenttothedefaultroutewhichisusuallyahostatyourISPoralocalgatewayconnectedtotheInternet.OnaLinuxsystem,networkinterfaceshavestandardnames,forexample:

lo—Theloopbackinterface,connectedtothehostitself,mostlyfortesting

anddiagnosiseth0oreno1—ThefirstEthernetinterfacewlan0—ThefirstWirelessnetworkinterface.ppp0—ThefirstPointtoPointProtocolnetworkinterfacevirbr0—Avirtualbridge interfaceusedbyvirtualmachines 1 foraddresstranslationsinordertoconnecttotheoutsidenetwork.

On most Linux distributions, the NetworkManager daemon initializes andconfiguresavailablenetworksautomaticallyandusuallynoadmin interventionisneeded.Network,aGUItooltoconvenientlycontrolnetworkingcanoftenbefoundon the system settingspanel.Figure8.6 shows the entry display of thisnetworkcontroltool.

Figure8.6NetworkControlTool

TheWiredinterfaceshowsthatthehostisona1000Mb/sLANwithalocalIPaddress192.168.1.42, itsMACaddress,DNSserverstouse,andthedefaultroute 192.158.1.1, likely a router connected to the Internet. Any wirelessinterfacesarealsolisted,enablingyoutoturnthemon/off.Clickonthegearicontoedittheconfigurationforaselectedinterface.Forexample,youcanaddDNSservers, pick firewall zones (Section 8.5), and set the security mode andpasswordforawirelessinterface.TheGUItoolnm-connection-editorandtheCLIcommandnmclialsoenable

youtoeditnetworkinterfacesettingsfortheNetworkManager.The file /etc/resolv.conf,generatedbyNetworkManager, storesDNSservers

tobeusedbyyoursystem.The/etc/hostsfilecanstoreIPaddressesfordomainnamesandanyaliasesforlocalandoften-usedremotehoststohelpdomaintoIPmapping.Editthehostsfilewithanystandardtexteditor.Otherusefulcommandsfornetworkadmininclude

pinghost—testsifhostisresponding

8.5

dighost—looksuphostviaDNSwhoisdomain—looksupregistrationinformationfordomainifconfig—checksandmodifiesnetworkinterfacesettingsroutehost—displaysandmanipulatestheIProutingtabletraceroutehost—displaysnetworkroutetohost

FIREWALLSETTINGSA network firewall provides a line of defense against outside intrusion. Afirewallcanbeaspecialcomputeronthenetworktoprotectallinternalhosts.Ahome or small business router is such an example. A firewall can also be aprogram running on a host to protect that host. An operating system usuallycomeswithafirewallprogramandLinuxisnoexception.Evenbehindarouterfirewall, a host will usually run its own firewall program for host-specificprotection.Ingeneral,afirewallprotectsbylimitingnetworkingtoandfromtheoutside

byfollowingIPpacketfilteringrulesthatspecifypreciselywhatnetworktrafficisallowed inorout.Forexample, firewall rulescandisallowaccess tocertainservicesand/orportson thehost computeror subnethosts.Or theycan forbidnetworkingfromblacklistedhosts.Modern Linux systems come with a firewall daemon firewalld that allows

systemadmintomodifyfirewallsettingson-the-fly,withouthavingtorestartthefirewalld process or reboot the computer. Such a dynamic firewall makes itmuch easier for system admin than a static firewall based on iptables rules.Newer Linux distributions come with firewalld but not iptables. It isrecommendedthatyouusefirewalldinsteadofiptables.The GUI tool firewall-config (Figure 8.7) allows you to adjust firewall

settingsusedbyfirewalld.

8.6

Figure8.7FirewallConfigurationTool

The firewalldaemonhasanumberofpredefinedprotectionzones that offerdifferentlevelofprotectionfornetworkconnections.Figure8.7showstheeno1interface for pcenvy having the FedoraWorkstation as its default zone. Thefirewall-config tool allows you to modify the firewall runtime/permanentsettings:

ToassignconnectionsandinterfacestodesiredzonesTomodifysettingsandadd/deleterulesforeachzoneTocreatenewzonesofyourownTomakeruntimesettingspermanent

Apredefinedfirewallzone isagoodstartingpoint.Additionalpermissionscanbe added when needed and can be dropped when no longer necessary. Forexample, ifweneed the localWeb server tobecomeavailable,we can enablehttpandhttps services.To support incomingssh/sftp connections,wecanaddthesshservice.Tomake changes to the firewall from the command line use firewall-cmd.

Forexample,toenableHTTPSfortheWebserverwecanfirewall-cmd–add-service=https–permanentfirewall-cmd–reload

MANAGINGFILESYSTEMSANDDISKS

DiskPartitionsThe storage on a block device is usually divided into a number of separateregionsknownaspartitions,eachpartitioncanbeformattedandmanagedbytheoperatingsystem.Thus,wecanthinkofapartitionasa“logicaldisk.”OnLinuxeach partition is represented by its own special file. For example, /dev/sda1pointstopartition1onthediskdevice/dev/sda.Diskpartitionsaredefinedbyapartitiontable,storedatthebeginningofthe

storagedevice,thatliststhelocation,size,andotherattributesofeachpartition.The partition table followswell-defined formats—the oldMBR (Master BootRecord)orthenewGPT(GUIDpartitiontable).TheMBRisstoredattheverybeginning of amass storage device (sector 0 length=512 byte) and contains asimple boot loader and the partition table. MBR works with BIOS (BasicInput/OutputSystem)firmwareinterfaceatsystemboottime.GPT + UEFI (Unified Extensible Firmware Interface) is the new standard

replacingMBR+BIOS.GPTbetter supports larger storagedevices andUEFIprovides a special EFI System Partition (ESP) which contains programscompiledfor theEFIarchitecture.Suchfirmwareprogramscanbebootloadersfor different operating systems. See Section 8.8 for a discussion on systembooting.Youcanlistyourdiskpartitionswiththecommandlsblkwhichdisplaysblock

devices.Figure8.8showsasampledisplay.

Figure8.8lsblkDisplay

PartitionscanbemanagedwiththeCLItoolsfdisk(fordriveslessthan2TB),gdisk(forGPT),andparted(forMS-DOS,GPTandotherformats).TheGUItoolgparted(Figure8.9)makesitsimpletocreateandmodifydisk

partitions.

Figure8.9GpartedTool

Agoodandsimplewayofpartitioningaharddriveis:

Firstpartition—1GB,EFI(FAT)mountedat/boot/efiforbootingLinuxandperhapsalsootheroperatingsystems.Second partition—4GB,Linux filesystem (ext4)mounted at /boot for theLinuxkernelandrelatedfiles.Third(andlast)partition—Allremainingdiskspace,LinuxLVMphysicalvolumetobemanagedbyLVMforswap(36GB),/,/home,andsoon.

Alsousefulfordiskmanagementisthecommandgnome-disks(Figure8.10).

Figure8.10TheGnomeDisksTool

ManagingFilesystemsExisting filesystems that you can mount/umount are stored in the systemfilesystemtable/etc/fstab,firstcreatedwhentheLinuxdistributionisinstalled.

Hereisasample:

/dev/mapper/fedora-root/ext4defaults11UUID=91035a32-9e5a-4001-

851f-28d71244792d/bootext4defaults12UUID=62C3-23C9/boot/efi

vfatumask=0077,shortname=winnt02/dev/mapper/fedora-home/home

ext4defaults12/dev/mapper/fedora-swapswapswapdefaults00

Eachlinedescribesadifferentfilesystemwith6fieldsseparatedbyspaces—devicename,mountpoint,filesystemtype,mountoptions,andsoon.Pleaseseethemanpage for fstab fordetailsofeach field.Filesystems listed in /etc/fstabareautomaticallymountedatsystemboottime.Anadminusercanuseanyplaintexteditortoadd/deletefilesystemslistedin/etc/fstab.Thecommandfsckfile-sys...checks and optionally repairs the given filesystems.Without arguments, the

commandprocesseseachfilesystemlistedin/etc/fstabsequentially.

AddingaNewDiskOften,acomputerrunningLinuxhasonlyoneharddrive.Addingasecondharddrive can provide more storage as well as improve system performance.Furthermore,criticalfilescanbebackedupontheseconddrivejustincasethefirstdrivesomehowfails.Toaddanewharddrive,followthesegeneralsteps:

1. Physicallyinstalltheharddriveonyourcomputer.2. Createdesiredpartitionsonthenewdisk.3. Createfilesystemsforthepartitions.4. Mountthefilesystems.

Step 1 involves system shutdown, unplugging the computer, physicallyconnectingthenewharddrive,andreboot.Nowissueacommandsuchaslsblkorparted-ltoidentifythedevicespecialfileforthenewdisk.Ifthefirstdiskis/dev/sda,

thenewdiskisusually/dev/sdb.Withthenewdiskidentified,wecanperformstep2andpartitionthediskby

following information in Section 8.6.Use either theGUI gparted tool or theCLIparted tool.Eachpartitionbecomesalogicaldiskandhasitsownspecialdevicenamesuchas/dev/sdb1/dev/sdb2,andsoon.Tokeepthingssimple,itisnotabadideatohaveonebigpartitionfortheentirenewdisk.

Forstep3,wecanusethecommandmkfs-ttypedevice_namewhich creates a filesystem of the specific type for the partition given by

device_name.Type-specific filesystemcreationcommands includemke2fs (forext2/ext3/ext4 filesystem),mkswap,mkfs.fat, and others.After being createdthefilesystemisalsoidentifiedbydevice_name.Finallyinstep4,usethemountcommandtomounteachnewfilesystemtoa

desireddirectoryonthefiletree.Andenterthenewfilesystemsinto/etc/fstab.Alternatively, we can put the entire new disk under Logical Volume

Management.

LVMModernLinuxdistributionssupportLVM(LogicalVolumeManagement)whichusestheLinuxkernel’sdevicemappingabilitytodynamicallyassociatelogicalvolumes (LV) of storage to physical volumes (PV) on storage devices. Thus,LVMisasoftwarelayeroverdiskpartitionstoformlogicalvolumesthatmakemanagingharddrivereplacement,repartitioningandbackupmucheasier.UnderLVM,anumberofdiskpartitionsaredesignatedasPVs.ThePVsare

dividedintodisjointvolumegroups (VG).EachVGfunctionsasa logicaldiskandcanbe“partitioned”intooneormoreLVs.EachLVwillhaveitsownmountpointonthefiletree.Figure8.11showstheLVMarchitecture.

Figure8.11LVMArchitecture

Thecommandpvcreatedevice_nameinitializesadiskorpartitionasaPVtobeusedunderLVM.Weusedevice

namessuchas/dev/sda5asthenameofPVs.ThecommandpvdisplaydisplaysallPVsandtheirinformation.Tocreateavolumegroup(VG)usethe

command

vgcreatevolume_group_namepv1pv2...ToseeallVGsdovgdisplayToaddnewphysicalvolumestoavolumegroupusevgextendvg_namepv5pv6...TocreateanLV(logicalvolume)inaVGdolvcreate-ln-nlv_namevg_nameThe-loptiongivesthenumberofextents(eachextentis4MBbydefault).A

logicalvolumecanberesizedlaterwhenavailablespacechangesinitsvolumegroup (lvextend, lvreduce, lvresize). To see all LVs do lvdisplay. Here is asampledisplay.

---Logicalvolume---LVPath/dev/fedora/swapLVNameswapVGName

fedoraLVUUIDgJNdsi-osp2-1046-isc2-NXnT-nTEO-3V6awhLVWriteAccess

read/writeLVCreationhost,timepcenvy.localdomain,2017-03-21

11:23:04-0400LVStatusavailable#open2LVSize35.36GiBCurrent

LE9053Segments1AllocationinheritReadaheadsectorsauto-

currentlysetto256Blockdevice253:1---Logicalvolume---LVPath

/dev/fedora/homeLVNamehomeVGNamefedoraLVUUIDKJTOjw-TJnp-78oL-

xYF9-mChg-2yVx-8M4yonLVWriteAccessread/writeLVCreationhost,

timepcenvy.localdomain,2017-03-2111:23:04-0400LVStatus

available#open1LVSize266.00GiBCurrentLE68096Segments

1AllocationinheritReadaheadsectorsauto-currentlysetto

256Blockdevice253:2---Logicalvolume---LVPath

/dev/fedora/rootLVNamerootVGNamefedoraLVUUIDrjaDlE-wFPk-qbWo-

EbB6-XZwR-d7Ws-U5mUkILVWriteAccessread/writeLVCreationhost,

timepcenvy.localdomain,2017-03-2111:23:31-0400LVStatus

available#open1LVSize1.52TiBCurrentLE398458Segments

1AllocationinheritReadaheadsectorsauto-currentlysetto

256Blockdevice253:0

When installingLinuxonadesktopwith a singleharddrive,partition sizescanbeestimated roughlybecauseLVMallows filesystems tobe resizedeasilylaterbasedonsystemneedsandperformance.Onlargersystems,LVMmakesaddingandreplacingdisksmucheasier.Also,

LVM enables consistent backups by taking snapshots of the logical volumes.PleaseseeresourcesatthebookwebsiteformoreinformationandcommandsforLVM.

FileStorageQuotasThefilequotamechanismisdesignedtoallowrestrictionsondiskspaceusagefor individual users and/or groups. A separate quota can be set for eachuser/grouponeachfilesystem.Quotascanbeenforcedonsomefilesystemsand

8.7

notonothers.Forexample,inacomputersciencedepartment,onefilesystemforstudents may have quota enforced; at the same time, another filesystem forprofessors may have no quota enforced. The quota specifies limits on thenumber of files and disk blocks a user may occupy. There are two kinds oflimits: soft limits andhard limits. If a user-initiated operation causes the softlimit to be exceeded, awarning appears on the user’s terminal.The offendingoperationisallowedtocontinueifthehardlimitisnotexceeded.Theideaistoencourageuserstostaywithintheirsoftlimitsbetweenloginsessions.Inotherwords, exceeding the soft limit temporarily is all right, as long as the userreleasesfilespaceandreturnswithinthesoftlimitbeforelogout.Atlogintime,awarning is provided if any soft limits still are violated. After a few suchwarnings,theuser’ssoftlimitscanbeenforcedashardlimits.System admin can edit the filesystem table (/etc/fstab) to indicate which

filesystemsneedtosupportquotasanduse,forexample,quotacheck-cu/homequotacheck-cg/hometo enable, respectively, user and group quota enforcement for the /home

filesystem.The quotas for users and groups are kept in files (aquota.user and

aquota.group,forexample)locatedintherootdirectoryofthefilesystem.Foramountedfilesystem,itsrootdirectoryisitsmountpointonthefiletree.Thecommandedquota isused to set andchangequotas.Onlya superuser

can invoke edquota. The commandquota displays your disk usage and yourquotalimits.Asuperusercangivethiscommandanoptionaluseridtodisplaythe informationofa specificuser.Asuperuseralsocan turnonandoffquotaenforcingforentirefilesystemsusingthecommandsquotaonfilesys…quotaofffilesys…

FILEANDSYSTEMBACKUPForsafety,itisofcourseimportanttoregularlybackupkeyfilesonexternalharddrives,onanothercomputer,oronthecloud.The tar command (Chapter 6, Section 6.11) provides a convenient way to

gather files into one single archive file for backup. For example, to back up/home,thecommandtar-Jcpvf/backup_home.txz–one-file-system/home

creates an XZ-compressed file for /home. The option –one-file-systemexcludes files on a different filesystem. The compressed tar file can then betransferredtoanexternaldriveorsystem.Having backed up user home folders, you can backup the root folder (/) to

achieveafullersystembackupthisway(Ex:ex08/twopartBackup):

tar-Jcpvfbackup_root.txz--one-file-system--

exclude=/backup_root.tbz--exclude=/dev--exclude=/home--

exclude=/media--exclude=/mnt--exclude=/proc--exclude=/run--

exclude=/srv--exclude=/sys--exclude=/tmp/

IfyouhavenetworkaccesstoanotherLinuxsystemor,betteryet,adedicatedbackup server, then thersync command (Chapter7,Section7.6) can bemoreeffectiveandeasierfordoingbackups.Saypcmonkey isyourLinuxbackuphost andyouwish tobackuppctiger’s

Web data stored in /var/www. Assume you have userid ableAdmin on bothsystemsandyouhavearrangedno-passwordSSHfromonesystemtotheother.Then,youcanuse(Ex:ex08/rsyncBackup)rsync-Capz–rsh="ssh-lableAdmin"–hard-links–inplace/var/wwwpcmonkey:/backup/pctiger/www/The backup files will be located on pcmonkey in the folder

/backup/pctiger/www. The -C option conveniently excludes files normally notneeded for a backup. After the first backup, when the same command is runagainlater,rsyncwillonlytransferanyfilesthathavechangedsincelasttime.Notersynchas–includeand–excludeoptionstoaddandremovefiles/folders

from processing. The –delete option causes any file/folder not present in thesource to be deleted at the destination. See the manpage for rsync for manyotheroptions.To perform regularly scheduled automatic backups, place the preceding

commandasacrontabentrysuchas(Ex:ex08/rsyncCrontab):361***(followedbythedesiredcommandononeline)EachLinuxusercanusethecommandcrontab-etoeditthatuser’scrontable

ofscheduledcommands.The five leading fieldsofeachcrontabentry indicatetheschedule.Seesections1and5ofthemanpagesforcrontabfordetails.Thepreceding example specifies 01:36AM every day. The crond (cron daemon)examinescrontabentrieseveryminutetoexecutescheduledcommands.

BackupwithDéjàDupDéjàDup(day-ja-doop)fromGnomeisaGUIbackuptoolthatiseasytouseyetvery powerful. As a frontend toDuplicity, it features scheduling, encryption,

incrementalbackups,andsupportforcloudstorage.IfnotalreadyinyourLinuxdistribution,installitwithdnfinstalldeja-dupapt-getinstalldeja-dupYoucanfindtheDéjàDupicon(asafebox)intheApplications,Utilities,or

systemmenu.

Figure8.12DéjàDupBackupTool

Tousethistool,rundeja-dup-preferences(Figure8.12),turniton(enablingthetool),listallthedesiredfolderstobackup,anysubfolderstoignore,andthelocation to store the backup files. Then, click Back Up Now. You can alsoscheduleautomaticregularbackups.As backup storage, youmay use a local folder, a remote folder via FTP or

SSH,anexternalharddrive,oracloudservice(Figure8.13).

Figure8.13DéjàDupStorageChoices

If you have DropBox installed (Chapter 4, Section 4.1) you can use

8.8

$HOME/Dropbox as storage. But be careful, unless you have changed thedefault file sync settings for your DropBox account, any files deleted in$HOME/Dropboxwillbeautomaticallydeletedonthecloudaswell.Déjà Dup provides for scheduling of automatic backups. And you can run

deja-dup–backupmanuallyatanytime.Torestoreoneormorefilesusedeja-dup–restorefile1...Withnoarguments,acompleterestoreisdone.Torestoreamissingfolderdodeja-dup–restore-missingdirectory

SYSTEMBOOTINGUnderthecontrolofanoperatingsytem,acomputercanexecuteanyprograminitsRAM.Butwhenacomputerisfirstpoweredon,thereisnooperatingsystemyetinmemory.Thecomputer’sfirmwaremustfirstcausetheloadingofabootloader program whose job is to load in stages increasingly more capablesoftwareleadingtotheoperatingsystemkernel.ModernLinuxsystemsusetheUnifiedExtensibleFirmwareInterface(UEFI),

insteadofBIOS2,forthebootprocess.UEFIfeaturesitsownCPU-independentarchitecture,devicedrivers,andthe

abilitytomountpartitionsandreadcertainfilesystems.UEFIcanbeconsideredatinyOSrunningonacomputer’sfirmware.Whenacomputer ispoweredon, the firmware firstperforms thePowerOn

SelfTest(POST)thenrunstheUEFIcode.UEFIsearchesthesystemstorageforanEFISystemPartition(ESP)whichisaGPTpartitionlabeledwithaspecificGloballyUniqueIDentifier(GUID).TheusuallocationfortheESPis/boot/efi.The ESP contains applications compiled for the EFI architecture including

bootloadersandotherutilities.TheEFIcomeswithabootmanagerthatcanbootthe system from a default configuration or prompt the user to choose anoperatingsystemtoboot.WhenanOS-specificbootloaderisselected,manuallyorautomatically, it isreadintomemoryandtakesover thebootprocess.UEFIalso supportsSecureBoot that checks the signed software (Chapter 7,Section7.12) used in the booting process including the operating system so nothingunauthenticatedisusedorloaded.GRUB2isthelatestversionofGNUGRUB,theGRandUnifiedBootloader

which can be used for BIOS and for UEFI systems. On UEFI Linuxdistributions,theGRUB2bootloaderisusuallyfoundin/boot/efi/EFI.GRUB2canloadLinuxaswellasotheroperatingsystemssuchasWindows10.Atboot time,quicklypress akey suchasESCorSHIFT to enter theGRUB2

8.9

menuandchoosehowtoproceed.ModernLinuxdistributionsmap the legacy runlevels (0–6) to five systemd

runlevel targets, as part of many kinds of systemd targets. Each such targetrepresentsasoftwareunitinanorderedsequenceofunitsinthebootprocess.

poweroff.targetorrunlevel0.target—Systemshutdownrescue.targetorrunlevel1.target—Single-UserMode,nonetworkinterfaces,nodaemons,onlyrootloginallowedmulti-user.targetorrunlevel[2,3,4].target—Multi-UserMode,withnetworkanddaemons,noGUIgraphical.targetorrunlevel5.target—Multi-UserModeplusGUIreboot.targetorrunlevel6.target—Systemreboot

Themulti-user.target is normal for a Linux server box and graphical.target isnormalforaworkstation.Thecommandwho-rshows the runlevel.Tochange the runlevel (systemadminonly), the legacy

commandinitnumberstillworks.Butcommandssuchassystemctlisolatemulti-user.targetsystemctlisolategraphical.targetshouldbeusedinstead.Toseeorsetthedefaulttargetforthenextrebootusesystemctlget-defaultsystemctlset-defaulttarget_nameABASHscript(Ex:ex08/bootinfoscript)canbeobtainedandusedtodisplay

comprehensive boot-related information about disks, partitions, boot loaders,devices,filesystems,andsoon.

SELINUXInChapter6wetalkedaboutDiscretionaryAccessControl(DAC)(Section6.4).SELinux (Security Enhanced Linux) strengthens system security by providingMandatory Access Control (MAC) whose rules are applied after DAC.AccordingtoaRedHatdocument

“SELinux isan implementationofamandatoryaccesscontrolmechanism

in the Linux kernel, checking for allowed operations after standarddiscretionary access controls are checked. SELinux can enforce rules onfiles and processes in a Linux system, and on their actions, based ondefinedpolicies."

ThissectionprovidesabriefoverviewandintroductiontoSELinuxwhichisalargetopicrequiringentirebooksforafulltreatment.StandardLinuxusesDACthatcontrolsaccessbasedonuserid,groupid,and

rwx permissions set at users’ discretion.The approach has fundamental flaws.DAChasnowaytoaccountforfine-grainedusersecuritylevels,rolesusersandprocessesmayplay,andsensitivityclassificationsofdata.WithSELinux,built-insecuritypolicies,enforcedbytheLinuxkernel,govern

howallsubjects interactwith allobjects in the system.A subject is a runningprocess.Anobjectisafile,folder,device,port,socket,orprocess.Inlate2000,theUnitedStatesNationalSecurityAgency(NSA)releasedthe

firstversionofSELinux to theopen source softwaredevelopmentcommunity.With contributions frommany sources, includingRedHat andLinusTorvaldswhosuggestedamodularapproachforintroducingsecurityenforcementintotheLinuxkernel,SELinuxwas integrated into theLinuxSecurityModules (LSM)framework. The kernel can load different LSMs to implement well-definedsecurityschemes.NowSELinuxcanbeinstalledinmostmodernLinuxdistributionsincluding

Red Hat, Fedora, Ubuntu, Debian, and Hardened Gentoo. Some distributionsmayhaveSELinux turnedoff initially. In fact,Ubuntu,MintandotherspreferAppArmoroverSELinux.OurdiscussionherewillbemostlybasedonRedHat,CentOS,andFedora.

SELinuxStatusandEnforcingModesTocheckthestatusofSELinux,issuethecommandsestatusHereisasampledisplay

SELinuxstatus:enabledSELinuxfsmount:/sys/fs/selinuxSELinuxroot

directory:/etc/selinuxLoadedpolicyname:targetedCurrentmode:

enforcingModefromconfigfile:enforcingPolicyMLSstatus:

enabledPolicydeny_unknownstatus:allowedMaxkernelpolicy

version:31

SELinuxcanrunineitherofthesetwomodes

Enforcing—Allows/deniesaccessbyenforcingpolicyrules.

Permissive—Doesnot deny access but displays and logs anyviolationofpolicyrulesinstead.

Usesetenforce1 (or0) toset thecurrentmode toenforcing(permissive).Usegetenforcetodisplaythecurrentmode.Thecurrentmodepersistsuntilthenextreboot when it will be set according to the SELinux configuration file/etc/sysconfig/selinux.Hereisanexamplewherethetargetedpolicyisenforced.

#ThisfilecontrolsthestateofSELinuxonthesystem.#SELINUX=

cantakeoneofthesethreevalues:#enforcing-SELinuxsecurity

policyisenforced.#permissive-SELinuxprintswarningsinstead

ofenforcing.#disabled-NoSELinuxpolicyis

loaded.SELINUX=enforcing#SELINUXTYPE=cantakeoneofthesethree

values:#targeted-Targetedprocessesareprotected,#minimum-

Modifiedtargetedpolicy.Onlyselectedprocessesareprotected.#

mls-MultiLevelSecurityprotection.SELINUXTYPE=targeted

EditthisfiletomodifyhowSELinuxisappliedacrossreboots.Forexample,SELINUX=disableddisablesenforcementofSELinuxrules(notrecommended).Toswitchfromdisabled toenforcing(orenforcingfor thefirst time)set the

modetopermissiveintheconfigfilefirstandreboot.Seeingnowarningsafterreboot,changetheconfigurationfrompermissivetoenforcingandrebootagain.

SecurityContextsWith SELinux, each process/file is labeledwith its own security context. Thekernel uses context information and policy rules to make access controldecisions.Thesecuritycontextofaprocessisalsoknownasthedomainoftheprocess. Policy rules also govern whether a process can transition from onedomaintoanother.SELinuxallowsaccessonlyif there isaspecificpolicyrulepermittingsuch

access.Otherwiseaccessisalwaysdenied.A SELinux security context is an ordered list of security identities in the

formatse_user:role:type[:sensitivity:category]Thesensitivitylevelandcategoryrangepartsareoptional.Usually,theTargetedSELinuxPolicyisthedefaultandyoucanfinditstored

in /etc/selinux/targeted. The targeted policy focuses on type enforcementwithsomeattentionpaidtouserandroleidentities.Thesensitivityandcategoryarerarelyused.These_useridentityisusedtocollectusersintosecuritygroups.Forexample,

userswiththeguest_uidentityusuallywon’tbeabletorunexecutableslocatedin their own home directory or /tmp, or to run setuid programs. Use thecommandseinfo–usertoproducealistofallpossibleSELinuxuseridentities,suchas

sysadm_usystem_uxguest_urootguest_ustaff_uuser_uunconfined_u

Therole identitydifferentiatesbetweenfiles,executables,runningprocesses,and daemons. Only processes with certain roles are allowed to take certainactionsor transitiontocertaindomains.UndertheSELinuxtargetedpolicy,bydefault,usersareunconfined.Thus,thedomainofauser’sShellps-eZ|grepbashisnormallyunconfined_u:unconfined_r:unconfined_t:s0...bashThe type identities form thebulkofaccesscontrolunder the targetedpolicy

withalargenumberoftypesandrulesthatspecifywhichtypecanaccesswhichtype.Take the ApacheWeb server (Chapter 9) daemonhttpd, for example. The

commandls-Z/usr/sbin/httpdproducesthecontextforthatexecutablefilesystem_u:object_r:httpd_exec_t:s0/usr/sbin/httpdAndps-eZ|grephttpddisplaysthedomainforthedaemonprocesssystem_u:system_r:httpd_t:s0...httpdThus, a process able to execute /usr/sbin/httpd creates a child process

(Chapter3,Section3.2)thattransitionstotheprecedingdomain.ForafileintheWebserver’sdocumentspace(readablebyhttpd),ls-Zindex.htmldisplaysunconfined_u:object_r:httpd_sys_content_t:s0index.htmlForafolderorfilewritablebyscriptsrunbyhttpd,thecontextisunconfined_u:object_r:httpd_sys_rw_content_t:s0

MaintainingandManagingFileContextsOnSELinuxsystemsusingthetargetedpolicy,regularusersusuallywon’thavemuchinteractionwithSELinuxenforcementbecausethey,alongwiththerestofthe system, are unconfined_t. Only daemons are targeted and confined. This

usuallymeans,thestandardDACrules.However,therearefourcommonareasaregularuserneedstopayattentionon

SELinuxsystems:copying,moving,rsyncing,andarchiving/restoringfiles.Thegoal is todo thesewhilepreserving thecorrectSELinuxcontexts for filesanddirectories.Herearesometips.

Themv command, by default, preserves the file’s original contextwhichmaybe incorrect for itsnew location.Usemv -Z toautomatically set thecontextcorrectlyforthedestinationlocation.Whencreatinganewfile/folder,includingwiththecpcommand,itgetsacontextinheritedfromitsparentdirectoryautomatically.For the tar command, use the –selinux (–no-selinux) option to include(exclude)SELinuxcontextinformationinthearchive.For the rsync command, add the –xattrs option to preserve extendedattributesthatincludeSELinuxcontexts.

Sometimes a file/folder needs its context set correctly via the chcon (changecontext)command.Forexample,ifnecessary,ausermayuse(Ex:ex08/chcon)chcon-R-thttpd_user_content_t$HOME/public_htmltosetcorrectcontexts for theuser’spersonalWebdocumentspace(Chapter

9).SELinux uses auditd to logmessages in /var/log/audit/ to aid auditing and

troubleshooting. These are mostly AVCs (Access Vector Cache). They showoperationsdenied (or allowed)by theSELinux security server.AnadminusercancheckSELinuxwithausearch-mavctodisplayalldenials,oraddtheoption-tsrecent(-tstoday)fordenialsforthe

last10minutes(today).Admincanalsousethecommandsealert-btolaunchaGUIalertbrowser(Figure8.14).

Figure8.14SELinuxAlertBrowser

8.10

8.11

Toseethecorrectcontextsettings,issuethecommandsestatus-vtodisplaycorrectcontextsforfilesandprocesseslistedin/etc/sestatus.conf.Use the command restorecon or fixfiles to restore/fix contexts. Use the

commandsecon to display the context of a file or process.For example, (Ex:ex08/secon),secon-f/usr/sbin/httpdproduces

user:system_urole:object_rtype:httpd_exec_tsensitivity:

s0clearance:s0mls-range:s0

andthecommandsecon–pid‘pidofhttpd‘producessomethinglike

user:system_urole:system_rtype:httpd_tsensitivity:

s0clearance:s0mls-range:s0

Many other tools for SELinux exist for troubleshooting,managing policies,writingnewones,andsoon.Seethebookwebsiteforadditionalresources.

FORMOREINFORMATIONSeetheDNFpageatfedoraproject.org.SeeFirewallddocumentationatfedoraproject.org.SeetheAPTcompleteguideatitsfoss.com.SeeLVMHowToattheLinuxDocumentationProject.SeeSELinuxresourcesatthebookwebsite.SeeUEFIatuefi.org.

SUMMARYLinuxsystemadministrationandmanagementisapromisingcareerdirectionforwell-trainedLinuxpersonnel.BasicaspectsofLinuxsystemadmin,especiallyfor home and small LAN situations, include: user accounts, softwaremanagement, network configuration, disk and filesystem management,file/system backup, booting, and system security. SELinux adds MandatoryAccess Control (MAC) based on security policies to the standard LinuxDiscretionary Access Control (DAC) based on userid and file permissions.

8.12

UnderstandingandmanagingSELinuxcontextsareimportantforendusersandsystemadminsalike.

EXERCISES1. Where are userids and passwords for Linux users kept? How is the

passwordcheckedwhenauserlogin?2. WhatisLDAP?HowisitrelatedtouserauthenticationonLinuxsystems?3. Compareprosandconsforfilebackupwithtarandwithrsync.4. Explainthepurposeofsudoandwhyweshoulduseit insteadofloginas

root.5. Howcan tasksbescheduledforexecutiononaregularbasis?Pleasegive

examples.6. How to find the IP address assigned to your Linux system? What is a

defaultroute?7. ExplainthestructureofanIPpacket.8. WhatisDHCPreservation?Howisitconfiguredonarouter?9. HowdoesonefindtheIPv6addressofahostontheInternet?Pleasegivea

specificexample.10. How to find which daemon processes are running? What tools and

commandsareusefulformanagingdaemonprocesses?11. Whatisasubnet?FindthesubnetyourLinuxsystemisin.12. Whatisasubnetmask?13. Whatisahardwarefirewall?softwarefirewall?Wherearetheylocated?14. Whatisadiskpartition?AGPTpartition?15. WhatisBIOS?UEFI?EFI?Pleaseexplain.16. WhatisaGUID?WhatisanESP?Pleaseexplain.17. Listandexplainthetasksneededtoinstallanewharddrive.18. WhatisLVM?Whatbenefitsdoesitbring?19. UnderLVM,whatisaphysicalvolume?volumegroup?logicalvolume?20. FindoutwhatRAIDisandhowyourLinuxdistributionsupportsRAID.21. WhatisaSELinuxcontext?22. WhatisdomaintransitioninSELiux?23. Whereisthecontextforafilestored?24. ExplainwhyusingthemvcommandcancauseproblemsunderSELinux.25. FindoutaboutAppArmorandcompareitwithSELinux.26. ForstudentsinterestedinLinuxservermanagement,pleasefindoutabout

Ansible and other similar software packages that automate software

12

provisioning,configurationmanagement,andapplicationdeployment.Avirtualmachineisanotheroperatingsystemrunningunderthecontrolofahostoperatingsystem.MostUEFIimplementationsarebackwardscompatiblewithBIOS.

9.1

Chapter9

WebHosting:Apache,MySQL,andPHP

Startedintheearly1990sasafilesharingsystemamongphysicists,theWorldWide Web (WWW or simply Web) has grown rapidly to a globe-spanninginformation system that modern societies won’t do without even for a shortwhile. In a real sense, theWeb has leveled the playing field and empoweredindividualsandbusinesses,largeorsmall,allovertheworld.Akeyfactorforthisgreatsuccessisthelowcostofputtinginformationonthe

Web. You simply find a Web hosting service to position your files andprogrammingforyourwebsiteontheWeb.AnyInternethostcanprovideWebhosting if it has a good Internet connection and runs aWeb server and otherrelatedprograms.Accordingtonetcraft.com’sJune2017survey,amongallWebservers,afull

46% areApache, and amajority ofApache servers run on Linux systems.ALinux-ApacheWeb hosting environment usually also supports PHP for activepages and MySQL (the community version) or MariaDB for database-drivenwebsites.TheLinux,Apache,MySQL/MariaDB,andPHPcombination(knownas LAMP) works well to support Web hosting. An introduction to theseprograms,togetherwiththeirconfiguration,andoperationispresented.Inaddition tounderstanding thebigpictureand theunderlyingprinciples, a

practicalhands-onapproachguidesyou through the installation, configuration,testing,andadministrationofApache,PHP,andMySQLsoyoucanlearnLinuxWebhosting throughdoing.Root access onyourLinux is convenient, but notnecessary.

WHATISAWEBSERVER?AWeb server is a piece of software that runs on a particular host to supplydocuments to the Web. The host computer is called a server host and often

9.2

provides many network-based services including theWeb. Linux systems arewidelyused to runWebservers, and it is important forLinuxprogrammers tobecomefamiliarwithoperationsrelatedtotheWebserver.AWebserverlistenstoaspecificnetworkingportonthehostandfollowsthe

Hypertext Transfer Protocol to receive HTTP requests and send HTTPresponses.Thestandardportis80forHTTPand443forHTTPS.Butotherportsmaybeused.In response to an incoming request, a server may return a static document

from files stored on the server host, or itmay return a document dynamicallygeneratedbyaprogramindicatedbytherequest(Figure9.1).

Figure9.1WebServerOverview

A single-thread server handles one HTTP request at a time, while a multi-threadservercanhandlemultipleconcurrent requests.AserverhostmayhavemultiplecopiesofaWebserverrunningtoimprovethehandlingofrequests.ManydifferentbrandsofWebserversareavailablefromcompaniesandfrom

open-sourceorganizations.GlassFish is a freeWebserver thatcomeswith theJavaEEdistributionfromjava.sun.com.TheApacheWebserver,availablefreefrom theApache Software Foundation (apache.org), is widely used on Linuxsystems.ThepopularApacheusuallycomespre-installedonLinuxdistributions.

URLANDURIAnimportantcornerstoneoftheWebistheUniversalResourceLocator (URL,Chapter 7, Section 7.13) that allows Web clients to access diverse resourceslocatedanywhereontheWeb.Forexample,theHTTPURLhttp://ml.sofpower.comleads to the companionwebsite for this textbook.AnHTTPURL (Figure )

identifiesaWebserverrunningonaparticularhostcomputerandprovides thefollowinginformation:

9.3

AUniversalResourceIdentifier(URI)thatcorrespondstoalocalpathnameleadingtoatargetresource(afileorprogram)storedontheserverhostAnoptionalpathinfoindicatingatargetfile/folderlocationasinputdatatothetargetresourceAn optional query string providing key=value pairs as input data to thetargetresource

Figure9.2HTTPURLStructure

Thepartof theURLimmediatelyafter thehost:portsegment (Figure9.2) isreferredtoastheURI.TheWebserverusestheURItolocatethetargetresource,whichcanbea staticpage, anactivepage,or anexecutableprogram.AstaticpageisreturneddirectlyinanHTTPresponse.Anypathinfoandquerystringismade available, as input, to an active page or an executable program. TheresultingoutputisthenreturnedinanHTTPresponse.The set of files and directories made available on theWeb through aWeb

serverisknownasitsdocumentspace.Thedocumentroot is therootdirectoryfor the document space, and it corresponds to the URI /. In addition to thedocumentroothierarchy,therecanbeotherfilesanddirectoriesinthedocumentspace, for example, the /cgi-bin and the userid usually map to directoriesoutsidethedocumentroothierarchy.A Web server also works with other special directories (outside of its

documentspace)forserverconfiguration,passwords,tools,andlogs.AnURIisinterpreted relative to the document root, cgi-bin, or another directory, asappropriate.TheWebservercanenforceaccessrestrictions,specifiedintheWebserverconfigurationfiles,onanyfile/folderinthedocumentspace.

REQUESTPROCESSINGForeachincomingHTTPrequest,aWebserverexecutes thefollowingrequestprocessingcycle:

1. Acceptsclientconnection(viaTCP/IP;Chapter7,Section7.2)2. Processesrequest(fetchesandprocessespageorinvokesprogram)3. Sendsresponse4. Closesconnection(orkeepsitaliveunderHTTP1.1)

9.4

9.5

While processing a request, a busy website often will receive many newrequests. It isnormal tousemultiple servers (multiprocessing)and/ormultiplethreadswithinthesameserver(multithreading)tohandleconcurrentrequests.

RESPONSEANDCONTENTTYPESForeachincomingHTTPrequest,theWebserversendsbackanHTTPresponsecontaining the requested resource or an indication of error or some othercondition.An HTTP response has two parts: the headers and the body. The server

specifies the Content-Type header to indicate the media type of the responsebody. Standard MIME (Multipurpose Internet Mail Extensions) content types(Chapter6,Table6.3)areused.Themostcommoncontenttypeistext/html,butthere aremanyother types.For a static file, theWeb serveruses the filenameextension to infer its media type using a list often found in the file/etc/mime.types.Thelocationofthiscontenttypelistisconfigurable.Incaseofdynamiccontent,thosegeneratedbyserver-sideprograms,theWeb

serverreliesonthoseprogramstosetcontenttype.

THEAPACHEWEBSERVERApache is themostpopularWebserver,especiallyonLinuxsystems.Youcandownload and install the Apache HTTP server (Apache) from the ApacheSoftwareFoundation(httpd.apache.org)freeofcharge(Section9.6).However,yourLinuxwillmostlikelyhaveApachealreadyinstalled.Apache

isderivedfromtheNCSA1httpdprojectandevolved throughaseriesofcodepatches (thus, a patchy server). Apache, written in the C language, is opensourceandrunsonalmostallplatforms.Apacheisfast,reliable,multi-threaded,full-featured, andHTTP/1.1 compliant.AlthoughApache1.3 is still available,themostrecentstableApache2versionistheonetouse.Apachehasmanycomponents,including

Serverexecutable—TherunnableprogramhttpdUtilities—Forservercontrol,passwords,andadministrationFiles—Including server configuration files, log files, password files, andsourcecodefilesDynamic loadable modules—Pre-compiled library modules that can beloadedintothehttpdatrun-timeDocumentation

9.6 APACHEONLINUXBecauseofitsimportance,mostpopularLinuxdistributionscomewithApachealreadyinstalled.OtherwiseyoucaneasilyinstallApacheHTTPD.

InstallingApachewithPackageManagementWehavementioned thatmostLinuxdistributionscomewithApache installed.With root access, you can use the Linux package management (Chapter 8,Section8.2)commands

CentOS/Fedora:dnfinstallhttpddnfupdatehttpdUbuntu/Debian:sudo

apt-getinstallapache2sudoapt-getupdateapache2

to install/update your Apache server. See Chapter 8, Section 8.1 for adiscussionofthesudocommand.InstallingtheWebServergroupgivesmoresupportingprograms.

CentOS/Fedora:dnfgroupinstall'WebServer'dnfgroupupgrade'Web

Server'

IfyouwishtohavetheverylatestApacherelease,orifyoudon’thaverootaccess,youcaninstallApachemanuallyasdescribedinSection9.16.

RunningtheApacheWebServerNetworkingservers,theWebserverincluded,areautomaticallystartedasLinuxboots and stopped as Linux shuts down. To make sure, start system-config-servicesandlookforhttpdamongtheserviceentrieslisted(Section8.3).Alternatively, toenable/disable,start/stop,andrestartservices, thesystemctl

commandcanbeused.systemctlenablehttpd.servicesystemctldisablehttpd.servicesystemctlstarthttpd.servicesystemctlstophttpd.servicesystemctlrestarthttpd.serviceEnablingaservicewillstartitautomaticallyonsystemboot.Aservicecanbe

startedwithoutbeingenabled.You’llusuallyfindthedocumentrootat/var/www/html/andtheApachemain

configurationfileat/etc/httpd/conf/httpd.conf(CentOS/Fedora)/etc/apache2/apache2.conf(Ubuntu/Debian)

9.7

Often,themainconfigurationfilewillincludeothercomponentconfigurationfilessuchasphp.confandssl.conf.Tocheckifhttpd,oranyotherprocess,isrunning,youcanusepidofhttpdpidofprocess_Nameandseeifoneormoreprocessidsarefound.

ControllingtheApacheServerThe command apachectl (CentOS/Fedora) or apache2ctl (Ubuntu/Debian),usuallyfoundin/usr/sbin,canbeusedtocontrolthehttpdapachectlactionapache2ctlactionActionsofapachectl

Action Meaningstart Startshttpdifnotalreadyrunningstop Stopshttpdifrunningrestart Starts/restartshttpdgraceful Restartshttpd,respectingongoingHTTPrequestsconfigtestor-t ChecksthesyntaxofconfigurationfilesPossibleactionsarelistedinTable9.1.

APACHERUN-TIMECONFIGURATIONFeaturesandbehaviorsoftheApachehttpdcanbecontrolledbydirectiveskeptin configuration files. The main configuration file is usually httpd.conf (orapache2.conf). When httpd starts, it reads the configuration files first. Aftermakingchangestotheconfiguration,thehttpdneedstoberestartedbeforethenewconfigurationtakeseffect.UnlessyouhaveinstalledyourownApacheasanordinary user (Section 9.16), you’ll need root privilege tomodify theApacheconfigurationortorestartit.

ApacheConfigurationFileBasicsAnApacheconfigurationfile(httpd.conf,forexample)isatextfilethatcontainsconfigurationdirectives.Eachdirectiveisgivenonaseparatelinewhichcanbecontinuedtothenextlinebyacharacterattheendoftheline.Lines thatbeginwith thechar#arecommentsandare ignored.Acomment

must occupy the entire line. No end-of-line comments are allowed. There are

many different directives. Directive names are not case sensitive, but theirarguments often are. A directive applies globally unless it is placed in acontainerwhichlimitsitsscope.Wheninconflict,alocaldirectiveoverridesaglobaldirective.Themainconfigurationfileishttpd.conf,andothercomponentconfiguration

filesmayexistandareincludedbythemainfilewiththeIncludedirective.Forexample, onmanyLinux systems the configurationdirectory /etc/httpd/conf.d/stores component configuration files such as ssl.conf for SSL (secure socketlayer)tosupportHTTPS,andphp.confforPHP(Section9.17).ThedirectiveIncludeconf.d/*.confisusedtoincludeallsuchcomponentconfigurationfiles.To test your Apache configuration for syntax errors, use either one of the

followingcommands:apachectlconfigtesthttpd-tInadditiontothecentral(mainandcomponent)configurationfiles,thereare

alsoin-directoryconfigurationfilesknownasaccessfiles.Anaccessfile,oftennamed .htaccess, is placed in any Web-bound folder (your public_html, forexample) to provide configuration settings applicable for the file hierarchyrootedat thatparticular folder.Directives inanaccess fileoverride settings inthe central configuration files. The possibility of an access file and whatdirectives itmay contain are controlled by theAllowOverride directive in themain configuration file.The .htaccess files are especially useful for individualusers to configure their ownWeb spaces, usually the public_html under theirhomedirectories.

AboutConfigurationDirectivesConfiguration directives controlmany aspects of theApacheWeb server. Thehttpd.conf file has three main parts: Global Environment, main serverconfigurations, and virtual hosts configurations. Comments are provided foreach configuration directive to guide its usage. Apache has reasonable andpractical default settings for all the directives, making it easy to configure atypical server. Additional directives specify how loaded components work.Commonlyuseddirectivesinclude

Server properties: host identification (ServerName name), file locations(ServerRoot, DocumentRoot, ScriptAlias), network parameters (Listen[IP:]port),andresourcemanagement(StartServers,KeepAlive)Enablingoptional server features (Options) and in-directory configuration

overrides(AllowOverride)Accessrestrictionsanduserauthentication(Allow,Deny,Require,Satisfy,AuthName,AuthType,AuthFile)Contenthandling(AddHandler,AddType,AddOutputFilter)HTTP caching and content deflation (DeflateCompressionLevel,ExpiresActive,ExpiresByType,AddOutputFilterByTypeDEFLATE)Virtualhosts(NameVirtualHost)

Forexample,thedirectiveDirectoryIndexindex.htmlindex.phpsaysindex.html(orindex.php)isthedirectoryindexfilewhichisdisplayedif

the folder containing it is the target resourceof an incomingURI.Without anindexfile,alistingoffilenamesinthatfolderisgenerated(indexgeneration)fordisplay only if the Indexes option has been enabled. Otherwise, an error isreturned.

LoadingModulesApacheisamodularserver.Onlythemostbasicfunctionalitiesareincludedinthe core httpd. Many extended features are implemented as dynamicallyloadablemodules(.so)thatcanbeselectivelyloadedintothecoreserverwhenitstarts.Thisorganizationisveryefficientandflexible.Theloadablemodulesareplacedinthemodulesfolderundertheserverroot

directory,which is defined in themain configuration filewith the ServerRootdirective.Toloadacertainmodule,usethedirectiveLoadModulename_modulemodules/moduleFileName.soForexample,LoadModuledir_modulemodules/mod_dir.soloadsmoduledir)LoadModulephp5_modulemodules/libphp5.so(loadsmodulephp5)The dir module enables Apache to generate a directory listing. The php5

modulesupportsdynamicWebpagesusingthePHPscriptinglanguage(Section9.17).Configuration directives may be included conditionally, depending on the

presenceofaparticularmodule,byenclosingtheminan<IfModule>container.Forexample,

>IfModulemod_userdir.c<UserDirpublic_html>/IfModule<

saysifweareusingtheuserdirmodule, thentheWebfolderforeachLinuxuserispublic_html.

GlobalDirectivesTable9.2showssomemoredirectivesrelatingtohowtheApacheserverworksglobally (Ex: ex09/apacheGlobal.conf). The Alias and ScriptAlias directivesmapanincomingURItoadesignatedlocalfolder.

ContainerDirectivesConfiguration directives can be placed inside a container directive to subjectthem to certain conditions or to limit their scope of applicability to particulardirectories, files, locations(URLs),orhosts.Withoutbeing limited,adirectiveappliesglobally.ApacheGlobalDirectives

Forexample,

>IfModulemod_userdir.c<UserDirpublic_html>/IfModule<

enables the per-userWeb space (Ex: ex09/peruser.conf) and designates theuserfoldertobepublic_htmlonlyiftheuserdirmoduleisloaded.Also,considerthesetypicalsettings(Ex:ex09/docroot.conf)forthedocument

root/var/www/html:

>Directory"/var/www/html"<OptionsIndexesFollowSymLinks(1)Order

allow,deny(2)Allowfromall(3)AllowOverrideNone(4)>/Directory<

Within the directory /var/www/html, we allow index generation and thefollowing of symbolic links (line 1). The order to apply the access controldirectives is allow followed by deny (line 2), and access is allowed for allincomingrequests(line3+)unlessitisdeniedlater.

9.8

The AllowOverride (line 4) permits certain directives in .htaccess files. Itsarguments can be None, All, or a combination of the keywords Options,FileInfo, AuthConfig, and Limit. We’ll return to this topic when we discussaccesscontrolindetail(Section9.8).You’llalsofindthefollowingtypicalsetting(Ex:ex09/htprotect.conf)inyour

httpd.conf:

>Files~"^\.ht"<Orderallow,denyDenyfromall>/Files<

It denies Web access to any file whose name begins with .ht (Chapter 4,Section 4.4). This is good for security because files such as .htaccess arereadablebytheApacheWebserver,butwedon’twanttheircontentsexposedtovisitorsfromtheWeb.As<Directory>and<Files>workonthefilepathnamesonyourcomputer,

the<Location>containerworksonURIs.Wealsohave<DirectoryMatch>,<FileMatch>,and<LocationMatch>thatuseregularexpressionsasdefinedforegrep(Chapter4,Section4.4).

ACCESSCONTROLUNDERAPACHEWhatIsAccessControl?RunningaWebserveronyourLinuxsystemmeans thatyoucanmakecertainfiles and folders accessible from theWeb.However, you alsowant to controlhowsuchfilescanbeaccessedandbywhom.Tomakeafile/folderaccessiblefromtheWeb,youmustplaceitsomewhere

in the document space configured for your Web server. This usually meansplacingafile/folderunderthedocumentrootorinsideyourownpublic_htmlandalsomaking the file readable (the folder readable and executable) by theWebserverviachmoda+rfile(chmoda+rxfolder).Filesonyoursystemnotplacedundertheserverdocumentspaceornothavingtherightaccessmodes(Chapter6,Section)willnotbeaccessiblefromtheWeb.The Web server can be configured to further limit access. Access control

specifies who can access which part of a website with what HTTP requestmethods.Access control can be specified based on IP numbers, domains, andhosts,aswellaspasswords.Accessrestrictionscanbeappliedtotheentiresite,tospecificdirectories,ortoindividualfiles.Apache access control directives include Allow, Deny, Order, AuthName,

AuthType, AuthUserFile, AuthGroupFile, Require, Satisfy, < Limit > , and <

9.9

LimitExcept>.

AccessControlbyHostIfa file in theserverdocument spacehasnoaccesscontrol,access isgranted.The order directive specifies the order in which allow and deny controls areapplied.Forexample,orderallow,denyonlyaccessallowedbutnotdeniedarepermitted.Inthefollowing,ifaccessis

firstdeniedthenallowed,itisallowed.

orderdeny,allowdenyfromallallowfromhost1host2...

Onmonkey.cs.kent.edu,we have a set of pages reserved for use inside ourdepartmental local area network (LAN). They are placed under the folder/var/www/html/internal. Their access has the following restriction (Ex:ex09/folderprotect.conf):

>Location/internal<orderdeny,allowdenyfromallallowfrom

.cs.kent.edu>/Location<

Thus,onlyhostsinthe.cs.kent.edudomainareallowedtoaccessthelocation/internal.TheIPaddressofahostcanbeused.Forexample,allowfrom131.123grantsaccesstorequestsmadefromanyIPwiththeprefix131.123.Toenableuserstocontrolaccesstofilesandfoldersundertheirper-userWeb

space(public_html),youcanusesomethingsuchas(Ex:ex09/htaccess.conf)

>Directory/home/*/public_html<AllowOverrideAllOrder

allow,denyAllowfromall>/Directory<

inhttpd.conf.Thismeansuserscanplacetheirownaccesscontrolandotherdirectivesinthefile/public_html/.htaccess.

REQUIRINGPASSWORDSAllowingaccessonlyfromcertaindomainsorhostsisfine,butwestillneedawaytorestrictaccesstoregistereduserseitherforthewholesiteorforpartsofit. Each part of a site under its own password control is known as a securityrealm.Auserneedsthecorrectuseridandpasswordtologintoanyrealmbeforeaccessingthecontentsthereof.Thus,whenaccessingaresourceinsidearealm,ausermust firstbeauthenticated or verified as towho theuser is.TheApache

Web server supports two distinct HTTP authentication schemes: the BasicAuthentication and theDigestAuthentication. Some browsers lack support forDigest Authentication which is only somewhat more secure than BasicAuthentication.Let’slookathowtosetuserlogin.

SettingUpUserLoginunderApacheTo illustrate how to password protect Web files and folders, let’s look at aspecificexamplewherethelocation/WEB/csnotes/isafolderwewillprotect.Wefirstaddthefollowingauthenticationdirectivestothehttpd.conffile(Ex:

ex09/validuser.conf):

>Location"/WEB/csnotes/"<AuthName"WDP-1Notes"AuthType

BasicAuthUserFile/var/www/etc/wdp1prequirevalid-user>/Location<

TheAuthNamegivesanametotherealm.Therealmnameisdisplayedwhenrequestingtheusertologin.Thus,itisimportanttomaketherealmnameveryspecificso thatuserswillknowwhere theyare logging into.Figure9.3showssuchaloginpanelforaccessingtheexamplepackageonthisbook’swebsite.

Figure9.3HTTPBasicAuthenticationExample

TheAuthTypecanbeeitherBasicorDigest.TheAuthUserFilespecifiesthe

9.10

fullpathnameofafilecontainingregisteredusers.TheoptionalAuthGroupFilespecifiesthefullpathnameofafilecontaininggroupnamesandusersinthosegroups. The Require directive defines which registered users may access thisrealm.

valid-user(allusersintheAuthUserFile)userid1id2id3...(the

givenusers)groupgrp1grp2...(allusersinthegivengroups)

TheAuthUserFileliststheuseridandpasswordforeachregistereduser,withoneuserperline.Hereisasampleentryin/var/www/etc/wdp1.PWang:RkYf8U6S6nBqETheApacheutilityhtpasswd (htdigest)helpscreatepassword filesandadd

registeredusersfortheBasic(Digest)authenticationscheme.(Seethemanpagefortheseutilitiesforusage.)Forexample,htpasswd-c/var/www/etc/wdp1PWangcreates the file and adds an entry for user PWang, interactively asking for

PWang’spassword.Ifyouwishtosetupagroupfile,youcanfollowtheformatfor/etc/group,namely,eachlinelookslikegroup-name:userid1userid2...It isalsopossible tosetup loginfroman .htaccess file.Forexample,put in

.htaccessunderuserpwang’spublic_html

AuthUserFile/home/pwang/public_html/.htpasswordAuthName"Faculty

Club"AuthTypeBasicRequirevalid-user

Then,placein.htpasswordanyregisteredusers.If more than one Require and/or allow from conditions is specified for a

particular protected resource, then the satisfy any (if any condition ismet) orsatisfyall(allconditionsmustbemet)directiveisalsogiven.Forexample(Ex:ex09/flexibleprotect.conf),

>Location/internal<orderdeny,allowdenyfromallallowfrom

.cs.kent.eduAuthName"CSInternal"AuthTypeBasicAuthUserFile

/var/www/etc/csrequirevalid-usersatisfyany>/Location<

meansresourcesunderthe/internalcanbeaccessedbyanyrequestoriginatingfromthecs.kent.edudomain(nologinrequired)orausermustlogin.

HOWHTTPBASICAUTHENTICATIONWORKS

9.11

UponreceivinganunauthorizedresourcerequesttoarealmprotectedbyBasicAuthentication,theWebserverissuesachallenge:

HTTP/1.0401UnauthorizedWWW-Authenticate:Basicrealm="CS

Internal"

Upon receiving the challenge, the browser displays a login dialog boxrequestingtheuseridandpasswordforthegivenrealm.Seeingthelogindialog,the user enters the userid and password. The browser then sends the sameresourcerequestagainwiththeaddedauthorizationHTTPheader

Authorization:BasicQWxhZGRpbjpvcGVuIHNlc2FtZQ==

wherethebase64(Chapter7,Section7.10)encodedbasiccookiedecodes touserid:password. From this point on, the browser automatically includes thebasic cookiewith every subsequent request to the given realm. This behaviorpersistsuntilthebrowserinstanceisclosed.

HOWHTTPDIGESTAUTHENTICATIONWORKS

Unless conductedover a secure connection, such asSSL (secure socket layer)usedbyHTTPS(Section9.14),theBasicAuthenticationisnotverysecure.Theuseridandpasswordaresubject toeasyeavesdroppingoverHTTP.TheDigestAuthentication is an emerging HTTP standard to provide a somewhat moresecuremethodthanBasicAuthentication.WithDigestAuthentication,theserversendsachallenge(onasingleline)

HTTP/1.1401UnauthorizedWWW-Authenticate:Digestrealm="GoldClub"

nonce="3493u4987"

where the nonce is an arbitrary string generated by the server. Therecommended form of the nonce is anMD5 hash (Chapter 7, Section 7.12),which includes the client’s IP address, a timestamp, and a private key knownonlytotheserver.Uponreceivingthechallenge,thebrowsercomputes

str1=MD5(userid+password)str2=MD5(str1+nonce+

Resource_URI)

ThebrowserthensendstheauthorizationHTTPheader(ononeline)

9.12

9.13

Authorization:Digestrealm="GoldClub",

nonce="...",username="pwang",

uri="/www/gold/index.html",response="str2"

Theserververifiestheresponsebycomputingitusingthestoredpassword.From this point on, the browser includes the Digest Authentication header

witheveryrequesttothesamerealm.Theservermayelecttorechallengewithadifferentnonceatanytime.

Basicvs.DigestAuthenticationBasic Authentication is simple and works with all major browsers. DigestAuthenticationissomewhatmoresecure,butbrowsersupport is lesscomplete.Web servers, including Apache, tend to support both authentication schemes.When security is a concern, the best practice is to move from BasicAuthenticationoverHTTPdirectlytoBasicAuthenticationoverHTTPS(SecureHTTPoverSSL).

PASSWORDENCRYPTIONTheApache-suppliedhtpasswd tooluses thesameLinux/UNIXpassword/dataencryption method as implemented by the C library function crypt. In thisencryptionscheme,akeyisformedbytakingthelower7bitsofeachcharacterfromthepasswordtoforma56-bitquantity.Hence,onlythefirst8charactersofapasswordaresignificant.Also,a randomlyselected2-charactersalt fromthe64-character set [a-zA-Z0-9./] is used to perturb the standardDataEncryptionAlgorithm(DEA)in4096possibledifferentways.Thekeyandsaltareusedtorepeatedlyencryptaconstantstring,knownonlytothealgorithm,resultinginan11-character code. The salt is prepended to the code to form a 13-characterencryptedpasswordwhichissavedinthepasswordfileforregisteredusers.Theoriginalpasswordisneverstored.Whenverifyingapassword,thesaltisextractedfromtheencryptedpassword

and used in the preceding algorithm to see if the encrypted password isregenerated.Ifso,thepasswordiscorrect.

AUTOMATICFILEDEFLATIONApachetakesadvantageofmanyHTTP1.1featurestomakeWebpagesfastertodownload.Onesuchfeatureisautomaticcompressionofapagebeforenetworktransfer, resulting in significantly reduced file size and delivery time. This is

9.14

especially true for textual pages whose compression ratio can reach 85% ormore.Acompressedpageisuncompressedbyyourbrowserautomatically.Themod_deflatemodule for Apache 2.0 supports automatic (dynamic) file

compressionviatheHTTP1.1Content-EncodingandAccept-Encodingheaders.Thesetwoconfigurationdirectives(Ex:ex09/deflate.conf)

DeflateCompressionLevel6AddOutputFilterByTypeDEFLATEtext/html

text/plain\text/xmltext/cssapplication/x-javascript

\application/xhtml+xmlapplication/xslt+xml\application/xml

application/xml-dtdimage/svg+xml

indicate a list of content types for dynamic compression (using zlib) at theindicated compression level. Deflation adds a bit of processing load on theserversideandthehigherthecompressionlevel,theheaviertheprocessingload.CompressionwillonlytakeplacewhentheincomingHTTPrequestindicates

an acceptable compression encoding. The detection of browser compressionpreferencesandthesendingofcompressedoruncompresseddataareautomatic.Of course, any compressed outgoing page will carry an appropriate Content-Encodingresponseheader.TheAddOutputFilterByTypedirectiveneedsAllowOverrideFileInfotowork

in.htaccess.

HTTPSANDSSL/TLSWebservers supportHTTPS for securecommunicationbetween theclient andtheserver.

Figure9.4HTTPandHTTPS

HTTPS is HTTP (Hypertext Transfer Protocol) over Secure Socket Layer(SSL)orthenewerTransportLayerSecurity(TLS)protocol(Figure9.4).NoteHTTP and HTTPS use different server network ports, normally 80 and 443,respectively.SSL/TLSdevelopedfromSSL1.0,2.0,and3.0toTLS1.0,1.1,and1.2. SSL/TLS provides secure communication between client and server by

allowingmutual authentication, the use of digital signatures for integrity, anddataencryptionforconfidentiality.ToenableHTTPS,aserverneedstoinstallavalidWebservercertificate(Section7.9)andenableSSL/TLS.SSL/TLS may be placed between a reliable connection-oriented transport

protocollayer,suchasTCP/IP,andanapplicationprotocollayer,suchasHTTP(Figure9.5).

Figure9.5HTTPSProtocolLayers

Basically,TLSsetsupsecurecommunicationintwosteps:

1. Thehandshakephase—Mutualauthenticationandsecurelyagreeinguponarandomlygeneratedsessionkeytobeusedinthenextphase

2. The session data phase—Following the Record layer protocol, using thesession key for symmetric encryption (Section 7.8) of messages betweentheclientandserver

The handshake phase uses public-key cryptography (Section 7.9) for security,while the session data phase uses themore efficient symmetric encryption forspeed.EachnewSSL/TLSconnectionwillestablishanewsessionkey.Figure9.6illustratestheTLShandshakeprocessfromauserviewpoint.

9.15

Figure9.6BasicTLSHandshake

Allthisisabitoverwhelmingforbeginners.Don’tworry,wewilltalkaboutcryptography,digital signature,andall thatabit later in thischapter.But first,let’slookatthedigitalcertificate.

HTTPSSUPPORTFollow these three steps to setup SSL/TLS server certificate for AppacheHTTPD so that theWeb server will listen to port 443 and process incomingHTTPSrequests.

1. Make sure you have the necessary packages, openssl and mod_ssl,installed.

2. ObtainaservercertificatefromaCAsuchasDigiCertorEtrustandinstallthe encoded certificatemyserver.crt and private keymyserver.key in thedirectory/etc/pki/tls/certs/.

3. Modify the SSL configuration, /etc/httpd/conf.d/ssl.conf, for httpd asfollows:

DocumentRoot"/var/www/html"ServerName

mydomain:443SSLCertificateFile

/etc/pki/tls/certs/myserver.crtSSLCertificateKeyFile

/etc/pki/tls/certs/myserver.key

ThenrestarthttpdsystemctlrestarthttpdAlso make sure the firewall is not blocking https or port 443 (Chapter 8,

Section8.5).Forlocalortestingpurposes,youcangenerateaself-signedservercertificate

asfollows.GenerateanRSAprivatekey:

cd/etc/pki/tls/certs;makemyserver.keyopensslrsa-in

myserver.key-outmyserver.key

Filloutacertificatesigningrequest:makemyserver.csrSelfsignandgeneratethecertificate:

opensslx509-inmyserver.csr-outmyserver.crt\-req-signkey

myserver.key-days3650

9.16 MANUALINSTALLATIONOFAPACHEIfyouprefernot to installApachewithpackagemanagement,youmay installApache manually. The installation procedure follows the standard Linuxconfigure,make,installsequence.If you have root access, you will be able to install Apache in a system

directorysuchas /usr/localandassignport80 to it. Ifnot,youstill can installApacheforyourself(forexperimentation)inyourownhomedirectoryanduseanon-privilegedport,suchas8080.Let$DOWNLOADbethedownloadfolder,for example, either /usr/local/apache_src or $HOME/apache_src, and let$APACHE be the installation folder, for example, /usr/local/apache or$HOME/apache.TodownloadandunpacktheApacheHTTPserverdistribution, followthese

steps.

1. Download—Gotohttpd.apache.org/download.cgianddownloadthehttpd-version.tar.gz or the .tar.bz2 file, aswell as itsMD5 fingerprint file, intoyour$DOWNLOADfolder.

2. Integrity check—Use md5sum on the fingerprint file to check thedownloadedfile.

3. Unpack—From the $DOWNLOAD folder unpack with one of thesecommands.tarzxvpfhttpd-version.tar.gztarjxvpfhttpd-version.tar.bz2You’ll find a new Apache source folder, httpd- version, containing theunpackedfiles.

ConfigureandCompileNow you are ready to build and install the Apache Web server. Follow theINSTALL file and the Compiling and Installing section of the Apachedocumentationhttpd.apache.org/docs/version-number .You’llneedanANSICcompiler (gcc preferred) to compile, Perl 5 to make tools work, and DSO(DynamicSharedObject) support.These should already be in place on newerLinuxdistributions.FromtheApachesourcefolder,issuethecommand./configureoptionsto automatically generate the compilation and installation details for your

computer.TheINSTALLfilehasgoodinformationaboutconfiguration.Toseeallthepossibleoptions,givethecommand./configure–help.For example, the –prefix=serverRoot option specifies the pathname of the

serverrootfolder,andtheoption–enable-mods-shared=allelectstocompileallApachemodulesintodynamicallyloadablesharedlibraries.The recommended method (Ex: ex09/makeapache.bash) to configure and

compileis./configure–prefix=$APACHE–enable-mods-shared=allotherOptionsmakemakeinstallHere the Apache server root folder has been set to your installation folder

$APACHE as the destination for the results of the installation. TherecommendedotherOptionsare

--enable-cache--enable-disk-cache\--enable-mem-cache--enable-

proxy\--enable-proxy-http--enable-proxy-ftp\--enable-proxy-

connect--enable-so\--enable-cgi--enable-info\--enable-rewrite-

-enable-spelling\--enable-usertrack--enable-ssl\--enable-deflate

--enable-mime-magic

Eachoftheprecedingthreecommandswilltakeawhiletoruntocompletion.Aftersuccessfulinstallation,itistimetocustomizetheApacheconfiguration

file$APACHE/conf/httpd.conf.Followthesesteps:

1. ChecktheServerRootandDocumentRootsettings.Theseshouldbethefullpathnamesasgivenby$APACHEand$APACHE/htdocs,respectively.

2. Set the listening port: Listen 80 requires root privilege) Listen 8080 (noneedforrootprivilege)

3. Makeanyotherconfigurationadjustmentsasneeded.

NowyoucanstarttheApacheserverwith$APACHE/bin/apachectlstartIf the start is successful,youcan thenuseaWebbrowseron the samehost

computertovisithttp://localhost.localdomain:portandseetheApachewelcomepage,whichisthefile$APACHE/htdocs/index.htmlThen,testtheserverfromanotherhostonthesameLAN,withhttp://host:portwherehostisthedomainnameofyourserver.Makesurethatthefirewallon

the server allows both HTTP and HTTPS access (Section ). Otherwise, theApacheWeb server won’t be accessible from other hosts. On CentOS/Fedorafirewallconfigurationisanoptiononthesystem->adminmenu.ForUbuntuthe

9.17

gufwtoolishandyforthesamepurpose.It is recommended that you install PHP together with Apache. See Section

9.18fordetails.

WHATISPHP?PHP, a recursive acronym for PHP: Hypertext Preprocessor, represents apowerful and widely used program for generating dynamic Web content. ItevolvedfromanearlierprojectbyRasmusLerdorf,andPHP3.0wasreleasedinmid-1998. PHP has matured as an important server-side scripting tool and ismoving past version 7.2 at the time of thiswriting. In addition to serving theWeb, PHP can also be used as a Linux command for general-purpose textprocessing.AlthoughPHPrunsonmultipleplatforms,wewillfocusonPHPasanApache

servermoduleonLinux.Assuch,PHPexecutesaspartofApacheandinterpretscodeembeddedinWeb-boundpages todynamicallygeneratecontentfor thosepages.Forexample,anHTMLdocumentcontaining

>p<Itis>?phpecho(date("lM.d,Y"));?<,>br/<doyouknowwhere

yourprojectis?>/p<

generatesthetextItisThursdayJune.18,2018,doyouknowwhereyourprojectis?Thedatedisplayeddependsontheexacttimeofaccess.AnyPHPcodeisgivenwithinthePHPbracket<?php...?>andinterleaved

(embedded)withinnormalHTMLcode,orothertypesofcodeasthecasemaybe.Pagescontainingsuchembeddedcodesareoftencalledactive(ordynamic)pages,because theyarenotstaticandcontain informationgeneratedon theflybytheembeddedcode.Theembeddedcodeisneverseenbythereceiveroftheresultingdocument;itgetsreplacedbyanyinformationitgenerates(Figure9.7).

9.18

Figure9.7PHPCodeInterpretation

THEPHPMODULEFORAPACHEAnApacheserverisgenerallyexpectedtosupportPHP,anditisnothardtoaddthePHPmoduleforApache.WiththePHPmodule,theApacheWebserverwillbe able to interpretPHPcodes embedded in textual documents of any type astheyarebeingdeliveredtotheWeb(Figure9.7).MostLinuxdistributionswillhaveApache installedwithPHP already. For example, youmay find thePHPmodule libphp*.so already in the Apache modules folder (usually/etc/httpd/modules).You can also use the Linux package management facility to install/update

Apache+PHP:

dnfinstallhttpdphpphp-common(CentOS/Fedora)dnfupgradehttpd

phpphp-common(CentOS/Fedora)sudoapt-getinstallapache2php7.0

\php7.0-mysqllibapache2-mod-php7.0(Ubuntu/Debian)sudoapt-get

updateapache2php7.0\php7.0-mysqllibapache2-mod-php7.0

(Ubuntu/Debian)

InstallingthePHPModuleThis section describes how to install the PHPmodulemanually and add it toyourApacheserver.IfyoualreadyhaveApache+PHPinstalled,pleaseskipthissection.First,download thecurrentPHPrelease (php-version.tar.gzor .tar.bz2) from

www.php.net/downloads.php,checktheMD5fingerprint,andunpackintoyour$DOWNLOADfolderasbefore(Section9.16).Next, go to the PHP source code folder $DOWNLOAD/php-version to

configurethePHPmodule.Forexample(Ex:ex09/makephp.bash),

dnfinstallhttpdphpphp-common(CentOS/Fedora)dnfupgradehttpd

9.19

phpphp-common(CentOS/Fedora)sudoapt-getinstallapache2php7.0

\php7.0-mysqllibapache2-mod-php7.0(Ubuntu/Debian)sudoapt-get

updateapache2php7.0\php7.0-mysqllibapache2-mod-php7.0

(Ubuntu/Debian)

Thenchecktheconf.outputtoseeifyougettheselines:checkingiflibtoolsupportssharedlibraries...yescheckingwhethertobuildsharedlibraries...yescheckingwhethertobuildstaticlibraries...noIfyouneedtoredotheconfigurationstep,pleasefirstcleanthingsupwithmakedistcleanAftersuccessfulconfiguration,youarereadytocreatethePHPmodule.Enter

thecommandmakeItwilltakeawhile.Afteritisdoneyoushouldcheckthe.libs/foldertoseeif

thePHPmodulelibphp7.sohasbeencreated.Ifso,thenissuethecommandmakeinstallTheinstalldirectoryis$APACHE/phpasspecifiedbythe–prefixoption.The

install process also moves libphp7.so to the folder $APACHE/modules/ andmodifies$APACHE/conf/httpd.confforthehttpdtoloadthePHPmodulewhenitstartsbyaddingtheApacheconfigurationdirectiveLoadModulephp7_modulemodules/libphp7.soInaddition,youalsoneed toadda fewotherdirectives to tellApachewhat

filesneedPHPprocessing:AddHandlerapplication/x-httpd-ea-php70.html.htm.phpDirectoryIndexindex.phpindex.htmlAsstated,anytimeachangeismadetotheconfiguration,youneedtorestart

Apache(Section)inordertogetthenewconfigurationtotakeeffect.

TESTINGPHPTo test your Apache+PHP installation, you can create the page info.php (Ex:ex09/info.php)

>html<>head<>title<phpinfo>/title<>/head<>body<>?phpphpinfo();?

<>/body<>/html<

andplaceitunderthedocumentrootfolder.Then,visithttp://localhost.localdomain/info.phpfromyourWebbrowser.Thephpinfo()functiongeneratesapageofdetailed

information about your PHP installation, including version number, modules

9.20

loaded,configurationsettings,andsoon.As Apache starts, it loads the PHP module and also any PHP-specific

configuration ina fileusuallynamedphp.ini.The locationof this file (usually/etc/php.ini)isgivenastheLoadedConfigurationFileinthephpinfo()generateddisplay.

PHPCONFIGURATIONTheconfiguration file (php.ini) is readwhen thePHPmodule is loadedas theWeb server (httpd) starts. Any changesmade to php.ini will only take effectafterApacheisrestarted(Section9.9).PHP has toggle (on/off) and value configuration directives. You edit the

php.ini,whichcontainsasetofreasonabledefaults,tomakeanyadjustments.Forexample, ifyouare runningaWebdevelopmentsitewhereseeingerror

messages will help debugging PHP scripts, then you would set (Ex:ex09/php.ini)

;;;;EnableserrordisplayoutputfromPHPdisplay_errors=

Ondisplay_startup_errors=On

ForaproductionWebserver,youwoulddefinitelywanttochangetheseto

display_errors=Offdisplay_startup_errors=Offlog_errors=On;;;;

Enablesallerror,warning,andinfomsgreportingerror_reporting=

E_ALL;;;;Sendsmsgstologfileerror_log=>pathnameofa

designatederror.txtfile<

PHP also allows you to open any local or remoteURL for generating pagecontent.However,ifyoursitehasnoneedforopeningremoteURLsfromPHP,youmayincreasesecuritybysetting

allow_url_fopen=Off

PHPalsohasverygoodsupportforHTTPfileuploading.Ifyouwishtoallowthat,thenuse

file_uploads=On;;;;Usesomereasonablesize

limitupload_max_filesize=2M

PHP extensions provide optional features for many different purposes. Forexample, the gd extension supports manipulation of fonts and graphics fromPHP, and themysql extension provides a PHP interface toMySQL databases.DynamicallyloadableextensionsarecollectedinaPHPmodulesfolder(usually

9.21

9.22

/usr/lib/php/modules), but are set in the php.ini by the extension_dir directive.OnmanyLinuxsystems,theextensionsareloadedbydefaultthroughextension-specific .ini files in the folder /etc/php.d/. By editing these files you controlwhichextensionsareloadedwhenApache+PHPstarts.Toexamine the settingof allPHPconfigurationsdirectives,youcan simply

lookatthephpinfo()display(Section9.19).

PHPCOMMANDLINESCRIPTINGPHPcanbeusedfromthecommandline.ThisisusefulfortestingPHPcodeandfor taking advantageof thepowerofPHP towrite command-line scripts.YoucancreateaPHPexecutabletextfileasfollows

#!/usr/bin/php>?phpPHPcodelines....?<

thenyoucaninvokethescriptfromthecommandlinejustlikeaBASHscript.As an example, let’s write a PHP script (Ex: 09/echoback.php) which is aversionofechoback.sh(Chapter5,Section5.11).

#!/usr/bin/php>?php$output="\n";array_shift($argv);//loses

$argv[0]thecommandnameforeach($argvas$arg){$output="$arg"

.$output;}echo$output;?<

Wecanseethatcommand-lineargumentsarestoredinthePHPspecialarray$argv.Executethisscriptwithanyofthesecommandsphpechoback.phpABCDEphp-fechoback.phpABCDE./echoback.phpABCDEHere is an alternative implementation (Ex: 09/echoback2.php) using a PHP

forloop.

#!/usr/bin/php>?phpfor($n=$argc-1;$n<0;$n--){echo$argv[$n];

echo"";}echo"\n";?<

Youcanalsopassphpcodedirectlytophponthecommandline.

php-r'print_r(phpversion());echo"\n";'php-r

'print_r(phpinfo());echo"\n";'

DATABASESUPPORTFORTHEWEBAcomputerdatabase isasystemforconvenientlystoring, retrieving,updating,

9.23

and inquiring information for concurrent access by many users. Moderndatabasesarerelational;informationisstoredinmultipletables(Figure9.8)thatareinterrelated.

Figure9.8TheEMPLOYEESTable

A database system is SQL-compliant if it supports the Structured QueryLanguagestandardAPI(ApplicationProgrammingInterface).Forexample,thefollowing SQL SELECT query retrieves all rows from table EMPLOYEESwherethefieldLASTisWang:SELECT*FROMEMPLOYEESWHERELAST="Wang";Programs written in SQL can access and manipulate any SQL-compliant

database. Databases can be used for decision support, online transactionprocessing, personnel records, inventory control, user accounts, multi-useronlinesystems,andmanyotherpurposes.Adatabasecanalsomakewebsiteseasiertoconstruct,maintain,andupdate.

Ontheotherhand, theWebcanmakedatabasesaccessiblefromanycomputerconnectedtotheInternet.PHPprovidesexcellentsupportforusingdatabasesforandfromtheWeb.The

SQLiteextensionofPHPisafastSQLinterfacetoaflatfiledatabasethatcomeswithPHP(version5orlater).FormanysimpleWebapplications,SQLiteisjusttherightsolution.

MYSQLMore complicated websites with larger data loads will need heavier dutydatabasesystemsthanSQLite.Forthat,thefreeMySQLorMariaDBisoftentheright choice, especially incombinationwithLinuxandPHPbecausePHPalsohasexcellentbuilt-insupportforconnectingandqueryingMySQLandMariaDBdatabases.WewillfocusonMySQLbutMariaDBisentirelysimilar.MySQL is a freely available open-source relational database management

system that supportsSQL. It runsonLinux,MSWindows®,MacOSX®, and

other systems and can be used frommany programming languages, includingC/C++, Eiffel, Java, Perl, PHP, Python, and Tcl. TheMySQL database serversupports both local and network access. It supports a privilege and passwordsystemtospecifywhocanaccess/modifywhatinthedatabasesystem.MostLinuxdistributions comewithMySQL installed. If youcan locate the

commandmysql(oftenin/usr/bin)onyoursystem,then,mostlikely,youhaveMySQLalready.Tobesurelookformysqldbystartingsystem-config-servicesorbythecommandsystemctlstatusmysqld.Ifnot,orifyouwishtoinstallthelatestversionofMySQL,pleaserefertoSection1.25.

Initializing,Starting,andStoppingMySQLMySQL uses adefault database namedmysql for its own purposes, such asrecording registered users (userid and password), managing databases, andcontrolling access privileges. The commandmysql_install_db (in usr/bin/) isrun once to initialize the MySQL default database (usually located in/var/lib/mysql/mysql/) and is done automatically when the MySQL servermysqld isstartedfor theveryfirst time.Themysql_install_db scriptcontainsmanyinitializationsettingsforMySQL,andadjustingthesesettingsallowsyoutocustomizevariousaspectsofMySQL.Startingmysqldcanbedonewiththesystem-config-servicesGUItoolorthe

commandsystemctlstartmysqldThe same GUI and command-line tools can be used to stop/restart the

mysqld.Withmysqld started, MySQL client programs can communicate with it to

access/manipulatedatabasesservedbyit(Figure9.9).

Figure9.9MySQLServerandClients

MySQLRun-TimeConfigurationAsmysqld (thedatabase server) starts, it readsconfigurationvalues inmy.cnf

(usually kept in /etc or /etc/mysql). Specified are the data folder, the socket(Chapter12,Section12.6) location, the userid ofmysqld, and possiblymanyother settings. Edit my.cnf, and delete the line bind-address = 127.0.0.1, ifpresent,whichrestrictstheMySQLservertoaccessfromlocalhostonly.Itisalsorecommendedthatyouconsiderrunningalocal-access-onlyMySQL

serverratherthanonethatisnetworkenabled.ThelatterallowsMySQLclientsto access the server via a network which can mean security problems. Theformerwill limit access toMySQL clients on the same host,making itmuchmoresecure.Todothis,addtheconfigurationsettingskip-networkingtoboth the [mysqld] and the [mysqld_safe] sections inmy.cnf.Youneed to

restartmysqld after making changes to the configurations. See the MySQLdocumentationfordetailsaboutMySQLconfiguration.ItisagoodideatoruntheLinuxcommandmysql_secure_installationtoimprovethesecurityofyourMySQLinstallation.Afterstartingmysqld,youcanusenetstat,acommandtodisplaynetworking

statusandactivityonyoursystem,todoublecheck.Runthecommandnetstat-tap|grepmysqldIfyouseeadisplay,itmeansmysqldisallowingnetworkaccess.Ifyousee

no display, then only local clients are allowed access. The -tap option tellsnetstat to display all information related to TCP with names of programsinvolved.

AdministeringMySQLMySQLprotectsdatabasesbyrequiringauseridandpassword,and,dependingon what privileges the user has, various operations/accesses are allowed ordenied.Atthebeginning,MySQLhasanadministrator(root)andablankpassword.

Theveryfirstadministrativetaskistosetapasswordforroot.2mysqladmin-urootpasswordnew_passwordThe option -u specifies theMySQL userid root and the admin operation is

passwordsetting.Makesureyousavethepasswordforfutureuse.Let’sassumetherootpasswordisfoobar.TheMySQLrootistheuserwhocancreatenewdatabases,addusers,andset

privilegesforthem.mysqladmin-hlocalhost-uroot-pfoobarcreatelxuxtakes the hostname, userid, and password information and creates a new

databaselxux.Nowwecanaddpwangasauserwithallprivilegestouselxux.Onewayisto

use themysql toolwhich isacommand-line interface to theMySQLdatabaseserver.Givethecommandmysql-hlocalhost-uroot-pfoobarlxuxthenyouareworkingwithinmysql,andyoumayenterSQLqueries.Dothe

following(Ex:ex09/adduser.sql):

mysql<USEmysql;(settingdatabasenametomysql)mysql<SHOW

TABLES;(listingnamesoftables)+-----------------+|

Tables_in_mysql|+-----------------+|columns_priv||db||func||

host||tables_priv||user|+-----------------+mysql<INSERTINTO

user(Host,User,Password,Select_priv)-<VALUES('','pwang',

password('thePassword'),'Y');mysql<FLUSHPRIVILEGES;mysql<GRANT

ALLPRIVILEGESONlxux.*TOpwang-<IDENTIFIEDBY

'thePassword';mysql<FLUSHPRIVILEGES;mysql<quit

Theninformuserpwangabouthisuserid,password,anddatabasename.SeetheMySQLdocumentation formore informationonsettinguserprivileges.ToresetthepasswordforpwangusetheSQL

mysql<USEmysql;mysql<updateuserset

Password=PASSWORD('newOne')-<WHEREUser='pwang';

BecausePHPisoftenavailableonthesamehost, thefreephpMyAdmin tool(phpmyadmin.net) isoftenalso installed toenableMySQLadministrationovertheWeb.PhpMyAdmin(Section9.24)supportsawiderangeofoperationswithMySQL.ThemostfrequentlyusedoperationsaresupportedbytheWebbrowsersupplied GUI (managing databases, tables, fields, relations, indexes, users,permissions, and so on). Other operations are always doable via direct SQLstatements. Both the root user and any user for a specific database can dodatabaseadministrationthroughphpMyAdminfromanywhereontheWeb.

ResettingtheMySQLRootPasswordIt is important to not forget theMySQL root password.However, if you findyourself in such a situation, you can reset it. As Linux root, first stop themysqld:systemctlstopmysqldThenrunmysqldinsafemodewithoutsecuritychecking:/usr/bin/mysqld_safe–skip-grant-tables&Thenrunmysqlonthedefaultdatabasemysql:mysql-urootmysql

9.24

Thenupdatethepasswordforroot:

mysql<updateusersetPassword=PASSWORD('anything')-<WHERE

User='root';QueryOK,2rowsaffected(0.04sec)Rowsmatched:2

Changed:2Warnings:0mysql<flushprivileges;exit;

Nowkillthemysqld_safeprocessandrestartthemysqld.

INSTALLINGPHPMYADMINFirst,downloadthelatestversionfromphpmyadmin.netandunpackinyourWebdocument root folder (usually /var/www/html). For example(Ex:ex09/myadmin.install),cd/var/www/htmltarjxvpfphpMyAdmin-4.8.0-english.bz2rmphpMyAdmin-4.8.0-english.bz2mvphpMyAdmin-4.8.0-englishphpMyAdminTheresultingphpMyAdminfolder isnowinplaceunder theWebdocument

rootandyoucandisplayinstallationinstructionsandotherdocumentationwiththeURLhttp://localhost.localdomain/phpMyAdmin/Documentation.htmlTo install phpMyAdmin, you only need to do a few things. In the

phpMyAdmin folder create a configuration file config.inc.php by copying andeditingthesamplefileconfig.sample.inc.php.Itisrecommendedthatyoupickthecookieauthenticationmethodandsetupa

control user, as indicated by the sample configuration file, on your MySQLserver so anyone who has a MySQL login can use phpMyAdmin to managedatabasesaccessibletothatparticularuser.SeethephpMyAdmindocumentationforconfigurationdetails.Afterinstallation,theURLhttp://host/phpMyAdminreaches the on-Web MySQL admin tool for any valid user to manage the

databaseserver.(Figure9.10).

9.25

Figure9.10phpMyAdminTool

MariaDBcanalsousephpMyAdmin.Be sure to install the latestversionofphpMyAdmin.

INSTALLINGMYSQL/MARIADBMySQL/Mariadb may already come with your Linux distribution. If not, theLinuxpackagemanagementsystemmakesinstallationeasy.ForCentOS/Fedora,doasrootdnfinstallmysqlmysql-serverdnfupgrademysqlmysql-serverordnfinstallmariadbmariadb-serverdnfupgrademariadbmariadb-serverForUbuntu/Debian,dooneofsudoapt-getinstallmysql-serversudoapt-getupdatemysql-serverorsudoapt-getinstallmariadb-serversudoapt-getupdatemariadb-serverNow proceed to edit the my.cnf file (Section 9.23) and then start/restart

mysqld,theserverdaemonofMySQLorMariaDB(Section9.23).IfyouwishtoinstallApache+PHP+MySQL/MariaDBtoachieveLAMPallat

once,usethesecommands.CentOS/Fedora:

9.26

9.27

dnfinstallhttpdphpphp-commonmysql-servermysqldnfinstallhttpdphpphp-commonmariadb-servermariadbUbuntu:sudoapt-getinstalltaskselsudotaskselinstalllamp-serverRemembertheseinstallationsareveryniceasdevelopmentalsystems,butnot

secure enough as production systems. Enterprise editions of Linux will mostlikely includeaproductionWebserverwithLAMPandmore.Whatyou learnherewillapplydirectlytosuchproductionservers.Refer to dev.mysql.com/downloads/ at the MySQL site for manual

installation.

FORMOREINFORMATIONComplete information for the Apache Web server can be found athttpd.apache.org/.ThelatestreleasesanddocumentationforPHPareatphp.net/index.php.Thesitewww.mysql.comcontainsallcurrentreleasesandotherinformationforMySQL.See themariadb.orgwebsite for all information about theMariaDBopensourcesoftware.ThereisalsoasiteforbuildingLAMPserversatwww.lamphowto.com.There aremany textbooks on website development and design.DynamicWebProgrammingandHTML5,byPaulS.Wang,isagoodread.

SUMMARYAWeb server followsHTTP to receive requests and send responses. ItsmainfunctionistomapincomingURIstofilesandprogramsinthedocumentspacedesignatedfortheWeb.TheApachehttpdWebserversupportsdynamicmoduleloadingandrun-time

configuration,making it very easy to customize and fit the requirements of awiderangeofWebhostingoperations.Configurationdirectivescanbeplacedincentral files and in access files under individual folders within the documentspace.In addition to controlling features and behaviors of httpd, Apache

configurationscanspecifyaccesslimitationstopartsofthedocumentspaceandcanrequireloginwithHTTPBasicorDigestAuthentication.

9.28

PHPisapopularactivepagelanguagethatcangeneratedynamicWebpages.PHPscriptsareembedded in textual fileswithinanynumberof<?php ...? >brackets.PHPcanbeinstalledasanApachemoduleandwillinterpretembeddedPHP scripts as the Apache httpd delivers a response page. PHP can bedynamicallyconfiguredviathephp.inifile.PHP supplies a wide range of capabilities for the Web, including file

inclusion, form processing, local/remote file operations, file uploading, imageprocessing, sessioncontrol, cookie support, anddatabaseaccess.PHPcanalsobeusedasaCLItool.PHPhasabuilt-inlightweightdatabase,butalsoworkswellwiththeheavy-

dutyMySQLandMariaDBdatabase systems.Both supportmultipledatabasesprotectedbyuseridandpassword.Differentdatabaseusersmayhavedifferentaccess privileges and can be managed easily using Linux commands(mysqladmin,mysql,mariadbandsoon)ortheWeb-basedphpMyAdmintool.ThecombinationLinux,Apache,MySQL/MariaDB,andPHP(LAMP)forms

apopularandpowerfulWebhostingenvironment.The freelyavailableLAMPmakes a great developmental system, but should not be used as part of aproductionWebserverforsecurityreasons.

EXERCISES1. Findoutabouttheconfigurationfile/etc/nsswitch.conf.2. AssumingyourLinuxisrunningtheApacheWebserver,findtheversionof

Apacheserver,thehttpd.conffile,andthedocumentrootfolder.3. HowdoesonegoaboutfindingoutifyourLinuxsystemsupportsper-user

Webspace?4. Install your own Apache server with PHP support under your home

directory(Hint:useanon-privilegedport).Afterinstallation,startyourownhttpdandtestit.

5. HowdoesonefindoutifyourApachehasPHPsupport?Ifso,whereisthefilephp.iniandforwhatpurpose?

6. SetupyourApachetoautomaticallydeflate.html,.css,and.jsfiles.7. Install a server SSL certificate and test your HTTPD server for HTTPS

support.8. Lookatyourphp.iniandfigureouthowtoenable/disablePHPerroroutput.9. WriteaPHPscriptandtestitfromthecommandline.10. ConfigureyourApachetorequireapasswordonsomeWebfolder.Create

somevalidusersandtestyoursettingtomakesurethatitworks.

12

11. SetupsomedatabasetablesusingthePHPbuilt-inSQLite.TestyoursetupwithPHPcodeinaWebpage.

12. Install your ownMySQL under your home directory. You’ll be the rootdatabaseuser.Createanewtestdatabaseandsometablesusingthemysqltool.

13. InstallthephpMyAdmintool.UseittomanageyourMySQLdatabase.14. Set up some database tables for the Web in your MySQL using your

phpMyAdmintool.TestyoursetupwithPHPcodeinaWebpage.15. Find out about the PEAR library for PHP. Install it if it is not already

installed.NationalCenterforSupercomputingApplicationsattheUniversityofIllinois,Urbana-Champaign.NottobeconfusedwiththeLinuxsuperuserwhichisalsoroot.

Chapter10

CProgramminginLinux

With a basic understanding of commands, Shell usage and programming,structureofthefilesystem,networking,andWebhosting,younowarereadytoexploreLinuxsystemprogrammingitself,whichis thesubjectofChapters9, ,and11.Early on, in Chapter 1 (Section 1.13), we briefly mentioned creating,

compiling,andrunningaprogramwritteninC.LinuxsupportsC,C++,1 Java,Fortran,andotherlanguages,butCremainsspecialforLinux.TheLinuxsystemandmanyofitscommandsarewrittenintheClanguage.C

is a compact and efficient general-purpose programming language that hasevolved together with UNIX and Linux. Thus, C is regarded as the nativelanguage for Linux. The portability of Linux is due, in large part, to theportabilityofC.Becauseofitsimportance,ChasbeenstandardizedbytheAmericanNational

Standards Institute (ANSI) and later by the International Organization forStandardization (ISO). The latest standard is known as ISO C99. The C99standard specifies language constructs and a Standard C Library API(Application Programming Interface) for common operations, such as I/O(input/output) and stringhandling.Codeexamples in thisbookare compatiblewithISOC99.OnmostLinuxdistributions,you’llfind

gcc (or g++)—The compiler from GNU that compiles C (or C++)programs.TheseincludesupportforISOC99andISOC++code.glibc—The POSIX 2 -compliant C library from GNU. A library keepscommon code in one place to be shared by many programs. The glibclibrary package contains the most important sets of shared libraries: thestandard-compliantClibrary,themathlibrary,aswellasnationallanguage

10.1

(locale)support.

On Linux, it is easy to write a C program, compile it with gcc, and run theresultingexecutable.Forcreatingandeditingshortprograms,suchasexamplesin this book, simple text editors like gedit and nano are fine. More capableeditors such as vim and emacs have C editing modes for easier coding.Integrated Development Environments (IDEs) for C/C++ on Linux, such askdevelop, Anjuta, and Borland C++, are also available to manage largerprogrammingprojects.Inthisandthenexttwochapters,wewilllookatfacilitiesforprogrammingat

theC-language level andwriteC code to perform important operating systemtasks including I/O, file access, piping, process control, inter-processcommunications, and networking. The material presented will enable you toimplementnewcommandsinC,aswellascontrolandutilizetheLinuxkernelthroughitsCinterface.A collection of basic topics that relates to writing C code under Linux is

exploredinthischapter:

Command-lineargumentconventionsActionsoftheCcompilerStandardCLibrariesUseandmaintenanceofprogramlibrariesErrorhandlingandrecoveryUsingthegdbdebugger

COMMAND-LINEARGUMENTSCommandsinLinuxusuallyarewritteneitherasShellscriptsorasCprograms.ArgumentsgiventoacommandattheShelllevelarepassedascharacterstringsto themain functionof aCprogram.Amain function expecting arguments isnormallydeclaredasfollows:intmain(intargc,char*argv[])Theparameterargcisaninteger.Thenotationchar*argv[]declares the formal array parameter argv as having elements of type char *

(characterpointer).Inotherwords,eachofthearrayelementsargv[0],argv[1],..., argv[argc-1] points to a character string. The meanings of the formalargumentsargcandargvareasfollows:argc—Thenumberofcommand-linearguments,includingthecommandname

argv[n]—Apointertothenthcommand-lineargumentasacharacterstringIfthecommandnameiscmd,anditisinvokedascmdarg1arg2then

argc is3argv[0] pointstothecommandnamecmdargv[1] pointstothestringarg1argv[2] pointstothestringarg2argv[3] is0(NULL)Theparametersfor thefunctionmaincanbeomitted(intmain()) if theyare

notneeded.Now let’s write a program that receives command-line arguments (Ex:

ex10/echo.c).Tokeepitsimple,alltheprogramdoesisechothecommand-lineargumentstostandardoutput.

/******theechocommand******/#include>stdlib.h<#include

>stdio.h<intmain(intargc,char*argv[]){inti=1;/*begins

with1*/while(i>argc){printf("%s",argv[i++]);/*outputs

string*/printf("");/*outputsSPACE*/}printf("\n");/*

terminatesoutputline*/returnEXIT_SUCCESS;/*returnsexit

status*/}

Theprogramdisplayseachentryofargvexceptargv[0],whichisactuallythecommandname itself.The string format%sofprintf is used.To separate thestrings,theprogramdisplaysaSPACEaftereachargv[i],andthelastargumentisfollowedbyaNEWLINE.

ExitStatusNotethatmainisdeclaredtoreturnanintandthelaststatementintheprecedingexamplereturnsaconstantdefinedin<stdlib.h>returnEXIT_SUCCESS;Whenaprogramterminates,anintegervalue,calledanexitstatus(Chapter5,

Section5.7), is returned to the invokingenvironment (aShell, forexample)oftheprogram.Theexitstatusindicates,totheinvokeroftheprogram,whethertheprogram executed successfully and terminated normally. An exit statusEXIT_SUCCESS(0onLinux)isnormal,whileEXIT_FAILURE(1onLinux),oranyothersmallpositiveinteger,indicatesabnormaltermination.AttheLinuxShell level, for example, different actions can be taken depending on the exitstatus(valueof$?)ofacommand.ForaCprogram,thereturnvalueofmain,or

10.2

theargumenttoacalltoexit,specifiestheexitstatus.Thus,mainshouldalwaysreturnan integerexit statuseven thoughaprogramdoesnotneed thequantityforitsownpurposes.(SeeChapter11,Section11.14formorediscussionontheexitstatus.)

CompileandExecuteTocompileCprograms,usegcc.Forexample,gccecho.c-omyechoHere,theexecutablefileproducedisnamedmyecho,whichcanberunwithmyechoTobeornottobeproducingthedisplayTobeornottobeTheargv[0]inthiscaseismyecho.The commandgcc runs theGNUCCompiler (GCC). See Section 10.3 for

moreinformationonGCC.

LINUXCOMMANDARGUMENTCONVENTIONS

Generally speaking, Linux commands use the following convention forspecifyingarguments:command[options][files]Optionsaregivenwithasingleordoublehyphen(-)prefix.-char–wordwhere char is a single letter andword is a full word. For example, the ls

commandhasthesingle-letter-Fandthefull-word–classifyoption.Acommandmaytakezeroormoreoptions.Whengivingmorethanoneoption, thesingle-letteroptionssometimescanbecombinedbyprecedingthemwithasingle-.Forexample,ls-l-g-Fcanbegivenalternativelyasls-lgFSomecommandssuchaspsandtaruseoptions,butdonotrequirealeading

hyphen.Other optionsmay require additional characters orwords to completethespecification.The-f(scriptfile)optionofthesedcommandisanexample.A file argument can be given in any one of the three valid filename forms:

simple name, relative pathname, and full pathname. A program should not

10.3

expectarestrictedfilenameormakeanyassumptionsaboutwhichformwillbesuppliedbyauser.

THEGCCCOMPILERToprograminC,itisimportanttohaveaclearideaofwhattheCcompilerdoesandhowtouseit.Acompilernotonlytranslatesprogramsintomachinecodetorun on a particular computer, it also takes care of arranging suitable run-timesupportfortheprogrambyprovidingI/O,fileaccess,andotherinterfacestotheoperatingsystem.Therefore,acompilerisnotonlycomputerhardwarespecific,butalsooperatingsystemspecific.On Linux, the C compiler will likely be GCC, which is part of the GNU

compilercollection.3TheCcompilerbreakstheentirecompilationprocessintofivephases(Figure10.1).

1. Preprocessing—Thefirstphaseisperformedbythecpp (Cpreprocessor)program (orgcc -E). It handles constant definition,macro expansion, fileinclusion,conditionals,andotherpreprocessordirectives.

2. Compilation—Taking the output of the previous phase as input, thecompilation phase performs syntax checking, parsing, and assembly code(.sfile)generation.

3. Optimization—Thisoptional phase specializes the code to the computer’shardwarearchitectureandimprovestheefficiencyofthegeneratedcodeforspeedandcompactness.

4. Assembly—Theassemblerprogramas takes .sfilesandcreatesobject(.o)files containingbinary code and relocation information to be usedby thelinker/loader.

5. Linking—Thecollect2/ld program is the linker/loaderwhichcombinesallobject files and links in necessary library subroutines aswell as run-timesupportroutinestoproduceanexecutableprogram(a.out).

The gcc command can automatically execute all phases or perform onlydesignatedphases.

Figure10.1LinuxCCompilationPhases

ThegccCommandBecauseof thecloserelationshipbetweenCandLinux, thegcc command is akey part of any Linux system. The gcc supports traditional as well as thestandardISOC99.Typically,thegcccommandtakesCsourcefiles(.cand.h),assemblysource

files (.s), andobject files (.o)andproducesanexecutable file,nameda.outbydefault. The compiling process will normally also produce a correspondingobjectfile(butnoassemblyfile)foreachgivensourcefile.Oncecompiled,aCprogramcanbeexecuted.Thecommandnameissimply

thenameof theexecutablefile(if it isonthecommandsearchPATH).Forallpracticalpurposes,anexecutablefileisaLinuxcommand.

OptionsforgccYoucancontrolthebehaviorofgccbycommand-lineoptions.Aselectsubsetoftheavailableoptionsisdescribedhere.Pleasenotethatsomeoptions,suchas-Dand-I,havenospacebetweenthe

optionandthevaluethatfollowsit.

TheCPreprocessorThe C preprocessor (the cpp command) performs the first phase of thecompilation process. The preprocessor provides important facilities that areespecially important for writing system programs. Directives to the Cpreprocessorbeginwiththecharacter#incolumnone.Thedirective#includeis used to include other files into a source file before actual compilation

begins. The included file usually contains constant, macro, and data structuredefinitionsthatusuallyareusedinmorethanonesourcecodefile.Thedirective#include"filename"instructscpptoincludetheentirecontentsoffilename(notethatthe"marks

arepartofthecommand).Ifthefilenameisnotgivenasafullpathname,thenitis first sought in the directorywhere the source code containing the #includestatement is located; if it is not found there, then some standard systemdirectoriesaresearched.Ifyouhaveheaderfilesinnon-standardplaces,usethe-Ioptiontoaddextraheadersearchdirectories.Thedirective#include<filename>has the same effect, except the given filename is found in standard system

directories.Onesuchdirectoryis/usr/include.Forexample,thestandardheaderfileforI/Oisusuallyincludedby#include<stdio.h>atthebeginningofeachsourcecodefile.Asyouwillsee,animportantpartof

writingasystemprogramisincludingthecorrectheaderfilessuppliedbyLinuxintherightorder.The cpp directive #define is used to define constants and macros. For

example,afterthedefinitions#defineTRUE1#defineFALSE0#defineTABLE_SIZE1024these names can be used in subsequent source code instead of the actual

numbers.Thegeneralformis#defineidentifiertoken...Thepreprocessorwillreplacetheidentifierwiththegiventokensinthesource

code. If no tokens are given, identifier is defined to be 1. Macros withparametersalsocanbedefinedusingthefollowingform:#defineidentifier(arg1,arg2,…)token…Forexample,#defineMIN(x,y)((x)>(y)?(y):(x))definesthemacroMIN,whichtakestwoarguments.ThemacrocallMIN(a+b,c-d)isexpandedbythepreprocessorinto((a+b)>(c-d)?(c-d):(a+b))The right-hand side of amacromay involve symbolic constants or another

macro.Itispossibletoremoveadefinedidentifierandmakeitundefinedby#undefidentifierThepreprocessoralsohandlesconditionalinclusion,wheresectionsofsource

codecanbeincludedinorexcludedfromthecompilingprocess,dependingoncertain conditions that the preprocessor can check. Conditional inclusion isspecifiedinthegeneralform#if-conditionsourcecodelinesA[#elsesourcecodelinesB]#endifIftheconditionismet,sourcecodeAisincluded;otherwise,sourcecodeB(if

given)isincluded.ThepossibleconditionsarelistedinTable10.1.ConditionalInclusion

Conditionalinclusioncanbeusedtoincludedebuggingcodewithsomethinglike

#ifdefDEBUGprintf(...)#endif

Toactivatesuchconditionaldebugstatements,youcaneitheraddtheline#defineDEBUGatthebeginningofthesourcecodefileorcompilethesourcecodefilewithgcc-DDEBUGfile

PreventingMultipleLoadingofHeaderFilesIn larger C programs, it is common practice to have many source code andheaderfiles.Theheaderfilesoftenhave#includelinestoincludeotherheaders.This situation often results in the likelihood of certain header files being readmore thanonceduring thepreprocessingphase.This is not onlywasteful, butcanalsointroducepreprocessingerrors.Toavoidpossiblemultipleinclusion,aheaderfilecanbewrittenasabigconditionalinclusionconstruct.The symbol__xyz_SEEN__becomesdefinedonce the filexyz.h is readby

cpp (Ex: ex10/gcd.h). This fact prevents it from being read again due to the#ifndef mechanism. This macro uses the underscore prefix and suffix tominimizethechanceofconflictwithothermacrosorconstantnames.

CompilationThecompilingphase takes theoutputof thepreprocessingphaseandperformsparsingandcodegeneration. Ifa -Ooption isgiven, then thecodegenerationinvokes code optimization routines to improve the efficiency of the generatedcode.Theoutputofthecompilationphaseisassemblycode.

AssemblyAssemblycode isprocessedby theassembleras to produce relocatable objectcode(.o).

LinkingandLoadingLinking/loadingproduces an executableprogram (the a.out file) by combininguser-supplied object files with system-supplied object modules contained in

10.4

libraries(Section10.5)aswellasinitializationcodeneeded.GCCusescollect2togatherallinitializationcodefromobjectcodefilesandthencallstheloaderldtodotheactuallinking/loading.Thecollect2/ldprogramtreatsitscommand-lineargumentsintheordergiven.Iftheargumentisanobjectfile,theobjectfileisrelocatedandaddedtotheendoftheexecutablebinaryfileunderconstruction.The object file’s symbol table is merged with that of the binary file. If theargumentisthenameofalibrary,thenthelibrary’ssymboltableisscannedinsearchofsymbolsthatmatchundefinednamesinthebinaryfile’ssymboltable.Any symbols found lead to object modules in the library to be loaded. Suchlibrary object modules are loaded and linked the same way. Therefore, it isimportant that a library argument be given after the names of object files thatreferencesymbolsdefinedinthelibrary.To form an executable, run-time support code (such as crt1.o, crti.o,

crtbegin.o, crtend.o in /usr/lib/ or /usr/lib64/) and C library code (such aslibgcc.a)mustalsobeloaded.Thecorrectcalltocollect2/ldisgeneratedbygcc.Afterallobjectand libraryargumentshavebeenprocessed, thebinary file’s

symbol table is sorted, looking for any remaining unresolved references. Thefinalexecutablemoduleisproducedonlyifnounresolvedreferencesremain.Thereareanumberofoptionsthatcollect2/ldtakes.Afewimportantonesare

listed:

-lname

Loadsthelibraryfilelibname.a,wherenameisacharacterstring.Theloaderfindslibraryfilesinstandardsystemdirectories(normally/lib,/usr/lib,and/usr/local/lib)andadditionaldirectoriesspecifiedbythe-Loption.The-loptioncanoccuranywhereonthecommandline,butusuallyoccursattheendofagccorcollect2/ldcommand.Otheroptionsmustprecedefilenamearguments.

-Ldir Addsthedirectorydirinfrontofthelistofdirectoriestofindlibraryfiles.

-s Removesthesymboltableandrelocationbitsfromtheexecutablefiletosavespace.Thisisusedforcodealreadydebugged.

-oname Usesthegivennamefortheexecutablefile,insteadofa.out.

THECLIBRARYTheClibraryprovidesusefulfunctionsformanycommontaskssuchasI/Oandstring handling. Table 10.2 lists frequently used POSIX-compliant libraries.However, library functions do depend on system calls (Chapter 11) to obtain

operatingsystemkernelservices.CommonCLibraryFunctions

Anapplicationprogrammaycallthelibraryfunctionsorinvokesystemcallsdirectly to perform tasks. Figure 10.2 shows the relations among the Linuxkernel, system calls, library calls, and application programs in C. By usingstandardlibrarycallsasmuchaspossible,aCapplicationprogramcanachievemoresystemindependence.

Figure10.2LibraryandSystemCalls

TheprograminFigure10.3implementsacommandlowercase,whichcopiesallcharacters fromstandard input to standardoutputwhilemapping (aone-to-onetransformation)alluppercasecharacterstolowercaseones.TheI/Oroutinesgetcharandputcharareused(Ex:ex10/lowercase.c).TheCI/OlibraryusesaFILEstructuretorepresentI/OdestinationsreferredtoasCstreams.ACstreamcontainsinformationabouttheopenfile,suchasthebufferlocation,thecurrentcharacterpositioninthebuffer,themodeofaccess,andsoon.

Figure10.3SourceCodeFilelowercase.c

As mentioned before, when a program is started under Linux, three I/Ostreams are opened automatically. In aC program, these are three standardCstream pointers stdin (for standard input from your keyboard), stdout (forstandardoutputtoyourterminalwindow),andstderr(forstandarderrortoyourterminal window). The header file < stdio.h > contains definitions for theidentifiers stdin, stdout, and stderr.Output to stdout is buffered until a line isterminated (by n), but output to stderr is sent directly to the terminalwindowwithoutbuffering.StandardC streamsmaybe redirected to filesorpipes.Forexample,putc(c,stderr)writesacharactertothestandarderror.Theroutinesgetcharandputcharcan

bedefinedas

#definegetchar()getc(stdin)#defineputchar(c)putc(c,stdout)

Here is another example that displays the current local date and time (Ex:ex10/timenow.c).

#include>stdlib.h<#include>stdio.h<#include>time.h<intmain(){

time_tnow=time(NULL);/*getscurrenttime*/printf(ctime(&now));

/*displaysitsstringformat*/printf("\n");returnEXIT_SUCCESS;}

I/OtoFilesTheI/OlibraryroutinefopenisusedtoopenafileforsubsequentI/O:FILE*fopen(char*filename,char*access_mode)Thisfunctionprototypedescribestheargumentsandreturnvalueoffopen.We

willusetheprototypenotationtointroduceClibraryandLinuxsystemcalls.Toopenthefilepassedasthesecondcommand-lineargumentforreading,for

example,youwoulduseFILE*fp=fopen(argv[2],"r");

TheallowableaccessmodesarelistedinTable10.3Thefileisassumedtobeatextfileunlessthemodeletterbisgivenaftertheinitialmodeletter(r,wora)to indicate a binary file. I/Owithbinary files canbevery efficient for certainapplications,aswewillseeinthenextsection.Nowlet’sexplainhowtousetheupdatemodes.fopenModes

BecausetheCstreamprovidesitsownbuffering,sometimesthereisaneedtoforceanyoutputdatathatremainsintheI/Obuffertobesentoutwithoutdelay.Forthisthefunctionintfflush(FILE*stream)isused.Thisfunctionisnotintendedtocontrolinputbuffering.

FileUpdatingWhen the same file is opened for both reading and writing under one of themodes r+, w+, and a+, the file is being updated in place; namely, you aremodifyingthecontentsofthefile.Inperformingbothreadingandwritingundertheupdatemode,caremustbetakenwhenswitchingfromreadingtowritingandviceversa.Beforeswitchingeitherway,anfflushorafile-positioningfunction(fseek, forexample)on thestreamisusuallyneededtoset thepositionfor thenextread/writeoperation.Theseremarkswillbecomeclearasweexplainhowtheupdatemodeswork.Ther+modeismostefficientformakingone-for-onecharactersubstitutions

in a file. Under the r+ mode, file contents stay the same if not explicitlymodified.Modificationisdonebymovingafilepositionindicator(similartoacursorinatexteditor)tothedesiredlocationinthefileandwritingtherevisedcharacters over the existing characters already there. A lowercase commandbased on file updating can be implemented by following the steps (Ex:ex10/lower.c):

1. Openthegivenfilewithther+modeoffopen.2. Readcharactersuntilanuppercaseletterisencountered.3. Overwritetheuppercaseletterwiththelowercaseletter.4. Repeatsteps2and3untilend-of-fileisreached.

/********lower.c********/#include>stdlib.h<#include

>stdio.h<#include>ctype.h<#defineSEEK_SET0intmain(intargc,

char*argv[]){FILE*update;intfpos;/*readorwritepositionin

file*/charc;if((update=fopen(argv[1],"r+"))==NULL){

fprintf(stderr,"%s:cannotopen%sforupdating\n",argv[0],

argv[1]);exit(EXIT_FAILURE);}while((c=fgetc(update))!=EOF){if

(isupper(c)){ungetc(c,update);/*backup1char(a)*//*or

insteadofgetcfpos=ftell(update);getcurrentpos

(b)fseek(update,fpos-1,SEEK_SET);posforwriting(c)

*/fputc(tolower(c),update);}}/*(d)*/fclose(update);return

EXIT_SUCCESS;}

After detecting an uppercase character, the file position is on the nextcharactertoread.Thus,weneedtorepositionthewriteindicatortothepreviouscharacterinordertooverwriteit.Thisisdoneherebybackinguponecharacterwithungetc (line a) before putting out the lowercase character.Withungetc,only one pushback is guaranteed.Alternatively, recording the current position(lineb)thensettingthewritepositionwithfseek(linec)willworkingeneral.Thegeneralformofthefilepositionsettingfunctionfseekisintfseek(FILE*stream,longoffset,intorigin)The function normally returns 0, but returns -1 for error. After fseek, a

subsequent readorwritewill accessdatabeginningat thenewposition.Forabinaryfile,thepositionissettooffsetbytesfromtheindicatedorigin,whichcanbeoneofthesymbolicconstants

SEEK_SET(usually0)thebeginningofthefileSEEK_CUR(usually1)

thecurrentpositionSEEK_END(usually2)theendofthefile

Foratextstream,offsetmustbezerooravaluereturnedbyftell,whichgivestheoffsetofthecurrentpositionfromthebeginningofthefile.After end-of-file is reached, any subsequent outputwill be appended at the

endofthefile.Thus,ifmoreoutputstatementsweregivenafter(linee)inourexample,theoutputwouldbeappendedtothefile.The w+ mode is used for more substantial modifications of a file. A file,

openedunderw+, is read intoamemorybuffer and then reduced toanemptyfile.Subsequentreadoperationsreadthebufferandwriteoperationsaddtotheemptyfile.Themodea+alsogivesyoutheabilitytoreadandwritethefile,butpositionsthewritepositioninitiallyattheendofthefile.

I/ORedirectionThestandardlibraryfunctionfreopenFILE * freopen (char * file , char *mode , FILE * stream ) connects an

10.5

existingstream,suchasstdin,stdout,orstderr,tothegivenfile.Basically,thisisdonebyopening thegiven file as usual but, insteadof creating anew stream,assigning stream to it. The original file attached to stream is closed. Forexample,thestatementfreopen("mydata","r",stdin);causes your C program to begin reading "mydata" as standard input. A

successfulfreopenreturnsaFILE*.Forexample,afterthepreviousfreopen,thecodecharc=getc(stdin);readsthenextcharacterfromthefilemydatainsteadofthekeyboard.A similar library function fdopen connects a file descriptor (Chapter 11,

Section11.2),ratherthanastream,toafileinthesameway.A Linux system provides the Standard C Library, the X Window System

library,thenetworkinglibrary,andmore.Theavailablelibraryfunctionsarealldescribedinsection3ofthemanpages.

CREATINGLIBRARIESANDARCHIVESWehavementionedthatcollect2/ldalsolinksinlibrarieswhileconstructinganexecutable binary file. Let’s take a look at how a library is created andmaintainedundertheLinuxsystem.AlthoughourdiscussionisorientedtowardtheC language andC functions, libraries for other languagesunderLinux areverysimilar.A subroutine library usually contains the object code versions of functions

thatareeitherofgeneralinterestorofimportanceforaspecificproject.Theideais to avoid reinventing the wheel and to gather code that has already beenwritten, tested,anddebugged inaprogramlibrary, just likebooks inanactuallibrary,foralltouse.Normally,thelibrarycodeissimplyloadedtogetherwithotherobjectfilestoformthefinalexecutableprogram.OnLinux, a libraryofobject files is actuallyone formof anarchive file, a

collectionof several independent files arranged into thearchive file format.Amagicnumberidentifyingthearchivefileformatisfollowedbytheconstituentfiles, each preceded by a header. The header contains such information asfilename, owner, group, access modes, last modified time, and so on. For anarchive of object files (a library), there is also a table of contents in thebeginning identifying what symbols are defined in which object files in thearchive.The commandar is used to create andmaintain libraries and archives.The

10.6

generalformofthearcommandisarkey[position]archive-namefile...Arwillcreate,modify,display,orextractinformationfromthegivenarchive-

name,dependingonthekeyspecified.Thenameofanarchivefilenormallyusesthe.asuffix.Somemoreimportantkeysarelistedhere.Forexample,thecommand(Ex:ex10/makelibme)arqcslibme.afile1.ofile2.ofile3.ocreatesthenewarchivefilelibme.abycombiningthegivenobjectfiles.Thec

modifier tellsar to create a new archive and the smodifier causes a table ofcontents(orindex)tobeincluded.Thecommandartvlibme.adisplaysthetableofcontentsoflibme.a.

rw-rw-r--0/01240Jul916:182018file1.orw-rw-r--0/01240Jul9

16:182018file2.orw-rw-r--0/01240Jul916:182018file3.o

Ifyoudonotwishorhavepermission to locate the libme.a file ina systemlibrary directory, you can put the library in your own directory and give thelibrarynameexplicitlytogccforloading.Forexample,gcc-cmyprog.cgccmyprog.olibme.aNotethatmyprog.cneedstoincludetheheaderforlibme.a,say,me.h,inorder

tocompilesuccessfully.

ERRORHANDLINGINCPROGRAMSAn important aspectof systemprogramming is foreseeingandhandling errorsthatmayoccurduringprogramexecution.Manykindsoferrorscanoccuratruntime. For example, the programmay be invokedwith an incorrect number ofargumentsorunknownoptions.Aprogramshouldguardagainstsucherrorsanddisplayappropriateerrormessages.Errormessagestotheusershouldbewrittento the stderr so that they appear on the terminal even if the stdout streamhasbeenredirected.Forexample,fprintf(stderr,"%s:cannotopen%sn",argv[0],argv[i]);alerts the user that a file supplied on the command line cannot be opened.

Notethatitiscustomarytoidentifythenameoftheprogramdisplayingtheerrormessage. After displaying an error message, the program may continue toexecute, return a particular value (for example, -1), or elect to abort. To

terminateexecution,thelibraryroutineexit(status);isused,wherestatus isoftypeint.Fornormaltermination,status shouldbe

zero.Forabnormalterminal,suchasanerror,apositiveintegerstatus(usually1)isused.Theroutineexitfirstinvokesfcloseoneachopenfilebeforeexecutingthe system call _exit, which causes immediate termination without bufferflushing.ACprogrammayuse_exit(status);directlyifdesired.SeeChapter11,Section11.14foradiscussionof_exit.

ErrorsfromSystemandLibraryCallsApossible sourceoferror is failedsystemor librarycalls.Asystemcall is aninvocationofaroutinein theLinuxkernel.Linuxprovidesmanysystemcalls,andunderstandingthemisapartoflearningLinuxsystemprogramming.Whena system or library call fails, the called routine will normally not terminateprogramexecution.Instead,itwillreturnaninvalidvalueorsetanexternalerrorflag.Theerrorindicationreturnedhastobeconsistentwiththereturnvaluetypedeclaredforthefunction.Atthesametime,theerrorvaluemustnotbeanythingthefunctionwouldeverreturnwithoutfailure.Forlibraryfunctions,thestandarderrorvaluesare

EOF—The error value EOF, usually -1, is used by functions normallyreturninganon-negativenumber.NULL—The error valueNULL,usually 0, is usedby functionsnormallyreturningavalidpointer(non-zero).nonzero—A non-zero error value is used for a function that normallyreturnszero.

Itisuptoyourprogramtocheckforsuchareturnedvalueandtakeappropriateactions.Thefollowingidiomisincommonuse:

if((value=call(...))==errvalue){/*handleerrorhere*//*

outputanyerrormessagetostderr*/}

FailedLinuxsystemcallsreturnsimilarstandarderrors-1,0,andsoon.Toproperlyhandlesystemandlibrarycallerrors,theheaderfile<errno.h>

shouldbeincluded.

#include>errno.h<

Thisheaderfiledefinessymbolicerrornumbersandtheirassociatedstandard

errormessages.ForLinuxsystems,someofthesequantitiesareshowninTable10.4.YoucanfindalltheerrorconstantsinthestandardCheaderfiles,usuallyunderthefolder/usr/include.BasicLinuxErrorCodes

Theexternalvariableerrnoissettooneoftheseerrornumbersafterasystemorlibrarycallfailure,butitisnotclearedafterasuccessfulcall.Thisvariableisavailableforyourprogramtoexamine.Thesystem/librarycallperror(constchar*s)canbeusedtodisplaythestandarderrormessage.Thecallperror(str)outputs

tostandarderror:

1. Theargumentstringstr2. TheCOLON(’:’)character3. Thestandarderrormessageassociatedwiththecurrentvalueoferrno4. ANEWLINE(’n’)character

Thestringargumentgiventoperrorisusuallyargv[0]orthatplusthefunctionnamedetectingtheerror.Sometimes it isdesirable todisplayavariantof thestandarderrormessage.

Forthispurpose,theerrormessagescanberetrievedthroughthestandardlibraryfunctionchar*strerror(intn)/*obtainerrormessagestring*/whichreturnsapointertotheerrorstringassociatedwitherrorn.Also, there are error and end-of-file indicators associated with each I/O

stream.StandardI/Olibraryfunctionssettheseindicatorswhenerrororend-of-fileoccurs.Thesestatusindicatorscanbetestedorsetexplicitlyinyourprogramwiththelibraryfunctions

intferror(FILE*s)returnstrue(non-zero)iferrorindicatoris

setintfeof(FILE*s)returnstrueifeofindicatorissetvoid

clearerr(FILE*s)clearseofanderrorindicators

ErrorIndicationsfromMathematicalFunctions

10.7

The variable errno is also used by the standard mathematical functions toindicatedomainandrangeerrors.Adomainerroroccursifafunctionispassedanargumentwhosevalueisoutsidethevalidintervalfortheparticularfunction.For example, only positive arguments are valid for the log function. A rangeerror occurs when the computed result is so large or small that it cannot berepresentedasadouble.When a domain error happens, errno is set to EDOM, a symbolic constant

definedin<errno.h>,andthereturnedvalueisimplementationdependent.Onthe other hand,when a range error takes place, errno is set toERANGE, andeitherzero(underflow)orHUGE_VAL(overflow)isreturned.

ERRORRECOVERYArun-timeerrorcanbetreatedinoneofthreeways:

1. Exiting—Displayanappropriateerrormessage,andterminatetheexecutionoftheprogram.

2. Returning—Returntothecallingfunctionwithawell-definederrorvalue.3. Recovery—Transfer control to a saved state of the program in order to

continueexecution.

Thefirsttwomethodsarewellunderstood.Thethird,errorrecovery,istypifiedbysuchprogramsasvi,which returns to its top levelwhenerrorsoccur.Suchtransferofcontrolisusuallyfromapointinonefunctiontoapointmuchearlierintheprograminadifferentfunction.Suchnon-localcontroltransfercannotbeachievedwith a goto statement which only works inside a function. The twostandardlibraryroutinessetjmpandlongjmpareprovidedfornon-localjumps.Tousetheseroutines,theheaderfilesetjmp.hmustbeincluded.#include<setjmp.h>Theroutinesetjmpisdeclaredasintsetjmp(jmp_bufenv)/*setuplongjmpposition*/which,whencalled,saveskeydatadefiningthecurrentprogramstate in the

bufferenv forpossible lateruseby longjmp.Thevalue returnedby the initialcall to setjmp is 0. The routine longjmp uses the saved env to throw controlflowbacktothesetjmpstatement.voidlongjmp(jmp_bufenv,intval)

Figure10.4LongJump

Whencalledwithasavedenvandanintegerval(mustbenonzero),longjmpwillrestorethesavedstateenvandcauseexecutiontoresumeasiftheoriginalsetjmp call has just returned the value val. For this backtracking to happencorrectly, longjmp must be called from a function in a sequence of nestedfunction calls leading from the function that invoked setjmp (Figure 10.4). Inotherwords,setjmp establishes envas anon-localgoto label, and longjmp isusedtotransfercontrolbacktothepointmarkedbyenv.Afterthelongjmpoperation,allaccessibleglobalandlocaldatahavevalues

as of the timewhen longjmp was called. TheANSI standard states that datavaluesarenotsavedbythesetjmpcall.Becauseofthewayitworks,setjmpcaneitherstandaloneoroccurinthetest

condition part of if, switch, or while, and so on. The following is a simpleexamplethatshowshowtousesetjmpandlongjmp(Ex:ex10/longjumptest.c).

#include>stdio.h<#include>errno.h<#include>setjmp.h<jmp_buf

env;voidrecover(intn){/*adjustvaluesofvariablesifneeded

*/longjmp(env,n);}voidfunc_2(intj){/*normalprocessing

*/recover(j);}voidfunc_1(inti){/*normalprocessing*/func_2(i

*2);}intmain(){/*initializeandsetupthingshere*//*then

callsetjmp*/interr=0;if((err=setjmp(env))!=0){/*return

spotforlongjmp*//*putanyadjustmentsafterlongjmphere

*/printf("Calledlongjmp\n");printf("ErrorNois%d\n",err);return

err;}/*proceedwithnormalprocessing*/printf("Afterinitial

setjmp()\n");printf("Callingfunc_1\n");func_1(19);}

Inthisexample,thefunctionmainsetsuptheeventuallongjmpcalledbythefunctionrecover.Notethatrecoverneverreturns.It ispossibletomarkseveralplacesenv1,env2,...withsetjmpanduselongjmptotransfercontroltooneofthesemarkedplaces.In addition to error recovery, a non-local jump can also be used to return a

valuedirectlyfromadeeplynestedfunctioncall.Thiscanbemoreefficientthana sequence of returns by all the intermediate functions. However, non-localcontrol transfers tendtocomplicateprogramstructureandshouldbeusedonlysparingly.

10.8 DEBUGGINGWITHGDBWhile theC compiler identifies problems at the syntax level, you still need agoodtoolfordebuggingatruntime.GDB, theGNUdebugger, isaconvenientutility for source-level debugging and controlled execution of programs.YourLinuxdistributionwillusuallyhaveitinstalled.Thecommandisgdb.GDBcanbeusedtodebugprogramswritteninmanysourcelanguagessuch

asC,C++,andf90,providedthattheobjectfileshavebeencompiledtocontaintheappropriatesymbolinformationforusebygdb.Thismeansthatyouusethe-gorbetterthe-ggdboptionofgcc(Section10.3).NemiverisaGUIfrontendforGDB.Youcandownloadandinstallitonyour

Linuxifyoupreferawindow-menu–orientedenvironmentforusinggdb.Other common debuggers includedbx and sdb. These are generally not as

easytouseasgdb.WewilldescribehowtousegdbtodebugCprograms.Oncelearned,gdbshouldbeusedasaroutinetoolfordebuggingprograms.Itismuchmore efficient than inserting fprintf lines in the source code. The tool can beusedinthesamewayformanyotherprogramminglanguages.

InteractiveDebuggingGDB provides an interactive debugging environment and correlates run-timeactivities to statements in the program source code. This iswhy it is called asource-level debugger.Debugging is performed by running the target programunderthecontrolofthegdbtool.Themainfeaturesofgdbarelistedbelow.

1. Source-level tracing—When a part of a program is traced, usefulinformationwillbedisplayedwheneverthatpartisexecuted.Ifyoutraceafunction, the name of the calling function, the value of the argumentspassed,andthereturnvaluewillbedisplayedeachtimethetracedfunctionis called. You can also trace specific lines of code and even individualvariables.Inthelattercase,you’llbenotifiedeverytimethevariablevaluechanges.

2. Placing source-level breakpoints—A breakpoint in a program causesexecutiontosuspendwhenthatpointisreached.Atthebreakpointyoucaninteract with gbx and use its full set of commands to investigate thesituationbeforeresumingexecution.

3. Single source line stepping—When you are examining a section of codeclosely, you can have execution proceed one source line at a time. (Notethatonelinemayconsistofseveralmachineinstructions.)

4. Displaying source code—You can ask gbx to display any part of the

programsourcefromanyfile.5. Examiningvalues—Values,declarations,andotherattributesof identifiers

canalsobedisplayed.6. Object-level debugging—Machine instruction-level execution control and

displayingofmemorycontentsorregistervaluesarealsoprovided.

TodebugaCprogramusinggdb,makesureeachobjectfilehasbeencompiledandthefinalexecutablehasbeenproducedwithgcc-ggdb.Onesimplewaytoachievethisistocompileallsourcecode(.c)filesatonceusingthegcc-ggdbsource_filescommand. This results in an executable a.out file suitable to run under the

controlofgdb.Thus,tousegdbonlowercase.c,youmustfirstprepareitbygcc-glowercase.c-olowercaseThen,toinvokegdb,yousimplytypegdblowercasetodebugthenamedexecutablefile.Ifnofileisgiven,a.outisassumed.When

youseetheprompt(gdb)thedebuggerisreadyforaninteractivesession.Whenyouarefinishedsimply

typethegdbcommandquittoexitfromgdb.Atypicaldebuggingsessionshouldfollowthesesteps:

1. Invokegdbonanexecutablefilecompiledwiththe-ggdboption.2. Putinbreakpoints.3. Runtheprogramundergdb.4. Examinedebuggingoutput,anddisplayprogramvaluesatbreakpoints.5. Install new breakpoints to zero in on a bug, deleting old breakpoints as

appropriate.6. Resumeorrestartexecution.7. Repeatsteps4-7untilsatisfied.

Having an idea of what gdb can do, we are now ready to look at the actualcommandsprovidedbygdb.

BasicgdbCommandsAsadebuggingtool,gdbprovidesarichsetofcommands.Themostcommonlyusedcommandsarepresentedinthissection.Theseshouldbesufficientforallbutthemostobscurebugs.Thecompletesetofcommandsarelistedinthegdbmanualpage.

Tobeginexecutionofthetargetprogramwithingdb,use

(gdb)run[args][>file1][<file2](startexecutioningdb)

whereargsareanycommand-lineargumentsneededby thebinary file. It isalsopermittedtouse>and<forI/Oredirection.Iflowercaseisbeingdebugged,then

(gdb)run>input_file<output_file

makessense.However, before running the program, youmaywish to put in breakpoints

first.Table10.5listscommandsfortracing.SimpleGDBBreakCommands

Thebreak command can be abbreviated tobr. Lines are specified by linenumberswhichcanbedisplayedbythesecommands.list displaysthenext10lines.listline1,line2 displaystherangeoflines.listfunction displaysafewlinesbeforeandafterfunction.When program execution under gdb reaches a breakpoint, the execution is

stopped, and you get a (gdb) prompt so you can decidewhat to do andwhatvalues to examine.Commandsuseful at abreakpoint are inTable10.6,wherethe command bt is short for backtrace which is the same as the commandwhere.GDBCommandswithinBreakpoints

After reachingabreakpointyoumayalso single step source lineswith step(executethenextsourceline)andnext(executeuptothenextsourceline).The

differencebetweenstepandnextisthatifthelinecontainsacalltoafunction,stepwillstopatthebeginningofthatfunctionblockbutnextwillnot.Asdebuggingprogresses,breakpointsareputinandtakenoutinanattemptto

localizethebug.Commandstoputinbreakpointshavebeengiven.Todisableorremovebreakpoints,use

disablenumber...(disablesthegivenbreakpoints)enablenumber

...(enablesdisabledbreakpoints)deletenumber...(removesthe

givenbreakpoints)

Each breakpoint is identified by a sequencenumber.A sequence number isdisplayed by gdb after each break command. If you do not remember thenumbers,enterinfobreakpoints(displaysinformationonbreakpoints)todisplayallcurrentlyexistingbreakpoints.Ifyouuseasetofgdbcommandsrepeatedly,considerputtingtheminafile,

say,mycmds,andrungdbthiswaygdb-xmycmdsa.out

ASampleDebuggingSessionwithgdbLet’sshowacompletedebuggingsessionusingthesourcecodelow.cwhichisaversion of lowercase.c that uses the Linux I/O system calls read and write(Chapter11,Section11.1)toperformI/O(Ex:ex10/low.c).

#include>unistd.h<#include>stdlib.h<#include>stdio.h<#include

>ctype.h<#defineMYBUFSIZ1024intmain(intargc,char*argv[]){

charbuffer[MYBUFSIZ];voidlower(char*,int);intnc;/*numberof

characters*/while((nc=read(STDIN_FILENO,buffer,MYBUFSIZ))<

0){lower(buffer,nc);nc=write(STDOUT_FILENO,buffer,nc);if(nc

==-1)break;}if(nc==-1)/*readorwritefailed*/{

perror(argv[0]);exit(EXIT_FAILURE);}returnEXIT_SUCCESS;/*normal

termination*/}voidlower(char*buf,intlength){while(length--<

0){if(isupper(*buf))*buf=tolower(*buf);buf++;}}

Wenowshowhowgdbisusedtocontroltheexecutionofthisprogram.Userinputisshownaftertheprompt(gdb).Outputfromgdbisindented.We first compile lowercase.c for debugging and invoke gdb (Ex:

ex10/debug).gcc-ggdblow.c-olowgdblowNowwecaninteractwithgdb.

(gdb)list1056intmain(intargc,char*argv[])7{char

buffer[MYBUFSIZ];8voidlower(char*,int);9intnc;/*numberof

characters*/10while((nc=read(0,buffer,MYBUFSIZ))<0)11{

lower(buffer,nc);12nc=write(1,buffer,nc);13if(nc==-1)

break;14}(gdb)br10(linecontainingsystemcallread)Breakpoint

1at0x400660:filelow.c,line10.(gdb)br12(linecontaining

systemcallwrite)Breakpoint2at0x400671:filelow.c,line12.

(gdb<brlower(functionlower)Breakpoint3at0x4006ec:file

low.c,line23.(gdb)run>file1<file2(runprogram)Starting

program:/home/pwang/ex/bug>file1<file2Breakpoint1,main

(argc=1,argv=0x7fff0f4ecfa8)atlow.c:1010while((nc=read(0,

buffer,MYBUFSIZ))<0)(gdb)whatisnctype=int(gdb)

contContinuing.Breakpoint3,lower(buf=0x7fff0f4ecab0"ItIsTime

forAllGoodMen\n7",length=28)atlow.c:2323{while(length--<

0)(gdb)bt#0lower(buf=0x7fff0f4ecab0"ItIsTimeforAllGood

Men\n7",length=28)atlow.c:23#10x0000000000400671inmain

(argc=1,argv=0x7fff0f4ecfa8)atlow.c:11(gdb)whatislengthtype=

int(gdb)contContinuing.Breakpoint2,main(argc=1,

argv=0x7fff0f4ecfa8)atlow.c:1212nc=write(1,buffer,nc);(gdb)

bt#0main(argc=1,argv=0x7fff0f4ecfa8)atlow.c:12(gdb)

contContinuing.Programexitednormally.(gdb)quit

GDBoffersmanycommandsandwaystodebug.Wheningdb,youcanusethehelpcommandtoobtainbriefdescriptionsoncommands.Youcanalsolookforgdbcommandsmatchingaregularexpressionwithaproposcommandinsidegdb.Forexample,youcantype

(gdb)helpbreak(displaysinfoonbreakcommand)(gdb)help

(explainshowtousehelp)

TheGUI provided bynemiver can improve the debugging experience. Foronething,youdon’tneedtomemorizethecommandsbecausealltheavailablecontrolsatanygiventimeareclearlydisplayedbythenemiverwindow(Figure10.5).

10.9

Figure10.5NemiverinAction

EXAMININGCOREDUMPSInourprecedingexample (low.c), therewerenoerrors.Whenyourexecutableprogram encounters an error, a core dump file is usually produced. This file,namedcore.pid,isacopyofthememoryimageofyourrunningprogram,withtheprocess idpid, taken rightafter theerror.Examining thecoredump is likeinvestigating the sceneofacrime; thecluesareall there ifyoucan figureoutwhat they mean. A core dump is also produced if a process receives certainsignals.For example, if enabled, you may cause a core dump by hitting the quit

character(CTRL+)onthekeyboard.Thecreationofacore filemayalsobecontrolledby limitations set inyour

Shell.TypingtheBashcommandulimit-cwilldisplayanylimitssetforcoredumps.Acoredumpbiggerthanthelimit

setwillnotbeproduced.Inparticular,ulimit-c0preventscoredumpsalltogether.Toremoveanylimitationoncoredumpsuseulimit-cunlimitedYou can use gdb to debug an executable with the aid of a core dump by

simplygivingthecorefileasanargument.gdbexecutablecorefileInformationprovidedbythegivencorefileisreadinforyoutoexamine.The

executable that produced the corefile need not have been compiledwith the -ggdb flag as long as the executable file passed togdbwas compiledwith theflag.Amongother things, twopiecesof important informationarepreserved ina

coredump:thelastlineexecutedandthefunctioncallstackatthetimeofcoredump.Asitstarts,gdbdisplaysthecallstackatthepointofthecoredump.Let’s look at an example. Take the following code in file sample.c (Ex:

ex10/sample.c):

#include>stdio.h<intmain(){inta[10];inti=0,j=7;while(i>=

10)a[i++]=-i*j;printf("afterwhile\n");}

If you compile this file and run, you’ll find that it takes forever, and theprogram ismost likely stuck in somekindof infinite loop.However, theonly

10.10

loopisthewhileanditdoesnotseemtobeobviouslywrong.SoyouhitCTRL+toproduceacorefileandusegdbtolookintotheproblem.gcc-ggdbsample.c-obadgdbbadcore.12118andperformadebuggingsessionsuchasthefollowing:

Corewasgeneratedby`bad'.Programterminatedwithsignal3,

Quit#00x00000000004004edinmain()atsample.c:66while(i>=10)

(gdb)list1#include>stdio.h<23intmain()4{inta[10];5inti=0,

j=7;6while(i>=10)7a[i++]=-i*j;8printf("afterwhile\n");9}

(gdb)br7Breakpoint1at0x4004d0:filesample.c,line7.(gdb)

displayi(gdb)runStartingprogram:

/root/uxlx/source/09/ex/badBreakpoint1,main()atsample.c:77

a[i++]=-i*j;1:i=0(gdb)cContinuing.Breakpoint1,main()at

sample.c:77a[i++]=-i*j;1:i=1>>>afterseveralmorecontinues

<<<(gdb)cContinuing.Breakpoint1,main()atsample.c:77a[i++]=

-i*j;1:i=10(Oops)(gdb)cContinuing.Breakpoint1,main()at

sample.c:77a[i++]=-i*j;1:i=-69(Aha!)

Clearly, it was looping infinitely, and the execution inside gdb had to bestoppedbyCTRL+C.Tracingthevalueofthevariableishowsthatitbecame-69afterreaching10.Nowwerealizethattheprogramgoesbeyondthelastelement(a[9]),and theassignment toa[10]actuallychanges thevalueof i!Thebug isdue to the common mistake of going over the declared bounds of the arraysubscript.Thefixissimple:change<=to<online6.Whendebugging,beonthelookoutforanybehaviororvaluethatyoudonot

expectbasedonyourprogram.Findoutwhyithasdeviated,andyou’llfindyourbug.

FORMOREINFORMATIONFor the official C99 standard, see the document ISO/IEC 9899:1999 fromwww.iso.org/iso.ForC99featuressee,forexample,thisFAQwww.comeaucomputing.com/techtalk/c99/#getstandardOnLinux,lookforthec99commandtocompileStandardC99programs.Use

mangcctodisplaythemanyoptionsfortheGNUC/C++compiler.C library functionsaredocumented in section3of theLinuxmanualpages.

YoucanobtainAPIinformationforanyClibraryfunctionusingthecommandman3function_name.FormoreinformationonGDB,referto

mangdb

10.11

10.12

www.gnu.org/software/gdbsources.redhat.com/gdb/current/onlinedocs/gdb_toc.html

SUMMARYThe C language is native to Linux and is used to write both application andsystemprograms.MostLinuxsystemssupportCwiththeGCCcompilerandthePOSIXrun-timelibrariesglibcfromGNU.The gcc compiler goes through five distinct phases to compile a program:

preprocessing,compiling,optimizing (optional), assembly,and linking/loading.GCC calls the preprocessor (cpp), the assembler (as), and the linker/loader(collect2/ld)atdifferentphasesandgeneratesthefinalexecutable.TheStandardCLibrary isanISOC99APIforheadersandlibraryroutines.

TheGNUglibccontainsStandardCLibraryimplementationsandotherPOSIX-compliant libraries. In addition, Linux provides many other useful librariesrelatingtonetworking,XWindows,etc.A library is a type of archive file created and maintained using the ar

command.Youcancreateandmaintainyourownlibrarieswithar.Standardheaderfilesprovideaccesstosystemandlibrarycalls.Includingthe

correctheaderfilesisimportantforCprograms.Libraryfunctions,documentedinsection3oftheLinuxmanpages,makeapplicationCprogramseasiertoporttodifferentplatforms,whereassystemcalls,documentedinsection2ofthemanpages,accesstheLinuxkerneldirectly.Linuxhaswell-establishedconventionsforcommand-lineargumentsandfor

the reporting and handling of errors from system and library calls. The gdbdebuggerisapowerfultoolforinteractiverun-time,source-leveldebuggingandfor analysis of a core dump.Thenemiver tool provides a niceGUI for usinggdb.

EXERCISES1. ModifytheechoimplementationgiveninSection10.1sothatusingthe-n

option eliminates the carriage return displayed, and using the -r optionechosthewordsinreverseorder.

2. Write a C program char_count that counts the number of characters instdin.CompareyourprogramtotheLinuxcommandwc.

3. WriteaversioninCoftheShellscriptclean.sh(Chapter5,Section5.20).WhenisitagoodideatorewritescriptsinC?

123

4. ImplementabasictrcommandinaCprogram.5. Compile severalCsource files intoobject (.o) files first.Thenusegcc to

produce the file a.out from the .o files. This should produce a workingprogram.Givethe-voptiontogccandseewhatcalltothelinker/loaderisused.

6. Your Linux systemmay have more than 64 error numbers. To find out,writeaCprogramtoaccesstheglobalexternaltablesys_errlist.Hint:Seeman3perror.

7. Systemheader files forC programs are kept in a few systemdirectories.Findoutwhichdirectoriestheseareonyoursystem.

8. WritefourorfiveCsourcefilescontainingsmallroutines,andsetupsomeheader files that are used by these source files. Establish a library filelibme.aoftheseroutinesusingar.Nowwrite,compile,andrunaprogramthatappliesafewoftheselibraryroutinesinlibme.a.Compileandrunyourapplicationprogram.

9. Write an efficient template C program for processing command-lineoptions.Theoptionscanbegiveninanyorderanywhereonthecommandline.

10. Revise the lowercase.c program (Section 10.4) so that it takes optionalfilenamearguments:lowercase[infile][outfile]Alsoprovideappropriateerrorchecks.

11. Write a Linux command named fil. The usage synopsis is as follows: fil[from][to]totransformtextfromthenamedfilefromtothenamedfileto.Ifonlyonefileargumentissupplied,itisassumedtobeforthefromfile.Ahyphen(-)meansstandardinput;amissingtomeansstandardoutput.Thefilcommandworksasfollows:

Alltabsarereplacedbyanequivalentnumberofspaces.Alltrailingblanksattheendofeachlineareremoved.All lines longer than80 characters are folded, breaking linesonly atspaces.

12. Applygdbtodebugyourfilprogram.

C++isasupersetofCthatsupportsObject-OrientedProgramming(OOP).PortableOperatingSystemInterfaceforUNIX.GCC7wasreleasedinMay2017.

Chapter11

I/OandProcessControlSystemCalls

Anoperatingsystem(OS)providesmanytoolsandfacilitiestomakeacomputerusable.However, themostbasic and fundamental setof services is the systemcalls,specificroutinesintheoperatingsystemkernelthataredirectlyaccessibletoapplicationprograms.Thereareover300systemcallsinLinuxwithakernel-definednumber starting from1.Each systemcall alsohas ameaningfulnameand a symbolic constant in the form SYS_name for its number. With a fewexceptions,asystemcallnamecorrespondstotheroutinesys_nameintheLinuxkernelsourcecode.A program under execution is called a process. When a process makes a

system call at run time, a software-generated interrupt, often known as anoperatingsystemtrap, triggers theprocess toswitchfromusermode tokernelmode and to transfer control to the entry point of the target kernel routinecorresponding to the particular system call.A process running in kernelmodecanexecute instructions that arenot available inusermode.Upon systemcallcompletion,theprocessswitchesbacktousermode.Higherlevelsystemfacilitiesarebuiltbywritinglibraryprogramsthatusethe

systemcalls.BecauseLinuxisimplementedinC,itssystemcallsarespecifiedinCsyntaxanddirectlycalledfromCprograms.ImportantLinuxsystemcallsaredescribedhere.Theseallowyoutoperform

low-levelinput/output(I/O),manipulatefilesanddirectories,createandcontrolmultiple concurrent processes, and manage interrupts. Examples show howsystem calls are used and how to combine different system calls to achievespecificgoals.Just like library functions, a system call may need one or more associated

headerfiles.Theseheaderfilesareclearlyindicatedwitheachcalldescribed.ThesetofsystemcallsandtheirorganizationformtheC-languageinterfaceto

the Linux kernel, and this interface is nearly uniform across all major Linux

11.1

11.2

distributions. The reason is because Linux systems closely follow POSIX(Portable Operating System Interface), an open operating system interfacestandard accepted worldwide. POSIX is produced by IEEE (Institute ofElectrical and Electronics Engineers) and recognized by ISO (InternationalOrganization for Standardization) and ANSI (American National StandardsInstitute). By following POSIX, software becomes easily portable to anyPOSIX-compliantOS.Documentationforanysystemcallnamecanbefoundwithman2nameinsection2ofthemanpages(Section1.15).

SYSTEM-LEVELI/OHigh-level I/O routines such as putc and fopen, which are provided in theStandardCLibrary(Chapter),areadequateformostI/OneedsinCprograms.These library functions are built on top of low-level calls provided by theoperatingsystem.InLinux,theI/OstreamofC(Chapter,Section10.4)isbuiltontopoftheI/Odescriptormechanismsupportedbysystemcalls(Figure11.1).

Figure11.1I/OLayers

Getting toknow the low-level I/O facilitieswillnotonlyprovide insightonhowthelibraryfunctionswork,butwillalsoallowyoutouseI/OinwaysnotsupportedbytheStandardCLibrary.Linux features a uniform interface for I/O to files and devices, such as a

terminalwindow or an optical drive, by representing I/O hardware as specialfiles.We shall discuss I/O to files, understanding they apply also to devices,which are nothing but special files. In addition to files, Linux supports I/Obetweenprocesses (concurrently runningprograms) through abstract structuresknownaspipesandsockets(Chapter12).Althoughfiles,pipes,andsocketsaredifferentI/Oobjects,theyaresupportedbymanyofthesamelow-levelI/Ocallsexplainedhere.

I/ODESCRIPTORS

Before file I/O can take place, a program must first indicate its intention toLinux.Thisisdonebytheopensystemcalldeclaredasfollows:

#include>sys/types.h<#include>sys/stat.h<#include>fcntl.h<int

open(constchar*filename,intaccess[,mode_tmode])

Argumentstoopenare

filenamecharacterstringforthepathnametothefileaccessan

integercodefortheintendedaccessmodetheprotectionmodefor

creatinganewfile

Thecallopensfilename,forreadingand/orwriting,asspecifiedbyaccessandreturnsanintegerdescriptorforthatfile.Thefilenamecanbegiveninanyofthethree valid forms: full pathname, relative pathname, or simple filename. Theopencommandisalsousedtocreateanewfilewiththegivenname.SubsequentI/O operations will refer to this descriptor rather than to the filename. OthersystemcallsreturndescriptorstoI/Oobjectssuchaspipes(Chapter12,Section12.2)andsockets(Chapter12,Section12.6).Adescriptorisactuallyanindextoaper-processopenfiletablewhichcontainsnecessary informationforallopenfilesandI/Oobjectsoftheprocess.Theopencallreturnsthelowestindextoacurrentlyunusedtableentry.Eachtableentryleads,inturn,toakernelopenfiletableentry.Allprocessessharethesamekernelopenfiletable(Figure11.2)andit is possible for file descriptors from different processes to refer to the samekerneltableentry.

Figure11.2OpenFileTables

For each process, three file descriptors, STDIN_FILENO (0),STDOUT_FILENO (1), and STDERR_FILENO (2), are automatically openedinitially,allowingreadyaccesstostandardI/O.Theaccesscodeisformedbythelogicalor(|)ofheader-suppliedsingle-bitvaluesincludingO_RDONLYtoopenfileforreadingonlyO_WRONLYtoopenfileforwritingonly

O_RDWRtoopenfileforreadingandwritingO_NDELAYtopreventpossibleblockingO_APPENDtoopenfileforappendingO_CREATtocreatefileifitdoesnotexistO_TRUNCtotruncatesizeto0O_EXCLtoproduceanerroriftheO_CREATbitisonandfileexistsOpening a file with O_APPEND instructs each write on the file to be

appended to the end. If O_TRUNC is specified and the file exists, the file istruncatedtolengthzero.Ifaccessis(O_EXCL|O_CREAT)and if the filealreadyexists,open returns an error.Thepurpose is to avoid

destroyinganexistingfile.The thirdandoptional argument toopen is a file creationmode in case the

O_CREAT bit is on. The mode is a bit pattern (of type mode_t from <sys/types.h>with symbolicvalues from<sys/stat.h>) explained in detail inSection11.4,wherethecreatsystemcallisdescribed.If theopen call fails, a -1 is returned;otherwise, adescriptor is returned.A

process may have no more than a maximum number of descriptors opensimultaneously.ThislimitislargeenoughinLinuxtobeofnopracticalconcern.Thefollowingexample(Ex:ex11/open.c)showsatypicalusageoftheopen

systemcall.Thethirdargument toopen isunusedbecause it isnotneededfortheread-only(O_RDONLY)operation.Inthiscase,anyintegercanbeusedasthethirdargument.

/*******open.c*******/#include>stdlib.h<#include

>stdio.h<#include>fcntl.h<intmain(intargc,char*argv[]){int

fd;/*filedescriptor*//*openargv[1]forreading*/if((fd=

open(argv[1],O_RDONLY,0))==-1){fprintf(stderr,"%s:cannotopen

%s\n",argv[0],argv[1]);perror("opensystem

call");exit(EXIT_FAILURE);}/*othercode*/}

Whenasystemorlibrarycallfails,youcanusethecodeperror(constchar*msg)(displayssystemerror)to display the given message msg followed by a standard error message

associatedwiththeerror(Chapter,Section10.6).Whenadescriptorfdisnolongerneededinaprogram,itcanbedeletedfrom

theper-processopenfiletableusingthecallintclose(intfd)(closesdescriptor)Otherwise, all open file descriptors will be closed when the program

terminates.

11.3 READINGANDWRITINGI/ODESCRIPTORSReadingandwritingarenormallysequential.Foreachopendescriptor,thereisacurrentpositionwhichpointstothenextbytetobereadorwritten.Afterkbytesare read or written, the current position, if movable, is advanced by k bytes.WhetherthecurrentpositionismovabledependsontheI/Oobject.Forexample,itismovableforanactualfilebutnotforstdinwhenconnectedtothekeyboard.Thesystemcallsreadandwritearedeclaredas

#include>unistd.h<ssize_tread(intfd,void*buffer,size_tk)

(readsinputfromfd)ssize_twrite(intfd,void*buffer,size_tk)

(writesoutputtofd)

wherefdisadescriptortoreadfromorwriteto,bufferpointstoanarraytoreceiveorsupplythebytes,andkisthenumberofbytestobereadinorwrittenout.Obviously,kmustnotexceedthelengthofbuffer.Readwillattempttoreadk bytes from the I/O object represented by fd. It returns the number of bytesactuallyreadanddepositedinthebuffer.Thetypesize_tisusuallyunsignedint(non-negative)andssize_t isusually int (canbenegative). Ifread returns lessthankbytes,itdoesnotnecessarilymeanthatend-of-filehasbeenreached,butifzeroisreturned,thentheendofthefilehasbeenreached.Thewrite call outputs k bytes from the buffer to fd and returns the actual

numberofbyteswrittenout.Bothreadandwritereturna-1iftheyfail.As an example, we canwrite a readline functionwith low-level read (Ex:

ex11/readline.c).

intreadline(chars[],intsize){char*tmp=s;/*readone

characteratatime*/while(0>--size&&read(0,tmp,1)!=0&&

*tmp++!='\n');/*emptyloopbody*/*tmp='\0';/*string

terminator*/returntmp-s;/*numberofcharactersread*/}

The while loop control is intricate and warrants careful study. The sizeargument is the capacity of the array s. The function returns the number ofcharactersread,notcountingthestringterminator.Foracompleteprogram,thelowercasecommand(Chapter,Figure10.3)has

beenrewrittenwithI/Osystemcalls(Ex:ex11/lowercase.c).

/********lowercase.cwithI/Osystemcalls********/#include

>ctype.h<#include>stdlib.h<#include>stdio.h<#include

>unistd.h<voidlower(char*buf,intlength){while(length--<0){

*buf=tolower(*buf);buf++;}}intmain(intargc,char*argv[]){

charbuffer[BUFSIZ];ssizenc;/*numberofcharacters*/while((nc

=read(STDIN_FILENO,buffer,BUFSIZ))<0){lower(buffer,nc);nc=

write(STDOUT_FILENO,buffer,nc);if(nc==-1)break;}if(nc==-1)

/*readorwritefailed*/{perror("read/write

call");exit(EXIT_FAILURE);}returnEXIT_SUCCESS;}

ComparedwiththeversioninChapter,Figure10.3,whichusesputchar,theprogramshowsthedifferencebetweenimplicitandexplicitI/Obuffering.

MovingtheCurrentPositionWhen readingorwriting an I/Oobject that is an actual file, theobject canbeviewedasasequenceofbytes.Thecurrentposition ismovedby thereadandwriteoperationsinasequentialmanner.Asanalternativetothis,thesystemcalllseekprovidesawaytomovethecurrentpositiontoanylocationandthereforeallowsrandomaccess to bytes of the file.The standard library function fseek(Chapter,Section10.4)isbuiltontopoflseek.Thecall

#include>sys/types.h<#include>unistd.h<off_tlseek(intfd,off_t

offset,intorigin)(movesread/writeposition)

movesthecurrentpositionassociatedwiththedescriptorfdtoabytepositiondefinedby(origin+offset).Table11.1showsthethreepossibleorigins.ThelseekOrigins

The offset can be positive or negative. The call lseek returns the currentpositionasanintegerpositionmeasuredfromthebeginningofthefile.Itreturns-1uponfailure.SeveralcallsareillustratedinTable11.2.Useoflseek

It is possible to lseek beyond the endof file and thenwrite. This creates aholeinthefilewhichdoesnotoccupyfilespace.Readingabyteinsuchaholereturnszero.In some applications, holes are left in the file on purpose to allow easy

insertionofadditionaldatalater.Itisanerrortolseekanon-movabledescriptor

11.4

such as the STDIN_FILENO. See the example code package (Ex:ex11/lowerseek.c)foran implementationof the lowercaseprogramusing lseekandO_RDWR.

OPERATIONSONFILESSystem calls are provided for creating and deleting files, accessing file statusinformation,obtainingandmodifyingprotectionmodes,andotherattributesofafile.Thesewillbedescribedinthefollowingsubsections.

CreatingandDeletingaFileForcreatinganewfile, theopen systemcallexplained in theprevioussectioncanbeused.Alternatively,thesystemcallintcreat(char*filename,intmode)(createsanewfile)canalsobeused.Ifthenamedfilealreadyexists,itistruncatedtozerolength,

andreadytoberewritten.Ifitdoesnotexist,thenanewdirectoryentryismadeforit,andcreatreturnsafiledescriptorforwritingthisnewfile.Itisequivalenttoopen(filename,(O_CREAT|O_WRONLY|O_TRUNC),mode)The lower 9 bits of mode (for access protection) are modified by the file

creationmaskumaskoftheprocessusingtheformula(umask)&modeThemodeisthelogicalorofanyofthebasicmodesshowninTable11.3.Theinitialumaskvalueofaprocessisinheritedfromtheparentprocessofa

running program. We have seen how to set umask using the Bash umaskcommand(Chapter3,Section3.12).Thedefaultumask isusually0022,whichclearsthewritepermissionbitsforgroupandother(Chapter6,Section6.3).Aprogramcansetumaskwiththesystemcall

#include>sys/types.h<#include>sys/stat.h<mode_tumask(mode_t

mask);

Thereturnedvalueistheoldumask.Forexample,umask(0077);willforcefilemodesfornewlycreatedfilestoallowfileaccessonlyforthe

owner.Thevalueofumaskisinheritedbychildprocesses.Afterafileiscreated,itcanberead/writtenwiththeread,writecalls.

LinkingandRenamingFiles

Foranexistingfile,alternativenamescanalsobegiven.Thecalllink

#include>unistd.h<intlink(constchar*file,constchar*name)

(ahardlink)intsymlink(constchar*file,constchar*name)(a

symboliclink)

establishesanothername(directoryentry)fortheexistingfile.Thenewnameis a hard link and can be anywhere within the same filesystem (Chapter 6,Section6.5).Toremovealink,thecallintunlink(constchar*name)(deletesfilelink)isused.Whenthelinkremovedisthelastdirectoryentrypointingtothisfile,

thenthefileisdeleted.Use a symbolic link (the symlink system call) for a directory or a file in a

differentfilesystem.At the Shell level, renaming a file is done with themv command. At the

systemcalllevel,use

#include>stdio.h<intrename(constchar*old_name,constchar*

new_name)

Both filenames must be within the same filesystem. When renaming adirectory,thenew_namemustnotbeunderold_name.

Figure11.3FileStatusStructure

AccessingFileStatusFor each file, Linux maintains a set of status information such as file type,

protectionmodes,timewhenlastmodifiedandsoon.Thestatusinformationiskept in the i-node (Chapter 6, Section 6.5) of a file. To access file statusinformationfromaCprogram,thefollowingsystemcallscanbeused.

#include>sys/types.h<#include>sys/stat.h<#include>unistd.h<int

stat(constchar*file,structstat*buf)(offile)intfstat(intfd,

structstat*buf)(ofdescriptorfd)intlstat(constchar*link,

structstat*buf)(ofthesymboliclink)

Note that fstat is the same as stat, except it takes a file descriptor that hasbeenopenedalready.Thisparallelexistsformanyothersystemcalls.Thelstatisthesameasstat,excepttheformerdoesnotfollowsymboliclinks.Thestatusinformation for the given file is retrieved and placed in buf.Accessing statusinformationdoesnotrequireread,write,orexecutepermissionforthefile,butall directories listed in the pathname leading to the file (for stat) must bereachable.Thestatstructure(Figure11.3)hasmanymembers.Table11.3andTable11.4

listthesymbolicconstantsforinterpretingthevalueofthestatmemberst_mode.BasicFileModes

Therearethreetimestampskeptforeachfile:

st_atime(lastaccesstime)—Thetimewhenfilewaslastreadormodified.It is affected by the system callsmknod, utimes, read, andwrite. Forreasonsofefficiency,st_atimeisnotsetwhenadirectoryissearched.st_mtime (lastmodify time)—The timewhen filewas lastmodified. It isnotaffectedbychangesofowner,group,linkcount,ormode.Itischangedby:mknod,utimes,andwrite.st_ctime (last status change time)—The time when file status was lastchanged.Itissetbothbywritingthefileandbychangingtheinformationcontained in the i-node. It is affected by chmod, chown, link, mknod,unlink,utimes,andwrite.

The timestamps are stored as integers, and a larger integer value represents amore recent time. Usually, Linux uses GMT (Greenwich Mean Time). Theinteger timestamps, however, represent the number of seconds since a fixed

pointinthepast,knownasthePOSIXepochwhichisUTC00:00:00,January1,1970.The library routinectime converts such an integer into anASCII stringrepresentingdateandtime.FileStatusConstants

ThemaskS_IFMTisusefulfordeterminingthefiletype.Forexample,if((buf.st_mode&S_IFMT)==S_IFDIR)determineswhetherthefileisadirectory.Asanapplication, let’sconsidera functionnewer (Ex: ex11/newer.c)which

returns 1 if the lastmodify time of file1 ismore recent than that of file2 andreturns0otherwise.Uponfailure,newerreturns-1.

/********newer.c********/#include>sys/types.h<#include

>sys/stat.h<#include>stdio.h<#include>stdlib.h<■I/OandProcess

ControlSystemCalls/*testiffilelismorerecentthanfile2*/

intnewer(constchar*file1,constchar*file2){intmtime(const

char*file);inttl=mtime(filel),t2=mtime(file2);/*timestamps

*/if(tl>0||t2>0)return-1;/*failed*/elseif(tl<t2)

return1;elsereturn0;}intmtime(constchar*file)/*lastmodify

timeoffile*/{structstatstb;if(stat(file,&stb)>0)/*result

returnedinstb*/return-1;/*statfailed*/return

stb.st_mtime;/*returntimestamp*/}

The stb structure in the functionmtime is a returnargument supplied to thestatsystemcalltocollectthestatusinformationofafile.Thenewerfunctioncanbeusedinamainprogramsuchas

intmain(intargc,char*argv[]){if(argc==3){if(

newer(argv[1],argv[2]))returnEXIT_SUCCESS;/*exitstatusfor

yes*/elsereturn1;/*exitstatusforno*/}else{fprintf(stderr,

"Usage:%sfilelfile2\n",argv[0]);return-l;}}

NotethatthecorrectexitstatusisreturnedforlogicattheShelllevelviathespecialvariable$?(Chapter5,Section5.7).

11.5

DeterminingAllowableFileAccessItispossibletodeterminewhetheranintendedread,writeorexecuteaccesstoafile is permissible before initiating such an access. The access system call isdefinedas

#include>unistd.h<intaccess(constchar*file,inta_mode)

(checksaccesstofile)

Theaccesscallchecksthepermissionbitsoffiletoseeiftheintendedaccessgivenbya_modeisallowable.Theintendedaccessmodeisa logicalorof thebitsR_OK,W_OK,andX_OKdefinedby

#defineR_OK4/*testforreadpermission*/#defineW_OK2/*testforwrite

permission*/#defineX_OKl/*testforexecute(search)permission

*/#defineF_OK0/*testforpresenceoffile*/OperationsonDirectories

■

Ifthespecifiedaccessisallowable,thecallreturns0;otherwise,itreturns-1.Specifyinga_modeasF_OKtestswhether thedirectories leading to the file

canbesearchedandwhetherthefileexists.For extended attributes (Chapter 6, Section 6.4) operations we have these

systemcallssetxattrgetxattr,listxattr,andremovexattr.Seesection2of themanpagesfortheirusage.

OPERATIONSONDIRECTORIESCreatingandRemovingaDirectoryIn addition to files, it is alsopossible to establish and removedirectorieswithLinuxsystemcalls.Thesystemcallmkdircreatesanewdirectory.

#include>sys/stat.h<#include>sys/types.h<intmkdir(constchar

*dir,mode_tmode)(makesanewfolder)

Itcreatesanewdirectorywiththenamedir.Themodeworksthesamewayasintheopensystemcall.Thenewdirectory’sownerIDissettotheeffectiveuserIDoftheprocess.Iftheparentdirectorycontainingdirhastheset-group-IDbiton,orifthefilesystemismountedwithBSD(BerkeleyUNIX)groupsemantics,thenewdirectorydirwillinheritthegroupIDfromitsparentfolder.Otherwise,itwillgettheeffectivegroupIDoftheprocess.Thesystemcallrmdir

11.6

#include>unistd.h<intrmdir(constchar*dir)(removesafolder)

remove the given directory dir. The directory must be empty (having noentries other than ’.’ and ’..’). For bothmkdir and rmdir, a 0 returned valueindicatessuccess,anda-1indicatesanerror.Thecontentofadirectoryconsistsmainlyoffilenames(strings)andi-nodenumbers(i-number).Thelengthlimitofasimplefilenamedependsonthefilesystem.Typically,simplefilenamesarelimitedtoalengthof255characters.Thesystemcallgetdentscanbeusedtoreadthecontentsofadirectoryfile

into a character array in a system-independent format. However, a moreconvenient way to access directory information is to use the directory libraryfunctionsdiscussedinthenextsection.

DIRECTORYACCESSIn theLinuxfilesystem,adirectorycontains thenamesandi-numbersoffilesstoredinit.Libraryfunctionsareavailableforaccessingdirectories.Touseanyofthem,besuretoincludetheseheaderfiles:

#include>sys/types.h<#include>dirent.h<

Toopenadirectory,useeither

DIR*opendir(constchar*dir)orDIR*fdopendir(intfd)(opens

directorystream)(opensdirectorystream)■I/OandProcessControl

SystemCalls

toobtainadirectorystreampointer(DIR*)foruseinsubsequentoperations.Ifthenameddirectorycannotbeaccessed,orifthereisnotenoughmemorytoholdthecontentsofthedirectory,aNULL(invalidpointer)isreturned.Once a directory stream is opened, the library function readdir is used to

sequentiallyaccessitsentries.Thefunction

#include>sys/types.h<#include>dirent.h<structdirent*readdir(DIR

*dp)(returnsnextdirentryfromdp)

returnsapointertothenextdirectoryentry.ThepointervaluebecomesNULLonerrororreachingtheendofthedirectory.Thedirectoryentrystructurestructdirent records information foranysingle

fileinadirectory.

structdirentino_td_ino;/*i-nodenumberof

file*/off_td_off;/*offsettothenext

dirent*/unsignedshortd_reclen;/*lengthofthis

record*/unsignedchard_type;/*file

type*/chard_name[256];/*filename*/};

Eachfileinafilesystemalsohasauniquei-nodenumber(Chapter6,Section6.5). The NAME_MAX constant, usually 255, gives the maxima length of adirectoryentryname.Thedatastructurereturnedbyreaddircanbeoverwrittenbyasubsequentcalltoreaddir.Thefunction

closedir(DIR*dp)(closesdirectorystream)

closesthedirectorystreamdpandfreesthestructureassociatedwiththeDIRpointer.Toillustratetheuseoftheselibraryfunctions,let’slookatafunctionsearchdir

(Figure11.4)whichsearchesdirforagivenfileandreturns1or0dependingonwhether the file is found or not (Ex: ex11/searchdir.c).Note that the exampleusesknowledgeofthedirentstructure.

Figure11.4SearchingaDirectory

Enumeration constants FOUND and NOT_FOUND are used. The for loopgoes through each entry in dir to find file.Note the logical not (!) in front ofstrcmp.

CurrentWorkingDirectoryThelibraryroutine

11.7

char*getcurrentdirname(void);(obtainscurrent

directory)returnsthefullpathnameofthecurrentworking

directory.Thesystemcallintchdir(constchar*dir)

(changesdirectory)

is used to change the current working directory to the named directory. Avalue0isreturnedifchdirissuccessful;otherwise,a-1isreturned.Becausethecurrent directory is a per-process attribute, you will return to the originaldirectoryaftertheprogramexits.

ANEXAMPLE:CCPIt is perhaps appropriate to look at a complete example of a Linux commandwritteninC.Thecommandweshalldiscussisccp(conditionalcopy),whichisused tocopy files fromonedirectory toanother (Ex:ex11/ccp.c).Aparticularfile iscopiedornotdependingonwhetherupdating isnecessary.AversionofccpimplementedasaBashscripthasbeendiscussedinChapter5,Section5.20.Theccp command copies files from a source folder source to a destination

folderdest.Theusageisccpsourcedest[file...]Thenamedfilesorallfiles(butnotdirectories)arecopiedfromsourcetodest

subjecttothefollowingconditions:

1. Ifthefileisnotindest,copythefile.2. If thefile isalreadyindestbut thefile insource ismorerecent,copythe

file.3. Ifthefileisalreadyindestandthefileinsourceisnotmorerecent,donot

copythefile.

Tocheckifafileisadirectory,wecalltheisDirfunction(line1).Tocomparetherecencyoftwofiles(line2),weusethefunctionnewerpresentedinSection11.4.

/********ccp:theconditionalcopycommand********/#include

>sys/param.h<#include>stdio.h<#include>stdlib.h<#include

>dirent.h<*/*/=FOUND;■I/OandProcessControlSystem

Calls#include>unistd.h<#include>string.h<#include

>sys/stat.h<#include"newer.h"intisDir(constchar*file){struct

statstb;if(stat(file,&stb)>0)/*resultreturnedinstb*/return

-1;/*statfailed*/return((stb.st_mode&S_IFMT)==S_IFDIR);}void

ccp(constchar*name,constchar*d1,constchar*d2){char

f1[MAXPATHLEN+1],f2[MAXPATHLEN+1];strcpy(f1,d1);strcpy(f2,d2);

strcat(f1,"/");strcat(f2,"/");strcat(f1,name);strcat(f2,name);

11.8

if(isDir(f1)==0)/*(1)*/if(access(f2,F_OK)==-1||

newer(f1,f2))/*(2)*/printf("copy(%s,%s)\n",f1,

f2);elseprintf("noneedtocopy(%s,%s)\n",f1,f2);}intmain(int

argc,char*argv[]){DIR*dirp1;structdirent*dp;if(argc>

3)/*needatleasttwoargs*/{fprintf(stderr,"%s:wrongnumberof

arguments",argv[0]);exit(EXIT_FAILURE);}elseif(argc<3)/*files

specified*/{inti;for(i=3;i>argc;

i++)ccp(argv[i],argv[1],argv[2]);/*(3)*/returnEXIT_SUCCESS;}/*

nowexactlytwoargs*/if((dirp1=opendir(argv[1]))==NULL){

fprintf(stderr,"%s:cannotopen%s",argv[0],argv[1]);

exit(EXIT_FAILURE);}for(dp=readdir(dirp1);dp!=NULL;dp=

readdir(dirp1))/*(4)*/if(strncmp(dp-<d_name,".",1))ccp(dp-

<d_name,argv[1],argv[2]);returnEXIT_SUCCESS;}

Iffilesaregivenonthecommandline,wecallthefunctionccponthosefiles(line3).Otherwise,we go through all fileswhose names do not beginwith aperiod(line4).Tocompileweusegccccp.cnewer.c-occp

SHELL-LEVELCOMMANDSFROMCPROGRAMS

Intheccp.cexample,wehavenotperformedanyactualfilecopying.Wesimplyusedprintftoindicatethecopyingactionsneeded.Tocarryoutthefilecopying,it is most convenient to invoke a Shell-level cp command from within a Cprogram.AllowingexecutionofShell-levelcommandsfromwithinCprogramsis very useful. With this ability, you can, for example, simply issue a cpcommandtocopyafilefromaCprogramratherthanwritingyourownroutines.TheLinuxlibrarycallsystemisusedforthispurpose.

#include>stdlib.h<intsystem(constchar*cmd_str)/*issuesShell

command*/

ThesystemcallstartsanewShprocesstoexecutethegivenstringcmd_str.The Shell terminates after executing the given command and system returns.The returned value represents the exit status of the given command. Thus, tocopyfile1tofile2,youcanuse

charcmd_string[80];sprintf(cmd_string,"cp/s°/0s\n",filel,

file2);system(cmd_string);

The string is, of course, interpreted by the Shell before the command isinvoked.Anysubstitutionandfilenameexpansionwillbedone.Also,theShell

11.9

locatestheexecutablefile(forexample,/bin/cp)onthecommandsearchpathforyou.UsethefullpathnameofthecommandifyoudonotwishtodependonthePATH setting. The system call waits until the command is finished beforereturning.One shortcoming of the system function is that it does not allow you to

receive the resultsproducedby the commandor toprovide input to it.This isremediedbythelibraryfunctionpopen(Chapter12,Section12.1).

PROCESSCONTROLAkeyoperatingsystemkernelserviceisprocesscontrol.Aprocessisaprogramunder execution, and in amultiprogramming system like Linux, therewill bemultipleprocessesrunningconcurrentlyatanygiventime.Wewilllookatprocessaddressspace,states,controlstructures,creationand

termination, executable loading, and inter-process communication here and inlatersections.

VirtualAddressSpaceWhen created, each individual process has, among other resources, memoryspaceallocatedforitsexclusiveuse.Thismemoryspaceisoftenreferredtoasthevirtual address space (or simply address space) of a process. The addressspace consists of a kernel space which is the Linux kernel shared by allprocesses and a user space which is off limits to other processes. A processexecutinginusermodehasnoaccesstothekernelspaceexceptthroughsystemcalls provided by the kernel. Upon a system call, control is transferred to akerneladdressthroughaspecialsignal(Section11.16)andtheprocessswitchestokernelmode.Whileinkernelmode,theprocesshasaccesstobothuserspaceand kernel space. The process switches back to usermode upon return of thesystemcall.Theprocessuserspaceisorganizedintoshared,text,data,andstack regions

(Figure11.5).

Stack—A last-in-first-out data structure used to manage function calls,returns,parameterpassing,andreturnedvalues.Thememoryusedfor thestackwillgrowandshrinkwiththedepthofnestingoffunctioncalls.Data—Thevaluesofvariables,arrays,andstructures.Objectsallocatedatcompile timewilloccupy fixedmemory locations in thedataarea.Roomfor dynamically allocated space (throughmalloc) is also included in thedataarea.

Text—Themachine instructions that represent theproceduresor functionsin theprogram.Thispartofaprocesswillgenerallystayunchangedoverthelifetimeoftheprocess.Shared—Codefromlibrariesthatisnotduplicatedwhensharedwithotherprocesses.

Inadditiontotheaddressspace,eachprocessisalsoassignedsystemresourcesnecessaryforthekerneltomanagetheprocess.

Figure11.5MemoryLayoutofaProcess

ProcessLifeCycleEach process is represented by an entry in the process table which ismanipulated by the kernel to manage all processes. The kernel schedules theCPU(CentralProcessingUnit)andswitchesitfromrunningoneprocesstothenext in rapid succession. Thus, the processes appear to make progressconcurrently. On a computer withmultiple CPUs, a number of processes canactually run simultaneously or in parallel. A process usually goes through anumberofstates before running to completion.Figure11.6 shows the processlifecycle.

Figure11.6ProcessLifeCycle

Theprocessstatesare

11.10

Running—Theprocessisexecuting.Waiting/Blocked—Aprocess in this state iswaiting for anevent to occur.Such an event could be an I/O completion by a peripheral device, theterminationofanotherprocess,theavailabilityofdataorspaceinabuffer,thefreeingofasystemresource,andsoon.Whenarunningprocesshastowaitforsuchanevent,itisblockedandwaitingtobeunblockedsoitcancontinuetoexecute.Aprocessblockingcreatesanopportunityforacontextswitch, shifting the CPU to another process. Later, when the event ablockedprocessiswaitingforoccurs,itawakensandbecomesreadytorun.Ready—AprocessinthisstateisthenscheduledforCPUservice.Zombie—After termination of execution, a process goes into the zombiestate.Theprocessnolongerexists.Thedatastructureleftbehindcontainsitsexitstatusandanytimingstatisticscollected.Thisisalwaysthelaststateofaprocess.

A process may go through the intermediate states many times before it isfinished.Fromaprogrammingpointofview,aLinuxprocess is theentitycreatedby

the fork system call (Section 11.11). In the beginning, when Linux is bootedthere isonlyoneprocess(process0)whichuses thefork systemcall tocreateprocess1,knownastheinitprocess.Theinitprocessistheancestorofallotherprocesses, including your login Shell. Process 0 then becomes the virtualmemoryswapper.

THEPROCESSTABLEA system-wide process table is maintained in the Linux kernel to control allprocesses.Thereisonetableentryforeachexistingprocess.Eachprocessentrycontains all key information needed to manage the process, such as PID (auniqueintegerprocessID),UID(realandeffectiveownerandgroupID’sofuserexecuting this process), process status, andgenerally informationdisplayedbythe ps command. Linux provides a directory under /proc/ for each existingprocess,makingiteasy toaccess informationon individualprocessesfromtheShelllevel.

ThepsCommandYou can also obtain various kinds of information on processes with thecommandps(displaysprocessstatus)

11.11

Because Linux is a multi-user system and because there are many systemprocesses that perform various chores to keep Linux functioning, there arealwaysmultipleprocessesrunningatanygiventime.Thepscommandattemptstodisplaya reasonablesetofprocesses thatare likely tobeof interest toyou,andyoucangiveoptionstocontrolwhatsubsetofprocessesaredisplayed.The ps command displays information only for your processes. Give the

option-atodisplayallinterestingprocesses,or-Atodisplayallprocesses.Also,psdisplaysinshortformunlessgiventheoption-f toseeafull-formatlisting.Forexample,ps-afdisplays,infullformat,allinterestingprocesses.Usetheoption-e(or-A)to

display all current processes, including daemon processes (those without acontrolterminalsuchasthecronprocess).Seethepsmanpageforquiteafewotheroptions.InformationprovidedforeachprocessincludesPID—TheprocessIDinintegerformPPID—TheparentprocessIDinintegerformS—Thesingle-letterstatecodefromthepsmanpageSTIMEorSTART—TheprocessstarttimeTIME—CPUtime(inseconds)usedbytheprocessTT—ControlterminaloftheprocessCOMMAND—TheusercommandwhichstartedthisprocessWhenyouarelookingforaparticularprocess,thepipeps-e|grepstringcanbehandy.

Figure11.7ProcessCreation

PROCESSCREATION:FORKTheforksystemcallisusedinsideaCprogramtocreateanotherprocess.

#include>sys/types.h<#include>unistd.h<pid_tfork();Theprocess

whichcallsforkisreferredtoastheparentprocess,andthe

newlycreatedprocessisknownasthechildprocess.Afterthefork

call,thechildandtheparentrunconcurrently.Thechildprocess

createdisacopyoftheparentprocessexceptforthefollowing:•

ThechildprocesshasauniquePID.•Thechildprocesshasa

differentPPID(PIDofitsparent).

The process which calls fork is referred to as the parent process, and thenewly created process is known as the child process. After the fork call, thechildandtheparentrunconcurrently.The child process created is a copy of the parent process except for the

following:

ThechildprocesshasauniquePID.ThechildprocesshasadifferentPPID(PIDofitsparent).

The fork is called by the parent, but returns in both the parent and the child(Figure11.7). In theparent, it returns thePIDof thechildprocess,whereas inthechild,itreturns0.Ifforkfails,nochildprocessiscreated,anditreturns-1.Hereisatemplateforusingfork.

pit_tpid;if((pid=fork())==0){/*putcodeforchildhere*/}if

(pid>0){/*forkfailed,puterrorhandlinghere*/}/*fork

successful,putremainingcodeforparenthere*/

The following simple program (Ex: ex11/simplefork.c) serves to illustrateprocess creation, concurrent execution, and the relationshipsbetween thechildandtheparentacrosstheforkcall.

/********simplefork.c********/#include>sys/types.h<#include

>unistd.h<#include>stdlib.h<#include>stdio.h<intmain(){pid_t

child_id;child_id=fork();/*processcreation(l)*/if(child_id==

0)/*childcodebegin(2)*/{printf("Child:Mypid=/dandmy

parentpid=/d\n",getpid(),

getppid());_exit(EXIT_SUCCESS);/*childterminates(3)*/}/*child

codeend*/if(child_id>0)/*remainingparentcode*/{

fprintf(stderr,"forkfailed\n");exit(EXIT_FAILURE);

}printf("Parent:Mypid=/d,spawnedchildpid=/d\n",getpid(),

child_id);returnEXIT_SUCCESS;}

Aftercallingfork (line1),yousuddenlyhave twoprocesses, theparentandthechild,executingthesameprogramstartingatthepointwhereforkreturns.Thechildandparentexecutedifferentcodesectionsinourexamplebecause

of the way the program is written. The child only executes the part under if(child_id==0) (line2).At the endof the child code (line3), itmust terminate

11.12

execution.Otherwise,thechildwouldcontinueintothecodemeantonlyfortheparent.The_exitsystemcallisslightlydifferentfromlibraryfunctionexitandisexplainedinSection11.14.Notealsothataprocesscanusethesystemcallsgetpid() and getppid() to obtain the process ID of itself and its parent,respectively.Theaboveprogramproducesthefollowingoutput.

■I/OandProcessControlSystemCallsChild:Mypid=19603andmy

parentpid=19602Parent:Mypid=19602,spawnedchildpid=

19603

Tofurtherillustratetheuseoffork,wecanwriteaprogramwheretheparentand child run concurrently (Ex: ex11/concurrent.c). The child computes thepartial sums, and the parent calculates the partial products, of an array ofintegers.

/********concurrent.c********/#include>sys/types.h<#include

>unistd.h<#include>stdlib.h<#include>stdio.h<#defineDIM8int

main(){pid_tpid;inti,ans,arr[DIM]={1,2,3,4,5,6,7,8};pid=

fork();if(pid==0)/*childcodebegin*/{ans=0;for(i=0;i>DIM;i++){

ans=ans+arr[i];printf("Child:sum=%d\n",ans);sleep(1);/*1

secdelay*/}_exit(EXIT_SUCCESS);}/*childcodeend*/if(pid>0){

fprintf(stderr,"forkfailed\n");returnEXIT_FAILURE;}ans=1;for

(i=0;i>DIM;i++){ans=ans*arr[i];printf("Parent:product

=%d\n",ans);sleep(2);/*2secdelay*/}returnEXIT_SUCCESS;}

Both parent and child have access to their own copies of the array arr, thevariableans,andsoon.Thefactthatbothprocessesareassigningvaluestoansconcurrently does not matter because the programs are running in differentaddressspaces.Thechilddelays1secondaftereachoutputline,buttheparentdelays2seconds,givingeachotherachancetograbtheCPUandrun.Hereisonepossiblesetofoutputbythisprogram.

Child:sum=1Parent:product=1Child:sum=3Child:sum=

6ProgramExecution:execRoutines■305Parent:product=2Child:sum

=10Parent:product=6Child:sum=15Child:sum=21Parent:product=

24Child:sum=28Child:sum=36Parent:product=120Parent:product=

720Parent:product=5040Parent:product=40320

Dependingon the relativespeedofexecutionandothersystem load factors,theoutput lines from theparentand thechildcanbe interleaved inadifferentway.

PROGRAMEXECUTION:EXECROUTINES

A process can load and execute another program by overlaying itself with anexecutablefile.Thetargetexecutablefileisreadinontopoftheaddressspaceof theveryprocess that is executing, overwriting it inmemory, and executioncontinues at the entry point defined in the file. The result is that the processbeginstoexecuteanewprogramunderthesameexecutionenvironmentastheoldprogram,whichisnowreplaced.Thisprogramoverlaycanbeinitiatedbyanyoneoftheexeclibraryfunctions,

includingexecl,execv,execve,andseveralothers,eachavariationofthebasicexecvlibraryfunction.

#include>unistd.h<externchar**environ;intexecv(constchar

^filename,char*constargv[]);

where filename is the full pathname of an executable file, and argv is thecommand-linearguments,withargv[0]beingcommandname.Thisexecvcalloverlaysthecallingprocesswithanewexecutableprogram.If

execv returns, an errorhasoccurred. In this case thevalue returned is -1.Theargument argv is an array of character pointers to null-terminated characterstrings.Thesestringsconstitutetheargumentlisttobemadeavailabletothenewprocess.Byconvention,atleastoneargumentmustbepresentinthisarray,andthefirstelementofthisarrayshouldbethenameoftheexecutedprogram(i.e.,the last component of filename). To the calling program, a successful execvneverreturns.Otherexecfunctionsmaytakedifferentargumentsbutwillworkthesameway

asexecv.Toavoidconfusion,wewillrefertoallofthemasanexeccall.Anexec call is often combinedwith fork to produce a new processwhich

runsanotherprogram.

1. ProcessA(theparentprocess)callsforktoproduceachildprocessB.2. ProcessBimmediatelymakesanexeccalltorunanewprogram.

An exec call transforms the calling process to run a new program. The newprogramisloadedfromthegivenfilenamewhichmustbeanexecutablefile.Anexecutable file is either a binary a.out. or an executable text file containingcommands for an interpreter.An executable text file beginswith a line of theform

■I/OandProcessControlSystemCalls#!interpreter

Whenthenamedfileisanexecutabletextfile, thesystemrunsthespecifiedinterpreter,givingitthenamedfileasthefirstargumentfollowedbytherestoftheoriginalarguments.Forexample,aBashscriptmaybeginwiththeline

#!/bin/bash

andanShscriptwith

#!/bin/sh

Asforanexecutablebinary,LinuxhasadoptedthestandardELF(ExecutableandLinkingFormat)whichbasicallyprovidesbettersupportforthelinkinganddynamicalloadingofsharedlibrariesascomparedtotheoldUNIXa.outformat.Thecommandreadelf-ha.outdisplaystheheadersectionoftheexecutablea.out.Doaman5elftoreadmoreabouttheELFfileformat.Thefollowingattributesstaythesameafteranexeccall:

ProcessID,parentprocessID,andprocessgroupIDProcessownerID,unlessforaset-useridprogramAccessgroups,unlessforaset-groupidprogramWorkingdirectoryandrootdirectorySessionIDandcontrolterminalResourceusagesIntervaltimersResourcelimitsFilemodemask(umask)SignalmaskEnvironmentvariablevalues

Furthermore, descriptorswhich are open in the calling process usually remainopen in the new process. Ignored signals remain ignored across an exec, butsignals thatarecaughtarereset to theirdefaultvalues.SignalhandlingwillbediscussedinSection11.16.

Example:ASimpleShellAs an example, let’s write a program that is a very simple Shell (Ex:ex11/myshell.c)performingthefollowingtasks:

1. Displayingaprompt2. Readingacommandlinefromtheterminal3. Startingabackgroundprocesstoexecutethecommand4. Displayinganotherpromptandgoingbacktostep1

Thiscycleisimplementedbythemainprogram:

/********myshell.c********/#include>sys/types.h<#include

>unistd.h<#include>stdlib.h<#include>stdio.h<#include

>string.h<#defineMAXLINE80intmain(){charcmd[MAXLINE];void

background(char*cmd);for(;;){printf("myshready//");/*Displays

prompt*/fgets(cmd,MAXLINE,stdin);/*Readscommand*/if(

strcmp(cmd,"exit\n")==0)returnEXIT_SUCCESS;

background(cmd);/*Startsbackgroundjob*/}return

EXIT_FAILURE;/*Exitsabnormally*/}

The function background prepares the argv array and starts a child process,whichthencallsexecvtoperformthegivencmdwhilebackgroundreturnsintheparentprocess.

#defineWHITE"\t\n"#defineMAXARG20voidbackground(char*cmd){

char*argv[MAXARG];intid,i=0;/*Tofillinargv*/argv[i++]=

strtok(cmd,WHITE);while(i>MAXARG&&(argv[i++]=strtok(NULL,

WHITE))!=NULL);if((id=fork())==0)/*Childexecutes

backgroundjob*/{execv(argv[0],argv);_exit(EXIT_FAILURE);/*

execvfailed*/}308■I/OandProcessControlSystemCallselseif(

id>0){fprintf(stderr,"forkfailed\n");

perror("background:");}}

Aftertheprogramiscompiledandnamedmysh,runitandenteracommandstringasfollows:

myshmyshready%/bin/ls-l

ThedirectorylistingproducedthiswayshouldmatchtheoneobtainedinyourusualShell.Infact,virtuallyanyLinuxcommandexecutedwithfullpathnamewillbehavethesame.TypeexitfollowedbyENTERtoquitfromthemyshprogram.Theexeclroutineisaconvenientalternativetoexecvwhenthefilenameand

theargumentsareknownandcanbegivenspecifically.Thegeneralformis

intexecl(constchar*name,constchar*arg0,constchar*argn,NULL)

Forexample,

execl("/bin/ls","ls","-l",NULL);

Since fork copies the entire parent process, it is wasteful when used inconjunction with an exec call to create a new execution context. In a virtualmemorysystem,thesystemcall

11.13

intpid;pid=vfork();

shouldbeused in conjunctionwith anexeccall.Unlike fork,vfork avoidsmuchofthecopyingoftheaddressspaceoftheparentprocessandisthereforemuchmoreefficient.However,don’tusevforkunlessitisimmediatelyfollowedbyanexeccall.

SYNCHRONIZATIONOFPARENTANDCHILDPROCESSES

Aftercreatingachildprocessbyfork,theparentprocessmayrunindependentlyorelecttowaitforthechildprocesstoterminatebeforeproceedingfurther.Thesystemcall

#include>sys/types.h<#include>sys/wait.h<pid_twait(int

*t_status);

searches for a terminated child (in zombie state) of the calling process. Itperformsthefollowingsteps:

1. Iftherearenochildprocesses,waitreturnsrightawaywiththevalue-1(anerror).

2. Ifoneormorechildprocessesareinthezombiestate(terminated)already,waitselectsanarbitraryzombiechild,freesitsprocesstableslotforreuse,stores its termination status (Section 11.14) in *t_status if t_status is notNULL,andreturnsitsprocessID.

3. Otherwise,waitsleepsuntiloneofthechildprocessesterminatesandthengoestostep2.

Whenwaitreturnsaftertheterminationofachild,thevariable(*t_status)isset,and it contains information about how the process terminated (normal, error,signal,etc.)Youcanexaminethevalueof*t_statuswithpredefinedmacrossuchas

WIFEXITED(*t_status)(returnstrueifchildexited

normally)WEXITSTATUS(*t_status)(returnstheexitstatusofchild)

Seeman2waitforothermacrosandforadditionalformsofwait.A parent process can control the execution of a child process much more

closely by using the ptrace (process trace) system call. This system call isprimarily used for interactive breakpoint debugging such as that supported bythegdbcommand(Chapter,Section10.8).Whenthechildprocessistracedby

11.14

itsparent,thewaitpidsystemcallisused,whichreturnswhenthespecificchildisstopped(suspendedtemporarily).Let’s look at a simple example of the fork and wait system calls (Ex:

ex11/wait.c).Here the parent process calls fork twice and produces two childprocesses. Each child simply displays its own process ID and terminates. Theparent process calls wait twice to wait for the termination of the two childprocesses.Aftereachwait,theprocessIDandthewaitstatusaredisplayed.

/********wait.c********/#include>sys/types.h<#include

>sys/wait.h<#include>unistd.h<#include>stdio.h<#include

>stdlib.h<intmain(){pid_tpid1,pid2,pid;intstatus;if((pid1=

fork())==0)/*childone*/{printf("childpid=%d\n",getpid());

_exit(EXIT_SUCCESS);}printf("forkingagain\n");if((pid2=fork())

==0)/*childtwo*/{printf("childpid=%d\n",getpid());

_exit(EXIT_FAILURE);}printf("firstwait\n");pid=

wait(&status);printf("pid=%d,status=%d\n",pid,

WEXITSTATUS(status));printf("2ndwait\n");pid=

wait(&status);printf("pid=%d,status=%d\n",pid,

WEXITSTATUS(status));return

EXIT_SUCCESS;WIFEXITED(*t_status)WEXITSTATUS(*t_status)(returns

trueifchildexitednormally)(returnstheexitstatusofchild)}

Notethatthesecondchildinthisexamplereturnsanexitstatus1onpurpose.

PROCESSTERMINATIONEvery running program eventually comes to an end.A processmay terminateexecutioninthreedifferentways:

1. Theprogramrunstocompletionandthefunctionmainreturns.2. Theprogramcallsthelibraryroutineexitorthesystemcall_exit.3. Theprogramencountersanexecutionerrororreceivesaninterruptsignal,

causingitsprematuretermination.

Theargumentto_exit/exitistheprocessexitstatusandispartoftheterminationstatus of the process. Conventionally, a zero exit status indicates normalterminationandnon-zeroindicatesabnormaltermination.Thesystemcallvoid_exit(intstatus)terminatesthecallingprocesswiththefollowingconsequences:

1. AlloftheopenI/Odescriptorsintheprocessarenowclosed.2. Iftheparentprocessoftheterminatingprocessisexecutingawait,thenit

isnotifiedoftheterminationandprovidedwiththechildterminationstatus.3. Iftheterminatingprocesshaschildprocessesyetunfinished,thePPIDsof

11.15

all existing children are set to 1 (the init process). Thus, the new orphanprocessesareadoptedbytheinitprocess.

MostCprogramscall the library routineexitwhichperformsclean-upactionsonI/Obuffersbeforecalling_exit.The_exitisusedbyachildprocesstoavoidpossibleinterferencewithI/Obufferssharedbyparentandchildprocesses.

THEUSERENVIRONMENTOFAPROCESSTheparametersargcandargvofaCprogramreference theexplicitargumentsgiven on the command line. Every time a process begins, another array ofstrings, representing the user environment, called the environment list, is alsopassed to theprocess.Thisprovidesanotherway throughwhich tocontrol thebehaviorofaprocess.Ifthefunctionmainisdeclaredas

intmain(intargc,char*argv[],char*arge[])

thenargereceivesadditionalvaluesfortheenvironmentlistwhichisalwaysavailableforaprocessintheglobalarrayenviron:

externchar**environ

Eachenvironmentstringisintheformname=valueAlthoughdirectaccesstoenvironispossibleinaCprogram,itissimplerto

accessenvironmentvaluesinaCprogramwiththelibraryroutinegetenv:

#include>stdlib.h<char*getenv(constchar*name)TheUser

EnvironmentofaProcess■311

This routine searches the environment list for a string, of the formname=value, that matches the given name and returns a pointer to value orNULLifnomatchfornameisfound.Withgetenvwecanwriteasimpletestprogram(Ex:ex11/envtest.c).

/********envtest.c********/#include>stdlib.h<#include

>stdio.h<intmain(intargc,char*argv[],char*arge[]){char*s;s

=getenv("PATH");printf("PATH=%s\n",s);returnEXIT_SUCCESS;}

You can set environment values at the Shell level.WithBash, a variable isexportedtotheenvironmentasshowninChapter3,Section3.10.Environmentvariablesandtheirvaluesarecontainedintheenvironmentlist.Frequentlyusedenvironment variables include PATH, HOME, TERM, USER, SHELL,

DISPLAY,andsoon(Chapter3,Section3.10).In Bash, we can also pass additional environmental values to any single

commandbysimplylistingthembeforethecommand.Forexample,

gccenvtest.c-oenvtestfoo=bar..../envtest

AttheClevel,theexeclandexecvlibrarycallspasstotheinvokedprogramtheircurrentenvironment.Thesystemcall

#include>unistd.h<intexecve(constchar*file,char*constargv[],

char*constenvp[]);

can be used to pass an environment array envp containing additionalenvironmentalvaluestothenewprogram(Ex:ex11/execve.c).

/*passingenvironmentwithexecve*/#include>unistd.h<#include

>stdlib.h<char*envp[3];intmain(intargc,char*argv[]){

envp[0]="first=foo";envp[1]="second=bar";

envp[2]=NULL;execve("target-program",argv,envp);

exit(EXIT_FAILURE);/*execvefailed*/}

Example:CommandSearchThewhichcommandwhichcmdname...locates the given commandnames (or aliases) on the command searchpath

definedbytheenvironmentvariablePATH(Chapter3,Section3.10).Itdisplaysthe full pathname of each command when located or an error message. Toillustrate theuseofsystemandlibrarycallsfurther,asimplifiedversionof thewhichcommandisimplementedhere.The programmywhich that follows is the same as the which command,

except it takes only one command and no aliases (Ex: ex11/mywhich.c). Theappropriateheaderfilesareincludedatthebeginning:

File:mywhich.cUsage:mywhichcmdname/***/#include>stdio.h<#include

>sys/param.h<#include>unistd.h<#include>string.h<#include

>stdlib.h</*forMAXPATHLEN*//*foraccess*//*forstrncpy*//*

forgetenv*/inthas_command(char*name,char*dir){intans=-l;char

wd[MAXPATHLEN+l];getcwd(wd,MAXPATHLEN+l);if(chdir(dir)==0){

ans=access(name,F_OK|X_OK);chdir(wd);}returnans==0;}/*l*/

/*2*//*3*//*4*/

Before changing, the current working directory is saved (line 1). Note thatgetcwdisalibraryfunctionandnotasystemcall.Ifthedirectoryisaccessible(line2),theexistenceofanexecutablefile,notdirectory,istested(line3).The

11.16

working directory is restored (line 4).The function has_command returns 1 ifthecommandisfound;otherwise,itreturns0.Themainprogramextractsindividualdirectoriesontheenvironmentvariable

PATHandcallshas_commandtolocatethegivencommand:

intmain(intargc,char*argv[]){char*path=getenv("PATH");/*5*/char

dir[MAXPATHLEN+l];intdir_len;char*pt=path;while(

dir_len=strcspn(path,":"))/*6*/{strncpy(dir,path,

dir_len);/*7*/dir[dir_len]='\0';/*8*/if(has_command(argv[l],dir)

){printf("/s//s\n",dir,argv[l]);returnEXIT_SUCCESS;}path+=

dir_len+l;/*9*/}printf("/snotfoundon\n/s\n",argv[l],pt);

returnEXIT_FAILURE;InterruptsandSignals■313}

ThemainprograminitializespathwiththevalueoftheenvironmentvariablePATH(line5).Thefirstdirectoryonpathiscopiedasastringintothevariabledir(line6-8)andisusedinacalltohas_command.Ifthecommandisnotfoundinthisdirectory,pathisadvancedtothenextdirectory(line9)andtheiterationcontinues.

INTERRUPTSANDSIGNALSBasicConceptsWealreadyknowthataprogramexecutesasanindependentprocess.Yet,eventsoutside a process can affect its execution. The moment when such an eventwould occur is not predictable. Thus, they are called asynchronous events.Examplesofsuchevents includeI/Oblocking,I/Oready,keyboardandmouseevents,expirationofatimeslice,aswellasinterruptsissuedinteractivelybytheuser. Asynchronous events are treated in Linux using the signal mechanism.Linuxsendsacertainsignaltoaprocesstosignifytheoccurrenceofaparticularevent. After receiving a signal, a process will react to it in a well-definedmanner. This action is referred to as the signal disposition. For example, theprocessmaybeterminatedorsuspendedforlaterresumption.Thereisasystem-defined default disposition associated with each signal. A process normallyreactstoasignalbyfollowingthedefaultaction.However,aprogramalsohastheabilitytoredefineitsdispositiontoanysignalbyspecifyingitsownhandlingroutineforthesignal.SomeLinuxSignals

Therearemanydifferentsignals.Forinstance,typingCTRL+onthekeyboardusually generates a signal known asquit. Sending the quit signal to a processmakesit terminateandproducesacoreimage filefordebugging.Eachkindofsignal has a unique integer number, a symbolic name, and a default actiondefinedbyLinux.Table11.5showssomeofthemanysignalsLinuxhandles.Acompletelistofallsignalscanbefoundwithman7signal.

SendingSignalsYoumaysendsignalstoprocessesconnectedtoyourterminalwindowbytypingcertaincontrolcharacterssuchasCTRL+,CTRL+C,andCTRL+ZtypedattheShelllevel.Thesesignalsandtheireffectsaresummarizedbelow.

CTRL+CSIGINTterminatesexecutionofforeground

processCTRL+\SIGQUITterminatesforegroundprocessanddumps

coreCTRL+ZSIGTSTPsuspendsforegroundprocessforlaterresumption

Inadditionto thesespecialcharacters,youcanusetheShell-levelcommandkill to send a specific signal to a given process. The general form of thekillcommandiskill[-sig_no]processwhereprocess is a process number (or Shell jobid). The optional argument

specifiesasignalnumbersig_no.Ifnosignalisspecified,SIGTERMisassumedwhichcausesthetargetprocesstoterminate.RecallthatweusedkillinChapter3,Section3.6wherewediscussedjobcontrol.InaCprogram,thestandardlibraryfunctionintraise(intsig_no)(sendssig_nototheprocessitself)isusedbyaprocesstosendthesignalsig_notoitself,andthesystemcall

intkill(pid_tpid,intsig_no)(sendssig_notoprocesspid)is used to send a specified signal to a process identified by the given

numericalpid.

SignalDeliveryandProcessingWhenasignalissenttoaprocess,thesignalisaddedtoasetofsignalspendingdeliverytothatprocess.Signalsaredeliveredtoaprocessinamannersimilartohardwareinterrupts.If thesignalisnotcurrentlyblocked(temporarilyignored)bytheprocess,itisdeliveredtotheprocessbythefollowingsteps:

1. Block further occurrences of the same signal during the delivery andhandlingofthisoccurrence.

2. Temporarily suspend the execution of the process and call the handlerfunctionassociatedwiththissignal.

3. Ifthehandlerfunctionreturns,thenunblockthesignalandresumenormalexecutionoftheprocessfromthepointofinterrupt.

Thereisadefaulthandlerfunctionforeachsignal.Thedefaultactionisusuallyexitingorcoredump(Table11.5).Aprocesscanreplaceasignalhandlerwithahandler function of its own.This allows the process to trap a signal and dealwithitinitsownway.TheSIGKILLandSIGSTOPsignals,however,cannotbetrapped.

SignalTrappingAfter receiving a signal, a process normally (by the default signal handlingfunction)eitherexits(terminated)orstops(suspended).Insomesituations,itisdesirable to react to specific signals differently. For instance, a process mayignore the signal, delete temporary files before terminating, or handle thesituationwithalongjmp.Thesystemcallsigactionisusedtotraporcatchsignals.

#include>signal.h<intsigaction(intsignum,conststructsigaction

*new,structsigaction*old);

where signum is the number or name of a signal to trap. The new (old)structure contains the new (old) handler function and other settings. Thehandlingactionforsignumisnowspecifiedbynew,andtheoldactionisplacedinold,ifitisnotNULL,forpossiblelaterreinstatement.Thestructsigactioncanbefoundwithman2sigaction,butyoubasicallycan

useitinthefollowingway:

structsigactionnew;new.sa_handler=handler_function;

new.sa_flags=0;

Thehandler_functioncanbearoutineyouwriteoronethatisdefinedbythesystem.Ifhandler_functionisSIG_IGN,thesignalissubsequentlyignored.Ifitis SIG_DFL, then the default action is restored. The new handler normallyremains until changed by another call to sigaction. Default actions of somesignals are indicated in Table 11.5. The sa_flags control the behavior of thesignalhandling.Forexample,sa_flags=SA_RESETHANDautomaticallyresetstothedefaulthandlerafterthenewsignalhandleriscalledonce.Wenowgiveasimpleexamplethatusesthesigactionsystemcalltotrapthe

SIGINT(interruptfromterminal)signalandaddsonetoacounterforeachsuchsignalreceived(Ex:ex11/sigcountaction.c).Toterminatetheprogramtypectrl+orusekill-9.

#include>signal.h<#include>stdio.h<voidcnt(intsig){staticint

count=0;printf("Interrupt=/d,count=/d\n",sig,++count);}int

main(){structsigactionnew;structsigactionold;

new.sa_handler=cnt;new.sa_flags=0;sigaction(SIGINT,&new,&old);

printf("BegincountingINTERRUPTs\n");for(;;);/*infiniteloop

*/}316■I/OandProcessControlSystemCalls

If the signal handler function, such as cnt here, is defined to take an intargument(forexample,sig),thenitwillautomaticallybecalledwiththesignalnumber that caused a trap to this function.Of course, counting the number ofsignalsreceivedisof limitedapplication.Amorepracticalexample,cleanup.c,has to dowith closing and deleting a temporary file used by a process beforeterminatingduetoauserinterrupt(Ex:ex11/cleanup.c).

#include>stdio.h<#include>signal.h<#include>stdlib.h<FILE

*tempfile=NULL;charfilename[32];voidonintr(){externFILE*

tempfile;if(tempfile!=NULL){printf("closinganddeleting

%s\n",filename);fclose(tempfile);

unlink(filename);}exit(EXIT_FAILURE);}/*Installsonintr()handler,

ifSIGINTisnotbeingignored*/voidsigtrap(intsig){struct

sigactionnew;structsigactionold;new.sa_handler=SIG_IGN;

new.sa_flags=0;sigaction(SIGINT,&new,&old);if(old.sa_handler

!=SIG_IGN){new.sa_handler=onintr;sigaction(sig,&new,

&old);}}intmain(){externcharfilename[32];externFILE*

tempfile;sigtrap(SIGINT);/*trapSIGINT*/sprintf(filename,

"/tmp/%d",getpid());/*tempfilename*//*opentemporarystream

forreadingandwriting*/tempfile=fopen(filename,"w+");/*

othercodeoftheprogram*/for(;;)sleep(3);/*removetemporary

filebeforetermination*/fclose(tempfile);unlink(filename);

returnEXIT_SUCCESS;}

11.17

11.18

Inthisexample,trappingofSIGINTisdoneonlyifitisnotbeingignored.Ifaprocess runswith its signal environment already set to ignore certain signals,thenthosesignalsshouldcontinuetobeignoredinsteadoftrapped.Forexample,the Sh arranges a background process to ignore SIGINT generated from thekeyboard.IfaprocessproceedstotrapSIGINTwithoutcheckingtoseeif it isbeingignored,thearrangementmadebytheShellwouldbedefeated.Furthermore, as with interactive utilities such as the vi editor, it is often

desirabletousethekeyboardinterrupttoaborttothetoplevelwithinaprogram.This can be easily done by combining signal trapping with the longjmpmechanism(Chapter,Section10.7).Generally,whenthesignalhandlerfunctionreturnsorwhenaprocessresumes

afterbeingstoppedbyCTRL+Z(SIGTTSP),aprocessresumesattheexactpointatwhichitwasinterrupted.Forinterruptedsystemcalls,theexternalerrnoissettoEINTR,andthesystemcallreturns-1.Ifinterruptedwhilereadinginputfromthekeyboard,aprocessmayloseapartiallytypedlinejustbeforetheinterrupt.

FORMOREINFORMATIONFor a list of Linux system calls, see the HTML version of the man page forsyscall,which isasystemcallused tomakeallsystemcalls.Youcanfind theman page from the resources page on the book’s companion website. Theexample code package for this book has an example (Ex: ex11/sysopen.c)demonstratinghowtousesyscall.ThePOSIXstandarddocumentationcanbepurchasedfromIEEE.

SUMMARYAllopenI/OchannelsarerepresentedbyI/Odescriptors.WithI/Odescriptors,the Linux kernel treats file, device, and inter-process I/O uniformly. ThisuniformityprovidesgreatflexibilityandeaseinI/Oprogramming.ForI/O,aCprogrammay use the low-level system calls or the higher level standard I/Olibrary routines. I/O descriptors are identified by small integers. Three pre-opened descriptors 0, 1, and 2 give each process access to the standard input,output,anderroroutput,respectively.Inadditiontoacompletesetoffilemanipulationcalls,Linuxalsooffersaset

oflibraryfunctionsforaccessingdirectories.File-anddirectory-relatedsystemcallsaresummarizedinTable11.6.FileandDirectorySystemCalls

11.19

Linux supports multiprogramming. Processes are created with fork,terminatedwith_exit,overlaidwithanotherexecutableprogramwithexec,andsynchronized with wait. Interrupt signals can be sent from one process toanotherbykilland trappedbysigaction.The*environ[]arraycontainsstring-valuedenvironmentvariablesforaprocesswhichcanbeconsultedwithgetenv.

EXERCISES1. WhatisthedifferencebetweenafiledescriptorandaCfilestream?Please

explain.2. Explaintheeffectoftheumaskvalues077and022.3. Do cat /proc/sys/fs/file-max to see the limit on themaximum number of

openfilesforyoursystem.4. The Linux command pwd displays the current working directory. Write

yourownversionofthiscommand.5. WriteaLinuxcommandtestaccessthattakesanaccessflag(-r,-w,andso

on)andafilenameascommand-lineargumentsandreturnsanexitstatusof0or1dependingonwhetherthespecifiedaccessispermittedornot.

6. WriteaLinuxcommandrmoldthattakesadatestringandremovesallfilesolderthanthegivendateinthecurrentdirectory.Ifthecommandisinvokedwiththe-iflag,thentheprogramwillgointointeractivemodeandaskstheuserattheterminalforapprovalbeforeactuallydeletingafile.

7. Writeyourownversionofasimplecpprogram(filetofile)usinglow-levelI/O.

8. Write a program which will print out the information given by the statsystemcallforeachfilegivenasitsargument.

9. How is a child process produced?How does a parent process obtain thePIDofachildprocess?HowdoesachildobtainthePIDofitsparent?How

doestheparentprocesslearnabouttheterminationofachildprocess?10. WhatisthedifferencebetweentheCexit()functionandthe_exit()system

call?Whereshouldeachbeused?11. Consider the simpleShell inSection .Addawait call to theprogramso

thattheShellwaitsuntilthechildprocesshasfinishedbeforedisplayingthenextprompt.

12. Modify the simple Shell in the previous exercise so that it uses thecommandsearchpath.

13. Writeyourownversionofthesystemlibrarycall.14. Writeaprogramthatprints thevalueof theenvironmentvariablesPATH,

HOME,USER,andTERMandothervariablesspecifiedasargumentsonthecommandline.

15. Write a program nls which is similar to the ls command but which, bydefault,displaysregularfilesanddirectoriesseparately.

16. Writeaprogram,usingamixtureofCandShellcommandsifyouwish,toprovideafacilitywhichtakesaCsourceprogramasinputandgeneratesalistofcorrectlyformattedincludestatementsforsystemheaderfiles.

17. Linux provides the flock system call to aid themanagement ofmutuallyexclusiveoperations.Findouthowthisworksandhowitisusedtoachievemutualexclusion.

18. The Linux system calls semctl, semget, and semop support semaphores.Find out how semaphores work and how they can be used to achievemutualexclusion.

12.1

Chapter12

Inter-processandNetworkCommunication

ThemanyapplicationsdiscussedinChapter7clearlyillustratetheconvenienceandtheenormouspotentialnetworkingcanbring.Herewewilldescribehowtowrite C programs for networking and illustrate how some of the Linuxnetworkingcommandsareactuallyimplemented.As mentioned before, a networking application usually involves a client

processandaserverprocess,residingondifferenthostsoronthesamehost.Atthe C program level, networking simply means communication between suchindependentprocesses.We consider two types of inter-process communication (ipc): ipc between

relatedprocessesandipcbetweenunrelatedprocesses.Forprocessesrelatedbyfork,ipccanbearrangedwithI/Oredirectionandthepipesystemcall.Betweenunrelatedprocesses,ipcisusuallyperformedthroughthesocketmechanism.Aprocesscommunicatesthroughitsownsocketwithanothersocketattached

to a different process. Sockets belong to different address families, and onlysockets within the same address family can communicate with one another.Within the same address family, different types of sockets support differentnetworking protocols. Familiarity with sockets is essential to networkprogramming. The topic is presented in detail, andmany code examples helpillustratehowclientsandserversworktogether.

OPENINGAPROCESSFORI/OIn the previous two chapters,we became familiarwith I/O to/from files usingeitherCstreamsorLinuxkernel filedescriptors,but I/Obetweenprocesses isnot very different. The simplest ipc involves a parent process and a childprocess.Theparentinitiatesthechildtorunsomeprogramandsendsinputtoorreceivesoutputfromthechild.TheStandardCLibraryfunctionpopen

12.2

#include>stdio.h<FILE*popen(constchar*cmd_string,char*mode)

createsachildprocesstoexecutesh-ccmd_stringand establishes a read or write stream (FILE *) to the child. The stream

establishediseitherforreadingthestandardoutputorwritingthestandardinputofthegivencommand,dependingonwhethermodeis"r"or"w".Onceopened,thestreamcanbeusedwithanyoftheStandardCI/OLibrary

functions.Finally,thestreamcreatedbypopencanbeshutdownbyintpclose(FILE*stream)Asanapplicationofpopen,let’swriteasimpleprogramthatisaversionofls,

butlistsonlythenamesofsubdirectoriesinagivendirectory(Ex:ex12/lsdir.c):

/********lsdir.c********/#include>stdio.h<#include>stdlib.h<int

main(intargc,char*argv[]){inti,count,total=0;size_t

len=1024;char*line=malloc(len);if(argc<1)chdir(argv[1]);/*

readsoutputoflscmd*/FILE*in=popen("/bin/ls-ldF*\n","r");

while(getline(&line,&len,in)<0){/*readsonelineofinput

*//*ifadir,displaysline*/if(line[0]=='d')

printf(line);}pclose(in);/*closesstream*/free(line);return

EXIT_SUCCESS;}

The program uses the Linux command ls with the option -ldF to list thecurrent working directory. The output is read, one line at a time, using thestandard library function getline. If a line begins with the character d (adirectory), then it isdisplayedby theparentprocess.Otherwise,we ignore thelineandmoveontothenext.Hereisasampleoutput.drwx——2pwangfaculty40962018-08-0716:49Art/drwx——2pwangfaculty40962018-08-0820:31ex/drwx——2pwangfaculty40962018-08-0716:49info/Thepopen function relies on the basic pipe mechanism which is our next

topic.

IPCWITHPIPEApipe isadirect (inmemory) I/Ochannelbetweenprocesses. It isoftenusedtogether with the system calls fork, exec,wait, and _exit to make multipleprocessescooperateandperformpartsofthesametask.Apipeisaflexibletooltoarrangeipcamongfork-relatedprocesses.AttheShelllevel,youcanconnectcommandsintoapipeline.Thepipecanbe

thought of as a first-in-first-out character buffer (Figure 12.1) with a read

descriptorpointingtooneendandawritedescriptorpointingtotheotherend.Tocreateapipe,thesystemcall

IPCwithpipe■323#include>unistd.h<intpipe(intfildes[2])

isusedwhichestablishesabufferandtwodescriptors:

fildes[0](forreadingthepipe)fildes[1](forwritingthepipe)

Figure12.1PipebetweenProcesses

Thepipe system call is used in conjunction with subsequent fork calls toestablishmultiple processes having access to the same pipe, thereby allowingthemtocommunicatedirectly(Figure12.2).

Figure12.2Pipeafterfork()

Thepipe call returns0 for successor -1 for failure.Consider the followingpieceofcode:

intfildes[2];pipe(fildes);/*settingupthepipe*/if(fork()==0){

close(fildes[1]);/*childwillreadfildes[0]

*/_exit(0);}close(fildes[0]);/*parentwillwritefildes[1]*/

After the fork, both parent and child have their copies of fildes[0] andfildes[1] referring to thesamepipebuffer.Thechildcloses itswritedescriptorandtheparentclosesitsreaddescriptorbecausetheyarenotneededinthiscase.Nowthechildprocesscanreadwhattheparentwritesintothepipe.ToperformI/Othroughapipe,youusethereadandwritesystemcallsonthe

pipefiledescriptors.Thecallreadremovescharactersfromthebuffer,whereaswriteaddsthem.Thecapacityofthepipebufferisusually4096characters,butthe buffer size is system dependent.Writing into a full pipe buffer causes theprocesstobeblockeduntilmorespaceisavailableinthebuffer.Readingmorecharactersthanthereareinthebufferresultsinoneofthefollowing:

1. Returningend-of-file(0)ifthebufferisemptyandthewriteendofthepipehasbeenclosed

2. Returningwhat is left in thepipe if thebuffer isnotemptyand thewriteendofthepipehasbeenclosed

3. Blockingthereadingprocesstoawaitthearrivalofadditionalcharactersifatleastonefiledescriptortothewriteendofthepiperemainsopen

The example (Ex: ex12/p2cpipe.c) below shows a parent process writing themessage"Hellothere,fromme."toachildprocessthroughapipe(Figure12.1).

/********p2cpipe.c********/#include>unistd.h<#include

>stdio.h<#include>stdlib.h<#include>string.h<#include

>sys/wait.h<intmain(intargc,char*argv[]){intp[2];inti,

status;pid_tpid;charbuffer[20];pipe(p);/*settingupthepipe

*/if((pid=fork())==0)/*inchild*/{close(p[1]);/*child

closesp[1]*/while((i=read(p[0],buffer,6))!=0){buffer[i]=

'\0';/*stringterminator*/printf("%dchars%sreceivedby

child\n",i,buffer);}_exit(EXIT_SUCCESS);/*childterminates*/}/*

inparent*/close(p[0]);/*parentwritesp[1]*/write(p[1],"Hello

there,",sizeof("Hellothere,")-1);write(p[1],"fromme.",

sizeof("fromme.")-1);close(p[1]);/*finishedwritingp[1]*/while

(wait(&status)!=pid);/*waitingforpid*/if(status==0)

printf("childfinished\n");elseprintf("childfailed\n");return

EXIT_SUCCESS;}

Afterthefork,bothparentandchildhavethefiledescriptorsp[0]andp[1].Inorder to establish the parent as the sender and the child as the receiver ofcharactersthroughthepipe,thechildclosesitsownp[1]andtheparentclosesitsownp[0].Theparentprocesswritestothepipe"Hellothere"and"fromme."intwoseparatewritecallsandclosesitswritedescriptor(p[1]).Inthemeantime,thechildreadsthepipeanddisplayswhatitgets,sixcharactersatatime(justtoshow multiple read operations). The following output is produced by thisprogram:

6chars:Hello:receivedbychild6chars:there,:receivedby

childIPCwithpipe■3256chars:from:receivedbychild3chars

:me.:receivedbychildchildfinished

By closing its p[1], the parent causes the pipe’swrite end to be completelyclosed—no processes can write to the pipe. This condition causes the finalsuccessfulreadinthechildprocesstoreturnwiththelast3characters.Thenextreadbythechildreturns0,indicatingendoffile.

PipebetweenTwoCommands

Now let’s show how a Shell may establish a pipe between two arbitraryprogramsbycombiningpipe,fork,andexec.Acommandmypipeline takesasarguments twocommandstrings separated

byathefirstcommandtothestandardinputofthesecondcommand.Thus,

mypipeline/bin/ls-l%/bin/greppwang

shouldworkasexpected(sameasls-l|greppwang).Ofcourse,weshalluseapipebetweenthetwoprocesses;oneexecutingthefirstcommandandtheotherthesecond.Thekeyinthisexampleisconnectingstdout inthefirstprocesstothewriteendofthepipeandconnectingstdininthesecondprocesstothereadendofthepipe.Thiscanbeaccomplishedbythedup2systemcall(Figure12.3).

Figure12.3PipeandI/ORedirection

intdup2(intfd,intcopyfd)Dup2duplicatesanexistingI/Odescriptor,fd,whichisasmallnon-negative

integerindexintheper-processdescriptortable.Theduplicateentryismadeinthedescriptor tableatanentryspecifiedby the indexcopyfd. If thedescriptorcopyfd is already in use, it is first deallocated as if a close(copyfd) had beendone first. The value returned is copyfd if the call succeeded; otherwise, theerrorvaluereturnedis-1.After dup2, both fd and copyfd reference the same I/O channel. In the

followingprogram(Ex:ex11/mypipeline.c),dup2isusedtoidentifydescriptor1(inchildone)withthewriteendofapipeanddescriptor0(inchildtwo)withthereadendofthesamepipe.

/********mypipeline.c********/#include>unistd.h<#include

>stdio.h<#include>stdlib.h<326■Inter-processandNetwork

Communication#include>string.h<intmain(intargc,char*argv[]){

intp[2];inti,pid1,pid2,status;argv++;/*loseargv[0]*/for(i=

1;i>=argc;i++)if(strcmp(argv[i],"%")==0){argv[i]=

'\0';/*breakintotwocommands*/break;}pipe(p);/*settingupthe

pipe*/if((pid2=fork())==0)/*childone*/{

close(p[0]);dup2(p[1],1);/*1becomesaduplicateofp[1]

*/close(p[1]);execv(argv[0],argv);/*thiswritesthepipe

*/_exit(EXIT_FAILURE);/*baderrorexecvfailed*/}if((pid1=fork

12.3

12.4

())==0)/*childtwo*/{close(p[1]);dup2(p[0],0);/*0becomesa

duplicateofp[0]*/close(p[0]);execv(argv[i+1],&argv[i+1]);/*

thisreadsthepipe*/_exit(EXIT_FAILURE);/*baderrorexeclfailed

*/}/*parentdoesnotusepipe*/close(p[0]);close(p[1]);while

(wait(&status)!=pid2);/*waitingforpid2*/if(status==0)

printf("childtwoterminated\n");elseprintf("childtwo

failed\n");returnEXIT_SUCCESS;}

BecauseopenI/Odescriptorsareunchangedafteranexeccall,therespectiveprograms in the two stages of the pipeline execute as usual, reading standardinputandwritingstandardoutput,notknowingthatthesedescriptorshavebeendiverted to a pipe. The same principles are used by the Shell to establish apipeline.Aftercompilationintomypipeline,wecanrunthecommand

./mypipeline/bin/ls-l%/bin/fgrep'.c'anditshouldbeentirely

equivalenttols-l|fgrep'.c'

CONNECTINGAFILEDESCRIPTORTOAFILESTREAM

Thedup2systemcallredirectsI/Oatthefiledescriptorlevel.Atthefilestreamlevel,we have seen (Chapter , Section 10.4) the Standard C Library functionfreopen,whichreconnectsanexistingfilestreamtoanotherfile.In addition to these two mechanisms, there is also the standard library

function fdopen, which establishes a stream that connects to an existing filedescriptor.

FILE*fdopen(intfd,char*mode)

Thefunctionfdopenestablishesafilestreamwiththegivenfiledescriptorfd.Themodemustbecompatiblewiththatofthedescriptorfd.The fdopen call is usefulwhen converting an fd into a stream for usewith

Standard C I/O Library functions. For instance, a pipe descriptor can beconnectedtoastreaminthisway.

TWO-WAYPIPECONNECTIONSAsanapplication,let’sseehowaparentprocesscanpasssomeinputtoachildprocess and then receive the resultsproduced.To theparent, thechildprocesssimplyproducesawell-definedresultbasedontheinputgiven.Thedesiredipc

canbeachievedbyestablishinga two-waypipe,anoutgoingandan incomingpipe,betweentheparentandchildprocesses(Figure12.4).

Figure12.4ATwo-WayPipe

The outgoing pipe is used by the parent to send input to the child and theincoming pipe is used to receive results returned by the child. The functionpipe_2way (Ex: ex12/pipe2way.c) is defined for this purpose. Given thecommandstringscmd,pipe_2waywillestablishaprocess torunthecommandand return the quantities piped[0] and piped[1], the read end of the incomingpipeandthewriteendoftheoutgoingpipe,respectively.

intpipe_2way(char*cmd[],intpiped[]){intpid,wt[2],

rd[2];pipe(rd);/*incomingpipe:readbyparent

*/pipe(wt);/*outgoingpipe:writetochild*/if((pid=vfork())==0)/*

inchild*/{close(wt[1]);dup2(wt[0],0);/*0identifiedwith

wt[0]*/close(wt[0]);close(rd[0]);dup2(rd[1],1);/*1identified

withrd[1]*/close(rd[1]);execv(cmd[0],cmd);/*executegiven

command*/perror("execvfailed");/*normallynot

reached*/_exit(EXIT_FAILURE);}/*inparent*/close(wt[0]);piped[1]

=wt[1];close(rd[1]);piped[0]=rd[0];Figure12.4ATwo-WayPipe■

Inter-processandNetworkCommunicationreturn0;}

The returnparameter, piped, is filledwith the twoproperdescriptorsbeforethe function returns. To test pipe_2way, let’s write a program that sendscharacterstothecommandlowercase and receives the transformed stringback.The latter is performed

bythereadlfunction

intreadl(intfd,chars[],intsize){char*tmp=s;while(0>—

size&&read(fd,tmp,1)!=0&&*tmp++!='\n');*tmp='\0';/*string

terminator*/return(tmp-s);}

Nowthemainprogramtotestpipe_2wayis

/********pipe2way.c********//*headers,readl,andpipe_2way

functions*/#defineSIZE256intmain(){intpd[2];char*str[2];char

test_string[]="IPCWITHTWO-WAYPIPE.\n";charbuf[SIZE];char*tmp

=buf;str[0]="./lowercase";str[1]=NULL;pipe_2way(str,pd);/*

writetolowercaseprocess*/write(pd[1],test_string,

strlen(test_string));readl(pd[0],buf,SIZE);/*readlowercase

12.5

process*/printf("Receivedfromlowercaseprocess:\n%s",buf);

returnEXIT_SUCCESS;}

Ifyoucompileandrunthisprogram,

gcclowercase.c-olowercasegccpipe2way.c./a.out

you’llseethedisplay

Receivedfromlowercaseprocess:ipcwithtwo-waypipe.

NETWORKCOMMUNICATIONInter-process communication so far works for processes related by fork.Extendingipc tounrelatedprocessesexecutingondifferenthostsachieves truenetworking.Fornetworkcommunication,independentprocessesmustbeabletoinitiate and/or accept communication requests in an asynchronous manner,whetherthecommunicatingprocessesareonthesamecomputerorondifferenthosts in a network. The standard Linux ipc today was first introduced byBerkeleyUNIXinthe1980s.Theschemeiscenteredonthesocketmechanismand supports the Internet protocols well. Its wide use contributed to theexplosivegrowthoftheInternet.Linuxipcprovidesaccesstoasetofcommunicationdomainscharacterizedby

theirprotocolfamily.Importantipcdomainsare

1. TheLocaldomainusestheLinuxsocket-typefileandthepipemechanismforcommunicationbetweenprocesseswithinthelocalLinuxsystem.

2. The Internet domains IPv4 and IPv6 use the corresponding Internetprotocolsforlocal-remotecommunications.

Other domains, for example, the ATMPVC domain (Asynchronous TransferModePermanentVirtualConnection),exist.The ipc communication domains are characterized by such properties as

addressing scheme, protocols, and underlying communications facilities. Thecentral mechanism is the socket. A socket is an endpoint of communicationwithinaspecificcommunicationdomain.Asocketmaybeassignedaname(thatis, an address) that allows others to refer to it. A process communicates(exchangesdata)throughitsownsocketwithanothersocketinthesamedomain,belonging to a different process.Thus, communication is conducted through apairof cooperating sockets, eachknownas thepeer of theother. In theLocaldomain,socketsarenamedwithfilesystempathnames, forexample, /tmp/soc.In the Internetdomain,a socketaddress ismorecomplicated. It consistsofan

12.6

address family, an IPaddress, and a transport layerportnumber. In the samedomain, different types of sockets use different communications protocols.Processescommunicatethroughsocketsofthesametype.Processesconnectedbysocketscanbeonverydifferentcomputersthatmay

usedifferentdatarepresentations.Forexample,anintis32bitsonsomesystemsbut 64 bits on others. Even when the data sizes agree, systemsmay still useeitherthehighorthelowbytetostorethemostsignificantpartofanumber.Inthisheterogeneousenvironment,dataaresentandreceived,at thesocket level,asasequenceofbytes.Thus,asequenceofASCIIcharacterscanusuallybesentandreceiveddirectlythroughsockets.Othertypesofdataneedtobeserializedinto a sequence of bytes before sending and to be deserialized from a bytesequenceintothelocaldatatypeatthereceivingend.

ClientandServerAsstatedinChapter7,anetworkserviceusuallyinvolvesaserverandaclient.A server process provides a specific service accessible through the networkcommunications mechanism. A client process provides user access to aparticular network service. A well-defined set of conventions must exist togovernhowservicesarelocated,requested,accepted,delivered,andterminated.This set of conventions comprises a protocol that must be followed by bothserverandclient.MostInternetservicesuseprotocolssittingontopofthebasictransportlayer

protocolTCP/IPorUDP/IP.Forexample,HTTP(theWebprotocol)sitsontopofTCP.InternetdomainsocketssupportTCPandUDP.

SOCKETSAsocketisanabstractionthatservesasanendpointofcommunicationwithinanetworkingdomain.Aprogramaccessesipcthroughthesocket.Inotherwords,thesocketistheipcmechanism’sinterfacetoapplicationprograms.Eachsocketpotentially can exchange datawith any other socket within the same domain.Eachsocketisassignedatypeproperty.Differenttypesofsocketsusedifferentprotocols.Thefollowingtypesofsocketsaregenerallysupported:

stream socket—Supports the bidirectional, reliable, sequenced, andunduplicated flow of datawithout record boundaries.When put to use, astreamsocketisconnectedtoanotherstreamsocket,andtheconnectedpairformsatwo-waypipeacrossthenetwork.Eachsocketinthepairiscalledthepeeroftheother.Asidefromthebidirectionalityofdataflow,apairof

connectedstreamsocketsprovidesaninterfacenearlyidenticaltothatofapipe. Within the Local domain, a pair of connected sockets is used toimplement a pipe. Stream sockets in the Internet domain use theTransmissionControlProtocol(TCP/IP).datagram socket—Provides bidirectional flow of data packets calledmessages.The communications channel is not promised to be sequenced,reliable, or unduplicated. That is, a process receiving messages on adatagram socket may find messages duplicated and, possibly, not in theorder in which they were sent. A datagram socket does not have to beconnectedtoapeer.Amessageissenttoadatagramsocketbyspecifyingits address. Datagram sockets closely model the facilities of packet-switchednetworks.Datagramsockets in theInternetdomainuse theUserDatagramProtocol(UDP/IP).rawsocket—Givesaccesstotheunderlyingcommunicationprotocolsthatsupportsocketabstractions.Thesesocketsarenormallydatagramoriented,althoughtheirexactcharacteristicsaredependentontheinterfaceprovidedbytheprotocol.Rawsocketsarenotintendedforthegeneraluser,butforthoseinterestedindevelopingnewcommunicationprotocolsorforgainingaccess to esoteric facilities of an existing protocol. Raw sockets in theInternetdomaingivedirectaccesstotheInternetProtocol(IP).

SocketConstants

The domains and standard socket types are defined in the header file <sys/socket.h>.SomedefinedconstantsforsocketsaregiveninTable12.1.

CreatingSocketsThesocketsystemcall

#include>sys/types.h<#include>sys/socket.h<intsocket(intdomain,

inttype,intprotocol)

isusedtocreateasocketoftheindicatedtypeinthegivendomain.Itreturnsa

descriptorthatisusedtoreferencethesocketinothersocketoperations.Definedconstants (Table12.1)areused tospecify thearguments. If theprotocol is leftunspecified(witha0value),anappropriateprotocolinthedomainthatsupportstherequestedsockettypewillbeselectedbythesystem.Forexample,s=socket(PF_LOCAL,SOCK_DGRAM,0);createsadatagramsocketforusewithintheLocaldomainsupportedbyUDP,

whereasthecalls=socket(PF_INET,SOCK_STREAM,0);createsanInternetstreamsocketsupportedbyTCP.

SocketAddressTypically,aprocessthatprovidesaspecificnetworkservicefirstcreatesasocketin an appropriate domain and of the appropriate type. Then an address isassignedtothesocketsothatotherprocessescanrefertoit.Thesocketaddressisimportantbecauseaclientprocessmustspecifytheaddressofasockettosendamessageormakeaconnection.Therefore,

1. Aserverprocessmustassignitssocketanaddressandmakeitknowntoallpotentialclients.

2. A client processmust be able to obtain the correct socket address of anyserveronanyhost.

Linuxsupportsmanydifferentnetworkingprotocolsandaddressfamilies.HerewewillfocusonlocalipcandtheInternet.

LocalandInternetSocketAddressesAlocalsocketaddressis justapathnameforasocket-typefileinthelocalfilesystem. An Internet socket address combines a host IP address (Chapter 7,Section7.19)andatransport layerportnumber.Standardnetworkservicesareassigned the sameportnumbersoneachhost.The file /etc/servicescontainsalistofservicesandtheirportnumbers.Itlistsonelineforeachservicewithfourfields:

AnofficialnameoftheserviceAuniquetransportlayerportnumberTheprotocoltouseAnyaliases(othernamesfortheservice)

Forexample,theentryssh22/tcp

specifiesthattheSecureShellserviceisatport22andusestheTCPprotocol.Sixteen bits (two bytes) are used for representing a port number. Standard

ports (below 1024) are privileged and their access restricted to widely usedserver programs with the right privilege. Port numbers 1024 and higher arereferredtoasnon-privilegedportsandareusedforotherapplications.Forsocketprogramswrittenbyregularusers,weneedtofindaport that isnotprivilegedandnotusedbyotherwell-knownservicesas listed in /etc/services.TheShelllevelcommand/sbin/sysctlnet.ipv4.ip_local_port_rangedisplayslocalportnumbersthatyoucanuseinsocketprogrammingexercises.

Figure12.5LocalDomainSocketAddressStructure

Datastructuresusedforsocketaddressesare

IntheLocaldomain,asocketaddressisstoredinasockaddr_unstructureusuallydefinedin<sys/un.h>(Figure12.5).In the Internet domain, a socket address is declared by the sockaddr_instructurecontainedin<netinet/in.h>(Figure12.6).

Figure12.6InternetSocketAddressStructure

Inpractice,Internetsocketaddressesareoftenusedinveryspecificways.

A clientmust construct a destination socket address to be used either inmakingaconnection(connect())totheserverorinsending(sendto())andreceiving(recvfrom())datagramswithoutmakingaconnection.Hereisatypical code sequence (minus error checking) for building an Internet

12.7

destinationsocketaddress.structsockaddr_ind—Createssocketaddrstructuredmemset(&d,0,sizeof(d))—Zerosoutthestructured.sin_family=AF_INET—SetsIPaddressfamilystructhostent*hep=gethostbyname(host)—Obtainshostentrystructurememcpy(&d.sin_addr,hep->h_addr,hep->h_length)—CopiesIPaddressintodd.sin_port=getservbyname(service,transport)->s_port—Sets standard portnumber

The IP address of a target host is usually obtained by consulting the domainnameserver(Chapter7,Section7.19)viathegethostbynamecall.Thestandardserviceport is retrievedwith thegetservbyname call (Section1.11).To use anon-standardport,setsin_porttohtons(port_number).

A server, on the other hand,must construct a service socket address andbind it to a socket for the server to receive incoming connections ordatagrams. The typical code sequence for building an Internet servicesocketaddressis

1. structsockaddr_ins—CreatesInternetsocketaddrstructures2. memset(&s,0,sizeof(s))—Zerosoutthestructure3. s.sin_family=AF_INET—SetsIPaddressfamily4. s.sin_port=getservbyname(service,transport)- > s_port—Sets port to

standardportnumber5. s.sin_addr.s_addr=INADDR_ANY—SetsserveraddrtoanylocalhostIP

address

TheconstantINADDR_ANYgetsyoutheIPaddressofthelocalhost.Tobindasocketaddresstoasocket,thesystemcallbind(intsoc,structsockaddr*addr,intaddrlen)isused,wheresocisasocketdescriptor,addr isapointer to theappropriate

addressstructure,andaddrlenisthesizeoftheaddress.Theparameteraddrcanreceivepointersoftypestructsockaddr_un*orstructsockaddr_in*.Let’s look at an example demonstrating Internet stream socket usage in a

clientprogram.

ATCPECHOCLIENTThestandardInternetechoservice isuseful in testingsockets.Theechoserver

canreceivemessagesfromanyclientconnected to itand thensends thatsamemessagebacktowhereitcamefrom.TheechoservicenormallyusesTCPandportnumber7.Theprogramtcp_echo.c isaclientprogramthatconnects to theechoserver

on any particular host and sends it amessage.Youmight say that this is ourHello World example of socket programming. The program is used in thefollowingway:gcctcp_echo.c-otcpEcho./tcpEchohost"AnyMessage"Theprogramstartswith thenecessaryheader filesandahelper function for

exitingonerror(Ex:ex12/tcp_echo).

/********tcpecho.c********/#include>stdio.h<#include

>stdlib.h<#include>sys/socket.h<#include>netinet/in.h<#include

>netdb.h<#include>string.h<#defineBSIZE1024voidQuit(constchar

*err){perror(err);exit(EXIT_FAILURE);}

The main program first checks for correct command-line arguments anddeclaresvariables.

intmain(intargc,char*argv[]){if(argc!=3){fprintf(stderr,

"Usage:%shost\"message\"\n",argv[0]);exit(EXIT_FAILURE);}int

soc;/*socketdescriptor*/charbuf[B_SIZE];structsockaddr_in

cl;/*clientsocketaddr(local)*/memset(&cl,0,sizeof(cl));struct

sockaddr_insr;/*serversocketaddr(remote)*/

Then,itfillseachfieldintheserversocketaddressstructuresrbyfirstzeroingout the structure (line A), assigning the address family (AF_INET for IPv4,line B), finding and setting the standard port number (line C) via thegetservbyname librarycall, andcopying thehost Internetaddressobtainedbygethostbyname (lineD) into the sin_addr fieldof the socket address structure(lineE).SeeSection12.11forinformationonthelibrarycalls.

memset(&sr,0,sizeof(sr));/*(A)*/sr.sin_family=AF_INET;/*

(B)*/sr.sin_port=getservbyname("echo","tcp")-<s_port;/*(C)*/hostent

*hp=gethostbyname(argv[1]);/*(D)*/if(hp==NULL){sprintf(buf,

"%s:%sunknownhost\n",argv[0],

argv[1]);Quit(buf);}memcpy(&sr.sin_addr,hp-<h_addr,hp-

<h_length);/*(E)*/

Withthetargetremoteserveraddresscompleted,theprogramcannowcreatealocalclient-sidesocket(lineF)inthePF_INETprotocolfamilyusingtheTCPprotocol and connect (line G) it to the server socket identified by the socketaddresssrwhichwasjustfilledin(linesA-E).

12.8

UsingDatagramSockets■335/*createssocket*/if(

(soc=socket(PF_INET,SOCK_STREAM,/*(F)*/IPPROTO_TCP))>0){

Quit("Problemcreatingsocket");}/*requestsconnectiontoserver

*/if(connect(soc,(structsockaddr*)&sr,/*(G)*/sizeof(sr))==-1){

close(soc);Quit("client:connect\n");}

Figure12.7TCP/IPSocketConnection

After successful connection of the local socket to the server socket, theprogramcanbegintoread/writethelocalsocketasafiledescriptor(linesHandI).Datawrittentothesocketgetssenttotheremotesocket,anddatasentbytheremotesocketcanbe read from the local socket.Becauseweareconnected tothestandardechoservice,theprogramshouldreadbackwhateverithadsentontotheserverinthefirstplace.

write(soc,argv[2],strlen(argv[2]));/*(H)*/read(soc,buf,

sizeof(buf));/*(I)*/printf("SERVERECHOED:%s\n",buf);

close(soc);returnEXIT_SUCCESS;}

Wecanusethisprogramtoaccesstheechoserviceonanactualhost../tcpEchomonkey.cs.kent.edu"Hereislookingatyou,kid."SERVERECHOED:Hereislookingatyou,kid.Refertotheexamplecodepackageforthecompletetcp_echo.cInternetclient

program.

USINGDATAGRAMSOCKETSTo further illustrate socket communication, let’s look at a simple exampleinvolving a sender process and a receiver process using Internet datagramsockets.The receiver is a server ready andwaiting to receive datagrams fromanysenderclientontheInternet(Figure12.8).

Figure12.8DatagramSocketCommunication

Thereceiverfirstcreatesablanksendersocketaddress.Thenitbuildsitsownsocket address self (line a) using port 8080 (line b) and the IP address of theserver host (INADDR_ANY line c). To run this server yourself, please find ausable UDP port on your host and modify line b accordingly (Ex:ex12/ireceiver.c).

/********ireceiver.c********//**SameheadersandQuit()helper

function**/#defineB_SIZE1024intmain(){structsockaddr_in

sender;memset(&sender,0,sizeof(sender));structsockaddr_in

self;/*(a)*/memset(&self,0,sizeof(self));

self.sin_family=AF_INET;self.sin_port=htons(8080);/*

(b)*/self.sin_addr.s_addr=htonl(INADDR_ANY);/*(c)*/

Nowwecancreateasockettoreceivedatagrams(lined)andbindtheaddressselftoit(linee).

soc=socket(PF_INET,SOCK_DGRAM,IPPROTO_UDP);/*(d)*/n=

bind(soc,(structsockaddr*)&self,/*(e)*/sizeof(self));if(n>

0)Quit("bindfailed\n");

Inaloop,thereceivercallsrecvfrom(linefandSection12.9)towaitforthenextincomingdatagram.Whenitarrives,themessageisreceivedinbuf,andthesendersocketaddressisstoredinthesenderstructure.Therecvfromcallblocksuntilanincomingmessageisreceived.Itreturnstheactuallengthofthemessageoranegativenumber if somethinggoeswrong. Incase thebuffer space is toosmall for the incomingmessage, the rest of themessagemaybediscardedbyrecvfrom.Touse it as a string,weplace a string terminator at the endof themessagereceived(lineg).

intsoc,n,len=0;charbuf[B_SIZE],client[INET_ADDRSTRLEN];

while(1){n=recvfrom(soc,buf,sizeof(buf)-1,/*(f)*/0,(struct

sockaddr*)&sender,&len);if(n>0){close(soc);Quit("recvfrom

failed\n");}buf[n]='\0';/*(g)*/inet_ntop(AF_INET,&

(sender.sin_addr),/*(h)*/UsingDatagramSockets■337client,

INET_ADDRSTRLEN);printf("Receivedfrom%d%s%dchars=%s\n",/*

(i)*/sender.sin_addr,client,—n,buf);if(strncmp(buf,"Stop",

4)==0)break;/*(j)*/}close(soc);returnEXIT_SUCCESS;}

Inthisreceiverexample,weusedtheinet_ntoplibraryfunctiontoconvertthesenderIPaddresstoaquadnotationstringinthecharacterbufferclient(lineh).Thereceiverdisplaystheinformationreceivedtostandardoutput(linei).Inourexample,ifthemessagereceivedstartswith“Stop”,thereceiverwillterminateexecution(linej).We can compile and run the receiver on a selected server host, say,

dragon.cs.kent.edu,andexperimentwith itbysendingmessages to itusing thenccommand(Chapter7,Section7.22):

gccireceiver.c-oireceiver(ondragon)./ireceivernc-u

dragon.cs.kent.edu8080(onanyotherhost)Hereisatest

message.Hereisanothertestmessage.StopCTRL+C

Thedisplaybythereceiverlookslike

Receivedfrom114170912165.25.13.6823chars=Hereisatest

message.

Asanotherexperiment,wecanwriteaclientprogram(isender.c)thatusesthesendto call (Section 12.9) to send datagrams to the receiver. Make sure thereceiver is running, on dragon, say, and then experiment with the sender asfollows.

gccisender.c-oisender./isenderdragon.cs.kent.edu8080Let’s

lookattheprogramisender.c(Ex:ex12/isender.c)./********

isender.c********//**headersandtheQuit()helperfunctions

**/intmain(intargc,char*argv[]){if(argc!=3){

fprintf(stderr,"Usage:%shostport\n",argv[0]);

exit(EXIT_FAILURE);}charbuf[]="Hellothere,itisme.";char

end[]="Stop.";structsockaddr_inreceiver;memset(&receiver,0,

sizeof(receiver));/*(1)*/receiver.sin_family=AF_INET;/*(2)*/338■

Inter-processandNetwork

Communicationreceiver.sin_port=htons(atoi(argv[2]));/*(3)*/struct

hostent*hp=gethostbyname(argv[1]);if(hp==NULL){

sprintf(buf,"%s:%sunknownhost\n",argv[0],

argv[1]);Quit(buf);}memcpy(&receiver.sin_addr,hp-<h_addr,/*(4)*/

Let’slookattheprogramisender.c(Ex:ex12/isender.c).

memcpy(&receiver.sin_addr,hp-<h_addr,/*(4)*/hp-<h_length);

After checking the command-line arguments, the server socket addressstructurereceiverisbuilt(lines1-4).AnInternetdatagaramsocketiscreated(line5)andusedtosendthemessage

12.9

inbuftothereceiversocketaddress(line6).

intsoc=socket(PF_INET,SOCK_DGRAM,0);/*(5)*/intn=sendto(soc,

buf,strlen(buf),0,/*(6)*/(structsockaddr*)&receiver,

sizeof(receiver));if(n>0){Quit("sendtofailed");}

printf("Sender:%dcharssent!\n",n);n=sendto(soc,end,

strlen(end),0,(structsockaddr*)&receiver,sizeof(receiver));

close(soc);returnEXIT_SUCCESS;}

SOCKETI/OSYSTEMCALLSForconnectedsockets, thebasicreadandwrite calls canbeused for sendingandreceivingdata:read(soc,buffer,sizeof(buffer));write(soc,buffer,sizeof(buffer));Eachprocessreadsandwritesitsownsocket,resultinginabidirectionaldata

flowbetweentheconnectedpeers.ThesocketI/Ocallsrecv(soc,buffer,sizeof(buffer),opt);send(soc,buffer,sizeof(buffer),opt);areexclusivelyforstreamsockets. If theargumentopt iszero, thentheyare

thesameasthewriteandread. IfopthastheMSG_PEEKbit turnedon, thenrecvreturnsdatawithoutremovingitsoalaterrecvorreadwillreturnthesamedatapreviouslypreviewed.Thesendtoandrecvfromsystemcallssendandreceivemessagesonsockets,

respectively. They work with any type of socket, but are normally used withdatagramsockets.

intsendto(intsoc,char*buf,intk,intopt,structsockaddr*to,

inttosize)

sends, via the socket soc, k bytes from the buffer buf to a receiving socketspecifiedbytheaddressto.Thesizeoftoisalsogiven.Thetoisapointertoanyvalid socket address, in particular, struct sockaddr_un or struct sockaddr_in.Most current implementations of struct sockaddr limit the length of the activeaddressto14bytes.Theoptparameterspecifiesdifferentoptionsforsendto/recvfromandworks

justliketheoptargumentforsend/recv.Thesendtocallreturnsthenumberofbytessentor-1toindicateanerror.Onthereceivingend,thecall

intrecvfrom(intsoc,char*buf,intbufsize,intopt,struct

sockaddr*from,int*fromsize)

12.10

receives, into the given buffer buf of size bufsize, amessage coming fromanothersocket. Ifnomessagesareavailable, thecallwaitsunless thesocket isnon-blocking (set via the fcntl system call). The peer’s address structure isreturned in *from and its size in *fromsize. The argument from is a resultparameterthatisfilledwiththeaddressofthesendingsocket.Thefromsizeisavalue-resultparameter;itinitiallyshouldcontaintheamountofspacein*from.Onreturn,*fromsizecontainstheactualsize(inbytes)oftheaddress*from.Thenumberofbytesreceivedisthereturnvalueofrecvfrom.

ShuttingDownSocketsTheclosesystemcallcan,ofcourse,beusedonasocketdescriptor:intclose(intsoc)Thereadandwritehalvesofasocketcanalsobeindependentlyclosedwith

theshutdownsystemcall.intshutdown(intsoc,intflag)closesthereadportionifflagis0,thewriteportionifflagis1,andboththe

readandthewriteifflagis2.Whenshutdowniscombinedwiththesocketpaircall,whichcreatestwoconnectedsocketsintheLocaldomain,thepipesystemcallcanbeemulatedexactly.

TCP-BASEDSERVERSWe have seen in Section 12.7 a TCP client that accesses the standard Echoservice.TCP-basedserversusestreamsockets.Astreamsocketisconnectedwithits

peertoformatwo-waypipebetweenaclientandaserver.Aclientprocessusesits socket to initiate a connection to a socketof a serverprocess, anda serverprocessarrangestolistenforconnectionrequestsandacceptsaconnection.Afteraconnectionismade,datacommunicationcantakeplaceusingtheread,write,recv,andsendI/Osystemcalls.Figure12.7illustratesserverandclientstreamsocketconnections.A server process binds a published address to a socket. To initiate a

connection,aclientprocessneedsto

1. Findthecorrectaddressofthedesiredserversocket.2. Initiateaconnectiontotheserversocket.

aswehaveseeninSection12.7.

AcceptingaConnection

Aserverprocesswithastreamsocket(Figure12.9)takesthefollowingstepstogetreadytoacceptaconnection:

1. CreatesasocketintheappropriatedomainoftypeSOCK_STREAM.2. Constructsthecorrectserversocketaddress,andbindsittothesocket.3. Indicatesawillingnesstoacceptconnectionrequestsbyexecutingthelisten

systemcall.4. Usestheacceptcalltowaitforaconnectionrequestfromanyclientandto

establishaconnection(Figure).

Figure12.9StreamSocketConnections

Thecallintlisten(intsoc,intn)initializesthesocketsocforreceivingincomingconnectionrequestsandsets

the maximum number of pending connections to n. After the listen call, theacceptcall

intaccept(intsoc,structsockaddr*addr,socklen_t*addrlen)

accepts connections on the stream socket soc on which a listen has beenexecuted. If therearependingconnections,acceptextracts thefirstconnectionrequestonthequeue,createsanewsocket(say,ns)withthesamepropertiesassoc,connectsthenewsocketwiththerequestingpeer,andreturnsthedescriptorofthisnewsocket.Theconnectionlisteningsocketsocremainsreadytoreceiveconnectionrequests.If no pending connections are present on the queue and the socket is not

markedasnon-blocking(say,withthefcntlsystemcall),acceptblocksuntilaconnection request arrives. If the socket is marked as non-blocking and nopendingconnectionsarepresentonthequeue,acceptwillreturnanerrorinsteadofblocking.Theacceptedsocket,ns,isusedtocommunicatewithitspeerandmaynotbe

used to accept additional connections. The argument addr is filled with theaddressoftheconnectedpeer.Again,theaddrlenisavalue-resultparameter.

AnExampleTCP/IPServerLet’s look at an example server (Ex: ex12/inetserver.c) that uses TCP/IP andforkschildprocessestotakecareofclientswhiletheparentprocesscontinuestomonitorincomingconnectionrequests.Theprogrambeginsbycheckingcommand-lineargumentsandpreparingthe

peerandselfsocketaddressstructures(linesuptoI).

TCP-BasedServers■341intmain(intargc,char*argv[]){if(argc

!=2){fprintf(stderr,"Usage:%sport\n",argv[0]);

exit(EXIT_FAILURE);}intsoc,ns;structsockaddr_inpeer;int

peer_len=sizeof(peer);memset(&peer,0,

sizeof(peer));peer.sin_family=AF_INET;structsockaddr_in

self;memset(&self,0,

sizeof(self));self.sin_family=AF_INET;self.sin_addr.s_addr=

htonl(INADDR_ANY);self.sin_port=htons(atoi(argv[1]));/*(I)*//*set

uplisteningsocketsoc*/if((soc=socket(PF_INET,SOCK_STREAM,

0))>0){Quit("server:socket");}if(bind(soc,(struct

sockaddr*)&self,sizeof(self))==-1){close(soc);

Quit("server:bind");}/*(II)*/listen(soc,1);/*(III)*//*accept

connectionrequest*/intpid;while((ns=accept(soc,(struct

sockaddr*)/*(IV)*/&peer,&peer_len))<=0){if((pid=fork())

==0)/*(V)*/action(ns,&peer);}close(soc);Quit("server:accept");}

Aftercreatingtheserversocketsocandbindingthelocaladdresstoit(lineII),we begin listening (line III) and accepting incoming connections (line IV) onsoc.Whenacceptreturns,weforkachildprocesstoperformtheservice(lineV),

definedentirelybytheactionfunction.Theparentcallsacceptagainforthenextconnection.The action function repeatedly reads the incoming data, echos it back, and

displaysthedatareceived(lineVI).Whenthechildisdone, itcalls_exit(lineVII).

/*Performsservice*/intaction(intns,structsockaddr_in*peer){

intk;charbuf[256];char*client[INET_ADDRSTRLEN];

inet_ntop(AF_INET,&(peer-<sin_addr),client,INET_ADDRSTRLEN);

while((k=read(ns,buf,sizeof(buf)-1))<0)/*(VI)

*/{buf[k]='\0';printf("SERVERid=%dRECEIVEDFROM%s:%s\n",

getpid(),client,buf);write(ns,buf,k);342■Inter-processand

NetworkCommunication}printf("Child%dDone.\n",getpid());

close(ns);_exit(EXIT_SUCCESS);/*(VII)*/}

Runthisprogram,say,onport4900,by

gccinetserver.c-omyecho./myecho4900

12.11

andconnecttoitwith

nclocalhost4900nchost4900

Theexamplecodepackagecontainsthecompleteinetserver.cprogram.

NETWORKLIBRARYROUTINESLinuxprovidesa setof standard routines in the Internet networking library tosupportnetworkaddressmapping.Theseroutines,withthehelpoftheDNSanddatafilessuchas/etc/servicesand/etc/hosts,returnCstructurescontainingtheneeded information. Routines are provided for mapping domain names to IPaddresses, service names to port numbers and protocols, network names tonetworknumbers,andsoon.Wehaveseensomeuseofthesealready.Nowwewilldescribetheseroutinesinmoredetail.The header file < netdb.h > must be included in any file that uses these

networkinglibraryroutines.Forinstance,thelibraryfunction

#include>netdb.h<structhostent*gethostbyname(constchar*host)

consults theDNSand returnsapointer toahostent structure for thehostasfollows:

structhostent{char*h_name;/*officialnameof

host*/char**h_aliases;/*aliases*/inth_addrtype;/*addresstype:

PF_INET*/inth_length;/*lengthofaddress*/char**h_addr_list;/*IP

addresses(fromnameserver)*/};

ANULLpointerisreturnedforerror.Thehostargumentcanbegiveneitheras a domain name or as an IP address. In the latter case, no DNS query isnecessary.For example, to obtain the IP address of a host with the name

monkey.cs.kent.edu.,usestructhostent*hp;

hp=gethostbyname("monkey.cs.kent.edu.");

andthenumericalIPaddressisin

hp-<h_addr_list[0]/*IPaddress*/DaemonProcesses■343

whichcanbecopied into the sin_addr fieldofa sockaddr_in structure foratarget socket. If a partial domain name such as monkey is given, then it is

12.12

interpreted relative to the Local domain. The IP address is stored as bytes innetwork byte order: byte 0 is the most significant and byte 4 is the leastsignificant. This order is commonly known as big endian. The network byteordermayormaynotbethesameasthehostbyteorderusedtostorelongs,ints,and shorts on a particular computer system. There are big endian and littleendianCPUs.Thelibraryroutinehtonl(htons)isusedtotransformanunsignedint(unsignedshort)fromhost tonetworkorder.Theroutinentohl(ntohsdoestheopposite.Todeterminetheportnumberforstandardnetworkservices,usestructservent*getservbyname(constchar*service,constchar*proto)whichreturnstheportnumberofthegivenservicewiththegivenprotocolina

serventstructure:

structservent{char*s_name;char**s_aliases;ints_port;char

*s_proto;};

ANULLpointerisreturnedforerror.Forexample,

structservent*sp;sp=getservbyname("ssh","tcp");

getssp->s_porttobe22(afterconversionbyntohs),thedesignatedportfortheSSHoverTCPservice.Similar sets of library functions are provided to access the network and

protocoldatabases.Examplesaregetnetbynameandgetprotobyname.

DAEMONPROCESSESOnLinux,therearemanyhiddenprocessesthatworkquietlyinthebackgroundtoperformavarietyoftasksasthoughbymagic.Thesearetheso-calleddaemonprocesses, and they run concurrently with other active user processes. Forexample,

Thecrondaemon(usually/usr/sbin/crond)executescommandsatspecifieddatesandtimesscheduledthroughthecrontabcommand(Section).ThehttpdWeb server (usually /usr/sbin/httpd) is a daemon that handlesHTTPrequests(Chapter9).Several daemons, including rpc.nfsd, rpc.lockd, rpc.statd, andrpc.mountdprovidetheNetworkFilesystem(NFS)service(Section).Thenamed(usually/usr/sbin/named)istheInternetDNSserver(Section).Thesendmaildaemon(usually/usr/sbin/sendmail-bd)istheInternetemail

server.Thesshddaemon(usually/usr/sbin/sshd)isthesecureShellloginserver.The SystemControl daemon systemd performs system booting and, afterthat,manages systemprocesses.The systemctl command is supportedbysystemd.

Manyothernetwork serversnot listedhere runasdaemons,but therearealsoservers,suchastheXWindowserver,thatarenotconsidereddaemons.Newerworkstationshavemultiplehardwareprocessorstoexecuteseveralprocessesinparallel,resultingingreatlyincreasedsystemspeed.

ProgrammingaDaemonDaemon programs such as sshd, httpd, and sendmail -bd have these fourimportantcharacteristics:

1. Adaemonneverexits.2. Adaemonhasnocontrolterminalwindow.3. AdaemondoesnotusestandardI/O.4. Asystemdaemonisnormallystartedatboottime,iscontrolledbytheinit

process(process1),andcanberestartedifitdiesforsomereason.

InChapter9,Section9.6wepresentedhowaLinux is configured to start theApacheWebserveratboottime.Followthesameprocedureforotherservers.A process can disassociate itself from its control terminalwindowwith the

systemcallsetsid().

#include>unistd.h<pid_tsetsid(void);

The call creates a new session and a newprocess group. It sets the callingprocessasthesessionleaderandtheprocessgroupleader.Nocontrolterminalisassignedyet.Thecallingprocess is theonlyprocess in thenewprocessgroupandtheonlyprocessinthenewsession.Thus, a daemon process often executes the sequence in Figure 12.10 to

disassociateitselffromthecontrolterminalandtheparentprocess.

Figure12.10DisassociatingfromControlTerminalWindow

12.13

Onceorphaned,thedaemonprocessiscontrolledbytheinitprocess.

INPUT/OUTPUTMULTIPLEXINGProgramssuchas thehttpdand theXWindowserver require thecapability tomonitorormultiplexanumberofI/Odescriptorsatonce.On-linechatprogramsaregoodexamples.TheyneedtodealwithmanyI/Ochannelssimultaneously.Theselectsystemcallprovidesageneralsynchronousmultiplexingscheme.

#include>sys/select.h<intselect(intnfds,fd_set*readfds,

fd_set*writefds,fd_set*exceptfds,structtimeval*timeout)

The select call monitors the I/O descriptors specified by the bit masks*readfds,*writefds,and*exceptfds.Itchecksifanyofthe*readfdsisreadyforreading;ifanyofthe*writefdsisreadyforwriting;andifanyofthe*exceptfdshasanexceptionalconditionpending.Eachmasthasbit0 throughnfds-1.ThenthbitofamaskrepresentstheI/Odescriptorn.Thatis,ifbitnofamaskis1,thenfiledescriptornismonitored.Forexample,if*readfdshasthevalue1(a1in bit position 0), then I/O descriptor 0 is monitored for data available forreading.Thecallreturnswhenitfindsatleastonedescriptorready.Whenselectreturns, the bit masks are modified to indicate (in the same manner) the I/Odescriptors that are ready. The integer value returned by select is the totalnumberofreadydescriptors.The parameter timeout is a non-zero pointer specifying a maximum time

interval to wait before select is to complete. To affect a poll, the timeoutargument should be non-zero, pointing to a zero-valued timeval structure. Iftimeout is a zero pointer, select returns only when it finds at least one readydescriptor. The code fragment in Figure 12.11 is an example where selectmonitorsusingatwo-secondtimeout.

Figure12.11I/OMultiplexing

#include>sys/select.h<structtimevalwait;intfd1,fd2,read_mask,

nready;wait.tv_sec=2wait.tv_usec=0;read_mask=(1>>fd1)|(1

>>fd2)nready=select(32,(fd_set*)&read_mask,0,0,&wait);

Theintmaskscanaccommodatedescriptors0through31.Differentmethodsareusedtohandlealargernumberofdescriptors.Oneistouseseveralintsforamask.Linuxsystemsmaynotworkinthesamewayinthisregard.Let’slookataserverthatmonitorsastreamandadatagramsocketwithselect

(Ex:ex12/selectExample.c).

#include>stdlib.h<346■Inter-processandNetwork

Communication#include>sys/types.h<#include>sys/socket.h<#include

>sys/select.h<#include>netinet/in.h</*Internetdomainheader

*/#defineSERVER_PORT03900#defineSERVER_PORT13901intmain(){

intsoc_s,soc_d,s_mask,d_mask,read_mask,nready;/*setup

listeningsocketsoc*/structsockaddr_inaddr0=

{AF_INET};addr0.sin_addr.s_addr=htons(SERVER_PORT0);struct

sockaddr_inaddrl={AF_INET};addr0.sin_addr.s_addr=

htons(SERVER_PORT1);soc_s=socket(AF_INET,SOCK_STREAM,

0);/*A*/soc_d=socket(AF_INET,SOCK_DGRAM,0);if(soc_s>0||

soc_d>0){perror("server:socket");exit(EXIT_FAILURE);}if

(bind(soc_s,(structsockaddr*)&addr0,sizeof(addr0))==-1||

bind(soc_d,(structsockaddr*)&addr1,sizeof(addr1))==-1)

{perror("server:bind");exit(EXIT_FAILURE);}listen(soc_s,

3);/*B*//*monitorsockets*/s_mask=1>>soc_s;d_mask=1>>

soc_d;/*C*/for(;;){read_mask=s_mask|d_mask;/*D*/nready=

select(2,(fd_set*)&read_mask,0,0,0);/*E*/while(nready)/*F*/{

if(read_mask&s_mask){nready—;do_stream(soc_s);/*G*/}elseif

(read_mask&d_mask){nready—;do_dgram(soc_d);/*H*/}}/*endof

while*/}/*endoffor*/}

Thestreamsocketsoc_sandthedatagramsocketsoc_darecreated,boundtocorrectaddresses, andmade ready to receive input (linesA–B).After thebitmasksaresetcorrectlybybitshiftingoperations(lineC),theprogramgoesintoan infinite loop to monitor these two sockets (line D). When select (line E)returns, each of the ready descriptors is treated in a while loop (line F) andmonitoringisresumed.The functions do_stream (line G) and do_dgram (line H) each handle a

differentkindofreadysocket.A similar system call pselect is also available, which allows you to block

signalswhilemultiplexingI/O.

12.14TCPOUT-OF-BANDDATATCP/IPsocketssupporttwoindependentlogicaldatachannels.Normaldataaresent/received in-band, buturgentmessages can be communicated out-of-band(oob).Ifanabnormalconditionoccurswhileaprocessissendingalongstreamofdatatoaremoteprocess,itisusefultobeabletoalerttheotherprocesswithanurgentmessage.Theoobfacilityisdesignedforthispurpose.Out-of-band data are sent outside of the normal data stream and received

independently of in-band data. TCP supports the reliable delivery of only oneout-of-bandmessage at a time. Themessage can be amaximum of one bytelong.Whenanoobmessage isdelivered toa socket,aSIGURGsignal isalsosent to the receiving process so it can treat the urgent message as soon aspossible.Thesystemcalls,send(soc,buffer,sizeof(buffer),opt);recv(soc,buffer,sizeof(buffer),opt);withtheMSG_OOBbitofoptturnedon,sendandreceiveout-of-banddata.

Forexample,aTCP/IPclientprogramcanusethecode

send(soc,"B",1,MSG_OOB);

tosendtheone-characterurgentmessageBtoapeersocket.Totreatoobdata,areceivingprocess trapstheSIGURGsignal(Chapter11,

Section11.16) and supplies a handler function that reads the out-of-band dataandtakesappropriateaction.Forexample,thefollowingcodedefinesafunctionoob_handlerwhichreadstheoobdata.

intoobsoc;voidoob_handler(){charbuf[1];ssize_tk;k=

recv(oobsoc,buf,sizeof(buf),MSG_OOB);if(k<0){/*process

urgentmsg*/}}

Totreatsignalssentviaoob,forexample,thishandlerfunctioncancheckthereceivedmessagetoseewhichoobbyteisreceivedandusekill(SIGXYZ,getpid());tosendsomesignaltoitself(Ex:ex12/oob.c).TheSIGURGsignal,indicatingpendingoobdata,istrappedwith

#include>signal.h<#include>fcntl.h<structsigactionnew;struct

sigactionold;oobsoc=ns;/*nsisInternetstreamsocket

*/new.sa_handler=oob_handler;new.sa_flags=0;sigaction(SIGURG,&new,

&old);348■Inter-processandNetworkCommunication

Toensurethattheprocessisnotifiedthemomenturgentoobdataarrives,the

12.15

12.16

followingcodesshouldalsobeexecuted:

#include>unistd.h<#include>fcntl.h<if(fcntl(ns,F_SETOWN,

getpid())>0){perror("fcntlF_SETOWN:");_exit(EXIT_FAILURE);}

ThecoderequeststhatwhenaSIGURGassociatedwiththesocketnsarises,itissenttotheprocessitself.ThefcntlfilecontrolcallsetstheprocesstoreceiveSIGIOandSIGURGsignalsforthefiledescriptorns.You’ll find a program (Ex: ex12/inetserverOOB.c) in the example code

packagewhichaddstheout-of-banddatacapabilitytotheinetserver.cprogram.

FORMOREINFORMATIONConsult section 7 of the Linux man pages for all supported socket addressfamilies. For AF_INET see ip(7), for AF_INET6 see ipv6(7), for AF_UNIX(same as AF_LOCAL) see unix(7), for AF_APPLETALK see ddp(7), forAF_PACKETseepacket(7),forAF_X25seex25(7),andforAF_NETLINKseenetlink(7).ForLinuxkernelsocketsupportseesocket(7).For networking and network protocols see Computer Networking: Internet

ProtocolsinActionbyJeannaMatthews(Wiley).ForNetworkingonLinuxseeAdvanced Guide to Linux Networking and Security by Ed Sawicki (CourseTechnology).

SUMMARYLinux supports networking applications by providing a set of system-levelfacilitiesforipcamongdistributedprocesses.Networkservicesoftenuseaclientandservermodelwhereserverprocessesprovidespecificservicesaccessedbyclientprogramsthatactasuserorapplicationinterfaces.Differentsockettypessupport different networking protocols. Clients access servers by locating theserver’ssocketaddressandinitiatingarequest.The ipc hinges on the socket mechanism, which serves as endpoints for

communicationwithinanyspecificcommunicationdomain.TheLocaldomainandtheInternetdomainareusuallysupportedonLinux.Theformerisusedforcommunicationwithin the local Linux system. The latter supports the variousInternet protocols that exist in the Internet protocol family, including IP, TCP,andUDP.There are several typesof sockets.Streamsockets are connected in pairs to

supportabidirectionalcommunicationschannel,whichcanbelikenedtoatwo-

12.17

waypipe.Datagramsocketsmayormaynotbeconnectedandcansend/receivemessages similar to data packets. Raw sockets give access to the underlyingcommunicationprotocols that support socketabstractions.Rawsocketsarenotintended for the general programmer. A process uses its own socket tocommunicate across the networkwith a socket belonging to a remote process(the peer). The two socketsmust be of the same type. TheDNS and a set ofnetworking system calls combine to retrieve network addresses and serviceports. Library routines make it straightforward to find and construct socketaddressesinaprogram.Network server programs may run as daemon processes, divorced from

controlterminalwindowsandstandardI/O,torunconstantlybutquietlyinthebackground.MonitoringI/OwithselectorpselectenablesthemultiplexingconcurrentI/O.

Out-of-band data, supported by Internet stream sockets, can be used to sendurgentmessagessuchasinterruptstopeersockets.

EXERCISES1. Thesystem orpopen call executes an sh command.Howwould you get

suchacalltoexecuteacommandstringfortheBashShell?2. Is itpossible foraparentprocess tosenddata to thestandard inputof its

child?How?Is itpossible foraparentprocess to receiveoutput fromthestandardoutputofachildprocess?How?

3. RefertotheHellotherepipeexampleinSection12.2.Whatwouldhappenif the child did not close its descriptor p[1]?What would happen if theparentdidnotcloseitsdescriptorp[1]?

4. WriteaCfunctionpipe_std("Shell-command-string")whichcreatesachildprocess to execute any given regular Linux command. Furthermore, itconnects the filedescriptors0and1of thecalling (parent)process to thecorresponding descriptors of the child process.The usage of the pipe_stdfunctionisasfollows:

In the parent process, a call to pipe_std is made with a specificcommand string. This sets up the two-way pipe between the parentprocessandthechildprocess.Then,pipe_stdreturns.Nowintheparentprocess,filedescriptor0readsthestandardoutputofthechildprocess,andoutputtofiledescriptor1isreadasstandardinputbythechildprocess.Thisallowstheparentprocesstofeedinputtothechildprocessandcollectthechild’soutput.

Afterinteractionwiththechildprocessisover,theparentprocesscallsend_pipe_write(); end_pipe_read(); two additional functionsassociatedwithpipe_std, torestoretheparent’sfiledescriptors0and1.Since the parent process and the child process can form a circularproducer-consumer relationship, the danger of deadlock is alwaysthere.Itistheparentprogram’sresponsibility(notthatofpipe_std)toguardagainstdeadlock.

5. Whatdifferent systemcalls canbeused to read/write a socket?What aretheirdifferences?Includecallsnotcoveredinthetext.

6. Write a lowercase server that takesmessages from a client and turns alluppercasecharactersintolowercasebeforeechoingthemessagebacktotheclient.ImplementtheserviceusinganInternetdatagramsocket.

7. DothepreviousproblemwithanInternetstreamsocket.8. Addcodetoyourlowercaseserverthatcheckstheaddressandportnumber

oftheclientsocketandonlyacceptsrequestsfrom“allowable”clients.9. Use the out-of-bandmechanism of Internet stream sockets to sendLinux

signalstoaremoteprocess.10. Write a command serviceIP that takes a service name, such as ftp and a

hostname,suchasmonkey.cs.kent.edu,anddisplaystheIPaddressandportnumber.

11. Maxima isapowerfulprogramformathematicalcomputations. Install themaximapackage ifyourLinuxdoesnotalreadyhave it,and thenmake itintoanInternetserver.

12. Write achat applicationwheremultiple people can join in the same chatsession on different hosts. This problem requires a clear overview of theproblemandacarefuldesignbeforeimplementation.

AppendicesOnline

Theappendicesareonlineatthebook’swebsite(mml.sofpower.com)whereyoucanalsofindinformationupdatesandmanyotherusefulresources.

Appendix:SettingUpYourOwnLinuxforLearningSeemultiplewaystosetupyourownLinuxforeffectivelearning,includingonyourownWindows®orMac®laptop/desktop.

Appendix:SecureCommunicationwithSSHandSFTPSSH is a secure remote login program. It lets you log in and access a remotecomputer.SFTPisasecurefiletransferprogramthatallowsyoutouploadanddownloadfilestoandfromanothercomputer.

Appendix:PatternProcessingwithawkTheawk program is a powerful filter. It processes input one line at a time,applyinguser-specifiedawkpatternactionstoeachline.

Appendix:HowtoUSEvimCreatingandeditingtextfilesisbasictomanytasksonthecomputer.TherearemanytexteditorsforLinux,butvim(viiMproved)isavisualinteractiveeditorpreferredbymany.

Appendix:TextEditingwithviIn-depthcoverageof text editingconcepts, techniques, andmacroswith thevieditorareprovided.

Appendix:ViQuickReferenceMany editing commands are available undervi, and this quick reference cardcanbehandy.

Appendix:TheemacsEditor

Rather than operating in distinct input and command modes like vi, emacsoperatesinonlyonemode:Printablecharacterstypedareinsertedatthecursorposition.CommandsaregivenascontrolcharactersorareprefixedbyESCorctrl+x.

[1]

[2]

[3]

[3]

[5][6]

[7]

[8][9]

[10]

[11]

[12]

Bibliography

BlumRichard.LinuxCommandLineandShellScriptingBible.NewYork,NY,USA:JohnWiley&Sons,Inc.;2008.BovetDaniel P,CesatiMarco.Understanding theLinuxKernel. 3rd ed.California,USA:O’Reilly;2005.SoyinkaWale.Linux Administration: A Beginner’s Guide. 7th ed. New York, USA:McGraw-HillEducation;2015.LoveRobert. Linux Kernel Development. 3rd ed. Indianapolis, Indiana, USA: Addison-WesleyProfessional;2010.SchroderCarla.LinuxNetworkingCookbook.California,USA:O’Reilly;2007.SieverEllen, FigginsStephen,LoveRobert,RobbinsArnold.Linux in aNutshell. 6th ed. California,USA:O’Reilly;2009.MarkG.Sobell.APracticalGuidetoLinuxCommands,Editors,andShellProgramming,2ndEd.,PrenticeHall,NewJersey,USA,2009.SobellMarkG.APracticalGuidetoUbuntuLinux.3rded.NewJersey,USA:PrenticeHall;2010.Steidler-DennisonTony.RunYourOwnWebServerUsingLinux&Apache.Collingwood,Victoria,AU:SitePoint;2005.WangPaulS.DynamicWebProgrammingandHTML5.Florida,USA:Chapman&HallCRCPress;2012.WardBrian.HowLinuxWorks:WhatEverySuperuserShouldKnow.SanFrancisco,CA,USA:NoStarchPress;2004.YankKevin. Build Your Own Database Driven Web Site Using PHP & MySQL. Collingwood,Victoria,AU:SitePoint;2009.

WebsiteandExampleCodePackage

WebsiteThebookhasawebsiteusefulforinstructorsandstudents:http://mml.sofpower.comYoucanfindtheappendicesforthetextbookatthesite.Thesitealsooffersacompleteexamplecodepackagefordownloading,informationupdates,resources,orderinginformation,anderrata.ExampleCodePackageAllexamplesinthisbook,andafewmore,arecontainedinacodeexamplepackage.1Theentirepackagecanbedownloadedfromthewebsiteinonecompressedfile,MasteringModernLinux.tgzorMasteringModernLinux.zip.Thedownloadaccesscodeis2018MML.Thepackagecontainsthefollowingfilesanddirectories

ex01/ex03/ex05/ex07/ex09/ex11/guide.pdfex02/ex04/ex06/ex08/ex10/ex12/license.txtUnpacking

1. Placethedownloadedfileinanappropriatedirectoryofyourchoice.2. Go to that directory and, depending on the downloaded file, use one of

thesecommandstounpack

tarzxpvfMMLCode.tgztarjxpvfMMLCode.tbztarJxpvfMMLCode.txzunzipMMLCode.zip

ThiswillcreateafolderMMLCode/containingtheexamplecodepackage.

Index

*(Bashvariable),115.(Bashcommand),71.(directoryselfpointer),18..(parentdirectory),18/etc/passwd(File),77?(Bashvariable),118#(Bashvariable),114AAt(Bashvariable),82,115abort(libraryfunction),313accept(systemcall),340access(systemcall),294accesscontrol,DAC,154MAC,154ActiveServerPages(ASP),192Admincommand,apt,202aptitude,203ausearch,221chcon,221chgrp,152chown,152dig,207dnf,202fdisk,209firewall-cmd,208firewall-config,208fixfiles,222fsck,211gdisk,209getenforce,219gnome-disks,210

gnome-system-monitor,204gparted,209groupadd,199groupdel,199groupmod,199ifconfig,207killall,204lgroupadd,200lsblk,209luseradd,200lvcreate,212lvdisplay,212mkfs,211mknod,157mount,211nm-connection-editor,207nmcli,207parted,209,211pgrep,204pidof,204ping,207pkill,204ps,204,301pstree,204pvcreate,212pvdisplay,212quotacheck,214restorecon,222route,207sealert,221secon,222seinfo,220sestatus,218,222setenforce,219su,199sudo,201system-config-services,204systemctl,204,228top,204

traceroute,207useradd,199userdel,199usermod,199vgcreate,212vgdisplay,212visudo,200whois,207yumex-dnf,203

Adminuser,199AdvancedEncryptionStandard(AES),176alarm(libraryfunction),313alias(Bashcommand),67alias(Bashexpansion),67Aliases,domainname,170Apache,HTTPSsupport,239–240Apacheserver,186ApacheWebserver,227–236configuration,229–236directives,230–231filedeflation,237–238install,228–242modules,231PHPmodule,243–245

apache2ctl(regularcommand),229apachectl(regularcommand),229App,3Dprinting,94cloudstorage,93–94documentformatting,89–90drawinganddiagramming,90–91imageprocessing,91–92mathematics,94–96wordprocessing,88–89

Appstore,203Applicationlauncher,49Apps,95apt(admincommand),202

aptitude(admincommand),203ar(regularcommand),269arithmetic(Bashexpansion),64ARPANET,167as(regularcommand),259ASCIIarmored,179ASP(ActiveServerPages),192asy(vectorgraphics),90attr(filecommand),151ausearch(admincommand),221Authority,fornameserverzones,190awk(regularcommand),108

Backup,fileandsystem,214–216

basename(filecommand),162Bash,aliasexpansion,67arithmeticexpansion,64arrays,124–125braceexpansion,64commandexecution,57–58commandexpansion,69commandline,54command-lineprompt,54,72compoundcommand,55disableglobbing,70errorandinterrupt,138–139examplescripts,131–137expansions,63–71filequery,122–123filenameexpansion,70–71forloop,114–115function,128–131functionarguments,128–129functiondefinition,81functionexport,82functionreturnvalue,130–131functions,81–82

heredocument,126–127historyexpansion,64–66initfiles,77–78interactingwith,54–55jobcontrol,60–63numericalcomputation,121–122patterncondition,121processexpansion,69processnumber,139prompt,73quoting,79–81redefinebuilt-infunction,131scriptdebugging,137–138scriptexecution,113scriptinvoking,111–112scriptparameters,114specialcharacters,79–81stringexpansion,64testexpressions,118–119tildeexpansion,67–68untilloop,121variableexpansion,68–69variableexport,57variablemodifiers,125–126variables,71–72,123–124whileloop,120

Bashcommand,.,71alias,67bg,62break,122case,119cd,18continue,122declare,69dirs,71echo,54,64env,73eval,133

exit,31,63export,57fc,66fg,55,62for,114hash,57help,71history,64,107if,115–119jobs,61kill,63,314logout,63popd,71pushd,71set,55,59,69shift,119shopt,70source,71until,121which,57while,120

Bashexamplescript,clean,131–133cpp,133–135mput,136–137resize,137total,135–136

Bashvariable,*,115?,118AAt,82,115CDPATH,72DISPLAY,73EDITOR,72HISTFILE,66HISTSIZE,72HOME,72HOSTNAME,72OLDPWD,67,72

PATH,57,72PS1,72PS2,72PWD,72SHELL,72TERM,73USER,72#,114DISPLAY,43positional,123

bg(Bashcommand),62bind(systemcall),333BITNET,167Bootfiles,190Booting,216–218brace(Bashexpansion),64bzip2(Filecommand),160

Clibraryfunction,191Cprogram,mainfunctionarguments,256assembly,259compiler,258–260headerfiles,262library,264–265linking,263optimization,259preprocessor,260–262

calendar(filter),98cancel(regularcommand),32case(Bashcommand),119Casesensitivity,170cat(regularcommand),59cd(Bashcommand),18CDandDVD,48CDPATH,72CGI(CommonGatewayInterface),192chcon(admincommand),221chdir(systemcall),296

chgrp(admincommand),152chmod(filecommand),149chown(admincommand),152chroot(filecommand),162chsh(regularcommand),31,54closedir(libraryfunction),296cmp(filecommand),162comm(filecommand),162Command,Bashbuilt-in,15,54executionenvironment,72–73exitstatus,55interpretationcycle,54regular,15,54search,57searchpath,57

command(Bashexpansion),69Commandline,argumentconventions,258argumentsinC,256–257completion,55–56editing,55

Commands,jobcontrol,63useful,30–32

CommonGatewayInterface(CGI),192CompilingCprogram,257–260CompilingPrograms,27–28Completion,commandname,56filename,56programmable,56userhostname,56username,56variablename,56

ComprehensivePerlArchiveNetwork,140connect(systemcall),332Connections,inHTTP,192ContentTypeandFileSuffix,147

Content,typesof,186Coredumps,279–281Country-codetop-leveldomains,189cp(regularcommand),19cpp(regularcommand),260creat(systemcall),290Cryptography,175–177public-key,177symmetric-key,177

Cryptosystem,symmetric,176–177

ctime(libraryfunction),293

DAC,218Daemon,343–344daemon,93Datasink,59date(regularcommand),31DebuggingwithGDB,274–279declare(Bashcommand),69DECnet,167Defaultfilebrowser,48Desktop,appearance,42launchapplications,39launcher,36managefiles,40notificationarea,37overview,35–36startmenu,37windowlist,37workspace,37,41

desktop(GUI),8Desktopcomponents,36–38df(filecommand),162DHCPserver,205dia(diagramtool),90diff(filecommand),162dig(admincommand),207

dig(regularcommand),172Digest,ofmessage,182–183Digitalcertificate,178DigitalSignature,177Digitalsignature,183Directory,operationsinC,295–296stream,295–296dirs(Bashcommand),71Diskpartitions,209DISPLAY,73DISPLAY(Bashvariable),43dnf(admincommand),202DNS,169,190–191Resolvers,191Servers,190DNSservers,207dolphin(regularcommand),48DomainNameRegistrars,168Domainnames,168IPmappingto,190registrationof,189–191serversof,189service,190–191top-level,168Dotnotation,167Dropbox,93–94du(filecommand),162dup2(systemcall),325Dynamicserverpages,192

echo(Bashcommand),54,64EditingText,22–23EDITOR,72edquota(regularcommand),214EFISystemPartition,217emacs(text3editor),31Emailclient,thunderbird,25

Encoding,base64,179Encrypt/decryptwithGnuPG,180Enigmail,181env(Bashcommand),73eog(viewphoto),91Epoch,293eval(Bashcommand),133evince(PDFviewer),88evince(regularcommand),46ex(regularcommand),104execl(libraryfunction),305Executable,binaryfile,305,306ELF,306file,305textfile,305execv(libraryfunction),305execve(libraryfunction),305exit(Bashcommand),31,63exit(libraryfunction),310Exitstatus,118,257Cprogram,257expand(filter),97expand(regularcommand),97export(Bashcommand),57exportfs(regularcommand),158expr(calculatetool),94

fc(Bashcommand),66fcntl(systemcall),340,348fdisk(admincommand),209fdopen(libraryfunction),268,327fdopendir(systemcall),295fflush(libraryfunction),266fg(Bashcommand),55,62File,absolutepathname,17accesscontrol,20–22,149Bashinit,78

Bashlogin,78Bashlogout,78creationmask,77extendedattributes,151fullpathname,146group,153hidden,14mode,151/etc/passwd,77pathname,185relativepathname,18setgid,153setuid,153simplename,18status,150–154synchronization,174user/groupid,152xattr,151

Filebrowser,45–48Filecommand,attr,151basename,162bzip2,160chmod,149chroot,162cmp,162comm,162df,162diff,162du,162find,158getfattr,151gzip,160ln,148locate,160ls,149mount,158rm,149setfattr,151

shar,161size,162split,162tar,160touch,162umount,156uniq,162wc,162xz,160

Filenamesuffixes,27Filetypes,146–149directory,147link,148ordinary,147special,148symboliclink,148

filename(Bashexpansion),70FilesandDirectories,16–20Filesystem,mounted,156networked,158quotas,214superblock,157table,157Filesystemorganization,155filezilla(FTPtool),92Filters,100calendar,98expand,97fold,98head,96sort,99tail,96tr,97find(filecommand),158finger(regularcommand),31firefox(Webbrowser),26Firewall,filteringrules,207

permanentsettings,208settings,207–208zones,208firewall-cmd(admincommand),208firewall-config(admincommand),208firewalld,207fixfiles(admincommand),222fold(filter),98fold(regularcommand),98for(Bashcommand),114fork(systemcall),302freopen(libraryfunction),268fsck(admincommand),211fseek(libraryfunction),266fstat(systemcall),292ftell(libraryfunction),267FTP,92ftp,92

g++(regularcommand),255gawk(regularcommand),108gcalccmd(CLIcalculator),94gcc(regularcommand),255gccoptions,259–260gdb(regularcommand),274gdisk(admincommand),209gedit(regularcommand),46gedit(texteditor),22,31GETmethod,193getchar(libraryfunction),265getcwd(libraryfunction),312getdents(systemcall),295getenforce(admincommand),219getfattr(filecommand),151gethostbyname(libraryfunction),333getpid(systemcall),303getppid(systemcall),303gimp(photoprocessing),91gimp(regularcommand),61

GNOME3windowmanager,44GnomeShell,38GNOMEshellextension,38GNOMETerminal,Web,emaillinks,51copyandpaste,50starting,49–50GNOMEtweaktool,38gnome-calculator(calculator),94gnome-disks(admincommand),210gnome-help(regularcommand),51gnome-screenshot(screenshottool),91gnome-software(appstool),87gnome-system-monitor(admin,command),204gnome-terminal(regularcommand),48gnome-tweak-tool(regularcommand),38GnuPG,178–180encrypt/decrypt,180keycreation,178–180messagesigning,183–184gparted(admincommand),209gpg(regularcommand),178gpg2(regularcommand),178gprof(regularcommand),260groupadd(admincommand),199groupadd(regularcommand),153groupdel(admincommand),199groupmod(admincommand),199GRUB2bootloader,217GUI,8GUIapps,87–96GUID,217gvim(texteditor),31gzip(filecommand),160

Hall,Larry,140hash(Bashcommand),57head(filter),96

head(regularcommand),96help(Bashcommand),71Hiddenfiles,14HISTFILE,66history(Bashcommand),64,107history(Bashexpansion),64HISTSIZE,72HOME,72host(regularcommand),168,172HOSTNAME,72hostname(regularcommand),171HTML,187–188htonl(libraryfunction),343htons(libraryfunction),343htpasswd(regularcommand),237HTTPmethod,GET,193POST,193httpd(regularcommand),228HTTPS,238Hypertext,184HypertextMarkupLanguage(HTML),184–185,187–188HypertextPreprocessor(PHP),192HypertextTransferProtocol(HTTP),192–193

I/O,redirection,58–60standard,58I/Odescriptor,286–288I/Omultiplexing,345–346I/OredirectioninC,268ifconfig(admincommand),207import(screencapture),91Indexfiles,185Initfile,.bash_profile,78readline,56Inputfocus,39InstantMessaging,26

Integrateddevelopmentenvironment(IDE),187Integrity,ofmessage,183Internet,167–169address,167clients,169domainnamesystem,169–170introduction,169servers,169TLD,170InternetCorporationforAssignedNames,andNumbers(ICANN),167,168,189,190InternetProtocol(IP),167,190InternetServiceProviders(ISPs),190Internetworking,167iptablesoutdated,208IPv4,167IPv6,167ISP,205

JavaServerPages(JSP),192jobs(Bashcommand),61JSP(JavaServerPages),192

kdevelop(regularcommand),255keepassx2(passwordmanagertool),93Keyrepositories,178Keyserver,179Keyboardinputescapecharacter,56kill(Bashcommand),63,314kill(regularcommand),62,63kill(systemcall),314killall(admincommand),204konsole(regularcommand),53

Labels,forDNStreenode,170ld(regularcommand),263LDAP,200less(regularcommand),97lgroupadd(admincommand),200Librarycreating,268–269

Libraryfunction,abort,313alarm,313closedir,296ctime,293execl,305execv,305execve,305exit,310fdopen,268,327fflush,266freopen,268fseek,266ftell,267getchar,265getcwd,312gethostbyname,333,334htonl,343htons,343ntohl,343ntohs,343pclose,322popen,321putchar,265raise,314readdir,296setjmp,272

Libraryfunctions,common,264libreoffice(productivitytool),88LightweightDirectoryAccessProtocol,200link(systemcall),291Linux,documentation,29features,3–5filelocations,145–146filesystem,154–158helpanddocumentation,51history,1–2

networkingcommands,171–184philosophy,3versions,2–3

Linuxsystems,190listen(systemcall),340ln(filecommand),148locate(filecommand),160LogicalVolumeManagement,212–214Login,8–11logout(Bashcommand),63longjmp(libraryfunction),272look(regularcommand),31,100lp(regularcommand),32lpr(regularcommand),24,32lprm(regularcommand),32lpstat(regularcommand),32lqp(regularcommand),32ls(filecommand),149LSB,3lsblk(admincommand),209lstat(systemcall),292luseradd(admincommand),200lvcreate(admincommand),212lvdisplay(admincommand),212LVM,212–214extent,212logicalvolume,212volumegroup,212LVMarchitecture,212

MAC,218MACaddress,205MacromediaDreamweaver,187make(regularcommand),244man(regularcommand),29Manage,disks,209–211filesystems,209–211Markuptags,187

maxima(advacedmath),95MD5,161,182Messagedigest,182–183Messageintegrity,183MIME(MultipurposeInternetMail,Extensions),147,186Missingcommandinstall,203mkdir(regularcommand),19mkdir(systemcall),295mkfs(admincommand),211mknod(admincommand),157mknod(systemcall),292more(regularcommand),26mount(admincommand),211mount(filecommand),158mutt(regularcommand),25mv(regularcommand),19MySQL,administration,249–251configuration,248–249controlling,248install,251–252serverandclient,248mysql(regularcommand),247MySQLdatabaseserver,247–252mysql_secure_installation(regularcommand),248mysqld(regularcommand),248

nano(texteditor),31Nautilus,networkingin,170–171nautilus(regularcommand),45nc(regularcommand),194ncat(regularcommand),194nemiver(regularcommand),279NemiverGUIforGDB,274netstat(regularcommand),249Netword,gateway,205

Network,address,167–168clientandserver,169configuration,205–207defaultgateway,206interfacenames,206layer,166packetswitching,169services,165,167Networking,protocols,166–167sockets,330–333NetworkinginC,328–343NetworkManagerdaemon,206NIC,205nm-connection-editor(admincommand),207nmcli(admincommand),207Nodes,190nslookup(regularcommand),172ntohl(libraryfunction),343ntohs(libraryfunction),343OLDPWD,67,72opendir(systemcall),295Openingaprocess,321–322OperatingSystem,7

Package,repositories,201Packagemanagement,201–203APT,201–203DEB-based,202DNF,201–203RPM-based,202Packetswitching,169Pagemakers,187PAM,200parted(admincommand),209,211PartiallyspecifiedURLs,185passwd(regularcommand),31

Passwordencryption,237PasswordManager,92–93PATH,57,72pclose(libraryfunction),322PDF,pages,89passwordprotect,89viewer,88pdflatex(typesettingtool),89Per-userWebfolder,41Perlscriptinglanguage,descriptionof,139–140PersonalWebpages,186–187PGP/MIME,180pgrep(admincommand),204PHP,commandlineuse,246configuration,245debugging,245fileprocessing,244install,243–245security,245PHP(HypertextPreprocessor),192,242–243phpMyAdmin,250phpMyAdmininstall,250–251pidof(admincommand),204pidof(regularcommand),229ping(admincommand),207ping(regularcommand),172Pipe,60Pipetwo-way,327–328Pipelinebuilding,107–108pkill(admincommand),204PluggableAuthenticationModules,200popd(Bashcommand),71popen(libraryfunction),321Portnumbers,185POSIX,1Predicatefunction,130

Process,285background,55creation,302–305definition,203environment,310–313foreground,55infodisplay,204managing,203–205monitor,204synchronization,308–309termination,204,310process(Bashexpansion),69Processing,inHTTP,192Programexecution,305–309ps(admincommand),204,301PS1,72PS2,72pstree(admincommand),204ptrace(systemcall),309pushd(Bashcommand),71putchar(libraryfunction),265pvcreate(admincommand),212pvdisplay(admincommand),212PWD,72pwd(regularcommand),18,69

qpdf(PDFmanipulation),89Quadnotation,167Queries,inHTTP,193Queries,inHTTP,192quotacheck(admincommand),214quotaoff(regularcommand),214quotaon(regularcommand),214

raise(libraryfunction),314read(systemcall),288readdir(libraryfunction),296readelf(regularcommand),306recv(systemcall),347

recvfrom(systemcall),332Regularcommand,apache2ctl,229apachectl,229ar,269as,259awk,108cancel,32cat,59chmod,21–22chsh,31,54cp,19cpp,260date,31dig,172dolphin,48edquota,214evince,46ex,104expand,97exportfs,158finger,24–25,31fold,98g++,255gawk,108gcc,255gdb,274gedit,46gimp,61gnome-help,51gnome-terminal,48,53gnome-tweak-tool,38gpg,178gpg2,178gprof,260grep,fgrep,egrep,100groupadd,153head,96host,168,172

hostname,171htpasswd,237httpd,228kdevelop,255kill,62,63konsole,53ld,263less,97look,31,100lp,32lpr,24,32lprm,32lpstat,32lqp,32ls,20–21make,244man,29mkdir,19more,26mutt,25mv,19mysql,247mysqladmin,249mysqld,248nautilus,45nc,194ncat,194nemiver,279netstat,249nslookup,172passwd,31pidof,229ping,172pwd,18,69quotaoff,214quotaon,214readelf,306rm,19rmdir,19

rsync,174scp,174script,32sed,104sftp,174sort,99ssh,11,56,173su,20system-config-users,153tail,96tr,97unexpand,97vi,102wget,169,194which,311whois,189xclock,43xwininfo,44yelp,29,76

Regularexpression,basic,102–103grep,103grep,104matchingcommands,100–101

Relativedomainnames,170rename(systemcall),291Responses,inHTTP,192restorecon(admincommand),222rm(filecommand),149rm(regularcommand),19rmdir(regularcommand),19rmdir(systemcall),295Rootnameservers,190Rot13,174route(admincommand),207rsync(regularcommand),174

S/MIMEcertificate,180Samba,161

scheme,inURLs,185scp(regularcommand),174script(regularcommand),32seahorse(passwordmanager),93sealert(admincommand),221secon(admincommand),222SecureBoot,217Secureemail,180–182Thunderbird,181–182sed(regularcommand),104seinfo(admincommand),220select(systemcall),345SELinux,4,154,218–222contexts,219–220filecontext,221–222filecontexts,220logfile,221object,218setfilecontext,221statusandmodes,218–219subject,218targetedpolicy,219send(systemcall),347sendto(systemcall),332server,185Serverroot,185Server-sideprogramming,191,193Servicedaemons,203servicesconfiguration,204sestatus(admincommand),218,222set(Bashcommand),55,59,69setenforce(admincommand),219setfattr(filecommand),151setjmp(libraryfunction),272setsid(systemcall),344sftp,92sftp(regularcommand),174shar(filecommand),161SHELL,72

Shell,Globpattern,70interactive,78intro,12–16login,78non-interactive,78settingdefault,54ShellcommandfromC,299shift(Bashcommand),119shopt(Bashcommand),70shotwell(managephoto),91shutdown(systemcall),339Siblinglabels,170sigaction(systemcall),315Signal,concepts,313processing,314sending,314trapping,315–317Signals,313–317Signature,digital,177simple-scan(scanningtool),92size(filecommand),162Socket,address,331addressstructure,332datagram,330Internet,331local,331port,333raw,330stream,330socket(systemcall),331Socketout-of-banddata,347–348socketpair(systemcall),339Softwaremanagementtasks,202–203software-center(appstool),87sort(filter),99sort(regularcommand),99

source(Bashcommand),71split(filecommand),162SSH,keygeneration,174X11forwarding,173ssh(regularcommand),11,56,173SSL/TLS,238–239SSSD,200stat(systemcall),292sed,104–107string(Bashexpansion),64su(admincommand),199su(regularcommand),20Subnetmask,205sudo(admincommand),201sudoersfile,201symlink(systemcall),291system(systemcall),299Systemcall,_exit,310accept,340access,294bind,333chdir,296connect,332creat,290dup2,325fcntl,340,348fdopendir,295fileoperations,290–295fork,302fstat,292getdents,295getpid,303getppid,303I/O,286–290kill,314link,291listen,340

lseek,289–290lstat,292mkdir,295mknod,292opendir,295pipe,322–326pselect,346ptrace,309read,288recv,347recvfrom,332rename,291rmdir,295select,345send,347sendto,332setsid,344shutdown,339sigaction,315socket,331socketpair,339stat,292symlink,291system,299umask,291unlink,291utimes,292vfork,308wait,308waitpid,309write,288

Systemcontrol,39SystemSecurityServicesDaemon,200system-config-services(admincommand),204system-config-users(regularcommand),153systemctl(admincommand),204,228systemdprocessandservicemanager,204systemdtargets,217

tail(filter),96tail(regularcommand),96tar(filecommand),160TERM,73Terminalwindow,48–51Termination,inHTTP,192Texteditor,emacs,31gedit,22,31gvim,31nano,31vi,31vim,31thunderbird(Emailclient),25tilde(Bashexpansion),67TLS,phases,238Tool,asy(vectorgraphics),90dia(diagraming),90eog(viewphoto),91evince(PDFviewer),88expr(calculate),94filezilla(FTP),92ftp,92gcalccmd(CLIcalculator),94gimp(photoprocessing),91gnome-calculator(calculator),94gnome-screenshot(screenshot),91gnome-software(appstool),87import(screencapture),91keepassx2(passwordmanager),93libreoffice(productivity),88maxima(advacedmath),95pdflatex(typesetting),89qpdf(PDFmanipulation),89seahorse(passwordmanager),93sftp,92shotwell(managephoto),91

simple-scan(scanning),92software-center(manageapps),87xsane(scanning),92top(admincommand),204Top-leveldomain,168,189touch(filecommand),162tr(filter),97tr(regularcommand),97traceroute(admincommand),207TransmissionControlProtocol,167TransportLayerSecurity(TLS),238

UEFI,217umask(systemcall),291umount(filecommand),156UnderstandingGUI,39unexpand(regularcommand),97UniformResourceLocators(URLs),185uniq(filecommand),162unlink(systemcall),291until(Bashcommand),121URI,226–227URL,226–227USER,72UserDatagramProtocol,167useradd(admincommand),199userdel(admincommand),199usermod(admincommand),199Users,manage,199–201utimes(systemcall),292

variable(Bashexpansion),68vfork(systemcall),308vgcreate(admincommand),212vgdisplay(admincommand),212vi(regularcommand),102vi(texteditor),31vim(texteditor),31visudo(admincommand),200

W3Consortium,184wait(systemcall),308waitpid(systemcall),309Wayland,43wc(filecommand),162Web,184–186access,185browsers,169browsing,26databasesupportfor,247dynamicpagegeneration,191–192hosting,188–189HTTPrequest,227HTTPresponse,227hyperlinks,184publishing,186request,227server,225–226URLs,185Webbrowser,firefox,26wget(regularcommand),169,194which(Bashcommand),57which(regularcommand),311while(Bashcommand),120whois(admincommand),207whois(regularcommand),189windowID,44Windowmanager,43–44WorldWideWeb,184–186informationon,186–187personalpageson,186–187write(systemcall),288

Xserver,42XWindowSystem,42–44X11forwarding,173xattr,151Xattrcalls,295

xclock(regularcommand),43xsane(scanningtool),92xwininfo(regularcommand),44xz(Filecommand),160

yelp(regularcommand),29,76yumex-dnf(admincommand),203

Zones,ofnameservers,190