7/29/2019 Unit6 2 FinaL

1/39

SQL *LoaderWhat is SQL *loader

SQL*Loader (sqlldr) is the utility to use for highperformance data loads. The data can be loadedfrom any text file and inserted into the database.

OR Oracle utility which populates Oracle tables from

host files

Tables must be created first SQL*LOADER operates with control file

Data can be put into control file or stored in

separate data file.

7/29/2019 Unit6 2 FinaL

2/39

SQL LOADER utility is used to load data from

other data source into Oracle. For example,

if you have a table in FOXPRO, ACCESS or SYBASEor any other third party database, you can use

SQL Loader to load the data into Oracle Tables.SQL Loader will only read the data from Flat files.So If you want to load the data from Foxpro orany other database, you have to first convert thatdata into Delimited Format flat file or Fixed lengthformat flat file, and then use SQL loader to loadthe data into Oracle.

SQL *Loader

7/29/2019 Unit6 2 FinaL

3/39

SQL*Loader

Control file Data files

Parameter file

(optional

Discarded

Discard file(optional)

Log file

Database files

Inserted

Rejected

Bad file

RejectedSQL*Loader

Field processing

Record selectionAccepted

Selected

Oracle server

7/29/2019 Unit6 2 FinaL

4/39

Using SQL*Loader

$sqlldr scott/tiger \

control=ulcase6.ctl \>

log=ulcase6.log direct=true

ulcase6.ctl

SQL*Loader

EMP tableulcase6.log

7/29/2019 Unit6 2 FinaL

5/39

Marina G. Erechtchoukova 5

Using the SQL*LOADER

Create table

Prepare Control file and data file or place data

into control file

SQL*LOADER is executed from the HOST

environment

7/29/2019 Unit6 2 FinaL

6/39

Architecture Of SQL*Loader

7/29/2019 Unit6 2 FinaL

7/39

control file

What is control file ? The SQL*Loader control file contains information that

describes how the data will be loaded. It contains the

table name, column datatypes, field delimiters, etc.

Specifies the action: Insert

Replace

Append

Specifies the data file name and list of fields

Has an extension .ctl

7/29/2019 Unit6 2 FinaL

8/39

Sample Control FileLOAD DATA

INFILE fileName.dat'

BADFILE badFileName.bad

DISCARDFILE ' discardFileName.dsc'

REPLACE INTO TABLE "SchemaName"."TABLENAME"

FIELDS TERMINATED BY '| TRAILING NULLCOLS

(Colomn1Name INTEGER EXTERNAL,

Colomn3Name CHAR,

Colomn4Name SYSDATE,

Colomn5Name CONSTANT "SOMECONSTANT"

)

7/29/2019 Unit6 2 FinaL

9/39

Marina G. Erechtchoukova 9

Control File Contains Data

Control file is prepared in the same way

INFILE *

BEGINDATA clause is added Below the clause data are placed according to

the format declared in the control information

7/29/2019 Unit6 2 FinaL

10/39

Log FileThe log file is a record of SQL*Loader's activities during

a load session. It contains information such as the following:

The names of the control file, log file, bad file, discard file, and data file

The values of several command-line parameters

A detailed breakdown of the fields and datatypes in the data file that was

loaded Error messages for records that cause errors

Messages indicating when records have been discarded

A summary of the load that includes the number of logical records readfrom the data file, the number of rows rejected because of errors, thenumber of rows discarded because of selection criteria, and the elapsedtime of the load

Sample Log File

Sample.log

7/29/2019 Unit6 2 FinaL

11/39

Marina G. Erechtchoukova 11

Example: Data File

1, 'Paris'

2, 'Boston'

3, 'London4, Ottawa

5, Toronto

7/29/2019 Unit6 2 FinaL

12/39

Bad file

SQL*LOADER creates .bad file if an erroroccurs.

Bad file

Rejected records Same format as data files

Discard file

Records not satisfying conditions Same format as data files

parameter file

to specify commonly used command line options

7/29/2019 Unit6 2 FinaL

13/39

SQL*Loader loads data from external files into tables of an Oracle database.

It has a powerful data parsing engine that puts little limitation on the formatof the data in the data file.

You can use SQL*Loader to do the following: Load data across a network. This means that you can run the SQL*Loader client on a

different system from the one that is running the SQL*Loader server.

Load data from multiple datafiles during the same load session.

Load data into multiple tables during the same load session.

Specify the character set of the data.

Selectively load data (you can load records based on the records' values).

Manipulate the data before loading it, using SQL functions.

Generate unique sequential key values in specified columns.

Use the operating system's file system to access the datafiles.

Load data from disk, tape, or named pipe.

Generate sophisticated error reports, which greatly aid troubleshooting.

Load arbitrarily complex object-relational data.

Use secondary data files for loading LOBs and collections.

SQL*Loader Features

7/29/2019 Unit6 2 FinaL

14/39

Marina G. Erechtchoukova 14

Using the SQL*LOADER in ITEC Lab

Login onto sit.yorku.ca

Call SQL*LOADER utility

sqlldr login@studb10g Control_File System asks for your password

Enter your password

7/29/2019 Unit6 2 FinaL

15/39

Example Suppose you have a table in MS-ACCESS by name EMP, running under Windows O/S,

with the following structure.

EMPNO INTEGERNAME TEXT(50)

SAL CURRENCY

JDATE DATE

This table contains some 10,000 rows. Now you want to load the data from this table

into an Oracle Table. Oracle Database is running in LINUX O/S

7/29/2019 Unit6 2 FinaL

16/39

Start MS-Access and convert the table into comma delimited flat

(popularly known as csv) , by clicking on File/Save As menu. Let the delimited file namebe emp.csv.

Now transfer this file to Linux Server using FTP command

Go to Command Prompt in windows

At the command prompt type FTP followed by IP address of the server running Oracle.

FTP will then prompt you for username and password to connect to the Linux Server. Supply avalid username and password of Oracle User in Linux

For example:-C:\>ftp 200.200.100.111

Name: oracle

Password:oracle

FTP>

Now give PUT command to transfer file from current Windows machine toLinux machine.

FTP>put

Local file:C:\>emp.csvremote-file:/u01/oracle/emp.csv

File transferred in 0.29 SecondsFTP> Now after the file is transferred quit the FTP utility by typing bye command.

FTP>bye

Good-Bye

7/29/2019 Unit6 2 FinaL

17/39

Now come the Linux Machine and create a table in Oracle with the same structure as in MS-ACCESS by taking appropriate datatypes. For example, create a table like this

$sqlplusscott/tiger

SQL>CREATE TABLE emp (empno number(5),

name varchar2(50), sal number(10,2),

jdate date);

After creating the table, you have to write a control file describing theactions which SQL Loader should do. You can use any text editor to writethe control file. Now let us write a control file for our case study

$vi emp.ctl

1 LOAD DATA

2 INFILE /u01/oracle/emp.csv

3 BADFILE /u01/oracle/emp.bad

4 DISCARDFILE /u01/oracle/emp.dsc5 INSERT INTO TABLE emp

6 FIELDS TERMINATED BY , OPTIONALLY ENCLOSED BY TRAILINGNULLCOLS

7 (empno,name,sal,jdate date mm/dd/yyyy)

7/29/2019 Unit6 2 FinaL

18/39

After you have wrote the control file save it and then,call SQL Loader utility by typing the followingcommand

$sqlldr userid=scott/tiger control=emp.ctl log=emp.log

After you have executed the abovecommand SQL Loader will shows you the output

describing how many rows it has loaded. The LOG option of sqlldr specifies where the log file of

this sql loader session should be created. The log filecontains all actions which SQL loader has performed

i.e. how many rows were loaded, how many wererejected and how much time is taken to load the rowsand etc. You have to view this file for any errorsencountered while running SQLLoader.

7/29/2019 Unit6 2 FinaL

19/39

Methods of Loading data through

SQL *loader

SQL*Loader provides the following methods to load

data:

Conventional Path Loads

Direct Path Loads

External Table Loads

7/29/2019 Unit6 2 FinaL

20/39

Conventional Path Load A conventional path load executes SQL * loader reads a set of

records from a file and generate insert commands and passesthem to the oracle kernal. Oracle finds places for those records

in the free blocks in the table and updates any assocated

indexs.

In a Direct Path Load SQL * loader create formatted datablocks within server process and write them directly to the

data files. this requires occasional checks with the database to

get new locations for data blocks, but no other I/O with

database kernal is required. The result is a data load process that is dramatically faster

then conventional path mode.

7/29/2019 Unit6 2 FinaL

21/39

7/29/2019 Unit6 2 FinaL

22/39

Conventional and Direct Loads

Instance

SGA Shared pool

Array

insert

Conventional

ExtentmanagementData save

Data

save

Direct

pathTable

High water mark

Space used only by conventional load

7/29/2019 Unit6 2 FinaL

23/39

Using Direct-Load Inserts

INSERT /*+A PPEND */ INTO scott.emp

NOLOGGING

SELECT * FROM scott.old_emp;

Server processE MP table

High water mark

Blocks used by inserted rowsFree space after delete

Used block

7/29/2019 Unit6 2 FinaL

24/39

Parallel Direct-Load Insert

ALTER SESSION ENABLE PARALLEL DML;

INSERT /*+PARALLEL(scott.emp,2) */

INTO scott.emp NOLOGGING

SELECT * FROM scott.old_emp;

Slave process Slave process

High water mark

Temporary SegmentsFree space after delete

Used block

7/29/2019 Unit6 2 FinaL

25/39

Parallel Direct Loads

Temporary segments

load1.dat

load1.ctl

load2.dat

load2.ctl

load3.dat

load3.ctl

SQL*Loader

SQL*Loader

SQL*Loader

High water markmarkTable

7/29/2019 Unit6 2 FinaL

26/39

Comparing Direct and

Conventional Path LoadsConventional Load Direct Path LoadUses COMMITs to make changes

permanent

Usesdata saves

Redo log entries always generated Generates redo only under specific

conditions

Enforces all constraints Enforces only primary key, unique,and NOT NULL

INSERT triggers fire INSERT triggers do not fire

Can load into clustered tables Cannot load into clustered tables

Other users can make changes to

tables

Other users cannot make changes

to tables

7/29/2019 Unit6 2 FinaL

27/39

Data Blocks, Extents, and Segments

7/29/2019 Unit6 2 FinaL

28/39

Oracle allocates logical database space for all data in a

database.

The allocation of database space are data blocks, extents, and

segments.

Data blocks.

Oracle manages the storage space in the data files of a

database in units called data blocks.

A data block is the smallest unit of data used by a database.

segments.

A segment is a set of extents, each of which has been

allocated for a specific data structure and all of which are

stored in the same table space.

extents

An extent is a specific number of contiguous data blocks

allocated for storing a specific type of information.

7/29/2019 Unit6 2 FinaL

29/39

Multiple-Process Oracle Instance

7/29/2019 Unit6 2 FinaL

30/39

7/29/2019 Unit6 2 FinaL

31/39

Bulk Inserts:

Common Traps and Successful Tricks

If your data is not being inserted from a flat file,

SQL*Loader will not be a useful solution.But move a

large set of data from one table to another, there

are several common methods forimproving the performance of the data migration:

Tuning the structures (removing indexes and triggers)

Disabling constraints during the data migration Using hints and options to improve the transaction

performance

Isolating the rollback segments for the large transaction.

7/29/2019 Unit6 2 FinaL

32/39

Tuning the structures (removing indexes and triggers)

Disabling any triggers or indexes that are on the table intowhich data is being loaded.

if you have a row-level trigger on the target table, that trigger will beexecuted for every row inserted into the table.

disable the triggers prior to the data load.

the bulk operation will complete faster than the repeated triggerexecutions.

the bulk operations execute for all rows that have not already beenprocessed by the triggers.

Disabling constraints during the data migration you should consider disabling constraints on the table.

Once the data has been loaded, you can reenable the constraints.

Using hints and options to improve the transactionperformance Oracle has introduced for data migration tuning. Those options include

the following:

7/29/2019 Unit6 2 FinaL

33/39

The append hint for insert commands In Direct Path Loader, the APPEND hint loads blocks of data into a

table

Use of the APPEND hint may increase your space usage.The no logging option If you are performing a create table as select command, use the

nologging option

nologging option to avoid writing to the redo logs during the

operation.The parallel option Parallel Query uses multiple processes to accomplish a single task.

you can parallelize both the create table portion and the query.

If you use the parallel option, you should also use the nologgingoption otherwise the parallel operations will have to wait due toserialized writes to the online redo log files.

you should first investigate the target tables structuresto make sure youve avoided the common traps as above

7/29/2019 Unit6 2 FinaL

34/39

Isolating the rollback segments for the large

transaction. May require the creation of a new table space for new rollback segments.

Use SET TRANSACTION USE ROLLBACK SEGMENT command.

You can also use programming logic to force inserts to be

processed in arrays rather than as an entire set.

Example :-

COBOL and C support array inserts, thus reducing the size of the

transactions required to process a large set of data.

7/29/2019 Unit6 2 FinaL

35/39

Bulk Deletes: The truncate CommandTwo problems are generated when user deleted records.

1) Users attempt to delete all the rows from a table at once theycomplain that the rollback segments are too small and transaction is

too large.

2) The rows have all been deleted Even though the segment no longer

has any rows in it and still maintains all the space that was allocated toit. Therefore, deleting all those rows saved not a single byte of

allocated space.

truncate command resolves both of these problems. It is a DDL

command, not a DML command, so it cannot be rolled back.

Once you have used the truncate command on a table, its rows aregone, and none of its delete triggers are executed in the process.

However, the table retains all its dependent objectssuch as grants,

indexes, and constraints.

7/29/2019 Unit6 2 FinaL

36/39

The truncate command is the fastest way to delete large volumes of

data. Because it will delete all the rows in a table

If you use partitions, you can truncate one partition of a table without

affecting the rest of the tables partitions.

A sample truncate command for a table.

truncate table EMPLOYEE drop storage;

The EMPLOYEE tables rows are deleted

The drop storage clause is used to deallocate the non-initialspace from the table.

you can delete all of a tables rows and reclaim all but its initial

extents allocated space, without dropping the table.

The truncate command also works for clusters. the reuse storage option is used to leave all allocated space empty

within the segment that acquired it:

truncate cluster EMP_DEPT reuse storage;

7/29/2019 Unit6 2 FinaL

37/39

When this example command is executed, all the rows in the

EMP_DEPT cluster will be instantly deleted.

To truncate a partition, you need to know its name.

Example :-

the partition named PART3 of the EMPLOYEE table is truncated via

the alter table command:

alter table EMPLOYEE truncate partition PART3

drop storage;

The rest of the partitions of the EMPLOYEE table will be

unaffected by the truncation of the PART3 partition.

As an alternative, you can create a PL/SQL program that uses

dynamic SQL to divide a large delete operation into multiplesmaller transactions.

7/29/2019 Unit6 2 FinaL

38/39

Partitions You can use partitions to isolate data physically.

Example :-

you can store the data from one department in a separate

partition of an the EMPLOYEE table. If you perform a bulk data

load or deletion on the table, you can customize the partitions

to tune the data manipulation operation.

You can truncate a partition and its indexes without affecting the rest

of the table.

You can drop a partition, via the drop partition clause of the alter

table command.

You can drop a partitions local index. You can set a partition to nologging, reducing the impact of large

transactions.

the chief advantage of partitions lies in their ability to be

managed apart from the rest of the table.

7/29/2019 Unit6 2 FinaL

39/39

For example, being able to truncate a partition enables you to

delete a large amount of data from a table (but not all of the

tables data) without generating any redo information.

In the short term, the beneficiary of this performanceimprovement is the DBA; in the longer term, the entire

enterprise benefits from the improved availability of the data.