Unit6 2 FinaL

Embed Size (px)

Citation preview

  • 7/29/2019 Unit6 2 FinaL


    SQL *LoaderWhat is SQL *loader

    SQL*Loader (sqlldr) is the utility to use for highperformance data loads. The data can be loadedfrom any text file and inserted into the database.

    OR Oracle utility which populates Oracle tables from

    host files

    Tables must be created first SQL*LOADER operates with control file

    Data can be put into control file or stored in

    separate data file.

  • 7/29/2019 Unit6 2 FinaL


    SQL LOADER utility is used to load data from

    other data source into Oracle. For example,

    if you have a table in FOXPRO, ACCESS or SYBASEor any other third party database, you can use

    SQL Loader to load the data into Oracle Tables.SQL Loader will only read the data from Flat files.So If you want to load the data from Foxpro orany other database, you have to first convert thatdata into Delimited Format flat file or Fixed lengthformat flat file, and then use SQL loader to loadthe data into Oracle.

    SQL *Loader

  • 7/29/2019 Unit6 2 FinaL



    Control file Data files

    Parameter file



    Discard file(optional)

    Log file

    Database files



    Bad file


    Field processing

    Record selectionAccepted


    Oracle server

  • 7/29/2019 Unit6 2 FinaL


    Using SQL*Loader

    $sqlldr scott/tiger \

    control=ulcase6.ctl \>

    log=ulcase6.log direct=true



    EMP tableulcase6.log

  • 7/29/2019 Unit6 2 FinaL


    Marina G. Erechtchoukova 5

    Using the SQL*LOADER

    Create table

    Prepare Control file and data file or place data

    into control file

    SQL*LOADER is executed from the HOST


  • 7/29/2019 Unit6 2 FinaL


    Architecture Of SQL*Loader

  • 7/29/2019 Unit6 2 FinaL


    control file

    What is control file ? The SQL*Loader control file contains information that

    describes how the data will be loaded. It contains the

    table name, column datatypes, field delimiters, etc.

    Specifies the action: Insert



    Specifies the data file name and list of fields

    Has an extension .ctl

  • 7/29/2019 Unit6 2 FinaL


    Sample Control FileLOAD DATA

    INFILE fileName.dat'

    BADFILE badFileName.bad

    DISCARDFILE ' discardFileName.dsc'



    (Colomn1Name INTEGER EXTERNAL,

    Colomn3Name CHAR,

    Colomn4Name SYSDATE,



  • 7/29/2019 Unit6 2 FinaL


    Marina G. Erechtchoukova 9

    Control File Contains Data

    Control file is prepared in the same way

    INFILE *

    BEGINDATA clause is added Below the clause data are placed according to

    the format declared in the control information

  • 7/29/2019 Unit6 2 FinaL


    Log FileThe log file is a record of SQL*Loader's activities during

    a load session. It contains information such as the following:

    The names of the control file, log file, bad file, discard file, and data file

    The values of several command-line parameters

    A detailed breakdown of the fields and datatypes in the data file that was

    loaded Error messages for records that cause errors

    Messages indicating when records have been discarded

    A summary of the load that includes the number of logical records readfrom the data file, the number of rows rejected because of errors, thenumber of rows discarded because of selection criteria, and the elapsedtime of the load

    Sample Log File


  • 7/29/2019 Unit6 2 FinaL


    Marina G. Erechtchoukova 11

    Example: Data File

    1, 'Paris'

    2, 'Boston'

    3, 'London4, Ottawa

    5, Toronto

  • 7/29/2019 Unit6 2 FinaL


    Bad file

    SQL*LOADER creates .bad file if an erroroccurs.

    Bad file

    Rejected records Same format as data files

    Discard file

    Records not satisfying conditions Same format as data files

    parameter file

    to specify commonly used command line options

  • 7/29/2019 Unit6 2 FinaL


    SQL*Loader loads data from external files into tables of an Oracle database.

    It has a powerful data parsing engine that puts little limitation on the formatof the data in the data file.

    You can use SQL*Loader to do the following: Load data across a network. This means that you can run the SQL*Loader client on a

    different system from the one that is running the SQL*Loader server.

    Load data from multiple datafiles during the same load session.

    Load data into multiple tables during the same load session.

    Specify the character set of the data.

    Selectively load data (you can load records based on the records' values).

    Manipulate the data before loading it, using SQL functions.

    Generate unique sequential key values in specified columns.

    Use the operating system's file system to access the datafiles.

    Load data from disk, tape, or named pipe.

    Generate sophisticated error reports, which greatly aid troubleshooting.

    Load arbitrarily complex object-relational data.

    Use secondary data files for loading LOBs and collections.

    SQL*Loader Features

  • 7/29/2019 Unit6 2 FinaL


    Marina G. Erechtchoukova 14

    Using the SQL*LOADER in ITEC Lab

    Login onto sit.yorku.ca

    Call SQL*LOADER utility

    sqlldr login@studb10g Control_File System asks for your password

    Enter your password

  • 7/29/2019 Unit6 2 FinaL


    Example Suppose you have a table in MS-ACCESS by name EMP, running under Windows O/S,

    with the following structure.




    This table contains some 10,000 rows. Now you want to load the data from this table

    into an Oracle Table. Oracle Database is running in LINUX O/S

  • 7/29/2019 Unit6 2 FinaL


    Start MS-Access and convert the table into comma delimited flat

    (popularly known as csv) , by clicking on File/Save As menu. Let the delimited file namebe emp.csv.

    Now transfer this file to Linux Server using FTP command

    Go to Command Prompt in windows

    At the command prompt type FTP followed by IP address of the server running Oracle.

    FTP will then prompt you for username and password to connect to the Linux Server. Supply avalid username and password of Oracle User in Linux

    For example:-C:\>ftp

    Name: oracle



    Now give PUT command to transfer file from current Windows machine toLinux machine.


    Local file:C:\>emp.csvremote-file:/u01/oracle/emp.csv

    File transferred in 0.29 SecondsFTP> Now after the file is transferred quit the FTP utility by typing bye command.



  • 7/29/2019 Unit6 2 FinaL


    Now come the Linux Machine and create a table in Oracle with the same structure as in MS-ACCESS by taking appropriate datatypes. For example, create a table like this


    SQL>CREATE TABLE emp (empno number(5),

    name varchar2(50), sal number(10,2),

    jdate date);

    After creating the table, you have to write a control file describing theactions which SQL Loader should do. You can use any text editor to writethe control file. Now let us write a control file for our case study

    $vi emp.ctl


    2 INFILE /u01/oracle/emp.csv

    3 BADFILE /u01/oracle/emp.bad

    4 DISCARDFILE /u01/oracle/emp.dsc5 INSERT INTO TABLE emp


    7 (empno,name,sal,jdate date mm/dd/yyyy)

  • 7/29/2019 Unit6 2 FinaL


    After you have wrote the control file save it and then,call SQL Loader utility by typing the followingcommand

    $sqlldr userid=scott/tiger control=emp.ctl log=emp.log

    After you have executed the abovecommand SQL Loader will shows you the output

    describing how many rows it has loaded. The LOG option of sqlldr specifies where the log file of

    this sql loader session should be created. The log filecontains all actions which SQL loader has performed

    i.e. how many rows were loaded, how many wererejected and how much time is taken to load the rowsand etc. You have to view this file for any errorsencountered while running SQLLoader.

  • 7/29/2019 Unit6 2 FinaL


    Methods of Loading data through

    SQL *loader

    SQL*Loader provides the following methods to load


    Conventional Path Loads

    Direct Path Loads

    External Table Loads

  • 7/29/2019 Unit6 2 FinaL


    Conventional Path Load A conventional path load executes SQL * loader reads a set of

    records from a file and generate insert commands and passesthem to the oracle kernal. Oracle finds places for those records

    in the free blocks in the table and updates any assocated


    In a Direct Path Load SQL * loader create formatted datablocks within server process and write them directly to the

    data files. this requires occasional checks with the database to

    get new locations for data blocks, but no other I/O with

    database kernal is required. The result is a data load process that is dramatically faster

    then conventional path mode.

  • 7/29/2019 Unit6 2 FinaL


  • 7/29/2019 Unit6 2 FinaL


    Conventional and Direct Loads


    SGA Shared pool




    ExtentmanagementData save





    High water mark

    Space used only by conventional load

  • 7/29/2019 Unit6 2 FinaL


    Using Direct-Load Inserts

    INSERT /*+A PPEND */ INTO scott.emp


    SELECT * FROM scott.old_emp;

    Server processE MP table

    High water mark

    Blocks used by inserted rowsFree space after delete

    Used block

  • 7/29/2019 Unit6 2 FinaL


    Parallel Direct-Load Insert


    INSERT /*+PARALLEL(scott.emp,2) */

    INTO scott.emp NOLOGGING

    SELECT * FROM scott.old_emp;

    Slave process Slave process

    High water mark

    Temporary SegmentsFree space after delete

    Used block

  • 7/29/2019 Unit6 2 FinaL


    Parallel Direct Loads

    Temporary segments










    High water markmarkTable

  • 7/29/2019 Unit6 2 FinaL


    Comparing Direct and

    Conventional Path LoadsConventional Load Direct Path LoadUses COMMITs to make changes


    Usesdata saves

    Redo log entries always generated Generates redo only under specific


    Enforces all constraints Enforces only primary key, unique,and NOT NULL

    INSERT triggers fire INSERT triggers do not fire

    Can load into clustered tables Cannot load into clustered tables

    Other users can make changes to


    Other users cannot make changes

    to tables

  • 7/29/2019 Unit6 2 FinaL


    Data Blocks, Extents, and Segments

  • 7/29/2019 Unit6 2 FinaL


    Oracle allocates logical database space for all data in a


    The allocation of database space are data blocks, extents, and


    Data blocks.

    Oracle manages the storage space in the data files of a

    database in units called data blocks.

    A data block is the smallest unit of data used by a database.


    A segment is a set of extents, each of which has been

    allocated for a specific data structure and all of which are

    stored in the same table space.


    An extent is a specific number of contiguous data blocks

    allocated for storing a specific type of information.

  • 7/29/2019 Unit6 2 FinaL


    Multiple-Process Oracle Instance

  • 7/29/2019 Unit6 2 FinaL


  • 7/29/2019 Unit6 2 FinaL


    Bulk Inserts:

    Common Traps and Successful Tricks

    If your data is not being inserted from a flat file,

    SQL*Loader will not be a useful solution.But move a

    large set of data from one table to another, there

    are several common methods forimproving the performance of the data migration:

    Tuning the structures (removing indexes and triggers)

    Disabling constraints during the data migration Using hints and options to improve the transaction


    Isolating the rollback segments for the large transaction.

  • 7/29/2019 Unit6 2 FinaL


    Tuning the structures (removing indexes and triggers)

    Disabling any triggers or indexes that are on the table intowhich data is being loaded.

    if you have a row-level trigger on the target table, that trigger will beexecuted for every row inserted into the table.

    disable the triggers prior to the data load.

    the bulk operation will complete faster than the repeated triggerexecutions.

    the bulk operations execute for all rows that have not already beenprocessed by the triggers.

    Disabling constraints during the data migration you should consider disabling constraints on the table.

    Once the data has been loaded, you can reenable the constraints.

    Using hints and options to improve the transactionperformance Oracle has introduced for data migration tuning. Those options include

    the following:

  • 7/29/2019 Unit6 2 FinaL


    The append hint for insert commands In Direct Path Loader, the APPEND hint loads blocks of data into a


    Use of the APPEND hint may increase your space usage.The no logging option If you are performing a create table as select command, use the

    nologging option

    nologging option to avoid writing to the redo logs during the

    operation.The parallel option Parallel Query uses multiple processes to accomplish a single task.

    you can parallelize both the create table portion and the query.

    If you use the parallel option, you should also use the nologgingoption otherwise the parallel operations will have to wait due toserialized writes to the online redo log files.

    you should first investigate the target tables structuresto make sure youve avoided the common traps as above

  • 7/29/2019 Unit6 2 FinaL


    Isolating the rollback segments for the large

    transaction. May require the creation of a new table space for new rollback segments.


    You can also use programming logic to force inserts to be

    processed in arrays rather than as an entire set.

    Example :-

    COBOL and C support array inserts, thus reducing the size of the

    transactions required to process a large set of data.

  • 7/29/2019 Unit6 2 FinaL


    Bulk Deletes: The truncate CommandTwo problems are generated when user deleted records.

    1) Users attempt to delete all the rows from a table at once theycomplain that the rollback segments are too small and transaction is

    too large.

    2) The rows have all been deleted Even though the segment no longer

    has any rows in it and still maintains all the space that was allocated toit. Therefore, deleting all those rows saved not a single byte of

    allocated space.

    truncate command resolves both of these problems. It is a DDL

    command, not a DML command, so it cannot be rolled back.

    Once you have used the truncate command on a table, its rows aregone, and none of its delete triggers are executed in the process.

    However, the table retains all its dependent objectssuch as grants,

    indexes, and constraints.

  • 7/29/2019 Unit6 2 FinaL


    The truncate command is the fastest way to delete large volumes of

    data. Because it will delete all the rows in a table

    If you use partitions, you can truncate one partition of a table without

    affecting the rest of the tables partitions.

    A sample truncate command for a table.

    truncate table EMPLOYEE drop storage;

    The EMPLOYEE tables rows are deleted

    The drop storage clause is used to deallocate the non-initialspace from the table.

    you can delete all of a tables rows and reclaim all but its initial

    extents allocated space, without dropping the table.

    The truncate command also works for clusters. the reuse storage option is used to leave all allocated space empty

    within the segment that acquired it:

    truncate cluster EMP_DEPT reuse storage;

  • 7/29/2019 Unit6 2 FinaL


    When this example command is executed, all the rows in the

    EMP_DEPT cluster will be instantly deleted.

    To truncate a partition, you need to know its name.

    Example :-

    the partition named PART3 of the EMPLOYEE table is truncated via

    the alter table command:

    alter table EMPLOYEE truncate partition PART3

    drop storage;

    The rest of the partitions of the EMPLOYEE table will be

    unaffected by the truncation of the PART3 partition.

    As an alternative, you can create a PL/SQL program that uses

    dynamic SQL to divide a large delete operation into multiplesmaller transactions.

  • 7/29/2019 Unit6 2 FinaL


    Partitions You can use partitions to isolate data physically.

    Example :-

    you can store the data from one department in a separate

    partition of an the EMPLOYEE table. If you perform a bulk data

    load or deletion on the table, you can customize the partitions

    to tune the data manipulation operation.

    You can truncate a partition and its indexes without affecting the rest

    of the table.

    You can drop a partition, via the drop partition clause of the alter

    table command.

    You can drop a partitions local index. You can set a partition to nologging, reducing the impact of large


    the chief advantage of partitions lies in their ability to be

    managed apart from the rest of the table.

  • 7/29/2019 Unit6 2 FinaL


    For example, being able to truncate a partition enables you to

    delete a large amount of data from a table (but not all of the

    tables data) without generating any redo information.

    In the short term, the beneficiary of this performanceimprovement is the DBA; in the longer term, the entire

    enterprise benefits from the improved availability of the data.