Upload
cissy
View
25
Download
1
Embed Size (px)
DESCRIPTION
PTTP & IDL – External / Internal File Name Validation. Doug Clough SYNERGETICS Engineered Systems. 7 December 2009 www.ptagis.org. IDL – Loader Invocation. PTTP. Files are pre-processed by idl_validateFile.pl one-by-one as IDL runs. IDL – Loader Invocation. PTTP. - PowerPoint PPT Presentation
Citation preview
PTTP & IDL – External / Internal File Name Validation
Doug Clough
SYNERGETICS Engineered Systems
7 December 2009
www.ptagis.org
7 December 2009
Pacific States Marine Fisheries Commission
IDL – Loader Invocation
PTTP
Files are pre-processed byidl_validateFile.pl
one-by-one as IDL runs
7 December 2009
Pacific States Marine Fisheries Commission
IDL – Loader Invocation
PTTP
We must detect external-internal file name
mismatch before file is presented to IDL
Files are pre-processed byidl_validateFile.pl
one-by-one as IDL runs
PROBLEM:
7 December 2009
Pacific States Marine Fisheries Commission
If external and internal names don’t match …
Not only do we want to suppress file loading
• We want to prevent generation of sym-link in archival directory
• We want to return diagnostic message to sender
And what about other files in the PTTP transaction?
• Do we want to reject all of the files if one or more has name problem?
• Do we want to reject only the specific files that have problems?
In either case, we need to take action upstream of PTTP_Ack::genEvents()
Have a look at PTTP_Req::analyze()
• Loops through all files in the TX
• Calls FDVL_ArchiveTree::makeArchiveLink()– Insert code before this call to compare internal and external names– If the names match, create the link and continue as presently coded– Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED– This will suppress generation of an invocation event for the file
7 December 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
Symbolic link in
archive directory
is created first …
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_Req::analyze()
sub analyze { my ( $obj ) = @_;
$TX_ID = $obj->get("PTTP_Req.TX_ID"); $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID;
( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES"));
foreach $FileName ( @DataFiles ) {
# NOTE: Assume that all files of type 'INT' and 'MUL' # will be analyzed by TASS and SBCA. Leave it to # TASS_Monitor and SBCA_Monitor to decide which # files to analyze.
$FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE"); $ReqOper = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER");
if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) && ( $ReqOper =~ /LOAD/i ) ) {
# Capture $SiteCode for use below ... $FileName =~ /^\s*(...).*$/; $SiteCode = $1;
# Unless the file is already present in the final FDVL archive # location, create a symbolic link there pointing to the file in # $TX_Dir. FileType not important here; INT and MUL are equivalent. FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT');
$HasFilesToAnalyze = 'True'; } }
7 December 2009
Pacific States Marine Fisheries Commission
FDVL_ArchiveTree::makeArchiveLink()
sub makeArchiveLink { my ( $PhysicalPath , $ArchivalPath , $FullSpec , $CmdString ); my ( $SourceDir , $FileName , $FileType # INT, MUL, or TAG (INT and MUL give same result) ) = @_;
$PhysicalPath = "${SourceDir}/${FileName}"; $ArchivalPath = &sayAbsolutePath($FileName,$FileType); $FullSpec = "${ArchivalPath}/${FileName}";
unless ( -f $FullSpec ) {
# Create directories if they don't exist ... &checkDirs($FileName,$FileType);
$CmdString = "ln -s $PhysicalPath $FullSpec";
system("$CmdString");
} else {
print "makeArchiveLink ...\n" if ( $DEBUG ); print " Already exists: $FullSpec\n" if ( $DEBUG ); $DUMMY = <STDIN> if ( $DEBUG ); }
}
7 December 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
How are files presented
for loading?
7 December 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event(s)
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
For ‘Invocation’ event PTTP_Dispatcher …
• Invokes handler
• Handler reads possible Input file
• Handler performs action and queues‘Completion’ event
File Type Handler
INTERROGATION PTTP_intIDL
MULTIMON PTTP_mulIDL
TAGGING PTTP_FDVL
run_loader($evt){<copies $FileName to $INTERROGATION_DATA>}
One per file
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_intIDL::runLoader()
sub runLoader { my ( $TX_ID , $F_NAME , $F_TYPE , $RetVal , $IDL_Name , $CmdString , $DEBUG );
$DEBUG = 0; #$DEBUG = 1;
my ( $r_evt ) = @_;
$TX_ID = $$r_evt->get("PTTP_Event.TX_ID"); $F_NAME = $$r_evt->get("PTTP_Event.EventData.F_NAME"); $F_TYPE = $$r_evt->get("PTTP_Event.EventData.F_TYPE");
# Assemble file name for use by Interrogation Data Loader (IDL) ... $IDL_Name = $F_NAME . ":" . $TX_ID . ":" . $F_TYPE;
# Make file available for processing by IDL ... $CmdString = "cp " . "$PTTP_SERVER_PERSIST_DIR/$TX_ID/$F_NAME" . " " . "$PTTP_INT_DATA/$IDL_Name"; system("$CmdString");
}
7 December 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
What we want:
Only do this if external
and internal file
names match!
7 December 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
In other words:
Block event generation
if names don’t match
7 December 2009
Pacific States Marine Fisheries Commission
If external and internal names don’t match …
Not only do we want to suppress file loading
• We want to prevent generation of sym-link in archival directory
• We want to return diagnostic message to sender
And what about other files in the PTTP transaction?
• Do we want to reject all of the files if one or more has name problem?
• Do we want to reject only the specific files that have problems?
In either case, we need to take action upstream of PTTP_Ack::genEvents()
Have a look at PTTP_Req::analyze()
• Loops through all files in the TX
• Calls FDVL_ArchiveTree::makeArchiveLink()– Insert code before this call to compare internal and external names– If the names match, create the link and continue as presently coded– Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED– This will suppress generation of an invocation event for the file
7 December 2009
Pacific States Marine Fisheries Commission
If external and internal names don’t match …
Not only do we want to suppress file loading
• We want to prevent generation of sym-link in archival directory
• We want to return diagnostic message to sender
And what about other files in the PTTP transaction?
• Do we want to reject all of the files if one or more has name problem?
• Do we want to reject only the specific files that have problems?
In either case, we need to take action upstream of PTTP_Ack::genEvents()
Have another look at PTTP_Req::analyze()
• Loops through all files in the TX
• Calls FDVL_ArchiveTree::makeArchiveLink()– Insert code before this call to compare internal and external names– If the names match, create the link and continue as presently coded– Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED– This will suppress generation of an invocation event for the file
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_Req::analyze()
sub analyze { my ( $obj ) = @_;
$TX_ID = $obj->get("PTTP_Req.TX_ID"); $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID;
( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES"));
foreach $FileName ( @DataFiles ) {
# NOTE: Assume that all files of type 'INT' and 'MUL' # will be analyzed by TASS and SBCA. Leave it to # TASS_Monitor and SBCA_Monitor to decide which # files to analyze.
$FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE"); $ReqOper = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER");
if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) && ( $ReqOper =~ /LOAD/i ) ) {
# Capture $SiteCode for use below ... $FileName =~ /^\s*(...).*$/; $SiteCode = $1;
# Unless the file is already present in the final FDVL archive # location, create a symbolic link there pointing to the file in # $TX_Dir. FileType not important here; INT and MUL are equivalent. FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT');
$HasFilesToAnalyze = 'True'; } }
7 December 2009
Pacific States Marine Fisheries Commission
If external and internal names don’t match …
Not only do we want to suppress file loading
• We want to prevent generation of sym-link in archival directory
• We want to return diagnostic message to sender
And what about other files in the PTTP transaction?
• Do we want to reject all of the files if one or more has name problem?
• Do we want to reject only the specific files that have problems?
In either case, we need to take action upstream of PTTP_Ack::genEvents()
Have another look at PTTP_Req::analyze()
• Loops through all files in the TX
• Calls FDVL_ArchiveTree::makeArchiveLink()– Insert code before this call to compare internal and external names– If the names match, create the link and continue as presently coded– Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED– This will suppress generation of an invocation event for the file
7 December 2009
Pacific States Marine Fisheries Commission
Typical A_AST.txt file ...
PTTP_Ack DATE | 2009/03/26 10:57:06 FILES TST00305.A F_TYPE | INT REQUEST OPER | LOAD TST00305.B F_TYPE | INT REQUEST OPER | LOAD TST00305.C F_TYPE | INT REQUEST OPER | LOAD TST00305.D F_TYPE | INT REQUEST OPER | LOAD TST00305.E F_TYPE | INT REQUEST OPER | LOAD SENDER ACCT | ptoc ADDR | FrankenFish NAME | Testing MSM Client (John) STATUS | RECEIVED TX_ID | TX01DE6A28-98AC-4609-94E2-34983A311979
7 December 2009
Pacific States Marine Fisheries Commission
Typical A_AST.txt file ...
PTTP_Ack DATE | 2009/03/26 10:57:06 FILES TST00305.A F_TYPE | INT REQUEST OPER | LOAD TST00305.B F_TYPE | INT REQUEST OPER | LOAD TST00305.C F_TYPE | INT REQUEST OPER | LOAD TST00305.D F_TYPE | INT REQUEST OPER | LOAD TST00305.E F_TYPE | INT REQUEST OPER | LOAD SENDER ACCT | ptoc ADDR | FrankenFish NAME | Testing MSM Client (John) STATUS | RECEIVED TX_ID | TX01DE6A28-98AC-4609-94E2-34983A311979
Change LOAD to REFUSED
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_Ack.pm
#------------------------------------------------------------------------------------------# I N I T I A L I Z A T I O N A N D D E C L A R A T I O N S#------------------------------------------------------------------------------------------
# Includes ...use PTTP_Message;use PTTP_Event;use HashAssemblage;@ISA = ("PTTP_Message","HashAssemblage");
# Initialize operation precedence ...@OpSeq = ( 'OPER' # File LOAD or CORRECTION , 'STAT' # File record-type statistics , 'HIST' # History of OPERations on file , 'TEST' # Move files to "test archive" - do NOT load );
# Initialize list of status values for each operation that# indicate completion or failure (i.e. final state) ...%FiniStat = ( 'OPER' => 'LOADED|REFUSED|INVALID|NET_DUPS|NET_ANOMALIES|SYS_FAIL|AUTH_FAIL|AGG_FAIL' , 'STAT' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL' , 'HIST' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL' , 'TEST' => 'ARCHIVED|FAILED|SYS_FAIL|AUTH_FAIL' );
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_Ack::genEvents()
# Generate an instance of PTTP_Event to initiate processing of first# request for each file named in the A_AST "To Do" list ...sub genEvents { my ( $FileList , @Files , $File , $FileType , $F_Name , $NextOp , $OpVal , $TX_ID , $evt , $EventData , $CanonicalDNA , $DBG_File );
my ( $ack # Reference to PTTP_Ack object instance ) = @_;
# Obtain list of files named in the "To Do" list ... $FileList = $ack->opFiles();
@Files = split(/\n/,$FileList);
# Generate an event for each file with un-fulfilled requests ... # NOTE: At this point, all of the files should qualify. foreach $File ( @Files ) {
$ack->genNextEvent($File);
}}
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_Ack::genNextEvent()
# Generate an instance of PTTP_Event to initiate processing of the# next request for the specified file ...sub genNextEvent { my ( $ack # Reference to PTTP_Ack object instance , $File ) = @_;
# Protect embedded '.' ... $F_Name = $File; $F_Name =~ s/\./\.\./g;
$NextOp = $ack->nextOp($File); $TX_ID = $ack->get("PTTP_Ack.TX_ID"); $OpVal = $ack->get("PTTP_Ack.FILES.$F_Name.REQUEST.$NextOp"); $FileType = $ack->get("PTTP_Ack.FILES.$F_Name.F_TYPE"); unless ( $NextOp =~ /None/i ) { $evt = new PTTP_Event(); $evt->set("PTTP_Event.TX_ID='$TX_ID'"); $evt->set("PTTP_Event.Source='PTTP_Client'"); $evt->set("PTTP_Event.Target='PTTP_Server'");
$EventData = "PTTP_Event\n" . " EventData\n" . " F_NAME | " . $File . "\n" . " F_TYPE | " . $FileType . "\n" . " " . $NextOp . " | " . $OpVal . "\n" ;
$evt->growStructure($EventData); $evt->persist(); } }
7 December 2009
Pacific States Marine Fisheries Commission
Typical A_AST.txt file ...
PTTP_Ack DATE | 2009/03/26 10:57:06 FILES TST00305.A F_TYPE | INT REQUEST OPER | LOAD TST00305.B F_TYPE | INT REQUEST OPER | LOAD TST00305.C F_TYPE | INT REQUEST OPER | REFUSED TST00305.D F_TYPE | INT REQUEST OPER | LOAD TST00305.E F_TYPE | INT REQUEST OPER | LOAD SENDER ACCT | ptoc ADDR | FrankenFish NAME | Testing MSM Client (John) STATUS | RECEIVED TX_ID | TX01DE6A28-98AC-4609-94E2-34983A311979
7 December 2009
Pacific States Marine Fisheries Commission
PTTP_Ack::nextOp()
# Returns the 'next operation' in sequence for a given FileName.# If all operations are complete, returns 'None'.sub nextOp { my ( $F_Name , $OpName , $ReqString , @Requests , $Request , $ReqStat , $FinalStat );
my ( $obj , $FileName ) = @_;
$F_Name = $FileName; $F_Name =~ s/\./\.\./g; # Protect embedded '.' $ReqString = $obj->listKeys("PTTP_Ack.FILES.$F_Name.REQUEST"); ( @Requests ) = split(/\n/,$ReqString);
foreach $OpName ( @OpSeq ) { foreach $Request ( @Requests ) { if ( $OpName =~ /$Request/i ) { $ReqStat = $obj->get("PTTP_Ack.FILES.$F_Name.REQUEST.$OpName"); $FinalStat = $FiniStat{$OpName}; unless ( $ReqStat =~ /$FinalStat/ ) {
# Generate return value ... return $OpName; } } } }
# Generate return value ... return 'None';}
REFUSED is a FinalState
7 December 2009
Pacific States Marine Fisheries Commission
If external and internal names don’t match …
Not only do we want to suppress file loading
• We want to prevent generation of sym-link in archival directory
• We want to return diagnostic message to sender
And what about other files in the PTTP transaction?
• Do we want to reject all of the files if one or more has name problem?
• Do we want to reject only the specific files that have problems?
In either case, we need to take action upstream of PTTP_Ack::genEvents()
Have another look at PTTP_Req::analyze()
• Loops through all files in the TX
• Calls FDVL_ArchiveTree::makeArchiveLink()– Insert code before this call to compare internal and external names– If the names match, create the link and continue as presently coded– Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED– This will suppress generation of an invocation event for the file
PTTP & IDL – External / Internal File Name Validation
Doug Clough
SYNERGETICS Engineered Systems
7 December 2009
www.ptagis.org
That’s all, folks!