View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Software
TestingConformanceDiagnostics And
Mark W. Skall, Division ChiefMark W. Skall, Division Chief
Candy Leatherman, SecretaryCandy Leatherman, Secretaryemail: [email protected]: [email protected]
Tel. No.: 301-975-3262Tel. No.: 301-975-3262
Div. Web Site: http://www.itl.nist.gov/div897/Div. Web Site: http://www.itl.nist.gov/div897/
Program ReviewProgram Review
Division 897 Funding - FY00
STRS - $5,207K
ITS - $585K
OA - $1,615K
Employees: 35 Full Time 3 Part Time 10 Intermittent 5 Guest Researchers
Projected Balance:Projected Balance:$257,062$257,062
(+ $637,570 transferred to (+ $637,570 transferred to 2001 OA Reserve)2001 OA Reserve)
SDCT’sSDCT’s GOALGOAL
To improve quality of software in industry through the development of:To improve quality of software in industry through the development of: TechnologyTechnology MeasurementsMeasurements StandardsStandards
Specific Strategy:Specific Strategy: Conformance TestsConformance Tests Reference ImplementationsReference Implementations Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data Standards developed jointly with Standards developed jointly with industryindustry
Philosophy:Philosophy: Concentrate on key areas at forefront of technologyConcentrate on key areas at forefront of technology Get involved early and partner with industryGet involved early and partner with industry Fill industry voidFill industry void Transfer technology and move onTransfer technology and move on
Excerpts from Assessment Panel Report – Excerpts from Assessment Panel Report – 6/26/006/26/00
Division ReviewDivision Review
““The planning and documentation methods used by SDCT could serve as a The planning and documentation methods used by SDCT could serve as a model for other ITL divisions. The process has used clearly-defined criteria to model for other ITL divisions. The process has used clearly-defined criteria to select state-of-the-art programs with clearly defined priorities and goals, well-select state-of-the-art programs with clearly defined priorities and goals, well-identified roles, and measurable contributions to national and international identified roles, and measurable contributions to national and international standards organizations. Examples include the work on conformance tests for standards organizations. Examples include the work on conformance tests for XML, Distributed Interactive Learning Systems, and for DASE . . .Directions XML, Distributed Interactive Learning Systems, and for DASE . . .Directions can change quickly within the software industry. The division carefully can change quickly within the software industry. The division carefully monitors these changes and acts appropriately. . . .Industry has provided many monitors these changes and acts appropriately. . . .Industry has provided many testimonials concerning the great value of the division’s standards work, testimonials concerning the great value of the division’s standards work, including: ‘The OASIS-NIST XML Conformance test suite is critical for our including: ‘The OASIS-NIST XML Conformance test suite is critical for our industry’, Norbert Mikula, CTO, OASIS; ‘NIST made strong contributions industry’, Norbert Mikula, CTO, OASIS; ‘NIST made strong contributions (X3D) and resolved our knottiest problems’, Don Brutzman, Board of (X3D) and resolved our knottiest problems’, Don Brutzman, Board of Directors, Web3D consortium, and ‘The high quality test suite and certification Directors, Web3D consortium, and ‘The high quality test suite and certification program is an invaluable resource for ATA’, Robert Peel, Director of program is an invaluable resource for ATA’, Robert Peel, Director of Airworthiness and Standards”Airworthiness and Standards”
Excerpts from Assessment Panel Report – Excerpts from Assessment Panel Report – 6/26/006/26/00
Laboratory Level ReviewLaboratory Level Review
““New projects are startedNew projects are started based on the importance of the work to U.S. industry, based on the importance of the work to U.S. industry, and the work viewed by the panel generally had specific, focused goals. An and the work viewed by the panel generally had specific, focused goals. An example of this approach is the work on XML done in SDCT. This project is example of this approach is the work on XML done in SDCT. This project is notable because it addresses a major standard on which industry was making notable because it addresses a major standard on which industry was making little progress, and it also effectively leveraged skills and approaches developed little progress, and it also effectively leveraged skills and approaches developed in a now terminated project on VRML. . .Another example of the improved in a now terminated project on VRML. . .Another example of the improved planning process is the existence of clear termination criteria, which have been planning process is the existence of clear termination criteria, which have been useful in sunsetting projects. For example, the Real-Time Java project and the useful in sunsetting projects. For example, the Real-Time Java project and the work on RBAC satisfied their completion criteria and therefore ended this work on RBAC satisfied their completion criteria and therefore ended this year.”year.”
Standards and Conformance Testing GroupStandards and Conformance Testing Group
Software Quality GroupSoftware Quality Group
Interoperability GroupInteroperability Group
Software Diagnostics and Conformance Testing
• Development of conformance testsDevelopment of conformance tests
• Development of reference implementationsDevelopment of reference implementations
• Research into better ways to do conformance testingResearch into better ways to do conformance testing
• Develop standards jointly with industryDevelop standards jointly with industry
Standards andConformance Testing Group
Group Leader: Lynne RosenthalGroup Leader: Lynne RosenthalSupy Proj. Leader: Lisa CarnahanSupy Proj. Leader: Lisa Carnahan
Clare Lucey, SecretaryClare Lucey, Secretary
Standards andConformance Testing Group
Mary BradyMary BradyLisa CarnahanLisa CarnahanLaurent Ciarletta (GR)Laurent Ciarletta (GR)Anthony Cincotta (S)Anthony Cincotta (S)Chris DabrowskiChris DabrowskiAlden DimaAlden DimaLeonard GallagherLeonard GallagherLeonard GebaseLeonard GebaseNeil Gima (S)Neil Gima (S)Alan GoldfineAlan Goldfine
Martha GrayMartha GrayMichael KassMichael KassClare LuceyClare LuceyThomas LogueThomas LogueCarmelo Montanez-RiveraCarmelo Montanez-RiveraThomas RhodesThomas RhodesRichard RivelloRichard RivelloJacqeline SchneiderJacqeline SchneiderJohn TebbuttJohn TebbuttMarie-Noelle Terasse (GR) Marie-Noelle Terasse (GR) Mark ZimmermanMark Zimmerman
Division 897Standards and Conformance Testing
Group
• Develop methods to automate software testingDevelop methods to automate software testing
• Develop software diagnostic toolsDevelop software diagnostic tools
• Develop reference dataDevelop reference data
• Formal methodsFormal methods
Software Quality Group
Group Leader: John BarkleyGroup Leader: John BarkleyDebbie Blackstone, SecretaryDebbie Blackstone, Secretary
Software Quality Group
Tamer Ahmed Tamer Ahmed (GR)(GR)Paul Ammann (F)Paul Ammann (F)Debbie BlackstoneDebbie BlackstoneDavid Brinkley (F)David Brinkley (F)Paul BlackPaul BlackNeva CarlsonNeva CarlsonJohn CherniavskyJohn CherniavskyAnthony CincottaAnthony Cincotta
Keith Gallagher Keith Gallagher (F)(F)Roger Gima (S)Roger Gima (S)Michael KooMichael KooMary LaamanenMary LaamanenJames LyleJames LyleWilliam MajurskiWilliam MajurskiDouglas WhiteDouglas White
Division 897Software Quality
Group
• Ensure Federal agency requirements are input into voluntary standards Ensure Federal agency requirements are input into voluntary standards committees - work through Federal CIO Councilcommittees - work through Federal CIO Council
• Help Federal agencies and industry achieve interoperability through Help Federal agencies and industry achieve interoperability through application of Division productsapplication of Division products
• Provide technical support to voluntary standards committeesProvide technical support to voluntary standards committees
• Support NIST paperless office effort utilizing digital signaturesSupport NIST paperless office effort utilizing digital signatures
InteroperabilityGroup
Acting Group Leader: Lisa CarnahanActing Group Leader: Lisa CarnahanVacant, SecretaryVacant, Secretary
InteroperabilityGroup
Daniel AllenDaniel AllenBruce Bargmeyer (RA)Bruce Bargmeyer (RA)Frederick BolandFrederick BolandGary FisherGary FisherLarry Fitzwater (GR)Larry Fitzwater (GR)
Elizabeth FongElizabeth FongKathryn HarvillKathryn HarvillRoy MorganRoy MorganJudith NewtonJudith NewtonGertrude Gertrude SherwoodSherwood
Division 897Interoperability
Group
StrategyStrategy
•Conformance TestsConformance Tests
Reference ImplementationsReference Implementations
Standards developed jointly with industryStandards developed jointly with industry
Diagnostic TestsDiagnostic Tests
Research to reduce costs of testingResearch to reduce costs of testing
Reference DataReference Data
Development of Conformance TestsDevelopment of Conformance Tests
• Emphasis on newer technologiesEmphasis on newer technologies
• Tests developed in parallel with standardsTests developed in parallel with standards
• Tests developed before implementationsTests developed before implementations
• Need buy-in from industryNeed buy-in from industry
• Need high impactNeed high impact
• Need technology transferNeed technology transfer
Development of Conformance Tests (cont’d)Development of Conformance Tests (cont’d)
• Legacy activitiesLegacy activitiesCGMCGMSQLSQLPOSIXPOSIXAda and other programming languageAda and other programming languageVRMLVRML
• CurrentCurrent activities activitiesXML and companion standards (DOM, XSL, etc.) XML and companion standards (DOM, XSL, etc.) Digital TV Digital TV X3DX3DLearning Technologies (IMS)Learning Technologies (IMS)
XML Conformance TestingXML Conformance Testing
Project Team:Project Team:
Mary BradyMary BradyCarmelo MontanezCarmelo MontanezRicky RivelloRicky RivelloMark ZimmermanMark Zimmerman
Electronic CommerceElectronic Commerce
XML Technologies
B2C(business-to-consumer)
InteractiveTelevision
(DASE, SMPTE)
Internet / World Wide Web
B2B(business-to-business)
Manufacturing(ICM)
DistributedLearning
(IMS)
Electronic Commerce Growth– Growth is exponential, already outpacing last year’s predictions– Changing the way businesses do business– Revolutionizing the way we consume and play
Advances are necessary in:– Tools for describing / sharing information
XML technologies
– Applying IT to solve vertical market demands NIST provides:
– XML Technologies Conformance Testing
– NIST developed tests, available on-line, for use in testing applications– NIST technical leadership, neutral third-party – Unbiased Feedback of Specification Errors
– Vertical Markets Applied IT research
Why NIST?
HTML or XML?
Figure 1: HTML Purchase Order
<H1>Purchase Order</H1><H1>Purchase Order</H1>
<UL><UL>
<LI><b> Mary Brady</b> <LI><b> Mary Brady</b>
<LI>NN Rm 572, Gaithersburg, MD, 20899<LI>NN Rm 572, Gaithersburg, MD, 20899
<LI>1234<LI>1234
<LI><I>Trees</I><LI><I>Trees</I>
<LI>50.00<LI>50.00
<LI>20<LI>20
<LI>1000.00<LI>1000.00
<LI>Be sure to water the root ball day of <LI>Be sure to water the root ball day of deliverydelivery
<LI>Susan Carscadden<LI>Susan Carscadden
<LI>NIST, Bldg 301, Gaithersburg, MD, <LI>NIST, Bldg 301, Gaithersburg, MD, 2089920899
</UL</UL
What is XML?•Domain-specific languages•Data separate from display•Self-describing data•Schemas
<Purchaser><Purchaser><Name> Mary Brady </Name><Name> Mary Brady </Name><Address Street=NN Rm 572 <Address Street=NN Rm 572
City=Gaithersburg, State=MD City=Gaithersburg, State=MD Zip=20899</Address>Zip=20899</Address>
</Purchaser></Purchaser><Item><Item><Part_number>1234</Part_number><Part_number>1234</Part_number><ItemDescription>Trees</<ItemDescription>Trees</
ItemDescription>ItemDescription><Cost>50.00</Cost><Cost>50.00</Cost><Quantity>20</Quantity><Quantity>20</Quantity><TotalCost>1000.00</TotalCost><TotalCost>1000.00</TotalCost><Instructions>Be sure to water the root <Instructions>Be sure to water the root
ball day of delivery</Instructions>ball day of delivery</Instructions></Item></Item><ShipTo><ShipTo><Name>Susan Carscadden</Name><Name>Susan Carscadden</Name><Address Street=NIST, Bldg 301 <Address Street=NIST, Bldg 301
City=Gaithersburg State=MD City=Gaithersburg State=MD Zip=20899</Address>Zip=20899</Address>
</ShipTo></ShipTo>
Figure 2: XML Purchase Order
XML TechnologiesXML Technologies
XML Syntax, Information Set, Schema, Linking, Fragment
Registries Repositories
Namespaces
Programming Languages
DOM
Electronic Commerce, Distance Learning, Health CareManufacturing, Interactive Televison
TransformationsFormatting Objects
Stylesheets Query Language
Signatures
Information Description
Ratings Libraries Control
Domain Specific
Boa
rds
Med
ical
Edu
catio
n
Pro
cure
men
t
...Aut
o P
arts XML.ORG
BIZTalk
E-BusinessLearning Objects
XML/EDI
... ...
XML Conformance TestingXML Conformance Testing
OASIS XML ConformanceOASIS XML Conformance Sun, IBM, Fuji Xerox, Sun, IBM, Fuji Xerox,
DataChannel, MicroStar, DataChannel, MicroStar, W3C membersW3C members
Why NIST?Why NIST? Technical LeadershipTechnical Leadership Coalesced industry Coalesced industry
partnerspartners18 months
First DeliverableFirst Deliverable XML Test SuiteXML Test Suite XML files & XSL XML files & XSL
StylesheetStylesheet6 months
XML Test Suite•1000 XML Tests
•DTD + 4000 lines of XML•400 lines of XSL Stylesheet
Future WorkFuture Work XML StyleSheets, XML StyleSheets,
SchemasSchemas XML NS, XLink, XPtrXML NS, XLink, XPtrContinued Support
DOM Conformance TestingDOM Conformance TestingW3C, OASISW3C, OASIS
Sun, IBM, Microsoft, Sun, IBM, Microsoft, Netscape, OracleNetscape, Oracle
XML ‘98XML ‘98 Met with W3C WG ChairMet with W3C WG Chair NIST asked to develop NIST asked to develop
teststestsXML ‘98
NIST ReleasedNIST Released Fundamental, Extended Fundamental, Extended
(ECMAScript)(ECMAScript) Interactive test harnessInteractive test harnessJune ‘99
DOM Test Suite• 300 ECMAScript Tests
• 14,500 lines of code
Future WorkFuture Work Fundamental, Extended Fundamental, Extended
(Java)(Java) HTML (ECMAScript & Java)HTML (ECMAScript & Java) DOM Level 2DOM Level 2
Continued Support
FY00 Progress -TestsFY00 Progress -Tests Completed XML Test Suite (release 1)
Completed XML Test Suite (release 2)
– Incorporated new XML tests and fixes Completed DOM-Ecmascript Test Suite
(release 1) Completed DOM-Java Test Suite (release
1) Develop XSLT Test Suite
– partnership with Lotus Corp. and OASIS
FY00 Progress -Committee FY00 Progress -Committee LeadershipLeadership
Chair OASIS Conformance and XML Chair OASIS Conformance and XML Testing CommitteesTesting Committees
Provide leadership for XSLT and Schema Provide leadership for XSLT and Schema Testing committeesTesting committees
Provide leadership for UN/CEFACT Provide leadership for UN/CEFACT ebXML initiative (conformance and ebXML initiative (conformance and reg/rep)reg/rep)– Provide conformance guidance– Develop conformance clause – Harmonize OASIS and ebXML efforts
FY01 PlansFY01 Plans
Complete XSLT test suite Develop test suites for XML Schema
and other XML Technologies Develop XML-based automated test
tools to improve test development Continue to chair OASIS testing
committees
FY00 ResourcesFY00 Resources 3.5 FTE3.5 FTE $700K (STRS)$700K (STRS)
FY01 ResourcesFY01 Resources 5 FTE Travel required to collaborate in
standards and test development efforts Equipment: 10K
XML Conformance TestingXML Conformance Testing
Customers:Customers:– IT industry: e.g., IBM, Sun, Microsoft, Oracle, IT industry: e.g., IBM, Sun, Microsoft, Oracle,
DataChannel, DocumentumDataChannel, Documentum
– Vertical industries: e.g., Education (IMS), Financial Vertical industries: e.g., Education (IMS), Financial (e.g., Dun and Bradstreet), Travel (e.g., Sabre)(e.g., Dun and Bradstreet), Travel (e.g., Sabre)
Impact:Impact:– Conformance test suites have been used to Conformance test suites have been used to
improve the quality of XML/DOM processors improve the quality of XML/DOM processors (many of which are embedded in other XML (many of which are embedded in other XML software solutions). These are used in many software solutions). These are used in many vertical markets by millions of customers.vertical markets by millions of customers.
– Feedback to standards developers improve the Feedback to standards developers improve the specificationsspecifications
Interactive TV Interactive TV
Project Team:Project Team:
Alan GoldfineAlan GoldfineJohn BarkleyJohn BarkleyDoug WhiteDoug WhiteLen GebaseLen Gebase
Interactive TVInteractive TV
• TV and the Internet are each recognized as “a technology …TV and the Internet are each recognized as “a technology …that seems to change everything” (BW, Oct 4 ‘99)that seems to change everything” (BW, Oct 4 ‘99)
• Interactive TV is the convergence of TV and the InternetInteractive TV is the convergence of TV and the Internet
• Profound impact on Electronic Commerce expectedProfound impact on Electronic Commerce expected
• WebTV: 1 million subscribers, 350 hrs/wk of programmingWebTV: 1 million subscribers, 350 hrs/wk of programming
• AOLTV: projected 2 million subscribers in 2 yearsAOLTV: projected 2 million subscribers in 2 years
• ITL partnering to develop the standards and tests required ITL partnering to develop the standards and tests required for the for the success of Interactive TV success of Interactive TV
Interactive TV
• Wireless phones do emailWireless phones do email
• Internet does broadcast radio and TVInternet does broadcast radio and TV
• Broadcast TV references links to Web pagesBroadcast TV references links to Web pages
Traditionally separate media are converging:Traditionally separate media are converging:
Currently, for the viewer:Currently, for the viewer:
accessed separatelyaccessed separately}}TV:TV:
(non-interactive)(non-interactive)
OROR
Web:Web:(interactive)(interactive)
Interactive TVInteractive TV
With DASE/DDE, TV and Web are converging into a With DASE/DDE, TV and Web are converging into a singlesingle medium: medium:
DASE/DDEDASE/DDE
DemonstrationDemonstration
Current production:Current production:
Web site with associated productsWeb site with associated products Golf matches on TVGolf matches on TV
Interactive TVInteractive TV
DASE/DDE: DASE/DDE: singlesingle medium medium
DASE/DDEDASE/DDE
DemonstrationDemonstrationInteractive TVInteractive TV
• Members include: Members include: ABC, CBS, IBM, Intel, Lucent, ABC, CBS, IBM, Intel, Lucent, Microsoft, NBC, Warner BrothersMicrosoft, NBC, Warner Brothers
• DASE (DTV Application Programming Environment): DASE (DTV Application Programming Environment): Standard for a platform independent, high-level Standard for a platform independent, high-level abstraction for integrating DTV and the Internetabstraction for integrating DTV and the Internet
• DASE specifies programming environment for DTV DASE specifies programming environment for DTV receiversreceivers
• Downloadable DASE applications portable across Downloadable DASE applications portable across different TV set topsdifferent TV set tops
Advanced Television Systems Committee (ATSC)Advanced Television Systems Committee (ATSC)
Interactive TVInteractive TV
Interactive TVInteractive TV
• Joint project with the High Performance Systems and Joint project with the High Performance Systems and Services Division (895)Services Division (895)
•Chair DASE Conformance Working Group where key Chair DASE Conformance Working Group where key players include: players include: ABC, Gateway, Microsoft, SunABC, Gateway, Microsoft, Sun
• Partner with Unisoft to develop conformance testsPartner with Unisoft to develop conformance tests
• Reference implementation for programming environment Reference implementation for programming environment
NIST’s Role in ATSC DASE:NIST’s Role in ATSC DASE:
• Members include: Members include: ABC, CBS, IBM, Intel, Microsoft, ABC, CBS, IBM, Intel, Microsoft, NBC, Warner BrothersNBC, Warner Brothers
• DDE (Declarative Data Essence): Standard for Internet DDE (Declarative Data Essence): Standard for Internet content and bindings to analog and digital streams content and bindings to analog and digital streams (formerly ATVEF)(formerly ATVEF)
• Declarative Content: HTML, CSS, ECMAScript, DOM, Declarative Content: HTML, CSS, ECMAScript, DOM, Triggers (for syncing the two mediums)Triggers (for syncing the two mediums)
Society of Motion Picture and Society of Motion Picture and Television Engineers (SMPTE):Television Engineers (SMPTE):
Interactive TVInteractive TV
Interactive TVInteractive TV
• Contribute text to specificationContribute text to specification
• Partner with Unisoft to develop conformance tests Partner with Unisoft to develop conformance tests
• Develop conformance tests for triggers and stream binding to Develop conformance tests for triggers and stream binding to integrate with existing Unisoft DDE element testsintegrate with existing Unisoft DDE element tests
• Develop prototype receiver testbedDevelop prototype receiver testbed
• Participate in Applications Data Essence (ADE) Study GroupParticipate in Applications Data Essence (ADE) Study Group
NIST’s Role in SMPTE DDE:NIST’s Role in SMPTE DDE:
FY00 ProgressFY00 Progress Chair DASE conformance task groupChair DASE conformance task group Conformance section of the DASE Conformance section of the DASE
Standard completedStandard completed Review of DASE conformance test Review of DASE conformance test
assertions from Unisoft completedassertions from Unisoft completed DDE Standard completed, adoption DDE Standard completed, adoption
underwayunderway DDE Bindings Standard underwayDDE Bindings Standard underway Test assertions for DDE triggers underwayTest assertions for DDE triggers underway Testbed for DDE receivers underwayTestbed for DDE receivers underway ADE Study draft completedADE Study draft completed
FY01 PlansFY01 Plans
Continue to chair DASE conformance Continue to chair DASE conformance groupgroup
DASE Standard adoptedDASE Standard adopted Continue DASE conformance test Continue DASE conformance test
development with Unisoftdevelopment with Unisoft DDE and DDE bindings Standards adoptedDDE and DDE bindings Standards adopted Develop DDE conformance tests for Develop DDE conformance tests for
triggers, integrate with Unisoft tests triggers, integrate with Unisoft tests Develop test assertions for DDE bindingsDevelop test assertions for DDE bindings Complete DDE Receiver testbedComplete DDE Receiver testbed Begin DDE-2 Standard developmentBegin DDE-2 Standard development
FY00 ResourcesFY00 Resources 2 FTE2 FTE $400K (STRS, ATP-$100K)$400K (STRS, ATP-$100K)
FY01 ResourcesFY01 Resources 4.5 FTE4.5 FTE Equipment: $20KEquipment: $20K Travel required for standards Travel required for standards
meetings and testing meetings and testing collaborationscollaborations
Interactive TVInteractive TV CustomersCustomers
– ITIT, e.g., IBM, Microsoft, e.g., IBM, Microsoft– broadcast TVbroadcast TV, e.g., NBC, DirecTV, e.g., NBC, DirecTV– entertainmententertainment, e.g., Disney, Universal, e.g., Disney, Universal– consumer electronicsconsumer electronics, e.g., RCA, Intel, e.g., RCA, Intel
ImpactImpact– New Medium: TV/Internet convergenceNew Medium: TV/Internet convergence– Interactive applications portable across receivers from different Interactive applications portable across receivers from different
manufacturersmanufacturers– ATSC and SMPTE invited NIST participationATSC and SMPTE invited NIST participation– NIST chairs DASE Conformance GroupNIST chairs DASE Conformance Group
X3D Conformance TestingX3D Conformance Testing
Project Leader:Project Leader:
Mike KassMike Kass
ObjectivesObjectives Continue the NIST/Web3D partnership begun in 1996Continue the NIST/Web3D partnership begun in 1996
– ITL created the VRML Test Suite (VTS) and Viper parserITL created the VRML Test Suite (VTS) and Viper parser Create an X3D Test Suite ( XTS )Create an X3D Test Suite ( XTS )
– Modify VRML test requirements and test descriptionsModify VRML test requirements and test descriptions– Translate existing VRML tests to X3D formatTranslate existing VRML tests to X3D format
Chair the Interoperability/Conformance WGChair the Interoperability/Conformance WG– Interface with developers and specification writers to resolve ambiguities in specificationInterface with developers and specification writers to resolve ambiguities in specification– Provide feedback to developers on browser conformanceProvide feedback to developers on browser conformance
FY00 ProgressFY00 Progress
Completed translation of VTS to X3D Test Suite (XTS)
Developed test harness to enable developers to contribute tests
Interest from DraW and Sun for using the test suite and making the results public
FY01 PlansFY01 Plans
Low level support - assisting community in using the VRML and X3D Test Suites
No new development planned. Maintenance as needed
FY00 ResourcesFY00 Resources 0.75 FTE $150K (STRS)
FY01 Resources 0.2 FTE (minimal) No Travel is expected
X3D Conformance TestingX3D Conformance Testing
Customers:Customers:– X3D Consortium including Sun, Sony X3D Consortium including Sun, Sony
DraW Computing, Shout, BlaxxunDraW Computing, Shout, Blaxxun
Impact:Impact:– Conformance test suites are being Conformance test suites are being
used to improve the quality of X3D used to improve the quality of X3D browsersbrowsers
– DraW Computing has said that they DraW Computing has said that they would make their test results publicwould make their test results public
Learning TechnologiesLearning Technologies
Project team:Project team:Martha GrayMartha GrayTom RhodesTom RhodesJacki SchneiderJacki SchneiderLisa CarnahanLisa Carnahan
Learning TechnologiesLearning TechnologiesLearning technology is: Learning technology is: Learning anytime, anyplaceLearning anytime, anyplace Interactive, distributive learning systems Interactive, distributive learning systems
In collaboration with industry, NIST/ITL is:In collaboration with industry, NIST/ITL is: defining requirements and specificationsdefining requirements and specifications developing prototypes and reference implementationsdeveloping prototypes and reference implementations developing testing techniques and toolsdeveloping testing techniques and tools
Why Now?Why Now? ““The Gartner Group estimates that companies in the United The Gartner Group estimates that companies in the United
States spent $58 billion on training in 1998, and of that, $9.4 States spent $58 billion on training in 1998, and of that, $9.4 billion went to IT-related training. They expect this figure to billion went to IT-related training. They expect this figure to grow to $10.5 billion this year.” grow to $10.5 billion this year.” Gilbane Report, July/Aug 1999Gilbane Report, July/Aug 1999
““Just as electronic commerce exploded over the past two Just as electronic commerce exploded over the past two years, the stage is now set for ‘e-learning’ to become the years, the stage is now set for ‘e-learning’ to become the next big wave in Internet-based applications” next big wave in Internet-based applications” Cisco Systems CEO Cisco Systems CEO and President John Chambersand President John Chambers
““Worldwide, the education and training market is over a Worldwide, the education and training market is over a trillion (US) dollars.” trillion (US) dollars.” IEEEIEEE
ProjectsProjects1.1. IMS Global Learning Consortium, Inc.IMS Global Learning Consortium, Inc.
Chair conformance teamChair conformance team– advise members on writing IMS specificationsadvise members on writing IMS specifications– ensure testability of specifications ensure testability of specifications – advise on how to test and set up testing programsadvise on how to test and set up testing programs
Develop Meta-data repository reference implementation Develop Meta-data repository reference implementation
2.2. IEEE and ISOIEEE and ISO Provide conformance expertise Provide conformance expertise Harmonization of efforts among specificationsHarmonization of efforts among specifications
3.3. DoD’s Advanced Distributive Learning (ADL)DoD’s Advanced Distributive Learning (ADL) Executive Order 13111 on Federal Training - NIST/ITL co-Executive Order 13111 on Federal Training - NIST/ITL co-
leaders with ADL and DOL. leaders with ADL and DOL.
FY00 ProgressFY00 Progress Continue collaboration with IMS, IEEE LT
committee and NIST/ATP Chair IMS conformance testing activities Developed IMS conformance test policies and
procedures Served as Technical Lead of SBIR solicitation,
assessed and recommend proposals Develop repository of learning objects (this
work coordinates with Reg/rep subproject)
FY01 PlansFY01 Plans
Continue to chair IMS conformance committee providing expertise to emerging IMS specifications
Develop a conformance test plan for applicable IMS specifications
FY00 ResourcesFY00 Resources 1.5 FTE1.5 FTE $300K (STRS, ATP)$300K (STRS, ATP)
FY01 ResourcesFY01 Resources 1 FTE Minimal travel required to attend
standards meetings and interoperability demos
Learning TechnologiesLearning Technologies
Customers:Customers:– IMS, DoL, DoD, Simon&Schuster, Sun, IBM, Apple, IMS, DoL, DoD, Simon&Schuster, Sun, IBM, Apple,
Microsoft, UNC, Calif State UMicrosoft, UNC, Calif State U
Impact:Impact:– Use of standard metadata for global learning will Use of standard metadata for global learning will
improve discovery, evaluation, and use/reuse of improve discovery, evaluation, and use/reuse of learning objectslearning objects
– NIST’s conformance expertise contributed to NIST’s conformance expertise contributed to clearer, less ambiguous, testable IMS clearer, less ambiguous, testable IMS specificationsspecifications
– Conformance test plans and procedures will Conformance test plans and procedures will enable IMS to determine compliant productsenable IMS to determine compliant products
ATA CGM Testing ProgramATA CGM Testing Program
FY00 ProgressFY00 Progress Completed final release of the ATA CGM
Test Suite Provided guidance to ATA Recognized Test
Laboratory on using the test suite and assessing clients
Chair Control Board - resolve questions of interpretation of the test suite and standard.
Initiated discussions on ATA's eCommerce activities - explore a possible XML testing program
FY01 PlansFY01 Plans
Continue to support ATA's CGM Test Program– No new CGM development planned– Maintenance as needed
Provide guidance on the use of NIST's XML test suite and its applicability and use in ATA
FY00 ResourcesFY00 Resources 0.25 FTE0.25 FTE $75K (STRS)$75K (STRS)
FY01 ResourcesFY01 Resources 0.25 FTE Travel required to support ATA test
program and potential for an ATA XML test program
CGMCGM
Customers:Customers:– Air Transport Association – Airlines and their Air Transport Association – Airlines and their
suppliers (e.g., United, Northwest, Airbus, suppliers (e.g., United, Northwest, Airbus, FedX, Boeing, Rolls-Royce)FedX, Boeing, Rolls-Royce)
– CGM Developers (e.g., Autotrol, Intergraphics) CGM Developers (e.g., Autotrol, Intergraphics)
Impact:Impact:– NIST Conformance test suite used by ATA NIST Conformance test suite used by ATA
CGM Test Program to validate CGM CGM Test Program to validate CGM interpretersinterpreters
– Quality, reliable CGM products used for Quality, reliable CGM products used for representing graphics in airframe representing graphics in airframe maintenance manualsmaintenance manuals
Operational Testing IssuesOperational Testing Issues
Test suites should be available on the InternetTest suites should be available on the Internet
Test suites should be free of chargeTest suites should be free of charge
Operational testing should be done by private sectorOperational testing should be done by private sector
Industry associations may issue certificatesIndustry associations may issue certificates
Technology Transfer from NIST to ATA
Developed CGM Interpreter Test Suite
Developed Test Program FrameworkDeveloped test procedures Formed and chair Control BoardDeveloped Test Lab criteria and evaluation criteria
Assessed Test Lab for ATA recognition
Serve as technical adviser
Completed transfer
Testing ServicesStatus of Software Operational Testing Services
TESTSERVICE
STARTDATE
TERMINATEDATE
CERTIFICATEOFFERED BY:
TESTING OFFEREDBY:
NSTL NSTLTerwilliger Terwilliger
SQL 1990 July 1 1997
EDS EDSIEEE Mindcraft, Perennial
(NVLAPs)POSIX 1991 Dec 31 1997
X/Open X/OpenAda 1985 July 1997 AJPO Recognized Test
LabsFortran78 1979 (GSA)
1986 (NIST)June 7 1998 EDS EDS
Cobol 85 1974 (Navy)1986 (NIST)
June 7 1998 EDS EDS
C 1989 Oct 1 1998 EDS and/orPerennial
EDS and/or Perennial
CGM 1994 Oct 1 1998 ATA ATA recognized TestLabs
Testing Services (cont’d)Testing Products or Services - Offered by Other Organizations - Developed with NIST Assistance
TEST SERVICE
START DATE
SPONSORED BY TYPE OF SERVICE
VRML 1997 VRML Consortium (VRMLC)
Self testing of browsers using NIST VTS. Self testing of VRML Content using NIST Viper
VRMLC is discussing a certification program
Spatial Data Transfer (TVP)
1998 US Geological Survey (USGS)
Test implementations (encoders, decoders, transfers) of SDTS
plans for EDS to issue Certificates and do testing
IMS 1999 IMS Test implementations of IMS specifications and prototypes
working with IMS to develop framework for certification program
StrategyStrategy
Conformance TestsConformance Tests
•Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data
Reference ImplementationsReference Implementations
3 Purposes3 Purposes
Used as a metric of comparison in conformance testingUsed as a metric of comparison in conformance testing
Used to sort out errors and ambiguities in standardsUsed to sort out errors and ambiguities in standards
Used as a prototypeUsed as a prototype
AromaPervasive Computing Project
Project Team:Project Team:Alden DimaAlden DimaBill MajurskiBill MajurskiMary LaamanenMary Laamanen
AromaAromaPervasive Computing ProjectPervasive Computing Project
Emerging trend towardEmerging trend toward– numerous, easily accessible, often inconspicuous numerous, easily accessible, often inconspicuous
computing devices and sensors; mobile or embedded in computing devices and sensors; mobile or embedded in the environment; connected to ubiquitous network the environment; connected to ubiquitous network structurestructure
Industry is asking forIndustry is asking for– Standards for service discovery, APIs, e-commerce servicesStandards for service discovery, APIs, e-commerce services– Testing and diagnostic tools to improve quality, Testing and diagnostic tools to improve quality,
interoperabilityinteroperability– Reference implementationsReference implementations
Aroma = multi-division effort to:Aroma = multi-division effort to:– Explore standardization and measurement issuesExplore standardization and measurement issues– Attract industry interest and partnershipsAttract industry interest and partnerships
Aroma PrototypeAroma Prototype (Aroma adaptor + Aroma projector)
Goals - To discover technical, standards and measurement issues Goals - To discover technical, standards and measurement issues related to:related to:– Connecting portable wireless devices to traditional networksConnecting portable wireless devices to traditional networks– Service discovery, self-configuration and dynamic resource sharingService discovery, self-configuration and dynamic resource sharing– Mobile code and dataMobile code and data– Software infrastructure needed to create and manage pervasive Software infrastructure needed to create and manage pervasive
services and applications services and applications
FY00 ProgressFY00 Progress
Present papers at NIST PC 2000, JavaOne, and International Pervasive Computing Workshop
CCo-chaired NIST Pervasive Computing 2000 conference
Developed Aroma Adapter-Projector with Div. 892
Measured performance of Aroma adapter Developed conceptual model based on the OSI
Reference Model Exploring low-end pervasive computing
– Embedded and 8,16-bit processors– Collaboration with US Naval Academy
FY01 PlansFY01 Plans
Co-chair NIST Pervasive Computing 2001 conference
Explore use of ADL in pervasive computing model
Implement a virtual private smart space (VPSS) test bed
Extend Aroma projector to include digital cameras and VPSS technology
Continue performance measurements Develop diagnostic and testing tools
FY00 ResourcesFY00 Resources
1.50 FTE $300K (STRS, OA-DISA)
FY01 Resources 3 FTE Travel required to conferences to present
work and collaborate with colleagues Equipment: $15K
Pervasive ComputingPervasive ComputingCustomers:Customers:
– IT industry: the usual suspectsIT industry: the usual suspects
– Consumers: e.g., GE, Sunbeam, Phillips, Northeast Consumers: e.g., GE, Sunbeam, Phillips, Northeast Utilities, XeroxUtilities, Xerox
– Universities: USNA, UMD, UC Berkeley Universities: USNA, UMD, UC Berkeley
Impact:Impact:– Solicited industry needs for standards and Solicited industry needs for standards and
measurementsmeasurements
– PC 2000 conference launched awareness of NIST as PC 2000 conference launched awareness of NIST as key playerkey player
– Papers presented at conference stimulated Papers presented at conference stimulated discussions on pc and standardsdiscussions on pc and standards
– Ref imp demonstrated feasibility of PCRef imp demonstrated feasibility of PC
– Enable widespread adoption and integration with Enable widespread adoption and integration with software tools to create, manage, measure and test software tools to create, manage, measure and test pervasive services and applicationspervasive services and applications
XML Registry/RepositoryXML Registry/Repository(Reg/Rep)(Reg/Rep)
Project Team:Project Team:
Lisa CarnahanLisa CarnahanLen GallagherLen GallagherMike KassMike Kass
Registry
DISCOVERY
RETRIEVAL
Internet
Registry
XML Registry/RepositoryXML Registry/RepositoryReference ImplementationReference Implementation
Built to OASIS specificationBuilt to OASIS specification Being used by both OASIS and Being used by both OASIS and
UN/CEFACT (ebXML) UN/CEFACT (ebXML) Built using IMS Meta-data Built using IMS Meta-data
specificationspecification Apply lessons learned to EC Apply lessons learned to EC
repositoriesrepositories
FY00 ProgressFY00 Progress
Revised the OASIS Registry and Repository Specification
Implement a reference implementation
Participate in ebXML Reg/Rep team– Ensure testability of specification– ensure compatibility between ebXML
and OASIS specifications
FY01 PlansFY01 Plans
Update reference implementation Develop conformance test
suite/data for OASIS specification Coalesce the ebXML and OASIS
specifications
FY00 ResourcesFY00 Resources 1.5 FTE $325K (STRS)
FY01 Resources 2 FTE Travel required to attend
standards meetings
XML Registry/RepositoryXML Registry/Repository CustomersCustomers
– EC/Enterprise Integrators: Data Channel, Documentum, EC/Enterprise Integrators: Data Channel, Documentum, Oracle, Microsoft, Sun, IBMOracle, Microsoft, Sun, IBM
– Consumers: Vertical market partnerships and supply-Consumers: Vertical market partnerships and supply-chains (e.g., automotive, insurance, travel, chains (e.g., automotive, insurance, travel, manufacturing)manufacturing)
ImpactImpact– Repositories allow efficient use of XML and eliminate Repositories allow efficient use of XML and eliminate
duplicationduplication– Enable SMEs to participate in EC marketplaces and Enable SMEs to participate in EC marketplaces and
supply chainssupply chains– Reduce cost of doing business through timeliness and Reduce cost of doing business through timeliness and
increased automationincreased automation
Internet Commerce for Internet Commerce for Manufacturing (ICM)Manufacturing (ICM)
(ITL,EEEL, MEL, MEP)(ITL,EEEL, MEL, MEP)
Project team:Project team:Tom RhodesTom RhodesLiz FongLiz FongKathy HarvillKathy Harvill
FY00 ProgressFY00 Progress Assumed project leadership (at request of EEEL) Defined standards-based framework for ICM testbed
(reference implementation) Conducted workshops and presented papers/demos
– NAMT, SIMA, and ATP reviews – Industry conferences: IPC, NEMI, ITEM2000
Collaborate with NEMI, IPC, Georgia Tech, UMBC Evaluating industry specifications (e.g., RosettaNet,
OAG, C-Net, eCo FW) for B2B interactions Evaluating applicability of agent and web
technologies
FY01 PlansFY01 Plans
Continue industry collaborations Demonstrate distributed ICM testbed
operations (with GaTech, NEMI, IPC partners)
Test and validate ICM information model in ICM testbed
Continue evaluation of applicable standards and technologies and their incorporation in ICM testbed
FY00 ResourcesFY00 Resources 1.5 FTE $350K (STRS-EEEL, ATP)
FY01 Resources 2 FTE (other FTE from EEEL, MEL,
MEP) Travel required to collaborate,
participate in standards meetings, and present work
ICMICM
CustomersCustomers– Consumers: PC assembly supply chain participantsConsumers: PC assembly supply chain participants– Consortia: NEMI, IPC, Rosetta NetConsortia: NEMI, IPC, Rosetta Net
ImpactImpact– Reduce the cost of circuit-board manufacturing by Reduce the cost of circuit-board manufacturing by
enabling the rapid exchange of build information and enabling the rapid exchange of build information and change notices for all supply-chain participants change notices for all supply-chain participants
– Demonstrates the use of XML in a vertical industryDemonstrates the use of XML in a vertical industry
StrategyStrategy
Conformance TestsConformance Tests Reference ImplementationsReference Implementations
• Standards developed jointly Standards developed jointly with industrywith industry
Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data
Forward-Looking StandardsForward-Looking Standards
Role-Based Access Control (RBAC)Role-Based Access Control (RBAC)
Virtual Reality Modeling Language (VRML)/X3DVirtual Reality Modeling Language (VRML)/X3D
XML and companion standards (DOM, CSS, etc)XML and companion standards (DOM, CSS, etc)
JavaJava
ADLADL
IMSIMS
DASE/SMPTEDASE/SMPTE
OASIS Registry/RepositoryOASIS Registry/Repository
GISGIS
STANDARDS COMMITTEES AND RELATED ACTIVITIES
NATIONAL•NCITS (Previously ANSI accredited X3) Management Level JTC1-TAG Technical Committees H2 (Database) H3 (Computer Graphics) L1 (Geographic Information Systems) L8 (Information Representation) J22 (Java) R1 (Real Time Systems)
IEEE Standards Board 1484
INTERNATIONAL•ISO SC32 (NCITS-L8, NCITS-H2) SC22 (NCITS-J22) SC24 (NCITS-H3) TC211 (NCITS-L1) SC 36 (IEEE 1484)
•ECMA TC39 (JavaScript)
OTHER•World Wide Web Consortium XML, DOM, XSLT, CSS•IMS Global Learning, Inc. •Object Mgmt. Group•OASIS•Web3D Consortium•ebXML (UN/CEFACT)
FEDERAL GOVERNMENT
CIO Council Interoperability Committee Standards Working Group
Real-Time JavaReal-Time Java
Project team:Project team:
Lisa CarnahanLisa CarnahanAlden DimaAlden Dima
Assistance asked by industry providers and government usersAssistance asked by industry providers and government users
Collaborate with Sun, HP, IBM, New Monics, FTLCollaborate with Sun, HP, IBM, New Monics, FTL
Hosted continuing workshops for real-time Java requirementsHosted continuing workshops for real-time Java requirements
Addresses real-time issues in both Java embedded and Java server Addresses real-time issues in both Java embedded and Java server application environmentsapplication environments
NIST coalesced industry to produce one set of requirements leading NIST coalesced industry to produce one set of requirements leading to one specto one spec
Open consensus forum for requirementsOpen consensus forum for requirements
BOTH camps (Sun and HP, Microsoft, etc.) have used NIST-led BOTH camps (Sun and HP, Microsoft, etc.) have used NIST-led requirements document to produce specsrequirements document to produce specs
Real-Time JavaReal-Time Java
EE TimesEE Times
“ “Interestingly, both the splinter group Interestingly, both the splinter group and Sun are using as a serious basis for and Sun are using as a serious basis for their efforts a ‘requirements document’ their efforts a ‘requirements document’ for real-time (Java) extensions prepared for real-time (Java) extensions prepared by a separate group working under the by a separate group working under the auspices of the National Institute of auspices of the National Institute of Standards and Technology (NIST). That Standards and Technology (NIST). That is raising hopes that both standards is raising hopes that both standards might ultimately come together.”might ultimately come together.”
Real-time Java Requirements Group
•“Requirements for Real-time Extensions to the Java Platform” (final release -September, 1999)
“The National Institute of Standards and Technology (NIST) has organized a real-time Java requirements group. This is an open group that is defining requirements for real-time Java. Their requirements document will be a valuable input to the experts group,…”
JSR-000001 Real-time Specification for Java (Call for forming the RT Java Experts Group sponsored by Sun Microsystems; NIST member)
Sun Microsystems’ sponsored RT Java Expert Group
“The J Consortium uses real-time requirements developed by the Requirements Group for Real-Time Extensions for the Java Platform, an effort sponsored by the
National Institute of Standards and Technologies (NIST)….”
CUPERTINO, Calif., September 24, 1999[Press Release -J Consortium™ Unveils First Public Real-Time Specification For The Java Platform]
The J Consortium
•NIST’s Requirements Group effort dissolved after successful completion of its goal.
FY00 ProgressFY00 Progress
Real-Time Java Requirements document Real-Time Java Requirements document completedcompleted
Worked with both spec development Worked with both spec development groups to help coalesce industrygroups to help coalesce industry
Member of Sun RT Experts Group - Member of Sun RT Experts Group - contributed to specificationcontributed to specification
Worked with J Consortium to help Worked with J Consortium to help develop specificationdevelop specification
Project sunsetProject sunset
FY00 ResourcesFY00 Resources 1.25 FTE1.25 FTE $250K (STRS)$250K (STRS)
FY01 ResourcesFY01 Resources 0 (Project sunset)0 (Project sunset)
Real-Time JavaReal-Time Java
CustomersCustomers– Sun, IBM, HP, NewMonics and users of real-Sun, IBM, HP, NewMonics and users of real-
time Javatime Java
ImpactImpact– Coalesced industry to produce one set of Coalesced industry to produce one set of
requirementsrequirements– 2 spec efforts compatible2 spec efforts compatible– New book on Real-time Java mentions NISTNew book on Real-time Java mentions NIST– Presented to and endorsed by VCATPresented to and endorsed by VCAT
ADL ADL (Architecture Description (Architecture Description
Language)Language)
Project Team:Project Team:Chris DabrowskiChris DabrowskiJohn TebbuttJohn Tebbutt
7
ADLs specify (at a minimum)
Use well-defined semantics & syntax
•Lexical and/or graphical language
•Usually use a formal descriptive method to specify aspects (e.g., behavior or constraints)
Support automated analysis & verification of design
•Simulation of execution•abstract formal theory (e.g. posets, fsm)•basis for conformance testing to Ref. Arch.
•Use of ‘formal methods’ approaches to:
•determine internal consistency
•conformance to ‘architectural style rules’
Job AssignorInterface
SchedulerInterface
Plan SelectorInterface
•System structure
•Components & Components Interfaces
•Defined Connections
Job AssignorInterface
Underlying ModuleThat Implements
Behavior
Do_Task(?Job)
Schedule_Job(?Job)
Architecture Description Architecture Description LanguageLanguage
Focus on high-level structure of the Focus on high-level structure of the overall application overall application (i.e., systems architecture)(i.e., systems architecture)
Provide features for modeling software Provide features for modeling software systems architecturesystems architecture– E.g., Structure, components, interfaces, interconnectionsE.g., Structure, components, interfaces, interconnections
Support automated analysis and Support automated analysis and verification of designverification of design
Enable testing during design phase Enable testing during design phase – Saves $ over testing at code phaseSaves $ over testing at code phase
Architecture Description Architecture Description Language (ADL)Language (ADL)
Software Architecture: “..an abstract system specification consisting primarily of functional components described in terms of their behaviors and interfaces and component-component interconnections” [Hayes-Roth, 1994]
ADL: “A language that provides features for modeling a software system’s conceptual architecture” [Medvidovic 1996]
Analysis Design Implement-tation
Testing Maintenance
Focus
Support
Why ADLs?
• Complexity of s/w - formal verification
• Software re-use - strong architecture description
• Testing during design phase - saves $ over testing at code phase
Standards CollaborationsStandards Collaborations– Propose additional functionality to SAE (first industry standards effort)Propose additional functionality to SAE (first industry standards effort)– Uniquely positioned to bring academic and industry communities together.Uniquely positioned to bring academic and industry communities together.– Apply developing AADL standard (based on Honeywell’s MetaH) to Unmanned Apply developing AADL standard (based on Honeywell’s MetaH) to Unmanned
Ground Vehicle: build additional functionality into AADL standardGround Vehicle: build additional functionality into AADL standard– Explore ADL relationships with IEEE and Open Group effortsExplore ADL relationships with IEEE and Open Group efforts
NIST’s Unique Role
Applied Research: Contribute to ADL body-of-knowledgeApplied Research: Contribute to ADL body-of-knowledge– Research application of ‘formal approaches’ to ADLsResearch application of ‘formal approaches’ to ADLs– Enhance ADL features for verifying conformance of application systems architectures to reference architectures (e.g. RCS)Enhance ADL features for verifying conformance of application systems architectures to reference architectures (e.g. RCS)– Publish successes and lessons-learned for industry awarenessPublish successes and lessons-learned for industry awareness– Provide feedback to appropriate ADL tool developersProvide feedback to appropriate ADL tool developers
Standards Role: Neutral Expert
Applied Research: Contribute to ADL body-of-knowledge
FY00 ProgressFY00 Progress
Participated in development of SAE’s Avionics ADL (AADL) standard
Explore ADL standardization efforts in IEEE and Open Group
Established partnerships with DISA, US Army, ITL/Security Division
Identify domain areas for applying ADL to explore standardization and testing issues – E.g., government smartcard, unmanned ground vehicle,
pervasive computing
FY01 PlansFY01 Plans Provide support for the formalization of AADL to
ensure its testability Develop prototype of US Army robotics architecture
– collaboration with MEL– dependent on DISA funds
Develop prototype of Smartcard Architecture w. Div 893
Demonstrate and document the benefits and functionality of ADLs
Produce a comparative study of different ADLs to support further development and standardization of ADLs
FY00 ResourcesFY00 Resources
1 FTE $200K (STRS)
FY01 Resources 2 FTE Equipment and Software: 10K Travel required to standards meetings
and to participate in conferences and industry meetings
Architecture Description Architecture Description LanguageLanguage
CustomersCustomers– Consumers: Industries that rely on component-based Consumers: Industries that rely on component-based
systems (e.g., manufacturing, avionics, real-time)systems (e.g., manufacturing, avionics, real-time)– ADL tool vendorsADL tool vendors
ImpactImpact– Moved ADL from University research to industry Moved ADL from University research to industry
standardstandard– Provide uniform descriptions of the structure and Provide uniform descriptions of the structure and
function of software. function of software. – Increases software quality by reducing errors through Increases software quality by reducing errors through
testing in design phase.testing in design phase.– Increase the producibility, quality, and reusability of Increase the producibility, quality, and reusability of
component-based software systems. component-based software systems.
CIO Council Standards Working GroupCIO Council Standards Working Group
Technology Transfer Act/OMB Circular A-119 direct federal agencies to Technology Transfer Act/OMB Circular A-119 direct federal agencies to use voluntary consensus standards in lieu of government unique use voluntary consensus standards in lieu of government unique standardsstandards
NIST directed to coordinate federal technical standards activities with NIST directed to coordinate federal technical standards activities with private sector technical standards activitiesprivate sector technical standards activities
NIST/ITL initiated effort to create Standards Working Group within Federal NIST/ITL initiated effort to create Standards Working Group within Federal CIO CouncilCIO Council
Standards Working Group approved late last year by the CIO Council Standards Working Group approved late last year by the CIO Council Interoperability CommitteeInteroperability Committee
Standards Working Group to be focal point of discussion, definition, and Standards Working Group to be focal point of discussion, definition, and coordination of federal requirements for voluntary IT standards, and will coordination of federal requirements for voluntary IT standards, and will assess the results of voluntary IT standards in meeting requirementsassess the results of voluntary IT standards in meeting requirements
Standards Working Group chaired by a member of this Division, and will Standards Working Group chaired by a member of this Division, and will be supported by this Division in its efforts to provide a unified voice to the be supported by this Division in its efforts to provide a unified voice to the voluntary IT standards community for Federal IT standards technical voluntary IT standards community for Federal IT standards technical requirementsrequirements
FY00 ProgressFY00 Progress Prototype of web based support software for
collection, evaluation, and status reporting on Federal Government technical requirements for voluntary standards developed and demonstrated.
Electronic Documents conference held featuring speakers from private industry, Federal Government, and voluntary standards organizations.
Meetings of the Standards Working Group held on an as needed basis.
Standards requirements forwarded to appropriate voluntary standards committees as the requirements are approved.
FY01 PlansFY01 Plans Continue development of web based support
software for Standards Working Group. Make software available for possible use by
other Working Groups within the EIEIT structure. Meetings of the Standards Working Group held
on an as needed basis. Standards requirements forwarded to
appropriate voluntary standards committees as the requirements are approved.
Consider holding second Electronic Documents conference.
FY00 ResourcesFY00 Resources 1.5 FTE $325K (STRS, OA-CIO Council)
FY01 Resources 1.5 FTE – Combination of time from
Group Leader and others in Interoperability Group
CIO Council Standards CIO Council Standards Working GroupWorking Group
CustomersCustomers– Other Federal AgenciesOther Federal Agencies
ImpactImpact– Federal agencies input requirements Federal agencies input requirements
to standards groups and utilize to standards groups and utilize voluntary consensus standardsvoluntary consensus standards
– Eliminates duplicative standardsEliminates duplicative standards
StrategyStrategy
Conformance TestsConformance Tests Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry •Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data
Year 2000 Problems - Project Leader: Gary Fisher
Published bulletin and produced Web site• Raised awareness of issues• Proposed methods for managing conversion
• Defined types of tests needed for date/time testing
• Under evaluation by IEEE
Developed test assertions
Issued change notice to FIPS 4-1 on date formats• Recommend use of four-digit format• Discouraged use of two-digit format
Held international symposium June 9-10, 1997• Management and technical issues• Market development of tools and services• Case studies by companies and governments
Developed source code scanner
Participate in Interagency Year 2000 Task Force
FY00 ProgressFY00 Progress
Y2K web site completedY2K web site completed Embedded system study completedEmbedded system study completed
– Report sent to KoskinenReport sent to Koskinen– Problems highlighted and impacts Problems highlighted and impacts
reportedreported– Produced new emphasis on embedded Produced new emphasis on embedded
systemssystems Project sunset Project sunset
FY00 ResourcesFY00 Resources
0.25 FTE0.25 FTE $50K (STRS)$50K (STRS)
FY01 ResourcesFY01 Resources 0 (Project sunset)0 (Project sunset)
Year 2000Year 2000
CustomersCustomers– All businesses that use computersAll businesses that use computers– Consumers who use computersConsumers who use computers
ImpactImpact– Y2K rollover went without incidentY2K rollover went without incident
StrategyStrategy
Conformance TestsConformance Tests Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry Diagnostic TestsDiagnostic Tests
• Research to reduce costs of testingResearch to reduce costs of testing
Reference DataReference Data
Automatic Generation of Spec Based Tests Automatic Generation of Spec Based Tests Using Formal MethodsUsing Formal Methods
Project Team:Project Team:
Paul BlackPaul BlackBill Majurski Bill Majurski Jim LyleJim LyleMicheal KooMicheal Koo
Industry NeedIndustry Need•Software testing is enormous cost to industry:Software testing is enormous cost to industry: 30% - 50% of budget30% - 50% of budget
Development may be 1000’s of staff-yearsDevelopment may be 1000’s of staff-yearsTesting cost may be $ millionsTesting cost may be $ millions
• Current methods: mostly hand generated testsCurrent methods: mostly hand generated tests
InefficientInefficientNot rigorous enoughNot rigorous enough
•Software testing: key to reducing security Software testing: key to reducing security incidents:incidents:
““More than 75% of the incidents…are the direct result of …bugs.” More than 75% of the incidents…are the direct result of …bugs.” and “…more real testing - not 'beta testing' in the marketplace - and “…more real testing - not 'beta testing' in the marketplace - will yield programs that are less vulnerable to attack.” will yield programs that are less vulnerable to attack.”
(Business(Business Week, February 28: "Locking Out the Hackers")Week, February 28: "Locking Out the Hackers")
• Conventional Conventional software specificationssoftware specifications are in are in natural language and so lack precisionnatural language and so lack precision
AmbiguitiesAmbiguitiesInconsistenciesInconsistenciesErrorsErrors
• Conventional Conventional software testingsoftware testing problems: problems:Exhaustive testing not possibleExhaustive testing not possibleTest case selection subjectiveTest case selection subjective
Result = costly and time-consuming testingResult = costly and time-consuming testing
ProblemsProblems
Requirements
Specification
SourceCode
EnglishEnglishLanguageLanguage
ProgrammingProgrammingLanguageLanguage
EnglishEnglishLanguageLanguage
HumanProgrammers
SoftwareTests
review &review &inspectinspect
Current Test Development MethodologyCurrent Test Development Methodology
review &review &inspectinspect
Requirements
Specification
SourceCode
ProgrammingProgrammingLanguageLanguage
MathematicalMathematicalLogicLogic
TestGenerator
SoftwareTests
informalinformalproofproof
formalformalproofproof
Advanced Test Development MethodologyAdvanced Test Development Methodology
MathematicalMathematicalLogicLogic
• Method for automatically generating tests from Method for automatically generating tests from formal specificationsformal specifications
• Associated coverage metricAssociated coverage metric
• Publications and presentationsPublications and presentations
ICFEM (98), WSEST(99), HASE (99), DASC (99)ICFEM (98), WSEST(99), HASE (99), DASC (99) GMU, UMBC, College Park, Brisbane, NRLGMU, UMBC, College Park, Brisbane, NRL
• Feasibility collaborations leading to Feasibility collaborations leading to commercializationcommercialization
Argus, a secure computer operating systemArgus, a secure computer operating system Ford Motor Research Lab: powertrain control systemsFord Motor Research Lab: powertrain control systems Northrop Grumman: fighter radar computersNorthrop Grumman: fighter radar computers
AccomplishmentsAccomplishments
FY00 ProgressFY00 Progress Feasibility tests on models from Ford and Feasibility tests on models from Ford and
ArgusArgus Initial formalization of BLAST (Basic Local Initial formalization of BLAST (Basic Local
Alignment Search Tool: identifies similar DNA Alignment Search Tool: identifies similar DNA sequences) Algorithm completed in sequences) Algorithm completed in collaboration with CSTLcollaboration with CSTL
Hosted Workshop on Testing Software Using Hosted Workshop on Testing Software Using Formal MethodsFormal Methods
Hosted Tutorial on Formal MethodsHosted Tutorial on Formal Methods 3 papers published3 papers published 2 papers submitted and accepted2 papers submitted and accepted 4 papers submitted4 papers submitted 9 talks (6 invited)9 talks (6 invited)
FY01 PlansFY01 Plans
Commercialize ATG Method with Ford Commercialize ATG Method with Ford and software company (tbd)and software company (tbd)
Continue feasibility studies with ArgusContinue feasibility studies with Argus Complete BLAST formalizationComplete BLAST formalization
FY00 ResourcesFY00 Resources
2.5 FTE2.5 FTE $500K (STRS-Competence)$500K (STRS-Competence)
FY01 ResourcesFY01 Resources 2.5 FTE2.5 FTE Equipment: $5KEquipment: $5K Travel required for validation collaborations Travel required for validation collaborations
and presentationsand presentations
Automatic Generation of Automatic Generation of TestsTests
CustomersCustomers– Software developersSoftware developers– Software end users, e.g., web surfersSoftware end users, e.g., web surfers– Security administratorsSecurity administrators– Researchers in software testing methodologiesResearchers in software testing methodologies
ImpactImpact– Reduce the cost of software testing (50% of Reduce the cost of software testing (50% of development costs)development costs)
– Improve software qualityImprove software quality– Reduce security incidents (75% due to bugs)Reduce security incidents (75% due to bugs)– Ford and Argus interested in commercialization of ATG Ford and Argus interested in commercialization of ATG methodologymethodology– Strongly endorsed by VCATStrongly endorsed by VCAT
Software Testing by Statistical Software Testing by Statistical MethodsMethods
(Competence Project)(Competence Project)
Objective: develop new methods for software testing Objective: develop new methods for software testing based on stochastic processes and metrics, in order tobased on stochastic processes and metrics, in order to– Improve the quality of software developmentImprove the quality of software development
– Develop reliability estimates that software correctly adheres to Develop reliability estimates that software correctly adheres to its specification its specification
Joint project Joint project – Software Diagnostics and Conformance Testing Division (897)Software Diagnostics and Conformance Testing Division (897)
– Statistical Engineering division (898)Statistical Engineering division (898) Collaboration with other NIST Labs (MEL, EEEL, CSTL)Collaboration with other NIST Labs (MEL, EEEL, CSTL)
Project Team:Project Team:
Len GallagherLen GallagherLynne RosenthalLynne Rosenthal
ProblemProblem
•Exhaustive testing not possibleExhaustive testing not possibleCan show presence of errors, not their absenceCan show presence of errors, not their absence
•Degree of correctness not quantifiedDegree of correctness not quantifiedCan derive subjective level of confidence, not Can derive subjective level of confidence, not
quantitative measurequantitative measure
•Test case coverage not scienceTest case coverage not scienceStrategy is intuitive, based on anecdotal Strategy is intuitive, based on anecdotal
evidenceevidence
Conformance Tests
??????conforminconformin
gg
Software Implementation
100% non conforming100% non conforming
FY00 ProgressFY00 Progress
Published paper on Coverage Model of Mutation Testing (with Paul Black)
Produced paper on applying statistical methods to component based testing
Produce paper on Test Input Needed to Improve Reliability
Analysis of effort to apply statistical method (partitioning) to new test development
FY01 PlansFY01 Plans
Continue effort to apply statistical method (partitioning) to new test development
Apply statistical method to pervasive computing applications
FY00 ResourcesFY00 Resources
1 FTE $200K (STRS-Competence, plus $100K
given to ITL/SED)
FY01 Resources 1 FTE 200K competence funding (plus 100K
given to ITL/SED)
Software Testing by Software Testing by Statistical MethodsStatistical Methods
Customers:Customers:– Conformance Test DevelopersConformance Test Developers
– OurselvesOurselves
– Users of conformance testsUsers of conformance tests
Impact:Impact:– Improve our knowledge and skills in designing Improve our knowledge and skills in designing
and developing quality software. and developing quality software.
– Quantitative measures associated with Quantitative measures associated with conformance testing. conformance testing.
Quantum InformationQuantum Information
Project Team:Project Team:
Paul BlackPaul BlackNeva CarlsonNeva Carlson
Quantum InformationQuantum Information
Goal: to develop information Goal: to develop information representations and computing machines representations and computing machines that take advantage of quantum effectsthat take advantage of quantum effects
Joint effort of physicists and computer Joint effort of physicists and computer scientistsscientists
FY00 ProgressFY00 Progress
Invited to develop joint proposal for Invited to develop joint proposal for Nanotechnology Initiative with PhysicsNanotechnology Initiative with Physics
Invited by NIST Physicists to present Invited by NIST Physicists to present Basic Concepts in Theory of ComputationBasic Concepts in Theory of Computation
Developed Quantum Information DemoDeveloped Quantum Information Demo Developed Quantum Information Developed Quantum Information
BibliographyBibliography Attended weekly Physics Quantum Attended weekly Physics Quantum
Information seminarInformation seminar
FY01 PlansFY01 Plans
Develop products for Nanotechnology Develop products for Nanotechnology Quantum Information projectQuantum Information project
Organize Workshop on Computer Organize Workshop on Computer Science in Quantum InformationScience in Quantum Information
Research questions in testing Quantum Research questions in testing Quantum AlgorithmsAlgorithms
FY00 ResourcesFY00 Resources .5 FTE.5 FTE $100K (STRS – Physics $100K (STRS – Physics
Lab?)Lab?)
FY01 ResourcesFY01 Resources 1.5 FTE (depends on 1.5 FTE (depends on
Nanotechnology Initiative Nanotechnology Initiative or $ from Physics)or $ from Physics)
Equipment: $0Equipment: $0 Travel anticipatedTravel anticipated
Quantum InformationQuantum Information CustomersCustomers
– Researchers in computational theory, and Researchers in computational theory, and algorithm design and algorithm design and analysisanalysis
– Developers of large computer models, e.g., Developers of large computer models, e.g., global weatherglobal weather– CryptanalistsCryptanalists
ImpactImpact– Computing machines of unprecedented Computing machines of unprecedented capability and small sizecapability and small size
– Solutions to intractable computational Solutions to intractable computational problems, e.g., problems, e.g., factoring, searchingfactoring, searching
– ITL staff invited to present at Physics seminars and ITL staff invited to present at Physics seminars and participate in participate in Physics Lab effortsPhysics Lab efforts
StrategyStrategy
Conformance TestsConformance Tests Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing
• Reference DataReference Data
Error, Fault, and Failure (EFF)Error, Fault, and Failure (EFF)Data Collection and Analysis Data Collection and Analysis
Project Team:Project Team:
Dolores WallaceDolores WallaceMichael KooMichael KooTony CincottaTony CincottaJason RagerJason Rager
Error, Fault, and Failure (EFF)Error, Fault, and Failure (EFF)Data Collection and Analysis Data Collection and Analysis
• Repositories of reference data on Repositories of reference data on software failuressoftware failures
• Tools to create and manage repositoriesTools to create and manage repositories
• Tools to analyze software failure dataTools to analyze software failure data
• Research in taxonomies to characterize Research in taxonomies to characterize software failuressoftware failures
FY00 ProgressFY00 Progress Repository systems:Repository systems:
– – Completed public repository with predefined Completed public repository with predefined queries queries
– – Completed internal repository with dynamic Completed internal repository with dynamic queries queries
Data: 1000 data points addedData: 1000 data points added
Paper on failures in medical devices Paper on failures in medical devices finalizedfinalized
Papers: report on NASA data submitted Papers: report on NASA data submitted to to NASANASA
Project was sunsetProject was sunset
FY00 ResourcesFY00 Resources 1.5 FTE1.5 FTE $300K (STRS, ATP - $160K)$300K (STRS, ATP - $160K)
FY01 ResourcesFY01 Resources 0 (Project sunset)0 (Project sunset)
Error, Fault, and FailuresError, Fault, and Failures CustomersCustomers
– Software DevelopersSoftware Developers– Researchers in Software Development MethodologiesResearchers in Software Development Methodologies
ImpactImpact– Identifies the stages in the software life cycle whose practices Identifies the stages in the software life cycle whose practices
can be improvedcan be improved– Endorsed by the Software Division of the Endorsed by the Software Division of the
American Society for QualityAmerican Society for Quality– Project methodologies and tools used in Project methodologies and tools used in
graduate programs at Johns Hopkins and the graduate programs at Johns Hopkins and the University of LouisvilleUniversity of Louisville
– EFF Tool used by RST CorporationEFF Tool used by RST Corporation
Other Agency ProjectsOther Agency Projects NIJ - Software and Tool VerificationNIJ - Software and Tool Verification
– Project Team: Gary Fisher, Len Gebase, Jim Lyle, Project Team: Gary Fisher, Len Gebase, Jim Lyle, Jackie Schneider, Mark ZimmermanJackie Schneider, Mark Zimmerman
VA - Health Care Information SystemsVA - Health Care Information Systems– Project Leader: Bill MajurskiProject Leader: Bill Majurski
GSA - EC Standards RoadmapGSA - EC Standards Roadmap– Project Leader: Mary BradyProject Leader: Mary Brady
EPA - Metadata Descriptions and EPA - Metadata Descriptions and RegistriesRegistries– Project Leader: Judy NewtonProject Leader: Judy Newton
NIJNIJNational Software Reference Library & Beta National Software Reference Library & Beta
TestingTesting
National Software Reference LibraryNational Software Reference Library GOAL: Reference Data Set of file signatures (hash) of GOAL: Reference Data Set of file signatures (hash) of
COTS fileCOTS file Used during examination of digital evidence to identify Used during examination of digital evidence to identify
pertinent files and eliminate others pertinent files and eliminate others BOTTOM LINE: One computer may have 5,000-20,000 BOTTOM LINE: One computer may have 5,000-20,000
files. NSRL data set can eliminate 40-95% of those files files. NSRL data set can eliminate 40-95% of those files and save hundreds of staff-hours.and save hundreds of staff-hours.
Beta Testing ProjectBeta Testing Project GOAL: Verification of computer forensic tools and their GOAL: Verification of computer forensic tools and their
use in analyzing digital evidenceuse in analyzing digital evidence Used to ensure tool usage yields objective, repeatable, Used to ensure tool usage yields objective, repeatable,
reproducible, results that hold up in court.reproducible, results that hold up in court.
FY00 ProgressFY00 Progress
Developed NSRL data dictionary and database structure for hash codes
Established contacts with Microsoft, Adobe, et al to supply software– CBD announcement, Press release
Identified hash algorithms Creating Reference Data Set (i.e, file signatures)
Developed draft test plan for beta testing Develop procedures and tests for imaging tools
FY01 PlansFY01 Plans Establish the reference data set and
distribution mechanism – (SQL database with web interface)
Develop test metrics for efficacy of hash codes used in the reference data set
Test the NSRL web site and make it available to approved users
Continue to develop tests and procedures for forensic tools
Issue test reports and test suites (publish on web site)
FY00 ResourcesFY00 Resources
3.5 FTE3.5 FTE $700K (OA-NIJ)$700K (OA-NIJ)
FY01 ResourcesFY01 Resources 3.5 FTE3.5 FTE Travel required for training courses Travel required for training courses
and presentationsand presentations
NIJNIJ
Customers:Customers:– Law Enforcement Agencies (e.g., FBI, Treasury, Law Enforcement Agencies (e.g., FBI, Treasury,
DCFL, state and local agencies)DCFL, state and local agencies)
– Software companies (e.g., Adobe, Microsoft)Software companies (e.g., Adobe, Microsoft)
Impact:Impact:– Improve the efficiency of forensic examination Improve the efficiency of forensic examination
of suspect computer files/date (NSRL data set of suspect computer files/date (NSRL data set can eliminate 40-95% of suspect files and save can eliminate 40-95% of suspect files and save hundreds of staff-hours).hundreds of staff-hours).
– Provide testing procedures for forensic Provide testing procedures for forensic software tools to ensure the analysis process software tools to ensure the analysis process and results holds up in courtand results holds up in court
Healthcare Information SystemsHealthcare Information Systems
Project Team:Project Team:
Bill Majurski Bill Majurski Tony CincottaTony CincottaRoger GimaRoger GimaJohn BarkleyJohn Barkley
GSA - EC Standards GSA - EC Standards RoadmapRoadmap
EPA - Metadata RegistriesEPA - Metadata Registries
Industry TestimonialsIndustry Testimonials ““The OASIS-NIST XML Conformance Test Suite is critical for our The OASIS-NIST XML Conformance Test Suite is critical for our
industry.”industry.”– Norbert Mikula, CTO, OASISNorbert Mikula, CTO, OASIS
““NIST made strong contributions (X3D) and resolved our knottiest NIST made strong contributions (X3D) and resolved our knottiest problems.”problems.”
– Don Brutzman, Board of Dir., Web3D ConsortiumDon Brutzman, Board of Dir., Web3D Consortium ““Your expertise helped advance conformance and spec Your expertise helped advance conformance and spec
development.”development.”– Steve Griffin, IMS Technical DirectorSteve Griffin, IMS Technical Director
““The high quality test suite and certification program is an The high quality test suite and certification program is an invaluable resource for ATA.”invaluable resource for ATA.”
– Robert Peel, Director of Airworthiness and StandardsRobert Peel, Director of Airworthiness and Standards ““NIST’s participation on DASE is vital for the U.S. industry.”NIST’s participation on DASE is vital for the U.S. industry.”
– Lisa McKnight, Multimedia Business DevelopmentLisa McKnight, Multimedia Business Development
NIST Requested Division NIST Requested Division PresentationsPresentations
eCommerce - XML and DASEeCommerce - XML and DASE– Presented to Undersecretary ShaversPresented to Undersecretary Shavers– Presented to NIST Director and DeputyPresented to NIST Director and Deputy
VRML, Real-Time Java, Auto GenerationVRML, Real-Time Java, Auto Generation– Presented to VCATPresented to VCAT
XML/DOM Conformance TestingXML/DOM Conformance Testing– Presented at ITL Town MeetingPresented at ITL Town Meeting
VRML Conformance TestingVRML Conformance Testing– Presented at ITL Town MetingPresented at ITL Town Meting
VmViewVmView– Presented to Congressional staffPresented to Congressional staff
ICM Project BriefingICM Project Briefing– Presented at ATP National MeetingPresented at ATP National Meeting
NIST TestimonialsNIST Testimonials ““Software testing work . . . is highly appropriate serving a Software testing work . . . is highly appropriate serving a
broad set of companies who would not, on their own make broad set of companies who would not, on their own make these investments.”these investments.”
– VCAT ReportVCAT Report ““Way to go folks. You did us proud.”Way to go folks. You did us proud.”
– Brian Belanger, Exec. Director, VCATBrian Belanger, Exec. Director, VCAT ““The model used in the software testing program - to partner The model used in the software testing program - to partner
with industry, rather than developing tests and transferring with industry, rather than developing tests and transferring them is excellent and will become a model for other them is excellent and will become a model for other programs at NIST.”programs at NIST.”
– Ray Kammer’s feedback of the ITL Program ReviewRay Kammer’s feedback of the ITL Program Review ““A wonderful testimonial to your work.”A wonderful testimonial to your work.”
– Ray Kammer’s response to the ATA letter.Ray Kammer’s response to the ATA letter.
PrioritiesPrioritiesCore ProgramCore Program
1.1. XML Conformance testingXML Conformance testing
2.2. Interactive TVInteractive TV
3.3. Automatic Generation of TestsAutomatic Generation of Tests
4.4. Pervasive ComputingPervasive Computing
5.5. XML Registry/RepositoryXML Registry/Repository
----------------------------------------------------------------------------------------------------------------------
6.6. Computer Forensics – NIJComputer Forensics – NIJ
7.7. CIO Council Standards Working GroupCIO Council Standards Working Group
Priorities – Next LevelPriorities – Next Level
1.1. ADLADL
2.2. Health Care Information Systems – VAHealth Care Information Systems – VA
3.3. CGMCGM
4.4. Learning TechnologiesLearning Technologies
5.5. Internet Commerce for ManufacturingInternet Commerce for Manufacturing
6.6. Software Testing by Statistical MethodsSoftware Testing by Statistical Methods
7.7. Quantum InformationQuantum Information
8.8. Other OA Projects (GSA, EPA)Other OA Projects (GSA, EPA)
Division IssuesDivision Issues We’re a victim of our success - we’re in We’re a victim of our success - we’re in
constant demandconstant demand We need people!!!!!We need people!!!!! We need more discretionary fundingWe need more discretionary funding
– STRS vs. OA/ATPSTRS vs. OA/ATP Travel demands are increasingTravel demands are increasing Demonstrations - good and badDemonstrations - good and bad Membership in consortia is essentialMembership in consortia is essential Collaborations with other DivisionsCollaborations with other Divisions
ROSENTHAL ROSEN
SKALL
BARKLEY
We also consult with distinguished scientistsWe also consult with distinguished scientists
Remember, Al and the SDCT Division jointly invented the Internet.Remember, Al and the SDCT Division jointly invented the Internet.