View
215
Download
0
Category
Tags:
Preview:
Citation preview
1
A Data Management System (DMS) for A Data Management System (DMS) for Air Quality Management AgenciesAir Quality Management Agencies
Prepared by:Prepared by:Mark StoeltingMark Stoelting
Bay Area Air Quality Management DistrictBay Area Air Quality Management DistrictSan Francisco, CASan Francisco, CA
andandEric GrayEric Gray
Sonoma Technology, Inc.Sonoma Technology, Inc.Petaluma, CAPetaluma, CA
Presented at:Presented at:U.S. EPA 2008 National Air Quality ConferenceU.S. EPA 2008 National Air Quality Conference
Portland, ORPortland, ORApril 6-9, 2008April 6-9, 2008
2
DMS: Why, What, How, WhoDMS: Why, What, How, Who
• Why build a DMS?Why build a DMS?
• What is it?What is it?
• How does DMS work?How does DMS work?
• What are the DMS capabilities?What are the DMS capabilities?
• Who is involved?Who is involved?
• Who paid for DMS?Who paid for DMS?
• Where is it going?Where is it going?
3
Why the DMS?Why the DMS?
• New technology: instruments, New technology: instruments, communications, data accesscommunications, data access
• Real-time data accessReal-time data access
• Improve data quality (QC checks)Improve data quality (QC checks)
• Graphical data reviewing toolsGraphical data reviewing tools
• Minimize/eliminate need for paperMinimize/eliminate need for paper
• Track instruments and performanceTrack instruments and performance
• Shared need by all monitoring agencies Shared need by all monitoring agencies
4
What is DMS?What is DMS?
• Data Management SystemData Management System• Database to ingest, process, export dataDatabase to ingest, process, export data• ArchiveArchive
• Programmable database with Programmable database with optionaloptional featuresfeatures• Process sub-hourly data (1 minute)Process sub-hourly data (1 minute)• Collect instrument metadataCollect instrument metadata• Track instruments and performance (OpCodes)Track instruments and performance (OpCodes)• Automatic Quality ChecksAutomatic Quality Checks• Export (AirNow, AQS)Export (AirNow, AQS)• Notification (email, process controls)Notification (email, process controls)
5
Simple Data StructureSimple Data Structure
DATA
SITE PARAMETER
INSTRUMENT
{OpCode}
Site,Parameter,Date_Time,OpCode,Data,[alpha_data]Site,Parameter,Date_Time,OpCode,Data,[alpha_data]
70701111,CO_chamt,10/05/2007 08:16,0,48.1580,,CO_chamt,10/05/2007 08:16,0,48.1580,
(optional, assigned in database)
DATE_TIME
6
Sites, Parameters, InstrumentsSites, Parameters, InstrumentsEstablish MonitoringSites
Create Parameters
CreateInstruments
7
Assign Parameters to Instruments Assign Parameters to Instruments and Instruments to Sitesand Instruments to Sites
Assign Parametersto Instruments
Assign Instrumentsto Sites
8
Automated Quality Checks/Export
RawData
AmbientAir Quality
DataRangechecks
Instrumentchecks
Site checks
Network (off-site)checks
AIRNow
DistrictWeb Page Exports filtered by QC code
9
Automatic Quality ChecksAutomatic Quality ChecksAssign AutomaticQuality ControlChecks toParameters
10
Data Input ProcessData Input Process• Ingests data files generated by datalogger or Ingests data files generated by datalogger or
data formatting softwaredata formatting software• Duplicate input data is automatically rejectedDuplicate input data is automatically rejected• Automated Quality Checks establish initial Automated Quality Checks establish initial
QC codeQC code• Min/max, zero drift, sticking, rate-of-change, compareMin/max, zero drift, sticking, rate-of-change, compare
• QC conditions can activate process/alarmQC conditions can activate process/alarm• Electronic chain-of-custody createdElectronic chain-of-custody created• Sub-hourly data aggregated to 1-hr averagesSub-hourly data aggregated to 1-hr averages• Automated data export to AIRNow and Air Automated data export to AIRNow and Air
District web siteDistrict web site
11
DMS StatusDMS StatusStation Reporting Status
Event Log
QC Log
12
Manual Data ReviewManual Data Review
• Time-series graphical editing and QC Time-series graphical editing and QC tools for staff data reviewtools for staff data review
• DMS records all reviews and manual DMS records all reviews and manual adjustments to QC status and data adjustments to QC status and data values (electronic chain-of-custody)values (electronic chain-of-custody)
• Tracks manual data reviewer statusTracks manual data reviewer status
• Lock final data prior to exportLock final data prior to export
• Exports final data for regulatory Exports final data for regulatory submission to AQS or other agenciessubmission to AQS or other agencies
13
Time Series Edit/QC ScreenTime Series Edit/QC Screen
14
Scatter Plot AnalysisScatter Plot Analysis
15
Other DMS FeaturesOther DMS Features
• Data access security (user rights Data access security (user rights management)management)
• Maintains instrument equipment Maintains instrument equipment databasedatabase
• Supports electronic instrument and site Supports electronic instrument and site logslogs
• Daily instrument performance reports Daily instrument performance reports (using OpCodes)(using OpCodes)
16
DMS Software RequirementsDMS Software Requirements
• DMS host computer:DMS host computer:• Windows Server 2000/2003 or XP/VistaWindows Server 2000/2003 or XP/Vista• Microsoft SQL Server 2005 (all editions)Microsoft SQL Server 2005 (all editions)
• DMS Client Software:DMS Client Software:• .NET 2.0 Framework.NET 2.0 Framework• Windows 2000, XP, VistaWindows 2000, XP, Vista
• Future tools may require Java and ERSI Future tools may require Java and ERSI arcGISarcGIS
17
DMS HardwareDMS HardwareDMS Server:DMS Server:• Minimum server requirements – Pentium 4 Minimum server requirements – Pentium 4
laptoplaptop• Must support Microsoft WindowsMust support Microsoft Windows• Hardware determined by processing needsHardware determined by processing needs• Laptop processes hundreds of thousands of Laptop processes hundreds of thousands of
records per hour and store millionsrecords per hour and store millions• Server can process millions of records per Server can process millions of records per
hour and store billionshour and store billionsDMS Client:DMS Client:• Pentium III class PC or betterPentium III class PC or better• Broadband connection to DMS ServerBroadband connection to DMS Server
18
DMS Performance BenchmarksDMS Performance Benchmarks
• AIRNow DMC DMS will be running in August AIRNow DMC DMS will be running in August on a Quad Core Intel processor with 8 GB of on a Quad Core Intel processor with 8 GB of RAM. The operating system will be Windows RAM. The operating system will be Windows Server 2003 64-bit EditionServer 2003 64-bit Edition
•• Currently, AIRNow DMC processes 12 million Currently, AIRNow DMC processes 12 million records a day for over 5000 sitesrecords a day for over 5000 sites
•• The DMS system can process 22 million The DMS system can process 22 million records per hour or ½ billion a day (historical records per hour or ½ billion a day (historical data import)data import)
•• The Bay Area DMS system will process 1 The Bay Area DMS system will process 1 million data records per daymillion data records per day
19
DMS Storage BenchmarksDMS Storage Benchmarks
• The database will grow by 100 MB for The database will grow by 100 MB for every one million recordsevery one million records
• One billion records can be stored in 100 One billion records can be stored in 100 GB dataGB data
• DMS query performance has been DMS query performance has been tested with ½ billion data recordstested with ½ billion data records
• AIRNow DMC will test performance for AIRNow DMC will test performance for over a billion records by fallover a billion records by fall
20
DMS Can Be …DMS Can Be …
• An instrument monitoring systemAn instrument monitoring system
• A data analysis systemA data analysis system
• A data Quality Check and Review A data Quality Check and Review systemsystem
• A data export tool (to AirNow, AQS)A data export tool (to AirNow, AQS)
• A data warehouse for historical dataA data warehouse for historical data
21
Agency DeploymentAgency Deployment• DMS developed using AIRNow technology in DMS developed using AIRNow technology in
20042004• DMS to replace AIRNow technology by fall DMS to replace AIRNow technology by fall
20082008• DMS in use by BAAQMD, SCAQMD, and DMS in use by BAAQMD, SCAQMD, and
CARB by end of 2008CARB by end of 2008• AIRNow International will be based on DMS AIRNow International will be based on DMS
for 2009for 2009• Stable DMS distribution package available for Stable DMS distribution package available for
agency use by early 2009agency use by early 2009• Looking for 3 or 4 additional agency Looking for 3 or 4 additional agency
development partners in 2008development partners in 2008
22
Development and FundingDevelopment and Funding• DMS was development and funded by DMS was development and funded by
agencies with an interest in the resultagencies with an interest in the result• Contributors include:Contributors include:
• Bay Area AQMD (BAAQMD)Bay Area AQMD (BAAQMD)• EPA Region IX, 103 GrantEPA Region IX, 103 Grant• EPA National Environmental Information EPA National Environmental Information
Exchange Network (NEIEN) Challenge GrantExchange Network (NEIEN) Challenge Grant• South Coast AQMD (SCAQMD)South Coast AQMD (SCAQMD)• EPA (AirNow and AirNow International)EPA (AirNow and AirNow International)• California Air Resources Board (CARB) soonCalifornia Air Resources Board (CARB) soon
• New DMS features must be available to all New DMS features must be available to all agenciesagencies
• New features must not reduce DMS New features must not reduce DMS functionality to other agenciesfunctionality to other agencies
23
Future DMS DevelopmentFuture DMS Development
• Enhanced view/edit featuresEnhanced view/edit features• Viewing tools - fingerprint plotsViewing tools - fingerprint plots• Statistical tools - box plots, reportsStatistical tools - box plots, reports
• Support for toxics, PM filter data, Support for toxics, PM filter data, Special Purpose MonitoringSpecial Purpose Monitoring
• Expanded reporting capabilities Expanded reporting capabilities
• Instrument Control ChartsInstrument Control Charts
• GIS mapping capability, web services, GIS mapping capability, web services, and NEIEN XML data exchange nodeand NEIEN XML data exchange node
24
DMS SummaryDMS Summary
• Simple, scalable, cost-effectiveSimple, scalable, cost-effective
• Fast, secure, and efficient operationFast, secure, and efficient operation
• Comprehensive quality control toolsComprehensive quality control tools
• Applicable to a wide variety of dataApplicable to a wide variety of data
• Free to public agencies under a share-Free to public agencies under a share-ware use agreementware use agreement
25
Contact InformationContact Information
Mark StoeltingMark Stoelting
Bay Area Air Quality Management DistrictBay Area Air Quality Management District
mstoelting@baaqmd.govmstoelting@baaqmd.gov
(415) 749-4619(415) 749-4619
Eric Gray Eric Gray
Sonoma Technology, Inc.Sonoma Technology, Inc.
eric@sonomatech.comeric@sonomatech.com
(707) 665-9900(707) 665-9900
Recommended