Embed Size (px)
Citation preview
AGRESEARCH SHOW AND TELL
Greg Peyroux, Information Solutions [email protected]
Alan McCulloch, Software [email protected]
Information Services Software Development And Specialists Team
Development, Web and Data Services
OVERVIEW
Demonstrations of applications, services and data of interest, discussion, expressions of interest
• Group Structure• Team
• Who and What We Do• New Direction and Business Focus
• Development and Management Tools• Open source and cross platform tools and
frameworks• Examples
INFORMATION SERVICES GROUP
Information Technology and Knowledge Services
• Knowledge• Library • Records management a• Knowledge advisory
• IT• Technical• User Support• Development, Web, Data and Bioinformatics
Recent History of AgResearch:•Series of Reviews
•‘Fit for Purpose’ Initiatives
•Significant Team Turnover
Opportunity to innovate and improve
THE TEAM – REBUILT AND REPOSITIONED
How do you know that you add value?
•Restructured team: Aligned to Outcomes
•Emphasis on “All AgResearch” + the Bigger Picture
•Focus on strategic goals
•Greater Management and Business Awareness
•Transparency: Work request portal on intranet
THE TEAM – REBUILD AND REPOSITION
Development Services•Priority on solutions for science and farms
•Request Analysis
– Determine optimum solution
»Alternative approach e.g. open source
»Reuse/repurpose existing system
»Purchase shrink wrapped S/W or consultancy
»Develop
•Close-off, Feedback and Executive Reporting
A NEW DIRECTION FOR DEVELOPMENT
THE TEAM – REBUILDING
Is B “Best”?
Desperately Seeking:•Fresh Ideas and Concepts
• Interns•Effective Communication Skills• Increased Science Capability:
• PHD, MScs, BScs• Statistical, Research + Technician Experience
•Critical Mass
THE EXTENDED TEAMSoftware Development And Specialists Team (Information Solutions)
Development, Web and Data Services
• Developers– Web, SharePoint– Windows, Mobile Devices– Lab (LIMS) and Farm Systems
• Bioinformatics
• Specialists– GIS– ERP analyst– Reporting Specialist– Database administrator– Records
TEAM IMPACT• Broad Span
– Internal – All AgResearch
– Science
– Support
– Farms
– External
– Farms
– Meat processors
– Commercial Labs
– Accountants and consultancy firms
Group and Team Structures Create Consistent , Aligned Service
Is B “Best”?
Microsoft Stack
• NET framework• Migrating to TFS 2010• SQL Server• BizTalk• SharePoint 2010 (CMS, team, documents, My Sites etc)
Keen to collaborate on:• Public Records Act• Research Data Management• Mapping Systems
Open Source and Cross Plaform Tools (Alan)
DEVELOPMENT TOOLS
Is B “Best”?
Organisation-Wide Data Integration
• Support
• Farm-Systems
• Science data
• Location
• Field Data recording
KEY PROJECTS
INTEGRATION
• Resolution Farm Management and Planning
Integration of Information Systems
INTEGRATION - LOCATIONZ
• Resolution Farm Management and Planning
INTEGRATION - LOCATION
• Resolution Farm Management and Planning
Global Location Integration and Management
• Spatial Location• Silver light application – draw, annotate, search and view• Import
• GPS points• SHP• KML etc
• Known• Desktop Application
• Primarily for sample tracking
• Global Location ID• Web service to guarantee retrieval
OVERVIEW OF FARMING/RESEARCH DATA
Farm and Related Research Data Components
EnvironmentalEnvironmental AnimalAnimal
Animal locations and movements
Animal locations and movements
Milk yield and vat
Herd
Tests and results etc
Primary data setsPrimary data sets
Location/GIS-relatedPaddock/Transect
Points, Lines Polygons
Integrate into data warehouse, mine for traitse.g. Best calvers in the wet
SMART DATA RECORDING AND STORAGE
Dynamic task creation and recording
• Mobile device– User creates a custom task on their PC
– Records task with a range of checks and rich validation
– User uploads task data into a database
Excel Import and Metadata annotating
OVERVIEW OF FARMING/RESEARCH DATATask Recorder – Integrated Task Creation Framework
TOOLS FOR MANAGING PEOPLE, PRODUCTS, PROJECTS AND CHANGE
Ugh! Look at his new recording system, that won’t last!
• Porter’s 5 Forces• PEST• SWOT• Fatal Flaw• Stakeholder mapping• Tipping Point
• Use of levers• Incubate advocates• Plan for resistors
Value of Tools
MANAGEMENT TOOLS AND FRAMEWORKS
MANAGEMENT TOOLS
Kotter, J. (2007). Leading change: Why transformational efforts fail? Harvard Business Review , 85(1), 96-103.
Managing Change: Kotter’s 8 Steps to Transformation
OPEN SOURCE AND CROSS-PLATFORM ENVIRONMENTS AND FRAMEWORKS
Allan McCulloch
WINDOWS AND UNIX HPC RESOURCES
CROSS-PLATFORM SOFTWARE DEVELOPMENT - STATUS
EXTENDED DATA MODELLING
DESIGN PATTERNS FOR DATA MODELLING
COLLABORATION SUPPORT - WIKIS AND DRUPAL
WINDOWS AND UNIX HPC RESOURCES
We use the Condor scheduler - creates and manages both Windows and Unix compute pools. Mature, scalable, cross-platform
• Windows = desktop machines , jobs run in background. Users approached and volunteer machines to the pool (+ “captive” machines in training rooms and IS and departmental staff !) as and when required . Max size so far, simulation run on about 40 machines. Saved $$$ on leased high performance windows boxes.
• Unix = Centos Redhat / Centos based , HP blades (mix of 32 and 64 bit , 2 and 4 core, 2Gb RAM ). B-word data pre-processing mainly
“LARGE MEMORY” SMP UNIX MACHINES
• 1 x Redhat Linux database server (postgres , mysql, Oracle)
• Data warehousing • B-word• GIS data (postgres)
• 1 x Redhat Linux compute server
• Data Mining • Stats (R , ASREML)• B-word miscellany• Virtual machine provisioning (early days)
CROSS-PLATFORM (LINUX-WINDOWS) IN-HOUSE SOFTWARE DEVELOPMENT
• Java based middleware supporting various B-word web interfaces (not being further developed)
• Small scale python, perl and R scripting for specific problems or pipelines (/ bucket brigades). Unusual for us to develop anything significant from scratch – huge communities, many packages and solutions already out there.
• Python based Object-Relational system supporting mainly B-word databases
• Mathematical modellers developing C++ based models
EXTENDED DATA MODELLING
What is it ?
• ER : Entity-Relationship model : Chen (1976), formalized in Ng (1981). Standard Terminology.
• Extended Entity Relationship (EER) : little standard terminology , many contributors
• EER only one of a number of semantic models• Developed to meet complex requirements beyond data-
processing in business and industry – CAD/CAM db, image and graphics db, cartographic and geological db.
• Ref ElMasri, R.A. , Navathe , S.B. : Fundamentals of Database Systems
BRDF FRAMEWORK – SOME EER ASPECTS
• Inheritance used to provide a core data provenance and record identification and classification layer for all tables• Automatic date stamps and user ids for updates on any table• Keywords field for search engines• Record status for versioning• Class typeid for object-relational mapping support• Universal unique record identifier across all tables – really useful !
• Heterogenous lists can include member records from many tables – e.g. search engine results; projects manifests (subjects, samples, lab resources etc)
• Normalised free text comments attached to any record• Provenance of all data imports - records are linked to the data source
they came from .• Entity types and relationship types treated uniformly
BRDF FRAMEWORK
• Postgres based (….but portable to dbs like Oracle, SQLServer)• Easy to install - single setup script• Around 8 instances mostly b-word related but also hosting
genetics and epidemiological data• Python based object-relational mapping and web interface • About to begin work on a C# /.net based OR mapping and new
front-end• (Custom object relational mappings are cost-effective and work
well for data processing , but objects do not usually plug into standard GUI componentry , and so therefore are not cost effective for UI development )
• (and vice-versa !!! )
DESIGN PATTERNS FOR DATA MODELLING
• “Design Patterns” (Gamma, Helm etc al.) – reusable pieces of design in Object Oriented software engineering (e.g. observer, visitor, factory method, façade, flyweight,….)
• We need Design Patterns for databases to control complexity and cost
• Design patterns easier to define for EER based systems
IMPORT (IN) AND DISPLAY (OUT) DESIGN PATTERNS
STUDY-OBSERVATION DESIGN PATTERN IN BRDF FRAMEWORK
STUDY-OBSERVATION DESIGN PATTERN SUGGESTED DISCRETIZING CONTINUOUS MS OBSERVATIONS INTO
BINS
DRUPAL BASED COLLABORATION SUPPORT : MULTI-TENANTED DESIGN
APPLICATIONS
Some examples of applications
FARM APPLICATIONS
• Resolution Farm Management and Planning
• ETS Tools - Forestry
• Resolution Farm Management and Planning
FARM APPLICATIONS - ETS
• MAF lookup tables and rules
• Google Map interface
• Extended GPS support
SUMMARY
• Open source and cross platform• Windows and Unix HPC resources• Cross-platform Software development – status• Extended Data Modelling • Design patterns for data modelling• Collaboration support - wikis and Drupal
• Development and Management Tools• Data integration• Change management
• About Us• Team • Who and What We Do• Direction and Business Focus
AGRESEARCH SHOW AND TELL
Greg Peyroux, Information Solutions [email protected]
Alan McCulloch, Software [email protected]
Thank you for your attention
Questions
Please feel free to contact us
