2003/03/25天文学会春季年会 JVO の研究開発 (2) 全体進捗大石雅寿国立天文台＆総研大 [email protected]

2003/03/252003/03/25 天文学会春季年会天文学会春季年会

JVOJVO の研究開発の研究開発 (2)(2)全体進捗全体進捗

大石雅寿国立天文台　＆　総研大[email protected]


CollaboratorsCollaborators

NAOJ• Y. Mizumoto• N. Yasuda• Y. Shirasaki• M. Tanaka

Ochanomizu U.• Y. Masunaga

Fujitsu Ltd.• K. Monzen• K. Kawarai• Y. Ishihara• H. Yanaka• Y. Yamaguchi• M. Ishida• N. Saito


Supported bySupported by

• Research Grant from the MEXT

　（文部科学省　科研費

　　特定領域研究　　「情報学」　　松岡・下條組）

• Research Grant from the Sokendai

　　　（総合研究大学院大学　企画研究）


Data ProductivityData Productivity

• Nobeyama Radio ： ~1TB/yr

• Subaru@Hawaii ： ~20TB/yr

• ALMA （ planned ）　： ~PB/yr

Flood of excellent data (survey data)

Digitized & Archived


HPC + High Speed NETWORKHPC + High Speed NETWORK

+GRID Technology

→ it is possible to connect and seamlessly use remote DBs and computing facilities for data analyses

→ Virtual Observatory


JVO (will) consists ofJVO (will) consists of

• Seamless Access to Federated DBs

• Various Data Analysis tools

• Advanced Analysis such as Data Mining

• Federation with Real Telescopes

→ 　 crucial to control data quality in VO


JVO prototype : Ver. 1JVO prototype : Ver. 1

• Establish seamless access to federated DBs by using the Grid technology

• Define JVO Query Language, and implement its parser

• Test environment is in a closed subnet


JVOPortal

JVOPortal

Catalog Query ServiceCatalog Query Service

　　 Globus Toolkit

Catalog DB

ServiceRegistry

ServiceRegistry

ASTRONOMER

JVOBrowser

JVOBrowser

JVOViewer

JVOViewer

Data Archive ServiceData Archive Service

Data ManagementDB

Data

JVOContoroller

JVOContoroller

利用者独自サービス


Data Analysis ServicesData Analysis Services

UDDI Server（ FreeSoft)

WebBroser

Commands callable from

GT2

GlobusToolokit V2

Security ServiceSecurity Service Resource ManagementResource Management Data ManagementData Management

skycat

Query request

JVO Query Language

Parse the JVO QL into each query sequence

Resolve servers to request queries

GridFTP each query

2003/03/25 天文学会春季年会

JVO Query Languagecreate view myEROtable asselect s.Bmag,

s.Rmag,t.Hmag,t.Kmag,...,sr.BOX(POINT(s.ra,s.dec),w,h)

as Rimage,tk.BOX(POINT(s.ra,s.dec),w,h)

as Kimage,...

from SUBARU s,2MASS t,...,SUBARU.R sr,2MASS.K tk,...

where XMATCH(s,b,...) < 3 arcsecand(s.Rmag-t.Kmag) > 6 magandBOX(POINT(ra0,dec0), w0, h0)and...

Specify search area with the same syntax as cutout image specification.

Create view with the user specified name in JVO system.

Select attributes from each catalog server. Column names can be expressed in UCD.

Select the catalog server.

Select the image data server.

Cross-match distributed catalogs.

Query condition based on distributed catalog.

Select cutout images from each image data server.Image area can be specified by BOX or CIRCLE operand.

create view myEROtable asselect s.Bmag,

s.Rmag,t.Hmag,t.Kmag,...,sr.BOX(POINT(s.ra,s.dec),w,h)

as Rimage,tk.BOX(POINT(s.ra,s.dec),w,h)

as Kimage,...

from SUBARU s,2MASS t,...,SUBARU.R sr,2MASS.K tk,...

where XMATCH(s,b,...) < 3 arcsecand(s.Rmag-t.Kmag) > 6 magandBOX(POINT(ra0,dec0), w0, h0)and...

Specify search area with the same syntax as cutout image specification.

Create view with the user specified name in JVO system.

Select attributes from each catalog server. Column names can be expressed in UCD.

Select the catalog server.

Select the image data server.

Cross-match distributed catalogs.

Query condition based on distributed catalog.

Select cutout images from each image data server.Image area can be specified by BOX or CIRCLE operand.

詳細は，安田他の講演 (V27b) で


Java2SE　1.4

Apache　Tomcat

soapuddi

JDBC

UDDI Registory / Servlet Engine

Java2SE　1.4

JVO　Server

UDDI4J

JVO Controller

Globus　Toolkit　(Server)

JVO　Service(count)

SQLlibrary

Java2SE　1.4

Maintenance　Tool

UDDI4J

UDDI Maintenance Tool

Hypertextcontents

Solaris8 (Possible　at　Linux)

Redhat Linux

JVO Service

ORACLE

DB Service

Free　Software

Commercial　Software

In　House　Software

Function

ApacheＳＯＡＰ

① ServletDownLoad

Globus　Toolkit　(Client)

Redhat Linux

PostgreSQL　(DBMS)

DB Service

The　following　free　software　is　included　by　globus　toolkit.•　OpenSSL　(Secure　Socket　Library)•　OpenLDAP　(LDAP　Server)•　wu-ftpd　(ftp　server)

JVO Prototype System Architecture

③ Find　Service

Netscape　Communicator

Java2　Plugin

JVO　Client(servlet)

JVO Client

② RequestObservation

Monitoring

UDDIData

SUBARUSupCamz-band•　get　&　parse　GSDL

•　pre　condition　check•　execute　Service•　post　condition　check

•　register•　update•　delete

GSDL(copy)

JVO　Service(select)

SQLlibrary

JVO　Service(image)

JVO　Service(X　match)

ImageData

GSDLGridFTP

ImageData(copy)

⑥ spawn　Service

⑤ execute　Service

Check　status

SUBARUDataBase

JVOController

Globus　Toolkit

AVO

Globus　Toolkit

NVO

Internet

ImageData

SUBARUSupCami-band


JVO Service

SQLlibrary

SQLlibrary

JVO　Service(count)


Copy　imate　data　　　　　　by　GridFTP

Copy　imate　data　　　　　　by　GridFTP

JVO　Service　(X　match)

JVO　Service　(image)

③ get　GSDL

④ Get　GSDL　　　　by　GridFTP

GSDL

⑦ issue　SQL

④’ Get GSDL by GridFTP

⑦’ issue SQL

⑤’ execute Service

ImageData(copy)

SUBARUDataBase

Gb Ether

⑥’ spawn Service


JVO Service

SQLlibrary


JVO　Service　(X　match)

GSDL

Redhat Linux 2MassDataBase

PostgreSQL

DB Service

2Mass　DB

ImageData(copy)

ImageData


• JVO プロトタイプの GRID 技術については

　　石原他の講演 (V30a) を参照• データベース構築， user interface につい

ては　　白崎他の講演 (V28b) を参照• 検索サービスの構築については　　田中他の講演 (V29b) を参照


How it works ?How it works ?

• Read JVO QL

• Create “Work Flow”, and execute each step

• When necessary, access to the registry to expand the step of the work flow

• Thus the steps increases dynamically




Query Files can be stored in a local PC (WS)


Monitor Window: Refreshed automatically in every 15 seconds


VOTable as outputVOTable as output

• XML

• Standard data exchange format among VOs defined by the IVOA (see below)


Sample OutputsSample Outputs



FAQs on our PrototypeFAQs on our Prototype

Q) Contents of environment variables are not passed to the executing machines.

A) Prepare a shell script to get environment variables and another script to execute the command. Then GridFTP these shell scripts , and execute the shells in the remote machine.

Q) It is impossible to prepare all users’ account to all relevant machines.

A) Map all accounts on a portal to a single representative one, such as globus.


FAQsFAQs

Q) GT2 has too large overhead before starting actual jobs !! They are several seconds.

Request) Overheads must be < 0.1 seconds excluding the network delays.

A) Use NMI (NSF Middleware Initiative) where each key generation is quite fast through assembler.


FutureFuture• Prototype version 2 (in progress)

　 implement data analysis tools

　　 add more data (SXDF, SDSS, etc.)

• Interoperation with VOs in the world

We have joined the International Virtual Observatory Alliance, and are contributing standardizing protocols for interoperability.


VOs in the WorldVOs in the World

• NVO 　 (USA), AVO (EU),AstroGrid (UK)

• Germany, Canada, Russia, Australia

• India, China, JAPAN

Protocols to interconnect VOs

→ 　 International Virtual Observatory Alliance (IVOA)


サービスサービスサービスサービス

サービスサービス

他システム

処理データ管理データ管理

資源管理資源管理

セキュリティ管理セキュリティ管理

研究者

VO 用ブラウザ

VO 用ブラウザ

Webサーバ

Webサーバ

JVOポータル

JVOポータル

天体カタログ

天体カタログ DB

ワークフロー

データ管理データ管理資源管理資源管理セキュリティ管理セキュリティ管理


サービスサービス利用者独自サービス



サービスサービス天体カタログ

サービス

天体カタログサービス

GRID 環境 A

サービスレジストリ A

サービスレジストリ A

サービスレジストリ C

サービスレジストリ C

DB

DA/DB

Data


サービスサービスDB/DA

サービス

DB/DAサービス

サービスレジストリ B

サービスレジストリ B

仮想観測実行サービス

仮想観測実行サービス

MVC

MVC

MVC

GRID 環境 B

GRID 環境 C

VO 用Viewer

VO 用Viewer



天文台殿

天文台殿



他システム

pipeline

処理

データ管理データ管理





他システム






他システム



セキュリティ管理セキュリティ管理研究者

VO 用ブラウザ

VO 用ブラウザ

Webサーバ

Webサーバ

VOポータル

VOポータル

VO 用Viewer

VO 用Viewer

仮想観測指示


JVO Home PageJVO Home Page

http://jvo.nao.ac.jp/

Documents

2003/03/25天文学会春季年会 JVO の研究開発 (2) 全体進捗 大石雅寿 国立天文台 ＆ 総研大 [email protected]

2003/03/25天文学会春季年会 JVO の研究開発 (2) 全体進捗大石雅寿国立天文台＆総研大 [email protected]