11/50/50

IPSI Belgrade Ltd.IPSI Belgrade Ltd.

ARTSHOP GALLERYARTSHOP GALLERY

vm@ipsi.co.yuoffice@ipsi.co.yuhttp://www.ipsi.co.yu

22/50/50

AuthorsAuthors

Milutinovic Veljko

Toskov IvanVujovic Ivana

Skundric Nikola Milutinovic Darko

Stojanovski Aleksandar Nikezic Gavro

Anucojic Goran

Radakovic MiroslavMarinkovic Ivan

33/50/50

Introduction – IPSI BgdIntroduction – IPSI Bgd

IPSI BelgradeIPSI Belgrade is a company jointly founded by German and Serbian capital

Partners: - IPSI Fraunhofer, Darmstadt, Germany- Telecom Italia Learning Services, Italy- NYU, School of Continuous Professional Studies, USA- Instituto Tecnologico de Durango, Mexico

44/50/50

Introduction – IPSI BgdIntroduction – IPSI Bgd

- Workspaces of the Future- Environments for Cooperative Working and Learning- Virtual Information and Knowledge Environments- Mobile Interactive Media- Open Adaptive Information Management Systems- Publication Engineering and Technology- Hardware Design and Operating Systems- Networks and WWW

55/50/50

Introduction – IPSI BgdIntroduction – IPSI Bgd

Products:

• Advanced Virtual Gallery• Semantic Web tutorial and book development• The injection cache, the STS cache, VLSI Detection for Internet/Telephony Interfaces, Genetic Search with Spatial/Temporal Mutations, Browser Acceleration, Technology transfer, Testing the Infrastructure for EBI, Socratenon Distant Web Educating Machine, e-Tourism…

66/50/50

Problem StatementProblem Statement

- Creating Web based art gallery with “look and feel” of the real world exhibitions

- Visitor moves through the gallery by “walking with options”

77/50/50

Existing SolutionsExisting Solutions

- Musee national des Arts asiatiques http://www.museeguimet.fr/tour-guimet/index.html

- Web Server of the Galleria degli Uffizi in Florence http://www.uffizi.firenze.it

- The Distributed Interactive Virtual Environment (DIVE) http://www.sics.se/dive/

- The Web3D Repository http://www.web3d.org/vrml/artgal.htm

88/50/50

Proposed SolutionProposed Solution

- Virtual reality gallery

- Advanced search capabilities

- Visitor’s criteria based room generation

99/50/50

Why is it better?Why is it better?

- Dynamically generated gallery- Dynamically generated gallery

- Content based search engine- Content based search engine

- User satisfaction - User satisfaction

1010/50/50

Conditions and AssumptionsConditions and Assumptions

- PC- Internet connection- Internet Explorer 5.0 or higher - Netscape 7.0 - Cortona VRML plug-in for IE

1111/50/50

Analysis and ImplementationAnalysis and Implementation

- Application is written in ASP.NET using C# as code-behind, and ADO.NET for database access.- Database server is SQL Server 2000.- Communication with the database is entirely made through XML (using SQLXML3.0 framework).

- Queries are made in XPath, while adding, changing and deleting of the records is done through UpdateGrams.

- Application is optimized for Internet Explorer 5.0 or higher, at the 1024x768 screen resolution. Netscape 7.0 or higher is also supported.

- 3D gallery is completely generated on the server side (dynamically) using VRML.

1212/50/50

Track 1Track 1

Track Requirements:Track Requirements:• To stand as the integrative part for the other two tracks• To provide:

– User interaction– Database connectivity (database independent)– Search functions

(simple and advanced using Track3 output)– Information brokering between artists and buyers– Administration tools– Artworks management tools, etc.– Thin client (3D scene generation on server side)

1313/50/50

Track 1Track 1

Development Tools:Development Tools:• Application server platform:

– Windows XP Professional– IIS 5.1– MS SQL Server 2000

• Development platform:– ASP.NET– C# as code-behind.

• Communication with the underlying database:– XML & XSD using XPath queries (DB independent)– Currently using SQLXML3.0 add-on for ADO.NET

1414/50/50

Track 1Track 1

Database Design:Database Design:

1515/50/50

Track 1Track 1

Administrator Tools:Administrator Tools:

• Separate entry point:

http://<server_address>/artshop/admin

1616/50/50

Track 1Track 1

Users & Exhibitors:Users & Exhibitors:• Entry point:

http://<server_address>/artshop/index.htm

1717/50/50

Track 1Track 1Interesting Details:Interesting Details:

• Native XML DBMS under development at IPSI Fraunhofer• Practical testing of the XML/XPath database access• Dynamic addition (to the system) of new multimedia types• 3D view of search results

1818/50/50

Track 1Track 1Interesting Details:Interesting Details:• Application that can connect on the fly to any DBMS which

supports XML/XPath is an interesting and possibly useful idea (user just has to set one XML file containing local field mapping, and one XSD to map the database fields to the pre-defined scheme)

• Cons:– XPath queries are lot less powerful then standard SQL queries– Inherently, loss of speed

(one complex SQL query had to be simulated with couple of XPath queries and additional processing in the code).

– For now, SQLXML3.0 does not support complete XPath standard.

1919/50/50

Track 1Track 1Errors Made:Errors Made:

• Initially, content analysis, picture processing, and adding data to database were completely separated (as specified in the contract), with the idea of later (partial) integration.

• Turned out to be a bad idea(required a lot of interventionfrom the ArtShop system administrator when adding artworks).

2020/50/50

Track 1Track 1Lessons Learned:Lessons Learned:• Problem solved by complete integration of forementioned tasks

into the one system process which monitors input directory, automatically schedules picture processing and content analysis, and takes care of updating of all necessary fields in all required databases.

• With that, we achieved maximum automation, reduced time needed for artwork addition, and reduced amount of data transferred through the Internet (between the administrator’s machine and the application host).

2121/50/50

Track 2Track 2

Image-Content-Oriented SearchImage-Content-Oriented SearchTrack requirements:Track requirements:

• Images used for extracting objects are artistic paintings• Image analyses• Extraction of the features• Create XML file for each image• Fetch the database with the features

2222/50/50

Track 2Track 2

Algorithm:Algorithm:

Open the picture

Load parameters, input and output directory

Determine the filter value

Put the picture into the reduced matrix

Determine histogram

Create objects

Merge objects into bigger objects

Create sorted array of objects

Create database objects, prepare them, and put them in XML file and tables in database

2323/50/50

Track 2Track 2

Determine histogram:Determine histogram: Create general histogram

Remove zero values

Sort histogram

Remove redundancies

Sort histogram

Refresh matrix

2424/50/50

Track 2Track 2

Regions forRegions for processing matrix: processing matrix:

P8

P4P5

P5P6P3

P3

P4P5

P3

P8

P8P8

P8

the first colon the last row

the last colon

the rest of the matrix

the last element in the last row

2525/50/50

Creating Creating objects:objects:

Any pixel left in the current region?

p8 = the current pixel

call matrix.unite_pixels method

p8.oi == 0 (it doesn’t belong to any object)

create new object

Any neighbour pixel left?

p8.oi != px.oi (don’t belong to the same object)

take the next pixel

take the next neighbour pixel

px = the neighbour pixel

true

true

true

false

true

false

false

false

2626/50/50

Merge Merge objects objects into into bigger bigger objects:objects:

Is Picture_Objects list empty?

q==true (are there any objects inside Border which colors after translation are the same as the color of Core object)

q = true; Edge[0] = Picture_Objects[0]

i < number of objects inside Edge list

Find all neighbour objects, which colors after translation are the same as the color of Core object, and put them into Border

Move all pixels from objects inside Edge to Core object

Remove objects that are inside Edge from Picture_Objects list

Is Border empty

q=false; add Core object to big list; Remove pixels belonging to Core object from Picture_Objects

false

true

true

Take ith object

i = i+1

Move all objects inside Border to Edge

false

true

false

true

false

2727/50/50

Track 2Track 2

Tools used in development:Tools used in development:• C# programming language – the chosen tool

– Advantage: Includes the best properties from other programming languages (C++, Java, Visual Basic)

– Disadvantage: slower processing speed than C++, which is not necessary in this application

• C++ - the best alternative tool

– Advantage: faster processing speed (unnecessary)

– Disadvantage: more complicated code, 50% of all bugs due to use of pointers

2828/50/50

Track 2Track 2

Original picture:Original picture:

2929/50/50

Track 2Track 2Picture after applyingPicture after applyinghistogram values:histogram values:

3030/50/50

Track 2Track 2Picture representedPicture representedthrough extracted objects:through extracted objects:

3131/50/50

Track 2Track 23D HSL space => 1D histogram3D HSL space => 1D histogram

Index of histogram array Hue Saturation Luminance Description

0 any any <=30 black

1 any any >=lummax white

2 any <20 >30<30+lum

the darkest grey

... … … … …

ilmax-1 any <20 >=30+(ilmax-3)*lum<30+(ilmax-2)*lum

the lightest grey

ilmax <=hue/2>=240-hue/2

>20<20+1*sat

>30<30+lum

the darkest red with the smallest saturation

ilmax+1 <=hue/2>=240-hue/2

>=20+1*sat< 20+2*sat

>30<30+lum

the darkest red with smaller saturation

… … … … …

ilmax+isat-2 <=hue/2>=240-hue/2

>=20+(isat-3)*sat< 20 +(isat-2)*sat

>30<30+lum

the darkest red with the biggest saturation

ilmax+1*(isat-1) <=hue/2>=240-hue/2

>20<20+1*sat

>=30+lum< 30+2*lum

darker red with the smallest saturation

ilmax+1*(isat-1)+1 <=hue/2>=240-hue/2

>=20+1*sat< 20+2*sat

>=30+lum< 30+2*lum

darker red with smaller saturation

… … … … …

ilmax+1*(isat-1)+isat-2 <=hue/2>=240-hue/2

>=20+(isat-3)*sat< 20 +(isat-2)*sat

>=30+lum< 30+2*lum

darker red with the biggest saturation

ilmax+2*(isat-1) <=hue/2>=240-hue/2

>20<20+1*sat

>=30+2*lum< 30+3*lum

dark red with the smallest saturation

3232/50/50

Track 2Track 23D HSL space => 1D histogram3D HSL space => 1D histogram

ilmax+2*(isat-1)+1 <=hue/2>=240-hue/2

>=20+1*sat< 20+2*sat

>=30+2*lum< 30+3*lum

dark red with smaller saturation

… … … … …

ilmax+(ilmax-3)*(isat-1)+1 <=hue/2>=240-hue/2

>=20+1*sat< 20+2*sat

>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum

the brightest red with smaller saturation

… … … … …

ilmax+(ilmax-3)*(isat-1)+(isat-2) <=hue/2>=240-hue/2

>=20+(isat-3)*sat< 20 +(isat-2)*sat

>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum

the brightest red with the biggest saturation

ilmax+1*(ilmax-2)*(isat-1) > hue/2< 3*hue/2

>20<20+1*sat

>30<30+lum

the darkest orange-red with the smallest saturation

ilmax+1*(ilmax-2)*(isat-1)+1 > hue/2< 3*hue/2

>=20+1*sat< 20+2*sat

>30<30+lum

the darkest orange-red with smaller saturation

… … … … …

ilmax+1*(ilmax-2)*(isat-1)+(ilmax-3)*(isat-1)+(isat-2)

> hue/2< 3*hue/2

>=20+(isat-3)*sat< 20 +(isat-2)*sat

>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum

the brightest orange-red with the biggest saturation

ilmax+2*(ilmax-2)*(isat-1) >=3*hue/2< 5*hue/2

>20<20+1*sat

>30<30+lum

the darkest red-orange with the smallest saturation

ilmax+2*(ilmax-2)*(isat-1)+1 >=3*hue/2< 5*hue/2

>=20+1*sat< 20+2*sat

>30<30+lum

the darkest red-orange with smaller saturation

… … … … …

ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+1 >=(2*ihmax-1)*hue/2<(2*ihmax+1)*hue/2

>=20+1*sat< 20+2*sat

>30<30+lum

the darkest magenta-red with smaller saturation

… … … … …

ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+(ilmax-3)*(isat-1)+(isat-2)

>=(2*ihmax-1)*hue/2<(2*ihmax+1)*hue/2

>=20+(isat-3)*sat< 20 +(isat-2)*sat

>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum

the brightest magenta-red with the biggest saturation

3333/50/50

Track 2Track 2Lessons learned:Lessons learned:

• It is impossible to extract objects using only colors as a criterion• It is impossible to extract objects,

even using textures, edges, different transformations as criteria• Semantics should be used in segmentation• Colors are the most important features in artistic paintings

3434/50/50

Track 3Track 3

Track requirements:Track requirements:• Possibility of moving through 3D galleries

• Automatic generation of 3D galleries based on user’s query

• Manual generation of 3D galleries

• User interface for image zooming

• Application for image processing

3535/50/50

Track 3Track 3Underlying algorithms:Underlying algorithms:

• Dynamic creation of gallery

• Creation of static galleries

• Algorithm for picture zooming

• Algorithm for picture processing

3636/50/50

Track 3Track 3

Creation of galleries:Creation of galleries:

• Validation of the created gallery

• Forming VRML files depending on users query

• Determining the number of pictures in the gallery

• Drawing a 2D floorplan based on the 3D gallery

• “Forest fire” algorithm for filling the floorplan with color

3737/50/50

Track 3Track 3

Picture processing:Picture processing:

• Loading image into memory

• Clone image into different-size copies

• Filtering of copies

• Parting of copies

3838/50/50

Track 3Track 3

Development tools:Development tools:

• C# in .NET Framework for programming image processing

• Macromedia Dreamweaver for programming zoom tool

• VRML Pad v2.0

3939/50/50

Track 3Track 3

Flowchart:Flowchart:

4040/50/50

Track 3Track 3

Creation of galleries:Creation of galleries:

• Making files based on users data

• Putting data on server so it can be available for artists

• Artist chooses which gallery he/she will be using for exhibition

• User can move through 3D world

• Selecting the textures for gallery

• Selecting the starting position of the user

4141/50/50

Track 3Track 3

NF filter details:NF filter details:

• If the new picture is smaller, every pixel is one pixel of the old picture.

• If the new picture is bigger, pixels are calculated based on the pixels surrounding the current.

4242/50/50

Track 3Track 3

Errors made:Errors made:

• Requests were not precise, so there was a gap at the end of the project between wanted and done

• Better results could be done with better using of ASP and XML

4343/50/50

Track 3Track 3

Lessons learned:Lessons learned:

• Every member of the team gets a part where his experience is dominating

• More planning at the start reduces a lot of work later

• Good communication between programmers can save a lot of time

4444/50/50

DemoDemo

4545/50/50

Future PlansFuture Plans

• Improving the existing 3D dynamic gallery

• Improving search engine capabilities

• Improving feature extraction algorithms and objects recognition

4646/50/50

Future Track 1Future Track 1

3D Multimedia Showroom Environment:3D Multimedia Showroom Environment:

• Implementing a general Web based 3D Multimedia Showroom Environment

• Exhibiting various MM data types: images, 3D objects, videos, audio, etc.

• Set of MM data types should be extendable

4747/50/50

Future Track 2Future Track 2

MM Object Feature Extraction:MM Object Feature Extraction:

• Implementing algorithms and software components for extracting features from MM data types (images, videos, 3D objects), in order to enable content based search

• System should be extendable (“plug-in”)

4848/50/50

Future Track 3Future Track 3

Semantic Abstraction of MM Feature Spaces:Semantic Abstraction of MM Feature Spaces:

• Developing methods and SW components which derive mapping from extracted features of MM objects to semantic concepts

• Using intelligent classification algorithms (Neural Networks, Fuzzy Classifier)

• Developing semantic query engine (answering questions, which could previously only be answered by humans)

4949/50/50

UsabilityUsability

• Art GalleriesArt Galleries

• MuseumsMuseums

• Exhibition FairsExhibition Fairs

5050/50/50

Instead of ConclusionInstead of Conclusion

IPSI Belgrade, office@ipsi.co.yuhttp://www.ipsi.co.yu