CrossMedia

 
CrossMedia Semantic Annotation and Search service
1 May 2010– 31 Dec 2012
 

The CrossMedia e-science platform supports collaborative research communities by providing a comfortable solution to jointly develop semantic- and media search algorithms on common and challenging datasets processed by novel feature extractors. Users form communities on the CrossMedia portal where they can jointly create, build and share search algorithms and media datasets in an iterative way. The system automatically builds indexes and also provides a testing facility as indexes are instantly included in the portal’s search interface to be tested with any desired media-based, semantic, or combined multimodal input. The generated indexes can be shared among research communities for reviewing or with the public for demonstration purposes. Thus, our portal presents an all-in-one solution for researchers: storage and processing capacity, flexible interfaces, built in index structure and innovative user interfaces.

Image removed.

The CrossMedia E-Science Portal aims to satisfy the domain-specific needs of researchers primarily in the image processing scientific field. Registered users can be engaged in different research groups by creating a new society or by joining the desired groups. Every group has its own separate private space in the portal where group members share and collaboratively work with group content including media collections (so-called albums), indexing algorithms and created media indexes.

Category-based and free text annotations can also be attached to media items enabling semantic search. Defining an index on the portal interface launches a series of automated operations that produce the desired index structure in the backend and finally results in making the generated index available for building search requests on the search user interface.

Every piece of group content can be (separately) shared at two different levels: either with other group(s) or with the public. This creates a wide variety of possibilities for inter-group collaboration and for demonstration purposes as well, e.g. setting public visibility on an index tree makes it available to any visitor of the site to use it on the search interface. For testing image descriptors and semantic indexes the portal provides an intuitive search user interface allowing visitors or group members to assemble complex, multipart queries and to adjust the options of query parts.

Image removed.

For demonstration purposes the platform was filled with 5 million images together with their metadata and MPEG-7 descriptors. We used named entity extraction to connect metadata elements with DBpedia resulting an RDF index, which we used to develop custom semantic query types such as querying by location or topic.

We used MPEG-7 image descriptors to implement our image retrieval solution. The color-structure, edge histogram and homogenous texture descriptors proved to be the most useful for content-based searching in images. The multimedia index structure was planned so that it enables the use of descriptors with variable dimensions and non-metric distances as well. Using the combination of in-memory, SSD and disk-based storage, we can build extra large indexes with fast response on queries. In this way, the disk-based CrossMedia solution provides similar speed as the well-known memory-based indexing solutions for image search and retrieval.

Participants

MTA SZTAKI

Department