MULTIPLE MEDIA INFORMATION SEARCH FRAMEWORK

Abstract

Nowadays, together with the increasing spread of online multimedia information, user’s information needs have become more complex. To fulfill information needs, users mainly rely on web search engines. The traditional ways of presenting search results are often unsatisfactory. Web search engines usually provide the search of multimedia documents that encapsulates different media objects. A web page is a most known example of a multimedia document. Web search approaches enable vertical search of media objects that belongs to different media formats. In vertical web search, the retrieval granules are media objects presented by employing non-blended integration mechanisms. The presentation of search results is also linear. In recent years, aggregated search approaches have emerged to satisfy user’s complex information needs. Aggregated search approaches retrieve, merge, rank, and present results to the users from separate vertical media sources. The search results are possibly integrated via blended, partial-blended, or non-blended integration mechanisms; however, the presentation is still linear. In aggregated search, the retrieved media objects have semantic and multimodal similarity relationships that are not exploited to support user’s exploration activities. Over the years, aggregated search tools have been investigated by the researchers. Aggregated search tools usually do not consider the multimodal nature of media objects in search results exploration activities. They address the exploration of only specific media types or a subset of them. Aggregated search tools enable only document-to-document browsing. Furthermore, retrieval granule are either media objects or multimedia documents. This thesis aims to address the issues that are related to the results exploration in aggregated search. The thesis focuses to provide a novel mechanism to explore results in aggregated search. The main objective is to give users a possibility to dynamically visualize and explore a search result space built over a repository of multimedia documents and their connected media objects in an integrated way. To do this a novel multiple media information search framework is proposed. Particularly, search framework initiates a search result space over the retrieved multimedia documents and their connected media objects. The search result space treats multimedia documents and media objects as retrieval granules. The search result space connects multimedia documents and media objects and media objects with each other via part-of and multimodal (textual, acoustic, and visual) similarity relationships respectively. The search result space is further exploited in results exploration activities. The search framework formally defines a set of components to provide an exploration of results in aggregated search of multiple media information. The search framework is further represented as an architecture that encapsulates framework components in data, search, data model, viii and interface layers. Data layer retrieves multimedia documents, media objects, and features associated with media objects. Search layer provides fielded search of media objects connected with the multimedia documents. Results representation layer initiates a search result space over multimedia documents and their connected media objects retrieved in aggregated search. Interface layer enables expression of multiple media based information needs and exploration of the search results. We realized the framework by implementing a full-fledged multiple media information search tool mainly to provide nonlinear interaction with the search results via full-blended integration, browsing, and visualization in an integrated way. The search tool instantiates a search result space via a particular graph data model on a publically available dataset of multimedia documents and their connected media objects. The search tool provides search results exploration by giving a results exploration mechanism and supporting various types of search tasks via particular results exploration interface components. We evaluated blended integration, browsing, and the connected search results exploration. We considered correctness and reachability factors in the effectiveness evaluation of blended integration and browsing respectively. Blended integration and browsing are compared further with the optimally ranked representation of search results, and they give satisfactory results. The effectiveness of search results exploration mechanism, search task support, and search interface components evaluated via task and scenario-based usability tests. In particular, we employed successful task completion, time-on-task, subjective measures of usability in supported search tasks, and overall user recommendations to evaluate results exploration mechanism. We tested the usability of lookup and exploratory search tasks via scenario-based evaluation. Furthermore, we conducted frequency usage, an interface component, and search results multi-representation analysis to highlight the usability of search interface components. The usability tests revealed that most users completed search task in given time constraints; search interface provides satisfactory results in subjective measures; users are satisfied with the search results exploration mechanism. The search interface supports lookup and exploratory search tasks to interact with the results. The users mostly spent time to interact with results presented via the linear list and browseable grid-based representation; however, users prefer all types of interactions given in the search interface to explore the results. Along with that, they like the browsing of multiple media information via grid and graph-based representations. Our framework provides nonlinear, multimodal, and unified exploration of results in multiple media information aggregated search in a usable way.