Development of mobile intelligent services for museum visitors based on machine learning with augmented reality technology

The rapid development of modern information technology requires new ways for users to interact with devices, that is, the development of new human–computer interfaces. New approaches to human–computer interaction do not use conventional graphical menus, forms or toolbars. They are based on interaction methods inherent only in human communication: instead of traditional controls, they use training examples, gestures and human speech. The specialized mobile services based on machine learning and augmented reality technology, as well as the Virtual Museum developed in this project, are aimed at preserving and popularizing the cultural heritage of the Republic of Kazakhstan, especially among young people, which will undoubtedly be facilitated by the virtualization of museum exhibits and their presentation using augmented reality technology.

Project Objectives

• Research and analysis of international experience in developing mobile museum applications for preserving the historical heritage of regions (virtual museums, virtual historical landscapes, virtual architectural monuments, virtual interiors, etc.).
• Development of an algorithm for the precise positioning of visual virtual objects relative to real-world objects based on augmented reality technology using simplified 3D models of objects and object scanning data.
• Development of an algorithm for determining the parameters of identified objects using simplified 3D models of objects and a database of museum exhibit descriptions.
• Development of the "Virtual Museum" mobile application with an artificial intelligence module that classifies and identifies museum exhibits using a mobile device camera, including the development of an algorithm for retraining the network on object data collected by the application.
• Testing of the developed "Virtual Museum" mobile application. Preparation of documentation on the use of the developed "Virtual Museum" application.

Novelty

The scientific novelty lies in developing machine learning models for mobile devices with reduced hardware and time costs that operate in real time, reducing the amount of data required for training, and introducing augmented reality technology to implement a digital museum.

A literature review of existing mobile applications for cultural organizations, namely museums, was carried out in order to understand and analyze the current state of the field and to formulate requirements for the solution being developed. The proposed analogues were studied and analyzed from various perspectives, such as existing solutions for developing museum exhibit classifiers, the architecture and implementation of the software solutions, the functionality they offer, and the cultural organizations serving as platforms for their implementation.

As a result of the analysis, general requirements for the "Virtual Museum" application were developed. The business requirements of the end user were described, possible types of museum visitor behavior scenarios were identified, as well as the desired result (response) from the application. A conceptual model of the future application was developed.

A user interface concept for the "Virtual Museum" application was proposed. Possible behavior scenarios for museum staff and visitors were identified.

The interface was also considered from the perspective of introducing augmented reality technology into the application, i.e. the concept was extended to embed models of recognized and classified exhibits and additional information about them into the interface.