Development of "advanced" tools facilitating high-quality and effective unstructured data analysis

To design and develop effective models, techniques and technologies (hereinafter, the toolkit) that assist in collecting, preparing, standardizing, analyzing, learning and recognizing structure in (un)structured data.

1. As part of the research project, a review was made of the largest online stores such as IKEA, Alibaba, Amazon, MediaMarkt, Kaspi.kz, Technodom and others. The analysis showed that a product catalog contains hidden information about the relationships between entities, the types and values of entity properties, and abstract models and properties specific to each format, which together can be used to build a predictive model for detecting structure in a previously unknown catalog. This makes it possible to automate the transformation of data from one format to another, allowing an online store to significantly expand its supplier base and product catalog:

– a systematic analysis was carried out of the features of existing methods, technologies and tools for in-depth big data analysis; – the problem statement was developed and the corresponding requirements formalized for a single format for transforming data from various sources and for a repository of heterogeneous data characterized by different scenarios; – a technical specification of the designed interfaces was developed; – the main idea was formed, i.e. to classify a new, previously unknown product into a previously known format based on textual and numerical statistical information (statistical data inference) and compare it with the models in the system; – the problem statement was formalized, using the Naive Bayes method as its basis. 2. A clustering algorithm was developed that sequentially applies lemmatization, tokenization, stop-listing, tf-idf and singular value decomposition. To find cluster groups, an algorithm based on DBSCAN was developed, using Euclidean distance to determine the distance between objects. The research proved that clustering accuracy depends on the ratio of the number of clusters to the number of objects in a cluster. The number of clusters is determined by the radius of each document, and the number of objects in a cluster by the average number of common objects, in this case words or terms. The clustering algorithm we describe can be used not only to classify groups but also for other purposes, such as finding association rules or finding groups of documents with similar meaning. An experimental analysis of the effectiveness of the developed machine learning technologies was carried out on various business cases, and for each case the most effective machine learning model was selected based on the calculated quality scores. 3. A prototype information system was developed as a tool facilitating high-quality data analysis to solve the task, using a microservice architecture based on Spring Cloud consistent with the requirements. The purpose of our tool is to receive data in various formats from all available sources, categorize it into the main format and then load it into the database in a unified form. Testing stages were carried out: testing and evaluation of results against selected key indicators, data collection, data categorization, data formatting, collection of results and loading data into a template. 4. An experimental analysis and verification of the developed models, methods, algorithms and software complex were carried out on a defined business problem, taking into account the correlation of various metrics. As a result of testing, a data correlation analysis was obtained; to date, the result has exceeded 70%. The developed models, methods and algorithms were also experimentally verified using the built-in data import functionality of ELK, which was used to load the data for processing. The technologies used included Python, Java, Elasticsearch and Kibana. ELK (E – Elasticsearch, L – Logstash, K – Kibana) is a widely used free search engine developed by Elastic together with the related Logstash data collection and log analysis engine and the Kibana analytics and visualization platform; these three products are designed to be used as an integrated solution called the Elastic Stack. 5. As a result of the research, articles were published in journals recommended by the CQASHE of the MES RK and in peer-reviewed international journals indexed in Scopus and Web of Science.

The project results can be commercialized.