Intellectual Technologies on Transport Intellectual Technologies on Transport Интеллектуальные технологии на транспорте 2413-2527 81650 10.20295/2413-2527-2024-137-45-55 Системный анализ, управление и обработка информации, статистика System analysis, management and information processing, statistics Системный анализ, управление и обработка информации, статистика Method of Transition from Data Warehouses to Geographic Information System Data Lakes Based on Lambda Architecture Метод перехода от хранилищ данных к озерам данных геоинформационных систем на основе Лямбда-архитектуры Абу Хасан Рахеб Abu Hasan Raheb ragheb1997@yandex.ru Кириенко Андрей Борисович Kirienko Andrey Borisovich anbokir@mail.ru Хомоненко Анатолий Дмитриевич Khomonenko Anatoly Dmitrievich khomon@mail.ru

доктор технических наук;

doctor of technical sciences;

 Петербургский государственный университет путей сообщения Императора Александра I Санкт-Петербург Россия Emperor Alexander I St. Petersburg State Transport University Saint-Petersburg Russian Federation Военно-космическая академия имени А. Ф. Можайского Санкт-Петербург Россия Mozhaisky Military Aerospace Academy Saint-Petersburg Russian Federation Петербургский государственный университет путей сообщения Императора Александра I Санкт-Петербург Россия Emperor Alexander I St. Petersburg State Transport University St. Petersburg Russian Federation Военно-космическая академия имени А. Ф. Можайского Санкт-Петербург Россия Mozhaisky Military Aerospace Academy Saint Petersburg Russian Federation 14 04 2024 14 04 2024 1 45 55 11 04 2024 https://itt-pgups.ru/en/nauka/article/81650/view

В статье рассматривается переход от традиционных хранилищ данных к озерам данных в геоинформационных системах с использованием Лямбда-архитектуры. Приводится обзор основных этапов перехода, включая планирование, сбор и обработку данных, запросы данных, аналитику данных и управление метаданными. Особое внимание уделяется взаимодействию озер данных и ГИС, а также примерному коду обработки больших данных на основе Лямбда архитектуры. Рассматриваются преимущества использования озер данных в ГИС и возможности интеграции современных технологий обработки данных.

This paper discusses the transition from traditional data warehouses to data lakes in geographic information systems using Lambda architecture. Provides an overview of the key transition steps, including planning, data collection and processing, data querying, data analytics, and metadata management. Particular attention is paid to the interaction of data lakes and GIS, as well as sample big data processing code based on Lambda architecture. The advantages of using data lakes in GIS and the possibilities o integrating modern data processing technologies are considered.

 озера данных хранилища данных Лямбда-архитектура геоинформационные системы метаданные обработка больших данных интеграция данных анали- тика данных переход от хранилищ данных data lakes; data warehouses; Lambda architecture; geographic information systems; metadata; big data processing; data integration; data analytics; transition from data warehouses

Ёcy, М. Т. Принципы организации распределенных баз данных = Principles of Distributed Database Systems. Fourth Edition / М. Т. Ёсу, П. Вальдуриес; пер. с англ. А. А. Слинкина. — Москва: ДМК Пресс, 2021. — 672 с. Ecy, M. T. Principy organizacii raspredelennyh baz dannyh = Principles of Distributed Database Systems. Fourth Edition / M. T. Esu, P. Val'duries; per. s angl. A. A. Slinkina. — Moskva: DMK Press, 2021. — 672 s. Bhattacherjee, S. RStore: A Distributed Multi-Version Document Store / S. Bhattacherjee, A. Deshpande // Proceedings of the 34th International Conference on Data Engineering (ICDE 2018), (Paris, France, 16–19 April 2018). — Institute of Electrical and Electronics Engineers, 2018. — Pp. 389–400. DOI: 10.1109/ICDE.2018.00043. Bhattacherjee, S. RStore: A Distributed Multi-Version Document Store / S. Bhattacherjee, A. Deshpande // Proceedings of the 34th International Conference on Data Engineering (ICDE 2018), (Paris, France, 16–19 April 2018). — Institute of Electrical and Electronics Engineers, 2018. — Pp. 389–400. DOI: 10.1109/ICDE.2018.00043. Leveraging the Data Lake: Current State and Challenges / C. Giebler, C. Gröger, E. Hoos, [et al.] // Big Data Analytics and Knowledge Discovery (DaWaK 2019): Proceedings of the 21st International Conference (Linz, Austria, 26–29 August 2019) / C. Ordonez, [et al.] (eds.). — Cham: Springer Nature, 2019. — Pp. 179–188. — (Lecture Notes in Computer Science. Vol. 11708). DOI: 10.1007/978–3–030–27520–4_13. Leveraging the Data Lake: Current State and Challenges / C. Giebler, C. Gröger, E. Hoos, [et al.] // Big Data Analytics and Knowledge Discovery (DaWaK 2019): Proceedings of the 21st International Conference (Linz, Austria, 26–29 August 2019) / C. Ordonez, [et al.] (eds.). — Cham: Springer Nature, 2019. — Pp. 179–188. — (Lecture Notes in Computer Science. Vol. 11708). DOI: 10.1007/978–3–030–27520–4_13. Lock, M. Maximizing Your Data Lake with a Cloud or Hybrid Approach / M. Lock; Aberdeen Group. — 2016. — 4 p. URL: http://technology-signals. com/wp-content/uploads/download-managerfiles/maximizingyourdatalake.pdf (дата обращения 12.01.2024). Lock, M. Maximizing Your Data Lake with a Cloud or Hybrid Approach / M. Lock; Aberdeen Group. — 2016. — 4 p. URL: http://technology-signals. com/wp-content/uploads/download-managerfiles/maximizingyourdatalake.pdf (data obrascheniya 12.01.2024). Extending Data Lake Metadata Management by Semantic Profiling / J.W. Ansari, N. Karim, S. Decker, [et al.] // Proceedings of the 15th International Extended Semantic Web Conference (ESWC 2018), (Heraklion, Crete, Greece 03–07 June 2018). — Springer International Publishing, 2018. — 15 p. URL: http://2018.eswc-conferences.org/wp-content/uploads/2018/02/ ESWC2018_paper_127.pdf (дата обращения 12.01.2024) Extending Data Lake Metadata Management by Semantic Profiling / J.W. Ansari, N. Karim, S. Decker, [et al.] // Proceedings of the 15th International Extended Semantic Web Conference (ESWC 2018), (Heraklion, Crete, Greece 03–07 June 2018). — Springer International Publishing, 2018. — 15 p. URL: http://2018.eswc-conferences.org/wp-content/uploads/2018/02/ ESWC2018_paper_127.pdf (data obrascheniya 12.01.2024) CoreDB: A Data Lake Service / A. Beheshti, B. Benatallah, R. Nouri, [et al.] // Proceedings of the 2017 ACM Conference on Information and Knowledge Management (CIKM ‘17), (Singapore, Singapore, 06–10 November 2017). — New York: Association for Computing Machinery, 2017. — Pp. 2451–2454. DOI: 10.1145/3132847.3133171. CoreDB: A Data Lake Service / A. Beheshti, B. Benatallah, R. Nouri, [et al.] // Proceedings of the 2017 ACM Conference on Information and Knowledge Management (CIKM ‘17), (Singapore, Singapore, 06–10 November 2017). — New York: Association for Computing Machinery, 2017. — Pp. 2451–2454. DOI: 10.1145/3132847.3133171. Data Lake Management: Challenges and Opportunities / F. Nargesian, E. Zhu, R.J. Miller, [et al.] // Proceedings of the VLDB Endowment. 2019. Vol. 12, Is. 12. Pp. 1986–1989. DOI: 10.14778/3352063.3352116. Data Lake Management: Challenges and Opportunities / F. Nargesian, E. Zhu, R.J. Miller, [et al.] // Proceedings of the VLDB Endowment. 2019. Vol. 12, Is. 12. Pp. 1986–1989. DOI: 10.14778/3352063.3352116. CLAMS: Bringing Quality to Data Lakes / M. Farid, A. Roatis, I.F. Ilyas, [et al.] // Proceedings of the 2016 International Conference on Management of Data (SIGMOD ‘16), (San Francisco, CA, USA, 26 June‑01 July 2016). — New York: Association for Computing Machinery, 2016. — Pp. 2089–2092. DOI: 10.1145/2882903.2899391. CLAMS: Bringing Quality to Data Lakes / M. Farid, A. Roatis, I.F. Ilyas, [et al.] // Proceedings of the 2016 International Conference on Management of Data (SIGMOD ‘16), (San Francisco, CA, USA, 26 June‑01 July 2016). — New York: Association for Computing Machinery, 2016. — Pp. 2089–2092. DOI: 10.1145/2882903.2899391. Keeping the Data Lake in Form: DS-kNN Datasets Categorization Using Proximity Mining / A. Alserafi, A. Abelló, O. Romero, T. Calders // Model and Data Engineering (MEDI 2019): Proceedings of the 9th International Conference (Toulouse, France, 28–31 October 2019) / K.-D. Schewe, N.K. Singh (eds.). — Cham: Springer Nature, 2019. — Pp. 35–49. — (Lecture Notes in Computer Science. Volume 11815). DOI: 10.1007/978–3–030–32065–2_3. Keeping the Data Lake in Form: DS-kNN Datasets Categorization Using Proximity Mining / A. Alserafi, A. Abelló, O. Romero, T. Calders // Model and Data Engineering (MEDI 2019): Proceedings of the 9th International Conference (Toulouse, France, 28–31 October 2019) / K.-D. Schewe, N.K. Singh (eds.). — Cham: Springer Nature, 2019. — Pp. 35–49. — (Lecture Notes in Computer Science. Volume 11815). DOI: 10.1007/978–3–030–32065–2_3. Dataset Discovery in Data Lakes / A. Bogatu, A. A.A. Fernandes, N.W. Paton, N. Konstantinou // Proceedings of the IEEE 36th International Conference on Data Engineering (ICDE 2020), (Dallas, TX, USA, 20–24 April 2020). — Institute of Electrical and Electronics Engineers, 2020. — Pp. 709–720. DOI: 10.1109/ ICDE48307.2020.00067. Dataset Discovery in Data Lakes / A. Bogatu, A. A.A. Fernandes, N.W. Paton, N. Konstantinou // Proceedings of the IEEE 36th International Conference on Data Engineering (ICDE 2020), (Dallas, TX, USA, 20–24 April 2020). — Institute of Electrical and Electronics Engineers, 2020. — Pp. 709–720. DOI: 10.1109/ ICDE48307.2020.00067. Goods: Organizing Google’s Datasets / A. Halevy, F. Korn, N.F. Noy // Proceedings of the 2016 International Conference on Management of Data (SIGMOD ‘16), (San Francisco, CA, USA, 26 June‑01 July 2016). — New York: Association for Computing Machinery, 2016. — Pp. 795–806. DOI: 10.1145/2882903.2903730. Goods: Organizing Google’s Datasets / A. Halevy, F. Korn, N.F. Noy // Proceedings of the 2016 International Conference on Management of Data (SIGMOD ‘16), (San Francisco, CA, USA, 26 June‑01 July 2016). — New York: Association for Computing Machinery, 2016. — Pp. 795–806. DOI: 10.1145/2882903.2903730. Sawadogo, P.N. On Data Lake Architectures and Metadata Management / P.N. Sawadogo, J. Darmont // Journal of Intelligent Information Systems. 2021. Vol. 56, Is. 1. Pp. 97–120. DOI: 10.1007/s10844–020–00608–7. Sawadogo, P.N. On Data Lake Architectures and Metadata Management / P.N. Sawadogo, J. Darmont // Journal of Intelligent Information Systems. 2021. Vol. 56, Is. 1. Pp. 97–120. DOI: 10.1007/s10844–020–00608–7. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics / M. Armbrust, A. Ghodsi, R. Xin, M. Zaharia // Proceedings of the 11th Annua Conference on Innovative Data Systems Research (CIDR 21), (11– 15 January 2021, Online). — 8 p. URL: http://cidrdb.org/cidr2021/ papers/cidr2021_paper17.pdf (дата обращения 12.01.2024). Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics / M. Armbrust, A. Ghodsi, R. Xin, M. Zaharia // Proceedings of the 11th Annua Conference on Innovative Data Systems Research (CIDR 21), (11– 15 January 2021, Online). — 8 p. URL: http://cidrdb.org/cidr2021/ papers/cidr2021_paper17.pdf (data obrascheniya 12.01.2024). Jensen, R., Shen, H., & Yue, P. (2017). Geo-Analytics: Integrating Geospatial Information Systems and Big Data Analytics. In Geographic Information Science (pp. 297–315). Springer International Publishing. Jensen, R., Shen, H., & Yue, P. (2017). Geo-Analytics: Integrating Geospatial Information Systems and Big Data Analytics. In Geographic Information Science (pp. 297–315). Springer International Publishing. Gao, S., & Liu, Z. (2021). A Review of Big Data and Geospatial Data Integration for Geocomputation and Decision Support. Remote Sensing, 13(2), 316. https://doi.org/10.3390/rs13020316. Gao, S., & Liu, Z. (2021). A Review of Big Data and Geospatial Data Integration for Geocomputation and Decision Support. Remote Sensing, 13(2), 316. https://doi.org/10.3390/rs13020316. International Network Performance and Security Testing Based on Distributed Abyss Storage Cluster and Draft of Data Lake Framework / B.-R. Cha, S. Park, J.-W. Kim // Security and Communication Networks. 2018. Art. No. 1746809. 14 p. DOI: 10.1155/2018/1746809. International Network Performance and Security Testing Based on Distributed Abyss Storage Cluster and Draft of Data Lake Framework / B.-R. Cha, S. Park, J.-W. Kim // Security and Communication Networks. 2018. Art. No. 1746809. 14 p. DOI: 10.1155/2018/1746809. Rituerto, Á., & Alvarez, J. M. (2019). Geo-Big Data: A Literature Review. ISPRS International Journal of Geo-Information, 8(11), 471. https://doi.org/10.3390/ijgi8110471. Rituerto, Á., & Alvarez, J. M. (2019). Geo-Big Data: A Literature Review. ISPRS International Journal of Geo-Information, 8(11), 471. https://doi.org/10.3390/ijgi8110471. Безворотных, А. В. Lambda architecture для корпоративного «Озера данных» / А. В. Безворотных; науч. рук. Р. И. Кузьмич // Молодость. Интеллект. Инициатива: Материалы X Международной научно-практической конференции студентов и магистрантов (Витебск, Беларусь, 22 апреля 2022 г.). — Витебск: Витебский гос. ун-т имени П.М. Машерова, 2022. — С. 6–8. Bezvorotnyh, A. V. Lambda architecture dlya korporativnogo «Ozera dannyh» / A. V. Bezvorotnyh; nauch. ruk. R. I. Kuz'mich // Molodost'. Intellekt. Iniciativa: Materialy X Mezhdunarodnoy nauchno-prakticheskoy konferencii studentov i magistrantov (Vitebsk, Belarus', 22 aprelya 2022 g.). — Vitebsk: Vitebskiy gos. un-t imeni P.M. Masherova, 2022. — S. 6–8. Implementing Big Data Lake for Heterogeneous Data Sources / H. Mehmood, E. Gilman, M. Cortes, [et al.] // Proceedings of the IEEE 35th International Conference on Data Engineering Workshops (ICDEW 2019), (Macao, China, 08–12 April 2019). — Institute of Electrical and Electronics Engineers, 2020. — Pp. 37–44. DOI: 10.1109/ICDEW.2019.00–37. Implementing Big Data Lake for Heterogeneous Data Sources / H. Mehmood, E. Gilman, M. Cortes, [et al.] // Proceedings of the IEEE 35th International Conference on Data Engineering Workshops (ICDEW 2019), (Macao, China, 08–12 April 2019). — Institute of Electrical and Electronics Engineers, 2020. — Pp. 37–44. DOI: 10.1109/ICDEW.2019.00–37. Marz, N. Big Data: Principles and best practices of scalable realtime data systems / N. Marz, J. Warren. — Shelter Island (NY): Manning Publications, 2015. — 328 p. Marz, N. Big Data: Principles and best practices of scalable realtime data systems / N. Marz, J. Warren. — Shelter Island (NY): Manning Publications, 2015. — 328 p. Sawadogo, P.N. Metadata Management for Textual Documents in Data Lakes / P.N. Sawadogo, T. Kibata, J. Darmont // Proceedings of the 21st International Conference on Enterprise Information Systems (ICEIS 2019), (Heraklion, Crete, Greece, 03–05 May 2019). — SciTePress, 2019. — Vol. 1. — Pp. 72–83. DOI: 10.5220/0007706300720083. Sawadogo, P.N. Metadata Management for Textual Documents in Data Lakes / P.N. Sawadogo, T. Kibata, J. Darmont // Proceedings of the 21st International Conference on Enterprise Information Systems (ICEIS 2019), (Heraklion, Crete, Greece, 03–05 May 2019). — SciTePress, 2019. — Vol. 1. — Pp. 72–83. DOI: 10.5220/0007706300720083. Visual Bayesian Fusion to Navigate a Data Lake / K. Singh, K. Paneri, A. Pandey, [et al.] // Proceedings of the 19th International Conference on Information Fusion (FUSION 2016), (Heidelberg, Germany, 05–08 July 2016). — Institute of Electrical and Electronics Engineers, 2016. — Pp. 987–994. Visual Bayesian Fusion to Navigate a Data Lake / K. Singh, K. Paneri, A. Pandey, [et al.] // Proceedings of the 19th International Conference on Information Fusion (FUSION 2016), (Heidelberg, Germany, 05–08 July 2016). — Institute of Electrical and Electronics Engineers, 2016. — Pp. 987–994. Munshi, A. A. Data Lake Lambda Architecture for Smart Grids Big Data Analytics / A.A. Munshi, Y. A.-R. I. Mohamed // IEEE Access. 2018. Vol. 6. Pp. 40463–40471. DOI: 10.1109/ACCESS. 2018.2858256. Munshi, A. A. Data Lake Lambda Architecture for Smart Grids Big Data Analytics / A.A. Munshi, Y. A.-R. I. Mohamed // IEEE Access. 2018. Vol. 6. Pp. 40463–40471. DOI: 10.1109/ACCESS. 2018.2858256. DataHub — A Metadata Platform for the Modern Data Stack. URL: http://datahubproject.io (дата обращения 25.12.2023). DataHub — A Metadata Platform for the Modern Data Stack. URL: http://datahubproject.io (data obrascheniya 25.12.2023).