toggle visibility Search & Display Options

Select All    Deselect All
 |   | 
Details
   print
  Record Links
Author (up) Fernandez Casani, A.; Garcia Montoro, C.; Gonzalez de la Hoz, S.; Salt, J.; Sanchez, J.; Villaplana Perez, M. doi  openurl
  Title Big Data Analytics for the ATLAS EventIndex Project with Apache Spark Type Journal Article
  Year 2023 Publication Computational and Mathematical Methods Abbreviated Journal Comput. Math. Methods  
  Volume 2023 Issue Pages 6900908 - 19pp  
  Keywords  
  Abstract The ATLAS EventIndex was designed to provide a global event catalogue and limited event-level metadata for ATLAS experiment of the Large Hadron Collider (LHC) and their analysis groups and users during Run 2 (2015-2018) and has been running in production since. The LHC Run 3, started in 2022, has seen increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. A new core storage service is being developed in HBase/Phoenix, and there is work in progress to provide at least the same functionality as the current one for increased data ingestion and search rates and with increasing volumes of stored data. In addition, new tools are being developed for solving the needed access cases within the new storage. This paper describes a new tool using Spark and implemented in Scala for accessing the big data quantities of the EventIndex project stored in HBase/Phoenix. With this tool, we can offer data discovery capabilities at different granularities, providing Spark Dataframes that can be used or refined within the same framework. Data analytic cases of the EventIndex project are implemented, like the search for duplicates of events from the same or different datasets. An algorithm and implementation for the calculation of overlap matrices of events across different datasets are presented. Our approach can be used by other higher-level tools and users, to ease access to the data in a performant and standard way using Spark abstractions. The provided tools decouple data access from the actual data schema, which makes it convenient to hide complexity and possible changes on the backed storage.  
  Address [Casani, Alvaro Fernandez; Montoro, Carlos Garcia; de la Hoz, Santiago Gonzalez; Salt, Jose; Sanchez, Javier; Perez, Miguel Villaplana] CSIC UV, Inst Corpuscular Phys IFIC, E-46980 Paterna, Spain, Email: alvaro.fernandez@ific.uv.es;  
  Corporate Author Thesis  
  Publisher Wiley-Hindawi Place of Publication Editor  
  Language English Summary Language Original Title  
  Series Editor Series Title Abbreviated Series Title  
  Series Volume Series Issue Edition  
  ISSN ISBN Medium  
  Area Expedition Conference  
  Notes WOS:001079548500001 Approved no  
  Is ISI yes International Collaboration no  
  Call Number IFIC @ pastor @ Serial 5706  
Permanent link to this record
Select All    Deselect All
 |   | 
Details
   print

Save Citations:
Export Records:
ific federMinisterio de Ciencia e InnovaciĆ³nAgencia Estatal de Investigaciongva