A Real Time Processing system for big data in astronomy: Applications to HERA

La Plante, P.; Williams, P. K. G.; Kolopanis, M.; Dillon, J. S.; Beardsley, A. P.; Kern, N. S.; Wilensky, M.; Ali, Z. S.; Abdurashidova, Z.; Aguirre, J. E.; Alexander, P.; Balfour, Y.; Bernardi, G.; Billings, T. S.; Bowman, J. D.; Bradley, R. F.; Bull, P.; Burba, J.; Carey, S.; Carilli, C. L.; Cheng, C.; Deboer, D. R.; Dexter, M.; de Lera Acedo, E.; Ely, J.; Ewall-Wice, A.; Fagnoni, N.; Fritz, R.; Furlanetto, S. R.; Gale-Sides, K.; Glendenning, B.; Gorthi, D.; Greig, B.; Grobbelaar, J.; Halday, Z.; Hazelton, B. J.; Hewitt, J. N.; Hickish, J.; Jacobs, D. C.; Julius, A.; Kerrigan, J.; Kittiwisit, P.; Kohn, S. A.; Lanman, A.; Lekalake, T.; Lewis, D.; Liu, A.; Macmahon, D.; Malan, L.; Malgas, C.; Maree, M.; Martinot, Z. E.; Matsetela, E.; Mesinger, A.; Molewa, M.; Morales, M. F.; Mosiane, T.; Murray, S.; Neben, A. R.; Nikolic, B.; Parsons, A. R.; Pascua, R.; Patra, N.; Pieterse, S.; Pober, J. C.; Razavi-Ghods, N.; Ringuette, J.; Robnett, J.; Rosie, K.; Santos, M. G.; Sims, P.; Smith, C.; Syce, A.; Thyagarajan, N.; Zheng, H.

doi:10.1016/j.ascom.2021.100489

As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.

A Real Time Processing system for big data in astronomy: Applications to HERA

La Plante P.;Williams P. K. G.;Kolopanis M.;Dillon J. S.;Beardsley A. P.;Kern N. S.;Wilensky M.;Ali Z. S.;Abdurashidova Z.;Aguirre J. E.;Alexander P.;Balfour Y.;Bernardi G.;Billings T. S.;Bowman J. D.;Bradley R. F.;Bull P.;Burba J.;Carey S.;Carilli C. L.;Cheng C.;DeBoer D. R.;Dexter M.;de Lera Acedo E.;Ely J.;Ewall-Wice A.;Fagnoni N.;Fritz R.;Furlanetto S. R.;Gale-Sides K.;Glendenning B.;Gorthi D.;Greig B.;Grobbelaar J.;Halday Z.;Hazelton B. J.;Hewitt J. N.;Hickish J.;Jacobs D. C.;Julius A.;Kerrigan J.;Kittiwisit P.;Kohn S. A.;Lanman A.;Lekalake T.;Lewis D.;Liu A.;MacMahon D.;Malan L.;Malgas C.;Maree M.;Martinot Z. E.;Matsetela E.;Mesinger A.;Molewa M.;Morales M. F.;Mosiane T.;Murray S.;Neben A. R.;Nikolic B.;Parsons A. R.;Pascua R.;Patra N.;Pieterse S.;Pober J. C.;Razavi-Ghods N.;Ringuette J.;Robnett J.;Rosie K.;Santos M. G.;Sims P.;Smith C.;Syce A.;Thyagarajan N.;Zheng H.

2021

Abstract

As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.

Scheda breve

Scheda completa

Scheda completa (DC)

	Anno di pubblicazione
	
				2021
			
	Settore Scientifico Disciplinare (validi fino a 24/06/2024)
	
				Settore FIS/05 - Astronomia e Astrofisica
			
	Titolo Rivista
	
				ASTRONOMY AND COMPUTING
			
	DOI
	
				https://dx.doi.org/10.1016/j.ascom.2021.100489
			
	Parole chiave
	
				Astronomy — Software; Data analysis — Physical sciences and engineering; Data analysis — Software; Development; Methods
			
	Appare nelle tipologie:
	
				1.1 Articolo in rivista

File in questo prodotto:

File	Dimensione	Formato
1-s2.0-S2213133721000433-main.pdf accesso aperto Tipologia: Published version Licenza: Creative Commons Dimensione 1.01 MB Formato Adobe PDF	1.01 MB	Adobe PDF
2104.03990.pdf accesso aperto Tipologia: Submitted version (pre-print) Licenza: Non pubblico Dimensione 808.33 kB Formato Adobe PDF	808.33 kB	Adobe PDF