The conclusion is the result of research led by two researchers from the High-Assurance Software Laboratory (HasLab) of the School of Engineering conducted at INESC TEC, which aims to ensure that scientists using supercomputers can carry out scientific studies, in areas such as medicine, natural sciences, climate change and others, faster. The results of the research work were presented in late February, at one of the most important conferences in storage systems: USENIX FAST.

One of the current issues is the large amount of digital data, e.g., genomic data, which these studies need to access efficiently. In addition, supercomputers host hundreds to thousands of studies simultaneously, making this process even more difficult, since studies compete for access to shared storage resources, where their data is kept.

As a result of this work, and as a solution to this problem, the researchers from the High-Assurance Software Laboratory (HASLab), in co-authorship with researchers from UT Austin and AIST, presented the paper “PAIO: General, Portable I/O Optimizations with Minor Application Modifications” in this year’s edition of the USENIX Conference on File and Storage (FAST), one of the most important conferences in the storage systems field. The event took place between February 22 and 24, in Santa Clara (USA).

“PAIO provides the essential I/O mechanisms for researchers to carry out their studies with fairness, with respect to the access to the required data. This is particularly important for studies with similar runtimes, ensuring that some do not finish within a few minutes or hours, while others require days or weeks”, said Ricardo Macedo.

Another contribution of the team is the Monarch system, entitled “Accelerating Deep Learning Training through Transparent Storage Tiering”, which will be presented in May at the 22nd edition of the International Symposium on Cluster, Cloud and Internet Computing (CCGrid’22).

The system proposes storage optimisations that accelerate the training of artificial intelligence models. “These optimisations can be used, for instance, to predict the spreading of diseases like COVID-19 much faster – in some cases reducing the training time by half” said João Paulo.

International collaborations contribute to advances in data storage

The work developed by the team of researchers is the result of the collaboration established with several renowned international institutions in the storage and high-performance computing fields, namely, the University of Texas at Austin (UT Austin), Texas Advanced Computing Center (TACC) and Hood College of the United States of America, as well as Japan’s National Institute of Advanced Industrial Science and Technology (AIST).

“Thanks to this partnership, the team is able to discuss with other experts which problems are associated with data storage, resulting from different types of applications on supercomputers”, mentioned João Paulo. In addition, these collaborations have also favoured “the validation of solutions developed in top infrastructures”, added the researcher.

It is important to mention that these results were obtained within the scope of the BigHPC and PAStor projects, funded by the COMPETE 2020 Programme and the Foundation for Science and Technology, in partnership with UT Austin – through the UT Austin Portugal Program.