Preserving data ecosystems
After the successful integration FastScan for Isilon, Qumulo, Spectrum Scale / GPFS and others, Atempo has built into its Miria platform a full version of FastScan for Lustre environments.
This new functionality is particularly important given the distributed structure of this HPC File System and the lack of a global solution on the market. Move, protect and archive billions of files, petabytes of data!
Atempo teams have some good news for Lustre users: we have developed a powerful mechanism for collecting data changes and moving new or changed data between storages. The change detection mechanism we call FastScan is integrated into our Miria platform. It works with Lustre core components to collect new, changed and deleted files at node level.
When a backup or archiving job is launched, Miria retrieves the list of new and changed files. It automatically splits the list into many sub-tasks sent to Miria Data Mover machines which adjust the number of simultaneous streams to balance file movement. Full file-tree access is preserved at all times.
This architecture cuts the long wait induced by filesystem tree walks and delivers early data movement. Coupled with the high performance and scalability of Miria, you have a very efficient data management solution for distributed filesystems!
Miria is incredibly powerful and feature-rich and should meet our future needs even though we’ve barely scratched the surface with the solution, we’re very impressed so far.”
Advances in HPC compute and storage workloads continue to drive data management challenges; as new storage technologies are leveraged comes the challenges of multiple storage silos, heterogeneous file systems and long-term storage requirements, this drives the need for reliable and efficient data orchestration tools.”
DiRAC (Distributed Research Utilizing Advanced Computing) is the integrated supercomputing facility for theoretical modeling and HPC-based research in particle physics, astronomy and cosmology and nuclear physics. It is a key part of the infrastructure supporting the UK’s Science and Technology Facilities Council (STFC) Frontier Science programme. Research scientists across the UK can apply for access to DiRAC’s powerful computing facilities.
All key points of DiRAC customer testimonial in under 4 mins
Dr Alastair Basden, Technical Lead for the DiRAC Memory Intensive Service, Durham University
