[en] Short communication

No abstract available

[en] Enormous data volumes and large, geographically dispersed user communities characterize the next generation of experiments in high energy physics and other scientific disciplines. Parallel processing will be integral to the solution of the information storage and retrieval problems that these experiments will engender. The authors describe several approaches to parallel query processing that have been implemented in the early stages of the PASS (Petabyte Access and Storage Solutions) project. These have been tested on an object-oriented persistent event store built from Fermilab CDF data, and evaluated on the 128-processor IBM SP-1 at Argonne National Laboratory, as well as on networks of workstations. They conclude with a discussion of scalability issues, and with a description of ongoing parallel query processing research

[en] The authors present a design and demonstration of a scientific data manager consisting of a low overhead, high performance object store interfaced to a hierarchical storage system. This was done with the framework of the Mark1 testbeds of the PASS project

[en] At a data rate of 200 hertz, event metadata records ('TAGs,' in ATLAS parlance) provide fertile grounds for development and evaluation of tools for scalable data mining. It is easy, of course, to apply HEP-specific selection or classification rules to event records and to label such an exercise 'data mining,' but our interest is different. Advanced statistical methods and tools such as classification, association rule mining, and cluster analysis are common outside the high energy physics community. These tools can prove useful, not for discovery physics, but for learning about our data, our detector, and our software. A fixed and relatively simple schema makes TAG export to other storage technologies such as HDF5 straightforward. This simplifies the task of exploiting very-large-scale parallel platforms such as Argonne National Laboratory's BlueGene/P, currently the largest supercomputer in the world for open science, in the development of scalable tools for data mining. Using a domain-neutral scientific data format may also enable us to take advantage of existing data mining components from other communities. There is, further, a substantial literature on the topic of one-pass algorithms and stream mining techniques, and such tools may be inserted naturally at various points in the event data processing and distribution chain. This paper describes early experience with event metadata records from ATLAS simulation and commissioning as a testbed for scalable data mining tool development and evaluation.

[en] In the ATLAS event store, files are sometimes 'an inconvenient truth.' From the point of view of the ATLAS distributed data management system, files are too small - datasets are the units of interest. From the point of view of the ATLAS event store architecture, files are simply a physical clustering optimization: the units of interest are event collections - sets of events that satisfy common conditions or selection predicates - and such collections may or may not have been accumulated into files that contain those events and no others. It is nonetheless important to maintain file-level metadata, and to cache metadata in event data files. When such metadata may or may not be present in files, or when values may have been updated after files are written and replicated, a clear and transparent model for metadata retrieval from the file itself or from remote databases is required. In this paper we describe how ATLAS reconciles its file and non-file paradigms, the machinery for associating metadata with files and event collections, and the infrastructure for metadata propagation from input to output for provenance record management and related purposes.

[en] The High Energy and Nuclear Physics Data Access Grand Challenge project has developed an optimizing storage access software system that was prototyped at RHIC. It is currently undergoing integration with the STAR experiment in preparation for data taking that starts in mid-2000. The behavior and lessons learned in the RHIC Mock Data Challenge exercises are described as well as the observed performance under conditions designed to characterize scalability. Up to 250 simultaneous queries were tested and up to 10 million events across 7 event components were involved in these queries. The system coordinates the staging of ''bundles'' of files from the HPSS tape system, so that all the needed components of each event are in disk cache when accessed by the application software. The caching policy algorithm for the coordinated bundle staging is described in the paper. The initial prototype implementation interfaced to the Objectivity/DB. In this latest version, it evolved to work with arbitrary files and use CORBA interfaces to the tag database and file catalog services. The interface to the tag database and the MySQL-based file catalog services used by STAR are described along with the planned usage scenarios

[en] The ATLAS event store employs a persistence framework with extensive navigational capabilities. These include real-time back navigation to upstream processing stages, externalizable data object references, navigation from any data object to any other both within a single file and across files, and more. The 2013-2014 shutdown of the Large Hadron Collider provides an opportunity to enhance this infrastructure in several ways that both extend these capabilities and allow the collaboration to better exploit emerging computing platforms. Enhancements include redesign with efficient file merging in mind, content-based indices in optimized reference types, and support for forward references. The latter provide the potential to construct valid references to data before those data are written, a capability that is useful in a variety of multithreading, multiprocessing, distributed processing, and deferred processing scenarios. This paper describes the architecture and design of the next generation of ATLAS navigational infrastructure.

[en] Traditional relational databases have not always been well matched to the needs of data-intensive sciences, but efforts are underway within the database community to attempt to address many of the requirements of large-scale scientific data management. One such effort is the open-source project SciDB. Since its earliest incarnations, SciDB has been designed for scalability in parallel and distributed environments, with a particular emphasis upon native support for array constructs and operations. Such scalability is of course a requirement of any strategy for large-scale scientific data handling, and array constructs are certainly useful in many contexts, but these features alone do not suffice to qualify a database product as an appropriate technology for hosting particle physics or cosmology data. In what constitutes its 1.0 release in June 2011, SciDB has extended its feature set to address additional requirements of scientific data, with support for user-defined types and functions, for data versioning, and more. This paper describes an evaluation of the capabilities of SciDB for two very different kinds of physics data: event-level metadata records from proton collisions at the Large Hadron Collider (LHC), and the output of cosmological simulations run on very-large-scale supercomputers. This evaluation exercises the spectrum of SciDB capabilities in a suite of tests that aim to be representative and realistic, including, for example, definition of four-vector data types and natural operations thereon, and computational queries that match the natural use cases for these data.

[en] Traditional relational databases have not always been well matched to the needs of data-intensive sciences, and to the needs of high energy physics data stores in particular. To address this mismatch, members of the database community and people involved with large scientific data stores in a variety of disciplines have inaugurated an open-source project, SciDB, that aims to develop and deliver database technologies suited to the needs of data-intensive sciences. This paper describes early experience using the first release of SciDB with an initial subset of high energy physics data structures and query patterns. It examines the early capabilities of SciDB, and describes requirements that further development must address if emerging database technologies such as SciDB are to accommodate the data structures, query patterns, computations, and use cases of high energy physics.