[en] At CERN LHCb experiment (Large Hadron Collider beauty), in order to reconstitute and filter events, a huge computing facility is required (currently about 1500 nodes). This computing farm, running SLC (Scientific Linux CERN), a Red Hat Enterprise Linux (RHEL) derivative, is net booted and managed using Quattor to allow easy maintenance. LHCb is also using around 500 disk-less Credit Card PCs (CCPCs) embedded in the front end electronic cards. To add flexibility to the disk-less nodes handling, we had the idea of using the file system union, which is usually used on Linux live media. The principle of the file system union is to join several file systems, at least one read-only (generally) and one read-write and use that union as a normal Linux file system. We have successfully tested this idea. The use of this new file system has helped to reveal some security vulnerability in the Linux kernel

[en] History has shown, many times computer logs are the only information an administrator may have for an incident, which could be caused either by a malfunction or an attack. Due to the huge amount of logs that are produced from large-scale IT infrastructures, such as LHCb Online, critical information may be overlooked or simply be drowned in a sea of other messages. This clearly demonstrates the need for an automatic system for long-term maintenance and real time analysis of the logs. We have constructed a low cost, fault tolerant centralized logging system which is able to do in-depth analysis and cross-correlation of every log. This system is capable of handling O(10000) different log sources and numerous formats, while trying to keep the overhead as low as possible. It provides log gathering and management, Offline analysis and online analysis. We call Offline analysis the procedure of analyzing old logs for critical information, while Online analysis refer to the procedure of early alerting and reacting. The system is extensible and cooperates well with other applications such as Intrusion Detection / Prevention Systems. This paper presents the LHCb Online topology, problems we had to overcome and our solutions. Special emphasis is given to log analysis and how we use it for monitoring and how we can have uninterrupted access to the logs. We provide performance plots, code modification in well-known log tools and our experience from trying various storage strategies. (authors)

[en] The LHCb experiment, one of the 4 large particle detector at CERN, counts in its Online System more than 2000 servers and embedded systems. As a result of ever-increasing CPU performance in modern servers, many of the applications in the controls system are excellent candidates for virtualization technologies. We see virtualization as an approach to cut down cost, optimize resource usage and manage the complexity of the IT infrastructure of LHCb. Recently we have added a Kernel Virtual Machine (KVM) cluster based on Red Hat Enterprise Virtualization for Servers (RHEV) complementary to the existing Hyper-V cluster devoted only to the virtualization of the windows guests. This paper describes the architecture of our solution based on KVM and RHEV as along with its integration with the existing Hyper-V infrastructure and the Quattor cluster management tools and in particular how we use to run controls applications on a virtualized infrastructure. We present performance results of both the KVM and Hyper-V solutions, problems encountered and a description of the management tools developed for the integration with the Online cluster and LHCb SCADA control system based on PVSS. (authors)

[en] The LHCb Data Acquisition system reads data from over 300 read-out boards and distributes them to more than 1500 event-filter servers. It uses a simple push-protocol over Gigabit Ethernet. After filtering, the data is consolidated into files for permanent storage using a SAN-based storage system. Since the beginning of data-taking many lessons have been learned and the reliability and robustness of the system has been greatly improved. We report on these changes and improvements, their motivation and how we intend to develop the system for Run 2. We also will report on how we try to optimise the usage of CPU resources during the running of the LHC ('deferred triggering') and the implications on the data acquisition.

[en] The first level trigger of LHCb accepts one million events per second. After preprocessing in custom FPGA-based boards these events are distributed to a large farm of PC-servers using a high-speed Gigabit Ethernet network. Synchronisation and event management is achieved by the Timing and Trigger system of LHCb. Due to the complex nature of the selection of B-events, which are the main interest of LHCb, a full event-readout is required. Event processing on the servers is parallelised on an event basis. The reduction factor is typically 1/500. The remaining events are forwarded to a formatting layer, where the raw data files are formed and temporarily stored. A small part of the events is also forwarded to a dedicated farm for calibration and monitoring. The files are subsequently shipped to the CERN Tier0 facility for permanent storage and from there to the various Tier1 sites for reconstruction. In parallel files are used by various monitoring and calibration processes running within the LHCb Online system. The entire data-flow is controlled and configured by means of a SCADA system and several databases. After an overview of the LHCb data acquisition and its design principles this paper will emphasize the LHCb event filter system, which is now implemented using the final hardware and will be ready for data-taking for the LHC startup. Control, configuration and security aspects will also be discussed

[en] The LHCb experiment at CERN has decided to optimise its physics reach by removing the first level hardware trigger for 2020 and beyond. In addition to requiring fully redesigned front-end electronics this design creates interesting challenges for the data-acquisition and the rest of the online computing system. Such a system can only be realized within realistic cost using as much off-the-shelf hardware as possible. Relevant technologies evolve very quickly and thus the system design is architecture-centred and tries to avoid to depend too much on specific technologies. In this paper we describe the design, the motivations for various choices and the current favoured options for the implementation, and the status of the R&D. We will cover the back-end readout, which contains the only custom-made component, the event-building, the event-filter infrastructure, and storage. (paper)

[en] This paper presents the design of the LHCb trigger and its performance on data taken at the LHC in 2011. A principal goal of LHCb is to perform flavour physics measurements, and the trigger is designed to distinguish charm and beauty decays from the light quark background. Using a combination of lepton identification and measurements of the particles' transverse momenta the trigger selects particles originating from charm and beauty hadrons, which typically fly a finite distance before decaying. The trigger reduces the roughly 11 MHz of bunch-bunch crossings that contain at least one inelastic pp interaction to 3 kHz. This reduction takes place in two stages; the first stage is implemented in hardware and the second stage is a software application that runs on a large computer farm. A data-driven method is used to evaluate the performance of the trigger on several charm and beauty decay modes.

[en] The LHCb collaboration has redesigned its trigger to enable the full offline detector reconstruction to be performed in real time. Together with the real-time alignment and calibration of the detector, and a software infrastructure to make persistent the high-level physics objects produced during real-time processing, this redesign enabled the widespread deployment of real-time analysis during Run 2. We describe the design of the Run 2 trigger and real-time reconstruction, and present data-driven performance measurements for a representative sample of LHCb's physics programme.