PAVEL NEYEZHMAKOV – PhD, First Deputy General Director of scientific work, National Scientific Center “Institute of Metrology” (Kharkov, Ukraine), Vice-President, COOMET Secretariat (Euro-Asian cooperation of national metrological institutions)
SERGIY ZUB – Senior Scientist of the NSC “Institute of Metrology”, Kharkov, Ukraine
STANISLAV ZUB – PhD, Doctoral candidate of Taras Shevchenko National University of Kiev, Kiev, Ukraine
A global computer infrastructure system called the “grid”, created in response to challenges resulting from the increasing complexity of experimental physical assemblies and information systems, is presented in this paper as an optimal IT platform for metrology tasks, i.e. the assurance of metrological traceability (often in geographically remote regions) and measurement data protection in global networks. The emphasis is placed on the use of a new com - ponent of the grid, the instrument element (IE), which is intended to ensure secure, remote, joint team work when monitoring and managing instruments and data generated and stored on distributed scientific measuring equipment using conventional grid resources. The article describes the variety of all possible IE applications within the available grid technology for those metrology tasks where IT support is required. The IE grid becomes an optimal environment which can be used to effectively perform measurement tasks which have the highest level of measurement data transfer, storage and processing safety, revealing new oppor - tunities to track measurement results and ensure a high level of reliability of these results.
International metrology organizations, including the OIML, play an important role in coordinating metro - logical activity in the globalized world . This activity is primarily intended to ensure metrological traceability, often though not always in geographically remote regions, as well as to achieve a high level of confidence in measurements.
The coordination and management of measurement processes often separated by long distances should be supported by state-of-the-art information technology. It is believed that this support could be based on the internet. However, this system provides no possibility to control the measurement and computation processes, and the data transferred are under constant threat of being falsified or even irretrievably lost.
The problems of interaction between remote devices as well as of measurement data protection in the global networks are being overcome [2,3,4,5], but the solutions are always partial, problem-dependent and take a lot of special effort.
Is there any way out? The answer is yes. Based on the same hardware (i.e. the same telecommunication elements as used by the internet) a brand new system of globally distributed computations can exist which ensures the highest possible level of safety for today which excludes even such troublesome phenomena as viruses and hacker attacks.
This system appears to meet the challenges of qualitative complexity of experimental physical assemblies and information systems which require new monitoring, management and operating approaches. The new paradigm of distributed computations is named the “grid”.
There are many implementations of this concept in the USA and in the European Union. However, the most wide-spread implementation of the grid concept in recent years is the gLite middleware. The gLite is used to support the most difficult physical installations created by mankind, the Large Hadron Collider (LHC). The development of this middleware is closely connected with the Conseil Européen pour la Recherche Nucléaire (CERN) experiments [13–16]. The most sophisticated measuring systems are now accumulated there and the problem encountered was the lack of computing resources to process the huge amount of information obtained during LHC experiments that were generated there for the first time.
Despite the historical roots of the European DataGrid (EDG) project, and then the LHC Computing Grid (LCG), the gLite middleware was developed as a general platform for constructing a distributed comput - ing network [10,11,16].
Figure 1 Grid system flowchart
With financial support from the European Union, the European scientific community launched the Enabling Grids for the E-sciencE (EGEE) project within which the gLite middleware was created. Remote control and data acquisition are part of the existing grid concept. However, grid developments are more often focused on the sharing of computational and storage resources. In fact, these resources are much needed for most applications and users.
2 Classical grid structure
A classical grid is composed of the following basic structural elements (see Figure 1): a set of computa - tional nodes (PC) with installed Worker Nodes (WNs); a set of Resource Centers (RC) comprising the Computing Element (CE) and the Storage Element (SE); and a set of basic Grid Services (the gLite middleware) with User Interfaces (UI) for access to Grid.
The gLite package is a complete solution for the grid including both the basic low-level programs and a number of high-level services. It accumulates the components of the best projects existing today such as the Condor and Globus, as well as the components developed within the LCG project . The gLite is compatible with such task planners as PBS , Condor and LSF, and contains the basic services that facilitate the creation of grid applications for any applied area. The Globus Toolkit developed by American scientists has actually become a world standard. It particularly includes a special HTTP-protocol for the use of the Grid Resource Allocation Management (GRAM); an extended version of the Grid File Transfer Protocol (GridFTP); the Grid Security Infrastructure (GSI); distributed data access based on the Lightweight Directory Access Protocol (LDAP); and remote data access via GASS (Globus Access to Secondary Storage) interface. A resource center is composed of two types of resources [10,11]: computing resources controlled via the Computing Element (CE), and data storage resources accessible via the Storage Element (SE) which makes it possible to store and transport data between similar resources or between a resource and a grid user. Basic grid services ensure the operation of the entire grid system and are divided into the following parts:
¨ Workload Management System (WMS);
¨ Data Management System (DMS) consisting of a file directory service and a metadata directory service;
¨ information and monitoring system (IS) which solves the problem of collecting and managing data on the status of the grid infrastructure;
¨ Grid Security Infrastructure (GSI) consisting of Certificate Authority (CA) issuing and supporting certificates;
¨ Virtual Organization Membership Service (VOMS);
¨ MyProxy Service (MP), the task of which is to provide the authentication and authorization between various grid system components;
¨ Logging and Bookkeeping (LB) System tracking task realization processes; and
¨ Accounting Subsystem (AS) designed to register the use of computing resources.
Thanks to gLite, the geographically distributed set of resources is represented as a single resource for users.
The gLite middleware plays the role of a personal computer operating system in the grid.
Figure 2 Interaction between the Instrument Element (IE)
and other grid components
3 Security in the grid
Security is at the heart of the grid. Its components are both special secure file transfer protocols such as GSIFTP, Secure RFIO, Gsidcap, and an authorization system which, in turn, is layered and deeply separates the grid resource access rights.
GSIFTP provides the functionality of the FTP protocol, but with the support of the Grid Security
Infrastructure (GSI). GSI ensures the security in in - secure public data networks by providing such services as authentication, file transfer confidentiality and unified grid logon. GSI uses X.509 digital certificates as user and resource identifiers. This protocol offers fast, safe and efficient file transfer and makes it possible to control file transfer between two storage elements separated from the user (third-party transfers), as well as to transfer data in several parallel flows.
What is new is the use of the Virtual Organization Membership Service (VOMS) which stores information on the affiliation of users to certain groups and virtual organizations and their role within them. User rights can be set depending on user affiliation to a particular virtual organization, group or role (with VOMS).
Thus, the grid security system possesses all known cryptography tools and offers a harmonious authorization and access rights system based on electronic certificates.
4 Instrument element
There are more and more cases today where in addition to resource distribution, close interaction between the instruments and the grid users must be ensured. Remote access is often necessary from any grid site.
The Grid Enabled Remote Instrumentation with Distributed Control and Computation (GridCC) project  was launched to meet these challenges. The project aims to use grid capabilities for secure, remote, joint team work when monitoring and managing instruments and data generated and stored on distributed scientific equipment using conventional grid resources.
The Instrument Element (IE) was developed within this project and has been successfully used in various scientific collaborations for remote interaction with devices in the grid environment.
The IE consists of a linked service collection ensuring the functionality for configuring, managing
and monitoring of measuring instruments outside the IE interface which allows their interaction with the rest of the grid. Figure 2 illustrates the interaction between the IE and its users and other grid components .
The key IE characteristics are described below .
The IE complies with the following functional requirements:
¨ uniform model of an instrument;
¨ standard grid access to the instruments;
¨ capability to communicate between different instruments belonging to different institutes and Virtual Organizations (VOs).
Remark: An IE user is not only a human being but also any application with certain rights specified in
an electronic grid certificate.
An IE user can have one of the following roles shown in Figure 3 :
1 Observer – can monitor the instrument;
2 Operator – can have access to an instrument configuration, as well as the right to control and monitor the instrument;
3 Administrator – can create an instrument configuration which then becomes accessible for 1 and 2.
Table 1 Nonfunctional requirements
At any moment there may be many observers and administrators, but only one operator which uses (i.e. controls) the instrument.
If the instrument is complex and has independent modules (as is the case for the LHC CMS detector
[12,13]) then several operators may each use a separate module.
The IE was developed based on the functional (see above) and technical requirements (see Table 1) . An instrument or a set of instruments can be both very simple and very complex. Therefore, the following instrument categories were introduced when creating an abstract model of the instrument:
¨ dummy instrument;
¨ smart instrument;
¨ smart instrument in an ad hoc network.
Figure 4 shows a uniform model of instrument control .
Figure 4 Uniform model of instrument control
Figure 5 shows a flowchart describing the instrument integration within the grid .
5 The grid and some metrology tasks
Those areas of metrology where the application of the grid could significantly improve the efficiency of the metrological activity are described below.
Figure 5 Instrument integration within the grid
5.1 Commercial transactions with consumers
Electricity meters, gas meters, heat meters, and water meters are constituent elements of the system of commercial transactions with end consumers and, in general, are in the legal metrology domain.
The authors expect that the grid, as a global distributed computing system in which security issues have systematically been solved to the highest level of modern information technology, will also be introduced in the banking sector, and commercial transactions with end customers will be integrated into this system.
5.2 Long-term storage and transfer of measurement data
The OIML specifies the legal metrology requirements for the long-term storage and transfer of measurement data, as well as for the uploading of new software versions for measuring instruments  in its Document OIML D 31:2008 General requirements for software controlled measuring instruments.
The grid has specifically been designed to solve similar tasks. It is known that the data hypothetically
demonstrating the existence of the Higgs boson in the LHC experiments are a negligible part in the whole array (~ 1015 bytes per year of collision results [10,11]). Therefore, when developing the grid instrumentation, the objective was set to prevent any distortion or loss of experiment data.
Thus, the use of the grid and its Instrument Element component automatically solve the legal metrology problems specified in OIML D 31.
5.3 Remote comparisons of measurement standards, measurement reference standards and devices
In many cases modern computer technology allows for remote comparisons of measuring instruments [4, 5]. There are international programs in clock synchronization, navigation and other areas within which remote comparisons are carried out.
The grid can move these approaches to an absolutely new level both qualitatively and quantitatively, providing better processing efficiency and simultaneous data security.
5.4 Key comparisons of primary measurement standards
These comparisons are often time-consuming and expensive. Compared to the traditional internet, the grid can provide more effective tools to carry out remote comparisons of measurement standards, uniform calculations, and safe data transfer. Most operations, sometimes even all of them, can be made remotely.
5.5 Linux OS embedded instruments
The software contained in measuring instruments is becoming more and more complex. Linux OS embedded instruments are becoming more widely used, and as measuring instruments the software component is under legal metrological control. The issues involved in testing such systems and ensuring the security of measurement data are crucial. The above mentioned requirements in OIML D 31 should be met, and although this is a new and challenging task , it is one which is already familiar to grid developers.
Today, LHC is the most complex measuring instrument in the history of our civilization where similar and many other tasks are solved using the grid. The tools the grid provides as well as the newly developed IE provide many opportunities to effectively and safely use measuring instruments with embedded Linux OS.
The authors propose to use the ‘thin client’ concept as another approach to solve measuring tasks with complex software. In other words, instruments should have minimum software integrating them into the grid, and the complex processing of raw data should be performed in the grid environment.
The choice of an approach is up to measuring instrument developers.
5.6 “Reference” software
One of the main methods to study measuring instrument software is to compare it with “reference”
software. However, the term “reference” software is not sufficiently defined. Besides, the comparison can be made based on both the results of code execution and the literal code matching.
In the future, the grid will be able to assign a clear meaning to the “reference” software concept and make the comparison on a new, higher level. In the authors’ opinion, leading metrological organizations could set and complete the task of creating a metrological reference software repository as one of the grid resources, similar to the gLite middleware repositories . In the
authors’ opinion, this is the very software to be used in key comparisons of measurement standards.
A relatively small group of highly skilled metrologists, mathematicians, and programmers would be
quite sufficient to create a “reference” program resource.
5.7 “Reference” test data
Besides the issue in section 5.6 there is also a problem to create a grid resource for “reference” test data  or, in a broader meaning, test tasks. Such tasks have been used for many years to test clusters within the computer experiment assurance system at the LHC based on the grid [12–15].
5.8 Smart electrical grid
Recently, much attention has been paid to so-called “smart meters” that allow energy savings to be made. It is stated in  that one of the major goals of the European Metrology Research Programme (EMRP) is “to develop a metrological measurement infrastructure in Europe to support successful implementation of a Smart Electrical Grid”.
Remark: Within the “Smart Electrical Grid” concept the term “grid” means mains electrical power,
which differs from the meaning of “grid” as an information computer network that is discussed in this paper.
EMRP specifies the following research areas in Smart Electrical Grid :
“a) Measurement framework for monitoring stability of smart grids via application of reliable and accurate Phasor Measurement units;
b) Traceable on-site energy measurement systems (smart meters) for ensuring fair energy trade;
c) Remote on-site measurement of power quality and efficiency; and
d) Modelling, simulation and network analysis of the system state of smart grids”.
The available grid concept realization - the gLite middleware - together with the Instrument Element
provide a complete set of tools to create applications for solving the above-mentioned problems in this area [20, 21].
Taking into account the conceptuality and integrity of the computing resources management system and the information security system, as well as the genericity of the instrument model (see above) in the IE, the development of grid-independent applications duplicating the solutions for these problems seems inexpedient to the authors, both with regard to time, financial, and manpower costs and to the quality of solving the problems.
A number of crucial metrological problems need to be adequately addressed by modern computer technologies. The authors are convinced that the grid is one such technology that meets today’s global challenges.
The creation of the Instrument Element as a grid component makes this system an optimal environment for effective monitoring, management and servicing of measuring resources which has the highest level of measurement data transfer, storage and processing security and which reveals new opportunities to track measurement procedures and assure a high level of confidence to these measurements.
In addition to the communities already using the grid [18,19]:
¨ earthquake community,
¨ environment community,
¨ experimental science community, metrologists can obtain an extremely powerful tool to
¨ realize their goals in the
¨ metrology community.
 P.I. Neyezhmakov, 20 years of COOMET: We measure together for a better tomorrow, OIML Bulletin,
Volume LII, Number 3, January 2011, p.32.
 P.I. Neyezhmakov, S.I. Zub, N. Zisky, J. Neumann, Application of technologies of safe data transfer in
metrology, Mathematics, Statistics and Computation to Support Measurement Quality International
Seminar, 30 June–2 July 2009, St. Petersburg.
 N. Zisky, Security system for trusted computing in metrology, PTB-BIPM workshop “Impact IT in
 A. Sand, G. Parkin, M. Stevens, A generic approach to distributed instrumentation, PTB-BIPM workshop
“Impact IT in Metrology”, 2007.
 H. Matsumoto, H. Yoshida, Recent status of remotecalibration (e-trace) system, PTB-BIPM workshop
“Impact IT in Metrology”, 2007.
 F. Thiel, U. Grottker, and D. Richter, The challenge for legal metrology of operating systems embedded in
measuring instruments, OIML Bulletin, Volume LII, Number 1, January 2011, p.7.
 K. Hossain, Global challenges for metrology, NCSL International Workshop and Symposium, 2010.
 OIML D 31:2008, General requirements for software controlled measuring instruments.
 H.R. Cook, M.G. Cox, M.P. Dainton, P.H. Harris, Methodology for testing spreadsheets and other packages
used in metrology. Report to National Measurement System Policy Unit, September 1999.
 S. Zub, L. Levchuk, P. Sorokin and D. Soroka, Grid middleware configuration at the KIPT CMS Linux
cluster, Nuclear Instruments and Methods in Physics Research A.
 L.G. Levchuk, D.V. Soroka, M.V. Voronko, S. Zub, LCG Middleware at the KIPT CMS Linux Cluster, Visnyk of Kharkov National University. Series: MIA. 2005 #5(703), pp.74–86.
 CMS Collaboration (S. Chatrchyan, ..., L. Levchuk, S. Lukyanenko, D. Soroka, S. Zub, ..., et al.),
Commissioning of the CMS experiment and the cosmic run at four tesla, arXiv:0911.4845. 2009.
 CMS Collaboration (S. Chatrchyan, ..., L. Levchuk, S. Lukyanenko, D. Soroka, S. Zub, ..., et al.), CMS data
processing workflows during an extended cosmic ray run, arXiv:0911.4842. 2009.
 O.O. Bunetsky, L.G. Levchuk, S.T. Lukyanenko, A.S. Pristavka, D.V. Soroka, P.V. Sorokin, S.S. Zub,
Preparation of KIPT (Kharkov) computing facility for CMS data analysis, Proceedings of the XXII
International Symposium on Nuclear Electronics and Computing (NEC’2009), Varna, Bulgaria, 7–14
September, 2009 (in print).
 S. Lukyanenko, O. Bunetsky, L. Levchuk, D. Soroka, P. Sorokin, S. Zub, On search for heavy Higgs in H → ZZ
→ 2μ2n decays with CMS, 12th Annual CMS/RDMS Collaboration Conference, IHEP, Minsk, Belarus,
14–19 September 2008.
 LHC Computing Grid – Technical Design Report, 2005, LCG-TRD-001 CERN-LHCC-2005-024.
 F. Lelli, E. Frizziero, M. Gulmini, G. Maron, S. Orlando, A. Petrucci S. Squizzato (2007), The many faces of the integration of instruments and the grid, Int. J. Web and Grid Services, Vol. 3, No. 3, pp. 239–266.
 Website of the GRIDCC project, http://www.gridcc.org
 Website of the DORII project, http://www.dorii.eu
 G. Maron, Instruments and Sensors on the Grid, 2007, Website of the GRIDCC project, http://www.gridcc.org
 L. Dickens , The GRIDCC project, 2007, Website of the GRIDCC project, http://www.gridcc.org
OIML BULLETIN VOLUME L I I I • NUMBER 3 • JULY 2012