Norha M Villegas
email: nvillega@cs.uvic.ca

Hausi A Müller

Gabriel Tamura

Laurence Duchien

Rubby Casallas
email: rcasalla@uniandes.edu.co

A Framework for Evaluating Quality-Driven Self-Adaptive Software Systems

Keywords: D.2.8 [Software Engineering]: Metrics-complexity measures, performance measures Software adaptation properties, software adaptation metrics, assessment and evaluation of self-adaptive systems, software

Over the past decade the dynamic capabilities of self-adaptive software-intensive systems have proliferated and improved significantly. To advance the field of self-adaptive and selfmanaging systems further and to leverage the benefits of selfadaptation, we need to develop methods and tools to assess and possibly certify adaptation properties of self-adaptive systems, not only at design time but also, and especially, at run-time. In this paper we propose a framework for evaluating quality-driven self-adaptive software systems. Our framework is based on a survey of self-adaptive system papers and a set of adaptation properties derived from control theory properties. We also establish a mapping between these properties and software quality attributes. Thus, corresponding software quality metrics can then be used to assess adaptation properties.

INTRODUCTION

Over the past decade, self-adaptation has increasingly become a fundamental concern in the engineering of software systems to reduce the high costs of software maintenance and evolution and to regulate the satisfaction of functional and extra-functional requirements under changing conditions. Even though adaptation mechanisms have been widely investigated in the engineering of dynamic software systems, their application to real problems is still limited due to a lack of methods for validation and verification of complex, adaptive, nonlinear applications [START_REF]Technology Horizons a Vision for Air Force Science & Technology During 2010-2030[END_REF].

After an exhaustive analysis of self-adaptive approaches, we concluded that adaptation properties and the corresponding metrics are rarely identified or explicitly addressed in papers dealing with the engineering of dynamic software systems. Consequently, without explicit adaptation properties it is impossible to assess and certify adaptive system behavior. In light of this, evaluation techniques, such as run-time validation and verification, are needed to advance the field.

To leverage the capabilities of self-adaptive systems, it is necessary to validate adaptation mechanisms to ensure that self-adaptive software systems function properly and users can trust them. To address this problem we propose a framework for evaluating self-adaptive systems, where adaptation properties are specified explicitly and driven by quality attributes, such as those defined in [START_REF] Barbacci | Quality attributes[END_REF]. Our framework provides (i) a set of dimensions useful to classify self-adaptive systems; (ii) a compendium of adaptation properties for control loops (i.e., in terms of the controller and the managed system); (iii) a mapping of adaptation properties to quality attributes to evaluate adaptation properties; and (iv) a set of quality metrics to evaluate adaptation properties and quality attributes.

To define adaptation properties for our framework, we analyzed existing self-adaptive approaches and investigated properties used in control theory. We then established a mapping between adaptation properties and software quality attributes. Then, we identified a set of metrics used to evaluate software quality attributes. The mapping of adaptation properties to quality attributes and their corresponding quality metrics constitute our framework. The actual evaluation of a self-adaptive system involves both the managed system (i.e., process) and the managing system (i.e., controller).

Borrowing properties and metrics from control theory and re-interpreting them for software-intensive self-adaptive systems is not a trivial task because the semantics of the concepts involved in adaptation for control theory are different than those for self-adaptive software. Moreover, existing self-adaptive software approaches do not generally address adaptation properties explicitly. Yet another important challenge is that, in general, self-adaptive software systems are nonlinear systems [START_REF] Hellerstein | Research challenges in control engineering of computing systems[END_REF].

Metrics to evaluate feedback control systems depend on the properties that result from the relationship between the control objectives, the target system's measured outputs, the disturbances affecting the system and how the target system is considered in the adaptation strategy [START_REF] Hellerstein | Feedback Control of Computing Systems[END_REF]. Our re-interpretation results from the analysis of several representative approaches and strategies that have been proposed to achieve behavior modification in a managed system. From the identified relationship and the analysis of these strategies we identified two main dimensions to classify and evaluate self-adaptive software. These dimensions arise from the way the strategy addresses (i) the managed system (i.e., the system to be controlled), and (ii) the controller itself. In the managed system dimension, we identified two groups. In the first one, the control paradigm, the managed system's behavior is modeled, evaluated and influenced without affecting its internal structure; in the second group, the software engineering or planning paradigm, it is the managed system's structure that is modeled and modified to influence the system's behavior. In the controller dimension we identified four types of control actions: (i) continuous signals that affect behavioral properties of the managed system; (ii) discrete operations that affect the computing infrastructure of the managed system; (iii) discrete operations that affect the processes of the managed system; and (iv) discrete operations that affect the managed system's software architecture. Hence, the nature of the adaptation strategy-structural or behavioral-and the way how the managed system is affected by the controller-control actions-define the classification of adaptive systems in our analysis.

Some previous papers have addressed the evaluation of self-adaptive software. In [START_REF] Meng | On evaluating self-adaptive software[END_REF], Meng proposed a mapping of fundamental concepts from control theory to self-adaptive software systems. In his vision, the fundamental properties to evaluate self-adaptation are stability and robustness. These two properties are analyzed and characterized in terms of what they imply for the programming paradigms, architectural styles, modeling paradigms, and software engineering principles. However, his evaluation model is descriptive and not being applied to any approach. In the taxonomy proposed by Salehie and Tahvildari, several representative projects addressing the adaptation of software systems were surveyed in terms of a set of adaptation concerns: how, what, when and where [START_REF] Salehie | Self-adaptive software: Landscape and research challenges[END_REF]. They also proposed a hierarchical view of self-* properties and discussed their relationship with quality factors of software systems [START_REF] Salehie | Autonomic computing: emerging trends and open problems[END_REF]. However, the scope of their work did not include the identification of metrics for evaluating self-adaptive software systems in light of the identified adaptation properties and quality attributes.

Our contribution in this paper differs from the aforementioned in the following aspects. First, we provide a reconciled definition of a more comprehensive list of properties found in control theory and contrast them in the analyzed approaches of self-adaptive software systems. Second, none of the studied contributions presents a comprehensive and unified list of adaptation properties applicable to software systems. Third, we provide a valuable foundation to evaluate adaptation properties in terms of quality attributes and metrics as widely practiced in the engineering of software systems.

The remainder of this paper is organized as follows. Section 2 presents our proposed model to characterize and classify self-adaptive software according to the dimensions mentioned above. Section 3 presents the analysis of selected adaptive systems based on the characterization model pre-sented in Sect. 2. Section 4 presents a compendium of the metrics and properties found in the analyzed approaches with their corresponding definitions. This section also presents our proposed mapping between adaptive properties and quality attributes and how this mapping can be used to evaluate adaptive systems. Section 5 discusses different aspects of our analysis and some challenges to be addressed for the evaluation of self-adaptation. Finally, Section 6 concludes the paper.

A MODEL TO CHARACTERIZE SELF-ADAPTIVE SOFTWARE

In this section we propose a model consisting of eight analysis dimensions to characterize self-adaptive software. This model constitutes a foundation for evaluating self-adaptive systems. For each of the analysis dimensions, the model considers a set of standardized classification options. These options resulted from combining classification attributes from recognized authoritative sources (e.g., Software Engineering Institute (SEI)) with those found in the set of papers that we analyzed. For instance, the set of options for the analyzed quality attributes as observable adaptation properties was identified mainly using the taxonomy proposed by an SEI study [START_REF] Barbacci | Quality attributes[END_REF]. This taxonomy provides a comprehensive characterization of software quality attributes, their concerns, factors that affect them and methods for their evaluation.

For each analysis dimension, we include the relevant options available from control theory, as follows.

• Adaptation goal. This is the main reason or justification for the system or approach to be self-adaptive. Adaptation goals are usually defined through one or more of the self-* properties, the preservation of specific quality of service (QoS) properties, or the regulation of non-functional requirements in general.

• Reference inputs. The concrete and specific set of value(s) and corresponding types that are used to specify the state to be achieved and maintained in the managed system by the adaptation mechanism, under changing conditions of system execution. Reference inputs are specified as (a) single reference values (e.g., a physically or logically-measurable property); (b) some form of contract (e.g., quality of service (QoS), service level agreements (SLA), or service level objectives (SLO); (c) goal-policy-actions; (d) constraints defining computational states (according to the particular proposed definition of state); or even (e) functional requirements (e.g., logical expressions as invariants or assertions, regular expressions).

• Measured outputs. The set of value(s) (and corresponding types) that are measured in the managed system. Naturally, as these measurements must be compared to the reference inputs to evaluate whether the desired state has been achieved, it should be possible to find relationships between these inputs and outputs. Furthermore, we consider two aspects on the measured outputs: how they are specified and how monitored. For the first aspect, the specification, the identified options were (a) continuous domains for single variables or signals; (b) logical expressions or conditions for contract states; and (c) conditions expressing states of system malfunction. On the other side, the options for monitoring are (a) measurements on physical properties from physical devices (e.g., CPU temperature); (b) measurements on logical properties of computational elements (in software e.g., request processing time; in hardware e.g., CPU load); and (c) measurements on external context conditions (e.g., user's profiles, weather conditions).

• Computed control actions. These are characterized, in the monitor, analyze, plan, execute, and knowledge (MAPE-K) loop context, and in particular by the nature of the output of the adaptation planner or controller [START_REF]An architectural blueprint for autonomic computing[END_REF]. These outputs affect the managed system to have the desired effect. The computed control actions can be (a) continuous signals that affect behavioral properties of the managed system; (b) discrete operations affecting the computing infrastructure executing the managed system (e.g., host system's buffer allocation and resizing operations; modification of process scheduling in the CPU); (c) discrete operations that affect the processes of the managed system directly (e.g., processes-level service invocation, process execution operations-halt/resume, sleep/respawn/priority modification of processes); and (d) discrete operations affecting the managed system's software architecture (e.g., managed system's architecture reconfiguration operations). The nature of these outputs is related to the extent of the intrusiveness of the adaptation mechanism with respect to the managed system and defines the extent of the adaptation mechanism with respect to exploiting the knowledge of either the structure or the behavior of the managed system in the adaptation process.

• System structure. Self-adaptive systems have two welldefined subsystems (although possibly un-distinguishable): (i) the adaptation controller and (ii) the managed system. One reason for analyzing controller and managed system structures is to identify whether a given approach implements the adaptation controller embedded with the managed system. Another reason is to identify the effect that the separation of concerns in these two subsystems has in the achievement of the adaptation goal. The analyzed approaches can be grouped into two sets: (i) those modeling the structure of the managed system to influence its behavior by modifying its structure; and (ii) those modeling the managed system's behavior to influence it directly. We consider the behavior model and the structure model as part of the system's structure. The identified options for the controller structure are variations of the MAPE-K loop with either behavioral or structural models of the managed system: (a) feedback control, that is, a MAPE-K structure with a fixed adaptation controller (e.g., a fixed set of transfer functions as a behavior model of the managed system); (b) adaptive control: a MAPE structure extended with managed system's reference and identification models of behavior (e.g., tunable parameters of controller for adaptive controllers-model reference adaptive control (MRAC) or model identification adaptive control (MIAC)); (c) reconfigurable control: MAPE-K structure with modifiable controller algorithm (e.g., rule-based software architecture reconfiguration controller). For the target system structure, the identified options are: (a) non-modifiable structure (e.g., monolithic system); and (b) modifiable structure with/without reflection capabilities (e.g., reconfigurable software components architecture). It is worth noting that not all options for system structure can be combined with any options for computed control actions. For instance, discrete operations affecting the computing infrastructure executing the managed system could be used to improve the performance of a monolithic system, whereas discrete operations affecting the managed system's software architecture would not make sense.

• Observable adaptation properties. By adaptation property we mean a quality (or characteristic) that is particular to a specific adaptation approach or mechanism. A quality can be a specific attribute value in a given state or a characteristic response to a known stimulus in a given context. Thus, observable adaptation properties are properties that can be identified and measured in the adaptation process. Given that we distinguish between the controller and the managed system in any self-adaptive system, we analyze observable adaptation properties also in both, the controller and the managed system. The identified observable properties in the controller are (a) stability; (b) accuracy; (c) settling-time; (d) small-overshoot; (e) robustness; (f) termination; (g) consistency (in the overall system structure and behavior); (h) scalability; and (i) security. For the managed system, the identified observable properties result from the adaptation process: (a) behavioral/functional invariants; (b) quality of service conditions, such as performance (latency, throughput, capacity); dependability (availability, reliability, maintainability, safety, confidentiality, integrity); security (confidentiality, integrity, availability); and safety (interaction complexity and coupling strength). Our proposed definitions for these properties are given in Sect. 4.2.

• Proposed evaluation. For the analyzed approaches, we used this element to identify the strategies proposed to evaluate themselves. Amongst the most used evaluation mechanisms are the execution of tests in real or simulated execution platforms, and the illustration with example scenarios.

• Identified metrics and key-performance indicators (KPIs). This element was identified from the analyzed approaches, the definition of metrics and KPIs that were used to measure the adaptation's variables of interest.

ANALYSIS OF SELF-ADAPTIVE APPROACHES

To validate our model for evaluating self-adaptive systems, we analyzed over 20 published approaches dealing with such systems. Of course, developing the model and analyzing the subject systems was an iterative process. The results of this analysis process are summarized in Tables 1 and2 below. Note that two of the dimensions (i.e., columns Adaptation Properties and Metrics in Table 1) are detailed in Sect. 4. Table 1 presents the characterization of selected adaptive approaches based on the evaluation model proposed in Sect. 2.

Average response time

Table 2 summarizes the characterization and classification of the studied self-adaptive systems.

Self-adaptive approaches range from pure control theory approaches to pure software engineering-based approaches with many hybrid approaches in-between. In control theorybased approaches, control actions are continuous signals that affect behavioral parameters of the managed system. The structure of the managed system in these approaches is generally non-modifiable while its behavior is modeled mathematically [START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF]. In contrast, software engineering-based approaches are characterized by implementing discrete control actions that affect the managed system's software architecture (i.e., the system structure). In these approaches the adaptation is supported by a model of the managed system's structure and reflection capabilities that allow the modification of the structure [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF][START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF][START_REF] Floch | Using architecture models for runtime adaptability[END_REF][START_REF] Garlan | Rainbow: Architecture-based self-adaptation with reusable infrastructure[END_REF][START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Léger | Reliable dynamic reconfigurations in a reflective component model[END_REF][START_REF] Mukhija | Runtime adaptation of applications through dynamic recomposition of components[END_REF][START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF][START_REF] Tamura | Qos contract-aware reconfiguration of component architectures using e-graphs[END_REF]. In hybrid adaptive systems, control actions are generally discrete operations that affect either the computing infrastructure executing the managed system or the set of processes comprising the managed system. Usually, the structure of the managed system is non-modifiable [START_REF] Baresi | Self-supervising BPEL processes[END_REF][START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF][START_REF] Cardellini | QoS-driven runtime adaptation of service oriented architectures[END_REF][START_REF] Ehrig | Formal analysis and verification of self-healing systems[END_REF][START_REF] White | Simplifying autonomic enterprise Java bean applications via model-driven development: A case study[END_REF]. It might be useful to classify hybrid apporaches further. For instance, we classified the approach proposed by Solomon et al. [START_REF] Solomon | Business process adaptation on a tracked simulation model[END_REF] between hybrid and software engineering-based approaches, given that their control actions affect the architecture of the managed system but the analyzer is based on a behavioral model of the managed system to decide when to adapt. These predictive mechanisms use control engineering techniques (i.e., Kalman Filters) to estimate adaptation parameters that require outputs that are not measurable on the actual managed system.

According to our proposed spectrum, most approaches were identified as software engineering-based and hybrid adaptive systems. With respect to the adaptation goal, we did not identify a relationship with our proposed spectrum. Thus, it is possible to address any of the adaptation goals along the entire spectrum. Concerning reference inputs, most approaches use contracts as the way to specify reference values for the adaptation goal and the corresponding measured outputs. All approaches that explicitly addressed monitoring monitor logical properties of computational elements (internal context), while two of them take into account the external context [START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Mukhija | Runtime adaptation of applications through dynamic recomposition of components[END_REF]. Regarding the controller structure, all approaches, except [START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF] and [START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF] that implement a simple feedback loop, implement either adaptive or reconfigurable control. Finally, the most common evaluation mechanism is the implementation of running examples based on simulated environments.

MEASURING ADAPTATION PROPERTIES

Our evaluation of a self-adaptive system has two aspects. The first one concerns the evaluation of desired properties for the managed system. In our analysis we focused only on desired properties that correspond to quality attributes of software systems. The second one relates to desired properties of the controller of the adaptation process.

In this section we present a set of properties and metrics useful to evaluate adaptation. For the identification of desired properties on the managed system, we based our analysis on the taxonomy of quality attributes for software systems proposed by SEI researchers [START_REF] Barbacci | Quality attributes[END_REF]. For properties re- lated to the controller, we based our analysis on the SASO properties identified by Hellerstein et al. in the application of control theory to computing systems [START_REF] Hellerstein | Feedback Control of Computing Systems[END_REF], and other properties identified in self-adaptive software systems surveys [START_REF] Ehrig | Formal analysis and verification of self-healing systems[END_REF][START_REF] Léger | Reliable dynamic reconfigurations in a reflective component model[END_REF][START_REF] Meng | On evaluating self-adaptive software[END_REF][START_REF] Mukhija | Runtime adaptation of applications through dynamic recomposition of components[END_REF]. Furthermore, we classified the identified adaptation properties according to how and where they are observed. Concerning how they are observed, some properties can be evaluated using static verification techniques while others require dynamic verification and run-time monitoring. We use the term observed as some properties are difficult to measure, despite the fact that controllers are designed to preserve them [START_REF] Tamura | Qos contract-aware reconfiguration of component architectures using e-graphs[END_REF]. With respect to where, properties can be evaluated on the managed system or the controller. On the one hand, some properties to evaluate the controller are observable on the controller itself or on both, the controller and the managed system; however, most properties can only be observed on the managed system. On the other hand, properties to evaluate the managed system are observable only on the managed system. In both cases, the environment that can affect the behavior of the controller or the managed system is also a factor worth considering.

Based on Sect. 2 and the analysis presented in Sect. 3, this section presents the foundations of a framework for the evaluation of self-adaptation, where properties observable on the managed system, either to evaluate the controller or the managed system, can be evaluated in terms of quality attributes. After analyzing several approaches on self-adaptive software systems and identifying adaptation properties, we propose a process for evaluating self-adaptation with which software engineers should be able to (i) identify required adaptation goals (i.e., the quality attributes that drive the adaptation of the managed system); (ii) identify adaptation properties to evaluate the controller, this must include the identification of properties that are observable on the controller, the managed system, or both; (iii) map quality attributes used to evaluate the managed system to properties that evaluate the controller but are observable on the managed system; and (iv) define metrics to evaluate properties observable on the managed system and the controller.

Quality Attributes as Adaptation Goals

If we intend to evaluate an adaptive software system we need to identify the motivation to build it-the adaptation goal. In general, adaptation can be motivated by the need of continued satisfaction of functional and regulation of nonfunctional requirements under changing context conditions. Nevertheless, as most analyzed contributions focus on nonfunctional factors, we based our analysis on software systems whose adaptation goals are motivated by quality characteristics of self-adaptive systems, such as self-configuring or self-optimizing, can be mapped to quality attributes. Following this idea, Salehie and Tahvildari discussed the relationships between autonomic characteristics and quality factors such as the relationship between selfhealing and reliability [START_REF] Salehie | Autonomic computing: emerging trends and open problems[END_REF].

Our main contribution is the application of quality attributes to evaluate self-adaptive software systems, as quality attributes are commonly used to evaluate desirable properties on the managed system. More importantly, we propose a mapping between quality factors and adaptation properties. This in fact introduces a level of indirection for evaluating adaptation properties that are not directly observable on the controller. The quality attributes that we analyzed in the approaches are the ones introduced in Sect. 2. In this subsection we present the definitions of the selected quality attributes, as well as the citations of the analyzed contributions whose adaptation goals are related to these quality attributes.

• Performance. Characterizes the timeliness of services delivered by the system. It refers to responsiveness, that means the time required for the system to respond to events or the event processing rate in an interval of time. Identified factors that affect performance are latency, the time the system takes to respond to a specific event [START_REF] Cardellini | QoS-driven runtime adaptation of service oriented architectures[END_REF][START_REF] Garlan | Rainbow: Architecture-based self-adaptation with reusable infrastructure[END_REF]; throughput, the number of events that can be completed in a given time interval-beyond processing rate as the desired throughput must also be observed in time sub-intervals [START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF][START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF][START_REF] White | Simplifying autonomic enterprise Java bean applications via model-driven development: A case study[END_REF]; and capacity, a measure of the amount of work the system can perform [START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF][START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF].

• Dependability. Defines the level of reliance that can justifiably be placed on the services the software system delivers. Adaptation goals related to dependability are availability, readiness for usage [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF][START_REF] Baresi | Self-supervising BPEL processes[END_REF][START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF][START_REF] Léger | Reliable dynamic reconfigurations in a reflective component model[END_REF][START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF][START_REF] White | Simplifying autonomic enterprise Java bean applications via model-driven development: A case study[END_REF]; reliability, continuity of service [START_REF] Baresi | Self-supervising BPEL processes[END_REF][START_REF] Ehrig | Formal analysis and verification of self-healing systems[END_REF][START_REF] Floch | Using architecture models for runtime adaptability[END_REF][START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Léger | Reliable dynamic reconfigurations in a reflective component model[END_REF][START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF]; maintainability, capacity to self-repair and evolve [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF][START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF][START_REF] Floch | Using architecture models for runtime adaptability[END_REF][START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF]; safety (from a dependability point of view), non-occurrence of catastrophic consequences from an external perspective (on the environment) [START_REF] Baresi | Self-supervising BPEL processes[END_REF]; confidentiality, freedom from unauthorized disclosure of information; integrity, non-improper alterations of the system structure, data and behavior [START_REF] Baresi | Self-supervising BPEL processes[END_REF].

• Security. The selected concerns of the security attribute are confidentiality, protection from disclosure; protection from unauthorized modification, integrity; and availability, protection from destruction [START_REF] Baresi | Self-supervising BPEL processes[END_REF].

• Safety. The level of reliance that can justifiably be placed on the software system as not generator of accidents. Safety is concerned with the occurrence of accidents, defined in terms of external consequences. The taxonomy presented in [START_REF] Barbacci | Quality attributes[END_REF] includes two properties of critical systems that can be used as indicators of system safety: interaction complexity and coupling strength. In particular, interaction complexity is the extent to which the behavior of one component can affect the behavior of other components. SEI's taxonomy presents detailed definitions and indicators for these two properties.

Adaptation Properties

An important part of our contribution is the identification of adaptation properties that have been used for the analyzed spectrum of adaptive systems, from control theory to software engineering, to evaluate the adaptation process. The identified adaptation properties are stated as follows. The first four properties, called SASO properties, correspond to desired properties of controllers from a control theory perspective [START_REF] Hellerstein | Feedback Control of Computing Systems[END_REF]; note that the stability property has been widely applied in adaptation control from a software engineering perspective. The remaining properties in the list were identified from hybrid approaches. Citations attached to each property refer either to papers where the property was defined or to examples of adaptive systems where the property is observed in the adaptation process.

• Stability. The degree in that the adaptation process will converge toward the control objective. An unstable adaptation will indefinitely repeat the process with the risk of not improving or even degrade the managed system to unacceptable or dangerous levels. In a stable system responses to a bounded input are bounded to a desirable range [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF][START_REF] Floch | Using architecture models for runtime adaptability[END_REF][START_REF] Lu | Performance specifications and metrics for adaptive real-time systems[END_REF][START_REF] Meng | On evaluating self-adaptive software[END_REF][START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF].

• Accuracy. This property is essential to ensure that adaptation goals are met, within given tolerances. Accuracy must be measured in terms of how close the managed system approximates to the desired state (e.g., reference input values for quality attributes) [START_REF] Cardellini | QoS-driven runtime adaptation of service oriented architectures[END_REF][START_REF] Solomon | Business process adaptation on a tracked simulation model[END_REF].

• Short settling time. The time required for the adaptive system to achieve the desired state. The settling time represents how fast the system adapts or reaches the desired state. Long settling times can bring the system to unstable states. This property is commonly referred to as recovery time, reaction time, or healing time [START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF][START_REF] Hellerstein | Feedback Control of Computing Systems[END_REF][START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Lu | Performance specifications and metrics for adaptive real-time systems[END_REF][START_REF] Meng | On evaluating self-adaptive software[END_REF].

• Small overshoot. The utilization of computational resources during the adaptation process. Managing resource overshoot is important to avoid the system instability. This property provides information about how well the adaptation performs under given conditionsthe amount of excess resources required to perform the adaptation [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF][START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF][START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF][START_REF] Lu | Performance specifications and metrics for adaptive real-time systems[END_REF][START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF].

• Robustness. The managed system must remain stable and guarantee accuracy, short settling time, and small overshoot even if the managed system state differs from the expected state in some measured way. The adaptation process is robust if the controller is able to operate within desired limits even under unforeseen conditions [START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF][START_REF] Meng | On evaluating self-adaptive software[END_REF].

• Termination (of the adaptation process). In software engineering approaches, the planner in the MAPE-K loop produces, for instance, discrete controlling actions to adapt the managed system (cf. Sect. 2), such as a list of component-based architecture operations. The termination property guarantees that this list is finite and its execution will finish, even if the system does not reach the desired state. Termination is related to deadlock freeness, meaning that, for instance, a reconfigurable adaptation process must avoid adaptation rules with deadlocks among them [START_REF] Ehrig | Formal analysis and verification of self-healing systems[END_REF][START_REF] Tamura | Qos contract-aware reconfiguration of component architectures using e-graphs[END_REF].

• Consistency. This property aims at ensuring the structural and behavioral integrity of the managed system after performing an adaptation process. For instance, when a controller bases the adaptation plan on dynamic reconfiguration of software architectures, consistency concerns are to guarantee sound interface bindings between components (e.g., component-based structural compliance) and to ensure that when a component is replaced dynamically by another one, the execution must continue without affecting the function of the removed component. These concerns help protect the application from reaching inconsistent states as a result of dynamic recomposition [START_REF] Mukhija | Runtime adaptation of applications through dynamic recomposition of components[END_REF]. Léger et al. define this property to complete the atomicity, consistency, isolation and durability (ACID) properties found in transactional systems that guarantee transactions are processed reliably [START_REF] Léger | Reliable dynamic reconfigurations in a reflective component model[END_REF] :

-Atomicity. Either the system is adapted and the adaptation process finishes successfully or it is not finished and the adaptation process aborts. If an adaptation process fails, the system is returned to a previous consistent state. -Isolation. Adaptation processes are executed as if they were independent. Results of unfinished adaptation processes are not visible to others until the process finishes. Results of aborted or failed adaptation processes are discarded. -Durability. The results of a finished adaptation process are permanent: once an adaptation process finishes successfully, the new system state is made persistent. In case of major failures (e.g., hardware failures), the system state can be recovered.

• Scalability. The capability of a controller to support increasing demands of work with sustained performance using additional computing resources. For instance, scalability is an important property for the controller when it must evaluate an increased number of conditions in the analysis of context. As computational efficiency is relevant for guaranteeing performance properties in the controller, scalable controllers are required to avoid the degradation of any of the operations of the adaptive process in any situation [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF][START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF][START_REF] Floch | Using architecture models for runtime adaptability[END_REF].

• Security. In a secure adaptation process, not only the target system but also the data and components shared with the controller are required to be protected from disclosure (confidentiality), modification (integrity), and destruction (availability) [START_REF] Barbacci | Quality attributes[END_REF].

Mapping Adaptation Properties and Quality Attributes

Once the adaptation goal and adaptation properties have been identified, the following step maps the properties of the controller, which are observable at the managed system, to the quality attributes of the managed system. Table 4 presents a general mapping between adaptation properties and quality attributes. These quality attributes refer to attributes of both the controller and the managed system depending on where the corresponding adaptation properties are observed.

According to Tables 3 and4, SASO properties, including stability, can be verified at run-time by observing performance, dependability and security factors in the managed system. Stability is one of the adaptation properties addressed in Océano, a dynamic resource allocation system that enables flexible SLAs in environments where peak loads are an order of magnitude greater than in the normal steady state. Quality attributes addressed in Océano concern dependability (i.e., availability and maintainability), and performance (i.e., throughput and capacity-scalability) [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF]. In the same way, the controller proposed by Parekh et al. to guarantee desirable performance levels (i.e., throughput and capacity) is also concerned with stability as adaptation property [START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF]. They apply an integral control technique to construct a transfer function that describes the system and the way the behavior of the managed system is affected by the controller. Baresi and Guinea propose a self-recovery system where service oriented architecture (SOA) business processes recover from disruptions of functional and non-functional requirements, to avoid catastrophic events (safety) and improper system state alterations (integrity), and to guarantee readiness for service (availability) and correctness of service (reliability) [START_REF] Baresi | Self-supervising BPEL processes[END_REF]. Accuracy is addressed by Cardellini et al. in the MOSES framework using adaptation policies in the form of directives to select the best implementation of the composite service according to a given scenario [START_REF] Cardellini | QoS-driven runtime adaptation of service oriented architectures[END_REF]. MOSES adapts chains of service compositions based on service selection using a multiple service strategy. It has been tested with some experiments to observe the behavior of the adaptation strategy in terms of its accuracy (i.e.,how close the managed system is to the adaptation goal). Multiple adaptation goals were used for these tests. The self-optimizing mechanism for business processes proposed by Solomon et al. also deals with accuracy as an adaptation property [START_REF] Solomon | Business process adaptation on a tracked simulation model[END_REF]. Their approach is based on a simulation model to anticipate performance levels and make decisions about the adaptation process. They developed a tuning algorithm to keep the simulation model accurate. The algorithm compensates for the measurement of actual service time to increase the accuracy of simulations by modeling errors, probabilities and inter-arrival times. Then it obtains the best estimate for these data such that a square root error between the simulated and measured metrics is minimized.

Settling time and small overshoot are addressed as adaptation properties in Océano [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF], the framework for developing autonomic Enterprise Java Bean (EJB) applications proposed by White et al. [START_REF] White | Simplifying autonomic enterprise Java bean applications via model-driven development: A case study[END_REF], and the self-recovery approach based on microreboots proposed by Candea et al. [START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF]. In Océano settling time is measured in terms of the time required for deploying a new processing node including the installation and reconfiguration of all applications and data repositories. In White's framework, settling time is evaluated in terms of the average response time required for autonomic EJBs to adapt. Finally, Candea's approach applies a recursive strategy that reduces mean time to repair (MTTR) by means of recovering minimal subsets of a failed system's components. If localized, minimal recovery is not enough, progressively larger subsets are recovered. In the controlbased approach, to ensure SLOs proposed by Parekh et al., small overshoot is addressed by avoiding that control values (e.g., MAXUSERS) are set to values that exceed their legal range. Root-locus analysis is used to predict the valid values of the maximum number of users. They divided the valid range into three regions to decide when control values reach undesirable levels. Based on empirical studies, they analyzed properties of the transfer function to predict the desired range of values [START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF].

One of the addressed adaptation properties in the selfmanagement approach for balancing system load proposed by Dowling and Cahill is robustness [START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF]. They aim to realize a robust controller by implementing adaptation an mechanism via decentralized agents that eliminate centralized points of failure.

Termination can be verified using static mechanisms. Ehrig et al. proposed a self-healing mechanism for a traffic light system to guarantee continuity of service (reliability) by selfrecovering from predicted failures (integrity) [START_REF] Ehrig | Formal analysis and verification of self-healing systems[END_REF]. In their adaptive solution, termination is addressed by ensuring deadlock-freeness in the managed system by statically checking the self-healing rules in such a way that the self-repairing mechanism never inter-blocks traffic lights in the same road intersection.

Scalability is also an adaptive property in K-Components, the agent-based self-managing system proposed by Downling and Cahill [START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF]. Scalability is addressed by evolving the selfmanagement local rules of the agents. Another approach where scalability is addressed as an adaptation property is Madam, the middleware proposed by Floch et al. for enabling model-based adaptation in mobile applications [START_REF] Floch | Using architecture models for runtime adaptability[END_REF]. Scalability is a concern in Madam for several reasons. First, its reasoning approach might result in a combinatorial explosion if all possible variants are evaluated; second, the performance of the system might be affected when reasoning on a set of a concurrently running applications competing for the same set of resources. They proposed a controller where each component (e.g., the adaptation manager) can be replaced at run-time to experiment with different analysis approaches for managing scalability.

Security was not addressed as an adaptation property by any of the self-adaptive systems analyzed. However, we propose the use of SEI's definition of security as a quality attribute and its corresponding quality factors to evaluate security on the controller [START_REF] Barbacci | Quality attributes[END_REF]. As presented in Table 3, security of the controller should be evaluated independently of the managed system. This means that ensuring security at the managed system does not guarantee security with respect to the adaptation mechanism.

Adaptation Metrics

Adaptation metrics provide the way of evaluating adap-tive systems with respect to particular concerns of the adaptation process [START_REF] Meng | On evaluating self-adaptive software[END_REF][START_REF] Reinecke | Evaluating the adaptivity of computing systems[END_REF]. Thus, metrics provide a measure to evaluate desirable properties. For instance, metrics to evaluate control systems measure aspects concerning the SASO properties (i.e., stability, accuracy, settling time, and small overshoot).

To characterize the evaluation of adaptive systems, we analyzed the variety of self-adaptive software systems to identify adaptation properties (i.e., at the managed system and the controller) that were evaluated in terms of quality attributes (cf. Sects. 3 and 4.1). Just as the evaluation of most properties is impossible by observing the controller itself, we propose the evaluation of these properties by means of observing quality attributes at the managed system. To identify relevant metrics, we characterized a set of factors that affect the evaluation of quality attributes such as speed, memory usage, response time, processing rate, mean time to failure, and mean time to repair [START_REF] Lu | Performance specifications and metrics for adaptive real-time systems[END_REF][START_REF] Barbacci | Quality attributes[END_REF]. These factors are an essential part of the metrics used to evaluate properties of both the controller and properties of the managed system [START_REF] Reinecke | Evaluating the adaptivity of computing systems[END_REF].

The evaluation of MOSES, the QoS-driven framework proposed by Cardellini et al. to adapt service-oriented business processes, is based on the following metrics to measure performance and reliability. Expected response time (Ru), the average time needed to fulfill a request for a composite service; expected execution cost (Cu), the average price to be paid for a user invocation of the composite service; and expected reliability (Du), the logarithm of the probability that the composite service completes its task for a user request [START_REF] Cardellini | QoS-driven runtime adaptation of service oriented architectures[END_REF].

For Océano, the following metrics were defined to evaluate dependability factors (e.g., availability) and performance factors (e.g., scalability in terms of throughput and capacity) [START_REF] Appleby | Océano -SLA based management of a computing utility[END_REF]: active connections per server-the average number of active connections per normalized server across a domain; overall response time-the average time it takes for any request to a given domain to be processed; output bandwidth-the average number of outbound bytes per second per normalized server for a given domain; data base response time-average time it takes for any request to a given domain to be processed by the back-end data base; throttle rate (T )-a percentage of connections disallowed to pass through Océano on a customer domain; admission rate-the complement of the domain throttle rate (1 -T ); and active servers-the number of active normalized-servers which service a given customer domain.

Average response time is a common metric used to evaluate performance in several adaptive approaches, such as the framework to develop autonomic EJB applications proposed by White et al. [START_REF] White | Simplifying autonomic enterprise Java bean applications via model-driven development: A case study[END_REF]. In K-Components, the self-adaptive component model that enables the adaptation of software components to optimize system performance, a load balancing function on every adaptation contract uses a cost function to calculate its internal load cost and the ability of its neighbors to handle the load [START_REF] Dowling | Self-managed decentralised systems using k-components and collaborative reinforcement learning[END_REF]. This cost function is defined as the addition of the advertised load cost and internal cost of the component (i.e., calculated as the estimated cost to handle a particular load type).

In the control-based approach proposed by Parekh et al., to achieve performance service level objectives the length of the queue of the in-progress client requests is the metric defined as control offered load, the load imposed on the server by client requests [START_REF] Parekh | Using control theory to achieve service level objectives in performance management[END_REF]. Baresi and Guinea proposed a metric to control reliability on the adaptation of business processes based on BPEL [START_REF] Baresi | Self-supervising BPEL processes[END_REF]. In their approach, reliability is calculated as the number of times a specific method responds to within two minutes over the total number of invocations. They also defined a KPI based on this metric such that reliability must be greater that 95% over the past two hours of operation.

In the adaptive middleware proposed by Kumar et al., a business value KPI is defined in terms of factors, such as the priority of the user accessing the information, the time of day the information is being accessed, and other aspects that determine how critical the information is to the enterprise [START_REF] Kumar | Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows[END_REF]. For this, they defined a utility function as a combination of some of these factors: utility(e gj-k ) = f ( ∑ dni, min(bni), b gj-k ), where i|eni ∈ M (e gj-k ). The business utility of each edge (e gj-k ), which represents data streams between operators that perform data transformations, is a function of the delay dni, the available bandwidth bni of the intervening network edges eni, and the required bandwidth b gj-k of the edge e gj-k .

In the self-healing approach based on recursive microreboots proposed by Candea et al., availability is evaluated in terms of mean time to recover (M T T R) [START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF]. To evaluate the availability of the system they defined two metrics,

availability (A = M T T F/(M T T F + M T T R)) and downtime of unavailability (U = M T T R/(M T T F + M T T R)),

where M T T F is the mean time for a system or subsystem to fail (i.e., the reciprocal of reliability), M T T R is the mean time to recover, and A is a number between 0 and 1. U can be approximated to M T T R/M T T F when M T T F is much larger than M T T R. Similarly, Sicard et al. define a metric for availability in terms of M T T R [START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF].

ResponseF requency/T imeU nit

Reboots [START_REF] Candea | Improving availability with recursive microreboots: a soft-state system case study[END_REF] A=MTTF/(MTTF+MTTR) U=MTTR/(MTTF+MTTR) Sicard et al. [START_REF] Sicard | Using components for architecture-based management: the self-repair case[END_REF] A=MTTR Table 5 summarizes our identified metrics to assess selfadaptive systems. Although these metrics are directly re-lated to the measurement of quality factors, we expect that these metrics will be useful for evaluating adaptation properties based on our proposed mapping between quality attributes and adaptation properties (cf. Sect. 4). The approach by Reinecke et al. [START_REF] Reinecke | Evaluating the adaptivity of computing systems[END_REF] supports our hypothesis. Their metric measures the ability of a self-adaptive system to adapt. They argue that adaptivity can be evaluated using a metametric named payoff which is defined in terms of performance metrics to measure the effectiveness of the adaptation process. That is, the optimal adaptive system is characterized by the fact that its adaptation decisions are always optimal (i.e., always yield the optimal payoff). To apply their metric it is necessary to (i) identify the adaptation tasks, (ii) define one or more performance metrics on these tasks (i.e., these metrics should reflect the contribution of these tasks toward the adaptation goal), (iii) define a payoff metric in terms of the performance metrics, and (iv) to apply the metric by observing the performance of the system.

DISCUSSION

We started the analysis phase for this work with 34 research papers with different proposals for self-adaptive software systems published over the past decade. From this set, 18 were filtered-out mainly because either they presented very generic proposals (i.e., with non-measurable selfadaptive properties) or they did not include enough information in the paper for characterization purposes.

From the analysis of the 16 remaining papers that we presented in the previous sections (and even considering the 18 papers that we filtered out), it is worth noting prevalent difficulty-or the lack of awareness of-to identify metrics to evaluate self-adaptive software. Nonetheless, some advances have been made based on concepts from control theory and the recognized importance of quality metrics and corresponding measures as the basis for understanding and improving processes. However, as we can conclude from our analysis, there are plenty of opportunities and challenges to be addressed, even when we consider concepts more abstract than metrics, such as the properties of self-adaptation. In the following we outline some of these opportunities and challenges.

First, most of the proposals focus only on self-adaptation mechanisms, not addressing explicitly the level of achievement of adaptation properties nor the adaptation properties themselves. On the one hand, it is known that even though control theory has defined standardized properties that a controller must realize (i.e., the SASO properties), self-adaptive software systems require additional properties, due to their discrete nature (e.g., termination). On the other hand, it is clear from the discussion in previous sections that quality attributes are a plausible option to evaluate some adaptation properties. However, it will be necessary (i) to evaluate if our proposed set of self-adaptation properties defined in Sect. 4.2 is general enough for self-adaptation mechanisms; (ii) to find standardized metrics to measure self-adaptation properties, which could be based on proposals such as the one by Reinecke et al. [START_REF] Reinecke | Evaluating the adaptivity of computing systems[END_REF]; and (iii) to analyze whether the proposal of measuring adaptation properties based on quality attributes is meaningful enough, and if they fulfill the conceptual definitions of corresponding properties of, for instance, control theory.

Second, the lack of awareness of adaptation properties as a goal to be measured results in a lack of evaluation meth-ods and metrics for these properties and for the adaptation mechanisms themselves. However, this trend could be reversed by designing self-adaptive mechanisms with implicit controllable and measurable properties. For some verifiable properties, this can be obtained, for instance, by developing new or using existing formal models as a basis for the selfadaptation process. For adaptation mechanisms with measurable properties, one main challenge is to develop mathematical behavior models based on the architecture itself of the target computing systems to be controlled.

Third, without declared evaluation methods and metrics it is very difficult to compare and to reason about the engineering of self-adaptation; for instance, from our analysis it resulted not possible to identify any measurable relationship between the adaptation goals and the evaluation of the adaptation strategies as such. From the structure point of view, it is clear that decoupling the controller from the managed system, with respect to evaluation, is a first critical step toward to be able to reason and control the properties of dynamic self-adaptation. However, several questions remain: does the system structure (i.e., controller and managed system) have any relationship with the quality of the adaptation approach? Do non-explicit controllers imply undefined adaptation properties? From the behavior point of view and considering the adaptation mechanism as a blackbox, how do we compare managed system behavior in the different phases of the adaptation process? Maybe in terms of stability and other properties such as settling time. Under which circumstances and characteristics of the managed system is an adaptation mechanism better than another? There is no available evaluation framework to compare adaptation mechanisms to help answer these questions.

CONCLUSIONS

Self-adaptive software evaluates and modifies its behavior to preserve the satisfaction of its functional requirements and regulation of non-functional requirements, under changing context conditions of execution. Researchers devised and proposed many diverse approaches and strategies to modify the behavior of a managed system. In this paper, as a result of our analysis, we proposed a classification of self-adaptive systems spanning the spectrum from control-based to software engineering-based approaches.

Many studied approaches did not identify nor address adaptation properties. Thus, the evaluation of adaptive systems is generally not addressed explicitly -neither in the controller nor in the managed system. Consequently, since adaptation properties are not identified in many approaches, metrics are not addressed either. Thus, validation mechanisms discussed for these approaches are usually limited to the evaluation of performance properties observed in the managed system, even when the adaptation goal is not related to performance quality attributes.

Future work will focus on the validation of the adaptation properties and their mapping to quality attributes as proposed in this paper through evaluation of existing adaptive systems.

Table 1 :

 1 Applying the characterization model to selected adaptive approaches

		JBeans [26]	Autonomic	White et al.		SCeSAME [24]	Tamura et al.		al. [23]	Solomon et			Sicard et al. [22]		al. [18]	Parekh et		CASA [17]	Glinz	Mukhija and				Léger et al. [14]			MWare [13]	Kumar et al.		Rainbow [9]	Garlan et al.		MADAM [8]	Floch et al.				Ehrig et al. [7]	nents [6]	Cahill KCompo-	Dowling and		MOSES [5]	Cardellini et al.				Candea et al. [4]			Guinea [3]	Baresi and		Océano [1]	Appleby et al.	Approach
	Self-protection	Self-optimization	Self-healing	Self-management			QoS preservation			Self-optimization		self-healing	Self-management			QoS preservation			Self-configuration	QoS preservation			Self-configuration	QoS preservation		Self-optimization	Self-configuration	Self-management			Self-repairing		Self-configuration	QoS preservation				Self-healing			Self-management			QoS preservation				Self-recovery				Self-recovery		Self-optimization	Self-management	Adaptation Goal
			QoS	SLOs-Contracts:		QoS	SLOs-Contracts:			Contracts: QoS	states	ing computational	Constraints defin-		value	Single reference				Contracts: QoS		states	ing computational	Constraints defin-			policy actions	Contracts: QoS;		SLOs-QoS	Contracts: SLAs-		SLOs-QoS	Contracts: SLAs-				Goal actions		QoS	Contracts: SLOs-			Contracts: QoS			QoS	Contracts: SLOs-	sions)	and regular expres-	funct. req. (logical	Contracts: SLAs,		Contracts: SLAs	Reference Inputs
		elements	of computational	SLOs/Logical properties	elements	of computational	SLOs/Logical properties	elements	of computational	SLOs/Logical properties		phase	Not explicit monitoring	elements	of computational	SLOs/Logical properties	context	elements, external	of computational	SLOs/Logical properties	computational elements	properties of	conditions/Logical	Malfunction	context	elements, external	of computational	SLOs/Logical properties	elements	of computational	SLOs/Logical properties	elements	of computational	SLOs/Logical properties	elem.	properties of comput.	conditions/Logic.	Malfunction	elements	of computational	SLOs/Logical properties	elements	of computational	SLOs/Logical properties	computational elements	properties of	conditions/Logical	Malfunction		elements	of computational	SLOs/Logical properties		elements	of computational	SLOs/Logical properties	Measured Outputs
		the managed system	affecting the process of	Discr. operations	system's soft. arch.	affecting the managed	Discr. operations	system's soft. arch.	affecting the managed	Discr. operations	system's soft. arch.	affecting the managed	Discr. operations	properties	affecting behavioral	Continuous signals		system's soft. arch.	affecting the managed	Discr. operations		system's soft. arch.	affecting the managed	Discr. operations		system's soft. arch.	affecting the managed	Discr. operations	system's soft. arch.	affecting the managed	Discr. operations	system's soft. arch.	affecting the managed	Discr. operations		the managed system	affecting the process of	Discr. operations	system's soft. arch.	affecting the managed	Discr. operations	the managed system	affecting the process of	Discr. operations		the managed system	affecting the process of	Discr. operations		the managed system	affecting the process of	Discr. operations		infrastructure	affecting computational	Discr. operations	Control Actions
		structure	modifiable	Adaptive control/Non-	structure (reflection)	control/Modifiable	Reconfigurable	structure (reflection)	control/Modifiable	Adaptive	structure (reflection)	control/Modifiable	Feedback	(mathematical model)	modifiable structure	Feedback control/Non-		structure (reflection)	control/Modifiable	Reconfigurable		structure (reflection)	control/Modifiable	Reconfigurable		structure (reflection)	control/Modifiable	Adaptive	structure (reflection)	control/Modifiable	Adaptive	structure (reflection)	control/Modifiable	Adaptive		structure	modifiable	Adaptive control/Non-	structure (reflection)	control/Modifiable	Reconfigurable	structure (reflection)	control/Modifiable	Reconfigurable		structure (reflection)	control/Modifiable	Adapt.		structure	modifiable	Adaptive control/Non-		structure (reflection)	control/Modifiable	Adaptive	System Structure
		Dependability (availab.)	Performance (throughput),	Settling time/QoS:	(contract)	consistency/QoS properties	Termination,			Accuracy/QoS: Performance		(reliab., availab.)	None/QoS: Dependability	(throughput, capacity)	overshoot/QoS: Performance	Stability, small			Performance	Consistency/QoS:		(availab., reliab.)	durab.)/QoS: Dependability	Consistency (atom., isol.,		(throughput, capacity)	overshoot/QoS:Performance	Settling time, small		(latency)	None/QoS: Performance	(reliab., maintainab.)	Performance, Dependability	Scalability/QoS:			Dependability (reliab.)	Termination/QoS:	(throughput, capacity)	scalability/QoS:Performance	Robustness,	(reliab., cost).	(latency), Dependability	Accuracy/QoS:Performance		(availab.)	time/QoS:Dependability	Small overshoot, settling		integrity, availab., reliab.)	QoS:Dependability (safety,	None/Behavioral,		performance (scalability)	(availab., maintainab.),	scalability/QoS:Dependability	overshoot and	Stability, settling time, small	Adaptation Properties
		effort savings	to probe development	Empirical experiments	Running example	probed with theorems.	Formal properties		test accuracy	Running example to	performance	to evaluate	Simulated experiments		Lotus Notes	Running examples on			evaluate performance	Running examples to			evaluate performance	Running examples to			real scenarios	Execution tests on		evaluate effectiveness	Running examples to	evaluate scalability	environment to	Simulated		example	properties. Running	Formal verification of			None		of the adapt. strategy	Based on the accuracy				Recovery-based tests			reliability-based tests	Functional and		the adaptation process	to test performance of	Based on settling time	Evaluation
							None			None		Table 5)	Availability (cf.			Offered load				None				None		Table 5)	function (cf.	Business utility			None			None				None		components	Load cost of		cost, reliability	resp. time, exec.		Table 5)	downtime (cf.	Availability,			(last two hours)	Reliability>=0.95	rate, active servers	rate, admission	bandwidth, throttle	time, output	per server,response	Active connections	Metrics

Table 2 :

 2 Characterization Summary

	Characteristic	Count [List of Approaches]
	Spectrum Classification
	Control Engineering	1 [18]
	Hybrid	5 [3, 4, 5, 7, 26]
	Hybrid-Software	1 [23]
	Software Engineering	9 [1, 6, 8, 9, 13, 14, 17, 22, 24]
	Adaptation Goal Specification
	Contract-based	14
	Monitoring Mechanisms
	Monitor internal context	15
	Monitor external context	2 [13, 17]
	Non specified	1 [22]
	Controller's Structure
	Feedback control	2 [18, 22]
	Adaptive control	9 [1, 3, 4, 7, 8, 9, 13, 23, 26]
	Reconfigurable Control	4 [5, 6, 14, 17, 24]
	Managed System's Structure
	Non-modifiable	4 [3, 7, 18, 26]
	Modifiable with reflection	12 [1, 4, 5, 6, 8, 9, 13, 14, 17, 22,
		23, 24]
	Adaptation Properties
	Settling time	4 [1, 4, 13, 26]
	Small overshoot	4 [1, 4, 13, 18]
	Scalability	3 [1, 6, 8]
	Stability	2 [1, 18]
	Accuracy	2 [5, 23]
	Termination	2 [7, 24]
	Consistency	3 [14, 17, 24]
	Robustness	1 [6]
	Security	0
	Quality Attributes
	Performance	10 [1, 5, 6, 8, 9, 13, 17, 18, 23,
		26]
	Dependability	7 [1, 3, 4, 5, 7, 8, 26]

Table 3 :

 3 Classification of adaptation properties according to how and where they are observed.

		Property	Where the
	Adaptation	verification	property is
	properties	mechanism	observed
	Stability	Dynamic	Managed system
	Accuracy	Dynamic	Managed system
	Settling Time	Dynamic	Managed system
	Small Overshoot	Dynamic	Managed system
	Robustness	Dynamic	Controller
	Termination	Both	Controller
	Consistency	Static	Managed system
	Scalability	Dynamic	Controller
	Security	Dynamic	Controller

Table 4 :

 4 Mapping adaptation properties to quality attributes

	Adaptation Properties	Quality Attributes
			Latency
		Performance	Throughput
	Stability	Dependability	Capacity Safety Integrity
		Security	Integrity
			Latency
	Accuracy	Performance	Throughput
			Capacity
	Settling Time	Performance	Latency Throughput
			Latency
	Small Overshoot	Performance	Throughput
			Capacity
	Robustness	Dependability	Availability Reliability
		Safety	Interact. Complex. Coupling Strength
	Termination	Dependability	Reliability Integrity
	Consistency	Dependability	Maintainability Integrity
			Latency
	Scalability	Performance	Throughput
			Capacity
			Confidentiality
	Security	Security	Integrity
			Availability

Table 5 :

 5 Metrics to evaluate quality attributes of a subject self-adaptive system being analyzed

	Quality	Approach	Metrics
	Attributes
		MOSES [5]	Response Time (Ru) Execution Cost (Cu)
				Response Time
	Performance	Océano [1]	Output Bandwidth Throttle Rate Admission Rate
		EJB	Frame-	Response Time
		work [26]
		K-Comp. [6]	Load Cost
		Parekh	et	Offered Load
		al. [18] Kumar	et	f (	∑	dni, min(bni), b gj-k )
		al. [13]
		MOSES [5]	Expected Reliability (Du)
	Dependability	Océano [1]	Active Connections per Server Active Servers
		Baresi	and
		Guinea [3]

Acknowledgments

This work was funded in part by the National Sciences and Engineering Research Council (NSERC) of Canada, IBM Corporation, CA Inc., Icesi University (Cali, Colombia), and Ministry of Higher Education and Research of Nord-Pas de Calais Regional Council and FEDER under Contrat de Projets Etat Region (CPER) 2007-2013.


G Chohan

C Pennington

J M Mackenzie

M Andrews

D Everington

R G Will

Rsg Knight

Dr Aje Green

The role of cerebrospinal fluid 14-3-3 and other proteins in the diagnosis of sporadic Creutzfeldt-Jakob disease in the United Kingdom: a 10 year review

Keywords: Cerebrospinal fluid, Creutzfeldt-Jakob disease, 14-3-3, S100b, tau protein

It is 10 years since the detection of cerebrospinal fluid (CSF) 14-3-3 was included in the diagnostic criteria for sporadic Creutzfeldt-Jakob disease (sCJD) by the World Health Organisation. Since that time other CSF proteins, such as S100b and tau protein have been proposed as surrogate markers for sCJD. We aimed to investigate the diagnostic value of each of these three proteins. CSF samples collected from patients who were referred to the National CJD Surveillance Unit as suspected cases of sCJD during the period 1997 -2007 were analysed for 14-3-3, S100b and tau protein. The sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of each of these markers either alone or in combination for the diagnosis of sCJD were assessed. The impact of CSF 14-3-3 analysis on the case classification of sCJD was investigated. CSF 14-3-3 had the greatest sensitivity (86%) when compared to tau protein (81%) and S100b (65%). The combination of a positive CSF 14-3-3 or an elevated tau protein with a raised S100b had the highest positive predictive power for sCJD. During the study period 100 patients were classified as probable sCJD solely on the basis of the clinical features and a positive CSF 14-3-3. The most sensitive marker for sCJD was a positive CSF 14-3-3. The analysis of CSF 14-3-3 plays a crucial role in the case classification of sCJD.

INTRODUCTION

Creutzfeldt-Jakob disease belongs to a family of fatal neurodegenerative diseases collectively known as human transmissible spongiform encephalopathies. Sporadic CJD (sCJD) remains the most prevalent form worldwide with an annual incidence of 1.0-1.5/million/year. [START_REF] Ladogana | Mortality from Creutzfledt-Jakob disease and related disorders in Europe, Australia and Canada[END_REF] Human spongiform encephalopathies are characterised by the accumulation of pathological prion protein (PrP) in the central nervous tissue. The pathological isoform is termed PrP Sc and this differs from the normal cellular isoform by its high content of ß-sheet structure and partial resistance to protease digestion. [2] Additional histological changes identified include spongiosis, neuronal loss and gliosis.

Neuropathological studies remain the only means of obtaining a definitive diagnosis; the initial diagnosis however is still dependent on the clinical phenotype, as defined by the WHO criteria. [START_REF]Report of a WHO consultation on global surveillance, diagnosis and therapy of human transmissible spongiform encephalopathies[END_REF]. Marked phenotypic heterogeneity is well documented in all human prion diseases and in sCJD this observation is yet to be explained. Early experimental transmission studies on primates, [START_REF] Gibbs | Creutzfeldt-Jakob disease: transmission to the chimpanzee[END_REF] subsequent epidemiological studies [START_REF] Will | Descriptive epidemiology of Creutzfeldt-Jakob disease in six European countries, 1993-1995[END_REF][START_REF] Brown | Creutzfeldt-Jakob disease in France: Epidemiological study of 170 patients dying during the decade 1968-1977[END_REF] and isolated case reports have allowed diagnostic criteria to be refined over the years, however early diagnosis and in turn accurate surveillance remains a challenge. Atypical forms of sCJD are well recognized, it has been postulated that such variation is partially dependent on, or associated with, genetic and molecular factors, PRNP codon 129 genotype and prion protein isotype respectively. The typical short duration disease phenotype being linked to methionine homozygosity and PrP Sc type 1. [2] The atypical and rarer variants (those defined as having a long duration of illness, young age of onset and unusual clinical or pathological features) are linked to valine homozygosity or heterozygosity at codon 129 and PrP Sc type 2.

The advent of novel diagnostic tests, specifically the cerebrospinal fluid (CSF) 14-3-3 protein, has also allowed improvement in classification over the last decade. The 14-3-3 has better diagnostic utility than investigative tests such as the EEG. [START_REF] Zerr | Analysis of EEG and CSF 14-3-3 proteins as aids to the diagnosis of Creutzfeldt-Jakob disease[END_REF][START_REF] Collins | Determinants of diagnostic investigation sensitivities across the clinical spectrum of sporadic Creutzfeldt-Jakob disease[END_REF] However the sensitivity of CSF 14-3-3 has been shown to vary, partially dependent on the genetic and molecular influences described above.[9] The detection of CSF 14-3-3 remains a supportive tool in the appropriate clinical context but as a solitary test, independent of clinical phenotype, has little value.

The major differential diagnoses of sCJD remain those of other irreversible neurodegenerative conditions; however a small proportion of patients may have a potentially treatable condition. Therefore CSF analyses, such as cell count and total protein, are an important early investigation in these patients and CSF 14-3-3 is often performed at this time. Many conditions associated with acute neuronal damage may result in a positive CSF 14-3-3 and thereby reduce the specificity of CSF 14-3-3 for sCJD. [START_REF] Collins | Determinants of diagnostic investigation sensitivities across the clinical spectrum of sporadic Creutzfeldt-Jakob disease[END_REF] Therefore other brain-specific proteins in the CSF may be of value as diagnostic markers. These additional markers include the CSF astrocytic marker S100b and the neuronal marker tau, protein in isolation or in combination.

In this study we aim to firstly review the sensitivity and specificity of each CSF protein in sCJD and also review the potential role of a combination of several markers to improve sensitivity and specificity in the clinical diagnosis of sCJD. Secondly, the impact of the CSF 14-3-3 on UK surveillance is of great importance and is therefore considered, specifically reviewing the number of cases of probable sCJD classified on the basis of CSF 14-3-3. In addition those cases classified as a probable case of sCJD on the basis of a positive 14-3-3 but found to have an alternative pathological diagnosis are investigated.

The additional role of combining neuronal markers in order to potentially exclude sCJD is reviewed.

MATERIALS AND METHODS

Patients

The National CJD Surveillance Unit (NCJDSU) was established in May 1990 to prospectively identify and record all suspected cases of sporadic CJD in the UK. The primary aim of the programme was to detect any change in the epidemiology of the disease that might be attributable to bovine spongiform encephalopathy, and a distinct clinical-pathological phenotype was described in 1996 (variant CJD).

CSF protein analysis

CSF samples are sent to the laboratory on dry ice and stored at -80°C prior to analysis.

For this study CSF 14-3-3, S100b and tau protein were analysed. Protein 14-3-3 in CSF was detected by western blotting after SDS-polyacrylamide gel electrophoresis (SDS-PAGE) with chemiluminescent visualization.[11-13] A positive, negative and two weak positive 14-3-3 controls were included on each run. These were controls from patients with neuropathologically confirmed sporadic CJD (positive control) or from patients with either a clinical or pathological diagnosis of an alternative disease (negative and weak positive controls). The relative immunoreactivity of positive, negative and weak positive 14-3-3 CSF samples is given in fig 1. The blots were independently assessed by two people (AJEG, MA, GC or CP) and only positive CSF 14-3-3 results were used for case classification. CSF S100b was measured using a previously reported sandwich enzyme linked immunosorbant assay (ELISA).

[11] A concentration of <0.5ng/ml was considered to be normal, whilst a concentration of >1.0ng/ml was considered to be diagnostic. CSF tau protein was measured using an enzyme immunoassay (Innotest hTAU-Ag, Innogenetics, Ghent, Belgium), according to the manufacturer's recommendations. A concentration of >1260pg/ml was considered to be diagnostic. [14] This assay measures total tau protein concentrations and as such measures both normally phosphorylated tau and hyper-phosphorylated tau. Some CSF samples had insufficient volume for all three analytes to be measured.

PRNP codon 129 genotype and PrP isotyping

Prion protein isotyping was performed on all suspected cases of prion disease where fresh brain tissue was received by the NCJDSU. Small quantities of cerebral cortex were homogenized, treated with proteases and the size and abundance of the three PrP Sc glycoforms was determined by Western blot analysis. [15] Genotyping for polymorphism at codon 129 of the PRNP gene was carried out on all available blood specimens. DNA was extracted from blood using standard techniques and analysed using the Helsinki method.

[16]

Statistical analysis

Comparison of age by Mann Whitney tests was carried out. Descriptive statistics were calculated for the sCJD and control patients. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and efficiency of each marker and for combinations of markers were obtained. To investigate the influence of stage of disease on the sensitivity of CSF 14-3-3 and tau protein, we divided the disease duration for each patient into equal thirds. The stage in which the LP was performed was noted and the sensitivity of each marker for sCJD in each of the three stages was calculated.

RESULTS

The sensitivity, specificity, PPV, NPV and efficiency of each neuronal marker for the diagnosis of neuropathologically confirmed sCJD cases are shown in Table 2. CSF 14-3-3 is the most sensitive marker and has a higher sensitivity than CSF tau, 86% and 81% respectively. CSF S100b does not have adequate sensitivity (65%) to be used as an isolated marker of sCJD. The difference in sensitivity between CSF 14-3-3 and tau protein is not influenced by the different numbers of cases investigated for each analyte.

The sensitivity of CSF 14-3-3 and tau protein calculated using those samples where both analytes were measured is 85% and 81% respectively. The specificity of CSF 14-3-3 and tau protein in those samples where both analytes were measured is 74% and 85% respectively. The combination of a positive CSF 14-3-3 with either an elevated S100b or an elevated tau protein increased the PPV of 14-3-3 from 83% to 94% or 91%, respectively (Table 2 and fig 2a,b). Likewise the combination of an elevated tau protein and an elevated S100b resulted in a higher PPV (95%) for sCJD when compared to using each marker alone (Table 2, fig 2c). However combining the tests either in pairs or taking all three markers together resulted in a reduction in sensitivity and a reduction in the efficiency of each marker. That is to say the ability of each marker to distinguish between those patients with sCJD and those that present with symptoms similar to sCJD but turn out to have an alternative diagnosis is reduced if they are combined (Table 2).

In the atypical subgroups, although the numbers are small, CSF 14-3-3 is more sensitive than an elevated tau (Table 3). This finding is particularly noticeable in those patients who are 50 years or less at the onset of the disease. However, in those sCJD cases that were negative for either CSF 14-3-3 or tau protein there did not seem to be a difference in the demographics of the patients or any obvious effect of codon 129 and PrP Sc isotype. In the 14-3-3 negative group there were 35 sCJD cases (27-81 years (mean 62.0 ± 10.9 years) at notification) and in the tau negative group there were 41 sCJD cases (44-81 years (mean 63.3 ± 9.9 years) at notification). The disease duration was also comparable 2-54 months (mean 15.1 ± 11.7 months) and 2-54 months (mean 14.4 ± 12.2 months)

respectively. These disease durations are much longer than those in sCJD cases who are positive for CSF 14-3-3 (mean 6.6 ± 6.8 months) or CSF tau protein (mean 6.5 ± 6.6 months). The effect of stage of disease on the sensitivity of CSF 14-3-3 and tau protein is shown in Table 4. The time of the LP and the date of the onset of disease were only available in 209 patients. Only 7% of patients had an LP within the first stage of disease, whilst 42% and 51% of patients had CSF samples taken in the second and third stages of disease respectively. Both CSF 14-3-3 and tau protein show comparable sensitivity in the first two stages of disease, however in the final stage of the disease CSF 14-3-3 is more sensitive than CSF tau protein. The time of the LP was calculated by expressing the time of LP from disease onset as a percentage of the total disease duration. The patients were classified into 3 groups depending on whether they had CSF samples taken in the first, second or third stage of the disease. Results expressed as number of positives over total number investigated. The 95% confidence limits are given on the second row.

During the study period 21 patients had a weak positive CSF 14-3-3. Of the 21 patients with weak positive CSF 14-3-3 results, 9 patients had neuropathologically confirmed sCJD whilst the remaining 12 patients had Alzheimer' disease (3), Lewy body disease (2), no neuropathological evidence of CJD ( 2), frontal lobe degeneration [START_REF] Ladogana | Mortality from Creutzfledt-Jakob disease and related disorders in Europe, Australia and Canada[END_REF], epilepsy (1), multi-focal leukoencephalopathy (1), angiotrophic lymphoma [START_REF] Ladogana | Mortality from Creutzfledt-Jakob disease and related disorders in Europe, Australia and Canada[END_REF] and no further information could be obtained on the final diagnosis in the remaining patient.

The most specific individual marker was CSF S100b however it also had the poorest sensitivity which limits its use as an isolated marker. CSF tau protein had a greater specificity than CSF 14-3-3 protein (85% vs 74%). Analysis of the control cases that had a positive CSF 14-3-3 and/or an elevated CSF tau protein showed that the most prevalent diagnoses included Alzheimer's disease, paraneoplastic syndrome and patients who clinically improved without an alternative diagnosis (Table 5). Out of a total of 242 cases of sCJD which had both CSF 14-3-3 and S100b measured only 10 had a negative 14-3-3 and a normal S100b concentration of <0.5ng/ml. Likewise out of 216 sCJD cases that had both CSF tau protein and S100b measured only 14 had concentrations of both markers within the normal range (fig 3a,b)

The impact of CSF 14-3-3 on UK surveillance over the last decade was assessed by examining the number of clinically probable cases classified as a result of a positive CSF alone. 163 probable cases that died without post mortem examination were identified.

No information regarding whether an EEG had been performed was available in 25 cases.

Of the remaining 138 cases, 3 had no EEG performed and 135 had an EEG reviewed by a senior member of the NCJDSU. Of these 135 patients, 38 had an EEG classified as highly suggestive or typical and hence appropriate for use in classification. The remaining 97 were classified as probable with the aid of a positive CSF 14-3-3 (72%).

Therefore over the study period at least 100 patients were classified as having probable sCJD on the basis of CSF 14-3-3 alone. Sixty-nine patients initially classified as probable sCJD on the basis of a positive 14-3-3 alone, with either an unsupportive or unobtainable EEG, had subsequent post-mortem examination. Neuropathological confirmation of sCJD was obtained in 66 of these cases. The remaining three cases were misclassified as probable sCJD but in fact had pathological confirmation of carcinomatosis of the meninges (14-3-3 positive, S100b 0.57ng/ml, tau protein 3201pg/ml) and Lewy body dementia (14-3-3 positive, S100b 0.57ng/ml, tau protein 1170pg/ml) and Alzheimer's disease (14-3-3 positive, S100b 0.94ng/ml, tau protein 1294pg/ml). This gives a misclassification rate of 4%.

DISCUSSION

It is ten years since a positive CSF 14-3-3 was added to the WHO diagnostic criteria for classifying probable sCJD. Since that time many studies have reported poorer sensitivity and specificity of CSF 14-3-3 than initially described, and have suggested that other markers of neuronal damage such as tau protein perform better. We have examined the diagnostic utility of CSF 14-3-3, tau protein and S100b analysis in the investigation of patients with suspected sCJD over a ten year period. In addition we have examined the overall impact that the inclusion of a positive CSF 14-3-3 has made on the diagnosis sCJD since it was introduced in 1997.

The sensitivity of a positive 14-3-3 in our study of sCJD is 86%, which is higher than that of tau protein (81%). The difference in sensitivity is more marked in the atypical forms of sCJD such as those who are younger than 50 years old at the onset of disease. In these cases the sensitivity of CSF 14-3-3 is 71% compared to 62% for CSF tau protein. This supports the findings of a larger European-wide study that found a positive CSF 14-3-3 was more sensitive than CSF tau protein in atypical sCJD cases.

[9] CSF 14-3-3 is also more sensitive than CSF tau protein in the final stage of disease. This is important as this the time at which the majority of CSF samples are taken.

A CSF S100b concentration of greater than 1.0ng/ml increases the PPV of a positive CSF 14-3-3 from 83% to 94%, and an elevated CSF tau protein from 90% to 95%. CSF S100b is routinely analysed by the NCJDSU for this reason. CSF S100b is also useful in excluding disease, with a negative CSF 14-3-3 and CSF S100b concentration of less than 0.5ng/ml having a NPV of 88%. Normal concentrations of CSF tau protein and CSF S100b concentration of less than 0.5ng/ml have a NPV of 84%.

There are no differences in the age of onset of disease, disease duration or codon 129 status of sCJD patients who are negative for CSF 14-3-3 and those that are negative for CSF tau. Therefore it is unlikely that the additional measurement of CSF tau will help improve the identification of CSF 14-3-3 negative sCJD cases.

Less than half the patients with weak positive CSF 14-3-3 results have sCJD and this suggests that the diagnostic utility of weak positive 14-3-3 results is limited. However, a repeat CSF 14-3-3 analysis in this group of patients may be of value if the clinical circumstances warrant it. [9] Elevated CSF tau protein has a better specificity for sCJD than CSF 14-3-3 (84% vs 74%). This is a similar finding to a recently published study where tau protein was felt to be the single best marker for sCJD with a specificity of 90%.

[17] It is however important to note that the diagnostic test accuracy and the differences reported by these various studies is partially dependent on the cut-off concentration of tau protein used. A universally agreed level for tau protein is not currently available. The diagnoses in the non CJD cases were similar for both tests. It is unclear why CSF tau protein should be of greater specificity than 14-3-3. The factors influencing the release of neuronal proteins in CJD and other conditions are not fully understood. Many studies have investigated the sensitivity of these two markers for the diagnosis of sCJD but very few have compared their specificity. Indeed specificity is highly dependent on the population investigated and therefore it is very difficult to compare individual studies. However the population investigated in this study is highly selected and consists of patients where the preceding pre-test probability of sCJD is high.

During the ten years since its introduction CSF 14-3-3 analysis has enabled 100 patients in the UK with suspected sCJD who died without a post-mortem and without supportive EEG data to be classified as probable sCJD. During this time only three patients have been mis-classified as probable sCJD on the basis of a positive CSF 14-3-3.

We conclude that within the UK population referred with a clinical suspicion of sCJD to the NCJDSU over the last decade, brain derived proteins such as 14-3-3, S100b and tau have immense diagnostic value (see conclusion Table 6). In our experience, the combination of 14-3-3 and S100b remains the best predictor of supporting or excluding sCJD as a diagnosis when employed in an alogorithmic manner using 14-3-3 detection as the primary screening marker before utilizing the S100b result. The importance of interpreting these results in the appropriate clinical context however is vital, especially as the phenotypic heterogeneity of sCJD remains wide.

Table 6

Conclusions regarding the use of CSF 14-3-3 and other markers in the diagnosis of sCJD CSF 14-3-3 as a sole marker has the highest sensitivity, particularly in the final stage of disease. As a sole marker CSF tau has the greatest specificity.

A combination of CSF 14-3-3 and elevated S100b or elevated tau protein and S100b has a greater PPV than CSF 14-3-3 alone

The combination of CSF 14-3-3 and tau protein in CSF 14-3-3 negative sCJD cases has not been shown to be of any additional value, even in phenotypically atypical cases

The combination of a negative CSF 14-3-3 and S100b is of value as a potential means of excluding sCJD

The differential diagnosis of sCJD in the 14-3-3 positive non-CJD cases remains as documented in previous studies

Figure 2a Figure 2b Figure 1c NegativeFigure 3a

 2a2b1c3a Figure 2aThe role of a positive 14-3-3 and a diagnostic S100b in the diagnosis of sCJD

Table 1

 1

	Global

Alternative pathological proven diagnosis Number of cases

  ). 1 patient clinically not thought to have had CJD and post mortem conducted but brain tissue not examined ** 1 patient had a western blot for PrP Sc which was negative, clinically this patient is thought to have a paraneoplastic process

	Huntington's disease	3
	Hashimoto's encephalitis	2
	Corticobasal degeneration	2
	Multi-system atrophy	1
	Granulomatous disease	1
	Central pontine myelinolysis	1
	Serotonin syndrome	1
	Table 1 Disease control cases (pathologically proven alternative diagnosis or
	alternative clinical diagnosis)
	Alzheimer's disease	14
	Biopsy/post-mortem showed no evidence of CJD	12*
	Malignancy/Paraneoplastic syndrome	10**
	Lewy body dementia
	Cerebral lymphoma
	Encephalitis
	Parkinson's disease
	Cerebrovascular disease
	Alzheimer's disease and ischaemic change
	Alzheimer's disease and Lewy body dementia
	Progressive multifocal leucoencephalopathy
	Normal pressure hydrocephalus
	Fronto-temporal dementia
	Corticostriatal degenerative disease
	Demyelination
	Alternative Clinical Diagnosis	Number of cases
	Dementia (unknown etiology) or clinically not CJD	22
	Clinically improved with or without steroids	22
	Alzheimer's disease	18
	Cerebrovascular disease/cerebral vasculitus
	Fronto-temporal dementia
	Lewy body dementia
	Paraneoplastic syndrome
	Psychiatric disorder

*

Table 2

 2 The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and efficiency for each marker and combination of CSF markers in neuropathologically confirmed sCJD. Figures in parentheses are 95% confidence limits.

					14-3-3	Tau	14-3-3	14-3-3, Tau
		14-3-3	Tau S100b	and	and	and	and
					S100b	S100b	Tau	S100b
	sCJD 1	210/245 175/216 158/243 151/242 127/216 162/216	123/216
	Not sCJD 1 44/171 20/135 17/169 9/169	7/135	16/135	6/135
	Sensitivity	86%	81%	65%	62%	59%	75%	57%
		(81, 90)	(75, 86)	(59, 71)	(57, 69)	(52, 65)	(69, 81)	(50, 64)
	Specificity	74%	84%	90%	95%	95%	88%	96%
		(67, 81)	(78, 91)	(84, 94)	(90, 98)	(90, 98)	(81, 93)	(91, 98)
	PPV	83%	90%	90%	94%	95%	91%	95%
		(77, 87)	(85, 94)	(85, 94)	(90, 97)	(90, 98)	(86, 95)	(90, 98
	NPV	78%	74%	64%	64%	59%	69%	58%
		(71, 84)	(66, 80)	(58, 70)	(58, 70)	(52, 66)	(61, 76)	(51, 65)
	Efficiency	81%	83%	75%	76%	73%	80%	72%
		(77, 85)	(78, 86)	(71, 79)	(72, 80)	(68, 77)	(75, 84)	(67, 76)

1

The figures given are the number of positive or negative results over the total number of samples investigated, Efficiency was defined as: True positives + True negatives Total number tested

Table 3

 3 Effect of age at onset of disease, disease duration and PRNP-codon 129 genotype on the sensitivity of CSF 14-3-3 and Tau protein.

		14-3-3 (%)	Tau protein (%)
	< 50 years at onset disease	12/17 (71%) (44, 90)	10/16 (62%) (35, 84)
	≥ 50 years at onset disease	198/228 (87%) (82, 91)	165/200 (83%) (77, 87)
	> 12 months disease duration	23/41 (56%) (40, 72)	21/39 (54%) (37, 70)
	≤ 12 months disease duration	184/199 (92%) (88, 96)	151/175 (86%) (80, 91)
	PRNP Codon 129-MM	106/121 (88%) (80, 93)	92/108 (85%) (77, 91)
	PRNP Codon 129-MV	32/44 (73%) (57, 85)	26/39 (67%) (50, 81)
	PRNP Codon 129-VV	27/29 (93%) (77, 99)	24/27 (89%) (71, 98)
	Results expressed as number of positives over total number investigated. Figures in parentheses
	are the sensitivity of the individual marker in each subset of sCJD. The 95% confidence limits
	are given on the second row.

Table 4

 4 Influence of time of CSF sampling on CSF 14-3-3 and tau protein positive results.

	Stage of disease	14-3-3 (%) Tau protein (%)
	First stage (0-33% disease duration)	9/14 (64%)	10/14 (71%)
		(35, 87)	(42, 92)
	Second stage (34-66% disease duration)	73/88 (83%)	72/88 (82%)
		(73, 90)	(72, 89)
	Third stage (67-100% of disease duration)	97/107 (91%)	89/107 (83%)
		(83, 95)	(75, 90)

Table 5

 5 Diagnoses in disease controls with positive CSF 14-3-3 and elevated tau protein cases

	Diagnosis (14-3-3 positive)	Pathological (26)	Clinical (18)
	Alzheimer's disease	7*
	Malignancy/Paraneoplastic	5
	Improved/ no evidence of CJD	3**
	Encephalitis/limbic encephalitis	3
	Lewy Body Dementia	1
	Parkinson's disease	1
	Cerebrovascular disease	0	2***
	Β cell Lymphoma/cerebral lymphoma	3
	Fronto-temporal Dementia	0
	Vasculitis	0
	Central pontine myelinolysis	0
	Multifocal demyelination	1
	Anoxic brain injury	1
	Normal pressure hydrocephalus	1
	Corticobasal degeneration	0
	Diagnosis (tau protein >1260pg/ml)	Pathological (12)	Clinical (6)
	Alzheimer's disease	3
	Malignancy/paraneoplastic	3
	Improved or no evidence of CJD	1
	Fronto-temporal Dementia	0
	Corticobasal degeneration	0
	Encephalitis	0
	Lymphoma	1
	Cerebrovascular disease	1***
	Multi focal demyelination	1
	Lewy body disease	1
	Anoxic brain damage	1
	*three had evidence of additional cerebrovascular disease and two also had evidence of LBD
	** one did not have brain examined
	***one had evidence of Alzheimer's disease

ACKNOWLEDGEMENTS

Surveillance of CJD is dependent on the co-operation of neurologists and neuropathologists throughout the UK. The authors are particularly grateful to the relatives of the patients for their co-operation and in particular to the relatives of the patient reported in this case report for their consent to publish. The CJD Surveillance Unit is funded by the Department of Health and the Scottish Home Office Department of Health.

CONFLICTS OF INTEREST

We have no conflicts of interest

LEGENDS

Figure 1

A western blotting illustrating the presence of positive, weak positive and negative CSF 14-3-3. Lanes 1, 2, 3 are positive for CSF 14-3-3, lanes 4, 5, 6 are weakly positive for CSF 14-3-3 and lanes 7, 8 and 9 are negative for CSF 14-3-3. Lanes 1 and 2 are from two patients with sporadic Creutzfeldt-Jakob disease (CJD), lanes 3 and 4 are from patients who have had a stroke and the remaining lanes contain CSF samples from patients who do not have CJD.