W Lloyd Bircher

Lizy John

Predictive Power Management for Multi-Core Processors

Predictive power management provides reduced power consumption and increased performance compared to reactive schemes. It effectively reduces the lag between workload phase changes and changes in power adaptations since adaptations can be applied immediately before a program phase change. To this end we present the first analysis of prediction for power management under SYSMark2007.

Compared to traditional scientific/computing benchmarks, this workload demonstrates more complex core active and idle behavior. We analyze a table based predictor on a quad-core processor. We present an accurate run-time power model that accounts for fine-grain temperature and voltage variation. By predictively borrowing power from cores, our approach provides an average speedup of 7.3% in SYSMark2007.

INTRODUCTION

The challenge in applying power management to increase efficiency and performance is identifying when to adapt performance capacity. In the case of controlling dynamic voltage and frequency scaling, the ubiquitous commercial solution has been to react to changes in performance demand. While this approach is simple, it performs suboptimally [4][22] for workloads with many distinct and/or short phases. Each time a workload transitions from a phase of low performance demand to a phase of high demand, reactive power management increases performance capacity some time after the transition. During the time between the change in demand and capacity, performance is less than optimal. Similarly, power consumption is sub-optimal on transitions from high to low demand. The amount of performance loss is proportional to the number of phase changes in the workload and the lag between demand and capacity. For increasing performance in power-limited situations, reactions must be fast enough to prevent overshooting the power limit or missing opportunities to increase performance.

To date analysis in this field has focused on predicting phase changes within scientific/computing workloads such as SPEC CPU. Since these workloads contain only active phases in which the processor is non-halted, a significant amount of valuable phase information is ignored. In a "real" system much of the time is spent idle waiting for user input or responses from the I/O subsystem. The power difference between the active and idle phase is much larger than what is observed within a fully active phase. To address this missed opportunity, we analyze a table-based phase predictor under the SYSMark2007 workload. This benchmark is better than conventional scientific/computing benchmarks at providing complex workload variation for power management analysis. GUI interaction, keypress delays, I/O, thread migration and a wide range of processor activity levels are provided. It includes popular desktop/personal computing applications such as Microsoft Word, Excel, PowerPoint, Adobe Photoshop, Illustrator, etc.

Unlike traditional, benchmarks which are composed of a single active phase, this one spends greater than 40% of the time in phases less than 1 second in duration. The analysis is performed using our proposed tabled-based activity predictor. This predictor simplifies existing implementations by using core activity as the predicted metric. It achieves accuracy of 98% for 43% of SYSMark2007. The remaining portion is detected as unpredictable and is handled reactively. To estimate power consumption of this approach, we present a detailed run-time power model for an AMD quad-core processor, utilizing core-level performance metrics. We improve upon existing models by accounting for leakage power and temperature effects and the impact of power management. This model has an average error rate of 0.89% across SPEC CPU 2006 and SYSMark2007.

2

PREDICTION FOR POWER MANAGEMENT Unlike prior research in prediction for power management [START_REF] Isci | Runtime Phase Monitoring and Prediction on Real Systems with Application to Dynamic Power Management[END_REF][6] [START_REF] Isci | Long-term workload phases: duration predictions and applications to DVFS[END_REF][17] [START_REF] Annavaram | The Fuzzy Correlation between Code and Performance Predictability[END_REF] we choose core activity and power as the predicted metric. Most others predict IPC [START_REF] Duesterwald | Characterizing and Predicting Program Behavior and its Variability[END_REF] [START_REF] Isci | Long-term workload phases: duration predictions and applications to DVFS[END_REF] or memory-boundedness [START_REF] Isci | Runtime Phase Monitoring and Prediction on Real Systems with Application to Dynamic Power Management[END_REF] [START_REF] Isci | Long-term workload phases: duration predictions and applications to DVFS[END_REF]. Their approach relies on detecting program phases of execution which benefit less from high processor clock frequency. These phases are usually defined as memory-bound. The degree of memory-boundedness is usually estimated using IPC, memory access intensity or a combination of both. The problem with this approach is that no application is 100% memory bound. Therefore, any reduction in processor clock frequency will result in a performance loss. These approaches rely on the tradeoff of a small performance loss for a larger power savings.

Instead we choose to predict core activity levels. In this case core activity is defined as the ratio of non-halted cycles to non-halted + halted cycles. Since by definition a halted (idle) core has 0% performance dependence on core frequency, there is no need to accept a performance loss in order to save power. This is the essence of existing c-state power management techniques [START_REF]BIOS and Kernel Developer's Guide for AMD Family 10h Processor[END_REF]. Since most existing dynamic power management (clock gating and power gating) use core halting as the key to entry, core activity can represent program phase changes which have the largest impact on power consumption. For the quad-core processor considered in this study, IPC variation is only responsible for power fluctuations of 20% of peak power compared to nearly 90% for core activity. Core activity is used by all existing Windows and Linux operating systems for control of DVFS. For this reason, it is also an appropriate prediction metric for out of band techniques such as power boosting [START_REF] Mcgowen | Temperature Control on a 90-nm Itanium Family Processor[END_REF][8] [START_REF] Charles | Evaluation of the Intel® Core™ i7 Turbo Boost feature[END_REF] and power limiting.

These techniques respond to changes in power consumption, as dictated by operating systems and applications, to improve performance or ensure safe operation

3

PROGRAM PHASE CHARACTERIZATION To understand power-relevant phases in real world workloads, we characterize the program behavior of the desktop/client SYSmark® 2007 benchmark [START_REF]An Overview of SYSMark 2007 Preview[END_REF]. This benchmark represents a wide range of desktop computing applications. The major categories are e-learning, video creation, productivity, and 3D. The individual subtests are listed in Table 1. This benchmark is particularly important to the study of dynamic power adaptations because it provides realistic user scenarios that include GUI interface in I/O interactions.

To illustrate the potential of this workload for the analysis of core activity prediction, we present traces of core activity level and duration. Figure 1 shows the number of cores used concurrently by the workload over time. At each point where the number of cores changes, a phase transition occurs as idle cores become active or vice versa. This is distinct from traditional benchmarks such as SPEC CPU which uses a fixed number of processors for the entire benchmark run. Therefore, there are essentially no core activity changes. As a result power fluctuations are much less since variation comes only from changes in pipeline utilization (IPC).

Table 1: Benchmarks Used: SYSmark® 2007

The actual number of phase changes in SYSMark is even greater since the traces in Figure 1 uses averaging of samples to smooth the trace for readability. Figure 2 quantifies actual phase variation under Windows Vista. The majority of phases are much shorter than one second. In all cases an average of at least 40% of phases are shorter than one second. There are two factors that cause high rates of phase changes: low overall activity and thread migration by the operating system. Considering only Figure 2 it might be concluded that two or more cores remain idle for almost the entire benchmark. However, this is not the case.

Figure 2 shows that all four cores experience large numbers of phase transitions. The reason is thread migration. Most threads only remain on any given core for 10s of milliseconds before being preempted. Since Windows does not aggressively maintain thread locality (application controlled affinity does), preempted threads tend to continuously migrate among all available cores. Since there are consistently more cores than are required by the workload, there are frequent transitions from idleactive and active-idle.

METHODOLOGY 4.1

Power Measurement To measure power consumption for the development of a power model, we instrument an AMD Phenom2 3GHz system. Processor core power consumption is measured using a hall-effect sensor placed in-line with the core power rail (Core VDD). This sensor produces an output voltage which is linearly proportional to current. We also measured voltage levels at the point where the current enters the processor socket. We perform all sampling at a rate of 1 MHz, using a National Instruments NIUSB-6259 [START_REF]National Instruments Data Acquisition Hardware[END_REF]. This granularity allows the measurement of most power phases which were sufficiently long to perform adaptations.

Though shorter duration phases exist, current adaptation frameworks are not able to readily exploit them.

Performance Counter Measurement

We use performance monitoring counters (PMCs) to track core activity and power. We develop an on-line, PMCbased, core-level power model. It is necessary to use a power model since it is impossible to measure core-level power consumption since all cores share a single power plane [START_REF]BIOS and Kernel Developer's Guide for AMD Family 10h Processor[END_REF]. Physical instrumentation thus cannot provide core-level power, whereas the power model using performance counters can provide power estimates from individual cores. Prior researchers have shown that accurate power models can be built using performance counter measurements [2][10] [START_REF] Bircher | Complete System Power Estimation: A Trickle-Down Approach based on Performance Events[END_REF]. Power models from prior researchers are extended to include leakage power, temperature effects and the impact of power management that are present in modern multi-core processors. These models are in fact preferred for dynamic power management since there is no need to measure power with out-of-band instrumentation. The accuracy of the models was verified with controlled core level and aggregate measurements.

To sample performance monitoring counters we develop a small kernel which provides periodic sampling of the four AMD performance counters. This kernel uses a device driver to give ring-0 access to user-mode applications. This approach is preferred over existing user-mode In all experiments the sampling overhead for performance counter access average less than 1%. Another benefit of our device driver is that it provides access to others registers besides performance counters. In particular, our approach requires access to model specific registers (MSRs) and PCI configuration registers. These registers allow our application to take control of processor frequency, voltage, power management. It also gives access to on-die processor temperature sensors. This is required to account for static power consumption. Finally, sampling is invoked at a user-specified periodicity user the built-in OS multimedia timer [START_REF]Windows Multimedia:timeEndPeriod[END_REF].

CORE ACTIVITY PREDICTOR

This predictor uses table-based prediction structures and the repetitive nature of core activity phases to predict performance demand and/or power consumption. The predictor is shown in Figure 3. (PHT) and predicted level. The distinction is that the predictor uses a PHT length of two entries and records the level of the next phase only. Since we define core activity to have only two possible values: 1 and 0 (100% and 0%), there is no need to record the level of each phase as is done in traditional run length encoding predictors [START_REF] Sherwood | Phase Tracking and Prediction[END_REF]. For example if the predicted level is 1, then the previous level was 0 and the level before that was 1. The PHT size is modest at 48 entries. This compares to Isci [START_REF] Isci | Runtime Phase Monitoring and Prediction on Real Systems with Application to Dynamic Power Management[END_REF] which uses a GPHR of 8 and PHT of 128 in order to obtain accuracy better than statistical predictors. Each predictor entry also contains several fields that capture quality, validity, and pending prediction status. The fields are described in Table 2.

3D

change is to occur. This is the most critical value produced by the predictor. It is used by the power manager to schedule changes in power/performance capacity of the system. This value allows for optimal selection of performance capacity given the anticipated duration of operation at a particular performance demand. Weight "Quality" of phase as a function of past predictions and duration. The weight is used by the power manager to determine if a prediction will be used or not. It is also used by the replacement algorithm to determine if the phase will be replaced if the predictor is full. All newly detected phases start with a weight of 1. If the phase is subsequently mispredicted, the weight is reduced by a fixed ratio.

Valid

Indicates whether this entry has a valid phase stored with a "true" or "false." Pending Indicates if this phase is predicted to occur again. This value is set "true" on the occurrence of the phase and remains true until the predicted transition time has past.

CORE-LEVEL CPU POWER MODEL

Using a real system instrumented for power measurement we develop polynomial, regression models for power consumption. The details of the model are given in Tables 3 and4. The model improves on existing on-line models [2][5][10] by accounting for power management and temperature variation. All model coefficients are tuned empirically using a real system instrumented for power measurement. Like existing models this one contains a workload dependent portion which is dominated by the number of instructions completed per second. In this case we use the number of fetched operations per second in lieu of instructions completed. The fetched ops metric for Phenom2 processors is preferred as it also accounts for speculative execution. The distinction of our model is that it contains a temperature dependent portion. Using workloads with constant activity, we vary processor temperature and voltage to observe the impact on static leakage power. Temperature is controlled by adjusting the speed of the processor's fan. Temperature is observed with 0.125 degree Celsius resolution using an on-die temperature sensor [START_REF]BIOS and Kernel Developer's Guide for AMD Family 10h Processor[END_REF]. This sensor can be accessed by the system under test through a built-in, on-chip register. Voltage is controlled using the P-State control register. This allows selection of one of five available voltage/frequency combinations. Voltage is observed externally using our power instrumentation. Like the workload dependent model, we tune the coefficients of the static power model using regression techniques. Note that the static power model is highly process dependent. Processors with different semiconductor process parameters require the model to be re-tuned. The dominant power management effects (voltage/frequency scaling, clock gating) are further accounted for using the gateable and ungateable power models. Gateable power is found by measuring the effect of enabling/disabling idle core clock gating. Ungateable represents the portion of power which cannot be gated. These components are also found experimentally. The resultant, average error in the model was 0.89%. The error distribution for SPEC CPU2006 and SYSmark2007 has the first standard deviation with less than 1% error. Worst-case error is 3.3%.

+ ݈ܾ݁ܽ݁ݐܽܩ ே ൰ + ܿ݅ݐܽݐܵ ௧,் Workload Dependent ((FetchOpsN/Sec)•CoeffF+(FloatPointOpsN/Sec) •CoeffFP+(DCAccessN/Sec) •CoeffDC) •Voltage 2 Gateable (%HaltedN) •CoeffGateable•Voltage 2 •FrequencyN Ungateable (%NonHaltedN) •CoeffUngateable•Voltage 2 •FrequencyN Static (Temp 2 •CoeffT 2 +Temp 1 •CoeffT 1 +•CoeffT 0 )VoltageN

RESULTS

This section provides power and performance estimates of our phase predictor in comparison to commercial reactive schemes. First we consider the prediction accuracy and coverage. Then, we estimate performance speedup due to borrowing power from under-utilized cores using prediction.

Performance Metric Definitions

To analyze the effectiveness of core activity prediction, we define three metrics: weighted match ratio, weighted hit ratio, and prediction coverage. These metrics differ from traditional branch predict / cache hit/miss metrics in that they describe the portion of the total program execution they represent. For example, a frequently predicted phase may only cover a small portion of the time if the phase is short. These get proportionally less weight. A long phase that occurs less gets a greater weight as it represents more time. A system with slow adaptation rates may assign more value in predicting long phases, while a fast-adapting system may need greater precision for shorter phases since they are viable for adaptation. These metrics are similar in function to those proposed by Isci [START_REF] Isci | Long-term workload phases: duration predictions and applications to DVFS[END_REF] for the analysis of a phase duration predictor. The weighted match ratio is a measure of predictor utilization weighted by phase length. When a phase transition occurs, if a phase of that length has been observed previously by the predictor, it is considered a match. This is similar to a tag match. It does not mean the prediction was correct --just that a similar phase has been seen recently. Weighted hit ratio indicates the outcomes of predictions weighted by phase length. For example, a 10ms phase prediction has twice the weight of a 5ms phase prediction. The net result is that weighted hit ratio favors correct outcomes on long-duration phases. Prediction Coverage. Due to the cost of misprediction, predictions are only considered valid if the outcome has a high probability of success, similar to a confidence predictor [START_REF] Jacobson | Assigning Confidence to Conditional Branch Predictions[END_REF]. Using the Weight value, maintained for each PHT entry, difficult-to-predict phases are ignored. This approach shifts the assessment of prediction quality from hit rate to the portion of the workload that can be accurately predicted. The power manager only adapts predictively for high-probability phases. The remainder of the time, the power manager behaves reactively.

7.2

Quantitative Comparison Table 5 summarizes the performance of our best case predictor configuration, 48-entry PHT, at anticipating phase changes. In all subtests, the predictor maintained an average weighted match ratio over 95%. This indicates that the predictor is fully utilized, and is not oversized. Increasing the PHT size beyond 128 caused weighted match ratio to drop below 50%. This suggests that the complete range of unique phases can be captured in a predictor of about 64 entries. Hit rates show diminishing returns above a PHT size of 48. Average weighted hit rates were also above 95% for all subtests (98% overall). Since our predictor only predicts phase transitions it has previously observed, the most critical result is prediction coverage percentage. In the worstcase, Productivity, was predictable 33.5% of the time. During that time, the predictions were almost always correct. For the cases in which a prediction is not possible, the predictor behaves as a reactive predictor. We compare these results to the reactive p-state slection scheme used by Windows Vista [START_REF]Processor Power Management in Windows Vista and Windows Server[END_REF]. This configuration is representative of the majority of commercial, operating system directed, reactive DVFS algorithms in use today [START_REF] Bircher | Analysis of Dynamic Power Management on Multi-Core Processors[END_REF]. Since the reactive "predictors" do not make use of any success metrics, the prediction coverage is always 100%. Note, weighted match ratio is not applicable here since no comparison is made to past phases. For the reactive scheme, the critical metric is weighted hit ratio.

Hits/misses are determined by configuring the Windows Vista DVFS algorithm to sample core activity on 100ms intervals [START_REF]Processor Power Management in Windows Vista and Windows Server[END_REF]. By tracing core activity and the resultant DVFS selections made by the operating system we are able to detect when correct predictions are made. For example, if an active phase occurs yet the operating system selects a low core frequency, a miss is counted. Since the AMD Phenom2 supports five DVFS frequency/voltage operating points, the "correct" frequency was determined using the operating systems own algorithm. The algorithm works on the concept of an ideal average activity. If the reactive scheme achieved the ideal activity with the selected frequency, it is counted as a hit. On average the reactive scheme is wrong 35% of the time, though it is able to "predict" 100% of the time. Note that since the predictor can identify the non-predictable phases it is possible to increase the coverage of the predictor by simply performing reactively when a prediction is not possible. If the workload is assumed to have a uniform distribution of predictable phases, then the predictor should equal it in non-predictable phases (57% of the time) and surpasses it in predictable phases.

Predictive Frequency Boosting

The second application of core activity prediction is boosting performance in active cores by borrowing power from under-utilized cores. To estimate the impact of our approach we scale power, performance, and core activity traces from a real system. For each point in the traces we consider the available power surplus and the probability that the predictor is able to identify a predictable phase. If it is determined that performance can be increased, we scale (shrink) the duration of the current phase at 75% of the frequency increase, to account for the speedup. Our analysis showed that, on average, SYSMark07 core frequency sensitivity is 75%. If a phase is not predictable, then no frequency change is applied.

The major distinction between this analysis and that of the previous section is that the predictor predicts power rather than core activity. Phases are still defined in terms of core activity. However, an additional field is included to record the next power level. This is similar to Duesterwald's cross-metric predictors [START_REF] Duesterwald | Characterizing and Predicting Program Behavior and its Variability[END_REF] which predict one metric based on another. Note that the maximum performance increase is assumed to be 20%. This limit is determined by critical path delay at the maximum supported voltage. This processor, like many multi-core processors produced in the last few years is performance limited by the total power budget, not critical path delay of a given core. If total processor power consumption is less than the maximum, due to under-utilized cores, active cores can reach higher frequencies (at higher voltages).

A comparison of average error results for the predictor versus a reactive scheme are presented in Figure 4. These results were somewhat surprising given the high prediction rates observed in the previous section. On average the predictor achieved 86% accuracy compared to 83% for the reactive scheme. This suggests that the 35% of execution time identified as predictable, yielded only three percentage points of increase in accuracy compared to the reactive scheme. Since core activity phases are being predicted with high accuracy, the likely cause of lower than expected accuracy is the use of cross-metric prediction. Since power is predicted indirectly using activity level, aliasing of different power levels could be causing increased error. Considering individual subtests, the results were better. In the best case, video creation accuracy was 86% compared to 77% for the reactive scheme. In all cases, the predictor outperformed the reactive scheme. The best-case speedup of 11.1% was achieved on elearning; a highly predictable and mostly single-threaded subtest. 3D had the lowest speedup at 5% partly due to low predictability, but more due to a lack of power surplus. Average speedup was 7.3%. The results are summarized in Figure 5. Speedup results are not provided for the reactive case is there is no foreknowledge of phase transition outcomes for avoiding boosting in unstable phases.

CONCLUSION

In this paper we have presented an analysis of core activity prediction in SYSMark2007. We show that this benchmark contains a large number of short duration phases that are relevant for power management. We demonstrate that these phases can be predicted with 98% accuracy 43% of the time. An estimate of performance increase due to power borrowing is presented, showing a potential speedup 7.3%. The analysis makes use of a temperature and voltage sensitive run-time power model that achieves 99.1% accuracy.

Figure 1 :Figure 2 :

 12 Figure 1: Concurrent Core Activity by SYSmark® 2007 Benchmark

Figure 3 :

 3 Figure 3: Power Phase PredictorLike traditional table-based predictors the main components are: a global phase history register (GPHR), pattern history table (PHT) and predicted level. The distinction is that the predictor uses a PHT length of two entries and records the level of the next phase only. Since we define core activity to have only two possible values: 1 and 0 (100% and 0%), there is no need to record the level of each phase as is done in traditional run length encoding predictors[START_REF] Sherwood | Phase Tracking and Prediction[END_REF]. For example if the predicted level is 1, then the previous level was 0 and the level before that was 1. The PHT size is modest at 48 entries. This compares to Isci[START_REF] Isci | Runtime Phase Monitoring and Prediction on Real Systems with Application to Dynamic Power Management[END_REF] which uses a GPHR of 8 and PHT of 128 in order to obtain accuracy better than statistical predictors. Each predictor entry also contains several fields that capture quality, validity, and pending prediction status. The fields are described in Table2.

Figure 4 :

 4 Figure 4: Core Power Accuracy Predictive vs Reactive

Figure 5 :

 5 Figure 5: SYSmark 2007 Speedup

Table 2 Predictor Field Descriptions Predictor Field Description

 2

		4
					E-Learning
	Active Cores	1 2 3
		0				Time
		4			Video Creation
	Active Cores	1 2 3
		0				Time
		4
					Productivity
	Active Cores	1 2 3
		0				Time
		4
						3D
	Active Cores	1 2 3
		0				Time
		100		VideoCreation			E-Learning
	Residency(%)	0 50	14 12 24 49	17 15 29 39	17 15 29 39	15 14 24 48	20 12 29 39	35 9 53	35 9 53	29 11 60
			Core0 Core1 Core2 Core3	Core0 Core1 Core2 Core3
		100		Productivity
	Residency(%)	0 50	6 4 22 67	12 5 32 52	12 5 32 52	13 5 34 48	26 3 6 64	38 24 24 14	38 24 24 14	18 17 6 59
			Core0 Core1 Core2 Core3	Core0 Core1 Core2 Core3

Length

Duration of phase. This is also the table index. When a phase is detected, it is used to index the prediction table. Level Predicted level at next transition. For the activity predictor this is active or idle. For power prediction this is the last power level seen when this phase occurred. Timestamp Records timestamp of when predicted phase

Table 4 : Measured Quantities (Model Inputs)

 4

	Quantity		Description
	N	Core Number.
	FetchOps	Micro operations fetched.	Includes
		speculative operations.
	FloatPointOps	Floating point operations retired. Accounts
		for difference in power between INT/FP.
	DCAccess	Data cache access. Accounts for power
		consumed in caches.
	%Halted	% of cycles in which the core was halted.
	%Halted	% of cycles the core was not halted.
	Voltage	Maximum requested voltage for all cores.
		Due to shared voltage plane.
	Frequency	Current core frequency. This can be read
		via AMD model specific register.
	Temperature	Current processor temperature. This can be
		read via AMD model specific register.
	Coeff	Model coefficient. The values are
		determined	empirically	using
		measurement/regression.

Table 3 : AMD Quad-Core Phenom2 Power Model Power Models Equation

 3

	Total Power	൬ ݐ݊݁݀݊݁݁ܦ݈݀ܽ݇ݎܹ ே + ଷ ேୀ	݈ܾ݁ܽ݁ݐܷܽ݃݊ ே

Table 5 : Prediction Outcomes -Core Activity Predictor, 48-Entry PHT

 5

		Weighted	Weighted	Prediction	Effective
	Workload	Match Ratio	Hit Ratio	Coverage	Hit Ratio
	E-Learning	92.8%	98.8%	57.0%	82.6%
	Productivity	95.5%	95.8%	33.5%	73.8%
	Video Creation	95.1%	97.7%	43.0%	76.4%
	3D	98.3%	99.1%	37.9%	72.7%
		Reactive
		Weighted	Weighted	Prediction	Effective
	Workload	Match Ratio	Hit Ratio	Coverage	Hit Ratio
	E-Learning	100%	66.4%	100%	66.4%
	Productivity	100%	65.2%	100%	65.2%
	Video Creation	100%	63.5%	100%	63.5%
	3D	100%	59.7%	100%	59.7%


Heather Hanson
email: hlhanson@us.ibm.com

Karthick Rajamani
email: karthick@us.ibm.com

What Computer Architects Need to Know About Memory Throttling

Memory throttling is one technique for power and energy management that is currently available in commercial systems, yet has has received little attention in the architecture community. This paper provides an overview of memory throttling: how it works, how it affects performance, and how it controls power. We provide measured power and performance data with memory throttling on a commercial blade system, and discuss key issues for power management with memory throttling mechanisms.

1 Memory throttling

Overview

Memory throttling is a power management technique that is currently available in commercial systems and incorporated in several proposed power and thermal control schemes, yet the underlying mechanisms and quantitative effects on power and performance are not widely known.

In a nutshell, memory throttling restricts read and write traffic to main memory as a means of controlling power consumption. A significant fraction of memory power is proportional to read and write bandwidth, and restricting bandwidth creates an upper bound for memory power.

Computer architects should be aware of memory throttling because it manipulates the instruction and data streams that feed the processor cores and can create both performance bottlenecks due to bandwidth overconstriction, as well as opportunities for improving performance through judicious power budgeting.

Simple control with a fixed memory throttle setting allows the full capacity of the memory to be available, with a regulated rate of access to limit power consumption, in enterprise systems with large memory configurations that if left unchecked would exceed the system's power budget.

Altering memory throttling dynamically tailors the access rate to workload demands and variable power allocation, useful for regulating DIMM temperature [START_REF] Wu | Dynamic memory throttling for power and thermal limitations[END_REF] [2] [START_REF] Sampson | Filter based throttling[END_REF], providing memory power control beyond power-down modes [START_REF] Hur | A comprehensive approach to DRAM power management[END_REF], and optimizing system-wide performance or power with techniques such as power shifting [START_REF] Felter | A Performance-Conserving Approach for Reducing Peak Power Consumption in Server Systems[END_REF].

Enforcing a power cap on the memory subsystem will become increasingly important for servers with large memory configurations to support data-intensive applications, and to support virtual systems that pack multiple software stacks-and their bandwidth requirementsinto a compact number of processing elements.

Even in systems where memory power is a small fraction of the total, memory throttling provides an essential function: in power-limited situations, every Watt trimmed from the memory budget is a Watt gained for use in processor cores and other components.

Comparison to CPU clock throttling

At a high level, memory throttling is analogous to the more familiar processor clock throttling. Clock throttling creates a duty cycle for the processor core's clock signal, where a time period is divided into distinct run and hold intervals. During the run interval, the clock signal runs freely. During the hold interval, the clock signal is gated to remain still, and computation halts.

Figure 1 illustrates run-hold duty cycles for clock throttling, not to scale. The period of one run-hold cycle is typically on the order of thousands of cycles. With long (relative to processor pipelines) duty-cycle periods, clock throttling behavior is a repeating sequence of a burst of activity followed by an idle period. During the burst of activity, processor cores operate normally; during the idle period, computation and memory requests halt and power consumption drops accordingly. In this way, the clock throttling mechanism acts as a governor for the power and performance of the system, with a direct control of the processor and indirect control of the memory system.

Similar to clock throttling's limits on processor cycles within a time interval, memory throttling regulates reads and write accesses within a time interval. There are several approaches to implementing memory throttling features. One technique is similar to clock throttling with run-hold duty cycles, where memory accesses pass through at the requested rate untouched during the run portion of a time interval, then are halted during the hold portion [START_REF] Hur | A comprehensive approach to DRAM power management[END_REF] [START_REF] Felter | A Performance-Conserving Approach for Reducing Peak Power Consumption in Server Systems[END_REF]. Another memory throttling mechanism is to periodically insert one or more idle clock cycles between every N th and (N + 1) th memory accesses [START_REF] Wu | Dynamic memory throttling for power and thermal limitations[END_REF], spreading out the accesses and lowering peak bandwidth. Yet another mechanism uses a bit mask to allow or block memory accesses, optimized to minimize interruptions to free running clock throttled CPU clock throttled MEM accesses Figure 1: Clock throttling maintains the original frequency during a portion of the time interval; memory throttling allows memory accesses within the time interval up to a quota. In this example, six memory accesses are allowed in each interval, intermittent in the first interval and sequential until the quota in the second and third intervals. real-time streaming data [START_REF] Kahn | Adaptive throttling of memory accesses, such as throttling RDRAM accesses in a real-time system[END_REF].

In this paper, we focus on an implementation within the memory controller that allows up to N transactions within a time period. The memory throttling time period in this implementation corresponds to a window of 32 frames, approximately 50 ns at a rate of one frame per 667 MHz bus cycle. Once the quota of N reads and writes is reached in a time period, any additional requests must wait for a future time period to proceed. Instructions waiting for the over-quota memory transactions must stall until they are processed. The POWER6 system used in these experiments uses a single memory throttle value universally applied to all DIMMS. Other implementations operate on finer granularities of unique throttle values, such as one memory throttling option on commercially available POWER7 systems that supports unique memory throttle values for each of two channel pairs within a memory controller.

Power and Performance

There are a few important distinctions to consider for memory throttling with respect to power and performance. First, memory throttling does not alter the DRAM clock in the manner that clock throttling alters the CPU clock signal. Memory throttling is usually implemented within the memory controller (processor chip or chipset) to restrict the memory request rate, and the DRAM itself is unchanged, unlike memory low-power states and power-down modes.

With quota-style memory throttling, the amount of additional latency due to throttling restrictions is a function of requested bandwidth rather than a fixed amount of busy time per interval. A workload whose bandwidth fits within the quota will proceed unchanged by the memory throttle mechanism. A steady workload with bandwidth needs beyond the quota will slow down as it waits for memory transactions to complete at a slower pace. A bursty workload with bandwidth spikes above the quota will effectively be smoothed out as ex-cess transactions are postponed and fulfilled during a later interval; latency for some individual transactions will increase though the average throughput may or may not be affected. Thus, the performance impact of memory throttling depends upon both the bandwidth and the time-varying behavior of workloads.

To the first order, memory power is linearly proportional to the sum of read and write bandwidth. In situations where memory bandwidth requirements fall below the throttle-enforced quota, bandwidth-and thus, power consumption-are unaffected by memory throttling. Rather, the memory throttling mechanism enforces an upper bound on memory power consumption, creating an effective tool for power budgeting.

Infrastructure

System

In this study, we characterized memory throttling with an IBM JS12 blade system [START_REF]IBM BladeCenter JS12 Express blade product description[END_REF] with one dual-core POWER6 processor with a 3.8 GHz clock rate, hosting a SLES10 linux operating system. A single memory controller on the POWER6 processor orchestrates reads and writes between both cores and main memory. The memory subsystem has a 16 GB capacity, configured as eight DDR2 667 MHz 2 GB (1Rx4) DIMMS.

Workloads

We characterized the system response to memory throttling with a set of micro-benchmarks with specific characteristics. Each benchmark maintains steady operation during execution, providing a single point for comparison, without complicating the analysis with phases or other time-varying behavior. This small suite of microbenchmarks covers a wide range of memory characteristics.

Two micro-benchmarks use the same floating-point workload, DAXPY, with distinct memory footprints. The small data set for DAXPY-L1 fits within the level-1 data cache, while the DAXPY-DIMM footprint is 8 MB, forcing off-chip memory accesses. By performing the same computation with different data set sizes, we are able to isolate effects due to the memory subsystem behavior.

Computation within the FPMAC kernel is similar in nature to DAXPY; the primary difference is that the floating-point multiply and accumulate algorithm in FP-MAC computes and stores with a single array while the DAXPY implementation uses two arrays, providing a different flavor of memory access pattern with the same data set size (8 MB) and similar compute load.

The RandomMemory-DIMM micro-benchmark generates random address locations for reads and writes within an 8 MB memory footprint. The memory access patterns defeat prefetching that would benefit FPMAC and DAXPY kernels' regular access patterns, exposing the full effects of memory latency at each throttle point. The FPMAC, DAXPY, and RandomMemory kernels are short C programs with heavy computational load or intensive memory accesses and very little overhead.

We also use a single calibration phase of the Java benchmark SPECPower ssj2008 that continuously injects transactions as fast as the system can process them, executing for a fixed period of time to provide insight into the effects of memory throttling on transactional workloads.

Measurements

The POWER6 Blade system used for the experiments is instrumented with on-board power sensors including memory power, and event counters for throughput in units of instructions per second (IPS), memory reads, and memory writes (among others).

The blade power management controller obtains event counter data via a dedicated I2C port on the processor chip [START_REF] Floyd | System power management support in the IBM POWER6 microprocessor[END_REF], averages the data over 256 ms, and sends measurements via an Ethernet connection to a separate workstation, where a monitoring program dumps a trace file for our analysis, without interfering with the memory characterization experiments.

Throttling Characterization

We characterized the blade system's response to memory throttling by executing four copies of a micro-benchmark (two cores, two threads each) with a fixed quota-style throttle while we recorded power and performance data, then changed the throttle setting and re-ran the same workload. Throttle values range from 1 to 32, out of a window of 32 accesses. A 100% throttle setting of 32/32 is unthrottled, 16/32 is 50% throttled, etc. As memory throttle settings are successively lower, memory bandwidth is more extensively throttled.

The amount of work performed by DAXPY-L1, DAXPY-DIMM, FPMAC-DIMM, and RandomMemory-DIMM at each throttle setting remained constant, and execution time varied. The single calibration phase of the SPECpower ssj2008 benchmark executed for a fixed time duration, and the amount of work completed varied with memory throttle setting.

We summarized memory characterization data by calculating the median value for memory bandwidth, throughput (IPS) and memory power over the observed intervals for each permutation of workload and memory throttle setting.

Bandwidth

Figure 2 charts memory traffic normalized to the peak traffic observed over this suite, in DAXPY-DIMM. The curves show three distinct regions of operation: bandwidth-limited where the bandwidth is a linear function of memory throttle, a bandwidth-saturated region, and a transitional portion between the limited and saturated regions.

Bandwidth-limited

In the bandwidth-limited region, increasing the available bandwidth by increasing the throttle value translates directly to higher memory traffic. Changing the throttle value within this region directly affects bandwidth, and thus has a direct effect on both performance and power. The bandwidth-limited region may include only very-throttled operation, such as the case of SPECPower ssj2008 phase, or a wider range of throttle values, as in the case of the DAXPY-DIMM and FPMAC-DIMM benchmarks with large memory footprints.

Transition

The transition region is critical for power and performance management with memory throttling. In this region, changing memory throttle setting does affect memory traffic, but in a more complex manner than in the bandwidth-limited region. The uncertainty in the relation between memory throttle and bandwidth within this region, and in the extent of the region itself, create a challenge for managing power and performance.

Memory throttle values that bound transition regions vary by benchmark, with transitions at lower throttle values for less-demanding workloads and higher throttle values for memory-intensive workloads.

Each workload has a gap between the maximum available and consumed bandwidth in the transition region, and the extent of the gap varies. For example, at a 30% throttle, RandomMemory-DIMM has not reached its saturation level, yet it consumes less bandwidth than other workloads at the same throttle setting. The knee of the curve is sharper for some workloads than other: the FPMAC-DIMM micro-benchmark has a particularly sharp transition, while the SPECPower ssj2008 phase has a much more gradual transition. Workloads with sharper transition are able to use more of the available bandwidth at a given throttle setting, up to the point of saturation. Workloads with more gradual bandwidth roll-off have other bottlenecks that also factor into limiting the rate of memory requests.

While the bandwidth regions are clearly visible in off-line analysis of the full throttle range, it is difficult at run time to discern whether the current workload(s) are in the transitional area. For example, at the 50% throttle level in Figure 2, the observed bandwidth for DAXPY-DIMM and FPMAC-DIMM workloads are nearly identical, yet DAXPY-DIMM is in the linear bandwidth-limited region and FPMAC-DIMM is at a very sharp transition point. Without knowing the bandwidth trends from neighboring throttle points, a controller would not know whether to expect a linear, non-linear, or no change in bandwidth for an incremental change in throttle value.

Bandwidth-saturated

In the bandwidth-saturated region, the flat portions of each curve in the graph, memory traffic does not change with memory throttle settings. Each workload settles to a unique saturation level. Other bottlenecks and the workload's data footprint limit the memory request rate.

On the blade system used to collect the data shown in Figure 2, the memory bus limits the available bandwidth for memory throttle settings 75% and above, meaning that throttle settings between 75-100% provide essentially the same amount of available bandwidth.

The DAXPY benchmarks illustrate two ends of the saturation spectrum. Bandwidth consumed by DAXPY-DIMM approaches the architectural limit of the memory bus; the other benchmarks in this collection are limited by other factors. The cache-resident dataset of DAXPY-L1 naturally limits its memory request rate, and the consumed bandwidth is so low that it is independent of the memory throttle setting, essentially in the bandwidthsaturated region throughout the entire range.

Increasing memory throttle settings beyond the saturation level has a negligible effect on bandwidth. It follows that increasing memory throttle settings beyond the saturation level will not improve performance, or draw more power. For example, no memory throttle setting would have any bearing on DAXPY-L1, nor would modulating memory throttle settings between 40-100% for the SPECPower ssj2008 calibration phase recorded in Figure 2.

Performance

Figure 3 plots performance (IPS) as a function of the memory throttle setting. Data are normalized to the peak throughput of individual benchmarks to factor out the effect of disparate throughput levels among benchmarks in the suite.

One advantage of power-cap control rather than continuous control is that when memory traffic is less than the limit imposed by memory throttling, performance is unchanged. Cache-resident DAXPY-L1 throughput is unaffected by memory throttling.

At the opposite end of the spectrum, DAXPY-DIMM shows noticeable throughput loss for throttling up to 65%. Remember that at 75%, the memory bus becomes the dominant bandwidth-limiting factor.

DAXPY-DIMM (and other workloads with similar characteris-tics) are sensitive to the majority of memory throttling values in the useful range up to 75%, and almost any power control via memory throttling would directly degrade performance.

SPECPower ssj2008 tolerates memory throttling without serious performance loss down to about 40% throttled, below which it has non-linear performance loss with memory throttling throughout its wide transition region. Kernels like FPMAC-DIMM with very short transition regions are dominated by linear performance loss in the bandwidth-limited region.

Workloads with a sharp-knee characteristic would show no response to changes in memory throttling in their bandwidth-saturation regions, until an incremental step down in memory throttling level tipped them into a bandwidth-limited region and suddenly dropped in performance.

Power

Figure 4 confirms that memory power consumption is linearly proportional to memory bandwidth on our system. Data are normalized to the maximum observed memory power measurement, in DAXPY-DIMM. The near-zero bandwidth requirements of the cache-resident DAXPY-L1 show that about 40% of memory power is not under throttle control in this system.

Measured memory power data points normalized individually per benchmark, shown in Figure 5, demonstrate where opportunity for power control lies. Memory throttling offers essentially no power control for corebound workloads such as DAXPY-L1, a small range of control for moderate-intensity workloads such as SPECPower ssj2008, and a large swing for memory intensive workloads such as DAXPY-DIMM and FPMAC-DIMM that have larger unthrottled memory power consumption.

Since quota-style memory throttling enforces an upper bound on power consumption, actual memory power consumption will be in the range between the static power levels and the memory power cap, depending upon run-time bandwidth demands.

Conclusion

Memory throttling exists in various forms in commercial systems, yet it has garnered little attention in architecture studies to date. Memory throttling can be used to enforce memory power budgets, enabling large memory configurations that would violate power constraints if left unthrottled, and also supporting dynamic techniques such as power shifting.

As with nearly all power management options, memory throttling comes with the price of performance penalties in some situations. We point out the regimes of power control with no performance loss, and where more extensive power reduction does degrade performance.

This paper characterized the effects of memory throttling on throughput performance and memory power on a commercial blade server system. We demonstrated the three regions of bandwidth response: bandwidthlimited, transition, and bandwidth-saturation. Understanding these regions and the workload characteristics that determine the interaction between throttle settings and bandwidth restriction enables wise choices in memory power management design.

Figure 2 :

 2 Figure 2: Memory traffic: total read and write accesses normalized to peak observed traffic, unthrottled DAXPY-DIMM. Each workload has unique characteristics for the three throttle regions: bandwidth-limited, transition, and bandwidth-saturated.

Acknowledgments

Thank you to our colleagues who contributed technical expertise and system administration support, especially Joab Henderson, Guillermo Silva, Kenneth Wright, and the power-aware systems department of the IBM Austin Research Laboratory.


Laura Keys
email: laurak@eecs.berkeley.edu

Suzanne Rivoire
email: suzanne.rivoire@sonoma.edu

John D Davis
email: john.d@microsoft.com

The

The Search for Energy-Efficient Bu ilding Blocks for the Data Center

teaching and research institutions in France or abroad, or from public or private research centers.

Introduction

Power consumption is a first-order design constraint in the data center (DC). Although still small in absolute terms, DC power consumption is growing rapidly, doubling between 2000 and 2005 [START_REF]Report on server and data center energy efficiency[END_REF]. The energy usage of the computational building blocks of the DC is critical to the overall power consumption, since it affects the design and operation of the cooling and power distribution infrastructure as well as the computational infrastructure [START_REF] Barroso | The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines[END_REF][START_REF] Koomey | Estimating total power consumption by servers in the U.S. and the world[END_REF][START_REF] Poess | Energy cost, the key challenge of today's data centers: a power consumption analysis of TPC-C results[END_REF].

Traditionally, the computational nodes in DCs operate with low system utilization but require high availability and fast response time. Researchers have therefore advocated the design of hardware whose power consumption is proportional to the system load [START_REF] Barroso | The case for energyproportional computing[END_REF]. However, there is a new class of DC benchmarks that use as many resources as are available. Many of these applications are I/O-and network-bound but exhibit phases of high CPU utilization. Dryad, Hadoop, MapReduce, and Condor are frameworks for this type of application [START_REF] Dean | MapReduce: simplified data processing on large clusters[END_REF][START_REF]Hadoop wiki[END_REF][START_REF] Isard | Dryad: distributed dataparallel programs from sequential building blocks[END_REF][START_REF] Thain | Distributed computing in practice: the Condor experience[END_REF].

In the past, research on these data-intensive workloads has assumed that the applications would be bottlenecked by low I/O bandwidth and high latency. However, the introduction of NAND flash-based solidstate drives (SSDs) virtually eliminates the disk seek bottleneck, enabling much higher I/O bandwidth and very low latency. Although SSDs do not yet provide the capacity of magnetic disk drives, SSDs can be very lowpower devices and have the ability to consolidate the storage system by providing far more IOPS, better feeding the processor with data [START_REF]Intel X18-M/X25-M SATA solid state drive product manual[END_REF].

In this paper, we characterize clusters across a variety of system types in order to find energy-efficient DC building blocks, with a focus on emerging dataintensive applications.

We initially characterize a variety of embedded, mobile, desktop, and server systems using singlemachine performance, power, and energy efficiency. Using these benchmarks as a guide to prune the system space, we build homogeneous clusters of the top three systems. We execute DryadLINQ applications on these clusters in order to understand their energy efficiency for different application types.

This paper makes the following contributions:  We characterize a wide range of real systems from embedded and mobile to desktop and server processors, focusing on single-thread and/or singlesystem performance.  We characterize homogeneous compute clusters composed of embedded, mobile, and server processors in the context of data-intensive applications to find the most energy-efficient computing infrastructure over a wide range of workloads.  We compare the energy efficiency of system classes that have not been compared in previous work, and we make this comparison across workloads with varying computational and I/O demands. The rest of this paper is organized is follows. Section 2 is an overview of related work in this area. Section 3 describes our experimental infrastructure and the hardware and software evaluated. Section 4 presents our experimental results. We further discuss these results in Section 5 and conclude with Section 6.

Related Work

A growing body of research proposes energy-efficient building blocks for cluster and DC computing, but this work has typically investigated only a limited subset of system types and/or applications.

One major trend is to propose building blocks for data-intensive computing that combine embedded processors, such as the Intel Atom, with solid-state disks. However, many of these proposed systems have been evaluated for only a single workload or against a limited set of alternative hardware.

For example, Szalay et al. propose "Amdahl blades," consisting of Intel Atom processors with SSDs, and present a scaling study comparing these blades to traditional high-end cluster nodes using data from a synthetic disk-stressing benchmark [START_REF] Szalay | Low-power Amdahl-balanced blades for data intensive computing[END_REF]. The Gordon system, designed by Caulfield et al., combines Atom processors with flash memory. It was evaluated against a Core 2 (single-core)-based server over a variety of MapReduce workloads using simulation and modeling rather than physical measurements [START_REF] Caulfield | Gordon: using flash memory to build fast, power-efficient clusters for data-intensive applications[END_REF].

The FAWN cluster, proposed by Andersen et al., consists of ultra-low-end embedded processors and high-end solid-state disks [START_REF] Andersen | FAWN: a fast array of wimpy nodes[END_REF]. A version using the Intel Atom was evaluated across a wide range of workloads [START_REF] Vasudevan | Energy-efficient cluster computing with FAWN: workloads and implications[END_REF]. This evaluation showed FAWN breaking the energy-efficient sorting record set by Beckmann in 2010 with similar hardware [START_REF] Beckmann | Energy-efficient sorting using solid state disks[END_REF]. The overall conclusion of the evaluation was that the FAWN hardware was superior to desktop-and server-class hardware for I/O-bound workloads and for memory-bound workloads with either poor locality or small working sets. However, high-end mobile processors were not evaluated in the FAWN study. Reddi et al. use embedded processors for web search and note both their promise and their limitations; in this context, embedded processors jeopardize quality of service because they lack the ability to absorb spikes in the workload [START_REF] Reddi | Web search using mobile cores: quantifying and mitigating the price of efficiency[END_REF].

Several studies have proposed high-end laptop hardware for energy-efficient DC computing. Rivoire et al. used a laptop processor and laptop disks to set an energy-efficient sorting record in 2007 [START_REF] Rivoire | JouleSort: a balanced energy-efficiency benchmark[END_REF], while Lim et al. proposed a laptop-processor-based building block for Web 2.0 workloads in 2008 [START_REF] Lim | Understanding and designing new server architectures for emerging warehouse-computing environments[END_REF]. However, these systems preceded the movement toward embedded processors and SSDs, and their conclusions must be revisited in light of these recent developments.

Finally, the CEMS servers proposed by Hamilton use a variety of desktop processors and a single enterpriseclass magnetic disk [START_REF] Hamilton | CEMS: low-cost, low-power servers for internet-scale services[END_REF]. These servers are evaluated using a CPU-bound webserver workload designed to exercise the CPU at varying utilizations up to 60%. Unlike much of the previous work, this study found that for this workload, the systems with the lowest power consumption were not the most energy-efficient systems.

System Overview

In this section, we describe the hardware platforms we examine, the benchmarks we use to evaluate them, and the infrastructure used to measure power.

Hardware

We consider a variety of systems based on embedded, mobile, desktop, and server processors. Table 1 provides a list of the important features of the systems under test (SUTs). All systems are running 64-bit Windows Server 2008 with support for Dryad and DryadLINQ jobs. We tried to provision the systems with 4 GB of DRAM per core when possible, but two of the embedded systems were only able to address a fraction of this memory. The industry-standard server system used 10,000 RPM enterprise hard disks, and the other systems each contained a single Micron RealSSD. This difference affected the server's average power by less than 10% and had a negligible effect on the system's overall energy efficiency.

Benchmark Details

We ran an assortment of benchmarks, some CPUintensive, others utilizing disk and network, in order to find the most energy-efficient cluster building block and see how robust this choice is across different types of workloads. A few of these benchmarks are used to evaluate single-machine performance, and the rest are DryadLINQ jobs dispatched to five-node clusters. We ran a single instance of each application at a time.

The single-machine benchmarks are as follows:  SPECpower_ssj 2008. This benchmark uses a CPUand memory-intensive Java webserver workload to probe the power usage of a SUT's CPU at various utilizations. Since the performance of this benchmark can vary drastically depending on the JRE used, we use the Oracle JRockit JRE tuned with platform-specific parameters based on similar reported benchmark runs.

Measurement Infrastructure

The measurement infrastructure consists of a hardware component to physically measure both the total system power and power factor and a software component to collect both the power measurements and application-level Event Tracing for Windows (ETW) metrics.

We use WattsUp? Pro USB digital power meters to capture the wall power and power factor once per second for each machine or group of machines. We use the API provided by the power meter manufacturer to incorporate measurements from the power meter into the ETW framework.

Evaluation

In this section, we first examine the single-machine performance of a range of machines. We use these results to identify the three most promising candidate systems for the cluster-level benchmarks. The results from both the single-machine and multi-machine benchmarks show that the mobile-class system consistently provides high energy efficiency on a wide range of tasks, while the other classes of systems are suitable for a more limited set of workloads.

Single-machine benchmarks

To pare down our list of systems, we used three single-machine benchmarks to characterize the systems' single-thread performance and power consumption. Based on this characterization, we can eliminate any systems that are Pareto-dominated in performance and power before proceeding to the cluster benchmarks.

Performance

We use SPEC CPU2006 integer benchmarks to compare the single-threaded SPEC-rate performance across all the platforms in Table 1. This benchmark, because it is CPU-intensive, should favor the processors with more complex cores. In addition to the dual-socket quad-core AMD Opteron server in Table 1 (SUT 4), we included two more Opteron servers: a dual-socket single-core server (2x1) with 8 GB of RAM and a dualsocket dual-core server (2x2) with 16 GB of RAM. These systems were included to quantify single-core performance improvements over time, as well as the benefits of additional cores. Figure 1 shows the per- core results, which are normalized to Intel Atom single-core-based system (SUT 1A).

There are two surprising results. First, the mobile Intel Core 2 Duo (SUT 2) has per-core performance that matches or exceeds that all of the other processors, including the server processors. Second, and more surprising, is the fact that the Atom processor performs so well on the libquantum benchmark. Overall, these results demonstrate that SUT 2 (Intel Core 2 Duo) and SUT4 (AMD Opteron 2x4) provide the highest singlethread performance.

Power consumption

Single-thread performance is not the only factor to consider when selecting the appropriate energy-efficient building blocks for the DC. Before diving into benchmarks that provide data on work done per Watt or Joule, we measure system power at idle and when running CPUEater at 100% utilization. Figure 2 shows power consumption at these two utilization points for all of the systems from Figure 1, ordered by the maximum system power under full CPU load. Surprisingly, the four embedded-class systems do not have significantly lower idle power than the other systems; in fact, the mobile-class system with a 25 W TDP processor has the second-lowest idle power. However, the 100% utilized systems result in a different ordering. The mobile-class system now has significantly higher power than the embedded systems, which use processors with 4-16 W TDPs.

Balancing performance and power

To confirm our conclusions based on examining performance and power separately, we used SPECpower_ssj to characterize the amount of work or operations done per watt. As Figure 3 shows, the Intel Core2 Duo system (SUT 2) and the Opteron (2x4) system (SUT 4) yield the best power/performance, followed by the Atom system (SUT 1B). These results reinforce our conclusions from looking at power and performance separately. Furthermore, this benchmark goes beyond the single-core performance measured by SPEC CPU 2006.

Multi-machine Dryad benchmarks

Based on the characterization from Section 4.1, we set up 5-node clusters of the three most promising systems (1B, 2, and 4) and ran the four DryadLINQ benchmarks: Sort, Primes, StaticRank, and WordCount.

Figure 4 shows the average energy usage for these benchmarks, normalized to the mobile system (SUT 2). It shows two versions of Sort that only differ by the number of data partitions, 5 or 20; the 20-partition version has better load balance.

The energy usage per task of SUT 2, the mobile Core 2 Duo-based server, is always lower than that of SUT 4, the Opteron-based server, across all the benchmarks, using three to five times less energy overall for the different benchmarks.

The relative energy usage of SUT 1B, the Atombased system, varies the most from benchmark to benchmark. It degrades significantly for Primes, which is the most CPU-intensive benchmark.

For this benchmark, the traditional server system (SUT 4) is more energy-efficient than the Atom-based system. 4. Normalized average energy usage for SUT 2, SUT 1B, and SUT 4 for each benchmark on each system and the geometric mean. SUT 4 has a performance advantage with four times the number of cores, enabling it to finish parallel and computationally intense tasks more quickly but with a significantly higher power envelope than SUT 1B. This advantage disappears, however, for StaticRank, which has a mix of CPU and I/O. SUT 4 can finish this job only slightly faster than SUT 2 or 1B, but it uses much more power. However, it should be noted that the partition size used for StaticRank is set by the memory capacity limitations of the mobile and embedded platforms. This biases the results in their favor, because at this workload size, SUT 4's execution is dominated by Dryad overhead.

More surprisingly, the Atom-based system is less energy-efficient for Sort than the mobile-CPU-based system. Previous work on platforms for sequential I/Ointensive workloads used Atom-based systems on the assumption that the I/O would be the bottleneck and the CPU would thus not be heavily utilized [START_REF] Beckmann | Energy-efficient sorting using solid state disks[END_REF][START_REF] Szalay | Low-power Amdahl-balanced blades for data intensive computing[END_REF][START_REF] Vasudevan | Energy-efficient cluster computing with FAWN: workloads and implications[END_REF]. However, the SSDs in these systems mitigate this bottleneck for Sort, placing more stress on the CPU. In contrast, the Atom-based system is most energyefficient for WordCount, which is the least CPUintensive of the four benchmarks.

These energy measurements on cluster benchmarks complement the results on single-machine benchmarks: low-power mobile class platforms have an advantage over high-power, high-performing server-class platforms as energy-efficient DC building blocks that do not skimp on performance. Their performance and power also are more robust over a wider range of benchmarks than the embedded-class systems.

Discussion

The results demonstrate a clear class of systems that is well suited for data-intensive computing. This result is somewhat surprising due to the interface limitations of real mobile-class systems. We discuss this result in more detail, and we follow that discussion with some of the system improvements that would be necessary to build a more compelling energy-efficient system, requiring minor modifications to today's components.

Energy Efficiency

Our results show that low-power embedded components are not necessarily ideal for energy efficiency, even for applications that are not normally considered CPU-intensive (e.g. Sort). With the increase in I/O capabilities provided by SSDs, our results indicate that embedded-class processors are not always sufficient to balance the I/O bandwidth. In fairness, one disadvantage that these systems had is that the chipsets and other components dominated the overall system power; in other words, Amdahl's Law limited the benefits of having an ultra-low-power processor. As the non-CPU components become more energy-efficient, this type of system will be more competitive.

Our results also confirm that standard servers are becoming more energy-efficient. We presented results from three consecutive generations of Opteron servers running SPEC benchmarks. Over time, these systems have maintained or improved single-thread performance, increased system throughput, and simultaneously reduced overall system power and energy. Until recently, embedded systems were the only systems that exhibited the same trends. This is a result of combining lower-power server processors with efficient power supplies and related components. However, there still is a long way to go.

The missing links

Research on energy-efficient DC building blocks has largely been limited to evaluations of existing hardware. While simulation provides the flexibility to derive any reasonable system imaginable, the runtimes for the applications used in this study make simulation of any type prohibitively expensive. For this data-intensive benchmark suite, the wall-clock runtime varied from just over 25 seconds (WordCount on SUT 4) to ~1.5 hours (StaticRank on SUT 1B). Therefore, this study was constrained to use existing hardware. However, there are several clear improvements that could be made to increase the energy efficiency of future datacenter hardware.

First, the embedded and mobile systems had very restrictive I/O subsystems, limited by the number of ports and overall bandwidth. Likewise, the network is also a limiting factor, which can be solved with more energy efficient designs and higher bandwidth, like 10 Gb solutions.

Finally, only configurations 3 and 4 supported ECC DRAM memory. Memory is the conduit to the processor, and memory errors are on the rise, especially for large systems [START_REF] Hamilton | CEMS: low-cost, low-power servers for internet-scale services[END_REF][START_REF] Yelick | How to waste a parallel computer[END_REF]. We view ECC as a requirement for any data-intensive computing system.

Our ideal system would couple a high-end mobile processor (like the Intel Core 2 Duo or AMD equivalent) with a low-power chipset that ECC for the DRAM, larger DRAM capacity, and more I/O ports with higher bandwidth.

Conclusions

Our results from small clusters demonstrate that systems built using high-end mobile processors and SSDs are the most energy-efficient system for dataintensive cluster computing across all the applications we tested. We compared systems across the spectrum of available hardware, including systems advocated by other researchers proposing solutions to this problem [START_REF] Szalay | Low-power Amdahl-balanced blades for data intensive computing[END_REF]. A concern with ultra-low-power embedded systems is that the chipset and peripherals can dominate the overall power usage, making these systems less energy-efficient than their processors alone. Our results also show that the successive generations of server systems are becoming more energy-efficient, as we expected. We were able to use single-threaded and single system benchmarks to filter the systems down to a tractable set in order to run a variety of large-scale benchmarks. The initial benchmark results were consistent with the data-intensive benchmark results. Moving forward, we expect that embedded processor systems will be overpowered by their I/O subsystem requirements for data-intensive applications in the near future. Furthermore, by optimizing the chipset and peripherals, even more energy-efficient systems can be built for this application space. These systems will use less power, reducing overall power provisioning requirements and costs.

Finally, there is a large body of future work that we would like to pursue. First, we would like to use OSlevel performance counters to facilitate per-application modeling for total system power and energy. Furthermore, we know of no standard methodology to build and validate these models. Likewise, developing standard metrics and benchmarks will make these comparisons easier in the future. These are essential for a framework that includes standardized metrics and benchmarks for system comparison.

Figure 1 .

 1 Figure1. Per-core SPEC CPU2006 integer performance normalized to the Atom N230 for the systems (embedded, mobile, desktop, and server processors) from Table1plus two legacy Opteron servers.

Figure 3 .

 3 Figure3. SPECpower_ssj results for four of the systems from Table1plus the two previous generations of Opteron servers.

Figure 2 .

 2 Figure 2. Power consumption at idle and at 100% CPU utilization for all the systems in Figure 1. The systems are shown in order from lowest to highest power consumption at 100% utilization.

Table 1 -

 1 Systems evaluated in this paper. Costs are approximate and given in US dollars at the time of purchase. Costs are not given for systems that were donated as samples. In the memory column, the star denotes the maximum amount of addressable memory.

	System Under Test	CPU	Memory	Disk(s)	System Information	Approx. cost
	1A (embedded)	Intel Atom N230, 1-core, 1.6 GHz, 4W TDP	4 GB DDR2-800	1 SSD	Acer AspireRevo	$600
	1B (embedded)	Intel Atom N330, 2-core, 1.6 GHz, 8W TDP	4 GB DDR2-800	1 SSD	Zotac IONITX-A-U	$600
	1C (embedded)	Via Nano U2250, 1-core, 1.6 GHz	2.37 GB DDR2-800*	1 SSD	Via VX855	sample
	1D (embedded)	Via Nano L2200, 1-core, 1.6 GHz	2.86 GB DDR2-800*	1 SSD	Via CN896/VT8237S	sample
	2 (mobile)	Intel Core2 Duo, 2-core, 2.26 GHz, 25W TDP	4 GB DDR3-1066	1 SSD	Mac Mini	$1200
	3 (desktop)	AMD Athlon, 2-core, 2.2 GHz, 65W TDP	8 GB DDR2-800	1 SSD	MSI AA-780E	sample
	4 (server)	AMD Opteron, 4-core, 2.0 GHz, 50W TDP	32 GB DDR2-800	2 10K RPM	Supermicro AS-1021M-T2+B	$1900

  Table 1 plus the two previous generations of Opteron servers.

	Performance to Power Ratio	(SSJ operations/W)	Intel Core2Duo, SUT 2 AMD Opteron (2x4), SUT 4 Atom (2-core) SUT 1 B AMD Athlon, SUT 3 AMD Opteron (2x2) Atom (1-core), SUT 1A AMD Opteron (2x1)
		Idle	10%	20%	30%	40%	50% CPU Utilization	60%	70%	80%	90%	100%


Alan G Yoder

Energy Efficient Storage Technologies for Data Centers

An impressive amount of work has been done to date on improving the electrical efficiency of various data center components. The data storage industry has begun to see the fruits of this effort, with increased power supply and fan efficiencies. However, storage presents other significant opportunities for energy conservation, through various types of capacity optimization, that are not captured in electrical efficiency discussions. As data storage uses on the order of 25% of IT power in an average data center, these other opportunities bear examination. This article presents a survey of emerging storage technologies which positively impact energy usage and presents current thinking in the storage industry regarding their relative effectiveness. It also attempts to set a baseline for configurations against which improvements in capacity and energy use can be made.

Introduction

The idea of optimizing storage is as old as data processing. Compression is one of the oldest and most venerable, as well as one of the best understood computational disciplines. Newer techniques such as data deduplication and advanced RAID, however, have only recently become well enough understood to admit of industry standard definitions [START_REF]The SNIA Dictionary, a lexicon of terms related to the storage industry[END_REF]. Many of these technologies offer far greater gains in efficiency than can be gotten via electrical optimizations (e.g. power supply efficiency, component idle modes). However, the gains offered by various technologies are confusing. Some, such as thin provisioning, are much better than it might seem at first blush. Others are much worse. Disk spindown, for example, has seen much interest, but few products have successfully shipped, and existing research leaves open the question of what to do about background data housekeeping tasks. Data protection requirements add further complexity to the modern data center storage picture. Many academic lab environments are content with relatively simple backup schemes. But advanced data centers often have recovery point objectives (RPOs) of zero, meaning that no transaction loss is acceptable. And their recovery time objectives (RTOs), the time allowed for recovery from a fault, may be minutes or even seconds. Even at less stringent sites, planned outages longer than a standard network delay window are often unacceptable. In addition, recent studies of power consumption in storage systems have outlined the complexity of behavior in the field, including the fact that maximum performance and maximum power may not match [START_REF] Allalouf | Storage Modeling for Power Estimation[END_REF] and that interactions between storage and servers must often be considered together [START_REF] Lange | The Next Frontier for Power/Performance Benchmarking: Energy Efficiency of Storage Subsystems[END_REF]. These issues have caused a disconnect between industrial and academic conceptualizations of exactly what "storage" means. Section 1 outlines a set of "standard" requirements for data center storage, with the reasoning behind them. Section 2 presents the storage optimizing technologies found noteworthy by the Storage Networking Industry Association (SNIA). Section 3 discusses the efficiency gains achievable by these technologies, as measured against the base configurations outlined in Section 1. Section 4 concludes. It should be noted that this paper deals almost exclusively with technologies and configurations suitable for primary storage. Space restrictions prevent a full discussion-which could easily equal this one in size-of secondary storage.

Section 1. Base storage configurations in industrial data centers

In the home and small or home office business sectors, almost all the data from almost all disk crashes can be recovered by a disk forensics expert for a fee of about $1000. This makes expensive RAID arrays and backup systems seem unnecessary. Indeed, a system like Apple's Time Machine 1 with a decent power conditioner and UPS, and an offsite copy of the Time Machine disk 1 All trademarks mentioned herein are the property of their respective owners.

Energy Efficient Storage Technologies for Data Centers

Alan G. Yoder, Ph.D.

NetApp, Inc. Storage Networking Industry Association (SNIA) to guard against site wide catastrophe, can seem like all the data protection an average small office needs. But this level of protection is inadequate for systems that must support 24/7 operation and thousands to hundreds of thousands of users. For Data Center class storage, the acronym RAS (reliability, availability and serviceability) is used to describe the requirements. In plain English, we can characterize these three qualities as follows:

• Reliability means that the system rarely if ever goes down or loses data. • Availability means that when it does go down, it doesn't stay down very long. A system that meets the "five nines" standard has an average downtime of less than 5.26 minutes per year. • Serviceability means the system can be serviced-replacing broken parts, upgrading firmware-without impacting availability.

Serviceability and availability requirements combine to mean that data center class equipment must have no single point of failure (no SPOF). The power supplies in such a piece of equipment are hot swappable and redundant, and therefore don't run at more than 50% capacity under normal circumstances. This means efficiency work should focus on 10-50% utilization scenarios. Variable speed fans are also critical to good efficiencies, as they too are redundant. The reliability and availability requirements lead to a downstream requirement for redundancy in the storage media. This has led to the use of RAID 1 (mirrored disks) in data centers as a policy mandate. Not surprisingly, data center class disk and system vendors have historically not struggled much against this seeming waste of rotating media. The availability requirement also means that some means of restoring quickly to a good copy of a dataset must be available. This is usually done through the use of point-in-time (PIT) copies, termed snapshots or clones. These are often augmented by write journals-also stored on redundant media-that can be replayed against the PIT copy to restore the dataset to its exact state at the time of the failure. Finally, a significant percentage of data center class gear is used for structured data, which is, at a first order, data generated and maintained by databases. These databases frequently are configured to use raw storage. Storage area networks (SANs) have provided data center operators with configurations that can be run and backed up in a more centralized and efficient manner than direct-attached disks. However, given that the SAN-attached array must emulate directattached disk, this class of equipment has traditionally required the use of high-performance fibre-channel (FC) or serial-attached SCSI (SAS) disk drives. Comparison of costs between these drives and the SATA drives available at Fry's Electronics neglects key differences and technology design choices [START_REF] Anderson | More Than an Interface -SCSI vs[END_REF], [START_REF] Whittington | Desktop, Nearline & Enterprise HDDs What's the difference?[END_REF]. The point of this exposition is that when comparing configurations in a data center context, the RAS requirements must be used to establish a baseline configuration against which improvements can be evaluated. The SNIA baseline configuration can be summarized as "no SPOF, FC drives, RAID 1".

Section 2. Storage optimizing technologies

While "moving the needle" (i.e. using less power to accomplish tasks) is important, there are other ways of saving not only energy, but equipment assets as well. If an array can be made to store twice as much data, the hardware savings are obvious. For this reason, updating arrays to new drive technologies every several years remains a tried and true way of saving on both equipment and power (the gains gotten from drive generations are so obvious and longstanding that they don't appear on the following list). SNIA's analysis indicates that there are five major capacity optimizing software technologies

Facilities optimization

The PUE metric [START_REF]PUE is discussed in the Metrics and Measurements whitepaper available[END_REF] developed by The Green Grid [START_REF]The Green Grid, an industry consortium[END_REF] nicely demonstrates the gains possible when optimizing facilities. Briefly, the PUE is the ratio between the amount of power entering a facility and the amount of power reaching the IT equipment. Traditional data centers often had PUEs of 2.5 and higher, meaning that 40% or less of the power actually reached the equipment it was intended to drive. Modern innovations include flywheel UPSs, air economizers, variable speed fans, hot aisle/cold aisle technologies, and improved monitoring and feedback control systems. Collectively, these have allowed the construction of facilities with PUEs of 1.25 on a routine basis, with better than that achievable. This is a doubling of overall power efficiency, more than can be gotten from any IT-side adjustment.

Power supply and fan efficiency

Traditional power supplies have had efficiencies in the 60-70% range. Efforts such as the 80 Plus Program [START_REF]The 80 Plus program[END_REF] and the US EPA's ENERGY STAR program [START_REF]ENERGY STAR Data Center Storage Specification. Under development[END_REF] have resulted in efficiency targets in the 80-95% range, depending on load.

Variable speed fans would seem to actually offer greater gains than this, because of the nearquadratic reduction in power required as they spin slower. Unfortunately, although the ENERGY STAR program for home PCs and small servers requires use of these, data on the amount of power that has been actually saved is scarce. More quantified study in this area needs to be done before even educated guesses are appropriate. 2 The word "ontology" is eschewed by this author.

Disk spindown

Spinning down unused disk drives has received a lot of attention, relative to other technologies. However, all analyses in the literature that the author has seen, such as PARAID [START_REF] Weddle | PARAID: A Gear-Shifting Power-Aware RAID[END_REF] and SRCMap [START_REF] Verma | SRCMap: Energy Proportional Storage using Dynamic Consolidation[END_REF], make simplifying assumptions that render the research unusable for primary storage.

First, what to do about background housekeeping activities such as parity and RAID scrubbing, absolutely essential in an enterprise-class array, is left as an open question. Second, accesses to spun down drives can cause latencies of 10 to 15 seconds. This relegates these systems to secondary storage (the SNIA defines "online", i.e. primary storage as having a max time to first data of 80ms).

One possible use for spindown in primary storage systems is for hot spares. A data center class array may have 1-2% or even more of its disks in hot spare mode, so that amount of the disk (and possibly the disk drawer fan) power can be reclaimed by this method.

SSDs

Solid state disks (SSDs) appear to have nearly ideal energy characteristics: energy used is proportional to IOPs, and data at rest is nominally free. This is a huge improvement over rotating media, which uses about 85% as much power at idle as it does when active [3]. But SSDs are not presently a viable drop-in replacement for traditional storage media, as the price points are simply not there. At present, they are useful for caching and storage tiering. This yields a power difference per unit data of about 6.05. In other words, it takes one sixth as much power to store data at idle on SATA as it does on FC. This is a significant savings in the power required for data at rest. But it has traditionally come at a significant cost-a reduction in response time and bandwidth, as shown in the table. This has historically made arrays using SATA drives useful mainly for secondary storage and specialized applications such as video capture. However, recent innovations in flash and solid state storage have enabled the construction of disk arrays consisting mainly of SATA drives, with large tertiary caches of flash or SSDs, and even quaternary caches of FC or SAS drives. These hybrid systems approach or even in some circumstances exceed the performance of FC and SAS arrays, at price points below those of FC or SAS storage. Given the present cost differential between traditional drives and SSDs, which shows few signs of narrowing significantly in the near future, it is possible that this class of system will become predominant until the value proposition of pure SSD storage becomes more compelling. The topic of optimal sizing of the various elements in a DRAM/NVRAM/flash/SSD/rotating media storage hierarchy is of great present interest to the industry. The SNIA's Solid State Storage Initiative (SSSI) is attempting to address the space, and has projects underway in several areas. [23]

High capacity drives with flash

Section 2b. Use of less redundancy

As explained in the first section, redundancy is a requirement for data center class data protection. This is reflected even in the Uptime Institute's data center tier framework; tier 3 data centers have redundant UPSs, while tier 4 data centers have redundant power coming from entirely separate power grids and suppliers [8].

RAID

The data protection requirements in a data center result in a requirement to survive the loss of any one storage media device in a RAID array3 without loss of data or access to data (a small performance degradation during "RAID rebuild" is considered acceptable). RAID 1-mirrored storage-achieves this admirably. It can survive the loss of many drives in an array, in fact, so long as no two of them belong to the same mirrored pair. RAID 5 (and the less common 4 and 3) also offer protection from a single disk failure per RAID group. However, the RAID groups are larger (typically 5 to 8 for RAID 5 vs. 2 for RAID 1), 4and reconstruction of a RAID group may affect performance more than in a mirrored array. These RAID group formats cannot tolerate as many failures overall as RAID 1, because of the larger RAID group sizes. Nonetheless, there are many scenarios in which data center operators decide that RAID 5 is worth the additional risk over RAID 1 to save on the number of drives that must be bought. Reductions of 37% or more are easily possible (5 disks in a RAID 5 group vs. 8 RAID 1 disks), with a corresponding reduction in the required fan and cooling power needed. RAID 6, when acceptable performance is offered, is another alternative. Due to the mathematics, larger RAID groups are used, and the economies work out roughly the same as for RAID 5. However, availability is better than for RAID 5 or even RAID 1. First, RAID 6 protects against the chance-becoming more significant with every disk drive generation-that a second drive will fail during a RAID reconstruct of today's large SATA drives. Second, in a group of 16 drives, any two can fail under RAID 6, while under RAID 1 7%, i.e. 1/(N -1) of the second failures would take out a mirrored pair, bringing down the array.

Delta Snapshots

Standard snapshots are point-in-time (PIT) copies. High-end storage arrays offer the ability to restore access quickly after a fault by remounting a broken volume on a recent snapshot, and possibly replaying a log. Delta snapshots work on a principle very similar to the vfork method used in Unix process management: pages (blocks) that are shared between the live system and the snapshot are not copied; blocks and the inodes or other metadata pointing to them are only copied when new data is written. Again, only anecdotal evidence is available, but for a snapshot kept one day, it appears to be quite rare that more than 10% of the data in it has been written to. A space savings estimate of at least 90% is therefore appropriate for a delta snapshot.

Section 2c. Committing less space

Storage containers are often over allocated to protect against future unplanned outages. In addition, much storage is wasted by never being allocated in the first place-in a sense the storage in the array itself has been over allocated. While there are no conclusive research results that we know of, anecdotal and widely accepted evidence gives a figure of roughly 30% for storage utilization across the industry [START_REF] Tyrrell | storage architect and veteran of over 600 data center visits[END_REF]. That is 30% of the usable capacity of that storage. Working backwards, storing 30 TB of data requires 100TB of formatted storage, which under RAID 1 would require 200TB of raw storage plus up to 10TB for the system. That means users of traditional storage may be paying for about 7 times as much capacity as they are using.

Thin provisioning

The technique of thin provisioning avoids much of this cost. The system allocates storage on demand, and the storage admin tracks the total amount of storage used and adds more as it is required. As with quotas on filesystems, the total amount of quota or allocation given out may greatly exceed the amount of real storage backing it. Estimates of utilization when thin provisioning is in place tend to run to 80%. The SNIA has begun using a conservative figure of 100% as the amount of gain to be expected in an average scenario (going from 35% to 70% utilization) [START_REF]SNIA Tutorials[END_REF].

Section 2d. Squeezing more data into available space

Finally, the data itself often offers opportunities for further reduction in space requirements. Compression ratios of 2:1 are commonly accepted in tape backup, but tape is a streaming format well suited to compression. Compressing to blockoriented storage is more difficult and less efficient on account of quantization roundoff, but is now being shipped in the marketplace. Data deduplication is in some ways more suited to block storage. The greatest gains are seen in backup scenarios, where similar datasets are repeatedly written to online or near-online media. But as these scenarios compete with tape, which uses no energy on the shelf, they are generally left out of "green" discussions. However, in-place dedup of live data is also an increasingly available alternative. Savings estimates vary considerably depending on data set and on whether any financial guarantees are attached to them. A recent vendor announcement gives a figure of 27% for an exabyte of field data.

[22]

Section 3. Savings

The following table summarizes the approximate savings available via the technologies discussed herein, as determined by the Green Storage Technical Working Group of the SNIA. The numbers given are therefore industry consensus numbers. They are relative to the component, so facilities optimization applies to the whole facility, for example, while power supply savings apply only to that percent of energy used by the power supplies and the things they power in the facility. There is no good way to tell how these savings will combine when multiple technologies are implemented at once. A marketplace-based metric, however, suggests that 50% is a conservative figure for the software technologies alone. Multiple vendors are now guaranteeing that their systems will use 50% less storage than a competing system without these technologies. While rigorous analyses and data comparisons are desirable, these have proven problematic to obtain. First, data sets vary; there is no known predictor for the energy and space consumption characteristics of a specific application on an arbitrary system. Second, large storage arrays are expensive and hard to procure. Third, storage vendors are notoriously shy about publication of performance statistics; non-disclosure agreements commonly prohibit equipment users from publishing the results of their internal research. For these and other reasons, the SNIA has so far used a consensus-based approach to obtain numbers when publishable empirical studies have been unavailable.

Section 4. Conclusion

Before concluding, a caveat. This article has not much discussed the performance implications of various technology choices, except peripherally. In general, thinking in terms of energy efficiency for data in flight is in its infancy. It is too early to discuss such efforts definitively. Even the SSD space is moving so rapidly that rational prediction is difficult. As a result, the various technologies discussed herein mostly pertain to the data at rest axis of the storage performance space. In other words, we have focused on quantities with units like energy$$/GB, where data in flight metrics would use units like energy$$/IOP [START_REF]SPC Benchmark 1/EnergyTM (SPC-1/ETM), Overview Presentation[END_REF].

In reality, energy use is just one vector in a manydimensional storage procurement decision space. The capacity optimizing technologies treated herein, however-delta snapshots, thin provisioning, advanced RAID, deduplication, compression, and the use of slower high-capacity drives with large caches-do not only decrease energy cost. They do so by allowing the purchase and powering of less storage media than would have otherwise been needed. This means that capital expenditure is favorably impacted along with ongoing operational expenditure. As a result, these technologies are powerful forces in the move toward green data centers. Systems shipping in the near future will have most or all of these technologies in place; future studies of data center class storage should evaluate their work and proposals in the overall context of systems loaded with a full suite of RAS and capacity optimization technologies.

Space does not allow a discussion of RAID concepts and types. Please see any of numerous sources on the Web, especially[START_REF]The SNIA Dictionary, a lexicon of terms related to the storage industry[END_REF] and[START_REF]RAID is a complex topic[END_REF].

See manufacturer's data sheets and best practice guides for recommended RAID group sizes.

Acknowledgements

Erik Reidel and the workshop reviewers made valuable comments and suggestions. Thanks go also to many collaborators at SNIA, especially Jody Glider and Patrick Stanko.


Ranganathan Parthasarathy

Parthasarathy Ranganathan

Saving the World, One Server at a Time!

come


Rodrigo Dominguez

David Kaeli

Challenges in Binary Translation for Desktop Supercomputing

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Challenges in Binary Translation for Desktop Supercomputing

Rodrigo Dominguez

David Kaeli

Computer Architecture Research Laboratory Northeastern University Boston, MA

Invited Paper

Abstract:

Given the fact that the microprocessor industry has jumped off the frequency scaling bandwagon due to power issues, the rate of development of aggressive many-core systems has picked up in the past few years. However, these many-core systems have taken a range of incarnations: graphics processors (GPUs) have emerged as the data-parallel architecture of choice; heterogeneous architectures (e.g., AMD Fusion and IBM Cell) have emerged, though present challenges to code developers and compiler writers; and general purpose CPUs continue to increase the number of cores based on Moore's Law, though they lack the data-parallel horsepower found on other architectural styles.

These many-core systems come equipped with a range of programming languages, runtime environments, and compiler frameworks. In the case of GPUs, the architecture of the programmable shader core continues to evolve today in an effort to adapt these graphics-oriented processors to the requirements of the scientific computing community. For this reason, both AMD and NVIDIA have defined intermediate representations (IRs) as part of their offerings.

The main role of any IR is to provide a stable instruction set architecture that spans multiple microarchitecture generations, similar to the concept of bytecodes in Java. In the case of AMD/ATI GPUs, the IR is called the Intermediate Language (IL), and for NVIDIA GPUs, the IR is called Parallel Thread Execution (PTX).

Our work explores the fundamental differences between PTX and IL and the benefits of each IR. We are enhancing a binary translation framework to translate PTX into IL, allowing applications compiled for NVIDIA's C for CUDA environment targeting NVIDIA GPUs to run on AMD/ATI GPUs. We will report on our results to date, and will shed some light on the challenges that lay before us.