Living in Peace and Wisdom on our Planet

IT managers are in a pickle these days: according to a recent report by the Uptime Institute, nearly half of data center managers expect to hit their maximum energy capacity in the next two years, even while IT demand keeps growing. At the same time, energy costs are making high-level computing an ever more costly core need for companies.

The result is a booming interest in energy-efficient data center solutions that won't break the bank. Ken Brill, the executive director of the Uptime Institute, spoke with GreenBiz Radio recently about some of the surprisingly easy ways to boost performance and drop IT costs at the same time, and what the Institute will unveil at its 2008 Symposium later this month.

Matthew Wheeland: All right. So, Ken, thanks for taking the time to speak with us. I know we've heard repeatedly about the issues companies are facing, in terms of their IT operations, the cost and performance demands. Can you explain a little bit what the drivers are behind this and what forms are these – are these challenges taking?

Ken Brill: Well, I think everybody is somewhat familiar with the increasing heat problems in the data center, trying to remove heat. But I believe that that is just a symptom of the larger problem. And I don't see very many people talking about what I see – what I see is the problem. Because I believe it's fundamentally economics, and the economics come from when I first started in this field, it took about 50 years for the cost of electricity to cumulatively equal the cost of the hardware.

And when you're in the environment in which I started, the cost of facilities was insignificant, and by facilities I mean power and cooling. The cost of facilities was insignificant relative to the investment and the hardware. Well, today, depending on where you are in the world, in high utility rate areas, like the northeast part of the U.S. or in Europe or in Japan, the cost of the utilities, just to support a single one-use server, will exceed the cost of the hardware itself in less than three years.

So, you've gone from 50 years to three years in a relatively short period of time. And the problem is that we all make assumptions. We build mental models of optimizations, based on when we started, and so many people are still carrying around in their heads this optimization that the server, that the cost of electricity and power and cooling and whatnot, is insignificant, relative to the cost of the hardware, and that's not true anymore. The cost of the facility, when you include power and cooling, the capital investment, depreciation, and just the operating costs, that annual cost today is almost equal to the cost of the hardware. Certainly, over three years, it exceeds the hardware.

MW: What's behind this? Is this an increase in performance, is it Moore's Law?

KB: It's the benefit of Moore's Law. I mean, what Moore's Law has said is the number of transistors on a chip can double, he originally said every 24 months, but it's actually worked out to about every 18 months. And that has happened, but what we've kept – we've failed to keep an eye on the ball. Because while the chip performance has been going up at about 3x every 24 months, the power efficiency has only been going up at 2x every 24 months.

So, you have more chips, but the energy efficiency of the transaction has not kept up with the rate at which the number of transistors has gone up, so you have absolute energy consumption going up. And this has been going on since 1965, and it has now come to crisis proportions. The absolute energy consumption of IT is going p at about a 10 percent per year rate, which, if you look at a national economy, we're looking at in the next five years having to build 10 1,000-megawatt coal or nuclear fired power plants just to keep up with the increase in electricity demand because of IT.

MW: And I think that's a really key point and something that the Uptime Institute made waves with last month, when you released a report saying that a significant percentage of data center managers expect to hit the wall, in terms of availability of electricity. Is this just a confluence of factors where you've got computers that are using more electricity at the same time you're also running out of available electricity?

KB: Well, I think what, we have enough electricity today. What the problem is that there are no new plants being built, that are sitting, waiting to be licensed. Last I heard that there were about between 100 and 200 plants waiting to be licensed. Now, it takes between five and 10 years to build them. But, in fact, they're just – we have the NIMBY problem: Not In My Backyard.

And so – you know, if energy consumption in IT goes up and we acquire 10 new plants to equal that, and by the way, 10 plants in the next five years, and then in the five years after that it's another 20. So, in the next 10 years, to keep up with the growth and energy consumption in IT, we would need to build 10 major power plants, and by the way, it's just not gonna happen, because the rest of the economy also needs these power plants, and none of them are getting licensed. And by the way, it takes between five and 10 years to build them once they've been licensed.

MW: So, what's the alternative? What are companies doing rather than lobbying for some sort of fast-track electricity plant to be built? What else can companies do?

KB: Well, the institute believes that IT could radically reduce its energy consumption, and it would save money in the process. And there are a number of things that people can do and we're gonna be talking about these at our upcoming symposium. But the first thing that people can do is to go through their data center and identify what is running that could just be turned off. Between 10 and 30 percent of the energy consumption in a typical data center is for servers and storage and other things that is no longer in use.

And it could just be turned – I mean, as literally, as simple as turning the switch. One of the problems that we have as a country is that we have downsized the management, data center management, and we have a perception that servers are very inexpensive. And it's cheaper, just basically, to let them run than to have the management talent that would be required to manage the servers optimally. Well, the average energy bill for a server is at least $700. Can be more than that, and if the server costs, and this is in a relatively low utility rate area. In other areas, it could be double what I just said.

You can see that in three years you can equal the cost of the hardware. So, by just turning off the server, you reduce this energy consumption. The other thing you do is that you buy back power and cooling capacity. Today, we have the largest building boom in data center construction that's ever occurred. And I question whether people have really thought out, do they need to be building it? Because if you could turn off loads you wouldn't need new stuff. You would create capacity in what you already have.

And then, let's talk about virtualization. I've just come back from our European network meeting, and one of our European companies presented a story where they started out in 2004, and they looked at their Win-tel environment, they did not look at their Unix environment or their mainframe environment, they just looked at the Win-tel environment, which was the less critical application. And they – over a three-year period, they took 3,150 servers and virtualized them and replaced those 3,150 servers which were consuming almost 2,000 kilowatts of energy. They replaced them with only 150 servers and saved 1,800 kilowatts of energy.

That brought them back, in terms of data center capacity, about $24 M of data center capacity, and it saved them about $7 M a year in electrical consumption.

MW: And that doesn't include any loss in performance?

KB: They would say that the performance actually improved, their disaster recovery is improved, and the ability to manage it is improved, and they need less head count. Because managing 150 is a lot simpler than managing 3,100.

MW: Right. Well, that's interesting. I mean, that just seems like there's no downside to it. So, what I hear you saying is you can cut back on the amount of machines you have, you can cut back on the amount of machines that you have, you can cut back on the amount of data center space you need, the amount of expenditures you have to make to build new data centers, if you just buy newer, more powerful, but also more energy efficient machines.

KB: Yes, but – here's the but.

MW: Right. There's always a but.

KB: The skills required to do that kind of thinking have atrophied, and are not common. One of the things that reported was to do this virtualization, and by the way, if you did the numbers, it's 30 to 1. Many people are talking about 10 to 1, 12 to 1, and they have a demonstrated example of 30 to 1. To get that kind of 30 to 1 ratio, they had to apply mainframe disciplines. And they are in shortage today. They had to do modeling, they had to do performance testing, and a whole series of things before they settled on the platform.

They used much more memory than you would normally do, but they were able to take what other people would do, at 10 to 12 to 1, they were able to more than double that by using a more optimized platform. Now, this is not easy work. This is not cool – in the technology sense, this is not cool. It's hard work, it's not valued, it takes a long time to do, if you do it successfully.

And the problem is for many CIOs, they have a tenure of three years, and to do a project like this – any time you do this you're gonna gore some cows, so you're gonna create some people who don't like what you're doing. And it may take longer to accomplish than your tenure in the job. So, we haven't created the proper incentive mix yet to do these sorts of efficiency things, because they're not they're not sexy. They do save large amounts of energy and costs.

MW: And I can see there also being an almost instinctive reaction to the idea of turning off everything we've invested money in over the last however many years, buying a whole new set of equipment just in order to save this money, but I can see it not computing, particularly quickly.

KB: Your point is exactly on. You don't do this. You do this while you're – when you're doing a technology refresh. So, you'd be spending the money anyway. But, you'd be spending less money, because instead of buying, and they did this 3,100 servers, they did this, in really, clumps of about 1,000.

And so, you know, there's the normal refresh cycle, and that's the time you should do this sort of work, so that you're not spending new money, you're just spending what you would have spent. Well, if you refreshed everything, you'd be spending 100 percent for new hardware, whereas if they spent only 1/30th for new hardware.

MW: Well, this seems like a good time to ask about the symposium, talking about all the skills that are lacking amongst most companies IT departments today. Tell me a little bit about the symposium. What are you going to cover, and how long have you been doing this?

KB: Well, the symposium is in Orlando, Florida, and it goes from the 27th of April to the 30th. We're having some really exciting people presenting and on topics that are very – have not been in any form that we know of. And the way we do this, it's very different than other events of its type. The institute really controls the program, we designed it ourselves, so we designed the day, so that if somebody came, they would walk away with a pretty good education about the things that are happening.

But this is not a sales -many things that occur today are really hidden sales processes. We work very hard not to have that happen. This is an intellectual exchange. So, this is the third year that we've done it. The first year, we sort of looked at it from a cooling standpoint, high density computing and we focused a lot of on cooling. This year, we're almost exclusively on economics because we think that's the fundamental issue, that cooling issues can be solved if we get the dollars.

But the problem is that this is a CIO, CFO, CEO, Board of Directors set of issues. Because we feel that the fundamental economics of IT are being changed, and I can say that simply and it used to be that facilities costs were 1 to 3 percent of IT's budget. For – for many companies, it's now up to 5 percent. For some of them, it's 10 percent. And as IT, as facilities becomes a bigger piece of IT's budget, because the power consumption that comes with the constant rate of IT spending is going up, as facilities becomes a bigger part of IT's budget, IT either has got to get more money out of the revenue stream of the organization.

And from what I hear from the CFO people, is IT's not gonna get a bigger percentage of that revenue stream. So, if facility costs are going up and revenue stream is constrained, that means that other things in IT must be cut back. And what are the things that can be cut back? Well, the biggest discretionary thing is new application development. In the long-term, I don't think that's good for the productivity of the world.

So, what we're trying to get across is we gotta pay attention to these facility costs, when we have to reduce them, but the only way we can do that is for the senior level executives to wake up and smell the coffee. Because these changes are happening largely invisibly. And the only time that the senior level people really understand it is when they're asked to approve a new data center investment for 10, 20, or $300 millino. Used to be you could buy a pretty good data center for $20 million bucks. Well, now it's up to $200 or $300 million. And that's when you get people's attention.

But at that point, it's too late. The decisions that resulted in having to spend all that money were made many, many months or years earlier, and so the whole point of symposium is to try to lay these issues out so that the senior level executives can understand that this is an economic crisis.

MW: And so, even with the same slice of the budget, you're saying that the IT department can improve performance and improve capacity or increase capacity, can do even more with the same slice of the budget?

KB: We believe, yes. We believe that there are 50 percent improvements available.

MW: Wow.

KB: And we have some documentation for that. I mean, just going through and turning off all the common host servers is a big one. Virtualization is another one. And buying more efficient IT hardware is another big one. At procurement organizations, they are not incented to look at performance per watt, they look at performance per dollar. Well, they also need to look at performance per watt, and if they do they will find dramatic differences between different manufacturers, and by being more energy conscious, they will actually be saving large amounts of money. Now, that gets me into the green IT awards program.

We will be presenting the first IT awards for greenness. And we have it in four cate – five categories. One is for strategy, one is for asset utilization and asset utilization would be the – you know, turning stuff off, virtualization, enabling the power save features. Another one would be for buying energy efficient hardware.

Another one is in operating their power and cooling more efficiently, and finally, beyond the data center, and some of the things there are setting up a program to shut off computers remotely, the thin client computing strategy. And we'll be presenting awards in each of those categories, and what we're trying to do there is to highlight, the people who will win these awards didn't do it for us. They didn't do it to win the award. These are things that they just did in the normal course of their business.

And we asked them, you know, "Tell us about them so we can recognize your achievement." So, all the things that we will be presenting awards for are things that other people could emulate.

MW: In talking about the difference between performance per watt, instead of performance per dollar, is this just a remnant, a holdover of sort of the old way of thinking about IT?

KB: Yes, part of – there's a new profession that is needed, that combines sort of capacity planning, economics, and some facilities engineering. It's a new professional, and so, you know, historically, people looked at performance per watt and one of the magic things was how much memory did you put with the processor in order to get a certain functionality?

Well, one of the things we're learning on the power supply, and this is just one tiny example. And we look at the power supplies in servers and we hold the manufacturers accountable for the efficiency of the power supplies at a very high load factor. Well, we're looking at the wrong end of the range, because of a whole variety of reasons, the typical load on a power supply is down somewhere in the 10 to 15 percent area, not up in the 80 to 90 percent area, which most people assume. Well, down in the 15 percent areas, the efficiency of the power supplies is terrible. And yet that's where they operate.

So, one of the things we need to do as an industry, is to change our specifications, look at efficiency at the bottom end of the curve, and then at the same time, we need to hold the manufacturers accountable for why are you over power supplying things? So, we're getting into somewhat technical areas, and we're combining engineering with economics, with both, with capacity planning, and yet, there's so much reward here. And it's virtually free.

MW: That's what I think is hardest to swallow, is that this really does seem to be virtually free. You just have to change the way you think about your IT structure.

KB: It is. The costs of making these power supplies efficient at partial load efficiency, is matters of under $5 in parts.

This is not new technology, this is known, but customers aren't demanding of the manufacturers this efficiency. And while the engineering people inside the various manufacturers know how to solve this problem, adding $5 to the building materials, they can't do it, not if it won't result in an increased market share.

And customers aren't demanding it, so the manufacturers can't offer it, even though they know how.

MW: Without going into too big of a tangent here, how do you incentivize this?

KB: Well, I think the biggest thing is, when you're buying this stuff is to start looking at performance per watt. There are two basic ways to do it. One is to use more efficient power supplies under actual load conditions, and the other is to use more efficient processors. And by the way, it's not just the processors, it's the memory. Because you can use a more efficient processor, and then lose all the benefits because you used a bad choice for memory, so it's looking at the whole enchilada, so to speak, not just the individual pieces.

Now, one of the things that will help this is the Energy Star rating for servers. That will take some of the black magic out of this, but that's gonna be toward the end of this year before that's out.

MW: And there was also some discussion about certifications for data centers, whether it's Energy Star certification for data centers, is this also something that's on the horizon?

KB: I think that there's an effort on that. The Institute is running our own parallel effort with that. We think that there's a very – when you get into a data center, understanding what its energy efficiency is, is subject to a lot of different factors: the weather conditions, the type of equipment that you've got installed, the percentage of load, what tier level it is, the skill of your technicians in running the building. While I commend the idea of what we're doing, I don't think it's going to be granular enough for many people to actually get benefits out of it.

We're working on a dashboard concept that will allow people to parce their energy consumption into different buckets, and then you can take each bucket and then benchmark yourself against other people. So, I think, until you get to the level of granularity that I'm talking about, you have to get more granular in order to figure out how to improve, I guess, is what I'm saying.

MW: Let's go back to the symposium for a moment. Also at the symposium, you are going to be unveiling some research you conducted with McKinsey, that does address some of these executive level issues that can help companies green their IT. What can you tell me about that, in advance of the report being published?

KB: Well, I'll just let a little, you know, a few little nuggets drift out. We found this interesting, because McKinsey and ourselves independently came to many of the same conclusions, and then we're collaborating in this report together. They bring much more of the economic side of it. We bring more the engineering side of it. But together, it's pretty powerful. We're saying, fundamentally, that for some organizations, they're not gonna be able to afford these costs.

And by the way, some people say, "Well, we'll outsource it." Well, the outsourcer has the same costs. Plus, they've gotta make a profit. So, the solution may be outsourcing, but people need to understand that outsourcing isn't going to change the fundamentals. The other couple things that we will be talking about and we'll be giving our analysis, but one is to move the data center assets – which used to be not very significant, but it is now measured in hundreds of millions of dollars for larger organizations – move that from the corporate real estate group to the IT organization.

And the reason for doing that is to make the CIO accountable for the entire stack of costs, so that he or she will see the benefit of some of the optimizations that we've been talking about. Because the costs are split between the two organizations, there's a fair amount of dysfunctional behavior that goes on. Another thing that we will be recommending is the appointment of an energy czar, with a mandate to make significant, quantified reductions in energy consumption.

In the fine print, we're saying that the energy czar is gonna make these energy consumption savings, and achieve the same or higher levels of availability, which we think is totally compatible. And one of the fears that I personally have, as we get into this leanness thing, is that we remember that the basic idea of IT, is business value.

And we can't compromise business value for energy efficiency. I think business value has to come first, and then we have to be efficient in how we deliver that value. And I believe that there's so much gold all over the floor that if we just will bend down, bend over and pick that gold up, we can make substantial savings.

MW: And it sounds to me, also like some of these recommendations are things I've heard repeatedly, that because there's a divide between who pays for the energy bill or the facilities cost versus who runs the data centers, then you don't see the savings that are possible. You don't see the waste that's happening.

KB: Right.

MW: So, it seems like these ideas could also result in a demand-side increase in what energy efficient computing products are out there, or energy efficiency tactics in data centers.

KB: Yeah, one of the things we're trying to do is to throttle demand. By getting all of the economics, creating a true TCO model. I mean, we don't want to stop the growth of IT. If IT is truly creating productivity in the economy, we certainly want that to occur, but we need to stop sub-optimal investments, and by sub-optimal investments, let me give an example. There's an example of a $22 million investment in blade servers. And if you looked at the ROI on the investment, you know, it looked terrific. But nobody talked to facilities.

So, the blade servers come in, and now the thing comes out that there's a $54 million investment needed in capital expenditure to be able to provide the power and cooling for the $22 million in blade servers. And not only that, the cost of operating those blade servers for three years was another $33 million, so the total investment was over $100 million. But, because it was split up between different organizations, a unified view is not taken. That's what we're trying to overcome.

MW: Great. That pretty much covers everything. Do you – is there anything else you want to talk about before I sign off?

KB: The basic message I'd like to make here is that we can talk about green or we can talk about profitability. To me, they're synonymous. And what I say to people is when you go home, do your kids talk about what you've done for the environment and how green what you've done. When you're at work, talk about the savings, and this is a skill that we need to develop as professionals: we need to be able to talk in terms of savings.

So often, we talk in a language that management doesn't understand, but they do understand dollars. And if you go to our website, you'll find all kinds of white papers that try to give background information, in terms of dollars so that you can – you can talk in language that people – that managers understand.

MW: All right, great, Ken, thanks so much for your time and for your insight today.

KB: And thank you for the – thank you, the audience listening in. I hope you found it useful.

... To listen to a podcast of this interview, Download the interview in mp3 format, listen to it in iTunes, or subscribe to the GreenBiz Radio RSS feed.

Matthew Wheeland is the managing editor of GreenBiz.com.