Seeking The Best Performance per Watt for Folding@Home
By
Gabriel Torres
on October 30, 2008
We got so excited in participating in the Folding@Home project that we built as many high performance systems we could running both the SMP and GPU clients. We were very happy with the results until we received our first electricity bill: our energy consumption more than doubled – and we haven’t even had our systems running 24/7 for 30 days! Since we still wanted to contribute as much as we can to Folding@Home, we decided to go in a quest to find out if there is a way to score lots of points at Folding@Home and, at the same time, not going bankrupt. We got all video cards we had available here in our lab to see which one provided the best performance/consumption ratio. Check it out.
If you are not familiar with the Folding@Home project, it is a project sponsored by Stanford University for using computers all around the world to make protein folding simulations in order to find cure for certain diseases. You can collaborate with the project by installing a client on your PC and, when your PC is idle, it automatically downloads, calculates and sends the results back to Stanford. Thus they can have the largest supercomputer in the world (made by the collaborative effort from all people participating in the project) without spending a dime.
If you want to collaborate more, you can install and run high-performance clients, like the SMP client (which recognizes more than one CPU or more than one CPU core; the standard client only recognizes one CPU core) and the GPU client (which uses the graphics chip from your video card to do the calculations, a technique called GPGPU, or General Purpose Graphics Processing Unit). These clients will complete the calculations at a faster pace but, on the other hand, will consume more power, increasing your electricity bill. Finding the “perfect” balance between performance and power consumption is the goal of this article.
For every completed work you send back to the university you get a certain number of points. The number of points will depend on the kind of client you have installed (standard, SMP, GPU, etc) and the kind of job you are running. The number of points you receive will be our metric for performance, as most people participating in Folding@Home in teams (like ourselves) are interested in achieving the highest possible score.
Now let’s show you the systems we built to collaborate with the project, their performance (i.e. the number of points they were giving us) and their electrical consumption. With this data you can have an idea of how much we were spending to have them turned on 24/7. We will do a lot of investigation on how to decrease consumption and, at the same time, keeping a high score.
But before we present you numbers, you need to understand more about power consumption. We measured consumption with a digital watt meter, which presents results in watts. With this instrument we were measuring the AC consumption of our system. This is not what the system was pulling from the power supply, because the power supply itself consumes and wastes power. The ratio between the power that the system is pulling from the power supply and the power that the power supply is actually pulling from the wall is called efficiency.
The higher efficiency is the better, as you will be wasting less energy. For example, if a certain system is pulling 200 W from the wall that means that your whole computer is pulling 200 W (and you will pay to the electricity company based on this amount) but the components that are connected to the power supply will be consuming less that than. If we take a typical power supply with 80% efficiency, the components would be pulling 160 W.
Suppose you replace your 80% efficiency power supply with another with 88% efficiency. Your system will still be pulling 160 W from the power supply, but your new unit will be pulling less from the wall: 182 W.
So off the bat one way to save on the electricity bill is replacing your power supply with another with higher efficiency. One way to discover your unit’s efficiency is reading the efficiency chart provided by the manufacturer. Another way is reading our reviews, where we measure this.
Watts (W) is the amount of power the equipment is consuming, but the electricity company charges you based on how much energy you are consuming, which is measured in kWh (kilowatt-hour). Energy is the amount of power you consume over time. So one kWh represents 1 kW (1,000 watts) consumed over one hour. Since we are going to assume that we will be running each machine 24/7, we will multiply the amount of watts by 0.72 (24 hours x 30 days / 1,000; the division by 1,000 is necessary to convert Wh into kWh) to have an estimate of the monthly consumption in kWh. Then we can simply multiply this number by the cost of each kWh to have an idea of the monthly cost to run each system 24/7. Of course the cost of electricity varies depending where you live; we are using the value of USD 0.1224800 per kWh, which is the electricity cost in our town on the day we published this article. On top of that we had other charges like franchise fee, green power financing, etc that we are not considering for simplicity.
As for performance, we measured how much each video card or CPU delayed to process 1% of the work load. By multiplying this time to 100, we had how much time each device would take to process the entire work unit. By dividing 86,400 (number of seconds in a day) from this number, we had the maximum number of work units the device can process per day. As we know how many points each work unit is worth, we can find out the maximum score we can expect from the device by multiplying the maximum number of work units the device can process per day by the number of points each work unit will give us. The result is the maximum score this device can give you per day, and this is the number we will be using.
Below you can see all systems we built to run Folding@Home. Please see previous page for a detailed explanation on consumption, cost and performance/score.
System # | CPU | Video Card | Motherboard | Memory | HDD | Power Supply |
1 | Core 2 Extreme QX 9650 (3 GHz) | None | 2 GB DDR2-800 Kingston KVR800D2N6/1GB | 160 GB | ||
2 | Core 2 Duo E6600 (2.4 GHz) | 4 GB DDR2-800 OCZ | 200 GB | |||
3 | Phenom 9600 | 2 GB DDR2-800 Kingston KVR800D2N6/1GB | 500 GB | |||
4 | Phenom 9700 | 2 GB DDR2-800 Kingston KVR800D2N6/1GB | 500 GB | |||
5 | Core 2 Extreme QX 9770 (3.2 GHz) | GeForce GTX 280 | 2 GB DDR3-2000 | 500 GB | ||
6 |
Now we are going to do a detailed analysis of our systems.
We measured consumption under three scenarios. First with the system running only the SMP client, then only with the GPU client, and then with both clients at the same time. We wanted to see if it would make sense to run both SMP and GPU clients at the same time.
System # | Client | Consumption (W) | Monthly Consumption (kWh) | Monthly Cost (USD) * |
1 | SMP | 121 | 87.12 | $10.67 |
2 | SMP | 185 | 133.20 | $ 16.31 |
2 | GPU | 228 | 164.16 | $ 20.11 |
2 | GPU + SMP | 202 | 145.44 | $ 17.81 |
3 | SMP | 196 | 141.12 | $ 17.28 |
3 | GPU | 171 | 123.12 | $ 15.08 |
3 | GPU + SMP | 213 | 153.36 | $ 18.78 |
4 | GPU | 430 | 309.60 | $ 37.92 |
5 | SMP | 335 | 241.20 | $ 29.54 |
5 | GPU | 441 | 317.52 | $ 38.89 |
5 | GPU + SMP | 468 | 336.96 | $ 41.27 |
6 | PS3 | 131 | 94.32 | $ 11.55 |
* USD 0.1224800 per kWh running 24/7.
Here we found out something very curious. Each GPU client puts at least one of the CPU cores working at 100%. On dual-core CPUs, one of the cores will always be working at 100% load (independently of the number of GPUs you have running Folding@Home). On quad-core CPUs, you will have one CPU core per GPU running at 100%. For example, with our Core 2 Duo when we started the GPU client the CPU load went straight to 50%, with one core being idle and the second core being 100% used. Because of this funny behavior, when we put the SMP and GPU clients to run at the same time consumption (and performance) lowered, as both clients were disputing the CPU. On our quad-core system with three video cards, CPU utilization was at 75%, with one CPU core being used per GPU running Folding@Home.
Notice that except on systems one and six, all other systems had a video card installed and the consumption presented when running the SMP client alone includes the consumption of the installed video cards in idle mode.
See how our Phenom 9600 running the SMP client was consuming more than a GeForce 9800 GT running the GPU client.
Since we were running all systems with the GPU and SMP clients at the same time (except on machine number 4), the estimated monthly cost for running these six machines was of USD 138. Ouch. The estimated daily score was of 30,715 points, with an estimated monthly score of 1,063,590.
On system 1 we were running Debian Linux 4.0 (64-bit), while on the other systems we were running Windows XP SP3 with Catalyst 8.10 drivers for ATI and 177.84 drivers for nVidia.
Now we want to see what we were getting in terms of score on Folding@Home by running these systems and do some preliminary analysis to see the most efficient configurations we had running. WU stands for work unit. Project is the number of the Folding@Home project each client was running at the time we collected our data, which will tell us how many points they will give us for each delivered work unit (click here to see the full table). We put the number of points given for each completed WU in parenthesis. The maximum daily performance is calculated by dividing 86,400 (number of seconds in a day) by the time to complete one work unit and the result multiplied by the points given to each completed work unit for that project.
Our metric for measuring efficiency will be points/kWh, which is calculated by dividing the maximum monthly performance by the monthly consumption in kWh. This index indicates how many points each system produces with each kWh consumed from the wall. So the higher this number, the better.
System # | Client | Project (Points) | Time to Complete One WU (seconds) | Max. Daily Performance (Points) | Max. Monthly Performance (Points) | Points/kWh |
1 | SMP | 5101 (2,165) | 66,000 | 2,834 | 85,020 | 976 |
2 | SMP | 2665 (1,920) | 150,000 | 1,106 | 33,180 | 249 |
2 | GPU | 5800 (480) | 5,800 | 7,150 | 214,500 | 1,307 |
2 | GPU + SMP | Above | 150,000 (CPU), 8,700 (GPU) | 5,873 | 176,190 | 1,211 |
3 | SMP | 2653 (1760) | 84,100 | 1,808 | 54,240 | 384 |
3 | GPU | 5014 (480) | 10,300 | 4,026 | 120,780 | 981 |
3 | GPU + SMP | Above | 84,100 (CPU), 15,400 (GPU) | 4,501 | 135,030 | 880 |
4 | GPU | 5651 (388) | 14,300 (HD 4850), 14,000 (HD 4870) | 2344 + 2394 = 4,738 | 142,140 | 459 |
5 | SMP | 2665 (1,920) | 79,500 | 2,087 | 62,610 | 260 |
5 | GPU | 5013 (480) | 8,200 (GTX 280), 8,100 (8800 GT), 8,500 (8800 GT) | 5,057 + 5,120 + 4,879 = 15,056 | 451,680 | 1,422 |
5 | GPU + SMP | Above | 79,500 (CPU), 8,200 (GTX 280), 8,100 (8800 GT), 8,500 (8800 GT) | 17,143 | 514,290 | 1,526 |
6 | PS3 | 5310 (110) | 26,100 | 364 | 10,920 | 116 |
You should understand something very important about Folding@Home scoring system. While work units assigned to nVidia-based video cards will almost always give you 480 points, the number of points given by work units processed by ATI-video cards and the Playstation 3 console can change quite often. The above results are based on the project that each client was running at the time we made our tests and do not reflect the best scores ATI and PS3 systems can achieve. Our ATI-based video cards were processing a work unit that gave 388 points, but there are work units that will give 548 points. Our PS3 was processing a work unit that gave 110 points, but there are work units that will give 330 points. The time for completing these units that give more points can be higher, however. Just as an exercise, we compiled the following table for systems four (ATI) and six (PS3) as if they were processing these other kinds of work units that give more points. We are doing this in order to not be accused of being unbiased or someone pointing out this potential flaw in our methodology in the future. For this exercise we will consider that the clients will process each work unit with the same performance, which may not be true in the real work.
System # | Client | Project (Points) | Time to Complete One WU (seconds) | Max. Daily Performance (Points) | Max. Monthly Performance (Points) | Points/kWh |
4 | GPU | 4743 (548) | 14,300 (HD 4850), 14,000 (HD 4870) | 3,311 + 3,382 = 6,693 | 200,790 | 648 |
6 | PS3 | 5305 (330) | 26,100 | 1,092 | 32,760 | 347 |
As you can see, even simulating the best performance these systems could achieve, both performance and efficiency were at levels below our other systems.
From the above results we learned interesting things about our systems:
Now we were curious to see if we used mid-range or even low-end video cards we would achieve better performance/power ratios. To do that we tested all video cards we had available.
After we saw that there is a huge difference in power consumption among different systems, we decided to build a mainstream system and measure the performance and power consumption of all video cards we could get our hands on installed on this system. Maintaining the whole system identical and changing only the video card is the correct way to evaluate video card performance.
The system we built had the following specs: Core 2 Duo E6600 (2.4 GHz), ASUS P5K-E/WIFI-AP Motherboard (Intel P35 chipset), 2 GB DDR2-800 (Kingston KVR800D2N6/1GB), 500 GB hard disk drive (Western Digital Caviar SE16), Zalman ZM-600HP power supply and Lite-On LH-20AIL optical drive. We were running Windows XP SP3 with Catalyst 8.10 drivers for ATI and 177.84 drivers for nVidia.
The results you can see below. Please keep in mind that consumption is the AC consumption for the whole system, not only for the video card. nVidia cards processed project 5800 (which gives 480 points per completed work unit), while ATI cards processed project 4743 (which gives 548 points per completed work unit).
We sorted the table below from the card with the best performance/kWh index to the worst.
Video Card | Time to Complete One WU (seconds) | Max. Daily Performance (Points) | Max. Monthly Performance (Points) | Consumption (W) | Monthly Consumption (kWh) | Monthly Cost (USD) * | Points/kWh |
GeForce GTX 260 | 5,800 | 7,150 | 214,500 | 214 | 154.08 | $ 18.87 | 1392 |
GeForce GTX 280 | 5,500 | 7,540 | 226,200 | 234 | 168.48 | $ 20.64 | 1342 |
GeForce 8800 GT | 8,100 | 5,120 | 153,600 | 160 | 115.20 | $ 14.11 | 1333 |
GeForce 9800 GT 1 GB | 9,300 | 4,459 | 133,770 | 170 | 122.40 | $ 14.99 | 1093 |
GeForce 8800 GTS | 10,100 | 4,106 | 123,180 | 188 | 135.36 | $ 16.58 | 910 |
Radeon HD 4830 | 17,800 | 2,660 | 79,800 | 158 | 113.76 | $ 13.93 | 701 |
Radeon HD 4870 | 12,600 | 3,758 | 112,740 | 225 | 162.00 | $ 19.84 | 696 |
Radeon HD 4850 | 15,100 | 3,136 | 94,080 | 189 | 136.08 | $ 16.67 | 691 |
GeForce 9500 GT | 21,700 | 1,911 | 57,330 | 119 | 85.68 | $ 10.49 | 669 |
GeForce 8600 GT | 25,300 | 1,639 | 49,170 | 126 | 90.72 | $ 11.11 | 542 |
Radeon HD 3870 | 21,100 | 2,244 | 67,320 | 178 | 128.16 | $ 15.70 | 525 |
GeForce 8500 GT | 58,900 | 704 | 21,120 | 108 | 77.76 | $ 9.52 | 272 |
Radeon HD 3450 (64-bit) | 288,000 | 116 | 3,480 | 111 | 79.92 | $9.79 | 43 |
* USD 0.1224800 per kWh running 24/7.
The results were quite interesting and we will talk more about them in the Conclusions.
Now that we know a lot more what is going on with our parts, we decided to replace parts our systems. Let’s see what we did and what happened.
After carefully reading the numbers presented on previous page, we decided to make the following changes to our setup:
We have now only three systems running Folding@Home, but consumption is now 560 W (403.20 kWh) from 1,565 W (1,127 kWh), what will cost us USD 49.38/month instead of USD 138/month. That is a 64% decrease.
Or course performance decreased. We will be now making 17,439 points/day or 523,170 points/month from 30,715 points/day or 1,063,590 points/month, so performance decreased 43%.
The beauty, as you can see, is that performance and consumption didn’t decrease at the same proportion. We had before a points/kWh index of 944 – i.e., we were making 944 points at Folding@Home for every kWh consumed – which increased to 1,297. So our efficiency increased 38%!
We found out several interesting things in our investigation. Here is a summary:
In the text we explained how to find out the points/kWh index of a system, but here is a more practical summary:
Originally at http://www.hardwaresecrets.com/article/640