Welcome to my website


Archive

Low watt, high power server- Part 2: Watt and heat

25 October 2011 23:24


This is blog post number 2 in my series about the build of my new home server setup and in this blog post I willlook at what wattages and heat the parts in the server is having, so in this post i will look at the watt and heat for disk drives, interface cards and RAM blocks.



How much wattage are the SSD and HDD using?
So to better know how much the disk drives are using i then look at what drives i was having and found that there was 4 different hard disk where there was 2 of each model and then i have 2 of the SSD.
So by this i can then measure the use of 2 disks at the same time and then just divide the result with 2 and then have a bit more precise measurement then if i only have one drive. So the setup is this, one Fluke 789 meter is measuring the 12V and another Fluke 789 is measuring the 5V and a Fluke 177 to measuring the amps for the 12V wire and then a Fluke 287 to measuring the amps for the 5V wire.
On the photo below you can see the 4 meter for measuring the disk drives and on the right there is the Fluke 345 Power Quality Clamp Meter there is measuring the wattages on the 230V AC.

I have done the measurement by connection 2 disk drive to the system and then using the average function on the fluke meters by first have the drives doing nothing (idle) and measuring the average amps over 10 minutes and then i have done the same 10 minutes average measurement when the drives was at the highest load that i was able to do by using the IOmeter software to run a benchmark.

Model

Size GB

Type

RPM

Disk Size

Watt Idle

Watt full load

Seagate Barracude 7200.9

80 GB

HDD

7200

3.5"

5.2944 W

6.7001 W

Western Digital Caviar SE WD800

80 GB

HDD

7200

3.5"

5.0583 W

5.6027 W

Western Digital Raptor WD360

36 GB

HDD

10000

3.5"

6.6774 W

8.8369 W

Seagate Momentus 7200.2

160 GB

HDD

7200

2.5"

1.0944 W

3.4462 W

Crucial M4 256GBSSD

256 GB

SSD

-

2.5"

0.5935 W

1.5882 W

So it is not just marketing talk that the SSD's are using much less wattage then the old hard disk drives and one SSD can have the same performance as many hard disk drives so with that the saving will be even better.
My old server is having 4x Seagate Barracude 7200.10 disks in a raid setup and that will give number around 20.8W in idle and 26,8W with high load for the disks, so if replaced by 3x SSD's there will use 1,7W idle and 4,7W with high load.... so this will give a saving at around 15-25W,This is a nice saving on the change from harddisk drives to SSD disk.
The only issue is the cost of the SSD, small GB size and limit lift time.




How much heat is there then coming from the disks?

So to make a comparable heat profile i have then power all the disk on for 4 hours in idle mode with just the power connector on and not any pc and then i run the disk at full load for 30 minutes to get them to heat up.
Then i have taken a thermal image with a Fluke TI25 thermal image camera of each pair of disks.
The image temperature color scale is set low point as 20 and high point as 65 degrades for each image to make sure colors and images are comparable.

HDD: Western Digital Caviar SE WD800 80GB 7200RPM 3.5"

The graph on the right shows the temperature across the to disks on the red line.
Seems to have okey temperature for 3,5" and the heat is coming from the main chip.


HDD: Western Digital Raptor WD360 36GB 10000RPM 3.5"

The graph on the right shows the temperature across the to disks on the red line.
The print board is have some hot spots, but for 10krpm disk i am a bit surprised that it is not hotter then this.


HDD: Seagate Barracude 7200.9 80GB 7200RPM 3.5"

The graph on the right shows the temperature across the to disks on the red line.
Wow that chip is getting very hot and the center of them is 72 degrades so if your using disks like this then your better make sure there is some type of air cooling across the disk.
This is when the disk is placed in free air on the floor so i can only guess that they will get even more hot when inside a computer case.


HDD: Seagate Momentus 7200.2 160GB 7200RPM 2.5"

The graph on the right shows the temperature across the to disks on the red line.
This is disks from a laptop and it is easy to see why laptops get so hot when they have hard disk drives installed.


SSD: Crucial M4 256GB 2.5"

The graph on the right shows the temperature across the to disks on the red line.
Wow, nice to see the SSD are so much cooler then the HDD's and if you look at the graph on the right you can see how low and cool the drives are in comparison to the other hard disk drives.

So this is super nice that i don't have the hard disk drives high heat in this new server and is just having the nice and cool SSD there is using alot less watt..


Wattages for RAM blocks.
By change the amount of memory on the mainboard and taking a number of reading with the Fluke 345 Power Quality Clamp Meter on the main 230V AC power, I have then calculated that 8GB(2x4GB) Corsair 8GB DDR3 1333Mhz XMS3 is using around 1,727W and 16GB(4x4GB) Corsair 8GB DDR3 1333Mhz XMS3 is using 3,454W, so from this it seems the ram blocks is not using much wattages and i think that the 8gb will cost more than the 16gb will in the long run because if there is only 8gb ram then the disk system will do a lot more swapping to disk and by this use extra watts.
so why not use 8gb blocks, it is because the 8gb blocks cost too much and if the prices was not that high then i will have installed 32GB(4x8GB) on the system, but this may come as a upgrade at a later time when the prices has come down, so for now it is only 16GB


Wattages for network cards.
So by doing the same as when taking measurement of the memory i have the cards changed around and taken a number of reading with the Fluke 345 Power Quality Clamp Meter on the main 230V AC power and from this i have found that the network cards cost around 2.5 to 4.5W per card and in the table below you can see the cards that i have tested with and how many watts they using.

Model

Interface

Ports

Total Watt

Watt per port

Intel PRO/1000 PT Desktop Adaptor

PCIe 1 Lane

1

2,272W

2,272W

Intel PRO/1000 MT Desktop Adaptor

PCI 32 bit

1

4,454W

4,454W

Intel I340-T4 Server Adaptor

PCIe 4 Lane

4

4,363W

1,090W

So because i have to make a Etherchannel of 3-4 ports between the server and the switch i will then use the Intel I340-T4 server adaptor as they have the lowest watt per port and i can use just one card and not have to use 4 cards, also it seems that Intel writes that is done a lot around the power management to keep the wattages down and be green and they list in the specs. That the card is using 4.3W and that is just what i have measured to use so it seems my readings for the cards are matching very well.


In the next post I will look at what power supply to use and the heat and cooling of the Mainboard, chipset, CPU, RAM and so on.


To see the other blog post in this series then click on the comment link and you will find a comment with links to each post in the series.

Tooms @ 25 October 2011 23:24 | Comment | Direct link
Low watt, high power server- Part 1: Need new server setup
9 October 2011 23:52

I am running my own home with VMware ESX and right now there is 7 VMs, this server setup is old and cannot keep up with what I need, so it is time to look at a new setup there is better than the old one and if it can use less wattages then it will be a plus

So to begin I then have a look at the setup I am running today to better see what I need the new one to do and where to improve the new one.

The old server is a single server with ESX 3.5 and is running 7 VMs
PartDescription
CPUAMD Athlon X2 4400+ 2.2ghz
2xcores and have a TDP = 89watt
MainboardASUS A8N-VM CSM
http://www.asus.com/Motherboards/AMD_Socket_939/A8NVM_CSM/
RAM4 x 1GB Kingston Hyber-X
Do to some bios issue then ESX can only see 2,8GB of the 4GB
Harddisk4 x 250GB 7200RPM of mixed Seagate and Samsung harddisks
In the Raid 5 this gives me a VMFS volume on 675GB
ControllerIntel SRCS16 6 SATA connections of 1.5 Gbits/s and 64mb cache
With battery backup installed
Network1 x Single port Intel gbit netcard
1 x dual port Intel gbit netcard
Other1 x DVD drive
Fan controllerT-Balancer BigNG with a number of big silent fans
UPSAPC Smart-UPS 1500 with network interface


So do to some BIOS issue there is only 2,8GB ram for the ESX and VMs to use and this then mean that it will swap a lot of memory out to the disk and is costing on performance.
For I/Os on the disk system the highest number I have seen in ESX esxtop is around 3950 cmds/sec.,

Here is a analyze of the load time and what wattage it is using, the analyze is showing what time is use and wattage from the power on to the VMs start to show idle time and is finish loading the stuff they do.
Time stampWatt used at that timeDescription
00:00124 - 126 wattPower on and the post process
00:02121 132 wattESX OS loading and VMs starting
00:19110 137 wattAll VMs is started and starting to be alive and reply to ping
00:49105 137 wattMost VMs is online and there services are working
01:19109 130 wattStarting to idle out and harddisk led is no on all time
01:40105 112 wattESX and VMs is now fully loaded the idle out

The power analyze is done with my Fluke 345 Power Quality Clamp Meter and is showing the Volt to be 233,1V AC RMS whit a power factor of 0.657PF

Here is some other wattage in others states
Power stateWatt
ESX 3.5 is load and no VMs started98 watt
In the BIOS screen 131 watt

So this is an old server and it is all too slow for me to be usable anymore and the wattage is use is all too high for what it is giving me of power and the only thing to say about it that it has been very stable and very silent design, In fact the server is so silent that I cannot hear it and it is below the background sound from outside the house.
So itservesme well and it is now time for a new setup.

So the new server setup will be something like this

  • Low wattage and most be lower than the 105-130watt that the old system use and my goal is to be at max 80 watt and lower be better.
  • Ultra low sound, I dont like fan and harddisk noise so it has to be silent
  • Low heat, the server is placed inside a closed rack without any cooling so it has to give away to low heat level as I can get it to do.
  • More CPU power, This is not the biggest deal as I most of the time not has max the CPU out and most newer CPU are much more power full then the old AMD x2 4400 is, so here i will try see if I can use a CPU there is having a low wattages but still has the CPU power that I need for the VMs.
  • Memory has been a big issue with only 2,8GB, so the goal is a ESX server with 16GB or more
  • Data volume, Harddisk is so last year and I think it will be all SSD to get the speed, low wattage, low heat and low noise.
    Data protection will be some raid level so the data is safe.
  • Backup, some way to do backup of the servers, because today I dont have this and the backup is power off the vm and manual copy.
  • Maybe multi ESX servers for the same storage system, so I also can have a test ESX server to play with and the VMs a storage on the same safe storage.
  • Maybe using some of the newer smart things like thin provision, Snapshot, deduplication, host cache and other smart new things.

So what are the plan and the new design?

I've donesome research and has now an idea for how this new setup will look like and it will most likely be two servers where one is the ESX server and the other one is a SAN/NAS server, yes I know it soundsa little
wild but I think it will give me all the things I need and still be with a low wattages.. I hope but testing will show.
So why two servers, if I only build one new ESX server then I will not get many of the new options and just get a server there is the same as the old one but faster at a lower wattage.
So the idea is to build a server there is a SAN/NAS server with all the smart options like
thin provision, Snapshot, deduplication, storage error alerting and many more things.

So now youre thinking what SAN is he talking about and
the answer to thatis a free software call Nexentastor there is very cool project there is found here http://www.nexentastor.org/ and it is free for private use up to 18TB.
I have used this product for some years now and it is super cool and the best thing for me is the deduplication option there is allowing me to storage a lot more data
on the SSDs then if I just used the SSD in an ESX server.

So NexentaStor gives me this

  • ZFS file system
  • Multi GB read cache (the server will have 16gb ram and about 14gb is used for read cache)
  • Online gzip compression
  • Online deduplication
  • Snapshot with almost no extra space cost(do to deduplication)
  • RAW volume access via ISCSI or Fiber channel(need adaptor and switch)
  • Share access via NFS and CIFS/SMB
  • Backup software can access storage data via NDMP, ISCSI, Fiber channel, NFS, FTP, Rsync or CIFS/SMB
  • Easy expanding volumes and shares.
  • Cool performance monitoring tools.
  • High performance I/O and Raid levels without the need for any high price raid controllers.
  • Can bundle multi networks interfaces into one multi gbit interface.
  • Support multi interface and VLANs
  • On the support list at VMware as a supported storage system, in fact they use it on VMworld for the lab systems.
  • And lot more.
But I will write more about the NexentaStor on a later blog post.


So the design for the SAN server is something like this:

Mainboard
CPU 64bit Intel
16gb ram (Gives a lot of read cache)
1x 64gb ssd for NexentaStor OS
3x 256gb ssd in raid z1(Storage for VMs)
4 port Intel 1gbit network interface there is setup as a EtherChannel so it work as one 4gbit link and multi Vlan



And the ESX server will look like this

Mainboard
CPU 64bit Intel
16gb ram (Gives a lot of read cache)
1x 64gb ssd for ESXi OS and host cache
4 port Intel 1gbit network interface there is setup as a EtherChannel so it work as one 4gbit link and multi Vlan



And last but also an important thing is the switch for connection them


1x Cisco 3750 (model 3750G-24TS) there is a 24 x 10/100/1000 ports and four SFP ports, 32-Gbps switching fabric, Layer 3, VLAN, QOS , IPv4 and IPv6 support,EtherChannel, LACP, PAgP, 128 MB DRAM and 16 MB Flash memory, and a lot more thing..

So this is a very super cool switch that I have got for a very good price from a firm bankruptcy sell.
The switch will replace an older
Cisco 3560 24port that I use today but that one is only a 10/100mbit switch but it is a layer 3 and very good, as a note this one I have posted another blog about back in time where I have made it silent and I will make blog about making the new switch silent also.


Now to the serverhardware.
In the research I have done I have found that I will go for the Intel 2100T CPU as it seems to have a low wattages and still have a lot of CPU power, so I think I will be good
enough for what I need and for storage I will have to go for SSDs as they are low noise, low wattage, high IO power and the only bad thing is the cost.
So I have been looking over of what hardware I already have and then order some of the other hardware parts so I can start doing some testing.

So here is the first setup.
MainboardAsus P8H67-M
6 sata port, 4 ram slot, onboard vga, onboard netcard, and other stuff
CPU Intel I3-2100T LGA1155 2,5ghz 2xcores with HT and 3mb cache
RAM16GB Corair XMS3 4*4gb 1333mhz DDR3
Harddisk OS1x 146gb 2 notebook harddisk
Harddisk DATA5x 3 harddisk of mixed 32gb and 80gbs
PSU300 fanless FSP-Group FSP300-60GNF

This setup is using around 35-60 watt and heat and noise is not low do to the harddisk noise and heat

So I have order some SSD to test how much better it will be.
MainboardAsus P8H67-M
6 sata port, 4 ram slot, onboard vga, onboard netcard, and other stuff
CPU Intel I3-2100T LGA1155 2,5ghz 2xcores with HT and 3mb cache
RAM16GB Corair XMS3 4*4gb 1333mhz DDR3
Harddisk OS64GB SSD Crucual M4
Harddisk DATA256GB SSD Crucual M4
PSU300 fanless FSP-Group FSP300-60GNF

This setup is using around 26-30 watt and there is no heat and noise


But I have been reading on the Internet that the normal PSUs are not very good when the power is low and there can be saved wattage by using a PSU there is call PicoPSU and that is micro PSU there is inside the computer and then there is a external PSU there is looking like a power supply for a notebook pc., so I have order one as I very much like the idea of getting the power supply outside the pc case to have less heat in the case and there for fan noise in the server will be a lot lower and if it will give lower wattages then this is just a plus
<><>
MainboardAsus P8H67-M
6 sata port, 4 ram slot, onboard vga, onboard netcard, and other stuff
CPU Intel I3-2100T LGA1155 2,5ghz 2xcores with HT and 3mb cache
RAM16GB Corair XMS3 4*4gb 1333mhz DDR3
Harddisk OS64GB SSD Crucual M4
Harddisk DATA256GB SSD Crucual M4
PSUPicoPSU-160-XT + 150W external power supply

So now the watt is down at 21 watt and can go up to 26 watt under high load, so 21 watt server that is cool and a very good start but it will not be that low when I am finish because I have to add some extra netcards and so on, but with 2 that servers it will still only be around 42 watt and then there is some room for adding the extra netcards and so on and still be well under the wattages of the old server..

Here is a photo of the test setup on the floor as it looks right now, so it need a bit more work before it is finish.

Watt Meter
Before you ask how I am measuring the wattages and how precise that is, I can say that I am using a Fluke 345
Power Quality Clamp Meter so it cannot get much better than this and I also have a Fluke 435 Power Quality Analyzer and a Fluke 1735 Power Quality Analyzer but the last two are not very good on low power but I will use them when I have the whole setup put together, but a lot more on this in a later blog.

This blog is part 1 in my build and I will post a number of blog post as I go along with the build.

Tooms @ 9 October 2011 23:52 | Comment | Direct link


Arkive list
2015 - November
2014 - August
2014 - April
2014 - March
2014 - February
2014 - January
2013 - December
2013 - November
2013 - October
2013 - August
2013 - June
2012 - December
2012 - September
2012 - August
2012 - May
2012 - April
2011 - November
2011 - October
2011 - September
2010 - May
2010 - March
2010 - January
2009 - December
2009 - November
2009 - October
2009 - August
2009 - July
2009 - May
2009 - April
2009 - March
2009 - February
2009 - January
2008 - December
2008 - November
2008 - August
2008 - January
2007 - December
2007 - November
2007 - October
2007 - August
2007 - July
2007 - April
2007 - March
2007 - February
2007 - January
2006 - December
2006 - November
2006 - September
2006 - August
2006 - July
2006 - May
2006 - April
2006 - March
2006 - February
2006 - January
2005 - December
2005 - November
2005 - October
2005 - September
2005 - August