HI:
i would like to set up a gpu computer for single-precision
calculations (molecular dynamics) that may last many days. Thus, consumer components may be OK, though of high quality.

it should be based on two/four GTX-470. ram requirements very modest: a few GB.

OS must be GPU-Linux 64 bit (i am familiar with debian amd64). Software raid 1.

I would appreciate advice on suitable motherboard, cpus, power source
and cage for large-diameter propellers.

thanks
francesco pietra

i forgot to mention that the X server will not be installed. graphics will be examined on a ssh-linked desktop.

GTX470s are getting hard to find now. The few that are around tend to be the same basic cost as GTX570s, which are probably better for your use (I'm guessing you want lots of CUDA cores).

GTX470- 448 CUDA cores, 607MHz core speed, 215watts.
GTX570- 480 CUDA cores, 732MHz core speed, 219watts.

As for motherboard/CPU, that depends. If you only want CUDA I dont think you need a SLI chipset motherbaord. If I'm right with that, then AMD systems are possible as well as intel.

Power supply, that would depend on your final setup, but I'd go for a corsair or seasonic power supply. With 2 x GTX 470s or anything like that, I'd be looking for at least 650-700watts.

'Cage for large-diameter propellers'? I guess you mean case, with large sized fan mounts.

1 members found this post helpful.

I would look at Supermicro.

http://www.supermicro.com/GPU/#SS

Yes, Supermicro has GPU-CPU mainboards but in my experience (I have a Supermicro motherboard for four-sockets multiple-CPUs) in Italy, through-Europe, Supermicro offers very inefficient technical support. They don't even know the values of potential at key position of the mainboards, so as to have efficient control from Linux of the temperature. They referred me to their "SuperDoctor", which needs a graphical interface, while computations are carried out on machine that do not have the X server, which is not needed and may only give problems.

At any event, I was asking for advice on specific components. The general architecture i know; where i have no expertise is in the available hardware to meet the task. Perhaps, i should summarize well known basic details of GPU-CPU computations. Thus, GPU molecular dynamics computing is very dependent on the specific software. Currently, only non-bonded interactions (which comprise the majority of interactions) are dealt with by the GPUs. Bonded forces and PME long-range forces are dealt with by the CPUs. In practice, currently a CPUs to GPUs 2:1 ratio is needed with all available molecular-dynamics software. And history teaches that software development is far slower that hardware development, particularly with GPUs, where a myriad of linkages are needed for parallelization. Obviously, CPUs are also needed for accessing data from disk and exchange data from nodes. Even in a shared memory machine (everything on a single motherboard), which is my aim, the CPUs direct the exchange. Therefore, there should be no major bottleneck on the mainboard-CPUs. This is my problem in finding the hardware. As the (classical) molecular mechanics software is compiled at single precision, my aim is to spare a lot of money by avoiding non necessary server components (server-type, such as Tesla and ECC ram would be needed by double-precision calculations, such as in quantum mechanics). The only server component needed for classical (newtonian) molecular mechanics are the power source (for four GPUs it will be 1300W minimum), the disks (2 tera in raid 1), and fans.

Perhaps such a mainboard exists with Gigabit, but i got lost with all their listing. Supermicro, other that for the problems alluded to above, is for server components as far as i know. Finally, I said GTX-470 but in view of your advice, it might be GTX-570 (unless buying second hand from eBay)

thanks
for advice
chiendarret

If CPUs to GPUs 2:1 ratio is the best setup, then I cant see you wanting 4 x GTX470/GTX570s. You cant get a 8 core desktop CPU.

6 cores are around, AMD Phenom X6 and some Intel i7 models. The AMDs are pretty afordable, but the intels are too expensive, the cheapest of the intel 6 core i7s (i7-970) is about $600 US. That is a LGA 1366 CPU as well, so there is extra expense for the motherboard and RAM. You can get a whole quad-core intel system for less than the CPU cost of the i7-970, and when you figure in the cost of the LGA 1366 board, triple channel RAM, etc, its a lot more.

You would also have the added expense of a bigger power supply- you would need 1000watts+ to run 4 x GTX470/570s. If you take corsair for example, an 1200-AX is about $280 (1000-HX is about $230) , where a nice 850-HX is only $140 or so.

Have you considered building 2 boxxen? It would be cheaper to build 2 x LGA 1155 CPUs, motherboards etc., with both setups running 2 x GTX470s/570s and , etc, than to get a single machine with an intel 6 core CPU and 4 GTX470s/570s running. IThat would also reduce the amount of heat output and make life a lot easier far as sourcing motherboaords go...to run 4 x GTX570s you need 4 x PCIe 16x slots, and there must be a double space between the slots so you can fit the cards in (all GTX570s are dual-slot cards)

Though I dont have the figures to know for sure, you might even find that a LGA1155 quad-core with 3 x GTX460s/GTX560Tis would do more work, use a similar amount of power, and be cheaper than 2 x GTX470s/570s. The onyl nasty part if getting a 3 x PCIe 16x LGA1155 motherboard, they are not cheap. But that would be offset by the savings on ther video cards and power supplies. GTX570s are about $350, you can get decent GTX460s for about $170.

I would avoid RAID1 unless you really need the space. You can get a SSD that is faster than RAID1 for about the same cost as 2 x 'normal' HDDs.

I would also really, really avoid getting GTX470s from ebay. You never know what overclocking they have been subjected to. There is also possible heat issues, (no airflow makes modern 'gaming' cards sick) you never know how well looked after teh ebay cards are.

BTW, I'm waiting for somebody to get back to me on CUDA on AMD boards, but if it works this is one of the setups I would consider-

AMD Phenom II X6 1075T ($180)
Coolermaster Hyper 212+ Heatsink/fan ($45)
Gigabyte GA-890FXA-UD5 ($180)
4GB DDR3 1600+ ($65 for decent stuff)
Corsair 850-HX ($140)
2/3x GTX460s. ($170 each)
OCZ Vertex II SSD 60Gb ($115)

$1065 with 2 x GTX460s. Sure, you could go for GTX470s/570s, but you would be looking at $700-1050 odd just for the cards. Not worth it IMO.

Case.....err, thats almost a personal choice, but the most you would need to spend would be about $160 on an antec 1200.

1 members found this post helpful.

Thanks, this was an extremely useful post. The idea of four GTX cards descended from my previous aim at a server-type motherboard. Then, I had the opportunity to run my simulations on tesla and 470, finding the two equivalent (as the simulation software is at single precision).

About the GTX 460, I have to verify if the simulation software has even be tried on these GTX. I hope so, your suggestion would be wonderful, also because we know that integrated GPU-CPU cards are under development. In fact, software developers in our area are waiting for them before trying to give to GPU more tasks. That hardware is never an investment is particularly true in this case. And particularly in Europe, where prices are so much higher than in USA. Also, sparing on the power source means sparing W/h, which are so expensive in my country (Italy).

As to two smaller boxes (which is also a fine suggestion) I have to verify if this has already been done on such a small scale. The software (NAMD) is one of the two best scaling, both on shared-memory machines and clusters (thousand of processor on high-end machines). However, those who use clusters have an electronic engineer in the staff, which is not the case here. Running truly parallelized software (by this I mean that system under study is subdivided into parts, imagine a protein subdivided into different groups of aminoacids and the different groups submitted to different CPUs or GPUs-CPUs, i.e. not a simple multiple running of the same procedure on the whole system) is quite easy on shared-mem machines. It can become problematic on cluster without hardware expertise.

thanks again.

I think ATI has their own library - ATI Stream...

http://developer.amd.com/gpu/AMDAPPS...s/default.aspx

To the OP, you might want to consider the GeForce GTX 590 then: dual GPU on one card with 3 GB of memory (1.5 GB / GPU). Purchase an SLI enabled motherboard (such as the Asus P8Z68-V PRO) and call it a day. If you get one of the new Sandy Bridge quad-core CPUs with HyperThreading (e.g. Core i7-2600K), then you'll have 8 logical CPU cores matched up with 4 physical GPUs.

http://www.anandtech.com/show/4330/asus-p8z68v-review/1
http://www.anandtech.com/show/4239/n...ngle-card-king
http://www.anandtech.com/show/4083/t...-2100-tested/3

Hope this helps...

EDIT: I should caveat my post with the statement that I have no idea to what extent Linux supports the motherboard or the video card.

Hmm....what setup was the tesla? I might have a better idea about balancing the CPU/GPU if I knew.

Sorry to make this more complex than it could be, and if this is info you already have. But there is a big differnce between some of those GPUs and CPUs, and what might seem to be balanced as far as CPU:GPU ratios might not be if checked in detail.

It might not have been. Seems to me from my quick look that NAMD is used mainly on server hardware, and they tend to use low level or 'server' level cards (nVidia quadro, ATI/AMD FireXXXX etc.). Since the GTX460 is a GF104, its almost an 'oddball' chip, and while its very similar to the GF100 that has been used in quadros, its not quite the same. There is one quadro that used GF104M (mobile), but how much that has been used to run NAMD I have no idea.

I wouldnt hold out much hope for the GPU on a CPU idea for your use. GPU on CPU tends to have a lot less power than a 'pure' GPU, so if it does work well then you should not need server/workstation or gamers cards for your calculations. A nice set of 3-4 normal 'desktop' (non-gaming) cards would do the job. GT220, GT430...a lot less power drain and heat, single slot cards. Much, much easier to setup and keep running than hot, power hungry cards.

Yeah, I guessed you might have clustering issues. A single box is always easier to work with. I only really mentioned it because its a lot cheaper, and a fair bit easier to setup 2 smaller cheaper computers rather than 1 monster.

Yeah, ATI stream (AMD FireStream now) is around. I have no idea if the software that the OP is using will work with FireStream though.....

I wouldnt even consider a dual-GPU video card for long term, high output use. The few I've seen used in those situations have all died before they should have (and I've seen a 7900GX2, 2 x 9800GX2s and a GTX295 die like that myself).

The GTX590s are also well over $700 even in the US. European prices would be even higher.

Hyperthreading does not give you a 'real' CPU. Maybe the software the OP is using will work in the prefered 2:1 ratio with hyperthreading, but without more data I wouldnt risk it. Nobody wants to blow $700+ on a video card, or set of video cards if its going to be limited by number of CPU cores.

To use CUDA you dont need SLI.

SLI is only useful for gaming, and to be honest is artifically limited (theres more than a few hacks to get SLI running with non-SLI boards). You shouldnt 'need' a SLI chipset to run cards in SLI, just nVidia wants the licencing fees. See the latest "nVidia anounces SLI for AMD 990X and 990FX chipsets" news. BTW, the kicker is its 'all or nothing'- either the motherboard manufacturers agree to pay the aprox $5 licence fee for every 990X/990FX motherboard, or they dont get SLI.

I wasn't implying that, rather that if the motherboard was SLI enabled, it was a quick rule of thumb to gauge if it could support multiple video cards (i.e. space and power issues), since the OP was primarily interested in consumer hardware.


That's why I wrote 'logical CPU cores' instead of 'physical CPU cores'.

True, but compared to a Tesla, they're cheap. Besides if you have reliability issues, that's what the warranty is for... (I believe EVGA offers a 'lifetime' one.)

Neither CUDA nor Stream have anything to do with the mainboard, they are purely related to the video-cards.

Just a sidenote.

I think that was the understanding throughout the thread...

I agree, I was only waiting to hear back because I know somebody who was having some nasty issues with CUDA and some motherboars. Funny enough, he said that it ran just fine on the AMD 790FX, but had the devils own time with it on some intel motherboard/CPU setup he replaced the 790FX with. The only reason he changed boards was because he didnt know there was a SLI hack, and he bought the SLI setup for folding and gaming.

Point, but I'd be wary of just any old SLI board. Or corssfire for that matter...

There is also the thorny question of PCIe x16 modes. Even that P8Z67-V PRO you linked runs x16/x0/x4 or x8/x8/x4 mode. The difference between x16 mode and x8 mode for gaming isnt that huge, but start running GTX590s in there and the real transfer mode has dropped to x4 (due to dual GPUs). Even with gaming that tends to chew over the same data, that can have a real impact....with NAMD it might have far more impact than with gaming.

Without doing a lot more research into NAMD I really dont know.

Neat, but doesnt negate the piont I was making. Unless you 100% know for sure that the 'extra' hyperthreading cores will work in the OPs perfered ratio of 2:1 CPU:GPU then you could be throwing away a lot of money on video cards, power supplies, etc..

Compared to quadros they are cheap as well. Tesla and quadros are made for long term, high output use. 'Desktop' or 'gaming' cards are not. Which is in part why the OP was asking here...

I've seen one friend of mine go through more video cards than is sane in a few years running maya and 3Dstudio max. Thats where one of the 9800GX2s I've seen dead came from.

I really wouldnt risk dual GPU cards for the sort of use the OP has in mind.

BTW, have you seen EVGAs warranty page?

http://www.evga.com/support/warranty/

"Limited lifetime", how long is that exactly?
You must register yoru product within 30 ways.
The OP would have to pay shipping to and from the US for RMAs.
Even something as simple as a sticker peeling off can void all warranty.
The ultimate 'get out of free jail card', they dont warranty against "Acts of God".