Re: Building a linux cluster
Carbon_Rod, Thu Jan 20 2011, 05:48AM

Depends what types of problems you are trying to solve....

Sometimes two $300 graphics cards running CUDA can outrun a partition with 16 cluster nodes.

Cluster nodes work just fine off a PXE booted OS, and workstations can be set to "dual" boot in this way during the weekends. Also, a node costs under $250 for the bare bones hardware: motherboard, 2 cpu, power supply, expensive fiber NIC, and maximum ram.

For $5k, you may want to look into Amazon cloud nodes with GPUs.

Cheers,

Re: Building a linux cluster
Conundrum, Thu Jan 20 2011, 07:44AM

cluster made of YLoD PS3's?

some university made one as a black hole simulator IIRC.

the fix is really easy but takes a while, if you do them in parallel it should take you about a day of work to do 8 or so.

Alternate idea (if you can't get ps3's), build a cluster using netbook motherboards overclocked to 40% faster than stock, with liquid cooling.
Easy to do if you have access to a 3-D prototyping jig to make the cooling blocks...
(plus they run windows and with minimal hackery will boot "native" with no hdd's required!)

-A

Re: Building a linux cluster
Sulaiman, Thu Jan 20 2011, 08:25AM

I've no idea what kinds of programmes you wish to run but
I used to maintain mainframe/mini/unix/pc systems
and I think that the two best approaches are
- A single large multi-processor unix system, shared O.S./memory/disk etc.
- use every spare computing cycle of every PC on the network
(cloud-computing, co-operative computing, NFS-like systems etc.)

Irrespective of the above, I would expect to gain more overall performance improvement by spending $900 on a training course for the software than an extra server. Mastering the software tools and programming wisely based on a good understanding of the data and rules can give >10x performance compared to a poorly (average) thought out system.

Re: Building a linux cluster
Carbon_Rod, Thu Jan 20 2011, 08:48AM

Or... try this:


cheers,

Re: Building a linux cluster
Steve Conner, Thu Jan 20 2011, 10:04AM


Well, why not fit each compute node with two of those too?

NVidia actually sell the "Tesla" GPU-based "supercomputers".

Re: Building a linux cluster
Pinky's Brain, Thu Jan 20 2011, 02:10PM

With Sandy Bridge you won't have ECC.

Infiniband is an option, even on this budget.

Why not use rackmount?

Re: Building a linux cluster
Nicko, Thu Jan 20 2011, 02:46PM

Hmmm. Don't think the i7 has on-chip graphics. The Gigabyte board supports the i5 chips that do have graphics, but nothing else. You'll need a cheapo PCIe graphics card.

I may be wrong - I often am...

Edit: It seems the 2700k does have graphics when using the H67 chipset - I'd still double check with the supplier...

Re: Building a linux cluster
Bjørn, Thu Jan 20 2011, 03:10PM

Sometimes two $300 graphics cards running CUDA can outrun a partition with 16 cluster nodes.

Most comparisons you find compare a single threaded badly written C program with a fully optimized GPU program.

If the comparison shows 100x improvement on the GPU:
Running 8 threads on an i7 gives 6x improvement so we are down to 16x
Spending the same amount of time on the C code as the CUDA code, 8x
Changing over to use SSE, 4x

The GPU is still faster but it is not always so fast that it is worth the extra effort.

Re: Building a linux cluster
Dr. Shark, Thu Jan 20 2011, 06:05PM

Lot's of good suggestions, especially on spending the money on training - I'll forward that to my colleagues Seriously though, they are biologists and neuroscientists and would rather throw money at the problem than changing any of the code. We have a huge repository of old MatLab that we are using and trying to change any of that would probably set our research back about 5 years.

In principle I like the CUDA approach, but I know a lot more about computers than anyone else in my lab combined and I pretty much gave up when I tried it a couple of months ago. It seems it should be so easy for what we are doing, filtering data, training neural networks etc, but all the easy toolboxes like PyCuda only get you so far. A real example, matrix multiplication is cheap and fast, but as soon as you need to slice an array in a weird way or access it element-wise you need to write your own kernels. And once you have to write low level code you have to know about optimal block size, what sits in cache vs. main memory, it really becomes so complex that it takes less time in total to just run the code on a slow (CPU) machine. Maybe this is starting to change as the lastest MatLab release has CUDA support, but for know I think Bjørn is right on.

Same applies to PS3's btw, we want an x86 environment so nobody has to write low level code. We briefly discussed using Amazon cloud services but the machines will be under full load pretty much 24/7 so it would not be economical.

My take on buying "big" machines with say a quad socket server board so we could have all 16 cores and storage in one machine was that it's a lot more expensive than spreading over a number of smaller nodes. Even a 12 core Mac Pro or equivalent Dell are more expensive than what I sketched out, and that is with a lot less storage and memory. Overall it seems to cost roughly twice as much as building your own. Future expandability is another big plus for the cluster.

Re: Building a linux cluster
Pinky's Brain, Thu Jan 20 2011, 09:02PM

Mac???? Srsly? Even Dell isn't real the one to look for for cheap servers, try Supermicro.

For a lot of cores on a single motherboard you could also consider AMD, the processors themselves are of course significantly slower ... orders of magnitude better interconnect bandwidth though.

A 4 CPU 2 GHz AMD system with 32 cores would be around 2500$ without memory.

PS. in a 1U rackmount case ... compact, but not something you want to be near.

Re: Building a linux cluster
Carbon_Rod, Fri Jan 21 2011, 02:51AM

PS3 scientific computing libraries generally require the right compiler. Yellow dog offered the most features, but the cost per bogomip is now higher than cheap x86 boards. However, most of the neat stuff about cell architecture (and some of the correct libraries you would need) is now quietly no longer served by IBM.


Usually, a good place to start is traditional OpenMP, and OpenCL for transparent GPU support.

A moment of silence for sun's SGE... Oracle is turning it into an abomination too.
=(

Re: Building a linux cluster
Steve Conner, Fri Jan 21 2011, 11:02AM

Carbon_rod: As the "Bogo" prefix suggests, Bogomips are not a measure of performance

I have a Mac, as do many of my colleagues, but I can see how Macs might not be the most cost-effective compute servers. The high-end desktop ones are ridiculously expensive.

Cray have a new "deskside" model out. I have no idea how much it costs.

Re: Building a linux cluster
Dr. Shark, Fri Jan 21 2011, 05:48PM

Since nobody pointed out any serious flaws like the RAM or processor having too many pins to fit into the main board (I had that the last time I build a computer ) I went ahead and ordered the parts for one compute node and the raid server.

I don't know much about bogomips, but I hear that Macs have the best bogomips / TFLOPS ratio. Seriously though, I think this comparison hits the nail on the head, there is no significant "apple tax" once you compare similar high quality systems, but there is a 100% convenience tax to buying a complete system compared to building you own.

The biggest challenge will now be building the RAID-5 or 6 array using the hardware raid controller in a Linux environment. I am sure there are lots of things that can go wrong, so I will be back with more questions soon.

Re: Building a linux cluster
Sulaiman, Fri Jan 21 2011, 07:45PM

First, if your repository is of MattLab files then you may (will probably) find them not suitable for a distributed processing environment, some re-programming/compiling may be required.
Contact your MattLab supplier and ask for suggestions.

Re: Building a linux cluster
Carbon_Rod, Sat Jan 22 2011, 12:16AM

@Steve McConner
What metric would you recommend to quote your Linux kernel efficiency on symmetric multiprocessors in a heterogeneous networked parallel computer framework?
Prove this is optimal and you get $$$... I even paired the puzzle for you...=P
6 292 35 125 365 45 228 209 182 150 499 143 81 53 644 463 22 33 415 278 165 74 71 360 19 147 82 169 479 478 75 312 244 38 67 773 98 18 501 162 174 388 295 129 260 86 89 146 65 8 367 99 714 118 54 23 80 827 34 176 57 390 569 11 977 94 532 68 24 1 26 152 85 196 88 161 790 124 131 189 96 122 16 79 252 126 14 106 139 83 200 32 290 60 446 10 772 163 117 104 9 359 372 329 215 130 21 7 107 78 76 247 1485 338 2 316 90 596 20 254 70 571 4 43 141 115 97 583 144 15 205 132 270 36 91 100 214 206 56 231 194 5 37 297 253 40 563 49 87 61 1051 13 46 109 12 137 44 311 63 510 41 216 363 171 72 29 240 186 39 84 202 128 177 30 421 62 48 351 411 31 156 17 153 242 190 111 112 291 198 164 102 279 509 208 376 52 319 58 296 108 142 238 3 66 370 64 294 201 226 50 133


@Sulaiman
Indeed, MatLab and it's 89MB worth of "hello world" libraries has an export code generator for C/C++ (DSP libs too)... but it is not going to be optimized code.

@Dr. Shark
It may sound silly now, but you may want to start thinking about your file system choices if 2TB+ drives are in use.... he he he...

Re: Building a linux cluster
Dr. Shark, Sat Jan 22 2011, 01:17AM

MatLab has a parallel processing toolbox, but in my experience it is far inferior to using Python with openMPI. It uses java for all communications between cluster nodes and it's really sluggish. That's kind of beside the point though, since our problems can easily be parallelized "manually". The data are organized by days and it's easy to just run one weeks worth of data on one computer, the next weeks data on another computer, and so on. So the problem is data-parallel in a really trivial way.

@Carbon rod, the fs is definitely a valid concern, I was planning to go with ext4 with lvm on top. lvm seems necessary for snapshots so we could run nightly backups without having to interrupt the computing jobs. This would also allow adding more/bigger drives to the raid as storage need grows. Any thoughts on that?

Re: Building a linux cluster
Carbon_Rod, Sat Jan 22 2011, 03:45AM

LVM is only really helpful if you are not using a RAID card as it slows down the i/o.
And... if you boot from it... there could be "Problems" later...

A Cluster file system does not usually use a traditional Journal like Ext4 or Ext3. Note too that only the newest distros' utilities will support disk scans into the TB... (you may have to do some compiling)

There are also other things to consider like maximum file counts and size limits.

Re: Building a linux cluster
Steve Conner, Sat Jan 22 2011, 10:22AM


I have absolutely no idea what you're talking about The only metric I understand is Number of happy customers * profit margin.

But you can't deny that Bogomips basically measure how fast the processor can execute the no-op instruction, and are therefore useless as a performance metric. (If you were a CPU designer, imagine how easy it would be to schedule all those NOPs in parallel for an industry-leading Bogomip rating. )

Some other things to think about re the original topic:

Integer or floating point math? For some applications, like financial trading, floating point isn't good enough, the application uses its own arbitrary precision numbers similar to Lisp's "Bignums". The FPU doesn't get used, and the processor's integer math performance is what matters.

For the vast majority of scientific applications, double-precision floating point is what's used. (Again, nothing to do with Bogomips.) When looking at MFLOPS figures, be sure to find out if they're single or double precision. Some FPUs can do both at the same speed, but others are twice as fast in single precision, and you can guess which figure will get quoted in the advertising literature.

Finally, you have to consider how your algorithms will map onto multiple cores. You said that your jobs can be easily parallelised by hand, so it may be worth checking whether your OS will allow you to schedule each job on one core, using only the memory most local to that core.

Re: Building a linux cluster
Dr. Shark, Sun Jan 23 2011, 07:07PM


Everything we ever need to do is in float 64. Our data comes from a 12 bit daq, so it would potentially make more sense to work in single precision, but MatLab does not like any other format that 64 bit doubles. MatLab is usually pretty quick to pick up new instructions sets like SSE and the new AVX so with any luck the code can still run reasonably fast.

MatLab can be told how many threads to use and since it does not really do intelligent threading byt itself (how could it?), it usually makes sense to run multiple instances each constrained to a single thread. No clue whether is is doing anything smart with cache and ram access though.


We are not really shooting for a cluster fs, I don't know nearly enough about linux to be able to pull this off. The idea is to have all storage in the central node and mount that via NFS (don't know if NFS 3 or 4) on all of the nodes. Ideally all data, code and software would be on the RAID and the compute nodes would just have Ubuntu and nothing else installed locally.

Re: Building a linux cluster
Dr. Shark, Thu Jan 27 2011, 12:47AM

Double posting like a boss:



got the first bunch of components today...



starting with the RAID node, which was completely dead at first. Turned out the memory was DOA, which took me a couple of hours to figure out.



The compute note went together smoothly, installing Ubuntu from a flash drive takes only 20 minutes. Impressive stuff!



both machines ready to rock and roll! The blue screen is the raid controller which has it's own BIOS. It's initializing the 8TB array, which unfortunately takes 5 hours.



Once I can get some software on the RAID and test everything is working as expected, I will order the remaining 8TB of disks and three more compute nodes.

Re: Building a linux cluster
Josh, Wed May 18 2011, 07:10AM

Interesting stuff.
I've had some interest in doing a dual i7 computer utilizing the 2600K i7's because it seems over the years rendering, comping and such with Blender is becoming a rather addictive hobby of mine.
Since CUDA and OpenCL is rather young I've been thinking CPU power.
Anyways good luck with the cluster,
I'll definitely keep reading your thread.