Hi,
I've observed rather poor GPGPU performance of Kepler GPUs so far. I am using GTX680 and I wanted to get 690 or K10, but I am hesitating. It seems that GeForce works fast only with float2 or float4 data types whereas Fermi or Radeon 79xx are fast for all data types.

I prepared a benchmark for testing the peak flop/s for GPUs. If you are using GTX 680, 690 or Kepler Tesla, would you mind downloading it from my website, running and sending me the results? I would really appreciate it. For others, it can be also a quite good benchmark. If you are using weaker GPU, you mind consider to decrease the problem size a bit.

It's available as a Windows binary or Linux source and runs using OpenCL (it's included in CUDA, so should be ok as long as you have it installed).

The Windows one looks like this:


You can download it from:
http://olab.is.s.u-tokyo.ac.jp/~kamil.rocki/FlopsCL.exe

Or linux source:
http://olab.is.s.u-tokyo.ac.jp/~kamil.rocki/FlopsCL_src_linux.zip

Just make and run flops, the output should show something like this:

It seems that GeForce works fast only with float2 or float4 data types whereas Fermi or Radeon 79xx are fast for all data types.

I take it you mean the GeForce versions of Kepler, as opposed to K10?

The prices of Tesla's are prohibitive for most, so the 690 would be the maximum choice. For double precision the K10 would be terrible as it's just 2 Kepler GK104s; 190 Gigaflops vs 665 for the M2090. For singles the official figures are 4577 Gigaflops, compared to 1331 for the M2090, but that doesn't indicate the float type.

Trimmed (5850) Cypress on W7x64, for 13108 Blocks:
(float/float2/float4/float8/float16)
(430.9/861.9/1722.8/1721.8/1438.8)
The Cypress is best optimized for float4 and float8.

Wouldn't run on W2K3x64, so I can't report GeForce Fermi results (GTX470).

I don't know what's predominantly being used here (float/float2/float4/float8/float16), or the proportions (I expect it might vary from one task type to another), but the results might be of some interest to the researchers/dev's.

What does the x in floatx mean? The number of elements in a vector you're working on?

Your 'x' is for the vector data type or width, so float4 is for a 4 vector data type (.x, .y, .z, .w) or (.s0, .s1, .s2, .s3). An atoms location and mass can be described by this and a second four_vector used for movement with the last ordinate not used (-).

Anyway, your low Kepler performance at 64% peak performance is very close to 66.6% - the limit which you can achieve without being able to utilize the super-scalar shaders. All Fermis you tested don't have any super-scalar shaders (GF100,GF110), but all smaller ones do. If my supposition is true you'l also see a bit less than 66.6% of peak performance for these.

MrS

Anyway, GPUGrid uses CUDA, and this thread specifically discusses and lists relative GPU performances for here. OpenCL isn't even relevant never mind float4 performance. Such discussion would have been better started in a new thread.

I don't see how any GK104 Kepler would have significantly different architecture in this respect. So it seems likely that the K10 will just perform akin to any GeForce implementation, such as a GTX670.

If you have to buy now, it probably makes sense to go for a 7970 or 7990 - It's generally the case that AMD offers better performance for OpenCL, so long as you don’t throw anything too complex at it.

If you don’t have to buy now, the NVIDIA Tesla K20 GPU is planned to be available in the fourth quarter of 2012. This will use a GK110 Kepler GPU. Architecturally, this GPU will be very different from GK104; NVidia suggests that it will provide 3 times more double precision compared to Fermi based Teslas. I don’t expect a $3000 K20 is the answer, but depends on what you are doing. If you can use multiple GPU’s then tot up the purchase and running costs of several GPU’s/systems and work out what’s the best for your needs.

The rumor is that Intel's illusive Xeon 'Phi' expansion cards might actually turn up around the end of the year, to try and muscle in on the Tesla, but I wouldn’t bet on one of those being both inexpensive and useful.

I don't know where AMD's Sea Islands are but the rumor is that the HD 8000 will be ~40% more powerful than the 7000, at high tide.

If I try bigger problem size (4096), it crashes on double16.

[s0 s1 s2 s3] is OpenCL specific. x,y,z,w is the mathematical nomenclature that was originally adopted by accelerator cards - derived from Cartesian co-ordinates I expect,

You would see the same super-scalar issue on the GTX460 and K10.

If I try bigger problem size (4096), it crashes on double16.

A good introduction to the super-scalar aspect of current nVidia GPUs is written by Anandtech. Initially they talk about Fermis with compute capability 2.1 (the small ones). Here you've basically got 3 blocks of 16 shaders in each shader-multiprocessor (SM). To utilize all 3 shader blocks one would need to issue 3 instructions per clock. However, there are only 2 schedulers working on 2 independent warps. Each can dispatch 2 instructions, but the 2nd instruction can only be a shader-instruction if it's independent of the 1st instruction.

And this is where your example creates a worst case scenario, if I read it correctly (I'm coding in Matlab, not assembly ;).

fma.rn.f64 %fd4, %fd3, %fd2, %fd3;

Here you write the result of the fma into %fd4 (right?).

fma.rn.f64 %fd5, %fd4, %fd3, %fd4;

And in order to execute this instruction, the previous result must alread be known, since %fd4 is used as an input argument here. In this case the super-scalar execution can not be used. The schedulers will issue one shader instruction each, which uses 2 of the 4 dispatch units. In the end only 32 of the 48 shader in the SM are utilized (there could also be some load/store action going on).

Later on in that article it is described how the SM is extended to an SMX in Kepler, featuring 128 shaders guaranteed to be utilized and an additional 64 super-scalar shaders, all arranged in groups of 32 and shared among 4 warp schedulers, each featuring 2 dispatch units.

MrS

You seem to be getting more than the expected numbers. Is it overclocked?