HDF5 and Ubuntu

http://micro.stanford.edu/wiki/Install_HDF5#Ubuntu

CUDA shared memory array variable

http://stackoverflow.com/questions/9187899/cuda-shared-memory-array-variable

You have two choices for declaring shared memory inside a kernel - static or dynamic. I presume what you are doing at the moment looks something like this:

#define BLOCK_SIZE (16)

__global__ void sgemm0(const float *A, const float *B, float *C)
{
    __shared__ float As[BLOCK_SIZE][BLOCK_SIZE];

}

and you would like to be able to easily change BLOCK_SIZE.

One possibility is to continue to use static shared memory allocation, but make the allocation size a template parameter, like this:

template<int blocksize=16>
__global__ void sgemm1(const float *A, const float *B, float *C)
{
    __shared__ float As[blocksize][blocksize];

}
template void sgemm1<16>(const float *, const float *, float *C);

Then you can instantiate as many different block size variants at compile time as you need.

If you want to dynamically allocate the memory, define it like this:

__global__ void sgemm2(const float *A, const float *B, float *C)
{
    extern __shared__ float As[];

} 

and then add the size of the allocation as an argument to the kernel call:

size_t blocksize = BLOCK_SIZE * BLOCK_SIZE;
sgemm2<<< gridDim, blockDim, sizeof(float)*blocksize >>>(....);

If you have multiple statically declared arrays which you wish to replace with dynamically allocated shared memory, then be aware that there is only ever one dynamic shared memory allocation per kernel, so multiple items exits within (share) that memory segment. So if you had something like:

#define BLOCK_SIZE (16)

__global__ void sgemm0(const float *A, const float *B, float *C)
{
    __shared__ float As[BLOCK_SIZE][BLOCK_SIZE];
    __shared__ float Bs[BLOCK_SIZE][BLOCK_SIZE];

}

you could replace it with:

#define BLOCK_SIZE (16)

__global__ void sgemm3(const float *A, const float *B, float *C)
{
    extern __shared__ buffer[];

    float *As = &buffer[0];
    float *Bs = &buffer[BLOCK_SIZE*BLOCK_SIZE];

}

and launch the kernel like this:

size_t blocksize = 2 * BLOCK_SIZE * BLOCK_SIZE;
sgemm3<<< gridDim, blockDim, sizeof(float)*blocksize >>>(....);

All are equally valid, although I personally favour the template version because it can allow other compiler optimisation like automatic loop unrolling that the dynamic version cannot without extra work.

Rapid Problem Solving Using Thrust

http://on-demand.gputechconf.com/gtc-express/2011/presentations/Rapid-Problem-Solving-Using-Thrust.pdf

CUDA Libraries - Thrust
http://www.bu.edu/pasi/files/2011/07/Lecture6.pdf

c++ terminate called without an active exception producer/consumer

std::thread - “terminate called without an active exception”, don't want to 'join' it

30.3.1.3 thread destructor [thread.thread.destr]

~thread();

If joinable() then terminate(), otherwise no effects. [ Note: Either implicitly detaching or joining ajoinable() thread in its destructor could result in difficult to debug correctness (for detach) or performance (for join) bugs encountered only when an exception is raised. Thus the programmer must ensure that the destructor is never executed while the thread is still joinable. — end note ]

What this means is that you should not let threads go out of scope without first calling either join() ordetatch().

Concept of producer/consumer
http://stackoverflow.com/questions/7381757/c-terminate-called-without-an-active-exception-producer-consumer

Partition camping bank conflicts GPU/CPU

http://forum.cs264.org/index.php?topic=223.0;wap2

Bank Conflict and Warp Serializing

One can use the warp_serialize flag when profiling CUDA applications to determine whether shared memory bank conflicts occur in any kernel.  In general, this flag also reflects use of atomics and constant memory.

NVIDIA CUDA Example for Matrix Transpose


http://docs.nvidia.com/cuda/samples/6_Advanced/transpose/doc/MatrixTranspose.pdf

Intracommunicator and MPI_COMM_WORLD

http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content_communicator.html

• MPI_COMM_WORLD is the initially defined universe intracommunicator for all processes to conduct various communications once MPI_INIT has been called.
• In MPI-1, where the processes of executing the MPI computation, MPI_COMM_WORLD is a communicator of all processes available for the computation. This communicator has the same value in all processes.
• In MPI-2, where processes can dynamically join an MPI execution, it may be the case that a process starts an MPI computation without having access to all other processes. In such situations, MPI_COMM_WORLD is a communicator incorporating all processes with which the joining process can immediately communicate. Therefore, MPI_COMM_WORLD may simultaneously have different values in different processes

SPMD

We compiled a single program -- we didn't compile a different program for each process-- and we did this in spite of the fact that process 0 is doing something fundamentally different from the other process. This is quite common in parallel programming. In fact, most MPI programs are written in this way. That is, single program is written so that different processes carry out different actions, and this is achieved by simply having the processes branch on the basis of their process rank. (SPMD) Single Program multiple data.


--An introduction to Parallel Programming,
Peter S. Pacheco
University of San Francisco

Exascale Computing

Definition(Wiki):

Exascale computing refers to a computer system capable of reaching performance of at least one exaflops. Such capacity would represent a thousandfold increase over the currently existing petascale^[1]

Parallel Numerical Algorithms

Chapter 10 – Iterative Methods for Linear Systems

UIUC
CS 544

Maybe a good book?
http://www.amazon.com/exec/obidos/ASIN/052143064X/ref=nosim/weisstein-20