Next Generation HPC?

Exciting Time to be Doing Large-Scale Scientific Computing

Large scale scientific computing, 1995-2005 (ish):

• ~20 years of stability
• Bunch of x86, MPI, ethernet or infiniband
• No one outside of academia was much doing big number crunching

There have been many projects from within scientific computing that have tried to provide a higher-level approach - most have not been wildly successful.

• Many were focussed to specifically on one kind of problem (HPF)
• Or solved one research group's problems but people working in other areas weren't enthusiastic (Charm++)

But new communties are reawakining these efforts with the sucesses they're having with their approaches.

And we know we can have both high performance and higher-level primitives from a parallel library I use pretty routinely.

It's called MPI.

These work extremely well.

Before available, how many researchers had to (poorly) re-implement these things themselves?

Nobody suggests now that researchers re-write them at low level “for best performance”.

Researchers constantly re-implementing mesh data exchanges, distributed-memory tree algorithms, etc. make no more sense.

For the rest of our time together, will introduce you to three technologies - one from within HPC, two from without - that I think have promise:

• Chapel - an HPC PGAS language from Cray
• Spark - a data-processing framework, originally out of UC Berkeley
• TensorFlow - data processing for "deep learning", from Google

All of these can be used for some HPC tasks now with the promise of much wider HPC relevance in the coming couple of years.

Chapel was one of several languages funded through DARPA HPCS (High Productivity Computing Systems) project. Successor of ZPL.

A PGAS language with global view; that is, code can be written as if there was only one thread (think OpenMP)

config const m = 1000, alpha = 3.0;

const ProblemSpace = {1..m} dmapped Block({1..m});

var A, B, C: [ProblemSpace] real;

B = 2.0;
C = 3.0;

A = B + C;

$ ./a.out --numLocales=8 --m=50000

Chapel, and ZPL before it:

• Separate the expression of the concurrency from that of the locality.
• Encapsulate layout of data in a "Domain Map"
• Express the currency directly in the code - programmer can take control
• Allows "what ifs", different layouts of different variables.

What distinguishes Chapel from HPL (say) is that it has these maps for other structures - and user can supply domain maps:

http://chapel.cray.com/tutorials/SC09/Part4_CHAPEL.pdf

Running the Jacobi example shows a standard stencil-on-regular grid calculation:

$ cd ~/examples/chapel_examples
$ chpl jacobi.chpl -o jacobi                   
$ ./jacobi                                     
Jacobi computation complete.
Delta is 9.92124e-06 (< epsilon = 1e-05)
# of iterations: 60

Lots of things do stencils on fixed rectangular grids well; maybe more impressively, Chapel's concurrency primitives allow things like distributed tree walks simply, too:

• Compiler still quite slow
• Domain maps are static, making (say) AMR a ways away.
  • (dynamic associative arrays would be a huge win in bioinformatics)
• Irregular domain maps are not as mature

Chapel: Pros

• Growing community
• Developers very interested in "onboarding" new projects
• Open source, very portable
• Using mature approach (PGAS) in interesting ways

Spark is a "Big Data" technology originally out of the AMPLab at UC Berkeley that is rapidly becoming extremely popular.

Hadoop came out in ~2006 with MapReduce as a computational engine, which wasn't that useful for scientific computation.

• One pass through data
• Going back to disk every iteration

However, the ecosystem flourished, particularly around the Hadoop file system (HDFS) and new databases and processing packages that grew up around it.

Don't need to justify Spark for those analyzing large emounts of data:

• Already widely successful for large-scale data processing

Want to show how close it is to being ready to use for more traditional HPC applications, too, like simulation:

• Simulation is "data intensive", too.

So for instance here, nothing starts happening in earnest until the plot_data()

    # Main loop: For each iteration,
    #  - calculate terms in the next step
    #  - and sum
    for step in range(nsteps):
        data = data.flatMap(stencil) \
                    .reduceByKey(lambda x, y:x+y)

    # Plot final results in black
    plot_data(data, usecolor='black')

For data analysis, Spark is already there - like parallel R without the headaches and with a growing level of packages.

Lots of typical statstics + machine learning.

For traditional high performance computing, seems a little funny so far: Scalapack-style distributed block matricies are there, with things like PCA, but not linear solves!

Graph support will enable a lot of really interesting applications (Spark 2.x - this year?)

Very easy to set up a local Spark install on your laptop.

JVM Based (Scala) means C/Python interoperability always fraught.

Not much support for high-performance interconnects (although that's coming from third parties - HiBD group at OSU)

Very little explicit support for multicore yet, which leaves some performance on the ground.

• Tensors only means limited support for, eg, unstructured meshes, hash tables (bioinformatics)
• Distribution of work remains limited and manual (but is expected to improve - Google uses this)

TensorFlow: Pros

• C++ - interfacing is much simpler than
• Fast
• GPU, CPU support, not unreasonble to expect Phi support shortly
• Great for data processing, image processing, or computations on n-d arrays

All of the approaches we've seen implicitly or explicitly constructed dataflow graphs to describe where data needs to move.

Then can build optimization on top of that to improve data flow, movement; optimization often leaves room for improvement.

These approaches are extremely promising, and already completely useable at scale for some sorts of tasks.

None will replace MPI yet, but any have the opportunity to make some work much more productive, and reduce time-to-science