Comm intermediate.cpp

 This program provides communication between nodes.       

comm_list is a list of mappings from a global space to a global space. Each node holds a portion of the lattice. When doing calculations often one node needs information from another node (usually when evaluating at the node boundaries). Each time there is a mapping, the entry is added.

Suppose we have two nodes as scene in figure 1. We use the linked_buffer_set function to do as follows. In figure 2, host 1 packs information from site A and site B into the linked buffer. Then host 2 unpacks the information. Likewise in figure 3, host 2 packs the information, and host 1 unpacks the information, it is the same linked buffer in both figures, so one host can either pack or unpack, but cannot do both at simultaneously. start_communication is when the gathering and sending occurs, and finish_communication is where the recieving and scattering occurs.

The way we decide which hosts send first, then recieve second and vice versa is the following algorithm.

If rank<host recieve, then send.

otherwise send, then recieve. This is handled by the send_recv_buffer_set function.

In order to get the information, we need to know the node rank where the data is stored, (this points to the origin of the data) then we need to find the offset, this is where the offset_list comes into play. The offest_list tells us the offset once we have indentified the host that we need to pull information from. (Such as Host 1, then the offset would be 1 (if we need the data from B)