Locality manager having memory and independent code, bus interface logic, and synchronization components for a processing element for intercommunication in a latency tolerant multiple processor

A method and an apparatus for reconciling communication and locality by enabling a user/programmer to write programs in an extended procedural language which explicitly manipulate locality. The multiprocessor includes a plurality of processing elements connected to each other each processing element having a unique address and including a central processing unit for executing local CPU code and reading and writing local data and a locality manager for controlling data movement between processing elements which are cooperating on a single task by executing local locality manager code which indicates when shared data can be released and when non-local data is requested.