When two nodes in a fabric share common variables, and they're constantly swapping those variables with each other, you create ?hotspots.? Hotspots occur when two nodes consume the bandwidth of the switch, and other processors can't get their data in or out of those two nodes.
The only way to detect hotspots is to have a statistical application that analyzes traffic patterns within the entire fabric. That's a large chunk of code from a software standpoint. Because fabrics are distributed architectures, none of the nodes knows what's going on in the rest of the architecture and cannot detect hotspots.
Another interesting problem that pops up is called ?loose sequential consistency.? It's also called the ?A-B/B-A? problem. Lets assume a processor wants to know the temperature of an oven. It wants to know if the fan is on too. It sends a request packet to the node controlling those I/O devices that says, ?What is the temp?? Then, that processor sends another request that asks if the fan is on.
The receiving processor reads the first request, but it now must configure the analog-to-digital chip and wait for the conversion to get the temp. Meanwhile, it reads the second request, goes out and reads the ?fan-on? bit, and sends that back to the original requester. Later, the CPU sends the answer to the ?temp? request. Now, the requester sees the answer to its second query first and out of order (B-A, not A-B).
Each CPU must create a transaction-numbering and time-stamping system to match up its received messages with its sent requests in a predetermined period of time. This application must run on every single CPU in the network. You can't take the data from the ?fan-on? request and use it to make decisions about the ?temp? request.
LCSN
Ethernet is a loosely coupled/shared-nothing architecture. Each node has its own resources and its own copy of the operating system. Also, each node is autonomous from the network. Determinism in such an architecture is virtually non-existent. No single node has any insight about the traffic patterns in the entire network.
The Ethernet 802 committees are adopting RDMA mechanisms (similar to those found in InfiniBand) in their new 10G specification. As a result, Ethernet fabric networks will move up and become SCSS architectures at the 10-Gbit/s level. Too many problems with LCSN architectures, like determinism, cannot be solved.
While we have many software and computer-science problems to solve with these different architectures, they do offer a step-function improvement in I/O and multiprocessing performance over buses. But none of the fabrics is going to be as deterministic as buses like VME.
Consequently, soft-real-time I/O applications like PCs and servers will accept fabrics first. Then, soft-real-time multiprocessing applications like clustered Linux servers will accept them.
Yet a bus such as VME, with its hard-real-time determinism, will continue as the only architecture that can meet the determinism requirements of many embedded applications. Fabrics will never replace certain buses because of the computer-science problems and the software burdens. But they do offer some tremendous benefits in certain applications that can effectively use TCSE or SCSS architectures.
When two nodes in a fabric share common variables, and they're constantly swapping those variables with each other, you create ?hotspots.? Hotspots occur when two nodes consume the bandwidth of the switch, and other processors can't get their data in or out of those two nodes.
The only way to detect hotspots is to have a statistical application that analyzes traffic patterns within the entire fabric. That's a large chunk of code from a software standpoint. Because fabrics are distributed architectures, none of the nodes knows what's going on in the rest of the architecture and cannot detect hotspots.
Another interesting problem that pops up is called ?loose sequential consistency.? It's also called the ?A-B/B-A? problem. Lets assume a processor wants to know the temperature of an oven. It wants to know if the fan is on too. It sends a request packet to the node controlling those I/O devices that says, ?What is the temp?? Then, that processor sends another request that asks if the fan is on.
The receiving processor reads the first request, but it now must configure the analog-to-digital chip and wait for the conversion to get the temp. Meanwhile, it reads the second request, goes out and reads the ?fan-on? bit, and sends that back to the original requester. Later, the CPU sends the answer to the ?temp? request. Now, the requester sees the answer to its second query first and out of order (B-A, not A-B).
Each CPU must create a transaction-numbering and time-stamping system to match up its received messages with its sent requests in a predetermined period of time. This application must run on every single CPU in the network. You can't take the data from the ?fan-on? request and use it to make decisions about the ?temp? request.
LCSN
Ethernet is a loosely coupled/shared-nothing architecture. Each node has its own resources and its own copy of the operating system. Also, each node is autonomous from the network. Determinism in such an architecture is virtually non-existent. No single node has any insight about the traffic patterns in the entire network.
The Ethernet 802 committees are adopting RDMA mechanisms (similar to those found in InfiniBand) in their new 10G specification. As a result, Ethernet fabric networks will move up and become SCSS architectures at the 10-Gbit/s level. Too many problems with LCSN architectures, like determinism, cannot be solved.
While we have many software and computer-science problems to solve with these different architectures, they do offer a step-function improvement in I/O and multiprocessing performance over buses. But none of the fabrics is going to be as deterministic as buses like VME.
Consequently, soft-real-time I/O applications like PCs and servers will accept fabrics first. Then, soft-real-time multiprocessing applications like clustered Linux servers will accept them.
Yet a bus such as VME, with its hard-real-time determinism, will continue as the only architecture that can meet the determinism requirements of many embedded applications. Fabrics will never replace certain buses because of the computer-science problems and the software burdens. But they do offer some tremendous benefits in certain applications that can effectively use TCSE or SCSS architectures.