Jack, H., ElMaraghy, W., Chien, T.W., “MPS (Message Passing Secretary) for Controlling a Manufacturing Workcell”
This document outlines a network based message passing program and its application in a manufacturing workcell to control events. The workcell is controlled by a number of programs, not necessarily on the same computer or in the same language. The networking scheme, called MPS (Message Passing Secretary), allows various programs to be quite ignorant of each other, while allowing complex, and robust interprocess communication. Some of the features of MPS based systems are: concurrent processes, fault tolerance, machine independence, and near real time response.
Early work in the development of control software for our workcell illustrated the problems which arise when using a single program to control multiple processes. The single program makes it difficult to control more than one process at once. The complexity of the controller, and the selection of programs was limited by the use of a single processor computer. As well, the number of controlled (interfaced) devices was limited by the number of serial ports on the computer.
The resultant computer problems lead to the subsequent development of a method which allowed the control program to be divided into smaller programs which utilized a crude file based communication method. This solved the previous problems, but increased the difficulty in communication. The programs had to have access to the same disk system, and had to be written to share files. And, although it is reasonable to expect a program to have knowledge of communication contents (i.e., requests, responses, or flags) they also had to be aware of the communication methods.
It was noticed that they were fast, efficient, and provided most of the basic functions for distributed processes. But, alone they did, not provide any functions for reasoning about communication sources and destinations. (Sockets were eventually chosen for our work)
This list of objectives required the addition of functions and information. If these functions were added to each program, speed would not be a problem, but the complexity of the communications would increase, and the system would be highly specific. This method is already used when designing high performance software, which a specific application. This method would also require that a complete set of information about other programs would have to be stored and maintained by each program. Any complex system of this design will have speed, but is difficult to develop and maintain. Assuming that a communication response time of one second is acceptable, we determined that a better approach was to have a separate communication server program which all of these processes talk through.
Thus, the MPS (Message Passing Secretary) was developed. The reader will note the similarity of the objectives (above) to those of a corporate secretary. Like the secretary, the job description calls for MPS to be responsible for the organization and the flow of information. Also, like the secretary, the message board is not concerned with the content of a message, and will not change the content of a message.
The clients are the processes which will pass messages via the MPS. They begin by informing the MPS of their presence. They identify themselves as belonging to a group, and having a priority (or sequence) within that group. Then, when one client issues a message, it is addressed to a remote group. But, the message must pass though the lower priority clients first. After it has passed through the remainder of the group in sequence, it is sent to the highest priority of the other group. The question of who may read the message next is resolved by the MPS. The detailed operations of the system will be explained in Section 2.
All of these devices are controlled through two controllers. The controller for the CRS robot controls the vice and the conveyor, as well as the robot. The Dyna controller is dedicated to the milling machine. Each controller is interfaced to separate serial ports on a Sun computer (running Unix). There were two separate controlling programs written to handle serial communications, communications to the devices (using device specific protocols), and accept requests received via the MPS. A Graphical User Interface (GUI) was developed to allow a user to issue requests to the devices, via the MPS.
A simple example of how the system works is a request for a robot move. The user requests a robot move with an input operation. This request is encoded, and it travels to the MPS. The MPS decides that the next client which may view the message is the robot controller. When the CRS robot control program requests available messages, it receives the request for a move, and then performs it.
Another program was written to test the interception (priority) feature of the MPS. This program has a higher priority than the dyna mill controller. Thus, it would intercept requests to the dyna mill controller, and normally pass them along. But, if the GUI had made a special request for a keytag (our sample test part), then it would convert this into NC code, and send a replacement message, which would direct the Dyna mill controllers to use the new NC code.
The MPS uses two types of information. The first set of information is about the clients, and their specific attributes. The second set of information is the messages which pass between the clients. When a new client wants to connect, it passes information about itself to the MPS. The MPS enters the information in its client database, and informs the client. After this it will route messages as if the client is there, and allow other applications to refer to that group by name. The diagram below illustrates a possible client structure stored in the MPS.
As the reader will note in Figure 2.1, each circle represents an individual process (a client instance). When a client identifies itself, it uses a group name and priority. Therefore client #4 is in group NC_PLANNING, with priority -20. When a message is sent to a specific group, it will start at the highest priority, and be passed to the lowest priority.
Considering the figure above, assume that a message has arrived at group NC_CAD, which requests the production of a certain customer prototype. The message board will keep the message until client #5 requests the message. Assuming that client #5 is an intercompany billing module, it takes record of the request, and make a billing note. It then writes the message with no changes. The MPS takes this message and holds it until client #3 requests a message. Assume that client #3 is a CAD workstation were the prototype is designed. When the design is complete, a message is sent to the group NC_PLANNING, to say that the part should be produced for a customer. The MPS examines this message, and notices that there is no client next in line, so it holds onto the message until client #6 requests messages. This continues on as messages flow through the MPS. If at some future date the billing module (client #5) is considered to be obsolete, it is simply removed from the message board, and its function disappears. On the same note it is as easy to add software.
In many systems the structure above is fixed. This means that alterations to the software must be done off-line, and with rigid control. This approach means that the system modules may start and stop at any arbitrary time with minimal effect to the system. This also serves as the basis for fault tolerance. If a module fails for some reason (e.g. power failure) the program is simply restarted, and the system continues as normal. If a technician wants to perform diagnostics on a system, they simply run a program which has a specific priority, and they may have a direct window into the system operation.
The structure shown above may have many features added. The simplest feature is concurrency. Figure 2.2 below shows the effect of concurrency. To achieve this the user only needs to run new programs. For example in the figure 2.2 we see group NC_PLANNING (also refer to Figure 2.1) which now has concurrent processes. These are clients #1, #7, and #8. The two new processes (#7 and #8) simply identified themselves to the MPS as members of NC_PLANNING with priority 0. The effect is that when a message is sent by client #6 now, any one of the three clients may pick up the message. This leads to an easily obtained concurrency.
These four programs are assembled together as seen in Figure 3.1 Below. The typical mode of operation for these programs is that group GUI will issue requests to both groups. The actions are in the form of motion and status requests for the ROBOT group, and milling requests for the DYNA group. Both groups have the ability to send messages to the GUI group also. These often come as responses to status requests, or failure errors.
The benefits of the scheme in Figure 3.1 became obvious when the development and testing of the software was underway. When a failure occurred, the program which failed was stopped (this was most often the GUI). It was fixed, recompiled, and restarted, without ever taking the workcell off-line. Also, the transaction log which the MPS keeps allowed the easy examination of events in the workcell, thus making it easier to locate the source of error.
The use of concurrency in this scheme becomes obvious when a process like the NC generator for keytags is used. If this process becomes a bottleneck, a second concurrent process can be started which relieves the overload. When the overload is passed, the second process is simply stopped. The result of this extra process can be seen in Figure 3.2 where client #5 has been run, and will relieve the load from the NC code generator.