1 - Introduction:
Early work in the development of control software for our workcell illustrated the problems which arise when using a single program to control multiple processes. The single program makes it difficult to control more than one process at once. The complexity of the controller, and the selection of programs was limited by the use of a single processor computer. As well, the number of controlled (interfaced) devices was limited by the number of serial ports on the computer.
The resultant computer problems lead to the subsequent development of a method which allowed the control program to be divided into smaller programs which utilized a crude file based communication method. This solved the previous problems, but increased the difficulty in communication. The programs had to have access to the same disk system, and had to be written to share files. And, although it is reasonable to expect a program to have knowledge of communication contents (i.e., requests, responses, or flags) they also had to be aware of the communication methods.
At this point we realized that we wanted to have many programs which communicate, but the communication methods available made this awkward. After examining the network tools,
- Signals - A mechanism for flags, or semaphores, between programs
- Sockets - Allow passing of messages and data from point to point
- RPC (Remote Procedure Call) - A remote process may be initiated
It was noticed that they were fast, efficient, and provided most of the basic functions for distributed processes. But, alone they did, not provide any functions for reasoning about communication sources and destinations. (Sockets were eventually chosen for our work)
Having a desire to overcome the previous problems, we considered their sources and then developed a list of objectives for an ideal communication system.
- Near real time response, in the range of one seconds or less. (this approach is not intended to compete with real-time systems).
- Buffering of messages between processes.
- General knowledge of processes, but no specifics.
- Maintain major methods of concurrent and distributed processing.
- Fault tolerance.
- Allow insertion into communication loop for testing analysis, and other processes.
- Allow message broadcast to unknown program.
- Keep running record of transactions for forensics, and restarting, in case of failure.
- Do not try to replace existing communication methods, but allow coexistence.
- To be independent of machine type, and language used.
This list of objectives required the addition of functions and information. If these functions were added to each program, speed would not be a problem, but the complexity of the communications would increase, and the system would be highly specific. This method is already used when designing high performance software, which a specific application. This method would also require that a complete set of information about other programs would have to be stored and maintained by each program. Any complex system of this design will have speed, but is difficult to develop and maintain. Assuming that a communication response time of one second is acceptable, we determined that a better approach was to have a separate communication server program which all of these processes talk through.
Thus, the MPS (Message Passing Secretary) was developed. The reader will note the similarity of the objectives (above) to those of a corporate secretary. Like the secretary, the job description calls for MPS to be responsible for the organization and the flow of information. Also, like the secretary, the message board is not concerned with the content of a message, and will not change the content of a message.
The clients are the processes which will pass messages via the MPS. They begin by informing the MPS of their presence. They identify themsleves as belonging to a group, and having a priority (or sequence) within that group. Then, when one client issues a message, it is adressed to a remote group. But, the message must pass though the lower priority clients first. After it has passed through the remainder of the group in sequence, it is sent to the highest priority of the other group. The question of who may read the message next is resolved by the MPS. The detailed operations of the system will be explained in Section 2.
As the workcell was the source of inspiration for this idea, it was also the ideal first test. The workcell is comprised of four active devices,
- a robot (CRS Plus),
- an NC milling machine (Dyna 2400),
- a pneumatic vice (controlled via CRS controller),
- a conveyor belt (controlled via CRS controller).
All of these devices are controlled through two controllers. The controller for the CRS robot controls the vice and the conveyor, as well as the robot. The Dyna controller is dedicated to the milling machine. Each controller is interfaced to separate serial ports on a Sun computer (running Unix). There were two separate controlling programs written to handle serial communications, communications to the devices (using device specific protocols), and accept requests received via the MPS. A Graphical User Interface (GUI) was developed to allow a user to issue requests to the devices, via the MPS.
A simple example of how the system works is a request for a robot move. The user requests a robot move with an input operation. This request is encoded, and it travels to the MPS. The MPS decides that the next client which may view the message is the robot controller. When the CRS robot control program requests available messages, it receives the request for a move, and then performs it.
Another program was written to test the interception (priority) feature of the MPS. This program has a higher priority than the dyna mill controller. Thus, it would intercept requests to the dyna mill controller, and normally pass them along. But, if the GUI had made a special request for a keytag (our sample test part), then it would convert this into NC code, and send a replacement message, which would direct the Dyna mill controllers to use the new NC code.
2 - The MPS - Message Passing Secretary:
The MPS uses two types of information. The first set of information is about the clients, and their specific attributes. The second set of information is the messages which pass between the clients. When a new client wants to connect, it passes information about itself to the MPS. The MPS enters the information in its client database, and informs the client. After this it will route messages as if the client is there, and allow other applications to refer to that group by name. The diagram below illustrates a possible client structure stored in the MPS.
Figure 2.1 - A Graphical Example of MPS Client Structure
As the reader will note in Figure 2.1, each circle represents an individual process (a client instance). When a client identifies itself, it uses a group name and priority. Therefore client #4 is in group NC_PLANNING, with priority -20. When a message is sent to a specific group, it will start at the highest priority, and be passed to the lowest priority.
Considering the figure above, assume that a message has arrived at group NC_CAD, which requests the production of a certain customer prototype. The message board will keep the message until client #5 requests the message. Assuming that client #5 is an intercompany billing module, it takes record of the request, and make a billing note. It then writes the message with no changes. The MPS takes this message and holds it until client #3 requests a message. Assume that client #3 is a CAD workstation were the prototype is designed. When the design is complete, a message is sent to the group NC_PLANNING, to say that the part should be produced for a customer. The MPS examines this message, and notices that there is no client next in line, so it holds onto the message until client #6 requests messages. This continues on as messages flow through the MPS. If at some future date the billing module (client #5) is considered to be obsolete, it is simply removed from the message board, and its function disappears. On the same note it is as easy to add software.
In many systems the structure above is fixed. This means that alterations to the software must be done off-line, and with rigid control. This approach means that the system modules may start and stop at any arbitrary time with minimal effect to the system. This also serves as the basis for fault tolerance. If a module fails for some reason (e.g. power failure) the program is simply restarted, and the system continues as normal. If a technician wants to perform diagnostics on a system, they simply run a program which has a specific priority, and they may have a direct window into the system operation.
The structure shown above may have many features added. The simplest feature is concurrency. Figure 2.2 below shows the effect of concurrency. To achieve this the user only needs to run new programs. For example in the figure 2.2 we see group NC_PLANNING (also refer to Figure 2.1) which now has concurrent processes. These are clients #1, #7, and #8. The two new processes (#7 and #8) simply identified themselves to the MPS as members of NC_PLANNING with priority 0. The effect is that when a message is sent by client #6 now, any one of the three clients may pick up the message. This leads to an easily obtained concurrency.
Figure 2.2 - An Example of Concurrency in an MPS Group
The mechanisms described are the basic functions required for the application of the MPS to workcell control. Other features will be described in later publications.
3 - The MPS model of the UWO CIM Cell:
Keeping in mind the components of the CIM Cell, the user can associate each one with a client. The various software components of the workcell are,
- A Graphical User Interface (group: GUI, priority 0)
- A Robot Driver (group: ROBOT, priority 0)
- A Milling Machine Driver (group: DYNA, priority 0)
- A Translator to make NC code for keytags from initials (group: DYNA, priority 10)
These four programs are assembled together as seen in Figure 3.1 Below. The typical mode of operation for these programs is that group GUI will issue requests to both groups. The actions are in the form of motion and status requests for the ROBOT group, and milling requests for the DYNA group. Both groups have the ability to send messages to the GUI group also. These often come as responses to status requests, or failure errors.
Figure 3.1 - The client structure of The workcell controller in MPS
The benefits of the scheme in Figure 3.1 became obvious when the development and testing of the software was underway. When a failure occurred, the program which failed was stopped (this was most often the GUI). It was fixed, recompiled, and restarted, without ever taking the workcell off-line. Also, the transaction log which the MPS keeps allowed the easy examination of events in the workcell, thus making it easier to locate the source of error.
The use of concurrency in this scheme becomes obvious when a process like the NC generator for keytags is used. If this process becomes a bottleneck, a second concurrent process can be started which relieves the overload. When the overload is passed, the second process is simply stopped. The result of this extra process can be seen in Figure 3.2 where client #5 has been run, and will relieve the load from the NC code generator.
Figure 3.2 - The Workcell Using a Second NC Code Generator for Keytags
