Patterns for real-world-aware and real-time solutions

Design patterns, originally introduced by architect Christopher Alexander,¹ are now an established software technique and the subject of many conferences such as those cited in References 2 and 3. Patterns in software architecture are intended to be reusable units of high-quality design that can simplify development and guide the developer toward high-quality solutions. Patterns also provide templates for model-driven programming and can be used to generate code for implementation by providing skeletal logic or support for particular quality-of-service (QoS) requirements.

We define real-world aware (RWA) solutions as applications that gather information from real-world sensors, actuate real-world devices, and integrate the sensed information and actuator behaviors into the typically transactional applications used to run an enterprise business. RWA solutions are usually distributed because information from many dispersed sensors and other sources is combined, summarized, and passed to a central processing point providing enterprise-wide optimizations and policy guidance. Because RWA solutions provide closed-loop control, they are usually also real time and require predictable end-to-end bounds on the latency of key functions that involve both processing and messaging. In addition to latency, they may also be required to meet specified levels of throughput, availability, security, and other nonfunctional QoS factors. RWA solutions are often complex, may have long lifetimes, and must allow for reuse and reconfiguration of existing subassemblies—hence, a component model is required.

In this paper we present a new pattern language characterizing programming styles, programming models, and middleware requirements useful for component-based RWA solutions. These patterns aid the design of systems in which edge components and end-to-end system interactions are required to meet specific real-time latency, predictability, and other QoS goals but may get higher-level optimization, direction, and support from business components executing in standard business middleware environments.

We follow the pattern documentation methodology formalized by Hohpe and Woolf.⁴ Our patterns are novel for current component reuse and business process integration technologies such as service-oriented architecture⁵ (SOA) and services component architecture⁶ (SCA). These technologies provide for assembling components with functionally correct behavior but need to be extended to address solutions with specified latency or other nonfunctional QoS properties needed for RWA systems. The SCA Policy Framework⁶ formalizes the annotation of components with policy and QoS expectations. A set of patterns for distributed systems and for distributed, real-time, and embedded (DRE) solutions is described in Reference 2. The ACE ORB^7-9 (TAO) is an object request broker targeted for object-oriented DRE solutions using some of these patterns in its implementation. The Component Integrated ACE ORB^10,11 (CIAO) is a middleware project that extends TAO to enable more general component integration, and it adds QoS aspects. More recently, the component synthesis using model-integrated computing (CoSMIC) approach¹² discusses the value of combining the CIAO framework with model-driven tooling.

The patterns developed in this paper for RWA and real-time solutions are distinct from those previously documented for business integrations, general distributed systems, and real-time object middleware implementation. We assume that business-component and DRE middleware, such as those discussed above, will evolve to address RWA solution needs. Similarly, the SCA Policy Framework could be enriched with RWA-relevant QoS annotations. Introducing a pattern language for RWA has the benefit of gathering in one place a catalog of relevant design, integration, and programming techniques for this emerging domain of software engineering. The examples provide use cases that can stimulate efforts to develop and test possible middleware and infrastructure extensions facilitating real-time and RWA solution development. Our pattern language suggests several specific modes for extending component specifications to address the needs of RWA and real-time solutions.

The nature of a pattern language is that more complex patterns are refined and broken down into related subpatterns. Figure 1 is the structure graph for the proposed pattern language. The patterns are discussed in the sections below bearing their pattern numbers and names. The numbers in red provide a subjective measure of our level of confidence in the validity and usefulness of each particular subpattern. Some patterns are more obvious than others and are easily recognizable. We have therefore given each pattern a confidence score ranging from 1 (least confidence reflecting our subjective assessment of prevalence, validity, and usefulness) to 5 (highest confidence).

Figure 1

To provide a complete description of the pattern language in a paper of reasonable length, we follow a two-layered approach. For a set of higher-confidence patterns (those scored at levels 3–5), we provide a complete description of eight patterns following the pattern template in Reference 4. Following that, we provide a short-form summary for the remaining patterns. All of the patterns have been described and reviewed in a patterns workshop.¹³ Together, this collection of patterns provides a starting set of approaches, concepts, and design techniques for componentized RWA and real-time solutions. Middleware and infrastructure support for these patterns would reduce costs and improve the quality of RWA solutions.

Known uses of this pattern include the following:

• In a financial services application, documenting that a market trade has occurred at the best price in all available markets

• In a real-time request-routing and load-balancing environment, ensuring that requests are routed in a manner that assures optimal responsiveness based on the current load data

• In a Federal Aviation Administration or transportation management environment, ensuring that critical commands are handled with current aircraft position and state information, such as the amount of fuel still available or monitored engine characteristics.

Value
This pattern helps organize application access to time-sensitive and rapidly varying data.

Context
This pattern is relevant under the following conditions:

• Streams of base information about real-world events arrive at a single site at which work requests using the received information must be processed. The arrival rate of base information is fixed.

• Each work request requires read access for just a few records in the received stream of data. The relevant data items for each work request must be identified, located, and accessed. The goal of the system is to support a given rate of work-request processing.

• Assurance must be provided that the data used in the work-request processing was current, i.e., the best data available and data that is known to be, for example, at most x milliseconds old.

Problem
How does one design an application to process incoming requests against information that must be assured to be current? Base information is arriving in a constant stream. The processing of arriving requests uses and depends on information from the base stream. However, the information in the base stream has a short, well-defined lifetime after which it is considered stale and must not be used in subsequent request handling. Lifetimes of single milliseconds and low tens of milliseconds for base data may be required.

Selection guidance
Use this pattern when the received data updates are needed at a single centralized processing site. If producers and consumers at multiple different locations have to share data with constraints on its timeliness, a messaging pattern as discussed in pattern 5, distributed shared real-time object, is more appropriate. This is not a cache pattern; caches are typically associated with maintaining local but possibly stale copies of data for dispersed users.

Forces
There is a natural tension between ensuring that no element of arriving data is lost and achieving minimal latency. Traditional (on disk) databases provide support for general search and retrieval but take the approach of requiring that data updates are transactionally consistent and completely written to durable (magnetic) storage before making them visible to any request processing. This tends to impose a base latency that will make the approach unsuitable for solutions with latency requirements in tens of milliseconds or less (e.g., a typical disk access time).

A second tension exists between the amount of base information stored and the flexibility of search and access patterns available to its users. Disk-oriented databases provide general methods for selecting specific data elements from a stored relation and accessing them rapidly using index information. Even for long-lived data, there are significant differences in optimal indexing techniques for file-systems-based and in-memory indexes.

Solution
The solution is to provide data that streams into an in-memory table in which individual data elements are time-stamped and expiry is managed. Dynamic indexes or hash tables are generated to allow efficient and fast lookup of the most recent relevant entries for use in requesting stream processing.

A first level of continuous operations is made available by storing arriving base data in at least two independent memories. If one memory fails, processing of incoming requests can failover (transfer on failure) immediately to an alternate memory. Both memories will do their own expiry processing. A second level of longer-term availability for audit purposes or to recover historical states of the system can be accomplished by streaming the base data to disk as it arrives. This requires that the persistent writing to disk can keep up with the arriving data rate. The state of in-memory tables at any point in past time can then be reconstructed from the audit log, but the trade-off is that there is no efficient search-and-retrieve capability for audit stream information once it has expired from memory.

In order to provide access to arriving information with minimal latency, real-time databases streamline and simplify the scopes over which transactional consistency is enforced. This is in contrast to databases that are used to support application servers and business record keeping where atomic (all or nothing) transactional update is enforced across general transactions that may include multiple database updates, message transmissions, and associated application logic.

Results
This pattern provides for management of time-sensitive data used to process a work stream. Timely handling of throughput and latency requirements of the work stream is likely to use the real-time component patterns 2–4 and 6–8.

The value of this pattern is that it contributes to the component-based assembly of real-time solutions by adding timeliness (and other nonfunctional QoS) contracts to component building blocks.

Context
Expect to use this pattern in complex real-time solutions that must be assembled from existing software subassemblies with defined real-time characteristics.

Problem
How do we get nonfunctional responsiveness requirements—and in particular, timing requirements—attached to appropriate parts of an application design and supported by tooling and middleware throughout the different phases of the real-time solution life cycle?

Selection guidance
This pattern is useful in real-time solutions that are complex enough to require component-level analysis of real-time nonfunctional behavior characteristics as a preliminary step to engineering the overall solution behavior.

Forces
Conventional component design is organized around an analysis of what is the simplest and most modular specification of its functional behavior with most of the information related to internal implementation choices hidden. In real-time and RWA solutions, nonfunctional requirements are as important as correct functional behavior. For example, the need for reliable and responsive behavior may often require that decisions be made locally (close to the sensors that provide input and actuators that must respond), even if this is based on a partial and slightly out-of-date view of the state of other parts of the system. There is a cost to having transactions be globally consistent, even when doing so provides the cleanest view of ideal system behavior.

Real-time solutions may also need to accommodate a family of heterogeneous implementations of a given component tuned to different deployment contexts but with uniform real-time behavior provided at the component interface level. Hiding the information concerning how a component implementation achieves its real-time behavior is as critical to the simplicity of a real-world solution as hiding the data structure is in implementing a conventional software component.

Solution
A conventional software services component that is consistent with SCA⁶ specifies ports through which it interacts with other components and also specifies the meaning and data types of the messages exchanged through those ports. A real-time software component should also specify its interface key patterns or cycles of external interactions and the timing statistics that will be met in the deployed instances of the component. Failure to meet this contract may be reported as an exception. The component may also explicitly express requirements for the processing and messaging characteristics of deployment platforms.

Results
Specifying the latency and throughput commitments of a component enables analysis of these behaviors in solutions that use the component. Pattern 3, distributed components with time budgets, addresses design and deployment of a component implementation to meet its real-time specification. The pattern-generated timeliness checks in the section “Additional short RWA solution patterns,” discusses automatically generated diagnostic and monitoring code useful in tuning a solution to meet its requirements. Pattern 4, adaptive configuration, and its subpatterns address how a component specification can be extended to define its connections to the external world, making it RWA as well as real time.

The general case of time budgets for top-level flows is a standard real-time engineering approach. In this pattern we have refined and shaped it to be a software design approach with a potential for tooling and middleware support.

Known uses of this approach include IBM Rational Rose Technical Developer¹⁴ (RRTD) which uses a special case summarized above. User code is supplied as set of interacting state machines in Unified Modeling Language (UML**) with associated timing requirements. The RRTD environment provides a runtime system that is an interpreter for the corresponding subset of UML diagrams. Tooling generates a representation of the source diagram interpretable by that environment. Some design-time analysis and validation of timing behavior is provided.

The Esterel synchronous design language uses similar validation techniques for embedded system designs.¹⁵

Value
This pattern is a first step in designing or specifying how a composite component meets it timeliness and other nonfunctional QoS commitments.

Context
This is a subpattern within pattern 2, components annotated with their real-time needs, used for timing requirements on composite components and subassemblies.

Problem
How can a real-time solution designer create reusable components with end-to-end latency with throughput specifications attached to them and have a reasonable expectation that these real-time requirements will be validated, assured, and monitored in all valid deployments of the component?

Selection guidance
Specifically, this is useful when the following occur:

• Components in a real-time solution have to be designed with specific end-to-end timing constraints on some of their flows or processing.

• When the component logic includes both processing and messaging, i.e., each deployment of the component may involve multiple target processors interacting through some network or connection fabric.

• The timing constraints are supposed to be met in all valid deployments on the component accounting for variability in factors such as target processor sets, loadings, and network connection speeds.

Forces
RWA solutions often need to be assembled from distributed components. A distributed component is a building block whose internal structure (not exposed in its interface) may consist of multiple program modules or subcomponents. A distributed component is deployed to a set of processors—a virtualized distributed processing system.

This is an extension of components defined in Java** Platform, Enterprise Edition¹⁶ (Java EE) and SCA.⁶ In Java EE, software components deploy to a single processor or execution container; SCA allows for composite components but does not provide a model for distributed deployment. Real-time solutions push us to a distributed component model because a designer has to attach end-to-end latency bounds to functions of a reusable component. The designer must also show the expected flow of processing and messaging interactions that will realize this function within the required time bound. A transactional business application with no time bounds and functionally correct behavior can always be organized as a set of single-module, single-site-of-deployment components; a real-time solution with end-to-end latency requirements must be built from distributed components with a time budget design for each reusable component.

A second related area of design trade-offs concerns the generality of the deployments intended for a reusable distributed component with end-to-end latency bounds. For instance, is a given component expected to be deployed only to a set of some number of gigahertz processors with no other workload, and each connected to its neighbors by a network with a message latency of some number of microseconds in order to achieve its specified latency properties? Such designs are not likely to lead to widely reusable component building blocks. However, the end-to-end latency delivered by the component may be a critical property required by the higher-level real-time solutions in which it is used.

Solution
The solution involves defining component timing interface contracts and component implementation timing budgets specified at levels that allow the following:

• Generic validation of component timing properties at design time

• Identification at design or compilation time of generic capabilities expected in the underlying execution infrastructure to meet timing contracts

• Subsequent checking and monitoring of underlying component and execution infrastructure capabilities during deployment and execution.

To proceed further with the solution, we need to define some concepts and terms. The building blocks of a real-time solution are organized as distributed components. Each distributed component has a component interface and a component implementation. The distributed component interface specifies how this component will interact with other components and with endpoint devices. The component interface will often include timeliness contracts specifying time properties that will be met by implementations of this component (most likely defined as annotations). It may also specify assumptions or expectations of the behavior and time properties of the partner components and endpoints to which it is connected.

The implementation of a distributed component shows how the processing of the component is organized. It orchestrates the functionality of contained program modules and subcomponents. The subcomponents, programming modules, and endpoints implementing a component may each be executing at different processing sites and interacting via messaging. A component implementation can be described using different styles ranging from event-driven, finite state machine hierarchies (used in RRTD¹⁴) to continuous feedback controllers. We present some example styles in patterns 6–8.

A timeliness budget is an assignment of time bounds or a time distribution for each part of the implementation of a function in the component interface on which there is a timing contract. In principle there are timing assumptions associated with processing, messaging, and subcomponent implementation steps.

With these concepts defined, the recommended methodology for the real-time solution designer is to organize the function of the real-time solution into distributed components with timing specifications, identify the program modules, provide a top-level flow implementation of the distributed component, and—for each end-to-end time bound on the component interface—provide a time-budget mapping to steps in the component orchestration.

Results
This design approach allows for tooling; middleware provides the following forms of support:

• Static validation of timeliness budget properties including the checking of subcomponent timeliness specifications, time path summation, and distribution properties

• Generic test execution and simulation-based validation of timeliness budget properties at the compile-and-test phase

• Generation of execution infrastructure checks to be used in subsequent deployment and runtime processing to ensure that the infrastructure is behaving as expected.

The combined effect of following this design methodology and having the tooling and middleware to support it is that the real-time solution will be assembled from reusable components that include predictable timing properties and enforceable assumptions as well as monitoring requirements on their execution platforms.

Substantial assurance of the timeliness behavior of a distributed component is available in a compile-and-test phase. The properties can then be used recursively in higher-level solutions using this component as part of the implementation.

A known use of adaptive configuration is in real-time continuous inventory control systems in a retail chain. A second example is for a broadly integrated soft real-time control system in a warship. Such a ship has to support being refitted periodically with changed armament types and quantities. It is highly desirable that this level of change be accommodated incrementally without invalidating the base behavior of the system and unchanged components connected into it.

Value
This pattern is a first step in enabling the design of RWA solution subassemblies adaptable for use in multiple physical environments and with specified endpoint entity connections.

Context
This pattern is useful in real-time systems, particularly those that interact with or control real-world sensors and actuators. They have to be reconfigurable to interact properly with different endpoint devices and different numbers of endpoint devices over time. Complex real-time systems almost always have this property because they are assembled and tested incrementally, first on small configurations of end devices and then on larger operational ones.

Problem
How does one organize the structure of a real-time application so that it is deployable to a wide variety of different system configurations with assurance that specified endpoint connections are made and that timing constraints are met in each configuration? If timing contract failures are possible, can clear advance warning of probable failures be provided at deployment time?

Selection guidance
Use this pattern when the real-time solutions or components do not necessarily need to be tied to one deployment environment; when there is a requirement to autonomically redeploy the solution or components in different environments based on usage and other external factors; and when there is a soft real-time solution and the flexibility of deployment is important.

Forces
When a change occurs in the endpoint devices with which a real-time application interacts, some adaptation must occur. The total cost of ownership of the application is likely to be unacceptably high if it has to be reconfigured every time the change occurs. This is particularly challenging, as the system would have to be formally retested and its timeliness properties revalidated for that particular configuration. Hence, a design approach is required that allows the system to be easily reconfigured to interact with different endpoint configurations while providing assurance that the required timeliness properties will be met in any new configuration—or, if there is significant risk that they will not be met, that advance warning is given.

Solution
The solution is to provide an adaptive system-component-coupling design that consists of a collection of real-time components with defined interfaces, interface timing requirements, and implementations supporting them; an adaptive schema expressing how different combinations and cardinalities of these components can be connected to provide configurations supporting different real-world system requirements; and an appropriate programming model, generated checking code, and component specification extensions for an adaptive configuration.

The idea is to construct the system out of components that have a component interface with defined real-time properties and an implementation supporting them. Major changes in system configuration are achieved by coupling different numbers of component instances into the system. Some form of hierarchical component connection and naming structure is used to provide orderly naming and logical connectivity to components. Attachment points are identified in this component configuration tree in which the number of attached components is variable.

Results
A central concept underlying this pattern is that useful real-time systems are grown organically by deploying components one at a time to match changing requirements in the structure of the physical world endpoints managed by the application or interacting with it. Particularly as soft real-time applications become more complex and their scope broadens, it will be unusual for all components of the system to be deployed at one time and then not reconfigured. This general pattern to address adaptive configuration may also be enriched using one or more of the subordinate patterns described in the section “Additional short RWA solution patterns.”

Known uses incorporating this pattern include the following: managing tracked objects generated from radar in a military scenario, managing order books in an exchange or auction scenario, or providing a multiplayer shared world in the context of high-performance multiplayer simulation or games.

Applications of this nature can use one of the standards below or one of their product implementations:

• Data Distribution Service for Real-time Systems is an Object Management Group standard defining interfaces for the distributed shared real-time object pattern¹⁷

• Real Time Innovations provides a product implementation of Data Distribution Service¹⁸

• In a simulation context, High Level Architecture–IEEE 1516 defines a similar distributed shared object pattern standard.¹⁹

Value
This pattern organizes interactions among independent subassemblies in a real-time solution using time-aware messaging.

Context
This pattern is useful whenever multiple independent applications, autonomous end users, or development organizations have to share real-time state information among themselves when timeliness of the information distribution is important.

Problem
How do we enable independent applications—possibly owned and developed by independent organizations and deployed in different places—to interact with defined bounds on the latency between emission and delivery of new information? The following is an equivalent formulation of the problem in a different context but with similar requirements: How do we provide shared distributed object middleware to end users of multiplayer games and virtual worlds so that a set of shared context objects can be both viewable and updatable to multiple dispersed autonomous users (each with their own goals and objectives) with tight bounds on the latency with which changes in the shared context are made visible to other interested users?

Forces
The forces at work in this pattern reflect the fact that in an environment of autonomous applications that require timely end-to-end behavior, each application is dependent on the timely delivery of updated state information from its partner applications. No individual application can take overall responsibility for this. Hence, a formal model governing the timing of the transfer of shared state information and the enforcement of this model by middleware is very helpful.

Solution
The pattern allows multiple end applications located at different nodes in a system to interact by means of shared objects with bounds on the lag until readers become aware of state updates. The solution is to provide support for real-time distributed shared objects typically implemented using a real-time publish-and-subscribe messaging infrastructure.

In this pattern, a number of applications potentially located at a number of independent processing nodes (in the Java case, each a real-time Sun JVM**) interact through a set of distributed shared objects. The time-dependent aspect of the pattern is a guarantee that each reader of the distributed shared object state will see updates within some bounded latency of the update being made regardless of which application was the writer. The pattern presumes that the processing nodes are linked by a messaging infrastructure of adequate bandwidth and predictable message delivery delay. Typically publish-and-subscribe messaging is used to distribute state updates of an object to all interested readers of that object. To ensure that a specific configuration of distributed shared objects is feasible and that readers are seeing updates with tolerable update latency, there are specific bounds on reading and writing rates that enable actual messaging patterns to be optimized for this. The applications are expected to declare their expected read and write rates. Topics have a fixed schema representing the shape of shared objects within a class and keyed topics allow the user application to declare an interest in specific object instances. Additionally, since the amount of bandwidth between readers and writers will most likely be fixed, the application must also specify priority for topics to influence the use of the bandwidth. Because the number of endpoints may vary dynamically, a discovery mechanism and failover support may be included.

The functional specification for a configured distributed shared object consists of rules, settings, and information defining the following:

• The nodes at which endpoint applications can reside

• The set of topics, i.e., shared object classes, supported

• The schema or state structure for each topic or object class

• For each topic, the keying structure and keying selection properties for instances of that topic

• For each endpoint application, bounds on its reading and writing rates and requirements for specific topics.

There may sometimes be a requirement to provide fairness as all readers of a state update see the update at the same time. In general this can only be done by increasing the latency with which updates are made available, and the amount of the increase is determined by the reader with the slowest connection, hence this is a significantly expensive option.

QoS, in which the delivery of messages is made reliable by hardening, is possible. This can ensure that particular objects or classes of objects have state that is recoverable after a node failure.

The presumption of the distributed shared object pattern is that a relatively uniform, fully connected mesh of connections is available between all nodes, avoiding the need for optimized multihop and reliable communications management. This allows the distributed shared object runtime to optimize the placement of state for objects, e.g., near primary readers, near primary writers, or centralized. In general, the pattern may be different for different state variables in each object instance. It is not necessary that all state of any one object instance be stored in the same place.

The value of this pattern is that it allows composite solutions to be engineered in which separate subassemblies use different real-time distributed system design styles.

Problem
What techniques are useful in specifying the implementation of a distributed real-time component in which pieces of user-provided logic, together with their annotation and top level flows, are assembled into a deployable component meeting the specified component interface?

Selection guidance
We have introduced the concept of real-time distributed component specifications and their annotations in previous patterns without being specific about the best technique for actually specifying the implementation of such a component. This pattern is likely to be useful when a choice must be made for an appropriate component implementation style for a real-time distributed component or when a real-time solution is to be assembled from multiple distributed components implemented using different techniques.

Forces
One driver of real-time component implementation techniques is the need to provide distributed programming model support. However, writing a distributed implementation to satisfy a real-time component interface specification is difficult, and wiring together a set of existing software assets into a reusable real-time distributed component with an interface is at least as difficult. Also, the available technologies for the component implementer to provide executable algorithmic logic are largely single site of execution, such as Java, C, or C++ objects, or finite state machines. Hence, typically it is the task of the distributed programming model support to wrap user-provided annotated code and use information from the top-level component implementation flows to generate a deployable distributed component.

There is no single widely recognized and standardized distributed programming model. The choice of best distributed component implementation in a particular context may depend on the particular behavioral characteristics required for that solution or on the taste and previous personal and team experiences of the implementers.

A final force at work is the need for clean separation of the component model from any particular choice of component implementation techniques. A key test of SOA was to show that client applications could use services without knowing whether the implementations were in C# or Java.

Extending this principle, it must be possible to assemble useful real-time solutions that combine distributed components implemented with different distributed programming models. The generic building block principle is critically important for real-time solutions.

The annotations for specifying timeliness, generating monitoring code, and enabling adaptive configuration in patterns 2–4 have been stated entirely at the component model level with no reference to any particular distributed programming model for implementing the component. This is essential because analysis and monitoring of end-to-end performance and latency, control of performance monitoring code, and adaptive configuration behavior will all depend on there being a single component model in which these concepts are understood and supported by at least one real-time middleware runtime system.

Solution
The solution to the problem of component implementation structure is based on three layers. First, at the base there is a layer that handles all real-time distributed components taking into account the timeliness properties of each component interface, the performance monitoring capabilities, and the component connection and adaptive configuration behavior that is independent of the component implementation method. Second, the middle layer has one or more distributed programming environments in which each real-time distributed component is implemented in a single distributed programming environment, although there could be multiple implementations. It also includes the tooling and support for the distributed programming environment wraps for user-provided code and logic modules in the component implementation using the top-level flow and annotations, which it assembles into a deployable component implementation artifact. Finally, the third layer at the top level in a component implementation consists of the user-provided code elements. These may be objects or state machines that provide specific algorithmic executable logic—Java, C, C++, or C#. The distributed programming environment of their component implementation interprets the annotations in the user-code elements and typically generates modules, each of which deploys to a single site at component deployment time.

We contend that the pattern of the three-layer separation of real-time component design, distributed-programming-model-aware environment, and user-supplied executable elements is a useful technique for the development of componentized real-time solutions and that it is the approach most likely to be supportable with helpful automated tooling programming models and middleware.

To further elaborate and illustrate the role of the distributed programming environment, we next introduce two subpatterns: pattern 7, asynchronous invocation coupling, and pattern 8, continuous control and dataflow coupling. There are other component assembly patterns that need to be able to interact with these new real-time patterns. Real-time complex event processing, RRTD real-time components, and traditional SOA components should all be able to be a part of a solution, including the more specialized real-time component styles proposed here.

This pattern helps the specification of composite real-time components using an asynchronous request distribution paradigm.

Context
This is an example of a design style for implementing composite components in an RWA solution. As we noted in pattern 6, particular contexts may motivate the use of different design styles. An example context for this pattern relates to the need for RWA solutions to be adaptable, allowing centralized processing to be replaced with more dispersed processing. A common occurrence in systems using radio-frequency identification (RFID) tags is that an initial system implementation uses a centralized controller to process the information from a set of RFID reader devices with limited processing capability. In a subsequent implementation, new, more intelligent reader devices become available, and responsiveness can be improved by migrating some formerly centralized processing out to the devices. Structuring components to facilitate this form of implementation change is of significant value. It may also be important to allow for mixed systems that involve both newer and older devices with the appropriate code deployment patterns.

Problem
What scripting style for implementing a real-time component with programming modules will allow convenient redistribution of the processing, e.g., easily allowing outbound function from a central controller to more intelligent devices?

Forces
In this component assembly style, the programmer's task is simplified. The behavior of each module is defined with case-specific logic explaining how it will handle arriving messages. Module interactions are specified in terms of logical message flows. The programmer loses some direct control of the exact runtime processing in a deployed component but in return can expect correct business-level logic behavior of the component when it is deployed to new contexts.

Solution
This design pattern can be used to construct a composite component using one or more event-handling program modules that interact using asynchronous message requests. It is a distributed variation on the proactor and reactor patterns described by Buschmann et al.² Event-processing modules interact in this way with each other or with the interfaces of the containing component. The component interfaces may be specified interactions (bound to other component interfaces at solution assembly time) or connections to physical endpoint nodes (PNodes) which are bound at component instance deployment.

We introduce the concept by characterizing the behavior of a pure asynchronous invocation component subassembly implementation. A component implemented with this pattern consists of one or more event-handling program modules wired together with asynchronous messaging paths. Figure 2(A) is an example.

Figure 2

Each asynchronous invocation programming module consists of the following:

• A user- or application-provided block of program logic with no embedded long-running operations requiring waiting or synchronous interactions with other program modules.

• One or more input queues, each with a well-defined signature specifying the data type of the request messages arriving on it.

• Each input queue is associated with an entry point in the program module, and the entry point expects a valid input request or message to be passed to it on each call.

• The body of the program module can include multiple SEND operations initiating asynchronous requests to send data to another program module (in the component), to actuator logical PNodes, or to containing component interfaces. The SEND statement will identify a port in the boundary of the program module. A component assembler will then establish the connection to a target destination, such as the input port in another program module or a port of the component using a graphical linking (arrow connector).

• An input queue may be fed from any combination of the following: SENDs in other program modules of this component subassembly, sensor logical PNodes associated with this component subassembly, or receiving interfaces of the containing component subassembly.

The semantics of the component execution are described by the following rules:

• Each program module could be executing on its own thread, which is potentially deployed to a separate processing node.

• Each program module is wrapped with module execution control that serially receives an input message: It calls the user-provided body and determines for each message sent to other modules whether to queue internally for asynchronous interactions or to serialize and send as a message.

This pattern is used and supported with tooling in the distributed responsive infrastructure virtualization environment project.²⁰

A known use of this pattern is the methodology and tooling support described in the e-Business Control Systems project by Ambrosio et al.²¹

Value
This pattern helps the design of RWA components using a continuous control and dataflow paradigm.

Context
This is a second example of a design style for implementing composite components in an RWA solution. As noted in pattern 6, real-time assembly, particular contexts may motivate the use of different design styles. This pattern is most likely to be useful when the particular function being provided in a component or the experience of the designer prompts conceptualizing the component as a continuous time feedback control system.

Problem
What style of component implementation will be natural for implementers to use to control system design, allowing specification of the flow of information between controllers and endpoints while relieving the user from designing and implementing specific messaging interactions?

Forces
Control feedback designers have a design viewpoint almost the opposite of that used by asynchronous process designers (as discussed in pattern 7). Control elements respond to a view of current system state by setting output control values in a style in which the frequency of execution of the control logic is a tunable parameter. The output controls are assumed to be setting a direction and rate of change that steers the state of the component back toward a goal state. This is a natural model when it is unrealistic to think of the state of some managed real-world entity being changed instantaneously by the action of a control element.

The forces governing this pattern are the tensions between presenting the digital feedback control designer with a cleanly specified distributed programming model for a continuous feedback functional design while allowing orthogonal specification and engineering of QoS characteristics and the generation of deployment-specific messaging and scheduling implementation code by the RWA middleware and infrastructure.

Solution
This design pattern can be used to construct a component out of one or more controller modules connected to each other and to the external interfaces of the component via defined flows of input and output state information. Each controller module is invoked periodically following some defined invocation rule. It can hold local state which persists from one invocation until the next. A controller module has well-defined input and output states, each consisting of a collection of typed input and output state variables. On each invocation of the controller, a current set of input state values is available for the program logic to read; a current set of output state values is available for the program logic to update. The logic of the controller is assumed to contain no long-running operations requiring lock resolutions or synchronous interactions with other nodes. Hence, latency analysis of the component can assume bounded processing time for each controller invocation. Figure 2(B) is an example of a continuous controller component.

Controller modules can interact via input and output state information flows, with each other, with logical PNodes, or with other interfaces of the containing component. The component designer specifies these dataflows as a wiring diagram. Sensor nodes connected to a component are a source of state information that can be directed to input the state of controllers in the component. The output state from controllers can be directed to actuator nodes attached to the component interface. Endpoint adapter logic may be provided to retrieve output from sensors and to drive commands to actuators in the component boundary.

Execution rules that control the frequency of execution are associated with each controller module. The frequency can be based on the static periodicity (i.e., every so many milliseconds), and it can also be determined by each time some subset of its input state variables is updated or by significant input state updates that use some specified predicate on the inputs.

From the perspective of a programming model, each controller can be considered to be an independent execution thread. The actual assignment of controllers to available physical processing engines and processes is deployment- and QoS-specific. The RWA middleware is responsible for wrapping the controller logic, invoking it in response to arriving input data following the specified execution pattern requirements, and for the local state of each controller between invocations.

A second point of simplification in this solution is that the RWA middleware generates, executes, and monitors logic to provide efficient messaging flows of input and output data between the controllers. Since the messaging requirements are likely to be specific to each deployment instance and QoS context, it is important to separate this from the user-provided logic in the controller and make it available for RWA infrastructure support.

This component design style is particularly natural to engineers designing control and DRE systems, but is less intuitive to transactional and business application programmers. It provides a design view that is an approximation of continuous time control. Successive invocations of a controller are related: Any inputs that have not changed since the previous execution are reused with their old value; any outputs that are not updated during the execution continue with their former value. This view contrasts with the invocation view used in business applications in which each service invocation or object method call comes with its own complete and independent set of input and output parameters and expects an independent service instance. The observed behavior of a system is expected to be dependent on the logic in the controllers but insensitive to the specific execution rules governing its frequency of execution.

Brief descriptions for the following three patterns are given below: logical endpoint roles in component boundaries, generated timeliness checks over component life cycle, and dynamic attachment points.

A reusable component designed to interact with physical endpoint nodes needs to be able to specify as part of its component interface how it should be connected to endpoints at instance deployment time. This specification includes the data types of messages sent to and received from the endpoint. It may also specify expected endpoint properties. The deployment infrastructure discovers endpoints and determines their properties (e.g., manufacturer and model number, the physical location and owner, and its function), which are used at deployment time to select which endpoints are to be connected to a particular component instance.

To support adaptive endpoint connection as described above, the containers into which components are deployed will not have fixed-processing capabilities as in Java Platform, Enterprise Edition¹⁶ (Java EE). The deployment infrastructure required for this pattern also has to allow for messaging and for dynamic discovery of physical endpoint nodes and their properties. Processing, messaging, and endpoint capabilities will be structured into adaptive, localized, self-organizing domains.

The solution to this pattern has three requirements: an appropriate component interface in which endpoint specifications can be included, a design technique showing how program module logic can interact with endpoints whose identity is not known until deployment, and a domain controller at runtime that can gather information about the properties of the available physical nodes in order to establish appropriate endpoint connections for each deployment.

The timeliness information on component interfaces described in pattern 3, distributed components with time budgets, may be used in multiple phases of the component life cycle to ensure that deployments of the component are valid and can be expected to meet all QoS characteristics relating to such factors as latency and throughput.

At component design time, a level of validation can be applied to ensure that time budgets defined for subelements or program module operations within the component are consistent with the time behavior commitments on the component interface. A model describing the abstract performance behavior of the platform on which the component will execute can extend the level of validation.

Significantly more information about the specific processing and messaging available to a particular deployment becomes available at deployment time. In pattern 3, adaptive configuration, we noted that there is considerable value in having the operator or deploying application informed at deployment time whether a specific deployed component instance is likely to meet its timeliness and other requirements. The deployment time aspect of this pattern is to have enough specification of timing requirements on the component for tooling to generate deployment time-checking code in order to determine whether the nonfunctional contracts on behavior expected for this component can be met in a particular deployment. Such checks will depend on a domain controller in the runtime system providing information on processing and messaging capabilities.

A variant of this pattern occurs when some engineering is expected on a newly deployed component to adjust its behavior to meet the requirements in the actual deployment context. Adjustments may include setting tunable constants and timeouts and identifying deployment-specific bottlenecks. This activity is dependent on having good monitoring of the time behavior of the executing elements. The generated code in this variant provides automatically generated measurement diagnostics that can be inserted, enabled, disabled, or removed to provide the system integrator with the necessary tuning information.

A final variant of this pattern involves generating runtime monitoring code that can provide early detection of a component failure within the specified timing limits. This monitoring code could detect changes in the performance characteristics of the underlying platform resulting from hardware or network failures that are likely to lead to a timing failure. The action taken when a performance failure is detected could be an operator alert or a predesigned failover to an alternate configuration.

The user-provided logic in program modules of a real-time component must be correct for all valid deployments. A program module that receives inputs from physical endpoints or attached subcomponents will often have to deal with different numbers of attached sources in each actual deployment of its subassembly—or even dynamically varying numbers of endpoints. For example, the number of thermostats in a household, the number of RFID antennas in a loading dock, and the number of radar systems on a ship are all likely to vary both by deployment and over time within each deployment.

The solution is to provide a programming model construct for user logic to interact with a dynamic number of attachment points. Such a solution should have the following properties:

• Can be used to interact with physical endpoints or with subassemblies

• Enables the middleware to discover elements with matching interface types that meet specified attach conditions, and to attach them

• Supports the program-controlled attachment of newly deployed subassemblies

• Enables the owning program to receive input from any valid currently attached element and to be informed of the identity of the source element

• Enables the owning program to send properly typed messages to a particular currently attached element using element identity

• Enables the owning application to specify a bound on the number of attached elements.

The patterns presented here were described using the approved pattern definition methodology, yet to some degree they also reflect real-time programming model concepts and ideas. This set of patterns is organized using formal pattern specifications. The collection of patterns provides a starting point for understanding and verifying our coverage of the real-time domain and for validating any tooling, middleware programming, and component models or runtime services for real-time solutions.

We are continuing the development of the pattern language and are updating the patterns as we discover more information and uses. There is a potential for broadening and adding to these patterns. Possible candidates include component interfaces and hierarchy, high-volume, low-information-density stream processing, out-of-envelope detection, and real-time provisioning and load balancing.

Many colleagues helped in identification, clarification, and discussion of these patterns and associated issues. We particularly thank Paul Giangarra, Tony Storey, and Branislav Selic. Other important insights came from Ron Ambrosio, Robert Berry, Han Chen, Penelope Gordon, Dimitrios Pendarakis, Michael Ward, and Mark Yao. Kyle Brown guided a preliminary version of the paper through the workshop described in Note 14.

*Trademark, service mark, or registered trademark of International Business Machines Corporation in the United States, other countries, or both.
**Trademark, service mark, or registered trademark of Object Management Group, Inc., or Sun Microsystems, Inc., in the United States, other countries, or both.

Accepted for publication October 31, 2007; Published online May 6, 2008.