Several CEP (Complex Event Processing) engines have emerged from the growth of event processing in general. All those engines have a common goal of enabling event-driven processes and applications that react to single and to a composition of events, occurring at different times and within different contexts. However, the volume of heterogeneous information could grow rapidly and require urgent processing, taking into account the critical dependencies between actions. One key solution is to enable the cooperation of different CEP engines in order to share the management of the event-oriented system.
These CEP engines could be vendor-specific with heterogynous interfaces. Therefore, the federation of these distributed CEP engines to enable cooperation and interoperability is essential and imposes specific challenges like the ones listed below:
* *Communication heterogeneity*: The communication through the federated CEP engines is essential and may need different types of communication protocols. For instance, when interacting in a private network, an asynchronous protocol, such as JMS, may be much faster and as reliable as a synchronous one. However, when leaving the private network, it may be blocked due to firewall restrictions, and its reliability may suffer as well. In that case, a synchronous approach like REST could become more suitable and useful
* *Heterogeneous CEP*: The actors have different tasks which may generate multiple complex events. These events can then be processed by other CEP engines that may belong to the same or to different domains. Each group of actors can have different kinds of complex event processing engines. In this point, the challenge consists in supporting the interaction among heterogeneous complex event processing engines.
* *Scalability*: This distributed system could process a huge volume of events and should be able to handle increasing loads in real-time while maintaining other performance criteria such as latency, coherence between actions, etc.
* *Adaptability*: In an evolutionary system, constraints, contexts and actions may change over time. Therefore, the possibility to deploy new rules at run-time is essential to deal with the new situations.
All these issues are clearly within the scope of the SocEDA project, which addresses them in a holistic manner and encompasses technical aspects amongst others.
To overcome those challenges we propose a new framework called DiCEPE (Distributed CEP Engine). The DiCEPE allows the federation of distributed CEP engines that can deal with the heterogeneity of the event’s sources, address *scalability*, provide the means to hierarchically control, and adapt the rules of the different CEP engines in the distributed environment. This approach will allow a domain-specific engine to be managed by the experts of the domain, while providing at the same time a way to create collaborations among different domains. The DiCEPE's architecture is based on the EPN (Event Processing Network), but it improves it by filling the gaps that are still present in the EPN model. Since our solution deals with many CEP engines, we describe it in terms of a concrete SOA platform. The DiCEPE is implemented on top of FraSCAti. However, most of the concepts used in DiCEPE are generic, and we believe that a similar implementation can be done on top of other component frameworks as well.
Moreover, the DiCEPE supports both *synchronous and asynchronous communication*. The synchronous communication is a direct communication where communicators are time synchronized. This kind of communication requires communicators to be connected at the same time. The DiCEPE supports REST protocol for this kind of communication. Asynchronous communications have a long history in Message-oriented Middleware (MOM) technologies. Client does not wait on a non-blocking call, the SCA run-time can perform multiple retries if a target service is not available without blocking the client. This is particularly important for remote communications where service providers may be rendered temporarily unavailable due to network interruptions. Asynchronous interactions have an additional advantage in that they generally improve application scalability. As asynchronous protocol DiCEPE supports JMS and WS-Notification.
This is useful, for example in the crisis scenario, where in a private network the JMS protocol is sometimes more appropriate for communication among CEP engines given its speed and reliability. But, when we scale the scenario to a city using the Internet, the REST protocol is the better suited, since it is less likely to get blocked by a Firewall (since it uses HTTP), and also has great reliability.
There is no universal communication protocol that takes into consideration all those aspects. For this reason the DiCEPE platform provides access to several communication protocols that can be used as needed. Therefore, DiCEPE provides a solution to the *communication heterogeneity* challenge.
In addition, the DiCEPE platform inherits its reconfiguration capabilities from FraSCAti, so a component can be stopped, modified and restarted at run-time, without service downtimes. This reconfiguration feature allows, for instance, the DiCEPE to handle the deployment of statement rules at run-time depending on the circumstances or context. With this capability, DiCEPE addresses the *adaptability* challenge.
The architecture of DiCEPE is composed of four parts: \(i) _Engine_, (ii) _Statement_, (iii) _Listener_, and (iv) _Context_; as presented in the next Figure.
{center}
!DiCEPE.jpg|border=1!
Figure Overview of the DiCEPE Architecture
{center}
* *{_}Engine{_}*: This component acts as the engine instance, by which Statement components, events, and outputs (Listener component) are registered.
* *{_}Statement{_}*: A Statement component is used for querying the inbound event streams. This component is registered within the Engine component. The Engine component is connected to one or many Statement components.
* *{_}Listener{_}*: A Listener component generates a new complex event when an action is detected. Each Listener component is associated to a Statement component.
* *{_}Context{_}*: A Context component collects information, like the number of statement rules deployed in the engine at run-time.
These CEP engines could be vendor-specific with heterogynous interfaces. Therefore, the federation of these distributed CEP engines to enable cooperation and interoperability is essential and imposes specific challenges like the ones listed below:
* *Communication heterogeneity*: The communication through the federated CEP engines is essential and may need different types of communication protocols. For instance, when interacting in a private network, an asynchronous protocol, such as JMS, may be much faster and as reliable as a synchronous one. However, when leaving the private network, it may be blocked due to firewall restrictions, and its reliability may suffer as well. In that case, a synchronous approach like REST could become more suitable and useful
* *Heterogeneous CEP*: The actors have different tasks which may generate multiple complex events. These events can then be processed by other CEP engines that may belong to the same or to different domains. Each group of actors can have different kinds of complex event processing engines. In this point, the challenge consists in supporting the interaction among heterogeneous complex event processing engines.
* *Scalability*: This distributed system could process a huge volume of events and should be able to handle increasing loads in real-time while maintaining other performance criteria such as latency, coherence between actions, etc.
* *Adaptability*: In an evolutionary system, constraints, contexts and actions may change over time. Therefore, the possibility to deploy new rules at run-time is essential to deal with the new situations.
All these issues are clearly within the scope of the SocEDA project, which addresses them in a holistic manner and encompasses technical aspects amongst others.
To overcome those challenges we propose a new framework called DiCEPE (Distributed CEP Engine). The DiCEPE allows the federation of distributed CEP engines that can deal with the heterogeneity of the event’s sources, address *scalability*, provide the means to hierarchically control, and adapt the rules of the different CEP engines in the distributed environment. This approach will allow a domain-specific engine to be managed by the experts of the domain, while providing at the same time a way to create collaborations among different domains. The DiCEPE's architecture is based on the EPN (Event Processing Network), but it improves it by filling the gaps that are still present in the EPN model. Since our solution deals with many CEP engines, we describe it in terms of a concrete SOA platform. The DiCEPE is implemented on top of FraSCAti. However, most of the concepts used in DiCEPE are generic, and we believe that a similar implementation can be done on top of other component frameworks as well.
Moreover, the DiCEPE supports both *synchronous and asynchronous communication*. The synchronous communication is a direct communication where communicators are time synchronized. This kind of communication requires communicators to be connected at the same time. The DiCEPE supports REST protocol for this kind of communication. Asynchronous communications have a long history in Message-oriented Middleware (MOM) technologies. Client does not wait on a non-blocking call, the SCA run-time can perform multiple retries if a target service is not available without blocking the client. This is particularly important for remote communications where service providers may be rendered temporarily unavailable due to network interruptions. Asynchronous interactions have an additional advantage in that they generally improve application scalability. As asynchronous protocol DiCEPE supports JMS and WS-Notification.
This is useful, for example in the crisis scenario, where in a private network the JMS protocol is sometimes more appropriate for communication among CEP engines given its speed and reliability. But, when we scale the scenario to a city using the Internet, the REST protocol is the better suited, since it is less likely to get blocked by a Firewall (since it uses HTTP), and also has great reliability.
There is no universal communication protocol that takes into consideration all those aspects. For this reason the DiCEPE platform provides access to several communication protocols that can be used as needed. Therefore, DiCEPE provides a solution to the *communication heterogeneity* challenge.
In addition, the DiCEPE platform inherits its reconfiguration capabilities from FraSCAti, so a component can be stopped, modified and restarted at run-time, without service downtimes. This reconfiguration feature allows, for instance, the DiCEPE to handle the deployment of statement rules at run-time depending on the circumstances or context. With this capability, DiCEPE addresses the *adaptability* challenge.
The architecture of DiCEPE is composed of four parts: \(i) _Engine_, (ii) _Statement_, (iii) _Listener_, and (iv) _Context_; as presented in the next Figure.
{center}
!DiCEPE.jpg|border=1!
Figure Overview of the DiCEPE Architecture
{center}
* *{_}Engine{_}*: This component acts as the engine instance, by which Statement components, events, and outputs (Listener component) are registered.
* *{_}Statement{_}*: A Statement component is used for querying the inbound event streams. This component is registered within the Engine component. The Engine component is connected to one or many Statement components.
* *{_}Listener{_}*: A Listener component generates a new complex event when an action is detected. Each Listener component is associated to a Statement component.
* *{_}Context{_}*: A Context component collects information, like the number of statement rules deployed in the engine at run-time.