High Performance (Functional) Distributed Computing with DDS and Scala

Angelo Corsaro

Download the slides here tl_files/debs2012/img/icons/icon_slides.png.

The Data Distribution Service (DDS) is standard for high-performance, scalable and real-time publish/subscribe that has experienced an extremely swift adoption across several application domains, such as, Air Traffic Control and Management, Aerospace, Defense, Automated Trading, Internet Applications, and SCADA.

To give two examples, (1) DDS is today at core of the next generation European Air Traffic Control and Management system known as SESAR, and (2) DDS powers the control system for hydroelectric power generator of the Grand Coulee Dam -- the biggest dam in the Americas and the 5th biggest in the world!

The swift adoption of DDS stems from its ability to (1) scale from small deployment to ultra-large scale systems; (2) deliver very high performance -- high throughput and ultra low latency; (3) control key non-functional properties of data dissemination through a rich set of Quality of Service (QoS) Policies; and (4) enable the design of type-safe incrementally evolvable distributed systems.

Scala (pronounced Skah-lah) is a scalable programming language designed by Martin Odesky at EPFL that is quickly gaining acceptance in several different application domains and at different scales ranging from trading to internet applications. The interest around Scala is motivated by its perfect blend of Object Oriented & Functional Programming abstractions, its static Type System equipped with a very advanced type inference, its extensibility, its native compatibility and interoperability with Java and C#, and its elegance, conciseness and safety properties.

The main aim of this tutorial is to introduce attendees to "functional" distributed computing with DDS and Scala. As such, the tutorial provides an in depth overview of the key elements that characterize DDS and explains how this perfectly blend into Scala.

More specifically, this tutorial:

  1. Introduces DDS' abstract coordination model and highlight its connection with Stream Processing and Functional Reactive Programming.
  2. Defines DDS' structural type system and explains how it fosters the design of evolvable and type-safe distributed systems.
  3. Describes the key DDS' QoS Policies and introduces some of the most common QoS Patterns.
  4. Shows how some classic distributed computing problems, such as distributed mutual exclusion, totally ordered multicast, and distributed queues can be implemented starting from DDS primitives.
  5. Shows how the combination of DDS and typed functional programming languages, such as Scala (read the functional subset of Scala), can be used to design elegant, efficient and astonishingly compact distributed applications using the Functional Reactive Programming paradigm.

The tutorial is self-contained and does not assume prior knowledge of DDS or Scala.


Angelo Corsaro, Ph.D.
Chief Technology Officer

Angelo Corsaro, Ph.D. is OpenSplice DDS Chief Technology Officer (CTO) at PrismTech. As CTO, Angelo directs the technology strategy, planning, evolution, and evangelism. Angelo leads the strategic standardization at the Object Management Group (OMG), where he co-chairs the Data Distribution Service (DDS) Special Interest Group. Angelo is a widely known and cited expert in the field of real-time and distributed systems, middleware, and software patterns, has authored several international standards and enjoys over 10+ years of experience in technology management and design of high performance mission- and business-critical distributed systems.

Angelo received a Ph.D. and a M.S. in Computer Science from the Washington University in St. Louis, and a Laurea Magna cum Laude in Computer Engineering from the University of Catania, Italy.


Event Processing under Uncertainty

Alexander Artikis, Opher Etzion, Zohar Feldman, Fabiana Fournier

Download the slides here tl_files/debs2012/img/icons/icon_slides.png.

Big data is recognized by Gartner as one of the three technology trends at the leading edge a CEO cannot afford to overlook in 2012. Big data is characterized by volume, velocity, variety and veracity ("data in doubt").

As part of the big data applications, many of the emerging event processing applications are required to process events that arrive from sources such as sensors and social media that have inherent uncertainties associated with them. Consider, for example, the possibility of incomplete data streams, streams including inaccurate data, inaccuracy about time and location of events, and imprecise definitions of event patterns.

In this tutorial we will classify the different types of uncertainty that may be found in event processing applications, discuss the implications of uncertainties on event representation and event processing functions, discuss methods to reduce and handle uncertainties within the event processing logic, and discuss Artificial Intelligence based methods for uncertainty handling.

These will be illustrated using a running example.

Download the slides.

Alexander Artikis is a Research Associate in the Institute of Informatics & Telecommunications at NCSR "Demokritos", in Athens, Greece. He holds a PhD from Imperial College London on the topic of norm-governed multi-agent systems. His research interests lie in the areas of artificial intelligence, temporal representation and reasoning, and distributed systems. He has published papers in related journals and conferences, such as the Artificial Intelligence Journal, the ACM Transactions on Computational Logic, and the ACM Transactions on Autonomous and Adaptive Systems. He worked on the highly successful EU FP7 PRONTO project, being responsible for the event recognition work-package. Dr. Artikis has served as a member of the program committees of several conferences and workshops.


Opher Etzion Opher Etzion is IBM Senior Technical Staff Member and Event Processing Scientific Leader in IBM Haifa Research Lab, Previously he has been lead architect of event processing technology in IBM Websphere, and a Senior Manager in IBM Research division, managed a department that has performed one of the pioneering projects that shaped the area of "complex event processing". He is also the chair of EPTS (Event Processing Technical Society). In parallel he is also an adjunct faculty member in the rank of professor at the Technion - Israel Institute of Technology. He has authored or co-authored around 80 papers in refereed journals and conferences, on topics related to: active databases, temporal databases, rule-base systems, complex event processing and autonomic computing, he is completing a book on event processing called: Event Processing in Action (with Peter Niblett) and co-authored the book "Temporal Database - Research and Practice", Springer-Verlag, 1998. Prior to joining IBM in 1997, he has been a faculty member and Founding Head of the Information Systems Engineering department at the Technion, and held professional and managerial positions in industry and in the Israel Air-Force. He is a senior member of ACM. He has supervised 6 PhD and 19 MSc students.


Zohar Feldman is a research staff member at IBM research Labs in Haifa, Israel. He received a M.Sc. degree in Operations Research and Analysis from the Technion, and he is currently pursuing a Ph.D on Monte Carlo planning. He has several years of experience in developing simulation models and optimization algorithms, and applying them in real-life applications. His main research interests include the areas of applied probability, stochastic optimization, artificial intelligence, Monte-Carlo methods, and stochastic simulation.

Contact information: +972 4 8281019, ZOHARF@il.ibm.com

Dr. Fabiana Fournier is a researcher at IBM Research - Haifa, holding a B.Sc. in Information Systems, and a M.Sc. and a PhD in Industrial Engineering from the Technion - Israel Institute of Technology. Dr. Fournier has over fifteen years of research and practical experience, and published numerous papers in the areas of organisational and process modelling, business transformation, and management. Her research areas also include business process management, business operations, enterprise architecture, and SOA. In 2008, Dr. Fournier received the IBM Outstanding Technical Accomplishment Award.

Contact information: +972 4 8296489, FABIANA@il.ibm.com

tl_files/debs2012/img/persons/speakers/fabiana_fournier .png

Web Solutions Platform (Wsp) Event System

Keith Hamilton

Download the slides here tl_files/debs2012/img/icons/icon_slides.png.

The Web Solutions Platform (Wsp) event system is a general-purpose distributed publish/subscribe event system designed for high performance and low latency. The event system allows developers to build loosely coupled applications with just a handful of lines of code. The code is open source.

The Wsp event system is currently deployed across >8,000 servers in Microsoft Bing to provide a very fast communications infrastructure. It is primarily used by the Pulse Real-Time monitoring system.

This talk will describe the features of the Wsp event system, its architecture, its API, scalability issues, demos, and building a sample application.

Keith Hamilton is a Software Architect at Microsoft in the Bing organization. He has worked at Microsoft for the past 18 years in many product teams. The last couple of years he has worked on a real-time application monitoring infrastructure used in Bing, the Wsp event system, and defining the next generation solution for storage, especially storage for big data. Prior, he designed and built a reliable messaging product used for Microsoft Business Network and worked on teams in Visual Studio, COM+, and Systems Management Server.


On Reliability in Publish/Subscribe Services

Christian Esposito

Download the slides here tl_files/debs2012/img/icons/icon_slides.png.

Publish/subscribe services are required in several long-term on-going industrial projects that envision a radical rethinking of software systems by integrating existing legacy systems in large-scale federating architectures. In fact, such systems are made of a constellation of systems that cooperate with each other by means of the event notification provided by publish/subscribe services over wide-area networks. Such services have met an enthusiastic success in implementing this large-scale federations thanks to their intrinsic decoupling properties that improve the offered scalability guarantees. However, a very important requirement of such federations is the capability of the adopted publish/subscribe service to tolerate faults occurring in the network and/or computing nodes composing the federation, without negatively affecting the provided event notification. Therefore, it is crucial that publish/subscribe services are equipped with proper methods to support reliable event notification.

In this talk, we present this topic of reliable event notification by introducing its definition, a model of the faults that have to be tolerated, the available methods to recover from such faults and how current publish/subscribe products deal with reliability.

Dr. Christian Esposito is currently a grant holder at the Institute of High Performance Computing and Networking (ICAR) of Italian National Research Council (CNR) working on developing a notification architecture for exchanging patient summaries across sanitarian structures in different parts of Italy. He also collaborates with Consorzio Inter-Universitario Nazionale per l'Informatica (CINI) and Department of Computer and Systems Engineering (DIS) at Università di Napoli Federico II. He is involved with CINI in the EU project called CRITICAL STEP on investigating safety standards for critical infrastructures, and performing robusting testing and fault injection campaigns on publish/subscribe services. On the other hand, he is participating for DIS to the Italian project called PRIN DOTS-LCCI on studing event notification strategies within the context of critical infrastructures.

Dr. Christian Esposito graduated in Computer Engineering at Università di Napoli Federico II in 2006, and got his PhD at the same university in 2009. During his PhD studies he has been collaborating with Selex-SI, a compagny of the FinMeccanica group, in investigating the use of publish/subscribe services for building the European-wide infrastructure for Air Traffic Management.



Advanced Design Patterns in Event Processing

Adrian Paschke, Paul Vincent, Alexander Alves, Catherine Moxey

See the slides at SlideshareSee the slides at Slideshare.

This Tutorial extends the work taught in last year’s Event Patterns tutorial (Architectural and Functional Design Patterns for Event Processing at DEBS 2011). The tutorial is based on the EPTS Reference Architecture (version 1) with additional design patterns for Event Processing from a top-down (Reference Architecture) and bottom-up (sample code from popular event processing tools) perspective, based on the 4 levels of Event Processing Functions:

  • event preparation
  • event analysis
  • complex event detection
  • event reaction

Of interest to attendees will be the:

  • Reference Architecture
  • Event Processing Design Patterns for functions of the Reference Architecture
  • Implementation Patterns and sample code for the Event Processing Patterns, using IBM, Oracle, TIBCO Software and open source solutions
  • Case Studies from the Event Processing world that relate to these Event Processing Design Patterns.


Professor Adrian Paschke is head of the Corporate Semantic Web chair (AG-CSW) at the institute of computer science at the Freie Universität Berlin (FUB). He is director of RuleML Inc., vice director of the Semantics Technologies Institute Berlin (STI Berlin) and leads the Berlin Semantic Web Meetup Group. He is Steering-Committee Chair of the RuleML Web Rule Standardization Initiative (RuleML), co-chair of the Reaction RuleML technical group, founding member of the Event Processing Technology Society (EPTS), co-chair of the EPTS Reference Architecture working group (EPTS RA), co-secretary of the OASIS Legal RuleML group, voting member of OMG, and active member of several standardization groups such as the the W3C Rule Interchange Format working group (W3C RIF), where he is editor of the W3C RIF standard and the OASIS Legal RuML standard.


Paul Vincent is the fellow Co-chair of the EPTS Reference Architecture Working Group, Member of the British Computer Society, co-chair of OMG PRR rules standard, and CTO for CEP and Business Rules at TIBCO Software. He holds an MSc in Intelligence Systems, and has presented CEP and rule tutorials and presentations at various events over the past years (OMG Workshops, BRForum, Semantic Tech Conference, DAMA, etc) as well as providing customer training and guidance for event processing users in Fortune 100 organizations.


Alexandre Castro Alves is a computer scientist, who has worked in the past decade on middleware; particularly around network management, CORBA, real-time distributed systems, J2EE, Web-Services, BPEL, OSGi, and most recently on Complex Event Processing (CEP). He is a co-author of the WS-BPEL 2.0 specification, a member of the steering committee for the Event Processing Technical Society (EPTS), the author of the book 'OSGi in Depth', and holds several patents around BPEL and event processing. He is currently employed by Oracle, previously having worked for BEA Systems as the architect for the product WebLogic Event Server, re-branded Oracle CEP.


Catherine Moxey is an IBM Senior Technical Staff Member in CICS Strategy and Planning, based at IBM Hursley near Winchester. Catherine has more than 20 years' development experience with IBM, primarily in CICS and System z but also in WebSphere Application Server web services technologies. She is the architect for the Event Processing support in CICS Transaction Server, and an active member of the Event Processing Technical Society. Catherine has written a number of papers on topics including CICS and event processing, and frequently speaks at conferences around the world.



Where Event Processing Grand Challenge Meets Real-Time Web: PLAY Event Marketplace

Nenad Stojanovic, Roland Stühmer, Philippe Gibert and Françoise Baude

Download the slides here tl_files/debs2012/img/icons/icon_slides.png.

In this tutorial we will present an analysis of the recent development in the realization of the Grand Challenge and elaborate on its implementation using the current development in the so called Real-time Web - a decentralized, global, Internet-like infrastructure, built upon widely-accepted open standards.

We will present a particular solution that has been developed in the scope of the EU project PLAY, that is aiming to develop an elastic and reliable platform for dynamic complex event-driven interaction in large highly distributed and heterogeneous federated service systems. The main outcome is an Event Marketplace Platform which offers and consumes real-time data of all sorts and across the whole Web.

We will explain the theoretical principles behind the Event Marketplace, discuss the research and technological challenges required for its efficient development, present the implementation and evaluation details and demonstrate several applications that have been built on the top of the Platform so far.