SnapLogic Embraces IoT in Latest iPaaS; Adds Value with Hadoop-Ready Integrations

With its Spring 2015 release, SnapLogic’s integration platform as a service introduces support for Internet of Things (IoT), via the MQTT protocol, innovative support for IoT data on Hadoop, along with plans to support other protocols and standards.   IDN talks with SnapLogic’s Jason Slater.

Tags: big data, cloud, integration, Hadoop, IoT, iPaaS, SnapLogic,

Jason Slater
solutions architect

"When it comes to IoT data, we see 100% alignment with our approach to big data and cloud application integration."

Integration & Web APIs
Enterprise-Grade Integration Across Cloud and On-Premise
June 25
Online Conference

With its Spring 2015 release, the SnapLogic integration platform as a service (iPaaS) introduces support for Internet of Things (IoT), via the MQTT protocol, innovative support for IoT data on Hadoop, along with plans to support other protocols and standards.   


By extending the SnapLogic Elastic Integration Platform for IoT, the company is extending the capability of its iPaaS from one-to-one and one-to-many to a new level of scale, one-to-millions, Jason Slater, a SnapLogic solutions architect, told IDN.


“When it comes to IoT data, we see 100% alignment with our approach to big data and cloud application integration,” Slater said. The update provides customers looking at IoT projects several crucial benefits, including seamless data validation, shaping and “message enrichment,” and capabilities that add context to data traffic for more actionable intelligence, he added.  

Driven by requests from current customers, SnapLogic Spring 2015 brings together its new IoT support with its expanding big data capabilities, The early IoT use cases indicate that integrating IoT data with Hadoop in a data lake is the direction the industry is headed, Slater said.


“Customers we talk to about their IoT strategy universally accept that traditional data warehousing approaches are not going to be feasible for this kind of big data storage and analytics… you simply can’t get ‘Things’ into a data warehouse,” he added.  While the world of consumable wearables is grabbing much of the media attention on IoT today, Slater said “industrial use cases are further along,” and of more immediate interest to the company.


Slater said the way SnapLogic combines IoT support with big data integration capabilities. This will remove much of the complexity of designing and delivering big data tasks. “Data scientists [and even business analysts] looking for big data insights don’t need to write Java code and complex queries,” Slater said.


SnapLogic’s iPaaS Architecture Brings Together IoT and Big Data

First, SnapLogic pipelines become integrated data flows, which can easily be assembled (snapped together) from SnapLogic Snaps, a library of pre-built intelligent connectors. These Snaps receive input from a website, application or data source – and return data to a requested endpoint. Snaps are designed to be smarter than traditional end-point connectors, as they can be inter-locked together one at a time to create customizable and configurable end-to-end pipelines.


The new MQTT Snap, which lets user build pipelines that connect directly to an MQTT broker for sensors, mobile and connected devices and stream data to analytical and other applications in real time.


Beyond MQTT, SnapLogic also has native and deep support for Hadoop 2.0. This is in large part thanks to SnapReduce, which enables SnapLogic to run natively on Hadoop as a YARN-managed resource (Yet another Resource Negotiator).  This approach lets SnapLogic “rapidly ingest IoT messages directly into a Hadoop cluster, and SnapReduce harnesses its highly scalable processing and delivers it to everyday users.”


SnapReduce offers real benefits to non-technical end users as well, Slater added. “From the perspective of end-user productivity and time to value, SnapReduce makes designing complex big data integration easy by dynamically generating MapReduce code behind the scenes, leveraging the Cascading framework.  This allows business analysts and data scientists to avoid writing Java code and complex queries and instead focus on what’s most important:  gaining insight from their big data,” he told IDN.


In large part, this YARN-friendly approach helps SnapLogic eliminate three top challenges of getting high-volume, high-speed IoT datasets into Hadoop – and ready for insights.  These steps are: acquire, prepare and deliver. 

  • Acquire:  The YARN-enabled SnapLogic HTML5 visual designer tool provides users a drag-and-drop way to bring data to Hadoop clusters, eliminating the need for complex coding. “Often, you need PhDs to write code or work with XML interfaces to get data into Hadoop. We remove that complexity and dramatically speed that up,” Greg Benson, SnapLogic’s chief scientist, told IDN. 
  • Prepare: SnapLogic tackles many time-consuming data prep tasks, including transform, aggregate, sort and even more complex correlation tasks to tie datasets together, (such as join, merge, map and more). “Users can even graphically transform and enrich data without any coding,” he said. [To further support data correlation, all data flows through SnapLogic as JSON documents and this enables it to handle unify treatment of data across multiple styles of integration, use cases and patterns.] 
  • Deliver: SnapLogic also can output data to the most popular BI, data visualization and other UI tools.


One other important ingredient in SnapLogic’s IoT / big data blend is high-throughput and high-performance. Earlier this year, SnapLogic increased the speed of its pipeline performance by 10-fold, to further support such IoT use cases. “Real-time data delivery is crucial to supporting modern business applications,” including IoT SnapLogic’s project manager Tim Lui told IDN in an earlier interview. 


This supercharged pipeline, called SnapLogic Ultra Pipeline, is updated in Spring 2015 to let users set up multiple receiving nodes to ensure there is no single point of failure and achieve higher availability and reliability. The Ultra Pipelines also natively handle high volumes of messages using HTTP, REST and now the MQTT protocol. “We support lightweight protocols such as MQTT via any standards-compliant external message broker. This allows us to be vendor and protocol agnostic . . .[and] can take IoT messages, validate, filter, rout and republish them to a JMS queue (for example, ActiveMQ or RabbitMQ),” Slater added.  

MQTT is just SnapLogic’s first journey on its IoT roadmap, Slater said. “In addition to MQTT, we’re working toward supporting a number of emerging standards to ensure IoT data is able to be managed effectively as part of an overall big data integration and data lake strategy,” Slater said.  Those include AMQP (Advanced Message Queuing Protocol), CoAP (Constrained Application Protocol), Open Mobile Alliance (OMA) Lightweight M2M, and ETSI Smart M2M, he said.


Other IoT additions include: New Snaps for HBase to allow customers to get data into Hadoop in an unbroken, streaming fashion. Also, to secure data, SnapLogic also added support for multiple instances authenticated with different Kerberos users on a single Hadoop cluster.

SnapLogic CEO Gaurav Dhillon underscored why SnapLogic is making an IoT commitment.  “The Internet of Things represents the next wave of big data in the enterprise,” he said in a statement. “As IT organizations move away from legacy data and application management approaches and toward Hadoop, Spark and the data lake, SnapLogic’s scale-out architecture and breadth of connectivity have become essential ingredients for the modern integration fabric.”


Other SnapLogic Spring 2015 Updates


Beyond IoT support, SnapLogic Spring 2015 continues to enrich iPaaS with features, flexibility and performance to make enterprises feel more confident about migrating to cloud-based integration, and away from traditional (on-premises) ESB and ETL (extraction, transformation and loading) approaches. Notable updates include:


Public Monitoring API:  In addition to an API for managing users and groups, a new public monitoring API has been introduced allowing users to proactively query pipeline status and take advantage of their existing monitoring tools to track the health of SnapLogic integrations.


Lifecycle Management Updates:  This allows users to manage pipeline development, creating and testing pipelines before moving them into production.  If anything fails during a promotion from development to test to production, the project cannot be promoted to the next phase.


Current SnapLogic customers can be provisioned with the new MQTT Snaps. Those looking to learn more can contact SnapLogic.