SnapLogic Ultra Pipelines Deliver Real-Time, Enterprise-Class Integration To iPaaS

SnapLogic is taking another step to make iPaaS more high-performance and enterprise-grade. SnapLogic’s Ultra Pipelines can access and integrate data at nearly real-time speeds and deliver more data-centric integration capabilities. IDN talks with SnapLogic’s Timothy Lui.

Tags: APIs, ESB, integration, iPaaS, pipelines, real-time, REST, Snaplex, SnapLogic, Ultra Pipelines,

Tim Lui
product manager

"As companies embrace ‘cloudification,’ they are looking for the cloud to deliver faster and easier ways to handle all data-driven projects"

Integration & Web APIs
Enterprise-Grade Integration Across Cloud and On-Premise
June 25
Online Conference

SnapLogic is taking another step to make iPaaS more high-performance and enterprise-grade. SnapLogic’s Ultra Pipelines can access and integrate data at nearly real-time speeds.


The goal is to supercharge iPaaS to deliver businesses more data-centric integration options beyond SaaS, and eliminate any performance trade-off when considering cloud-based integration, according to Tim Lui, the product manager behind SnapLogic’s Ultra Pipelines.


Lui described SnapLogic’s focus on iPaaS [integration platform as a service] performance – and why now.


“At first, companies were looking at cloud integration as simply a way to connect with or other business SaaS systems. But as companies embrace ‘cloudification’ more and more, they are looking for the cloud to deliver faster and easier ways to handle all types of data-driven projects. We’re hearing customers tell us ‘performance’ is key to this new data-centric conversation,” Lui told IDN.


SnapLogic Ultra Pipelines deliver up to 10x the performance boost over earlier versions by a couple of main innovations.


First, the pipelines receive input from a website or an application and return data to the requested endpoint. Further, the Ultra Pipelines are always active, resulting in significantly faster data delivery compared to legacy application and data integration technologies, he added. The improvements can be dramatic, with integration latencies cut by up to 80 percent or higher. “We regularly see cases where integration [updates] go from seven seconds to under two seconds, even milliseconds,” Lui added.


Interestingly, this growing thirst for high-performing integration is not limited to cloud-based iPaaS, he added. “Many data integration solutions – both traditional on-prem and iPaaS -- are struggling today to keep up with growing data volumes, complexities and locations,” Lui said.


The reason is simple: This pain from poor performing integrations is now being felt by business users – not just IT.


“Real-time data delivery is crucial to supporting modern business applications,” Lui explained, noting the list includes some of today’s hottest sectors, including e-commerce, cloud-based SaaS apps, analytics and even Internet of Things (IoT).


SnapLogic’s vice president of engineering Niraj Nagrani put SnapLogic’s focus on super-high integration this way: “The frictionless, real-time processing that Ultra Pipelines deliver is what companies need to take full advantage of the cloud and big data.”


Beyond real-time delivery, SnapLogic’s Ultra Pipelines can also offer guaranteed delivery of data requests, integrations and even end-to-end transactions. It does so thanks to a newly-added ability to correlate inbound and outbound messages. This feature ensures that if a data request fails, it is picked up immediately by another instance and restarted without delay. To help IT and business users know if all these capabilities are up-and-running, SnapLogic’s Ultra Pipelines also provides on-going and automatic monitoring of performance and status of requests.


Beyond speed and scale, SnapLogic’s Ultra Pipelines also deliver simplicity. They automate many complex integration-related tasks that often require IT to take manual steps. For instance, with Ultra Pipelines, IT no longer needs to manually invoke or schedule multi-step and multi-point integration processes, Lui said.


SnapLogic’s super-charged Ultra Pipelines also provide: low-latency processing; tight security via HTTPS-secured “listening” technology; and authentication by using bearer tokens, which avoids a performance hit.


Add up these improvements in speed, scale, security and visibility and perhaps it’s not a surprise that SnapLogic’s Ultra Pipelines are causing companies to give iPaaS solutions a second look for many mission-critical business use cases beyond SaaS integration.

A case in point is commerce, Lui said.


“E-commerce transactions are proving a big driver, not just for the higher speeds we can offer, but also for our Ultra Pipelines to guarantee delivery,” he noted. While iPaaS may not be known to deliver the speed and reliability needed for revenue-generating apps. SnapLogic is able to deliver the fast and reliable integration ecommerce needs, especially to support multiple look-ups across data located anywhere for all sorts of information (e.g. customer, account, inventory, pricing, etc.), Lui added.


“Something we find customers really like is they gain this added speed -- without having to give up [cloud] scalability. So, customers gain huge performance benefits and still get the elastic architecture we’ve always offered,” Lui said. SnapLogic is also delivering iPaaS solutions for big data / analytics, and digital marketing, he added.


SnapLogic’s Ultra Pipelines ‘Always-On’ Capabilities, Architecture

One key behind the success of SnapLogic’s Ultra Pipelines architecture is the implementation of an ‘always on’ approach.


“Unlike some iPaaS or on-prem integration solutions that require a trigger or an event (such as a write or update) to activate an integration, our real-time integration technology puts in place a long-running connection that is always listening and waiting for instructions,” he said. “That way, SnapLogic [Ultra Pipelines] are instantly ready to execute when there is new data,” he said.


Also contributing to the Ultra Pipelines’ performance boost is SnapLogic’s underlying iPaaS processing engine, the Snaplex.


The SnapLogic Snaplex is a self-upgrading, elastic execution grid that streams data between applications, databases, files, social and big data sources. From the cloud, the Snaplex scales up and down based on two key variables

  1. the volume of data being processed or
  2. the latency requirements of the cloud integration flow.

Further, the Snaplex allows data and process integration flows to be triggered based on events or scheduled jobs, called via REST APIs, or invoked programmatically via the SnAPI.


Ultra Pipelines process native JavaScript Object Notation (JSON) documents to deliver low-latency processing to hierarchical data. Further, they work with all of SnapLogic’s 300+ pre-built Snaps (pre-defined connectors to SaaS, on-premises apps and data endpoints).


The release of SnapLogic’s Ultra Pipelines comes just as a recent Gartner report finds “real-time integration” is becoming more important to the enterprise. This spike in demand come s in large part due to the explosion in ‘extended enterprise’ solutions – those that need to link together on-premises, cloud, mobile, analytics, social and other off-premises apps and data sources.


Gartner’s report, New Integration Strategies and Technologies Target the Digital Business. said in part:

“By 2017, more than two-thirds of all new integration flows will extend outside the enterprise firewall. The need for companies to interoperate and collaborate in near real time in the context of business moments is not easily addressed via integration competency centers (ICCs). Traditional approaches to integration ICCs typically utilize to fulfill integration via long-range resource allocation and planning cycles will not work.”

A recent SnapLogic blog post also noted an informal survey that points in the same direction. “[W]e asked attendees [at a recent conference]: “What is the future of the ESB at your company?” 67 percent said they’re looking to move to a more flexible / agile integration platform,” it read.