ClusterHQ Fills DevOps Gaps; Adds Automated Testing and Continuous Integration Tools for Docker Data Volumes

ClusterHQ aims to fill the growing need for better tools for automated testing and continuous integration to support DevOps, especially for building microservices-based apps. The company is bringing two key capabilities to its container data management solutions.  

Tags: apps, CD, CI, cloud, ClusterHQ, code management, collaboration, data management, data volumes, developer, DevOps, Docker, Git, microservices, testing, tools,

Flocker Hub

In today’s world of agile software development, it’s extremely hard to develop microservices-based applications without automated testing and continuous integration.


Even companies that take steps to put these in place can still face gaps, particularly when it comes to achieving effective data management across their entire application lifecycle.


ClusterHQ aims to fill this crucial gap with new additions to its container data management solutions, providing organizations with the flexibility to move data between diverse systems, ranging from laptops, test environments, data centers and clouds – complete with thorough version and access controls, according to ClusterHQ’s vice president for products Mohit Bhatnagar.  


“The flexibility to have data where you need it, when you need it, no matter the underlying infrastructure, has an immediate and positive impact on DevOps teams doing continuous integration, continuous delivery and automated testing,” Bhatnagar noted.


ClusterHQ’s engineers took their inspiration for these updates from the distributed version control, source code management and collaboration features known to Git and GitHub users.  In specific, ClusterHQ is adding:

FlockerHub, a version-control repository for Docker data volumes. It allows DevOps teams to store, share and keep track of any Docker data volume; distribute their data volumes to any host; all with access controls for users or servers to ensure data governance.


Bhatnagar described the approach as being “like GitHub for data” because it resembles how GitHub acts as a version-control repository for software source code.  FlockerHub is available initially as a ClusterHQ hosted service. 


Fli  (pronounced “fly”) is the command line interface (CLI) to FlockerHub. Fli allows DevOps teams to version-control Docker data volumes in the same way they use Git to version-control code. Fli can take incremental snapshots of any database or data volume that runs on Linux, and push those snapshots to FlockerHub where they can be pulled by any access-controlled user. 


Bhatnagar called Fli “a Git-like command line interface that runs on any Linux server or laptop, lets developers snapshot, clone, push and pull data volumes to FlockerHub.”  The name comes from the idea of helping make application data ‘fly,’ he added.

Bhatnagar explained how FlockerHub and Fli are designed to work together hand-in-globe. “Because these snapshots are incremental, each additional push or pull to FlockerHub sends only part of the data, which speeds up transfer times and reduces bandwidth consumption. Additionally, just like you can use Git to organize code into repos and branches, Fli provides the ability to organize data into volume sets and branches, making it simple to operate large-scale test fixture libraries,” he said.


In a recent blog post, ClusterHQ’s vice president of marketing Michael Ferranti, vice president of marketing at ClusterHQ, elaborated on the similarities the company sees between source code management and data management:  

Before Git, developers largely worked in isolation. You worked on your own version of code. Your teammates did too, whether they sat next to you or were half the planet away. Collaborating on complex code-related problems was difficult, unnatural, and at best a royal pain in the neck. Managing code versions in QA was manual and error prone. Continuous test and integration, let alone deployment, weren’t even fantasies.

Because testing was inherently hampered, you wrote very few test cases, so software was ill-tested before being pushed to production. And those production pushes were ever-traumatic. Code release cycles were measured in years or quarters. Rapid innovation – get a bug report or have a feature idea, quickly code it, test it, iterate, iterate, iterate – was impossible. Microservices? DevOps? Inconceivable.

Today, working with data suffers from many of the same problems. Data is hard to move around. Getting access to data often requires manual processes that take way too much time and as a result cost way too much money.

ClusterHQ engineers laid out some of the top use cases, where FlockerHub and Fli will help promote agile development, and keep stakeholders on the same page when it comes to accuracy of data:

Realistic test data for CI environments: The best tests include realistic test data. But getting data into continuous integration environments can be cumbersome. When combined with popular CI software like Jenkins, the data management capabilities provided by FlockerHub and Fli can accelerate incremental and parallel builds, speeding up testing dramatically.


On-demand staging for each commit: As developers work on feature branches, they often preview their code in a staging environment. FlockerHub volumes can be referenced directly in Docker Compose, allowing the Dockerized application to be spun up along with its data, providing a truly representative staging environment.


Commit and share error state for simplified debugging: Any developer who needs to work on an issue can pull the error state, along with the correct code version and Dockerized application environment, and have everything needed to isolate the bug.

Survey Finds Legacy Testing Stressed by DevOps and Microservices

If you are among the growing number of DevOps advocates that find older testing tools lacking, you are not alone.


In it’s first-ever Application Testing survey, ClusterHQ found many developers complaining that their legacy testing processes can’t keep pace with the needs of microservices-based apps, especially when it comes to speed and agility.


The survey revealed that an eye-popping 43 percent of app developers say they spend as much as 25 percent of their time debugging errors discovered in production – rather than being able to develop new features.


The survey identified three main challenges:  

  • An inability to fully recreate production environments in testing (33%)
  • Interdependence on external systems that makes integration testing difficult (27%)
  • Testing against unrealistic data before moving into production (26%)

Mark Davis, CEO of ClusterHQ, noted that “legacy software development practices, like relying on narrow subsets of synthetic data for testing, no longer cut it for teams focused on maximizing the amount of time they spend building features that users value. Forward looking software developer leaders understand that to deliver innovation to customers they must effectively manage the entire application lifecycle across a diverse range of infrastructures, a process that begins with identifying and eliminating bugs as early as possible so that teams can focus on adding end-user value.”