The Importance of Data in the DevOps Process
June 11, 2020

Razi Shoshani
SQream

Organizations and their data are continually growing. Over the years data technology has grown along with them, moving from a focus on centrally managed databases and data warehouses, to multiple fit-for-purpose systems that share data and are not managed in a unified manner.

With this, new challenges have arisen. Data flow blind spots, changes to data structure and data pollution are par for the course. Current data stores are varied and far from being uniform and consistent. Applications and solutions must process interface, integrate and process data from these disparate data stores in multiple formats including text, binary, XML and JSON to name a few.

So if in the past developers knew what was stored and what the data looked like, today they are challenged with data that is more complex, stored in silos, often causing long gaps between application development specification creation and the deployment of the integrated solution.

As organizations have struggled to meet these issues head-on, they've experienced increased strain on manpower and resources, bringing with it rising costs, and making it even more challenging to successfully integrate and manage the organization's growing data stores.

Following are some questions and answers focused on actions organizations can take to ease these growing pains and ensure clean, fast data processes.

What is one method used by DevOps to handle the challenges caused by these disparate data stores, which is growing exponentially and in varied formats?

Data-driven software development puts data in the center of the development process for applications that will be developed. It involves taking data assets originating in a variety of data sources and linking between these diverse assets into one data repository of data assets. This will enable developers to create a streamlined integration from their applications to their data stores, to search and query the data without the need for multiple data channels.

What should you know about your data when building your application specification?

It is critical that application developers understand their data as early on as possible in the specification development process. They should understand the structure of all data stores that they will need to access, how the various data stores can be accessed and joined as needed to enable querying and updating of data from the applications. They should also understand any regulations or other legal constraints of accessing the data as well as company specific data governance guidelines.

What other information should DevOps ensure they have about the data?

Developers should ensure they have basic information about their data before they begin to develop their applications. They should know origin of the data and where it currently resides, how clean the data is, who owns the data and what the value of the data is from a risk point of view. In addition, they should understand the current uses of the data, what decisions are made based on the data and when those decisions are made.

What can we do with data that is hard to manage?

To make your data easier to integrate, manage and analyze, you can implement several best practices:

■ Clean it, to reduce errors.

■ Make a practice of using standardized processes throughout the organization when handling data.

■ Check the data you upload for accuracy.

■ Scrub the data before uploading in order to remove duplications.

■ Ensure your team is all on-board and following the new data handling processes going forward.

What should you do to ensure that your data can be accessed and integrated smoothly?

There are a number of things you can do to ensure agile response from your database. Here are some of them:

■ Ensuring you are using hardware with specifications capable of supporting the demands of your system.

■ Making sure your hardware is set up according to best practices to meet your system requirements.

■ Utilizing connectors to enable the ability to integrate between your coding system using multiple coding languages.

■ Exercising database version control, so that all changes to the database are made with a single source of truth and ensuring error free updates.

How can developers provide quicker response times to change management requests?

Developers can have a major impact on the way an organization does business whether they know it or not. Even once the application is up and running they are inundated with a non-stop flow of change requests, bug fixed and other new requirements. Often these changes require a new query or report, or other data heavy task. To address these change requests as quickly as possible, developers should be able to access and query existing data ad-hoc or with minimal setup and query development time.

To streamline both the development process and the change requirement needed after the application has gone live, developers need access to data, and the ability to rapidly query that data. This is especially challenging as data is growing exponentially.

What can we do to make it easier to integrate and analyze large amounts of data?

Data can be stored on data acceleration platforms that utilize the power of a GPU database to more rapidly access and analyze massive amounts of data — multi-billion row datasets in seconds — from Machine Learning, to Geospatial Analysis, to complex advanced queries that take days to run in standard conditions. The acceleration platform brings power to the development process, significantly cutting time, reducing cost and risk.

Razi Shoshani is Co-Founder and CTO of SQream
Share this

Industry News

July 13, 2020

Docker announced a collaboration with Amazon Web Services (AWS) to simplify the lives of developers by allowing them to focus on application development, streamlining the process of deploying and managing containers in AWS from their local development environment.

July 13, 2020

Perforce Software announced the release of a combined JRebel and XRebel plugin for the Eclipse IDE.

July 13, 2020

Spectro Cloud announced that its first product—Spectro Cloud—is now generally available.

July 09, 2020

ShiftLeft released a new version of NextGen Static Analysis (NG SAST), including new workflows, purpose-built for developers that significantly improve security, while enhancing productivity.

July 09, 2020

RunSafe Security announced a partnership with JFrog that will enable RunSafe to supercharge binary protections via a simple plugin that JFrog users can deploy within their Artifactory repositories and instantly protect binaries and containers.

July 09, 2020

LeanIX closed $80 million in Series D funding led by new investor Goldman Sachs Growth.

July 08, 2020

Afi.ai introduced Afi Data Platform, a cloud-based replication and resiliency service that helps to monitor, predict downtime and recover K8s applications.

July 08, 2020

D2iQ announced the release of Conductor, a new interactive learning platform that enables enterprises to access hands-on cloud native courses and training.

July 08, 2020

SUSE entered into a definitive agreement to acquire Rancher Labs.

July 07, 2020

Micro Focus announced AI-powered enhancements to the intelligent testing capabilities of the UFT Family, a unified set of solutions designed to reduce the overall complexity of automating the functional testing processes.

July 07, 2020

Push Technology announced the launch of a new Service API capability for Diffusion Cloud, Push’s Real-Time API Management Cloud Platform.

July 07, 2020

Lightrun exited stealth and announced $4M in seed funding for the first complete continuous debugging and observability platform for production applications.

July 01, 2020

JFrog announced the launch of ChartCenter, a free, security-focused central repository of Helm charts for the community.

July 01, 2020

Kong announced a significant upgrade to open source Kuma, Kuma 0.6, available today.

July 01, 2020

Compuware Corporation, a BMC company, announced new capabilities that further automate and integrate test data and test case execution, empowering IT teams to achieve high-performance application development quality, velocity and efficiency.