Building Analytics Apps for External Users? Here's How to Get it Right
June 01, 2022

David Wang

Analytics has come a very long way in recent years, with transformational developments widening its impact beyond internal stakeholders and into the hands of external users. This brings with it a range of challenges, and when building analytics applications for customers, for example, one of the first considerations will be the choice of database backend.

While the main options will probably include PostgreSQL, MySQL, or even extending a data warehouse beyond its core BI dashboards and reports, it's important to keep in mind that analytics for external users can be revenue-impacting. As a result, choosing the right tool for the job is essential if organizations are to deliver a high-quality user experience.

Analytics Performance

For many users, the most frustrating element of their analytics experience is performance, and in particular, the wait-state of queries in a processing queue. It's one thing to have an internal business analyst wait a few seconds or even several minutes for a report to load; it's entirely different when analytics functionality is being offered to external users whose tolerance of processing delays will be much lower.

This problem is generally caused by the amount of data to analyze with the processing power of the database and the number of users and API calls. Collectively, this determines how well the database can keep up with the application.

However, there are a variety of approaches to building an interactive data experience with any generic OLAP database when there's a lot of data. The issue? These come at a price. For instance, precomputing all the queries makes the architecture very expensive and rigid, while aggregating the data first minimizes the insights, and limiting the data analyzed to recent events doesn't give users the complete picture. All of these ways involve making compromises.

There is, however, a no-compromise approach that can deliver an optimized architecture and data format built for interactivity at scale. This comes in the form of Apache Druid — a high-performance, real-time analytics database to power analytics applications for any number of users.

Druid employs a uniquely distributed and elastic architecture that prefetches data from a shared data layer into a near-infinite cluster of data servers. This architecture enables faster performance than a decoupled query engine like a cloud data warehouse because there's no data to move and more scalability than a scale-up database like PostgreSQL/MySQL.

Furthermore, Druid provides automatic, multi-level indexing that is built into the data format to drive more queries per core. This goes beyond the typical OLAP columnar format with the addition of a global index, data dictionary, and bitmap index. In doing so, it maximizes CPU cycles for faster crunching.

High Availability Should be a High Priority

To illustrate the value of these capabilities, consider this scenario: if a dev team is building a backend for internal reporting, does it really matter if it goes down for a few minutes or even longer?

For most, the answer is probably not and explains why there's always been tolerance for unplanned downtime and maintenance windows in classical OLAP databases and data warehouses.

But what if the dev team then needs to build an external analytics application that customers will use?

Any outages here can impact revenue, with a serious knock-on effect on issues as varied as team resources to customer satisfaction. As a result, resilience — both high availability and data durability — must be a priority when choosing a database for external analytics applications.

Delivering resilience means posing some important design criteria questions — can you protect from a node or a cluster-wide failure?

How bad would it be to lose data?

What work is involved to protect your app and your data?

The legacy approach to achieving greater resiliency is to replicate nodes and to remember to take backups. But when dev teams are building apps for customers, the sensitivity to data loss is much higher, and as a result, occasional backups aren't fit for purpose.

In contrast, Druid's core architecture is designed to withstand downtime without losing data (even recent events) by implementing high availability (HA) and durability based on automatic, multi-level replication with shared data in S3/object storage. This not only enables the HA properties dev teams expect but also a form of continuous backup that automatically protects and restores the latest state of the database even if an entire cluster is lost.

Cost-Performance Benefits

Building a database that delivers high concurrency means striking the right balance between CPU usage, scalability, and cost. Historically, addressing concurrency was a matter of allocating more hardware to the challenge, and while adding more CPUs certainly allows organizations to run more queries, it can easily become very expensive.

In contrast, databases like Apache Druid are built with optimized storage and query engine that drives down CPU usage. By only reading the data it needs to, the infrastructure can serve more queries in the same timespan.

This is also an important consideration when building external applications that will deliver the performance and resilience required both today and in the future. For those organizations focused on customer retention, being able to scale their infrastructure is key to remaining competitive.

David Wang is VP of Product Marketing at Imply
Share this

Industry News

July 25, 2024

Backslash Security introduced its Fix Simulation and AI-powered Attack Path Remediation capabilities.

July 25, 2024

Check Point® Software Technologies Ltd. announced the appointment of Nadav Zafrir as Check Point Chief Executive Officer.

July 25, 2024

Sonatype announced that Sonatype SBOM Manager, its Enterprise-Class Software Bill of Materials (SBOM) solution, and its artifact repository manager, Nexus Repository, are now available in AWS Marketplace, a digital catalog with thousands of software listings from independent software vendors that make it easy to find, test, buy, and deploy software that runs on Amazon Web Services (AWS).

July 24, 2024

Broadcom unveiled the latest updates to VMware Cloud Foundation (VCF), the company’s flagship private cloud platform.

July 24, 2024

CAST launched CAST SBOM Manager, a new freemium product designed for product owners, release managers, and compliance specialists.

July 24, 2024

Zesty announced the launch of its Insights and Automation Platform.

July 23, 2024

Progress announced the availability of Progress® MarkLogic® FastTrack™, a UI toolkit for building data- and search-driven applications to visually explore complex connected data stored in Progress® MarkLogic® platform.

July 23, 2024

Snowflake will host the Llama 3.1 collection of multilingual open source large language models (LLMs) in Snowflake Cortex AI for enterprises to easily harness and build powerful AI applications at scale.

July 23, 2024

Secure Code Warrior announced the availability of SCW Trust Agent – a solution that assesses the specific security competencies of developers for every code commit.

July 23, 2024

GFT launched AI Impact, a new solution that leverages artificial intelligence to eliminate technical debt, increase developer efficiency and automate critical software development processes.

July 23, 2024

Code Metal announced a $13M seed, led by Shield Capital.

July 22, 2024

Atlassian Corporation has achieved Federal Risk and Authorization Management Program (FedRAMP) “In Process” status and is now listed on the FedRAMP marketplace.

July 18, 2024

Mission Cloud announced the launch of Mission Cloud Engagements - DevOps, a platform designed to transform how businesses manage and execute their AWS DevOps projects.

July 18, 2024

Accelario announces the release of its free TDM solution, including database virtualization and data anonymization.