Vast Data Platform

VAST Data has developed a unique data platform designed to address the limitations of classic high performance computing (HPC) infrastructure in the AI era.

Their architecture is optimised for GPU-accelerated computing, combining the speed and scalability of HPC storage systems with the reliability of enterprise solutions.

VAST Data's platform is seeking to become the foundation for AI and deep learning at leading research institutions and GPU cloud providers.

Key Data Trends

Data is becoming more distributed geographically across sites and data centres. It's not just about data being on-prem or in the cloud anymore, but data being "anywhere".

The volume of unstructured data (files, images, video) is rapidly growing alongside structured data. Platforms need to bring together all data types.

There is a concept of the "future value of data" - organisations want to store data long-term for future use cases, not just immediate analytics.

Disruptive Technologies

• AI and deep learning are increasingly the primary applications driving data platform requirements. Platforms need to support the entire deep learning workflow.

• New hardware like storage-class memory and all-flash storage are enabling fundamental re-architectures of data platforms.

• Serverless computing, with triggers and functions, is being embedded into data platforms to enable developer agility.

Key Themes

• The emergence of a "new data platform" that unifies structured, unstructured, streaming data and AI/deep learning into one platform, going beyond previous big data and data warehouse architectures.

• Enabling computation and AI/ML to be pushed to wherever the data resides, rather than always moving data to the compute.

• The increasing importance of metadata management and catalogues to make sense of distributed data.

• Simplifying complex data infrastructures through unification of capabilities in the platform.

The journey

VAST Data has built a data platform designed to address the limitations of classic HPC infrastructure in the AI era.

Their journey began with the goal of solving the challenges associated with storing and processing vast amounts of data at scale, focusing on performance, scalability, and ease of use.

Solving the tiering problem

• Traditional storage systems often relied on tiering, where data would be moved between high-performance and low-cost storage based on age and access patterns.

• This led to bottlenecks in data analysis, as data constantly needed to be moved back and forth between tiers.

• VAST Data created an all-flash architecture to eliminate the need for tiering, providing high performance for all data, regardless of age or access frequency.

Disaggregated Shared Everything (DASE) Architecture

• VAST Data developed the DASE architecture to address the limitations of shared-nothing and shared-media architectures.

• DASE decouples the system's computational resources from its persistent data and system state.

• This allows for independent scaling of performance and capacity, as each component can be scaled separately.

• By eliminating the need for compute nodes to coordinate with each other, VAST clusters can scale linearly without the limitations of legacy architectures.

Enabling high-performance workloads

• VAST Data initially focused on the HPC market, where performance and scalability were critical.

• They worked with early adopters like NASA and NIH to validate their platform's performance and stability under the most demanding workloads.

• By solving the hard problems first, VAST Data established a strong foundation for their platform.

Expanding to enterprise and cloud

• After proving their platform's capabilities in HPC environments, VAST Data expanded their focus to enterprise and cloud markets.

• They added enterprise features such as snapshots, replication, encryption, and multi-tenancy to make their platform suitable for a broader range of use cases.

• VAST Data also introduced support for multiple protocols (NFS, SMB, S3) and application interfaces (CSI for Kubernetes) to integrate seamlessly with existing enterprise infrastructures.

Addressing the needs of AI and deep learning

• Recognising the growing importance of AI and deep learning workloads, VAST Data optimised their platform for these use cases.

• They partnered with NVIDIA to achieve "NVIDIA DGX SuperPOD" certification, leveraging GPUDirect Storage for high-bandwidth, low-latency data transfer between storage and GPUs.

• VAST Data also introduced support for a native table format (VAST Database) to accelerate data lake migrations and enable high-performance analytics on structured and semi-structured data.

Building a global namespace with VAST Data Platform

• To support the need for data access across geographies and clouds, VAST Data introduced the VAST Data Platform.

• The platform creates a global namespace (VAST Data Space) that spans edge, core, and cloud environments, allowing data to be accessed from anywhere without the need for manual data movement.

• This global namespace simplifies data management and enables efficient data sharing and collaboration across distributed teams.

Throughout their journey, VAST Data has focused on solving the challenges associated with storing and processing massive amounts of data at scale.

By developing a novel architecture, expanding to new markets, and continually optimising for emerging workloads like AI and deep learning, they have created a platform that addresses the limitations of traditional storage systems and enables organizations to unlock the full potential of their data.

Platform Details

The VAST Data Platform is built on a Disaggregated, Shared-Everything Architecture (DASE), which disaggregates the cluster's computational resources from its persistent data and system state.

This architecture allows for independent scaling of performance and capacity, overcoming the limitations of legacy shared-nothing and shared-media architectures. The platform consists of several key components:

VAST DataStore

An unstructured data repository that presents file, object, and table interfaces simultaneously, storing data as elements rather than files or objects.

It provides AI/HPC-class performance for all data, simplifying pipelines and offering archive economics for exabyte-scale volumes of data on AI-ready flash storage.

The core storage product provides enterprise NAS, file (NFS, SMB), and object storage (S3 compatible). VAST Data pioneered new ways of using NFS to run at local NVMe speeds and developed a driver to unlock the ability to use NFS at high speeds.

VAST DataBase

A fully ACID-compliant high-speed data lake that supports both transactional and analytical workloads at exabyte scale.

It eliminates the need for separate databases, data warehouses, and data lake platforms stitched together with complex ETL pipelines. The DataBase also manages the VAST Catalog, automatically recording metadata for every file and object ingested into the DataStore.

It aims to provide granular access to exabytes of data and quickly find specific data points.

The database functionality can be used to store custom metadata, such as the results of inferencing engines applied to image data. It is SQL-based and supports columnar lookups.

VAST DataSpace

Creates a global namespace over data centres, cloud, and edge, addressing the challenges of replication, remote data access, and consistency.

It introduces the concept of global lease management combined with intelligent data movement, ensuring consistency across multiple geographies without wasteful and expensive data movement.

VAST DataEngine

A distributed processing environment designed to power event-driven AI workflows. It natively integrates data processing and event notifications, enabling real-time evaluation or triggers and automating the critical task of AI discovery.

The Data Engine is an eventing engine that allows customers to run their own code and functions.

It enables triggers to be created by the file system, which can call customer-defined functions, such as running an inferencing model on an uploaded image and storing the output in the database.

VAST Data is not writing the code for the inferencing jobs but providing the infrastructure for customers to run their own functions.

Future Predictions

• VAST's platform is well-positioned to support the massive data and deep learning requirements driven by the current "AI wave".

• Selling data platforms will increasingly require appealing to developers and data scientists, not just IT infrastructure buyers.

• AI itself will increasingly make automated decisions about how to optimise data placement and computation.

• Hardware will continue to commoditise while value shifts to software and the data platform.

Summary of the VAST Data Platform

VAST Data was founded in 2015 and has experienced rapid growth and success.

The company raised a Series D round at a valuation of $9.1 billion over 2023 and has been cash flow positive for the past 8 quarters.

VAST Data has deployed and is storing 10 exabytes of data globally across its install base, with 60% of its business focused on HPC and AI workloads.

Founder's Vision

• In 2016, Renan, the founder of VAST Data, had a vision to build a data centre-scale computer that could be a "thinking machine."

• The roadmap involved developing a storage system, data management capabilities, and a transactional storage system.

Product Timeline

• The company launched its product in 2019, with the first revenue coming in 2020.

• VAST Data was the first enterprise storage system certified by NVIDIA to be SuperPOD certified.

• In the summer of 2022, VAST Data introduced new capabilities such as the VAST Database, VAST Data Space, and Data Engine.

Disaggregated Shared Everything (DASE) Architecture

• DASE is the core architecture that underpins VAST Data's capabilities.

• It separates the processing logic from the disk drives, allowing every CPU to access and feel like every disk in the cluster is locally attached.

• This architecture provides better scalability, eliminates east-west traffic, and improves reliability.

Comparison to Other Platforms

• VAST Data's platform aims to provide a comprehensive solution that combines storage, analytics, and eventing capabilities.

• It differs from platforms like Snowflake, which is a SaaS-based data warehouse focused on smaller datasets and fast queries.

• VAST Data's platform is designed to handle exabyte-scale data lakes while providing fast querying capabilities.

Target Market and Use Cases

• VAST Data is targeting cloud service providers, especially those focused on AI workloads, who need a comprehensive solution beyond just GPUs as a service.

• The platform is also relevant for enterprises looking for capabilities similar to those offered by public cloud providers in their own data centres.

• Use cases include image analysis, sentiment analysis, and object detection, where data can be stored in the file system, and metadata can be generated and stored in the database.

In summary, VAST Data has developed an innovative data platform that combines storage, analytics, and eventing capabilities, underpinned by its Disaggregated Shared Everything (DASE) architecture.

The platform aims to provide a comprehensive solution for managing and processing exabyte-scale data, targeting cloud service providers and enterprises focused on AI and HPC workloads.

VAST Data's approach is to build upon its storage foundation and extend its capabilities to compete with the offerings of public cloud providers, enabling customers to run advanced analytics and AI workflows on their data.