DASE (Disaggregated and Shared Everything)

The DASE architecture is designed to scale to tens of thousands of CPUs and exabytes of flash storage, making it suitable for massive-scale data processing and storage requirements.

Key Principles

1. Simplify every aspect of data and infrastructure management and processing.

2. Embrace standards to allow customers to bring their data and applications without refactoring.

3. Give customers ownership of their data and the ability to deploy it on their chosen infrastructure.

Architecture Overview

VAST Data has created a new distributed systems architecture that decouples the CPUs (which run the system's logic) from the underlying storage devices.

Decoupling CPUs from storage devices allows for independent scaling of compute and storage resources. This means that you can add more CPUs without necessarily adding more storage, or vice versa, depending on the workload requirements.

It eliminates the need to maintain a fixed ratio of compute to storage capacity, providing greater flexibility in system design and resource allocation.

Decoupling also enables the use of heterogeneous hardware components. You can mix and match different generations of CPUs and storage devices within the same cluster, allowing for incremental upgrades and cost optimisation.

This is done by interconnecting the CPUs and storage devices using a low-latency storage fabric. As a result, any CPU can access any storage device in the cluster, creating a global shared volume.

What is Low Latency Storage Fabric?

The storage fabric is a high-speed, low-latency network that interconnects the CPUs and storage devices.

The specific type of fabric is likely a high-performance interconnect such as InfiniBand or Ethernet with RDMA (Remote Direct Memory Access).

The low-latency nature of the storage fabric is crucial for enabling the CPUs to access storage devices efficiently, as if they were directly attached. This is essential for achieving high parallelism and performance in the DASE architecture.

DBoxes

VAST’s shared everything model allows any VAST server or CNode to have direct access to all the data, metadata, and system state directly.

In a DASE cluster, the system state is stored on NVMe SSDs in highly available NVMe enclosures known as DBoxes.

By eliminating the need for CNodes to communicate with each other VAST clusters can independently scale performance and capacity well beyond the limitations of legacy architectures.

Without the need to accommodate mechanical media, VAST implemented a new class of data protection algorithms that deliver superior resilience with radically lower overhead and fast disk rebuild.

The result is a data platform that scales to exabytes of capacity, and delivers TB/s of performance, at a cost point that makes tiering and caching obsolete.

Advantages

1. High Parallelism: Since all CPUs can access all data, the system can process data in parallel, making it suitable for high-performance computing, distributed databases, and AI applications.

2. Resilience: The architecture allows for the loss of CPUs or storage devices without impacting the system's availability. VAST Data clusters can achieve nearly six nines (99.9999%) of availability.

3. Scalability: The system can scale to tens of thousands of CPUs and exabytes of low-cost flash storage.

Data Management

VAST Data uses a persistent write buffer, implemented in storage class memory (SCM), as a landing zone for incoming data before it is moved to low-cost flash storage.

The write buffer allows for data classification based on life expectancy, ensuring that data is placed on the appropriate storage tier (short-term or long-term) to optimise SSD endurance and performance.

As data flows through the write buffer, the system creates consistency points in a log-structured manner. This enables efficient snapshots, replication, and data cataloguing without impacting performance.

The write buffer also serves as a staging area for data reduction techniques, such as similarity-based compression, which is applied globally across the entire dataset before data is written to flash storage.

VAST Data uses persistent write buffers (implemented in storage class memory) as a landing zone for incoming data before it is moved to low-cost flash storage. This allows the system to:

1. Classify data according to its life expectancy, preserving SSD endurance.

2. Create consistency points in a log-structured manner.

3. Enable additional services like snapshots, replication, and data cataloguing without impacting performance.

Cost Efficiency

VAST Data employs three approaches to reduce infrastructure costs:

1. Using low-cost, commodity flash storage.

2. Implementing locally decodable codes (LDC) for efficient data protection with minimal overhead (as low as 3%).

3. Utilizing similarity-based data reduction, which identifies similar data patterns across the entire cluster and compresses them, resulting in an average 4:1 data reduction ratio.

Deployment and Expansion

VAST Data's architecture allows for the creation of composable infrastructure, where resources can be allocated to different workloads or tenants without impacting others.

The system also supports the use of multiple generations of hardware (both storage and compute) within the same cluster, making expansions and upgrades seamless.

VAST Data Space

VAST Data Space is a global namespace that allows customers to deploy VAST clusters across edge locations, core data centres, and popular cloud platforms like AWS, Microsoft Azure, and Google Cloud Platform.

This creates a unified data plane that spans all computing environments.

In summary, VAST Data's DASE architecture is designed to provide unparalleled parallelism, efficiency, and resilience at any scale.

By decoupling compute and storage resources, employing innovative data management techniques, and leveraging cost-saving measures, VAST Data enables organisations to build a foundation for next-generation computing that can handle the demands of AI, big data, and high-performance computing workloads.