Skip to main content

Amazon FSx

1. Introduction​

Amazon FSx offers a range of file system solutions designed to meet diverse workload requirements, from high-performance computing to enterprise file sharing. Much like how RDS simplifies database management on AWS, FSx abstracts the underlying complexities of file system deployment, enabling you to focus on your applications without worrying about managing infrastructure. At its core, FSx allows organizations to deploy file systems such as Windows File Server, Lustre, NetApp ONTAP, and OpenZFS, each tailored to specific use cases, performance needs, and compatibility requirements. This chapter provides an in‐depth look at the different types of file systems available on FSx, their use cases, performance characteristics, and operational management features.

2. Overview of Amazon FSx​

Amazon FSx provides fully managed third-party file systems that cater to a broad spectrum of workloads. The service currently supports four primary file systems:

  • FSx for Windows File Server: A managed Windows-based file share offering native support for Windows protocols and integration with Active Directory.
  • FSx for Lustre: A high-performance, distributed file system optimized for machine learning, high-performance computing (HPC), and data processing workloads.
  • FSx for NetApp ONTAP: A robust solution for migrating on-premises NetApp ONTAP or NAS environments to AWS, with extensive protocol support and advanced storage features.
  • FSx for OpenZFS: A solution for those seeking to migrate workloads from existing ZFS environments to AWS, emphasizing performance and efficient data management.

Each of these file systems is designed to provide a fully managed, scalable, and cost-effective solution while leveraging the reliability and global infrastructure of AWS.

3. Amazon FSx for Windows File Server​

Amazon FSx for Windows File Server is engineered to offer a fully managed, native Windows file share with the following key features:

  • Protocol and File System Support:
    Supports the SMB protocol and Windows NTFS, enabling seamless file sharing and data management for Windows-based applications.

  • Active Directory Integration and Security:
    Native integration with Microsoft Active Directory ensures secure authentication and authorization. It supports Windows ACLs (access control lists) and user quotas for granular access control.

  • Cross-Platform Compatibility:
    Although optimized for Windows, the file system can also be mounted on Linux EC2 instances (using the Linux SMB client), providing flexibility in mixed-OS environments.

  • On-Premises Integration:
    For organizations with existing on-premises Windows file servers, the Microsoft Distributed File System (DFS) can be used to integrate FSx for Windows File Server into the on-premises infrastructure.

  • Performance and Scalability:
    Designed for high performance, it scales up to tens of gigabytes per second, supports millions of IOPS, and can handle hundreds of petabytes of data.

  • Storage Options:
    Offers SSD storage for workloads requiring very low latency (e.g., databases, media processing, analytics) and HDD storage for cost-sensitive, broad-spectrum workloads such as home directories.

    You can change your file system storage type from HDD to SSD using the AWS Management Console and AWS CLI. You can't change your file system storage type from SSD to HDD. If you want to change a file system's storage type from SSD to HDD, you will need to restore a backup of the file system to a new file system that you configure to use HDD storage.

  • High Availability and Disaster Recovery:
    With the option to configure the file system in Multi-AZ mode, FSx for Windows File Server can provide high availability through synchronous replication across two availability zones. Daily backups to Amazon S3 further enhance disaster recovery capabilities.

4. Amazon FSx for Lustre​

Lustre is an open-source, parallel distributed file system that was specifically engineered for large-scale cluster computing. It achieves extreme scalability and high performance by spreading data across multiple servers and disks, which is why it’s the file system of choice for many of the world’s fastest supercomputers.

Amazon FSx for Lustre is a fully managed service by AWS that delivers a high‐performance, scalable file system built on the open‐source Lustre file system. It’s designed to meet the needs of compute‐intensive workloads—such as high-performance computing, machine learning, big data analytics, and media processing—by providing sub-millisecond latencies, high throughput, and millions of IOPS. In addition, it integrates with Amazon S3, allowing you to seamlessly process and update cloud data stored there.

Together, Amazon FSx for Lustre and Lustre enable organizations to harness the power of high-performance storage without the overhead of managing complex infrastructure.

  • Purpose and Use Cases:
    Based on the Lustre file system, which is derived from Linux and cluster computing principles, FSx for Lustre is ideal for HPC workloads, machine learning, video processing, financial modeling, and electronic design automation.

  • Performance Characteristics:
    Provides massive throughput (up to hundreds of gigabytes per second), millions of IOPS, and sub-millisecond latency, making it suitable for data-intensive applications.

  • Storage Options:

    • SSD: Optimized for low latency and IOPS-intensive, random file operations.
    • HDD: Suitable for throughput-intensive workloads where large, sequential file operations are predominant.

  • Integration with Amazon S3:
    Seamlessly integrates with Amazon S3, enabling applications to treat S3 buckets as file systems. FSx for Lustre allows for writing computation outputs back to S3, and it features data lazy loading—loading data only when a client requests it—reducing both cost and latency.

  • Deployment Modes:

    • Scratch File System:
      Designed for temporary storage without data replication. Optimized for short-term processing where performance is paramount, though data is lost if the underlying server fails.
    • Persistent File System:
      Intended for long-term storage with data replication within the same availability zone. In the event of a server failure, data is transparently restored within minutes.

5. Amazon FSx for NetApp ONTAP​

Amazon FSx for NetApp ONTAP is a fully managed AWS service that brings the full power of NetApp’s renowned ONTAP storage operating system to the cloud. It provides enterprise-grade file storage that’s highly scalable, secure, and feature-rich—all without the need to manage the underlying storage hardware or software. With FSx for ONTAP, you gain access to robust data management capabilities like efficient data tiering, built-in snapshots, replication (SnapMirror), and data deduplication and compression, along with multi-protocol support (NFS, SMB, iSCSI, and NVMe over TCP) that make it easier to migrate, protect, and run your on-premises workloads in AWS.

NetApp ONTAP itself is NetApp’s proprietary operating system that powers its storage systems—whether deployed on-premises (in FAS, AFF, or ASA systems), as a software-defined solution (ONTAP Select), or in the cloud (Cloud Volumes ONTAP). ONTAP offers a unified storage platform with advanced features such as snapshots, replication, high availability, and automated tiering. It’s designed to optimize performance and efficiency while protecting data across various workloads and environments.

FSx for NetApp ONTAP offers comprehensive features for enterprise storage environments:

  • Multi-Protocol Support:
    Compatible with NFS, SMB, and iSCSI, allowing it to support diverse operating systems including Linux, Windows, and macOS.

  • Migration and Compatibility:
    Ideal for migrating existing on-premises ONTAP or NAS workloads to AWS, ensuring continuity and broad compatibility with services such as VMware Cloud on AWS, WorkSpaces, AppStream, EC2, ECS, and EKS.

  • Advanced Storage Features:
    Supports auto-scaling, allowing the file system to grow or shrink automatically based on demand. Additional features include snapshots, replication, data compression, and data de-duplication to optimize storage efficiency.

  • Instantaneous Cloning:
    Enables point-in-time instantaneous cloning, which is invaluable for testing new workloads and creating staging environments quickly.

6. Amazon FSx for OpenZFS​

OpenZFS is the open-source evolution of the ZFS file system originally developed by Sun Microsystems for Solaris. It combines a robust file system with an integrated volume manager, offering advanced features such as data integrity verification, self-healing through redundant data storage (using mirroring or RAID-Z), efficient data compression, deduplication, and instant snapshots. OpenZFS is used across multiple platforms—including Linux, FreeBSD, macOS, and even Windows via community projects—ensuring reliable, scalable, and feature-rich storage solutions for both enterprise and personal use.

Amazon FSx for OpenZFS is a fully managed file storage service by AWS that leverages the open-source OpenZFS file system. It enables you to migrate and run Linux-based file servers on AWS without changing your application code or data management practices. With FSx for OpenZFS, you benefit from features such as instant snapshots, data cloning, and on-demand replication, all built on the proven capabilities of OpenZFS. The service delivers high performance—with millions of IOPS, low-latency access, and scalable throughput—making it well-suited for a wide range of applications from development and testing to mission-critical, data-intensive workloads.

  • Protocol Support and Compatibility:
    Supports the NFS protocol across multiple versions, and is compatible with Linux, macOS, and Windows environments.

  • Performance and Efficiency:
    Capable of scaling up to 1 million IOPS with latency of less than 0.5 milliseconds. It offers high performance along with support for snapshots and data compression.

  • Cost Considerations and Features:
    While providing efficient and low-cost storage, FSx for OpenZFS does not support data de-duplication. However, similar to NetApp ONTAP, it offers point-in-time instantaneous cloning, facilitating rapid testing and deployment of new applications.

7. Operational Management for FSx for Windows File Server​

7.1. Resizing Volumes with AWS DataSync​

Resizing FSx volumes requires careful planning because a file system’s capacity can only be increased directly. To decrease the capacity, the recommended approach is:

  • Migration via DataSync:
    Create a new FSx file system with the desired smaller capacity. Use AWS DataSync to continuously synchronize data from the original file system to the new one. This method ensures that data transfer occurs in the background, allowing for near-zero downtime.

  • Capacity Consideration:
    Note that when restoring from a backup, the new volume must be at least the same size as the original. Therefore, migrating using DataSync is the optimal strategy when a reduction in capacity is needed.

7.2. Backup, Restore, and Disaster Recovery​

Disaster recovery and data protection are critical components of FSx operational management:

  • Automated Backups:
    For example, FSx for Windows File Server is automatically backed up daily to Amazon S3, ensuring that data is protected and can be restored if necessary.

  • Restore Procedures:
    In scenarios where a file system must be migrated (such as moving from a single-AZ to a multi-AZ deployment), backups can be used to restore the file system into a new configuration. However, restoring from backup to change capacity has limitations, reinforcing the need for migration strategies like DataSync.

  • Replication and High Availability:
    Single-AZ deployments provide in-AZ data replication, whereas Multi-AZ configurations replicate data synchronously between two availability zones, providing robust disaster recovery and automatic failover capabilities.

7.3. Single-AZ vs. Multi-AZ Tradeoffs​

When designing a solution with FSx for Windows File Server, you must weigh the tradeoffs between Single-AZ and Multi-AZ deployments:

  • Single-AZ Deployments:
    • Data replication is confined within a single availability zone.
    • Options include Single-AZ 1 (SSD-based) and Single-AZ 2 (offering both SSD and HDD storage).
    • This configuration might be suitable for less critical workloads or where cost considerations outweigh the need for high availability.
  • Multi-AZ Deployments:
    • Provides synchronous replication between two availability zones.
    • Automatically enables failover in the event of a file system failure, ensuring continuous availability.
    • This configuration is generally recommended for critical workloads where downtime is unacceptable.

7.4. Migrating Between Single-AZ and Multi-AZ Deployments​

Migrating from a single-AZ deployment to a multi-AZ deployment can be approached in two ways:

  • DataSync-Based Migration:
    Create a new multi-AZ FSx for Windows File Server and use AWS DataSync to continuously replicate data from the single-AZ system to the new multi-AZ system. This method allows the original system to remain operational while the migration occurs in the background, minimizing downtime.

  • Backup and Restore:
    Alternatively, you can perform a backup of the single-AZ file system, shut it down, and then restore it into a multi-AZ configuration. This method results in some downtime but may be quicker for certain scenarios. The choice between these approaches depends on the availability requirements and operational constraints of your environment.

8. Conclusion​

In this chapter, we explored each FSx offering in detail, examined their performance characteristics, and reviewed critical operational management practices—including volume resizing, backup and restore strategies, and the migration between single-AZ and multi-AZ deployments. These insights are essential for designing robust, high-performance file storage solutions on AWS and for making informed decisions on the appropriate FSx service based on workload requirements and operational objectives.