What is a Cluster?
A sharded cluster is a way to scale horizontally by distributing data across multiple replica sets. When a read or write operation is performed on a collection, the client requests the router (mongos) which will validate which shard the data is stored in via the configuration server.
A sharded cluster consists of shards, query routers, and config servers. Shards store data subsets, mongos provide an interface between clients and shards, and config servers keep configuration settings for MongoDB.
MongoDB supports high availability through Replica Sets, but that’s not all. To ensure your data is always available, you need to arm your operational teams with monitoring, alerting, security, anomaly or failure detection, automatic recovery/failover, backup management, and more.
Shard servers: Each shard is responsible for some indexed value in your database. It has a unique name that is used by the query router (mongos) to find its data. Each shard is also part of a replica set.
Config server: The config server stores metadata for the shards and communicates with mongos to determine which shards to query for data. It is recommended that you have at least three shards in your cluster.
Select a name for your database and specify the storage type for data and backups. Database names cannot exceed 63 characters and must contain letters, numbers, periods, colons, dashes, and underscores. You can also assign a project to the database. This enables you to group your database deployments according to your organizational needs.
A MongoDB cluster is a collection of database hosts you configure for replication. A primary host serves read and write requests and asynchronously replicates changes to secondary hosts in the cluster. You can choose a configuration type and host class when you create the cluster. You can also specify the number of hosts for each of these.
Scalability is the ability of a system to maintain and increase its level of performance under increased operational demands. It is a fundamental aspect of data management, and is especially important for projects that involve large amounts of data.
Horizontal scalability is achieved through sharding, which distributes data across several machines. This enables high throughput operations with large datasets. It is also possible to scale vertically by increasing the capacity of individual servers. This is often done by upgrading hardware resources such as CPU, RAM, and storage capabilities. This approach is more expensive than horizontal scalability, but it allows for greater flexibility and faster response times.
If you have a lot of read traffic and need to scale out your data, horizontal sharding is the way to go. Sharding helps distribute your data across a much larger number of physical servers than is possible with a single database server. Moreover, a well-thought out shard key combined with tagged sharding can significantly improve your read performance by enabling concurrent querying among individual shards.
If a MongoDB primary node fails, the balancer automatically elects another node to take over those roles and begins replicating changes to other cluster members, ensuring consistent data. This process, known as auto-splitting, ensures that a read operation never encounters a failed node and can complete without any delay.
In addition to shards, a sharded cluster also contains query routers (the so-called mongos) and config servers that store configuration settings and metadata about the cluster. Mongos act as an interface between clients and shards, helping them find the right ones for a targeted operation.
Fault Tolerance in MongoDB provides an extra layer of security to ensure that the system can continue operating even when a single host is down. The system will automatically elect a new primary from the remaining hosts if quorum is not met. This process takes up to 12 seconds and depends on factors such as network latency and the replication configuration settings (notably, retryWrites).
Replication in MongoDB is a mechanism that ensures data consistency and handles failover in distributed environments. It works by sending writes to the primary node and then to secondary nodes, which act as read replicas for applications.
For application scalability and performance, a typical architecture for a replica set is 3 members. Managed Service for MongoDB supports this configuration in the default settings. It also supports additional options such as the ability to retry writes (1 time only) and a write concern majority journal default. This option is useful for applications with a high number of writes.