How does sharding work in mongodb

The trade off is increased complexity in infrastructure and maintenance for the deployment. MongoDB supports horizontal scaling through sharding. A MongoDB sharded cluster consists of the following components:. MongoDB shards data at the collection level, distributing the collection data across the shards in the cluster.

MongoDB uses the shard key to distribute the collection's documents across shards. The shard key consists of a field or multiple fields in the documents.

You select the shard key when sharding a collection. To shard a populated collection, the collection must have an index that starts with the shard key. When sharding an empty collection, MongoDB creates the supporting index if the collection does not already have an appropriate index for the specified shard key. See Shard Key Indexes. The choice of shard key affects the performance, efficiency, and scalability of a sharded cluster. A cluster with the best possible hardware and infrastructure can be bottlenecked by the choice of shard key.

The choice of shard key and its backing index can also affect the sharding strategy that your cluster can use. Choose a Shard Key. MongoDB partitions sharded data into chunks. Each chunk has an inclusive lower and exclusive upper range based on the shard key. In an attempt to achieve an even distribution of chunks across all shards in the cluster, a balancer runs in the background to migrate chunks across the shards.

MongoDB distributes the read and write workload across the shards in the sharded cluster , allowing each shard to process a subset of cluster operations. Both read and write workloads can be scaled horizontally across the cluster by adding more shards. For queries that include the shard key or the prefix of a compound shard key, mongos can target the query at a specific shard or set of shards. These targeted operations are generally more efficient than broadcasting to every shard in the cluster.

Starting in MongoDB 4. Each shard will contain a subset of sharded data. Mongos: Mongos in sharding will act as a query router; it will be used to provide the interface between sharded cluster and client applications. Mongos is also known as a query router. Mongos is used to process the operations with shards and return results to the client. We have implemented more than one mongos instance to divide the load or client request.

Config Servers: Config server is used to store the metadata of the cluster server. This metadata contains information of cluster data set mapping. Query router or mongos is used this metadata information to perform operations on specific shards.

We can implement three config server sharded clusters in a production environment. In the above example, two app servers are used, connected to the shard by using a mongos instance. MongoDB is also used the shard key to distribute the collected data across the shards which we have used in sharding.

The shard key in MongoDB consists of a field consisting of every document in a target collection. To create the query router, we provide the log location, IP, and port of the server instance. Further, we need to define the config server to which this query router belongs. It also includes the network settings to the server instance. Finally, we have set replSetName allowing the data to be replicated.

Initiate the shard server. The replication with the default configuration is enabled using the initiate function. Then check the status of the initialization with the status function. Add the Shard to the clustet. Using the addShard command, we provide the replSetName with the IP address and the port of the shard instance.

Create the database. This database will be used in the sharding operation. Here the necessary steps to enable sharding:. Check sharding status. Using the status command, we check if the sharding is enabled to the database. The data in the collection will be sharded using the collection and the shard key.

Create the index and add a record. Create the index with personid as the field in descending order. The first step is to make sure the index personid is hashed. If not, this will result in an error, and sharding will fail. Verify if sharding is working as intended.

Use the getShardDistribution command to verify the status of the sharding operation. The above output describes that the personscollection is sharded in the ShardRepSet on the Shard server It consists of a single document in a single chunk.

The available document is the single record we entered into the collection. MongoDB sharding is a method to manage large data sets efficiently by distributing the workload across many servers without having any adverse effects on the overall performance of the database. Also, sharding provides the ability to efficiently scale the cluster for future requirements without a complex restructuring of the underlying hardware infrastructure.

This e-book is a general overview of MongoDB, providing a basic understanding of the database. These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Picking a shard key, that groups the documents together will make most of the queries go to a specific Shard. This can avoid scatter gather queries. One possible example might be a Geo application for the UK, where the first part of the key includes the postcode and the second is the address.

Due to the first part of the shard key being the postcode, all documents for that particular sort key will end up on the same Shard, meaning all queries for a specific postcode will be routed to a single Shard. The UK postcode works as it has a lot of possible values due to the resolution of postcodes in the UK.

This means there will only be a limited amount of documents in each chunk for a specific postcode. However, if we were to do this for a US postcode we might find that each postcode includes a lot of addresses causing the chunks to be hard to split into new ranges. The effect is that MongoDB is less able to spread out the documents and in the end this impacts performance. Depending on your Shard key the routing will work differently. This is important to keep in mind as it will impact performance.

We want to ensure we meet two specific goals. The first one is to write to multiple recipients on separate shards thus leveraging the write scalability. How does one go about getting the correct shard key?

The first part delivers the message to all its recipients. The Math. What if we need to lookup documents by multiple different identities like a user name, or an email address?