MongoDB Replica Set - Why we choose to do it

We had been running Atatus in Google Cloud from the beginning till Feb 2015. In Feb, we move to our own controlled cloud, where we control everything from database to servers.

With Google Cloud, we benefited from certain advantages and one was, we never had to worry about securing our data. Google Cloud always provides high availability and keeps a backup of your data and you get your peace of mind. With our own cloud, we had great control of what and how we do it. With great power comes great responsibility, so here, we had to worry about high availability and data backup.

What we had been doing

Initial days, we were running single primary MongoDB server. We were also backing up our data every day and pushing it to AWS S3. This was sufficient on a surface level, where we were contend that we were having a backup. This is certainly a good step towards securing your data, but this alone is not sufficient.

If our single primary MongoDB server has a downtime or temporarily becomes unavailable, even our user login would not work. Good that, that still hasn’t happened ;), and we want to keep it that way! The idea of setting the mongodb replica set has been ensuring high availability of data and create a robust failover system. This is important in any production environment where a database going down would have a negative impact on our business.

What is MongoDB Replica Set

MongoDB handles replication through an implementation called replication sets. Replication sets in their basic form are somewhat similar to nodes in a master-slave configuration. In replica set following types of server exist. Out of all, one server is always primary.

Primary: The primary member is the default access point for transactions with the replication set. It is the only member that can accept write operations

Secondary: It stores the data and these servers stays in sync with primary. If primary server goes down then these servers are for failover to the primary. We are using one of the secondary member for backing up to AWS S3. To make sure your backup MongoDB node does not become master, set its priority parameter to 0.

Arbiter: It stores the no data, it meant only for participate in election during failover.

Each replication set can have only one primary member at a time. This is because replication happens by copying the primary’s “oplog” (operations log) and repeating the changes on the secondary’s dataset. Multiple primaries accepting write operations would lead to data conflicts.

The critical part if, you always need to have odd number of servers inside the replica set. If a replication set has only one secondary member, an arbiter is required.

In this article, We will see how to setup 3 member cluster replica set.

1. Setup the each instance of replica set

Create and setup 3 new instances of mongodb server instance with stable MongoDB release. Once you have all your private IPs, add them to the hosts file.

Login to each machine, update /etc/hosts file with your private IPs of the 3 machines.

10.142.38.138 mongo1
10.142.38.139 mongo2
10.142.38.140 mongo3

Use your own IP addresses in place of the addresses in the above example. The names of the members in the replication set are also variables, so you may name them what you choose.

2. Edit mongodb.conf file

We need to do to begin the MongoDB configuration is stop the MongoDB process on each server.

service mongodb stop

Create a directory for MongoDB data

mkdir /mongodata

Edit the mongodb.conf file to add the private IP address, port and dbpath.

bind_ip=10.142.38.138
port=27017
dbpath=/mongodata

While still in the mongodb.conf file scroll to the bottom and add the following information:

fork=true
replSet=rs1

In this example, the sample replication set is rs1, however, you may change the name as you choose.

Start the replication member by issuing the following command in all machines

mongod --config /etc/mongodb.conf

3. Creating the Replication Set

Start the MongoDB client on only one member of the replication set with the command:

mongo --host mongo0

At the MongoDB prompt, switch to admin with the command:

use admin

Start the replication set by entering:

rs.initiate()

This will initiate the replication set and add the server you are currently connected to as the first member of the set. You can see this by typing:

rs.conf()

The output should look similar to the following:

{
    "_id" : "rs1",
    "version" : 8,
    "members" : [
    {
                "_id" : 0,
                "host" : "10.142.38.138:27017"

        }
    ]
}

Now you are ready to add secondary member:

rs.add("mongo2:27017")

Add an arbiter member:

rs.addArb("mongo3:27017")

To verify that the members have been added correctly run the rs.conf() command again.

rs.conf()

{
    "_id" : "rs1",
    "version" : 8,
    "members" : [
        {
            "_id" : 0,
            "host" : "10.142.38.138:27017"
        },
        {
            "_id" : 1,
            "host" : "mongo1:27017"
        },
        {
            "_id" : 2,
            "host" : "mongo2:27017"
        }
    ]
}

That’s it!

If this all looks simple, it’s because it is. We’ve been very pleased with how replica sets have smoothed out the bumps in handling operational emergencies as well as for general maintenance and scaling out.

If you have any other tips, share it in comments.