A Tale of Two DAGs – Deploying Exchange 2013 DAG Environment on Nutanix

As always, I try to blog on questions I get from the field and recently, I helped some colleagues and customers build a DAG on Nutanix 2 times last week ;).  The process is pretty straightforward, but there are some best practices that everyone should follow to ensure a healthy DAG environment.

Creating a two node Microsoft Exchange 2013 Database Availability Group


Ok, let me tell you how to make your Exchange 2013 Mailbox role deployment highly available (HA) by using a Database Availability Group (DAG).. Let’s get rocking’.

Before we get too deep into it, let’s first make sure you’re on the right plane. This flight will take you through configuring a DAG in Exchange 2013 with two nodes in the same site to make your Mailbox role highly available. In a later blog post I will be covering how to make your Exchange 2013 Mailbox role deployment site resilient by adding additional nodes in a remote location. If you’re still with me, buckle your seatbelt and keep your hands inside the cabin at all times.

Server Safety Check

Before we can take off, we need to do a pre-check routine on all our servers because the last thing we want is to get cruising at 20,000 users and then have it crash. Let’s discuss the operating system requirements.

Operating System

DAGs utilize Windows Failover Clustering so you’re going to need a Windows OS that supports that and since Exchange 2013 can only be installed on Server 2012 and Server 2008 R2 SP1, we’ll focus on those. If you’re running 2008 R2, you’ll want Enterprise or Datacenter. If you’re running Server 2012, you can use Standard or Datacenter. There’s nothing worse than installing Exchange only to find that you put the wrong OS on there when you go to configure the DAG. I have seen this happen many times at customers. Just in case you’re wondering, Microsoft does not support upgrading the OS once Exchange has been installed so your only option is to uninstall Exchange, install the correct OS, and then reinstall Exchange or build a new server with the correct OS. After you’ve got the correct OS, you’ll want to install the most current updates.

The Network

DAGs need to replicate and clients need to connect, so it’s only natural that you’d need two NICs to create a DAG. Microsoft supports a single NIC but it’s recommended you use two so you can separate the replication traffic from the client traffic. You don’t want the reseeding of a database to affect your users’ experience. Let’s talk about the two NICs real quick. The first thing I like to do is name them appropriately. I usually name my replication NIC REPL and my client NIC MAPI, that way there’s no confusion in this next step I take, and that’s to put them in the correct binding order.
The MAPI NIC needs to be listed first. This is how you do that.

  1. From the command prompt, type in NCPA.CPL to open up the Network Connections screen.
  2. Press the Alt key and that will display the File menu, then click on Advanced and select Advanced Settings…
  3. On the Adapters and Bindings tab, order the adapters so that MAPI is first and REPL is second.

Note: If you’re using network teaming, ensure the MAPI team is listed first and the REPL team is listed second.
Quick note about Network Teaming. I’m a fan of it because I don’t want a NIC failure to cause a database failover. NIC teaming is supported on client and replication networks but you have to configure each team so that only one NIC in the team is up and the other NIC is in standby, which means Active/Passive.

Last thing I want to point out is the IPv6 check box on each NIC. Everywhere I go I see admins – this box because they don’t “support” IPv6. First point is, unchecking that box doesn’t truly disable IPv6. Second point is, if you’re having issues with Exchange connectivity, chances are having the check box checked is not your issue. Lastly, Microsoft does not test Exchange with that box unchecked, so technically you’re in an unsupported configuration by unchecking it. All that was said to say, leave it alone even though you really, really want to uncheck it.

Client Network

This is the king of NICs because it gets it all, default gateways, DNS settings, and priority, and there can only be one! You’ll configure this just like you would any other NIC card for a server.

Replication Network

The replication network needs to be on a different subnet than the client network or the DAG won’t see them as two separate networks, obviously. For this post, I used for my MAPI network and for my replication network. If you have a separate network for backups or administration, I would use that. I wouldn’t create a dedicated network just for Exchange replication unless you have to. DAG replication networks can share. It learned how to in kindergarten. If you want to use more than one replication network, then each replication network will need to be on different subnets as well. There are a few things you need to do with this network. Let’s go through that because you get to uncheck stuff, just not the IPv6 box

  1. From a command prompt, type in NCPA.CPL to open the Network Connections window
  2. Right-Click the Replication NIC and go to Properties
  3. Uncheck Client for Microsoft Networks and File and Print Sharing for Microsoft Networks
  4. Select IPv4 and click Properties
  5. Input your IP Address and Subnet mask. Do NOT put in a Default Gateway or DNS servers.
  6. Click on Advanced…
  7. Click on the DNS tab and deselect Register this connection’s addresses in DNS
  8. Click on the WINS tab and select Disable NetBIOS over TCP/IP
    Wow that was a lot of steps. Ensure you click Ok to save all that stuff!

Nutanix Best Practices for Exchange Storage

  • Separate database and log files into separate containers
  • Databases should be on a container with in-line compression and EC-X enabled
  • The Windows NTFS Allocation unit size should be set to 64KB for Database Drives
  • Log Files should be on a container with no attributes and located on a volume with 4k block allocation size.
  • Archive Databases should be on a separate container from active databases with In-Line Compression and EC-X enabled

As per Microsoft Best Practices, Nutanix recommends to disable deduplication for the container hosting active Exchange Databases workloads. Deduplication does not provide significant benefits due to frequency of change in the Exchange Databases.
Alright, we’ve got our servers ready for the DAG, now it’s time to install Exchange. If you need help doing that, I’ve will be covering that in a future blog post. If you have Exchange already install, then move on forward.

Creating the DAG

Now the real fun begins! We get to create stuff. Let’s open up the Exchange Admin Center (EAC) and click on Servers –> database availability groups.
As you can see I already have a DAG but don’t worry, I didn’t get started without you. I’m going to create another one cleverly named DAG02. Click on the ‘+’ sign so we can create a new DAG.
In the new database availability group windows, You’ll give this a name, pick a witness server and give it an IP. Let’s talk about each of those real quick.

DAG Name

This name needs to be unique in the environment because once you join the first server to the DAG, a Cluster Name Object (CNO) will be created in Active Directory with this name. This CNO can be pre-staged, meaning you can create it manually before Exchange does. You might want to do that in scenarios where you don’t have rights to create Computer objects in Active Directory. At the time I’m writing this, Technet states that in Exchange 2013, if you’re DAG members are running 2012 that you have to pre-stage the CNO, but my testing has proven otherwise. Regardless, stay on the safe side and pre-stage the CNO if you’re OS is 2012.

Witness Server

You have to pick a server to host a share for the DAG. This share is only needed when there is an even number of nodes in the DAG, and it’s not utilized until you need it to maintain quorum. I’ll explain that in more detail in my Nerd Knowledge section below. Skip it if you don’t care. If you plan to have an odd number of nodes in your DAG you have to configure it regardless because Exchange will use it to adjust the quorum configuration automatically as you add and remove nodes from your DAG.

The next question is, “Rob, who should I give the honor of being the witness to?” Good question! We’ll, if you are separating your CAS and Mailbox roles out, pick a CAS server. In fact, if you leave the witness server blank and there is a CAS server in that site, Exchange will automatically pick it. If you’re not separating out the roles, then you have to pick a server. A good candidate would be a server that’s not going to be rebooted a lot but I wouldn’t have a dedicated witness server, it’s a waste of resources, so pick an existing file server or a server that’s within your area of control. Before you decide to pick a Domain Controller, review the witness bullet points below.

  • Witness Servers must be in the same forest and not be a member of the DAG.
  • Witness Server must have the Exchange Trusted Subsystem (ETS) group added to the local administrators group. If this is a Domain Controller, it needs to be added to Domain\Administrators. This is why you might want to not pick a domain controller. This gives the ETS administrative rights to the entire domain, not just that server. Granted ETS has crazy rights already, but putting the ETS in the Administrators group for the domain might make your security people nervous.
  • Witness Server must be running Server 2012, Server 2008 R2, Server 2008, Server 2003 R2, or Server 2003.

Few more notes about the Witness server. It does not need to be highly available, meaning don’t put it on a cluster, and a server can serve as the witness server for multiple DAGs but each DAG needs it’s own witness share. Also, you don’t need to specify a path when creating the DAG. Leave it blank and Exchange will create the proper folder structure. Lastly, I’m assuming you’re installing two or more nodes in one physical site. If you plan to put nodes in other sites, then you’ll want to pick a Witness server in the site that holds the majority of your users.

Nerd Knowledge: I mentioned that the Witness server is only used when there is an even number of nodes and only utilized when it’s needed to maintain quorum, and those of you that are nerds like me, I wanted to explain that a little more. Like I mentioned before, a Witness server is only used when there is an even number of nodes in the DAG, but Exchange will want to configure one so that it can automatically adjust the quorum configuration as you add and remove nodes. For example, if you have two nodes, under Failover Cluster Manager you’ll see the quorum configuration set to Node and File Share Majority. If you add a third node, the quorum configuration will change to Node Majority and the Witness server is not used at all.

Now to address the other statement of the Witness server only being utilized to maintain quorum. A DAG must “have quorum” to mount databases and if it loses quorum, you’re environment will go down. To maintain quorum there must be enough votes in the DAG and each server is a voting member. For example, if you have a two node DAG the number of votes to stay up is 2. That can be found by taking the number of nodes in the DAG, dividing it by 2, and then adding 1. In a two node DAG that looks like this (2 nodes / 2) +1 = 2. Again, that means in a two node DAG we need two votes to maintain quorum and keep our databases mounted.

I know what you’re thinking, “But, Rob, we only have two members in our DAG and we need two votes. Doesn’t that mean we can’t lose a server without dismounting all of our databases?” Fair question, young padawan so let me tell you. This is where the Witness server comes into play. In a two node DAG, in normal operation, a Witness server is not used because we don’t need it. We have our two voting members up, but if we lose one of those nodes, the surviving node will try to lock the witness server. If it successfully locks the witness server, it becomes special and gets a second vote, which gives us our two required votes to maintain quorum and life is good. However, if you reboot the witness server in this scenario, you’ll lose quorum and your mailboxes will dismount. Even worse, they won’t mount again until you either “shrink” your DAG, or get both nodes back up. That might sound confusing and I can cover that in another post because the length of this is getting out of control, but just remember that if you have one of your two DAG members down, don’t reboot your witness server or your surviving node.

Wow, that’s a lot, but I need to explain one more example to ensure we fully understand this. Let’s assume we have six nodes in our DAG. To maintain quorum we need four votes (6/2) + 1 = 4. We have an even number of nodes so we need a witness server, but we won’t use it until we need it to maintain quorum. If we lose one or two of the six nodes, we still have enough votes, so right now we don’t need the witness server. If we lose a third node, one of the surviving nodes will try and lock the witness server.

Whoever grabs it first wins and that surviving node gets an extra vote making it 2 +1 +1 = 4. Make sense? I hope so, cause I lost my own attention two paragraphs ago! LOL Let’s move on.

DAG IP Address

This is the IP address used by the DAG. It uses DHCP by leaving it blank or you can specify a static IP address here.

Now that you’ve put in all the information, click Ok. Finally!!
You can now see the DAG in the EAC, and if wanted to be a true nerd, you can open ADSIEDIT and go to

CN=Database Availability Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=<Org Name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<Domain>,DC=<Com>. You should see the msExchMBDatabaseAvailabilityGroup object.

You should also check the witness server to ensure the file share was created.

Adding Servers to the DAG

Now that we have an empty DAG and a witness server, we need to add servers to it. This is done by highlighting the DAG in the EAC and clicking the Managed DAG membership.
On the manage database availability group membership screen, click the ‘+’ sign to add servers to the DAG.
Select the servers you want to add to the DAG. You can select multiple servers here. I’m adding two servers at the same time. I’m skipping RED-15EXCH02 because it’s running 2008 R2 and the other two servers I’m adding are running 2012.

When you add the first Exchange server to the cluster, that server creates the CNO in Active Directory if you didn’t pre-stage it. Microsoft has reported that if you add Exchange servers to the DAG too quickly and Active Directory does not have time to replicate, the second Exchange server might not see the CNO and then create it’s own. This might be another reason to pre-stage the CNO before adding a node to the DAG or at least make sure you force replication after you add the first node to the DAG to replicate around the CNO.
Once the servers have been added to the DAG, you should see something like this.
Now that we’ve created our DAG, we can take a look at it by opening up the Exchange Management Shell (EMS) and running Get-DatabaseAvailabilityGroup DAG02 -Status | FL. That should give you something like this. You can notice a few things from this screenshot. You can see that Compression and Encryption are enabled between DAG members in other sites if we decided to extend this DAG to another location.

Note that it’s a DAG property not a network property so that means it’s either enabled for all networks in the DAG or none of them. Another thing worth noting is the port that the DAG replicates on, 64327. This is helpful when there is a firewall between your Exchange servers.
Now your screenshot might look a little different than mine because I ran Set-DatabaseAvailabilityGroup DAG02 -DatacenterActivationMode DagOnly. This prevents a split brain cluster from forming, so that makes this setting a must in all DAGs. I highly recommend you set it too.

Lastly, you can see that our witness server is configured and what server is hosting the Primary Active Manager.

Let’s take a quick look at the Failover Cluster Manager to see what happened there.
From above screenshot, you can see that we have a cluster named after the DAG and our two servers are added as nodes. You can also see that the cluster configuration is Node and File Share Majority. Remember, it’s because we have an even nodes in the cluster. If we added a third node, it would configure the cluster to use Node Majority.

Quick Tip: If you’re worried about your cluster configuration, witness server, or your DAG settings in general, try running Set-DatabaseAvailabilityGroup DAG02 from EMS. That’s right, no parameters. It will make sure you didn’t do anything stupid. I use this like Roy uses, “Have you tried turning it off and on again?” If you don’t know who Roy is and have never heard that phrase, you need to stop reading this post, open a Netflix account, if you don’t already have one, look up “IT Crowd”, watch the “The Speech” episode on Season 3, and come back here and post about how funny it was and how much you want to thank me for telling you about it.

One last thing I’d like to point out here are the DAG networks. If you open up the EMS, and run Get-DatabaseAvailabilityGroupNetwork, you’ll see your DAG networks. Remember that I have two DAGs, but you should see something similar to DAG02’s networks DAG02\MapiDagNetwork and DAG02\ReplicationDagNetwork01. Why did it put a ’01’ at the end of the replication network and not the MAPI network? Because MAPI is the highlander and there can only be one! You can also see which networks are enabled for replication. I’ve noticed that my MAPI network for DAG02 is configured for replication, but I’m going to fix that below.
In Exchange 2010, you had to configure your DAG networks manually, such as disabling replication on the MAPI network. If you run Get-DatabaseAvailabilityGroupNetwork, you can see that Exchange 2013 auto configured it for us. In fact, if you want to configure the DAG networks manually in Exchange 2013, you have to run Set-DatabaseAvailabilityGroup dag02 -ManualDagNetworkConfiguration $True. I’m going to run this command to disable replication on my MAPI network because the DAG enabled it for me and I want that disabled on that network to force everything through the replication network. This doesn’t meant that the DAG will never use the MAPI network to replicate because in an emergency situation when all replication networks are down, DAG02 will use the MAPI network if it has to so that replication continues. You can run the following command to see which network your DAG is using for replication. Get-MailboxDatabaseCopyStatus -Server red-15exch01 | fl name,incoming,outgoing. Running this from both servers in our DAG will show which network the log copying is coming in on.

Adding Database Copies

Well done so far! You’ve created a DAG and hopefully learned some stuff. Now that we do have this fancy DAG, we need to use it by creating Database copies. Let’s jump back over to the EAC and click on servers, then databases. One quick thing here. Am I the only one that hates it when menu items aren’t capitalized??? That drives me nuts that the menu options in the EAC are not capitalized, but I’m open to change. We’ll see how that goes. Ok, back on track.

Click on the database hosted by one of the member in our DAG and click on the “…” and select Add database copy. This will “seed” the database, meaning copy over the EDB file over to the server you pick and then copy over the log files. If you’re database is large, it could take some time depending on your bandwidth. If you’re doing this in a lab with an empty database, it should go pretty quick. You’ll want to do this for each database in the DAG that you want protected.DAG-CreatingDAG13
And we’re done!!!
In this post you learned how to create and get started on configuring a DAG. I talked about Nutanix Best Practices around DAG’s.  I talked way too much about the Witness Server.. Man there was a lot in this post, but I truly hoped you learned something even if it’s never to read one of my posts again. :).

As always, I encourage your feedback and any questions you might have..

Until next time, Rob

Exchange Backup Craziness – Log File Cleanup

Exchange Backup Craziness

“Often, you hear about something weird and un-supported, and feel like you have to share it”.

I often get calls and questions regarding backups and Exchange Server, and most backup technologies are not always working as required or as you would expect, but that’s off-topic.

One of the most common stories is that without a working Exchange Server backup, when you perform massive mailbox moves or no backup at all, the transaction logs will get piled and fill up the volume that they reside in. and then panic starts, “hey my databases were dismounted…” then of course the administrator realizes that the space on the log drive or volume has indeed ran out and now he needs to figure out what to delete. On Nutanix, we simply can solve this by extending the container that the logs live in, but what if you rely on snapshots for backups.

I had a customer reach out to me running Exchange 2007 with CCR ( Cluster Continuous Replication) on Nutanix. Yes, you heard me right, Exchange 2007 ;).  They are planning on migrating to Exchange 2013 in the next year or so, but need to get from A to B for budgetary reasons until then.  The only form of backup the customer has is to use Nutanix daily snapshots.  The customer understands the painful process of restoring an individual mailboxes from snapshots and not having up to date recovery that logs provide along with the point in time database backup, but its a risk they are willing to take as opposed to having nothing. They reached out to me and asked, how do cleanup logs that are piling up. And so here’s where this post comes in…

My blog article suggests that you cannot sustain downtime or interruption for your users while battling with deleting log files or restoring your working backup solution. If you can sustain a downtime (should be around minutes or so) the easiest method will be to enable Circular Logging on your database / storage group – see more here –

The customer needs to be able to purge the committed logs so they don’t fill up their disk space.  So how can you delete or purge Exchange server logs without any risk? well, in simple – you cannot, its built-in by design, because the whole idea of restoring an Exchange or for this matter any transnational database requires you to have a first – “full” backup of the database itself and all transaction logs that were generated since the date of the database creation date, or the last “successful” “full backup”.

Now here’s a nice method to “fake” a “full backup” or an on-demand transaction logs purge when you see you will be soon out of space, using the Exchange VSS writers and the diskshadow utility (available with Server 2008R22012R2) . This procedure also “proves” that a VSS backup for your Exchange Server will work normally.

Please note: This method was tested on an Exchange 2010 server with using a Nutanix block NX-3460-G4. Use this method on your risk.  This is not supported by Nutanix or Microsoft.. You should perform a “Snapshot” before and right after this process is done.

How to manually purge Exchange server logs – with ease

This example will show you how to purge the logs for a database that is located on Drive D, the log files of the databases are also located in Drive D. we will “fake backup” drive D and this will trigger the logs to be purged.Note: If you have separated your log files and database file in different drives, or you want to include additional databases in the “backup” you must include the additional drives in the process, so in the example below, you will “Add volume e:” after “Add volume drive d:” and so on…

  1. Open Command prompt
  2. Launch Diskshadow
    1. Add volume d:
    2. (optional, add one line for each additional drive to include) Add volume X:
    3. Begin Backup
    4. Create
    5. End Backup
  3. At this step you should notice the following events in the application log indicating that the backup was indeed successful and logs will now be deleted.

Here’s some screenshots of the process:Exchange Backup Craziness
The Diskshadow example screenshot.

ESE – Event ID 2005 – Starting a Full Shadow Copy Backup

MSexchangeIS – Exchange VSS Writer preparation.
ESE Event ID 224 – Logs are now purged

MSExchangeIS Event ID 9780 – Backup is now complete.

Final Note: although this example was tested against Exchange 2010, it should work just as fine with Exchange 2016/2013 & 2007

Until next time, Rob.

Exchange Server 2016 RTM Released: Forged in the cloud. Built for Web-Scale

Exchange Server 2016 is here and available to download!!!

What sets this version of Exchange apart from the past, is that it was forged in the cloud. This release brings the Exchange bits that already power millions of Office 365 mailboxes to your on-premises environment. And deploying Exchange 2016 on Nutanix, you can truly create the ultimate email web-scale environment.

Email remains the backbone of business communication and the one that workers consider the most essential tool for getting things done. Because of this, it’s vital to have a modern messaging infrastructure that meets today’s business expectations of scale. With the volume of email and other communications continuing to grow, people need tools that help them focus on what’s most important in their inboxes, schedules and interactions with others at work. And as the quantity of email data grows, so do the demands on IT to manage, preserve and protect it. This is why Web-Scale so important in an Exchange 2016 environment.

Web-Scale Fundamentals  
Exchange Server 2016
To help you meet these challenges with Exchange Server, Microsoft has deepened the integration between Exchange and other Office products, so your organization can be more productive and collaborate more effectively. They’ve made it easier to manage your email with new ways to focus on what’s important, work more efficiently, and accomplish more with your devices. Microsoft has also simplified the Exchange architecture and introduced additional recovery features.

Exchange 2016 builds on and improves features introduced in Exchange 2013, including Data Loss Prevention, Managed Availability, automatic recovery from storage failures, and the web-based Exchange admin center.

  • Better collaboration: Exchange 2016 includes a new approach to attachments that simplifies document sharing and eliminates version control headaches. In Outlook 2016 or Outlook on the web, you can now attach a document as a link to SharePoint 2016 (currently in preview) or OneDrive for Business instead of a traditional attachment, providing the benefits of coauthoring and version control.
    Exchange Server 2016
  • Improved Outlook web experience: Continuing our effort to provide you with a first class web experience across devices, Microsoft has made significant updates to Outlook on the web. New features include: Sweep, Pin, Undo, inline reply, a new single-line inbox view, improved HTML rendering, new themes, emojis, and more.
    Exchange Server 2016
  • Search: A lightning-fast search architecture delivers more accurate and complete results. Outlook 2016 is optimized to use the power of the Exchange 2016 back-end to help you find things faster, across old mail and new. Search also gets more intelligent with Search suggestions, People suggestions, search refiners, and the ability to search for events in your Calendar.
    Exchange Server 2016
  • Greater extensibility:  An expanded Add-In model for Outlook desktop and Outlook on the web allows developers to build features right into the Outlook experience. Add-ins can now integrate with UI components in new ways: as highlighted text in the body of a message or meeting, in the right-hand task pane when composing or reading a message or meeting, and as a button or a dropdown option in the Outlook ribbon.
    Exchange Server 2016
  • eDiscovery: Exchange 2016 has a revamped eDiscovery pipeline that is significantly faster and more scalable. Reliability is improved due to a new search architecture that is asynchronous and distributes the work across multiple servers with better fault tolerance. You also have the ability to search, hold and export content from public folders.
  • Simplified architecture: One Role…!  Exchange 2016’s architecture reflects the way we deploy Exchange in Office 365 and is an evolution and refinement of Exchange 2013. A combined mailbox and client access server role makes it easier to plan and scale your on-premises and hybrid deployments. Coexistence with Exchange 2013 is simplified, and namespace planning is easier.
  • High availability: Automated repair improvements such as database divergence detection make Exchange easier than ever to run in a highly available way. Stability and performance enhancements from Office 365, many of which were so useful that Microsoft shipped them in Exchange 2013 Cumulative Updates, are also baked into the product.

That’s just quick list of highlights; I encourage you to get a full view of what’s new by reviewing the Exchange 2016 documentation on TechNet.
Or, if you are in the mood for something more bite-sized, check out these short demo videos in which a few members of the Exchange team show off their favorite features:

Exchange 2016 will follow the same servicing rhythm as Exchange 2013, with Cumulative Updates (CUs) released approximately every three months that contain bug fixes, product refinements, and selected new investments from Office 365. The first CU is expected to arrive in the first quarter of 2016.

Until next time, Rob….