Microsoft Ignite 2017 Summary and Announcements

Ignite 2017 Key takeaways

This was the first year I have not attended Microsoft Ignite, due to unforeseen circumstances. But this didn’t stop me from covering Ignite 2017. So here we go…

Ignite 2017 this year has about 25k attendees. During the same time as Ignite, they are also running Microsoft Envision. This is more focused to business leaders across industries.  Its main focus is to have Business Leaders understand and manage their organizations in the Digital Age.

Ignite 2017 Attendee Breakout

  • 47 % ITI/IT Pros
  • 34% Developers
  • 19% ITDM.

Top Industries Attended

  • 34% IT and Software (flat YoY)
  • 20% Education
  • 9% Healthcare
  • 9% Manufacturing
  • 9% Professional & Business Services

Ignite Keynotes Summary and Links


Modern Workplace

Key Takeaways – Modern Workplace

Expanding Microsoft 365

  • Microsoft 365 Firstline offering and Microsoft 365 Education
  • New Windows 10 S devices from HP, Lenovo, Acer and Fujitsu starting at $275 USD

Intelligent personalized search power by Microsoft Graph

  • Bing for business
  • LinkedIn data integrated with Office 365 profile card
  • Office 365 search & discovery improvements
  • Windows 10 taskbar search

Intelligent Communications vision

  • Bring voice and video + new cognitive and data services into Micro Teams

Advances in Intelligent Security

  • Integrated Adminced threat Protection using Intelligent Security Graph
  • Better data protection and access control across Microsoft 365
  • New Compliance Manager, a single GDPR dashboard

Modernizing Business Process with Cloud and AI

Key Takeaways – Business Applications

New Microsoft Dynamics 365 AI Solutions

  • First solutions for customer care includes a virtual agent for customers, an intelligent assistant for support staff and conversational AI management tools, power by Microsoft AI
  • HP, Macy’s, and Microsoft already using this technology to improve customer satisfaction and handle more requests, more quickly

Modular apps for Dynamics 365

  • New modular apps are lightweight SaS services designed to transform one business process at a time
  • Work with Dynamics 3 business apps or can be used independently
  • Extend existing systems of record, integrate with Office 365 and augment with LinkedIn insights.
  • First to allow talent leaders and hiring managers to address a company’s most important asset, people
  • Attract: focused on recruiting | Onboard: helps you make new employees successful – Available later this year.

Deeper integration for PowerApps and Microsoft Flow + Office 365 and Dynamics 365

  • Rapidly build apps, automate tasks, simplify workflows and solve unique business problems.
  • Allow any business user familiar with InfoPath forms, Access databases or SharePoint list. This allows customers to build apps that help them achieve more, on a single no-code/low code platform.

Apps and Infra/Data and AI

  • Every customer is an AI customer

The Enterprise Cloud

Key Takeaways – Hybrid

Delivering true hybrid consistency

  • Azure Stack shipping through OEM partners including Dell EMC, HPE, and Lenovo
  • Database Migration Service (DMS)

Empowering customer to optimize costs

  • Azure Hybrid Benefit for SQL server
  • Azure Cost Management by CFloudyn – free to all Azure subscriptions

Key Takeaways – Intelligence

Any data, any place

  • SQL Server on Linux Windows and Docker availability with SQL Server 2017 GA’

One convenient workbench for data scientists and AI developers

  • Azure Machine Learning Updates

Build intelligent apps at global scale

  • Azure Cosmos DB and Azure Functions integration

Performance and Scale for mission-critical analytic apps

  • Azure SQL Data Warehouse preview release of new “optimized for compute” performance tier

Cloud for Good – Key takeaways

To empower nonprofits, Microsoft Philanthropies will:

  • Microsoft has announced they met their 2016 commitment to donate $1 billion in cloud computing resources to nonprofits
  • Continue the cloud donations program, and triple the number of nonprofits Microsoft serves over the next three years
  • Launch a new Tech for Social Impact group, and the first offers, announced this week include:
    • Microsoft 365 for Nonprofits
    • Nonprofit Surface discounts for the first time ever

To get more detailed information about these announcements, please see links below or check out the Ignite2017 Site.

Official Microsoft Blog
Office Blogs
EMS Blog
Dynamics Blog
Azure Blog
Hybrid Cloud Blog
Data Platform Blogs

Until next time, Rob.

Microsoft Exchange Best Practices on Nutanix

To continue on my last blog post on Exchange…

As I mentioned previously, I support SE’s from all over the world. And again today, I got asked what are the best practices for running Exchange on Nutanix. Funny enough, this question comes in quite often.  Well, I am going to help resolve that. There’s a lot of great info out there, especially from my friend Josh Odgers, which has been leading the charge on this for a long time.  Some of his posts can be controversial, but truth is always there.  He’s getting a point across.

This blog post will be updated on a regular basis as things change. It will also be moved to a permanent part of the resources section.  This is meant to be a general best practice guide to help with planning and maintaining a healthy Exchange environment on Nutanix.  I will specify hypervisor specifics when required.  Now on the post…..


Let’s start out with the basics…

MS Exchange on Nutanix Support

Nutanix provides a 100% supported solution for MS Exchange running on vSphere, Hyper-V or Acropolis Hypervisor using iSCSI (Block storage)
Here is a breakdown of supported configurations by hypervisor:

vSphere (ESXi)Use In-Guest iSCSI (Volume Groups) for full support
Hyper-V Use SMB 3.0
AHVUse native vDisks (iSCSI) – SVVP Certification for AHV

Also, check out Josh’s post “Fight the FUD – Support for MS Exchange on Nutanix” that outlines this very topic.  In summary, the customer has the choice to deploy in multiple configurations to suit their needs. But, one of the most often questions I get is, “does your SVVP Certification cover running Exchange on all your supported hypervisors?”  The answer is not simple.  The SVVP was submitted for the Acropolis Hypervisor, while this does not cover all of them, we technically are supported for all hypervisors as per Microsoft supported storage architectures.  Microsoft does not specifically mention Hyperconverged, it only mentions ISCSI in regards to SAN.  IMO, that covers ESXi and AHV.

Now let me explain….SAN’s are one of the biggest modern datacenter bottlenecks. Data has gravity, so co-locating storage and compute eliminates network bottlenecks = Hyperconverged is way better than SAN and hence SUPPORTED IMO 😉
To end this topic and move on, a Nutanix customer has the choice to deploy in multiple configurations to suit their needs.  Being pushed to one particular hypervisor for a customer is not always in their best interest.  Having choices now and later is a much better approach with the overall goal of simplifying the datacenter.   As Josh said in one of his blog posts ,”Running a standard platform and storage protocol for all workloads is a simple model which reduces the unnecessary complexity of multiple protocols and/or in-guest storage configurations”, I can’t agree more with that statement. 🙂

Exchange Performance on Nutanix

Now this subject will always be controversial and potentially subject to criticism.  Internal testing performed by the Nutanix Performace and Engineering team shows that AHV and Hyper-V performance are roughly the same from a hypervisor perspective and ESXi was 10% higher. That being said, usually, the next question is how is performance versus traditional SAN/NAS.  And again, I have to point out, it’s all about Data Locality. Can’t change the laws of physics. Data has gravity, hence we will always beat traditional SAN architecture.

Check out Josh’s posts on “Peak Performance vs Real World – Exchange on Nutanix Acropolis Hypervisor”.  It gives you a better understanding of are realistic benchmarks of Exchange in general and on Nutanix. I wholeheartedly agree with Josh when he says “Benchmarks are of little value without context specific to customer requirements!”  Spending close to over 15 years building and maintain Exchange systems, I learned one hard fact, no generic simulator (like JetStress) can show real world metrics.

Data Reduction Technologies with Exchange on Nutanix

1 vDisk per Database, 1 vDisk per DB Logs
1 Container with RF2, In-Line Compression & EC-X for Databases
1 Container with RF2 for Logs
Do not use Dedupe with MS Exchange!
Microsoft does not support Data deduplication (Note: Underlying storage deduplication such as Nutanix dedupe is not mentioned, but implied)

Data Reduction Estimates:

Rule of thumb: Always size without data reduction if possible.
Conservative assumption for compression for Exchange = 1.3:1
Aggressive assumption for compression for Exchange = 1.6:1
Conservative assumption for EC-X for Exchange = 1.1:1
Aggressive assumption for EC-X for Exchange = 1.25:1

Questions to ask yourself when planning an Exchange Environment:

How many Users? e.g.: 10000, 10000, etc.
How many user profiles do you need? e.g.: 2 , Standard and Executives
How large Mailbox (excluding archiving) per User? e.g.: 1GB, 2GB , 5GB
How many messages per day do you want to support per user? Light = 50 , Medium = 100 , Heavy = 150+

Do you require site resiliency?

These are among some of the basic questions you need to answer.  This is where the Exchange Server Role Calculator comes in. It’s a great tool, but like any tool, you do need to give it good input to get out good output. The function of the tool is as the name implies.

Exchange Server Role Calculator Defined

Now, at the time of this writing, version 7.8 is the latest and greatest. Now, do note, I would not call this tool perfect, but its gets you pretty close. Like anything else, the Exchange team is still learning real world behavior and this is where a good experienced Exchange engineer comes into play.
IMO..there is an Art and Science to sizing Exchange.  The days of Exchange just being a simple mail server are far over. These days, it’s much more complex with supporting multiple forms of ingress and egress traffic for different functions (Mobile, Web, SMTP, Skype Integration, etc.). Each of these different functions has varying load considerations and supports more visible features like Outlook Web Access and Exchange Activesync. Also, I still am of the opinion that it does not take into consideration the number of devices that 1 mailbox services.
Considering this complexity, you can see that undersizing or oversizing can happen easily.  If you size correctly at the beginning with Nutanix, then it just an easy scale out, buy as you need it situation. Then you know what happens, finally for the first time, predictability in your budgets.  I remember the days, not that long ago, when I had to have a client retire a SAN, not for space constraints, but for IO constraints.  And at the time, all I got from the client was “can’t we use it for something else” and ya, I’ve replied with “use it as a WSUS repository for patching the Exchange environment” 😉
During my next post, I will dive into the Exchange Role Calculator much more and go over some examples of sizing on Exchange. We’ll mainly focus on mailbox storage and then move on to other role sizing considerations.  I also plan to cover the other aspects to maintain a healthy Exchange environment (i.e. Message Hygiene, Global and Local Load balancing, Integrations and End User Experience) in subsequent posts.
Below are the Office Best Practices Guides from Nutanix and some public case studies.

Until next time, Rob…..

Nutanix Offical Best Practice Guides
MS Exchange on Nutanix / vSphere Best practice guide:

Public Case Studies for Nutanix customers using Exchange

Microsoft Exchange Documentor 1.0 – Planning Tool

Microsoft Exchange Documentor 1.0 - Planning Tool
Have you had a need to document your Exchange environment?  Do you need a tool to help you plan your upgrade of Exchange?  Is your documentation current for compliance purposes?

In my job at Nutanix, one of the things I do is support our sales engineers globally on Exchange and other Microsoft solutions.  And yes, on-premise Exchange is still going strong, even with O365 in the mix.  I still see a lot of engagements in the field with Exchange and usually around keeping their data out of the cloud or a dependence on some kind of line of business application.  In any case, one of the biggest challenges transitioning to Hyperconverged technologies like Nutanix is understanding what you currently have.  The complexities of legacy 3-teir infrastructure are hard enough.  You also need to understand the requirements of your current Exchange environment.  Depending on the size of your Enterprise, this can be a time burn and can have a margin of error.  You need to understand Exchange PowerShell and other touch points in the Exchange environment.
Exhange_AlwaysOn Microsoft Exchange Documentor 1.0 - Planning ToolI originally built this tool, when I was working as a System Integrator before I joined Nutanix.  Its original purpose was for documenting a client Exchange environment after I was finished building it out.  The tool is based on PowerShell and leverages Microsoft Word to generate the final document. Currently, the tool only documents Exchange 2010 and does a basic hardware inventory.  My vision for the tool is to support all current versions of Exchange including O365 and build out a front-end GUI with report options. Maybe eventually, becoming a planning tool for sizing and transitioning your Exchange environment to true cloud-like agile infrastructure.
I started a GitHub project and you can check it out here.

In this fast-paced world, we need as much automation as possible to make our jobs more efficient, so, we as engineers can have more time back at home or doing the things we enjoy :). To quote a line from Kirt in Star Trek: The Wrath of Khan “Hours instead of days! Now, we have minutes instead of hours.” pretty much sums the world we live in.

livelongprosper Microsoft Exchange Documentor 1.0 - Planning Tool

Live long and prosper…Until next time, Rob…..

Exchange Server Role Requirements Calculator Explained – Part 1

The Exchange Server Role Requirement Calculator is your one stop calculation tool for Exchange 2013\2016 design. The tool covers design calculations for both the Mailbox and Client Access server role. Exchange 2013 reduced the number of roles from previous versions of Exchange by making the design and implementation as simple as possible and Exchange 2016 furthers that into a single role.  The Server Role Requirement Calculator helps us to size virtually and it provides in-depth sizing of every component of the hardware like CPU, Memory, Network, Storage, Backup, servers, datacenter etc.

Exchange Server Role Calculator is an excel spread sheet with an option to input your requirements.  The current version v7.8 of the calculator introduces support for Exchange 2016! Yes, that’s right, you don’t need a separate calculator, v7.8 and later supports Exchange 2013 or Exchange 2016 deployments.
When you open the calculator you will find a new drop-down option in the Input tab that allows you to select the deployment version. Simply choose 2013 or 2016:
When you choose 2016, you will notice the Server Multi-Role Configuration option is disabled due to the fact that Exchange 2016 no longer provides the Client Access Server role.

As discussed in the Exchange 2016 Architecture and Preferred Architecture articles, the volume format best practice recommendation for Exchange data volumes has changed in Exchange 2016 as they now recommend ReFS (with the integrity feature disabled). By default, for Exchange 2016 deployments, the calculator scripts will default to ReFS (Exchange 2013 deployments will default to NTFS).
Before we can properly size, we need to understand the below criteria and input to build our design:

Once we have determined all of the above, we should be ready to input those details into the input worksheet of Exchange Server Role Requirement Calculator. The Input worksheet is divided into 7 Parts:

1. Environmental configuration: This is where you input details of the AD architecture, Server roles, Vitalization, number of DAGs, number of nodes in the DAG, number of nodes in primary and DR datacenter. It also includes the number of DB copies in both primary and secondary datacenter with some transport message configurations like message queue expiration and safety net expiration.SRC1
Environmental Configuration

2. Mailbox Configuration: This is where you may need to input user profile details based on your existing profile and the projected mailbox growth percentage. There is an option to input different types of Tier profiles. The Existing user profile can be determined by using the Exchange profile analyzer scripts, mentioned above. Profile analyzer helps us to determine the messages sent & received per mailbox /day and average message size.

This part also has an important on IOPS and CPU Megacycles multiplication factor, which plays an important role for the users while designing mobile users & application users- as there is no clear information about the CPU and IOP’s requirement.
SRC2Mailbox Configuration

3. Backup Configuration: There is where we need to input the details of the required backup type, backup frequency and the log truncation tolerance days, in case of backup failure.
SRC3Backup Configuration

4. Storage configuration: Storage configuration is one of the most important factors during the design. This is where we input the type of disk you have for the Exchange database. Microsoft recommends an inexpensive SATA Just Bunch of Disk (JBOD) in the event there are three or more database copies. Exchange 2013\2016 also provides an option of multiple Database / volume, allowing for more than one database/volume on high disk capacity of 2-4 TB.
Storage configuration

5. Processor Configuration: While this is considered “optional” to input factor, I would highly recommend you have the CPU cores determined with SPECint2006 Rate using the processor query tool and input under processor configuration for both primary and secondary datacenter mailbox servers.
SRC5Processor Configuration

6. Log Replication Configuration: This configuration is optional and it is only used if the database is replicated between physical locations. You just have to input the logs generated/ hour percentage, network bandwidth type, and available latency between the primary and secondary datacenter.
Log Replication Configuration

7. Environment Customization: The Environmental Customization is actually optional. You’ll need to input the server names from both primary and secondary datacenter and DAG names, which will be used in the production environment. These names will be used for pictorial representation of the DAG, servers, and database. It is also used for DB creation scripts, which we will be discussing in the next part.

In this part we have covered all available input options in the Exchange Server Role Requirements Calculator. The Exchange calculator will use the imputed data and calculate the role requirement, storage design, network configuration, backup configuration, DAG configuration etc. for the organization and updates the results in the result worksheets.

Check out Part 2 – Exchange Server Role Requirements Calculator Explained

Until next time, Rob…

A Tale of Two DAGs – Deploying Exchange 2013 DAG Environment on Nutanix

As always, I try to blog on questions I get from the field and recently, I helped some colleagues and customers build a DAG on Nutanix 2 times last week ;).  The process is pretty straightforward, but there are some best practices that everyone should follow to ensure a healthy DAG environment.

Creating a two node Microsoft Exchange 2013 Database Availability Group


Ok, let me tell you how to make your Exchange 2013 Mailbox role deployment highly available (HA) by using a Database Availability Group (DAG).. Let’s get rocking’.

Before we get too deep into it, let’s first make sure you’re on the right plane. This flight will take you through configuring a DAG in Exchange 2013 with two nodes in the same site to make your Mailbox role highly available. In a later blog post I will be covering how to make your Exchange 2013 Mailbox role deployment site resilient by adding additional nodes in a remote location. If you’re still with me, buckle your seatbelt and keep your hands inside the cabin at all times.

Server Safety Check

Before we can take off, we need to do a pre-check routine on all our servers because the last thing we want is to get cruising at 20,000 users and then have it crash. Let’s discuss the operating system requirements.

Operating System

DAGs utilize Windows Failover Clustering so you’re going to need a Windows OS that supports that and since Exchange 2013 can only be installed on Server 2012 and Server 2008 R2 SP1, we’ll focus on those. If you’re running 2008 R2, you’ll want Enterprise or Datacenter. If you’re running Server 2012, you can use Standard or Datacenter. There’s nothing worse than installing Exchange only to find that you put the wrong OS on there when you go to configure the DAG. I have seen this happen many times at customers. Just in case you’re wondering, Microsoft does not support upgrading the OS once Exchange has been installed so your only option is to uninstall Exchange, install the correct OS, and then reinstall Exchange or build a new server with the correct OS. After you’ve got the correct OS, you’ll want to install the most current updates.

The Network

DAGs need to replicate and clients need to connect, so it’s only natural that you’d need two NICs to create a DAG. Microsoft supports a single NIC but it’s recommended you use two so you can separate the replication traffic from the client traffic. You don’t want the reseeding of a database to affect your users’ experience. Let’s talk about the two NICs real quick. The first thing I like to do is name them appropriately. I usually name my replication NIC REPL and my client NIC MAPI, that way there’s no confusion in this next step I take, and that’s to put them in the correct binding order.
The MAPI NIC needs to be listed first. This is how you do that.

  1. From the command prompt, type in NCPA.CPL to open up the Network Connections screen.
  2. Press the Alt key and that will display the File menu, then click on Advanced and select Advanced Settings…
  3. On the Adapters and Bindings tab, order the adapters so that MAPI is first and REPL is second.

Note: If you’re using network teaming, ensure the MAPI team is listed first and the REPL team is listed second.
Quick note about Network Teaming. I’m a fan of it because I don’t want a NIC failure to cause a database failover. NIC teaming is supported on client and replication networks but you have to configure each team so that only one NIC in the team is up and the other NIC is in standby, which means Active/Passive.

Last thing I want to point out is the IPv6 check box on each NIC. Everywhere I go I see admins – this box because they don’t “support” IPv6. First point is, unchecking that box doesn’t truly disable IPv6. Second point is, if you’re having issues with Exchange connectivity, chances are having the check box checked is not your issue. Lastly, Microsoft does not test Exchange with that box unchecked, so technically you’re in an unsupported configuration by unchecking it. All that was said to say, leave it alone even though you really, really want to uncheck it.

Client Network

This is the king of NICs because it gets it all, default gateways, DNS settings, and priority, and there can only be one! You’ll configure this just like you would any other NIC card for a server.

Replication Network

The replication network needs to be on a different subnet than the client network or the DAG won’t see them as two separate networks, obviously. For this post, I used for my MAPI network and for my replication network. If you have a separate network for backups or administration, I would use that. I wouldn’t create a dedicated network just for Exchange replication unless you have to. DAG replication networks can share. It learned how to in kindergarten. If you want to use more than one replication network, then each replication network will need to be on different subnets as well. There are a few things you need to do with this network. Let’s go through that because you get to uncheck stuff, just not the IPv6 box

  1. From a command prompt, type in NCPA.CPL to open the Network Connections window
  2. Right-Click the Replication NIC and go to Properties
  3. Uncheck Client for Microsoft Networks and File and Print Sharing for Microsoft Networks
  4. Select IPv4 and click Properties
  5. Input your IP Address and Subnet mask. Do NOT put in a Default Gateway or DNS servers.
  6. Click on Advanced…
  7. Click on the DNS tab and deselect Register this connection’s addresses in DNS
  8. Click on the WINS tab and select Disable NetBIOS over TCP/IP
    Wow that was a lot of steps. Ensure you click Ok to save all that stuff!

Nutanix Best Practices for Exchange Storage

  • Separate database and log files into separate containers
  • Databases should be on a container with in-line compression and EC-X enabled
  • The Windows NTFS Allocation unit size should be set to 64KB for Database Drives
  • Log Files should be on a container with no attributes and located on a volume with 4k block allocation size.
  • Archive Databases should be on a separate container from active databases with In-Line Compression and EC-X enabled

As per Microsoft Best Practices, Nutanix recommends to disable deduplication for the container hosting active Exchange Databases workloads. Deduplication does not provide significant benefits due to frequency of change in the Exchange Databases.
Alright, we’ve got our servers ready for the DAG, now it’s time to install Exchange. If you need help doing that, I’ve will be covering that in a future blog post. If you have Exchange already install, then move on forward.

Creating the DAG

Now the real fun begins! We get to create stuff. Let’s open up the Exchange Admin Center (EAC) and click on Servers –> database availability groups.
As you can see I already have a DAG but don’t worry, I didn’t get started without you. I’m going to create another one cleverly named DAG02. Click on the ‘+’ sign so we can create a new DAG.
In the new database availability group windows, You’ll give this a name, pick a witness server and give it an IP. Let’s talk about each of those real quick.

DAG Name

This name needs to be unique in the environment because once you join the first server to the DAG, a Cluster Name Object (CNO) will be created in Active Directory with this name. This CNO can be pre-staged, meaning you can create it manually before Exchange does. You might want to do that in scenarios where you don’t have rights to create Computer objects in Active Directory. At the time I’m writing this, Technet states that in Exchange 2013, if you’re DAG members are running 2012 that you have to pre-stage the CNO, but my testing has proven otherwise. Regardless, stay on the safe side and pre-stage the CNO if you’re OS is 2012.

Witness Server

You have to pick a server to host a share for the DAG. This share is only needed when there is an even number of nodes in the DAG, and it’s not utilized until you need it to maintain quorum. I’ll explain that in more detail in my Nerd Knowledge section below. Skip it if you don’t care. If you plan to have an odd number of nodes in your DAG you have to configure it regardless because Exchange will use it to adjust the quorum configuration automatically as you add and remove nodes from your DAG.

The next question is, “Rob, who should I give the honor of being the witness to?” Good question! We’ll, if you are separating your CAS and Mailbox roles out, pick a CAS server. In fact, if you leave the witness server blank and there is a CAS server in that site, Exchange will automatically pick it. If you’re not separating out the roles, then you have to pick a server. A good candidate would be a server that’s not going to be rebooted a lot but I wouldn’t have a dedicated witness server, it’s a waste of resources, so pick an existing file server or a server that’s within your area of control. Before you decide to pick a Domain Controller, review the witness bullet points below.

  • Witness Servers must be in the same forest and not be a member of the DAG.
  • Witness Server must have the Exchange Trusted Subsystem (ETS) group added to the local administrators group. If this is a Domain Controller, it needs to be added to Domain\Administrators. This is why you might want to not pick a domain controller. This gives the ETS administrative rights to the entire domain, not just that server. Granted ETS has crazy rights already, but putting the ETS in the Administrators group for the domain might make your security people nervous.
  • Witness Server must be running Server 2012, Server 2008 R2, Server 2008, Server 2003 R2, or Server 2003.

Few more notes about the Witness server. It does not need to be highly available, meaning don’t put it on a cluster, and a server can serve as the witness server for multiple DAGs but each DAG needs it’s own witness share. Also, you don’t need to specify a path when creating the DAG. Leave it blank and Exchange will create the proper folder structure. Lastly, I’m assuming you’re installing two or more nodes in one physical site. If you plan to put nodes in other sites, then you’ll want to pick a Witness server in the site that holds the majority of your users.

Nerd Knowledge: I mentioned that the Witness server is only used when there is an even number of nodes and only utilized when it’s needed to maintain quorum, and those of you that are nerds like me, I wanted to explain that a little more. Like I mentioned before, a Witness server is only used when there is an even number of nodes in the DAG, but Exchange will want to configure one so that it can automatically adjust the quorum configuration as you add and remove nodes. For example, if you have two nodes, under Failover Cluster Manager you’ll see the quorum configuration set to Node and File Share Majority. If you add a third node, the quorum configuration will change to Node Majority and the Witness server is not used at all.

Now to address the other statement of the Witness server only being utilized to maintain quorum. A DAG must “have quorum” to mount databases and if it loses quorum, you’re environment will go down. To maintain quorum there must be enough votes in the DAG and each server is a voting member. For example, if you have a two node DAG the number of votes to stay up is 2. That can be found by taking the number of nodes in the DAG, dividing it by 2, and then adding 1. In a two node DAG that looks like this (2 nodes / 2) +1 = 2. Again, that means in a two node DAG we need two votes to maintain quorum and keep our databases mounted.

I know what you’re thinking, “But, Rob, we only have two members in our DAG and we need two votes. Doesn’t that mean we can’t lose a server without dismounting all of our databases?” Fair question, young padawan so let me tell you. This is where the Witness server comes into play. In a two node DAG, in normal operation, a Witness server is not used because we don’t need it. We have our two voting members up, but if we lose one of those nodes, the surviving node will try to lock the witness server. If it successfully locks the witness server, it becomes special and gets a second vote, which gives us our two required votes to maintain quorum and life is good. However, if you reboot the witness server in this scenario, you’ll lose quorum and your mailboxes will dismount. Even worse, they won’t mount again until you either “shrink” your DAG, or get both nodes back up. That might sound confusing and I can cover that in another post because the length of this is getting out of control, but just remember that if you have one of your two DAG members down, don’t reboot your witness server or your surviving node.

Wow, that’s a lot, but I need to explain one more example to ensure we fully understand this. Let’s assume we have six nodes in our DAG. To maintain quorum we need four votes (6/2) + 1 = 4. We have an even number of nodes so we need a witness server, but we won’t use it until we need it to maintain quorum. If we lose one or two of the six nodes, we still have enough votes, so right now we don’t need the witness server. If we lose a third node, one of the surviving nodes will try and lock the witness server.

Whoever grabs it first wins and that surviving node gets an extra vote making it 2 +1 +1 = 4. Make sense? I hope so, cause I lost my own attention two paragraphs ago! LOL Let’s move on.

DAG IP Address

This is the IP address used by the DAG. It uses DHCP by leaving it blank or you can specify a static IP address here.

Now that you’ve put in all the information, click Ok. Finally!!
You can now see the DAG in the EAC, and if wanted to be a true nerd, you can open ADSIEDIT and go to

CN=Database Availability Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=<Org Name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<Domain>,DC=<Com>. You should see the msExchMBDatabaseAvailabilityGroup object.

You should also check the witness server to ensure the file share was created.

Adding Servers to the DAG

Now that we have an empty DAG and a witness server, we need to add servers to it. This is done by highlighting the DAG in the EAC and clicking the Managed DAG membership.
On the manage database availability group membership screen, click the ‘+’ sign to add servers to the DAG.
Select the servers you want to add to the DAG. You can select multiple servers here. I’m adding two servers at the same time. I’m skipping RED-15EXCH02 because it’s running 2008 R2 and the other two servers I’m adding are running 2012.

When you add the first Exchange server to the cluster, that server creates the CNO in Active Directory if you didn’t pre-stage it. Microsoft has reported that if you add Exchange servers to the DAG too quickly and Active Directory does not have time to replicate, the second Exchange server might not see the CNO and then create it’s own. This might be another reason to pre-stage the CNO before adding a node to the DAG or at least make sure you force replication after you add the first node to the DAG to replicate around the CNO.
Once the servers have been added to the DAG, you should see something like this.
Now that we’ve created our DAG, we can take a look at it by opening up the Exchange Management Shell (EMS) and running Get-DatabaseAvailabilityGroup DAG02 -Status | FL. That should give you something like this. You can notice a few things from this screenshot. You can see that Compression and Encryption are enabled between DAG members in other sites if we decided to extend this DAG to another location.

Note that it’s a DAG property not a network property so that means it’s either enabled for all networks in the DAG or none of them. Another thing worth noting is the port that the DAG replicates on, 64327. This is helpful when there is a firewall between your Exchange servers.
Now your screenshot might look a little different than mine because I ran Set-DatabaseAvailabilityGroup DAG02 -DatacenterActivationMode DagOnly. This prevents a split brain cluster from forming, so that makes this setting a must in all DAGs. I highly recommend you set it too.

Lastly, you can see that our witness server is configured and what server is hosting the Primary Active Manager.

Let’s take a quick look at the Failover Cluster Manager to see what happened there.
From above screenshot, you can see that we have a cluster named after the DAG and our two servers are added as nodes. You can also see that the cluster configuration is Node and File Share Majority. Remember, it’s because we have an even nodes in the cluster. If we added a third node, it would configure the cluster to use Node Majority.

Quick Tip: If you’re worried about your cluster configuration, witness server, or your DAG settings in general, try running Set-DatabaseAvailabilityGroup DAG02 from EMS. That’s right, no parameters. It will make sure you didn’t do anything stupid. I use this like Roy uses, “Have you tried turning it off and on again?” If you don’t know who Roy is and have never heard that phrase, you need to stop reading this post, open a Netflix account, if you don’t already have one, look up “IT Crowd”, watch the “The Speech” episode on Season 3, and come back here and post about how funny it was and how much you want to thank me for telling you about it.

One last thing I’d like to point out here are the DAG networks. If you open up the EMS, and run Get-DatabaseAvailabilityGroupNetwork, you’ll see your DAG networks. Remember that I have two DAGs, but you should see something similar to DAG02’s networks DAG02\MapiDagNetwork and DAG02\ReplicationDagNetwork01. Why did it put a ’01’ at the end of the replication network and not the MAPI network? Because MAPI is the highlander and there can only be one! You can also see which networks are enabled for replication. I’ve noticed that my MAPI network for DAG02 is configured for replication, but I’m going to fix that below.
In Exchange 2010, you had to configure your DAG networks manually, such as disabling replication on the MAPI network. If you run Get-DatabaseAvailabilityGroupNetwork, you can see that Exchange 2013 auto configured it for us. In fact, if you want to configure the DAG networks manually in Exchange 2013, you have to run Set-DatabaseAvailabilityGroup dag02 -ManualDagNetworkConfiguration $True. I’m going to run this command to disable replication on my MAPI network because the DAG enabled it for me and I want that disabled on that network to force everything through the replication network. This doesn’t meant that the DAG will never use the MAPI network to replicate because in an emergency situation when all replication networks are down, DAG02 will use the MAPI network if it has to so that replication continues. You can run the following command to see which network your DAG is using for replication. Get-MailboxDatabaseCopyStatus -Server red-15exch01 | fl name,incoming,outgoing. Running this from both servers in our DAG will show which network the log copying is coming in on.

Adding Database Copies

Well done so far! You’ve created a DAG and hopefully learned some stuff. Now that we do have this fancy DAG, we need to use it by creating Database copies. Let’s jump back over to the EAC and click on servers, then databases. One quick thing here. Am I the only one that hates it when menu items aren’t capitalized??? That drives me nuts that the menu options in the EAC are not capitalized, but I’m open to change. We’ll see how that goes. Ok, back on track.

Click on the database hosted by one of the member in our DAG and click on the “…” and select Add database copy. This will “seed” the database, meaning copy over the EDB file over to the server you pick and then copy over the log files. If you’re database is large, it could take some time depending on your bandwidth. If you’re doing this in a lab with an empty database, it should go pretty quick. You’ll want to do this for each database in the DAG that you want protected.DAG-CreatingDAG13
And we’re done!!!
In this post you learned how to create and get started on configuring a DAG. I talked about Nutanix Best Practices around DAG’s.  I talked way too much about the Witness Server.. Man there was a lot in this post, but I truly hoped you learned something even if it’s never to read one of my posts again. :).

As always, I encourage your feedback and any questions you might have..

Until next time, Rob

Exchange Backup Craziness – Log File Cleanup

Exchange Backup Craziness

“Often, you hear about something weird and un-supported, and feel like you have to share it”.

I often get calls and questions regarding backups and Exchange Server, and most backup technologies are not always working as required or as you would expect, but that’s off-topic.

One of the most common stories is that without a working Exchange Server backup, when you perform massive mailbox moves or no backup at all, the transaction logs will get piled and fill up the volume that they reside in. and then panic starts, “hey my databases were dismounted…” then of course the administrator realizes that the space on the log drive or volume has indeed ran out and now he needs to figure out what to delete. On Nutanix, we simply can solve this by extending the container that the logs live in, but what if you rely on snapshots for backups.

I had a customer reach out to me running Exchange 2007 with CCR ( Cluster Continuous Replication) on Nutanix. Yes, you heard me right, Exchange 2007 ;).  They are planning on migrating to Exchange 2013 in the next year or so, but need to get from A to B for budgetary reasons until then.  The only form of backup the customer has is to use Nutanix daily snapshots.  The customer understands the painful process of restoring an individual mailboxes from snapshots and not having up to date recovery that logs provide along with the point in time database backup, but its a risk they are willing to take as opposed to having nothing. They reached out to me and asked, how do cleanup logs that are piling up. And so here’s where this post comes in…

My blog article suggests that you cannot sustain downtime or interruption for your users while battling with deleting log files or restoring your working backup solution. If you can sustain a downtime (should be around minutes or so) the easiest method will be to enable Circular Logging on your database / storage group – see more here –

The customer needs to be able to purge the committed logs so they don’t fill up their disk space.  So how can you delete or purge Exchange server logs without any risk? well, in simple – you cannot, its built-in by design, because the whole idea of restoring an Exchange or for this matter any transnational database requires you to have a first – “full” backup of the database itself and all transaction logs that were generated since the date of the database creation date, or the last “successful” “full backup”.

Now here’s a nice method to “fake” a “full backup” or an on-demand transaction logs purge when you see you will be soon out of space, using the Exchange VSS writers and the diskshadow utility (available with Server 2008\R2\2012\R2) . This procedure also “proves” that a VSS backup for your Exchange Server will work normally.

Please note: This method was tested on an Exchange 2010 server with using a Nutanix block NX-3460-G4. Use this method on your risk.  This is not supported by Nutanix or Microsoft.. You should perform a “Snapshot” before and right after this process is done.

How to manually purge Exchange server logs – with ease

This example will show you how to purge the logs for a database that is located on Drive D, the log files of the databases are also located in Drive D. we will “fake backup” drive D and this will trigger the logs to be purged.Note: If you have separated your log files and database file in different drives, or you want to include additional databases in the “backup” you must include the additional drives in the process, so in the example below, you will “Add volume e:” after “Add volume drive d:” and so on…

  1. Open Command prompt
  2. Launch Diskshadow
    1. Add volume d:
    2. (optional, add one line for each additional drive to include) Add volume X:
    3. Begin Backup
    4. Create
    5. End Backup
  3. At this step you should notice the following events in the application log indicating that the backup was indeed successful and logs will now be deleted.

Here’s some screenshots of the process:Exchange Backup Craziness
The Diskshadow example screenshot.

ESE – Event ID 2005 – Starting a Full Shadow Copy Backup

MSexchangeIS – Exchange VSS Writer preparation.
ESE Event ID 224 – Logs are now purged

MSExchangeIS Event ID 9780 – Backup is now complete.

Final Note: although this example was tested against Exchange 2010, it should work just as fine with Exchange 2016/2013 & 2007

Until next time, Rob.