Intro

In this blog post, I will show the configuration and „caveats“ of the configuration to segment the CVM network traffic. In special routing the CVM Backplane traffic LAN via dedicated physical uplinks.(Physical isolation)

I covered this during a configuration in a high secure environment of one of my customers.

Each traffic type needed to be segmented on 2 physical network uplinks ports of the ESX Servers. This is a requirement resulting from BSI (https://www.bsi.bund.de/DE/Home/home_node.html). The public German institute for cyber security and IT security regulations. And the customer needed to align with those regulations. But those requirements may also apply to other countries and regions in the world.

Maybe some of you find this blogpost useful if you ever have to face this „challenge“ 😉

Software levels during time of writing:

Reference Documents

The network segmentation feature of NUTANIX is primarily documented in the Security Guide, because it is, of course, a security feature.

So this is a good entry point.

AOS 5.11.1.2 Release Notes:

https://portal.nutanix.com/#/page/docs/details?targetId=Release-Notes-Acr-v5_11_1_2:Release-Notes-Acr-v5_11_1_2

AOS 5.11.x Security Guide:

https://portal.nutanix.com/#/page/docs/details?targetId=Nutanix-Security-Guide-v511:wc-network-segment-on-existing-cluster-backplane-physically-isolate-wc-t.html

AOS 5.11.x PRISM Element Web Console Guide:

https://portal.nutanix.com/#/page/docs/details?targetId=Web-Console-Guide-Prism-v511:Web-Console-Guide-Prism-v511

Overview to the environment

The customer has implemented 2 clusters. 1 management and 1 production cluster. All nodes based on DELL-XC 740xd models. Each node had 10 physical network ports. The management cluster hosts all management instances. F.e. vCenter and PRISM Central. And also a XRAY Test VM which we used as a validation and test environment.

The second cluster only hosts production VMs. At time of testing the production cluster was empty and no workloads where running.

The following figure shows a high-level logical architecture.

As you can see the CVM backplane LAN and the CVM management LAN are on separate physical uplinks. In addition, there are 2 different physical switch fabrics. The CVM Backplane network is also physically isolated and non-routed.

The following networks were available on the clusters:

(Only list relevant networks here which are necessary to configure network segmentation)

The ESX Hosts vmk0 interface is homed in „CVM management“. During configuration of the network, segmentation feature a new vmk interface (vmk2) is created on all ESX hosts. The vmk2 interface is homed in „CVM management“.

Same thing with CVM interfaces:

eth0=CVM management

eth2=CVM Backplane.

Special things to know about network segmentation

The following guidelines are not documented in the official documentation (AOS 5.11 documents). These are my personal experiences during configuration.

To configure the physical network segmentation feature the CVM management port groups AND the CVM Backplane portgroups need to be on the SAME virtual switch type. In my example, we used DV Switches.

Pro Tipp: DV Switch support is only available from AOS 5.11.1 and newer.

The eth2 interface of the CVMs need to be in an „un-configured“ state.

If you run a AOS release that is not compatible with the network segmentation feature, manually remove the eth2 interface and then update to a release which is compatible. After the upgrade a new eth2 interface will be generated.

Ports 443 and 80 need to be opened between vCenter and all CVMs as well as the clusters virtual IP . PRISM uses Port 80 to query the vCenter Server about virtual switch information.

Reference KB: https://portal.nutanix.com/#/page/kbs/detailstargetId=kA032000000TVQACA4

Once backend network is configured stargate and Cassandra only communicate over backend, however there is not (yet) “degraded node detection” over backend network

Preparations

Preparing and configuring the DVSwitches (VMware)

We created 2 new DV Switches. 1 for the management network and 1 for backplane network. The management network was configured with an active/passive load-balancing scheme. The backplane network was configured with an LACP link-aggregation algorithm.

You need to talk to your network people how you should configure the virtual switches so that they align with physical network configurations. Involve them in the whole configuration process.-

After all is set you can start the configuration in the following order:

1. Create 2 new DV Switches. Assign 2 physical uplinks per switch.
2. Dvs_management
3. Dvs_backplane
4. Configure LACP on the 2^nd DV Switch (dvs_backplane) only
5. Create 2 port groups per switch. Name them according to your naming scheme.
6. In our case this looks as follows:

dvs_management:

PG: Management_cvm

PG: Management_host

Important: As we use a native VLAN on the management switch we don’t need to tag a VLAN on the PG’s.

The PG’s use 2 Uplinks and are configured as 1 active uplink and 1 standby uplink

Dvs_backplane:

PG: Cvm-backplane_cvm

PG: Cvm-backplane_host

Important: The PG’s on the backplane switch need to be tagged with the correct VLAN AND the LAG Interface needs to be assigned as the ONLY uplink !

1. After switches and PGs are configured, migrate the management interfaces (vmk0 and CVM-eth0) off the standard switches to the new DV Switch. In a NUTANIX standard configuration the management and the backplane network is on the same virtual standard switch.
2. Important: It is good practice here to have the VMware vCenter on a separate VLAN. So that you don’t need to migrate both the vmk0 AND the vcenter.
   
3. If you do this you can loose all management functionality. And need to roll back the configuration which might become really ugly ;).
   

Preparations NUTANIX

The following preparations need to be done on the NUTANIX (PRISM) site:

1. Check DNS resolution of PRISM to the vCenter.
2. Check DNS resolution from the vCenter to the ESX Hosts and optional to the Clutser virtual IP.
3. In my case I configured the cluster from a workstation which was separated from the Management Domain and isolated from the management network.
   
4. So the DNS resolution did not work from my workstation to the vCenter and other entities. So I needed to configure my local „hosts“ file to have DNS J.
   
5. Register vCenter in PRISM

Configuring Backplane Network (PRISM)

Login to PRISM Element

Click on „Settings“

Then choose „Network configuration“ on the left

Click on „Configure“ in the first row of the table on the right.

Backplane LAN Configuration Dialoque opens:

In the „Host Port Group“ choose the Host PG of the Backend vSwitch we created earlier.

In the „CVM Port Group“ Fiels choose the CVM PG of the backend vswitch we created earlier.

Then enter appropriate Subnet and VLAN information.

The PG’s of the Backplane DV Switch are already tagged with a VLAN ID. So we don’t need to enter a VLAN ID here !

Finally click on „Verify and save“

The configuration starts and PRISM tells you the configuration is initiating.

The process now assigns IP addresses from the subnet to each host and each CVM. It normally starts from the first IP Address in the defined Range.

You can monitor the progress in the Task Log:

The Configuration takes about 5-15 minutes per CVM including a re-boot of each CVM.

If the configuration completes successfully, you can see the following log entries:

Check ESX Network configuration after successful configuration of the backplane network

ESX Host vmk Interface.

Check the vmk interfaces of each Host. There should be a vmk2 adapter present. If you already have more vmks configured the backend adapter may have another vmk number.

CVM Port Group + IP Adressen:

Now check the port groups and the IP addresses of each CVM. As you can see, the CVM should have 3 network adapters configured. Adapter 1 is management Adapter 3 is backplane.

The following IP addresses should be configured:

Check network settings in PRISM after successful configuration

Navigate to Settings –> Network configuration -> Backplane network.

There you can see all hosts and their associated IPs for VMK and CVM.

Validating the configuration with X-RAY

After successful configuration, you can use X-Ray to validate the network segmentation. Always use Tip: the newest X-Ray version!

I used version 3.6.0

Initial setup of X-RAY:

You must complete all of the following steps before you can start testing:

1 X-Ray deployment

https://portal.nutanix.com/#/page/docs/details?targetId=X-Ray-Guide-v36:v36-xray-install-vcenter-t.html

2 Configure X-Ray VM network settings

https://portal.nutanix.com/#/page/docs/details?targetId=X-Ray-Guide-v36:v36-xray-network-config-t.html

3 First connection to X-Ray UI:

https://portal.nutanix.com/#/page/docs/details?targetId=X-Ray-Guide-v36:v36-xray-vm-login-t.html

4 Setup a DHCP server in the network you want to run your test VMs in:

This could be accomplished by placing small windows VM into the test network where X-Ray runs and deploy the windows DHCP Server Feature with an appropriate IP scope/range.

Zero configuration networking often does not work in secure environments if networks are routed f.e.

X-Ray Dashboard (First log-in)

You first need to configure a test target:

Follow this procedure.

https://portal.nutanix.com/#/page/docs/details?targetId=X-Ray-Guide-v36:v36-xray-target-create-t.html

After creating a test target run the validation procedure. The validation procedure needs to complete successfully. Otherwise, you cannot run ANY tests! Troubleshoot any problem, which comes up during validation first.

After validation runs without any error you can start your first validation tests

In my case I started to run the „Four corners micro benchmark“

Tests -> Four corners micro benchmark

After you started the test routine the following should be monitored from vCenter:

Network load of the backplane network

Network load of the management network

Monitor the network interfaces throug VMware vCenter

In vCenter select any ESX Host of the cluster you run your tests on.

Then navigate to:

Monitor-> Performance -> Additional settings

In the diagram options select the vmnics of the backplane network :

Then you can see that some traffic passes the uplink adapters of the „Backplane vSwitch“.in this example these uplinks were vmnic0 and vmnic8

Check the load also on the management adapters. There should be no or marginal traffic passing there during testing.

Conclusion

The NUTANIX network segmentation feature is a good fit if you want to:

• Logically or physically, segregate network traffic types from each other
• Address high security network requirements.
• Assign dedicated network bandwidth to CVM Backplane Network. For storage operations.
• Address different network failure scenarios. I.e. if a physical switch fabric fails you do not lose management AND storage network.

The NUTANIX Platform incorporates many new features with AOS 5.11.x. Especially adressing security requirements demanded by various users and customers.