VNX FILE REPLICATION Part 1 and 2

Source: VNX File Replication part 1 and 2 | FVTOOL

//

VNX File Replication part 1/2 | FVTOOL

I was putting together a configuration for one of my customers. This were his requirements

  1. Replicate these filesystems and keep 31 daily snapshots (if
    possible). The file systems were deduplicated at the source side.
  2. Automatically delete daily snapshots 32 and older.
  3. He had nested folders with thousands of links under them.
  4. He needed a NFS target for his NFS target for his non-VNX data backups. This filesystem (NFS targets) did not need any snapshots as the backup software takes care of multiple copies.

I decided to put the configuration together in my Demo Lab and generate the scripts I would be using to create replication sessions for my customer. Now, I only have one VNX so I had to figure out how to do replication. I had the choice of putting together a VNX simulator that is pretty much the full blown VNX File(Celerra) running in a VM. The second choice I had was to do replication between the two data movers in my VNX. The VNX comes with two Data Movers, one one active and one standby.

To do this, I had to convert my standby data mover into an active data mover. You can do this in the Unisphere but since I did not want to reduce the number of screenshots in this blog I will show you how to do it in CLI. SSH over to the VNX file. To promote the standby DM into an active DM, you need to do two things. The Data Movers have the names server_2 for Data Mover 2 and server_3 for Data Mover 3

  1. Deselect server_3 as the standby data mover
  2. Change the role of server_3 from an active to a active data mover.
[nasadmin@hq-vnx-cs0 ~]$ server_standby server_2 -delete mover=server_3
server_2 : done
[nasadmin@hq-vnx-cs0 ~]$ server_setup server_3 -type nas
server_3 : reboot in progress 0.0.0.0.0.0.0.0.0.0.0.1.1.1.1.3.4.4. done 
server_3 : checking root filesystem ... done 
server_3 : verifying setup for Unicode fonts ... done 
server_3 : updating local and remote systems configuration ...
hq-vnx-cs0 : operation in progress (not interruptible)...
id         = 0
name       = hq-vnx-cs0
owner      = 0
device     = 
channel    = 
net_path   = 172.17.101.153
celerra_id = APM001211039272007
server_3 : configuring snmpd user...
 done 
done

I wanted to mirror his environment that Production volumes that have checkpoints and so I created a checkpoint schedule

[nasadmin@hq-vnx-cs0 ~]$ nas_fs -list
id      inuse type acl   volume    name                server
1         n    1   0     10        root_fs_1           
2         y    1   0     40        root_fs_common      1,2
3         n    5   0     73        root_fs_ufslog      
5         n    5   0     93        root_fs_d3          
6         n    5   0     94        root_fs_d4          
7         n    5   0     95        root_fs_d5          
8         n    5   0     96        root_fs_d6          
9         y    1   0     12        root_fs_2           1
13        y    1   0     14        root_fs_3           2
17        n    5   0     108       delete_me           
20        y    1   0     115       hq-vnx-vCD-NFS-IMAG 1
21        y    1   0     119       DATA                1
23        n    5   0     121       root_panic_reserve

Pulling information on the file system /DATA i got the following.

I created a daily checkpoint schedule

[nasadmin@hq-vnx-cs0 ~]$ nas_ckpt_schedule -create DATA_FS_CKPT_SCHED -filesystem DATA -recurrence once -start_on 2013-01-08 -runtimes 15:59 -cvfsname_prefix morning
Error 13422428162: User is not allowed to perform this operation due to the snapsure license not being enabled.
[nasadmin@hq-vnx-cs0 ~]$

In Navisphere I noticed that in Unisphere Create File System Checkpoint was greyed out. I needed that enabled.

Screen Shot 2013 01 08 at 3 23 37 PM

It seemed that I needed to enable the Snapsure license, which I did and also enabled the replicatorv2 license that I would be using in the configuration.

[nasadmin@hq-vnx-cs0 ~]$ nas_license -list
key                 status    value
site_key            online    4f 69 1f 36
cifs                online    
nfs                 online    
[nasadmin@hq-vnx-cs0 ~]$ nas_license -create snapsure
done
[nasadmin@hq-vnx-cs0 ~]$ nas_license -list
key                 status    value
site_key            online    4f 69 1f 36
cifs                online    
nfs                 online    
snapsure            online    
[nasadmin@hq-vnx-cs0 ~]$ nas_license -create replicatorV2 
done
[nasadmin@hq-vnx-cs0 ~]$ nas_license -list
key                 status    value
site_key            online    4f 69 1f 36
cifs                online    
nfs                 online    
snapsure            online    
replicatorV2        online    
[nasadmin@hq-vnx-cs0 ~]$ 

I ran my schedule again and i got an OK

[nasadmin@hq-vnx-cs0 ~]$ nas_ckpt_schedule -create DATA_FS_CKPT_SCHED -filesystem DATA -recurrence once -start_on 2013-01-08 -runtimes 16:10 -cvfsname_prefix afternoon
OK
[nasadmin@hq-vnx-cs0 ~]$ nas_ckpt_schedule -list
   Id Name                 State      Next Run                       Description
    2 DATA_FS_CKPT_SCHED   Pending    Tue Jan 08 16:10:00 PST 2013  

With this complete I could not setup replication between the two DataMovers. I was going to replicate the filesystem /DATA

First I checked the DataMover Interconnect over the loopback device existed. This is enabled by default. In the customers production system, I would be doing a 1:1 mapping of Prod Data Mover to DR Data Mover.

nas_cel -interconnect -list
id     name               source_server   destination_system   destination_server
20001  loopback           server_2        hq-vnx-cs0           server_2
30001  loopback           server_3        hq-vnx-cs0           server_3

I validated that the interconnects were ok by running the following command.

[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -validate id=20001
loopback: has 1 source and 1 destination interface(s); validating - please wait...ok
[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -validate id=30001
loopback: has 1 source and 1 destination interface(s); validating - please wait...ok
[nasadmin@hq-vnx-cs0 ~]$ 

Continued in Part 2

In Part 1, I verified that the loopback interface was available. I did not want to create replication using the loopback interface that would replicate to the same DataMover DM2–>DM2, but I wanted to put replication from DM2->DM3. I had to configure the IP interfaces on DM3 to get them up to receive the data coming in.

 ____________       ____________
|	     |     |	     	|		
|    DM2     | --> |	 DM3 	|
|____________|     |____________| 

To create the interfaces in DM3, I first created a I first created a FailSafe Network interface on DataMover 3. A FailSafe interface allows failover from one interface to another. It does not have any load balancing between the two interfaces. Data flows through one interface until it fails.

Create Network Device

I then created a virtual interface NAS-IP2 on the fail safe interface called FSN-1.

Create Network Interface

To create an interconnect you first have to establish a trust relationship between the Control stations, but since I was using DMs in one VNX, I did not have to create the trust relationship. To create the interconnect, I first checked the name of the cell manager.

[nasadmin@hq-vnx-cs0 ~]$ nas_cel -l
id    name          owner mount_dev  channel    net_path                                      CMU
0     hq-vnx-cs0    0                           172.17.101.153                                APM001211039272007

I used that name in the command to below to create the interconnect.



[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -create InterConn-DM2--DM3 -source_server server_2 -destination_system hq-vnx-cs0 -destination_server server_3 -source_interfaces ip=172.17.105.90 -destination_interfaces ip=172.17.105.91
operation in progress (not interruptible)...
id                                 = 20003
name                               = InterConn-DM2--DM3
source_server                      = server_2
source_interfaces                  = 172.17.105.90
destination_system                 = hq-vnx-cs0
destination_server                 = server_3
destination_interfaces             = 172.17.105.91
bandwidth schedule                 = uses available bandwidth
crc enabled                        = yes
number of configured replications  = 0
number of replications in transfer = 0
status                             = The interconnect is OK.

For the interconnect to work properly, it requires a peer interface that connects back from the destination DataMover to the source DataMover. I used this commands to create the peer interconnect to enable replication from DM3 to DM2.

[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -create InterConn-DM3--DM2 -source_server server_3 -destination_system hq-vnx-cs0 -destination_server server_2 -source_interfaces ip=172.17.105.91 -destination_interfaces ip=172.17.105.90
operation in progress (not interruptible)...
id                                 = 30003
name                               = InterConn-DM3--DM2
source_server                      = server_3
source_interfaces                  = 172.17.105.91
destination_system                 = hq-vnx-cs0
destination_server                 = server_2
destination_interfaces             = 172.17.105.90
bandwidth schedule                 = uses available bandwidth
crc enabled                        = yes
number of configured replications  = 0
number of replications in transfer = 0
status                             = The interconnect is OK.

I verified the interconnect.

[[nasadmin@hq-vnx-cs0 ~]$  nas_cel -interconnect -l
id     name               source_server   destination_system   destination_server
20001  loopback           server_2        hq-vnx-cs0           server_2
20003  InterConn-DM2--DM3 server_2        hq-vnx-cs0           server_3
30001  loopback           server_3        hq-vnx-cs0           server_3
30003  InterConn-DM3--DM2 server_3        hq-vnx-cs0           server_2
[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -validate id=20003
InterConn-DM2--DM3: has 1 source and 1 destination interface(s); validating - please wait...ok
[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -validate id=30003
InterConn-DM3--DM2: has 1 source and 1 destination interface(s); validating - please wait...ok

If you do not create a peer interface and instead just have one interconnect replication in one direction, you will get an error that looks similar to this when you try and validate

[nasadmin@hq-vnx-cs0 ~]$ nas_cel -interconnect -validate id=20003
InterConn-DM2--DM3: has 1 source and 1 destination interface(s); validating - please wait...failed

I now had a new interconnects shown in RED. I needed now to create a storage pool to hold my replicated file systems. I would name that DATA_REP. To create a Storage pool, you have to do it in two steps. Provision LUNs from the VNX Fibre channel side to the Data Movers. Once the Volumes are available to the DataMover, you can now create the Storage Pool.

I started by first two LUNs called DATA-03-FILE and DATA-04-FILE.

Create LUNs

I then added the two LUNs to the storage group ~filestorage. This is the LUN masking to the DataMover.

Add to storage group

I checked the devices that had been presented to the VNX file, I only had disk d8 and d9 currently presented.

[nasadmin@hq-vnx-cs0 ~]$  nas_disk -list
id   inuse  sizeMB    storageID-devID   type  name          servers
1     y      11260  APM00121103927-2007 CLSTD root_disk     1,2
2     y      11260  APM00121103927-2008 CLSTD root_ldisk    1,2
3     y       2038  APM00121103927-2009 CLSTD d3            1,2
4     y       2038  APM00121103927-200A CLSTD d4            1,2
5     y       2044  APM00121103927-200B CLSTD d5            1,2
6     y      65526  APM00121103927-200C CLSTD d6            1,2
7     y       1023  APM00121103927-0000 MIXED d7            
8     y      20479  APM00121103927-000F MIXED d8            1,2
9     y     102399  APM00121103927-0010 MIXED d9            1,2

I then ran the following command to have the DataMovers to rescan their HBAs to discover and save all SCSI devices.

[nasadmin@hq-vnx-cs0 ~]$ server_devconfig server_2 -create -scsi -all

Discovering storage on hq-vnx-cs0 (may take several minutes)
server_2 : done

Info 26306752254: server_2 : APM00121103927 reassigned LUN 0017 in storage group '~filestorage' from host id 0006 to 0019

Info 26306752254: server_2 : APM00121103927 reassigned LUN 0018 in storage group '~filestorage' from host id 0007 to 0018

Warning 17717198868: server_2 : The FAST-POOL-0 on APM00121103927 storage pool contains disk volumes that use different data service configurations. Creating and extending file systems using this storage pool may cause performance inconsistency.
[nasadmin@hq-vnx-cs0 ~]$ server_devconfig server_3 -create -scsi -all

Discovering storage on hq-vnx-cs0 (may take several minutes)
server_3 : done

On rechecking the disks available to the data movers, I now saw that I now had two disks d10 and d11 of size 60GB each.

[nasadmin@hq-vnx-cs0 ~]$ nas_disk -list
id   inuse  sizeMB    storageID-devID   type  name          servers
1     y      11260  APM00121103927-2007 CLSTD root_disk     1,2
2     y      11260  APM00121103927-2008 CLSTD root_ldisk    1,2
3     y       2038  APM00121103927-2009 CLSTD d3            1,2
4     y       2038  APM00121103927-200A CLSTD d4            1,2
5     y       2044  APM00121103927-200B CLSTD d5            1,2
6     y      65526  APM00121103927-200C CLSTD d6            1,2
7     y       1023  APM00121103927-0000 MIXED d7            
8     y      20479  APM00121103927-000F MIXED d8            1,2
9     y     102399  APM00121103927-0010 MIXED d9            1,2
10    n      61439  APM00121103927-0012 MIXED d10           1,2
11    n      61439  APM00121103927-0011 MIXED d11           1,2

I then created a Storage Pool to hold the replicated file system

[nasadmin@hq-vnx-cs0 ~]$ nas_pool -create -name DATA_REP -description 'Replication Destination' -volumes d10,d11 -default_slice_flag y
id                   = 45
name                 = DATA_REP
description          = Replication Destination
acl                  = 0
in_use               = False
clients              = 
members              = d10,d11
storage_system(s)    = APM00121103927
default_slice_flag   = True
is_user_defined      = True
thin                 = False
tiering_policy       = Auto-Tier/Optimize Pool
compressed           = False
mirrored             = False
disk_type            = Mixed
server_visibility    = server_2,server_3
is_greedy            = False
template_pool        = N/A
num_stripe_members   = N/A
stripe_size          = N/A

I now had space to send my replicated data. It was a storage pool DATA_REP. I now started the replication using the following commands.

[nasadmin@hq-vnx-cs0 ~]$ nas_replicate -create rep1 -source -fs DATA -destination -pool DATA_REP -interconnect InterConn-DM2--DM3 -max_time_out_of_sync 15
OK

And verified that replication was ok using the following commands

[nasadmin@hq-vnx-cs0 ~]$ nas_replicate -list
Name                      Type       Local Mover               Interconnect         Celerra      Status
rep1                      filesystem server_2-->server_3       InterConn-DM2--D+ hq-vnx-cs0   OK

[nasadmin@hq-vnx-cs0 ~]$ nas_replicate -info rep1
ID                             = 119_APM00121103927_2007_129_APM00121103927_2007
Name                           = rep1
Source Status                  = OK
Network Status                 = OK
Destination Status             = OK
Last Sync Time                 = 
Type                           = filesystem
Celerra Network Server         = hq-vnx-cs0
Dart Interconnect              = InterConn-DM2--DM3
Peer Dart Interconnect         = InterConn-DM3--DM2
Replication Role               = local
Source Filesystem              = DATA
Source Data Mover              = server_2
Source Interface               = 172.17.105.90
Source Control Port            = 0
Source Current Data Port       = 50325
Destination Filesystem         = DATA_replica1
Destination Data Mover         = server_3
Destination Interface          = 172.17.105.91
Destination Control Port       = 5085
Destination Data Port          = 8888
Max Out of Sync Time (minutes) = 15
Current Transfer Size (KB)     = 8461480
Current Transfer Remain (KB)   = 8173480
Estimated Completion Time      = Wed Jan 09 19:16:41 PST 2013
Current Transfer is Full Copy  = Yes
Current Transfer Rate (KB/s)   = 6022
Current Read Rate (KB/s)       = 3407
Current Write Rate (KB/s)      = 320
Previous Transfer Rate (KB/s)  = 0

As per the customer requirements, I setup daily recurring checkpoints on the source file system that would take place every day at 7:45pm keeping the last 20.

[nasadmin@hq-vnx-cs0 ~]$ nas_ckpt_schedule -create DATA_daily -filesystem DATA -recurrence daily -every 1 -start_on 2013-1-9 -end_on 2013-1-9 -runtimes 19:45 -keep 20 -cvfsname_prefix evening
OK

Running the file system list. I could now see that I had the primary file system checkpoint and the destination file system. The destination file system name was DATA_replica1.

[nasadmin@hq-vnx-cs0 ~]$ nas_fs -l
id      inuse type acl   volume    name                server
1         n    1   0     10        root_fs_1           
2         y    1   0     40        root_fs_common      1,2
3         n    5   0     73        root_fs_ufslog      
5         n    5   0     93        root_fs_d3          
6         n    5   0     94        root_fs_d4          
7         n    5   0     95        root_fs_d5          
8         n    5   0     96        root_fs_d6          
9         y    1   0     12        root_fs_2           1
13        y    1   0     14        root_fs_3           2
17        n    5   0     108       delete_me           
20        y    1   0     115       hq-vnx-vCD-NFS-IMAG 1
21        y    1   0     119       DATA                1
23        n    5   0     121       root_panic_reserve  
25        y    7   0     124       ckpt_DATA_FS_CKPT_S 1
34        y    1   0     129       DATA_replica1       2  <---- Destination File system
39        y    7   0     124       ckpt_DATA_daily_001 1  <-----Source file system checkpoint

I also created the checkpoints for the destination file system. And this is how the command looked like.

[nasadmin@hq-vnx-cs0 ~]$ nas_ckpt_schedule -create DATA_replica_daily -filesystem DATA_replica1 -recurrence daily -every 1 -start_on 2013-1-9 -end_on 2013-1-9 -runtimes 19:55 -keep 20 -cvfsname_prefix evening
OK

And this is how the file systems looked like with the Destination file system checkpoint. I would suggest that you take the checkpoint after the file system completed replicating so that you do not use up so much of the SavVol.

[nasadmin@hq-vnx-cs0 ~]$ nas_fs -l
id      inuse type acl   volume    name                server
1         n    1   0     10        root_fs_1           
2         y    1   0     40        root_fs_common      1,2
3         n    5   0     73        root_fs_ufslog      
5         n    5   0     93        root_fs_d3          
6         n    5   0     94        root_fs_d4          
7         n    5   0     95        root_fs_d5          
8         n    5   0     96        root_fs_d6          
9         y    1   0     12        root_fs_2           1
13        y    1   0     14        root_fs_3           2
17        n    5   0     108       delete_me           
20        y    1   0     115       hq-vnx-vCD-NFS-IMAG 1
21        y    1   0     119       DATA                1
23        n    5   0     121       root_panic_reserve  
25        y    7   0     124       ckpt_DATA_FS_CKPT_S 1
34        y    1   0     129       DATA_replica1       2
39        y    7   0     124       ckpt_DATA_daily_001 1
40        y    7   0     133       ckpt_DATA_replica_d 2

I hope this helps out someone out there.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s