EMC VNX – NEW SHUTDOWN OPTIONS

EMC VNX | David Ring//

A new feature with the release of VNX Rockies(Block OE 5.33 & File OE 8.1) was the ability to Shutdown the Entire Array using either a single command or via the ‘Power Off’ button in the Unisphere GUI. This feature is also available for first generation VNX storage system’s, from VNX OE code release 05.32.000.5.209 & 7.1.74.5 onwards.
These options are supported on Unified, Block and File systems.

Power Off via CLI
The new CLI option extends the nas_halt command to include a new switch to power down the entire system :
nas_halt –f –sp now
This will power off Control Stations, Data Movers and the Storage Processors.
usage: nas_halt [-f] [-sp] now
Perform a controlled halt of the Control Station(s) and Data Mover(s)
-f Force shutting down without prompt
-sp Shut down Storage Processors on unified platforms

Power Off via Unisphere
From Unisphere GUI navigate to the System List page:
SHUT1

Once you hit the ‘Power Off’ button a dialog box appears and from here you enter the array serial number in order to confirm shutdown:
SHUT2

Final Confirmation:
SHUT3

Note:
DAEs are not powered off.

EMC VNX – POOL LUN OWNERSHIP

EMC VNX | David Ring//

For every Pool LUN regardless of whether the LUN is thick or thinly provisioned there are 3 owner types assigned to each LUN:

Allocation Owner
Default Owner
Current Owner

There has been cases where inconsistencies in Current, Default & Allocation LUN ownership results in poor performance. Please review the EMC KB88169 written by Dave Reed for more detail.

Jon Klaus goes into good detail on the topic in his post ‘VNX Storage Pool LUN Ownership’

All three owner types need to align with the same ‘VNX Storage Processor’ (A/B) for best performance. If all 3 owners are aligned correctly then there is no need for the system to redirect I/O across the CMI to a peer Storage Processor. This is an example of a best practice LUN ownership configuration as viewable from the LUN properties in Unisphere:
VNX_LUN_OWN1

With VNX2 when you create multiple LUNs from Unisphere the ownership is evenly balanced between SP A/B, this results in a balanced LUN configration. With VNX1 you need to specify the owner under the advanced tab while creating the LUN(s) in Unisphere to keep the LUNs evenly balanced across Storage Processors, or you can create the LUNs using cli and specify the SP owner as follows:

naviseccli lun -create -type nonThin -poolName ‘PoolName’ -sp a -capacity 20 -sq gb -name ‘LunName_SPA’ -l 1

naviseccli lun -create -type nonThin -poolName ‘PoolName’ -sp b -capacity 20 -sq gb -name ‘LunName_SPB’ -l 2

To generate a report of all POOL LUN properties including Ownership values from Unisphere chose the system tab and click on the ‘Reports’ option:
VNX_LUN_OWN2
From the next window chose the reports window and select Pool LUNs:
VNX_LUN_OWN3
You will then be presented with a HTML report with details for each LUN:VNX_LUN_OWN4

You can also use the following Navicli command to generate a listing of all your LUNs Ownership values:
naviseccli -h SP_IP lun -list -alowner -default -owner

Example Output:
LOGICAL UNIT NUMBER 1
Name: LunName_SPA
Current Owner: SP A
Default Owner: SP A
Allocation Owner: SP A

LOGICAL UNIT NUMBER 2
Name: LunName_SPB
Current Owner: SP B
Default Owner: SP B
Allocation Owner: SP B

To lookup the ownership values on an individual LUN basis use the navicli command below or from Unisphere view the properties of the LUN (as per above):
naviseccli -h SP_IP lun -list -l 1 -alowner -default -owner
naviseccli -h SP_IP lun -list -l 2 -alowner -default -owner

Changing Ownership
Now we are going to look at how we can change the Pool LUN ownership types beginning with the Allocation owner. Changing the Allocation Owner is only recommended if you have an imbalance of SP LUN ownership in your system. The only way to achieve this (changing the Allocation Owner) is to create another LUN of equal characteristics but on the other Storage Processor and then migrate the source LUN to the newly created target LUN on the opposite SP as the following example demonstrates:

1. For example create a New pool LUN (target) to migrate the source LUN to:
naviseccli lun -create -type nonThin -poolName ‘PoolName’ -sp b -capacity 20 -sq gb -name ‘LunName_SPA’ -l 3

2. Migrate ‘LUN ID 1′ to the new LUN which now has an allocation owner of SP B (More on LUN migration HERE):
naviseccli -h SP_IP migrate -start -source 1 -dest 3 -rate asap

After the migration you can see that the Allocation Owner is now ‘SP B’ but the Current and Default Ownership’s remain with ‘SP A’:
naviseccli -h SP_IP lun -list -l 1 -alowner -default -owner

LOGICAL UNIT NUMBER 1
Current Owner: SP A
Default Owner: SP A
Allocation Owner: SP B

3. Changing the Default Owner using the LUN -modify command or change through the LUN properties window in Unisphere:
naviseccli -h SP_IP lun -modify -l 1 -sp B

4. To change the Current Owner you will need to execute a trespass command on the LUN using navicli or by right clicking on the LUN in Unisphere and click the trespass option:
naviseccli -h SP_IP trespass lun 1
If changing on multiple LUNs then running the trespass mine command from the SP will trespass all the LUNs that the SP has DEFAULT ownership of. For example to trespass LUNs with Default Ownership of ‘SP B’ but which are currently owned by ‘SP A’:
naviseccli -h SPB_IP trespass mine

After completing these 4 steps the LUN ID 1 now has Current and Default ownership to match the Allocation owner:
naviseccli -h SP_IP lun -list -l 1 -alowner -default -owner

LOGICAL UNIT NUMBER 1
Current Owner: SP B
Default Owner: SP B
Allocation Owner: SP B

Note: RAID Group LUNs do not have an allocation owner. With MCX Raid Group LUNs are truly symmetric active/active – Future enhancement for POOL based LUNs, until then please be aware of your LUN ownership values.

EMC VNX – FAST CACHE

EMC VNX – FAST CACHE | David Ring//

EMC ‘FAST Cache’ technology gives a performance enhancement to the VNX Storage Array by adding FLASH drives as a Secondary Cache, working hand-in-hand with DRAM Cache and enhancing overall Storage Array performance. EMC recommend firstly using available Flash Drives for FAST Cache and then adding Flash Drives as a tier to selected FAST VP pools as required. FAST Cache works with all VNX systems (And also CX4) and is activated by installing the required FAST Cache Enabler. FAST Cache works with traditional ‘Flare LUNs’ and ‘VP Pools’.

Note: FAST Cache is enabled at the Pool wide level and cannot be selective for specific LUNs within the Pool.

The initial configuration is quite simple, after adding the required quantity of drives you can create FAST Cache through the ‘System Properties’ console in Unisphere, which will enable FAST Cache for system wide use:
FC

Or you can use Navicli commands to create and monitor the initialization status of FAST Cache:
naviseccli -h SPA_IP cache -fast -create –disks disks -mode rw -rtype r_1
Check on the status of FAST Cache creation:
naviseccli -h SPA_IP cache -fast -info -status

If you wish to enable|disable FAST Cache on a specific VP_POOL:
naviseccli -h SPA_IP storagepool -modify -name “Pool_Name” -fastcache off|on
Check what Pools have FAST Cache enabled/disabled:
naviseccli -h SPA_IP storagepool -list -fastcache

If the FAST Cache configuration requires any modification then it first needs to be disabled. By disabling (destroying) FAST Cache all dirty blocks are flushed back to disk; once FAST Cache has completed disabling then you may re-create FAST Cache with your new configuration.

Configuration Options
FAST Cache configuration options range from 100GB on a CX4-120 to 4.2TB of FAST Cache on a VNX-8000.
CX4 Systems:
CX4-120 – 100GB
CX4-240 – 200GB
CX4-480 – 800GB
CX4-960 – 2000GB
VNX1 Systems:
VNX 5100 – 100GB
VNX 5300 – 500GB
VNX 5500 – 1000GB
VNX 5700 – 1500GB
VNX 7500 – 2100GB
VNX2 Systems:
VNX 5200 – 600GB
VNX 5400 – 1000GB
VNX 5600 – 2000GB
VNX 5800 – 3000GB
VNX 7600 – 4200GB
VNX 8000 – 4200GB

FAST Cache drives are configured as RAID-1 mirrors and it is good practice to balance the drives across all available back-end buses; this is due to the fact that FAST Cache drives are extremely I/O Intensive and placing more than the recommended maximum per Bus may cause I/O saturation on the Bus. Amount of FAST Cache drives per B/E Bus differs for each system but ideally for a CX/VNX1 system aim for no more than 4 drives per bus and 8 for a VNX2.

The order the drives are added into FAST Cache is the order in which they are bound, with the:
first drive being the first Primary;
the second drive being the first Secondary;
the third drive being the next Primary and so on…
You can view the internal private RAID_1 Groups of FAST Cache by running the following Navicli:
naviseccli -h SPA_IP getrg –EngineeringPassword

Note: Do not mix different drive capacity sizes for FAST Cache, either use all 100GB or all 200GB drive types. Also for VNX2 systems there are two types of SSD available:
FAST Cache SSDs are single-level cell (SLC) Flash drives that are targeted for use with FAST Cache. These drives are available in 100GB and 200GB capacities and can be used both as FAST Cache and as TIER-1 drives in a storage pool.
FAST VP SSDs are enterprise Multi-level cell (eMLC) drives that are targeted for use as TIER-1 drives in a storage pool (Not supported as ‘FAST Cache’ drives). They are available in three flavors 100GB, 200GB and 400GB.

Example ‘FAST Cache’ Drive Layout (VNX 8000)
The following image is an example of a VNX-8000 with the maximum allowed FAST Cache configuration (42 x 200GB FAST Cache SSDs). As you can see I have spread the FAST Cache drives as evenly possible across the BE Buses beginning with DAE0 in order to achieve the lowest latency possible:
VNX8000_FAST-Cache

FAST Cache Internals
FAST Cache is built on the ‘Unified LUN’ technology; thus the Data in FAST Cache is as secure as any other LUN in the CX/VNX array. FAST Cache is a nonvolatile storage that survives both power and SP failures and it does not have to re-warm after a power outage either.
There will be a certain amount of DRAM allocated during FAST Cache creation for the IO tracking of FAST Cache known as the ‘Memory Map’. This FAST Cache bitmap is directly proportional to the size of the FAST Cache. The memory allocation is something in the region of 1MB for every 1GB of FAST Cache created. So when FAST Cache is being enabled FLARE will attempt to grab approx 1/3rd the required memory from Read cache and 2/3rds from the Write cache and then re-adjusts the existing DRAM read and write caches accordingly.
With a compatible workload; FAST Cache increases performance by reducing the response time to hosts and provides higher throughput (IOPS) for busy areas that may be under pressure from drive or RAID limitations. Apart from ‘Storage Processors’ being able to cache read and write I/Os; the Storage Processors on the VNX also coalesce writes and pre-fetch reads to improve performance. However, these operations generally do not accelerate random read-heavy I/Os and this is where FAST Cache helps. FAST Cache monitors the storage processors’ I/O activity for blocks that are read or written to multiple times, with the third IO to any block within a 64K extent getting scheduled for promotion to FAST Cache, promotion is handled the same way as writing or reading an IO to a LUN. The migration process operates 24×7 using the ‘Least Recently Used algorithm’ in order to determine which data stays and which goes. The writes continue to be written to DRAM write cache but with FAST Cache enabled those writes are flushed to the Flash drives and so increasing flush speeds.
One important thing to note is that while performance of the VNX increases and IOPS figures increase with workload demands there will be an increase in the SP CPU utilization and this should be monitored. There are recommended guidelines on max throughput figures for particular arrays… more on this later.
It is important to know the type of workload on your LUN; as an example, log files are generally written and read sequentially across the whole LUN, in this scenario the LUN would not be a good candidate for FAST Cache as Flash drives are not necessarily better at serving large block sequential I/O when compared to spinning drives. Also Large block sequential I/O workloads are better served by having large quantities of drives, promoting this type of data to FAST Cache will normally result in the data being served by a lesser quantity of drives thus resulting in a performance reduction. Avoiding using FAST Cache on unsuitable LUNs will help to reduce the overhead of tracking I/O for promotion to FAST Cache.

Best Suited
Here I have listed conditions that you should factor when deciding if FAST Cache will be a good fit for your environment:
• VNX Storage Processor Utilization is under 70-percent
• There is evidence of regular forced Write Cache Flushing
• The majority I/O block size is under 64K (OLTP Transactions are typically 8K)
• The disk utilization of RAID Groups is consistently running above 60-70%
• Your workload is predominately random read I/O
• Your production LUNs have a high percentage of read cache misses
• Host response times are unacceptable

Useful EMC KB’s
Please see the following EMC KB articles for further reference (support.emc.com):
Please Note that some guidelines apply differently to Clar|VNX|VNX2.
KB 73184: Rules of thumb for configuring FAST Cache for best performance and highest availability
KB 14561: How to flush and disable FAST Cache
KB 15075: FAST Cache performance considerations
KB 82823: Should FAST Cache be configured across enclosures?
KB 15606: How to monitor FAST Cache performance using Unisphere Analyzer
KB 168348: [Q&A]VNX Fast Cache
KB 84362: Where do I find information about FAST Cache?

EMC VNX2 and VNX Future

Joe Chang : EMC VNX2 and VNX Future

EMC VNX2 and VNX Future

Update 2013-10: StorageReview on EMC Next Generation VNX
Update 2013-08: News reports that VNX2 will come out in Sep 2013

While going through the Flash Management Summit 2012 slide decks, I came across the session Flash Implications in Enterprise Storage Designs by Denis Vilfort of EMC, that provided information on performance of the CLARiiON, VNX, a VNX2 and VNX Future.

A common problem with SAN vendors is that it is almost impossible to find meaningful performance information on their storage systems. The typical practice is to cited some meaningless numbers like IOPS to cache or the combined IO bandwidth of the FC ports, conveying the impression of massive IO bandwidth, while actually guaranteeing nothing.

VNX (Original)

The original VNX was introduced in early 2011? The use of the new Intel Xeon 5600 (Westmere-EP) processors was progressive. The decision to employ only a single socket was not.

EMC VNX

EMC did provide the table below on their VNX mid-range systems in the document “VNX: Storage Technology High Bandwidth Application” (h8929) showing the maximum number of front-end FC and back-end SAS channels along with the IO bandwidths for several categories.

EMC VNX

It is actually unusual for a SAN storage vendor to provide such information, so good for EMC. Unfortunately, there is no detailed explanation of the IO patterns for each category.

Now obviously the maximum IO bandwidth can be reached in the maximum configuration, that is with all IO channels and all drive bays populated. There is also no question that maximum IO bandwidth requires all back-end IO ports populated and a sufficient number of front-end ports populated. (The VNX systems may support more front-end ports than necessary for configuration flexibility?)

However, it should not be necessary to employ the full set of hard disks to reach maximum IO bandwidth. This is because SAN systems are designed for capacity and IOPS. There are Microsoft Fast Track Data Warehouse version 3.0 and 4.0 documents for the EMC VNX 5300 or 5500 system. Unfortunately Microsoft has backed away from bare table scan tests of disk rates in favor of a composite metric. But it does seem to indicate that 30-50MB/s per disk is possible in the VNX.

What is needed is a document specifying the configuration strategy for high bandwidth specific to SQL Server. This includes the number and type of front-end ports, the number of back-end SAS buses, the number of disk array enclosures (DAE) on each SAS bus, the number of disks in each RAID group and other details for each significant VNX model. It is also necessary to configure the SQL Server database file layout to match the storage system structure, but that should be our responsibility as DBA.

It is of interest to note that the VNX FTDW reference architectures do not employ Fast Cache (flash caching) and (auto) tiered-storage. Both of these are an outright waste of money on DW systems and actually impedes performance. It does make good sense to employ a mix of 10K/15K HDD and SSD in the DW storage system, but we should use the SQL Server storage engine features (filegroups and partitioning) to place data accordingly.

A properly configured OLTP system should also employ separate HDD and SSD volumes, again using of filegroups and partitioning to place data correctly. The reason is that the database engine itself is a giant data cache, with perhaps as much as 1000GB of memory. What do we really expect to be in the 16-48GB SAN cache that is not in the 1TB database buffer cache? The IO from the database server is likely to be very misleading in terms of what data is important and whether it should be on SSD or HDD.

CLARiiON, VNX, VNX2, VNX Future Performance

Below are performance characteristics of EMC mid-range for CLARiiON, VNX, VNX2 and VNX Future. This is why I found the following diagrams highly interesting and noteworthy. Here, the CLARiiON bandwidth is cited as 3GB/s and the current VNX as 12GB/s (versus 10GB/s in the table above).

EMC VNX

I am puzzled that the VNX is only rated at 200K IOPS. That would correspond to 200 IOPS per disk and 1000 15K HDDs at low queue depth. I would expect there to be some capability to support short-stroke and high-queue depth to achieve greater than 200 IOPS per 15K disk. The CLARiiON CX4-960 supported 960 HDD. Yet the IOPS cited corresponds to the queue depth 1 performance of 200 IOPS x 200 HDD = 40K. Was there some internal issue in the CLARiiON. I do recall a CX3-40 generating 30K IOPS over 180 x 15K HDD.

A modern SAS controller can support 80K IOPS, so the VNX 7500 with 8 back-end SAS buses should handle more than 200K IOPS (HDD or SSD), perhaps as high as 640K? So is there some limitation in the VNX storage processor (SP), perhaps the inter-SP communication? or a limitation of write-cache which requires write to memory in both SP?

VNX2?

Below (I suppose) is the architecture of the new VNX2. (Perhaps VNX2 will come out in May with EMC World?) In addition to transitioning from Intel Xeon 5600 (Westmere) to E5-2600 series (Sandy Bridge EP), the diagram indicates that the new VNX2 will be dual-processor (socket) instead of single socket on the entire line of the original VNX. Considering that the 5500 and up are not entry systems, this was disappointing.

EMC VNX

VNX2 provides 5X increase in IOPS to 1M and 2.3X in IO bandwidth to 28GB/s. LSI mentions a FastPath option that dramatically increases IOPS capability of their RAID controllers from 80K to 140-150K IOPS. My understanding is that this is done by completely disabling the cache on the RAID controller. The resources to implement caching for large array of HDDs can actually impede IOPS performance, hence caching is even more degrading on an array of SSDs.

The bandwidth objective is also interesting. The 12GB/s IO bandwidth of the original VNX would require 15-16 FC ports at 8Gbps (700-800MBps per port) on the front-end. The VNX 7500 has a maximum of 32 FC ports, implying 8 quad-port FC HBAs, 4 per SP.

The 8 back-end SAS busses implies 4 dual-port SAS HBAs per SP? as each SAS bus requires 1 SAS port to each SP? This implies 8 HBAs per SP? Intel Xeon 5600 processor connects over QPI to a 5220 IOH with 32 PCI-E gen 2 lanes, supporting 4 x8 and 1×4 slots, plus a 1×4 Gen1 for other functions.

In addition, a link is needed for inter-SP communication. If one x8 PCI-E gen2 slot is used for this, then write bandwidth would be limited to 3.2GB/s (per SP?). A single socket should only be able to drive 1 IOH even though it is possible to connect 2. Perhaps the VNX 7500 is dual-socket?

An increase to 28GB/s could require 40 x8Gbps FC ports (if 700MB/s is the practical limit of 1 port). A 2-socket Xeon E5-2600 should be able to handle this easily, with 4 memory channels and 5 x8 PCI-E gen3 slots per socket.

VNX Future?

The future VNX is cited as 5M IOPS and 112GB/s. I assume this might involve the new NVM-express driver architecture supporting distributed queues and high parallelism. Perhaps both VNX2 and VNX Future are described is that the basic platform is ready but not all the components to support the full bandwidth?

EMC VNX

The 5M IOPS should be no problem with an array of SSDs, and the new NVM express architecture of course. But the 112GB/s bandwidth is curious. The number of FC ports, even at a future 16Gbit/s is too large to be practical. When the expensive storage systems will finally be able to do serious IO bandwidth, it will also be time to ditch FC and FCOE. Perhaps the VNX Future will support infini-band? The puprose of having extreme IO bandwidth capability is to be able to deliver all of it to a single database server on demand, not a little dribblet here and there. If not, then the database server should have its own storage system.

The bandwidth is also too high for even a dual-socket E5-2600. Each Xeon E5-2600 has 40 PCI-E gen3 lanes, enough for 5 x8 slots. The nominal bandwidth per PCIe G3 lane is 1GB/s, but the realizable bandwidth might be only 800MB/s per lane, or 6.4GB/s. A socket system in theory could drive 64GB/s. The storage system is comprised of 2 SP, each SP being a 2-socket E5-2600 system.

To support 112GB/s each SP must be able to simultaneously move 56GB/s on storage and 56GB/s on the host-side ports for a total of 112GB/s per SP. In addition, suppose the 112GB/s bandwidth for read, and that the write bandwidth is 56GB/s. Then it is also necessary to support 56GB/s over the inter-SP link to guarantee write-cache coherency (unless it has been decided that write caching flash on the SP is stupid).

Is it possible the VNX Future has more than 2 SP’s? Perhaps each SP is a 2-socket E5-4600 system, but the 2 SPs are linked via QPI? Basically this would be a 4-socket system, but running as 2 separate nodes, each node having its own OS image. Or that it is a 4-socket system? Later this year, Intel should be releasing an Ivy Bridge-EX, which might have more bandwidth? Personally I am inclined to prefer a multi-SP system over a 4-socket SP.

Never mind, I think Haswell-EP will have 64 PCIe gen4 lanes at 16GT/s. The is 2GB/s per lane raw, and 1.6GB/s per lane net, 12.8GB/s per x8 slot and 100GB/s per socket. I still think it would be a good trick if one SP could communicate with the other over QPI, instead of PCIe. Write caching SSD at the SP level is probably stupid if the flash controller is already doing this? Perhaps the SP memory should be used for SSD metadata? In any case, there should be coordination between what each component does.

Summary

It is good to know that EMC is finally getting serious about IO bandwidth. I was of the opinion that the reason Oracle got into the storage business was that they were tired of hearing complaints from customers resulting from bad IO performance on the multi-million dollar SAN.

My concern is that the SAN vendor field engineers have been so thoroughly indoctrinated in the SaaS concept that only capacity matters while having zero knowledge of bandwidth, that they are not be able to properly implement the IO bandwidth capability of the existing VNX, not to mention the even higher bandwidth in VNX2 and Future.

Updates will be kept on QDPMA Storage.

Published Monday, February 25, 2013 8:27 AM by jchang

MY TROUBLESHOOTING A HP FLEX 10 FCOE CONNECTION TO A CISCO FIBER CHANNEL MDS SWITCH

// <![CDATA[
(function() { var b=window,f="chrome",g="tick",k="jstiming";(function(){function d(a){this.t={};this.tick=function(a,d,c){var e=void 0!=c?c:(new Date).getTime();this.t[a]=[e,d];if(void 0==c)try{b.console.timeStamp("CSI/"+a)}catch(h){}};this[g]("start",null,a)}var a;b.performance&&(a=b.performance.timing);var n=a?new d(a.responseStart):new d;b.jstiming={Timer:d,load:n};if(a){var c=a.navigationStart,h=a.responseStart;0=c&&(b[k].srt=h-c)}if(a){var e=b[k].load;0=c&&(e[g](“_wtsrt”,void 0,c),e[g](“wtsrt_”,”_wtsrt”,h),e[g](“tbsd_”,”wtsrt_”))}try{a=null,
b[f]&&b[f].csi&&(a=Math.floor(b[f].csi().pageT),e&&0<c&&(e[g]("_tbnd",void 0,b[f].csi().startE),e[g]("tbnd_","_tbnd",c))),null==a&&b.gtbExternal&&(a=b.gtbExternal.pageT()),null==a&&b.external&&(a=b.external.pageT,e&&0=d&&b[k].load[g](“aft”)};var l=!1;function m(){l||(l=!0,b[k].load[g](“firstScrollTime”))}b.addEventListener?b.addEventListener(“scroll”,m,!1):b.attachEvent(“onscroll”,m);
})();
// ]]>
Source: Brian’s Virtually Useful Blog: VNX// <![CDATA[
var a="indexOf",b="&m=1",e="(^|&)m=",f="?",g="?m=1";function h(){var c=window.location.href,d=c.split(f);switch(d.length){case 1:return c+g;case 2:return 0//

First, some good links for documents about this:

Cisco, VMware, HP

The Problem:

Upon building a new HP chassis full of blades, hoping to connect my new blades to storage, I was build zoning rules connecting them to our EMC VNX, Inside of Cisco Fabric Manager I did not see the HP blades listed so that I could build zoning rules.  The other suspicious item was that inside of HP Virtual Connect Manager, inside of the “SAN Fabrics” tab, I was giving a warning for a status.

The Setup:

I had setup this up in a fully meshed fiber with NPIV enabled everywhere.  1 HP Chassis, two Flex 10 modules, two Cisco MDS switches, and one EMC VNX.  My flex 10 modules were each connected directly to both of the Cisco MDS Switches.   Each of the Cisco MDS’s were connected directly to each of the SP’s on the EMC VNX (fully meshed).

The Solution:

I worked on this for some time, then after changing the connections on the Flex 10’s to both go directly to the SAME Cisco MDS switch (removing the mesh from the Compute side), the HP Virtual Connect finally showed happy, and the Cisco MDS began to see all the HP Blades and I was able to connect storage.  So what did I do wrong originally?  I am sure there is a great reason, but what part of Fiber Channel for Dummies did I miss?

EMC VNX STORAGE POOL DESIGN & CONFIGURATION

// <![CDATA[
(function() { var b=window,f="chrome",g="tick",k="jstiming";(function(){function d(a){this.t={};this.tick=function(a,d,c){var e=void 0!=c?c:(new Date).getTime();this.t[a]=[e,d];if(void 0==c)try{b.console.timeStamp("CSI/"+a)}catch(h){}};this[g]("start",null,a)}var a;b.performance&&(a=b.performance.timing);var n=a?new d(a.responseStart):new d;b.jstiming={Timer:d,load:n};if(a){var c=a.navigationStart,h=a.responseStart;0=c&&(b[k].srt=h-c)}if(a){var e=b[k].load;0=c&&(e[g](“_wtsrt”,void 0,c),e[g](“wtsrt_”,”_wtsrt”,h),e[g](“tbsd_”,”wtsrt_”))}try{a=null,
b[f]&&b[f].csi&&(a=Math.floor(b[f].csi().pageT),e&&0<c&&(e[g]("_tbnd",void 0,b[f].csi().startE),e[g]("tbnd_","_tbnd",c))),null==a&&b.gtbExternal&&(a=b.gtbExternal.pageT()),null==a&&b.external&&(a=b.external.pageT,e&&0=d&&b[k].load[g](“aft”)};var l=!1;function m(){l||(l=!0,b[k].load[g](“firstScrollTime”))}b.addEventListener?b.addEventListener(“scroll”,m,!1):b.attachEvent(“onscroll”,m);
})();
// ]]>
Source: Brian’s Virtually Useful Blog: EMC VNX Storage Pool Design & Configuration// <![CDATA[
var a="indexOf",b="&m=1",e="(^|&)m=",f="?",g="?m=1";function h(){var c=window.location.href,d=c.split(f);switch(d.length){case 1:return c+g;case 2:return 0//

The EMC Whitepaper on VNX Best Practices h8268_VNX_Block_best_practices.pdf is the way to go, I used version 31.5 as it is the most current one available.

Storage Pools vs Raid Groups vs MetaLuns

From an design perspective, MetaLuns were basically replaced by Storage Pools, Storage pools allow for the large stripping across many drives that MetaLuns offers, but with a lot less complexity. MetaLuns are now generally used to combined/expand a traditional LUN.  Raid Groups have a maximum size of 16 disks, so for larger strips this isn’t a viable option.  For situations where guaranteed performance isn’t critical, go with Storage Pools, use Raid Groups if you need deterministic (guaranteed) performance.  The reason behind this is that you are probably going to create multiple LUNs out of your storage pool, so this could lead to one busy LUN affecting the others.

Raid Level Selection

Assuming your going with a Storage Pool, your options are Raid 5, 6, or 1/0.  If you are using large drives (over 1TB) then Raid 5 is not a good choices because of long rebuild times, Raid 6 is almost certainly the way to go.  Always use suggested drive numbers in the pools, Raid 5 is 5 disks, or a number that evenly divides by 5, Raid 6 & Raid 1/0 is 8 disks or a number that evenly divides by 8.  If you use less than the recommended you will be wasting space.

How Big of a Storage Pool do I start with?

Create a homogeneous storage pool with the largest number of practical drives.  Pools are designed for ease-of-use.  The pool dialog algorithmically implements many Best Practices.  It is better to start with a large pool, as you add disks to the pool, it does not (currently) restripe the disks, and therefore if you only added 5 disks to an existing 50 disk pool, the new LUNS would have much lower performance.   The smallest practical pool is 20 drives (four R5 raid groups).  It is recommended practice to segregate the storage system’s pool-based LUNs into two or more pools when availability or performance may benefit from separation.

I am only covering a small portion of what you need to know.

When dealing with storage, there are thousands of options, homogeneous drives vs. heterogeneous, Thick vs. Thin Provisioning, Fast VP Pools, Drive Speeds, Fast Cache and Flash Drives, Storage Tiering, the whitepaper above does a great of of detailing all of that, I won’t try to improve on what EMC has said.

EMC FAST: Whether to File and/or Block Tier

// // Source: Just Be Better… — EMC FAST: Whether to File and/or Block Tier// <![CDATA[
var customColor = '#ff6600';
var vimeoColor = '#ff6600';
var disableQuoteTitleFonts;
var disableHeaderNavFonts;
var slimAudioPlayer;

document.write(' .player, #page .video .video_embed, .photoset_narrow { display: none; } ‘);

// ]]>

Storage performance needs in today’s data center can change on a moment’s notice. Data that once needed the backing of a freight train today may only need the performance of a vespa tomorrow. Having the ability to react to the ever changing needs of one’s data in an automated fashion allows efficiencies never before seen in EMC’s midrange product line. Generally as data ages its importance lessens both from a business and usage perspective. Utilizing FAST allows EMC customers to place data on the appropriate storage tier based on application requirements and service levels. Choosing between cost (SATA/NL-SAS) and performance (EFD’s/SAS) is a thing of the past. Below are the what, when and why of EMC’s FAST. The intent is to help one make an informed decision based on the needs of their organization.

Block Tiering (What, When and Why)

The What: FAST for VNX/Clariion is an array based feature that utilizes Analyzer to move block based data (slices of LUNs). By capturing performance characteristics, it can intelligently make predictions on where that data will be best utilized. Data is moved at the sub LUN layer in 1G slices, eliminating the need and overhead with moving the full LUN. This could mean that portions of a LUN could exist on multiple disk types (FC, SATA , EFD) Migrations are seamless to the host and occur bidirectionally based on performance needs, ie. FC to SATA, FC to EFD, SATA to FC, SAS to NL-SAS, etc. FAST is utilized at the Storage Pool layer and not available within Traditional RAID Groups. To utilize FAST v2 (which is sub LUN tiering ) you must be at FLARE 30 or above (4.30.000.5.xxx), and have both Analyzer and FAST enabler installed on the array. Existing LUNs/Data can migrate seamlessly and non-disruptively into storage pools using the VNX/Clariion LUN migration feature. Additionally FAST operates with other Array based features such as Snapview, MirrorView, SAN Copy, RecoverPoint, etc, without issue. All FAST operations and scheduling is configurable through Unisphere.

The When: Automated tiering is a scheduled batch event and does not happen dynamically.

The Why: To better align your application service levels with the best storage type. Ease of management, as a requirement for FAST are storage pools. Storage pools allow for concise management and eased growth opportunities from one location. Individual RG and Meta LUNs management is not needed to obtain high end services levels with the use of SP’s and FAST. The idea going forth is to minimize disk purchasing requirements by moving hot and cold data to and fro disk types that meet specific service levels for that data. If data is accessed frequently then it makes sense that it lives on either EFD (enterprise FLASH drives) or FC/SAS. If data is not accessed frequently then it ideally should live on SATA/NL-SAS. By utilizing FAST in your environment, you are utilizing your Array in the most efficient manner while minimizing cap-ex costs.

File Tiering (What, When and Why)

The What: FAST for VNX File/Celerra utilizes the Cloud Tiering Appliance (or what was FMA, previously known as Rainfinity). The CTA utilizes a policy engine that allows movement of infrequently used files across different storage tiers based on last access times, modify times, size, filename, etc. As data is moved, the user perception is that the files still exist on primary storage. File retrieval (or recall) is initiated simply by clicking on the file, the file is then copied back to its original location. The appliance itself is available as a virtual appliance that can be imported into your existing VMware infrastructure via vCenter, or as a physical appliance (HW plus the software). Unlike FAST for VNX/CLARiiON, FAST for file allows you to tier across arrays (Celerra <-> VNX, Isilon or third party arrays) or cloud service providers (Atmos namely, other SP’s coming). The introduction of CTA to your environment is non-disruptive. All operations for CTA are configurable through the CTA GUI. In summary, CTA can be used as a Tiering engine, an archiving engine or a migration engine based on the requirements of your business. From an archiving perspective, CTA can utilize both Data Domain and Centera targets for long term enforced file level retention. As a migration engine, CTA can be utilized for permanent file moves from one array to another during technology refreshes or platform conversions. Note: CTA has no knowledge of the storage type, it simply moves files from one tier to another based on pre- defined criteria.

The When: Automated tiering is designed to running at scheduled intervals (in batches) and does not happen dynamically or continually I should say.

The Why: Unstructured data, data that exists outside of pre-defined data model such as SQL, is growing at an alarming rate. Think about how many word docs, excel spreadsheets, pictures, text files exist in your current NAS or general file-served environments. Out of that number what percentage hasn’t been touched since its initial creation? In that context, a fair assessment would be 50% of that data. A more accurate assessment would probably be 80% of your data. Archiving and Tiering via CTA simple allows for more efficient use of your high end and low end storage. If 80% of your data is not accessed or accessed infrequently it has no business being on fast spinning disk (FC or SAS). Ultimately this allows you to curb your future spending on pricey high end disk and focus more purchasing capacity for where your data should sit, on low end storage.

***Update***

As brought to my attention on the twitters (Thanks->@PBradz and @veverything), there is of course another option. Historically, data LUNs as used by the data movers for file specific data (CIFS, NFS) has only been supported on traditional RAID Group LUNs. With the introduction of the VNX, support has been extended to pool LUNs. This implies that you can utilize FAST block tiering for the data that encompasses those LUNs. A couple of things when designing and utilizing in this manner (more info here)…

  • The entire pool should be used by file only
  • Thick LUNs only within the pool
  • Same tiering policy for each pool LUN
  • Utilize compression and dedupe on the file side. Stay clear of block thin provisioning and compression.

There are of course numerous other recommendations that should be noted if you decide to go this route. Personally, its taken me a while to warm up to storage pools. Like any new technology it needs to gain my trust before I go all in on recommending it. Inherent bugs and inefficiencies early on have caused me to be somewhat cautious. Assuming you walk the line on how your pools are configured, this is a very viable means to file tier (so to speak) with the purchase of FAST block only. That being said there is still benefit to using the CTA for long term archiving primarily off array, as currently FAST Block is array bound only. Define the requirements up front so you’re not surprised on the backend as to what the technology can and can not do. If the partner you’re working with is worth their salt you’ll know all applicable options prior to that PO being cut…