|
Hello everyone,
First of all, my experience with storages (and especially HPE 3PAR) is very limited, so please bear with me.
Recently we migrated our Cluster storage from old HP P2000 G3 (600 Gb 15K SAS HDDs) to new HPE 3PAR 8200 (3.8 Tb SSDs).
Immediately after migration we have noticed an unusual problem – when we try to backup even 1 Virtual Machine, we experience:
1. Massive freezes and timeouts of multiple VMs at the same time (not only the one being backed up, but another VMs on the host as well);
2. Often freezes and limited responsiveness of host itself during the backup procedure;
3. The freezes last for 5 – 10 minutes, and after it the backup fails with “Transient VSS error 0x800423F3”.
During the freezes and timeouts there is no any unexpected high CPU or HDD load on the host. If anything, the disk activity even goes down.
Short infrastructure description:
1. Windows Server 2012 Datacenter cluster running Hyper-V role;
2. Consists of 3x nodes (HP Proliant DL380p Gen8);
3. Each node is connected to the SAN switches with 2x 8G FC (1x to every SAN switch);
4. HPE 3PAR 8200 is connected to the same SAN switches with 4x 16G FC (2x to every switch);
a. Port (N:S:P) Type WWN Rate Connected Devices Switch Port Switch
b. 0:0:1 Host <> 16 Gbps 3 hosts 1 FCSAN01
c. 0:0:2 Host <> 16 Gbps 3 hosts 1 FCSAN02
d. 1:0:1 Host <> 16 Gbps 3 hosts 0 FCSAN01
e. 1:0:2 Host <> 16 Gbps 3 hosts 0 FCSAN02
5. CPG is configured with RAID 5, 7+1;
6. There are 3 volumes for VMs
a. 1: Dedup and Compression enabled;
b. 2 and 3: Compression enabled;
7. All volumes are published to all 3 hosts with MPIO enabled
a. “Round robin”,
b. 4 paths,
c. All 4 are in “Active / Optimized” state;
8. Latest available Windows updates, drivers and firmware installed;
9. 3PAR runs OS ver. 3.3.1.269 (MU1)+P11,P15,P18,P19,P21,P25;
10. Backup solution: MS SCDPM 2012 R2;
The problem with freezes manifests itself only during backups -- there are no problems with normal day-to-day VM operation. No lock-ups, no crashes, no data corruption, no nothing out of ordinary.
As we currently understand, the problem most likely happens during the Shadow Copy creation. The problem appears only when we try to backup VMs from HPE 3PAR storage. The backups for the VMs that are still on old HP P2000 run fine, and as soon as we migrate problematic VMs back to P2000 from 3PAR, the problem disappears and we can backup them without any freezes. If we again move them to 3PAR – the problem returns.
Our attempt to contact HPE support ended with support people telling us that the HPE 3PAR 8200 + Server 2012 cluster + SCDPM combination is not tested for compatibility, so it might not work and we should go pester MS (not exactly a solution).
Right now we have a case with Microsoft, trying to find out if something is wrong with MS Software VSS provider. So far, no luck. Honestly, i’m not so sure SW VSS provider is to blame, as it works fine with VMs on HP P2000.
Unfortunately, from what we’ve read in the 3PAR documentation, 3PAR Hardware VSS provider does not work with MS Windows cluster + SCDPM 2012 R2. We tried it anyway, and ended up deadlocking one of CSVs, crashing all the VMs on that. So, that solution is out of the question.
Perhaps someone here has some experience with similar issue and can give some recommendations what to try? Because frankly we ran out of ideas where to look.
Thank you.
|
