storcli 的进阶使用
基础信息
查看磁盘管理基础信息
$ storcli64 show
Status Code = 0
Status = Success
Description = None
Number of Controllers = 1
Host Name = amd7302asus
Operating System = Linux4.15.0-55-generic
System Overview :
===============
-------------------------------------------------------------------------------------
Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth
-------------------------------------------------------------------------------------
0 LSIMegaRAIDSAS9270CV-8i 8 24 1 0 1 0 Msng On 1&2 Y 4 Opt
-------------------------------------------------------------------------------------
从上述结果分析，有一块型号为 LSIMegaRAIDSAS9270CV-8i 的 RAID 卡。它是 8 通道的，挂载了 24 块物理盘，有 1 个磁盘组（DG），有 1 个虚拟磁盘（VD），没有电池。
查看 RAID 卡信息
从上述信息看出，有 1 个 RAID 卡，那么可以通过 storcli /c0 show 查看其信息。
$ storcli64 /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
Controller = 0
Status = Success
Description = None
Product Name = LSI MegaRAID SAS 9270CV-8i
Serial Number = SV42234313
SAS Address = 500605b008f05ab0
PCI Address = 00:c1:00:00
TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR
-----------------------------------------------------------------------------
0 - - - - RAID0 Optl N 174.644 TB dflt N N none Y N
0 0 - - - RAID0 Optl N 174.644 TB dflt N N none Y N
0 0 0 16:1 18 DRIVE Onln N 7.276 TB dflt N N none - N
0 0 1 16:2 17 DRIVE Onln N 7.276 TB dflt N N none - N
0 0 2 16:3 20 DRIVE Onln N 7.276 TB dflt N N none - N
...
-----------------------------------------------------------------------------
Virtual Drives = 1
VD LIST :
=======
-------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
-------------------------------------------------------------
0/0 RAID0 Optl RW Yes RAWBC - ON 174.0 TB
-------------------------------------------------------------
Physical Drives = 24
PD LIST :
=======
-------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-------------------------------------------------------------------------------
16:1 18 Onln 0 7.276 TB SATA HDD N N 512B ST8000NM000A-2KE101 U -
16:2 17 Onln 0 7.276 TB SATA HDD N N 512B ST8000NM000A-2KE101 U -
16:3 20 Onln 0 7.276 TB SATA HDD N N 512B ST8000NM000A-2KE101 U -
...
-------------------------------------------------------------------------------
上述结果中，可以看出有 24 个物理设备，以及 enclosure id 号为16，硬盘大多是附在 enclosure 上不是直接附在 controller 上。
查看剩余空间
$ storcli64 /c0 show freespace
Controller = 0
Status = Success
Description = None
FREE SPACE DETAILS :
==================
-----------------------
ID DG AftrVD Size
-----------------------
1 0 0 660.75 GB
-----------------------
查看和设置 rebuildrate
$ storcli64 /c0 show rebuildrate
Controller = 0
Status = Success
Description = None
Controller Properties :
=====================
------------------
Ctrl_Prop Value
------------------
Rebuildrate 30%
------------------
$ storcli64 /c0 set rebuildrate=15
RAID 性能优化之 consistency check。
一致性检查是好事，但在进行一致性检查会影响性能。在没有强烈的需求下，可以降低一致性检查的策略，例如降低检查频率，减少性能影响。
查看 CC 信息。
$ storcli64 /c0 show cc
Controller = 0
Status = Success
Description = None
Controller Properties :
=====================
-----------------------------------------------
Ctrl_Prop Value
-----------------------------------------------
CC Operation Mode Concurrent
CC Execution Delay 168
CC Next Starttime 05/16/2020, 02:00:00
CC Current State Stopped
CC Number of iterations 267
CC Number of VD completed 0
CC Excluded VDs None
-----------------------------------------------
上述结果，可以分析，CC 的模式是并发，每隔 168 小时（即一周），下次执行时间为 2020-05-16 02:00:00。
$ storcli64 /c0 show ccrate
Controller = 0
Status = Success
Description = None
Controller Properties :
=====================
----------------
Ctrl_Prop Value
----------------
CC Rate 30%
----------------
上述是检查一致性率（可在 0％ 到 100％ 之间配置），它代表专用于运行检查一致性任务的系统资源的百分比。值为 0％ 时，检查一致性对于控制器的优先级最低，花费的时间最多，并且对系统性能的影响最小。检查一致性率为 0％ 并不表示检查一致性已停止或暂停。在 100％ 时，检查一致性是控制器的最高优先级。检查一致性时间最短，并且对系统性能的影响最大。
$ storcli /cx set consistencycheck|cc=[off|seq|conc] [delay=value]
$ storcli /cx set ccrate=<value>
# 一个推荐的配置
$ storcli /cx set cc=seq delay=1344
$ storcli /cx set ccrate=15
清除缓存
$ storcli64 /c0 flushcache
Controller = 0
Status = Success
Description = Controller and/or disk caches flushed successfully.
enclosure 管理
通过 RAID 挂载的磁盘，先是会被 enclosure 管理。服务器一般会有一个独立的存储背板，12 盘位或 24 盘位。
查看所有 enclosure 信息
$ storcli64 /c0/eall show
Controller = 0
Status = Success
Description = None
Properties :
==========
--------------------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------------------
16 OK 24 24 0 0 0 0 0 Port 4 - 7 x4 MP4242E000 x36-254.13.0.0
252 OK 8 0 0 0 0 0 1 Internal SGPIO
--------------------------------------------------------------------------------
从上述信息可以看出，enclosure id 为 16 的设备，有 24 槽位，同时也有 24 个物理盘，它的产品 ID 是 MP4242E000。除此之外，还有个内部的设备，八个槽位未被使用（主板上的 SATA）。
查看单个 enclosure 的信息
$ storcli64 /c0/e16 show all
Controller = 0
Status = Success
Description = None
Enclosure /c0/e16 :
==================
Information :
===========
Device ID = 16
Position = 1
Connector Name = Port 4 - 7 x4
Enclosure Type = SES
Status = OK
FRU Part Number = N/A
Enclosure Serial Number = N/A
ESM Serial Number = N/A
Enclosure Zoning Mode = N/A
Partner Device ID = Unavailable
Device Type = Enclosure
Inquiry Data :
============
Vendor Identification = SDATA
Product Identification = MP4242E000
Product Revision Level = 0d00
Properties :
==========
--------------------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------------------
16 OK 24 24 0 0 0 0 0 Port 4 - 7 x4 MP4242E000 x36-254.13.0.0
--------------------------------------------------------------------------------
出现了更详细的信息（虽然很多是N/A）。
查看 enclosure 的状态
$ storcli64 /c0/e16 show status
Controller = 0
Status = Success
Description = None
Enclosure /c0/e16 :
==================
Slot Info :
=========
------------
Slot Status
------------
0 OK
1 OK
2 OK
3 OK
4 OK
5 OK
6 OK
7 OK
8 OK
9 OK
10 OK
11 OK
12 OK
13 OK
14 OK
15 OK
16 OK
17 OK
18 OK
19 OK
20 OK
21 OK
22 OK
23 OK
------------
磁盘配置
视角聚焦到单独的物理盘上。
查看所有磁盘
$ storcli64 /c0/eall/sall show
Controller = 0
Status = Success
Description = Show Drive Information Succeeded.
Drive Information :
=================
-------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-------------------------------------------------------------------------------
16:1 18 Onln 0 7.276 TB SATA HDD N N 512B ST8000NM000A-2KE101 U -
16:2 17 Onln 0 7.276 TB SATA HDD N N 512B ST8000NM000A-2KE101 U -
16:3 20 Onln 0 7.276 TB SATA HDD N N 512B ST8000NM000A-2KE101 U -
...
-------------------------------------------------------------------------------
省略一些结果，该指令可以看出每一个磁盘的信息，容量大小，所属 DG，块大小，序列号等信息。
磁盘点灯，指示位置
$ storcli64 /cx/ex/sx start locate
$ storcli64 /cx/ex/sx stop locate
磁盘 rebuild
$ storcli64 /cx/ex/sx show rebuild
$ storcli64 /cx/ex/sx start rebuild
$ storcli64 /cx/ex/sx stop rebuild
设置磁盘状态
$ storcli64 /cx/ex/sx set good/offline/online/missing
good 空闲
online 盘上线
offline 盘下线
missing 掉线
磁盘热备
$ storcli64 /cx/ex/sx add hotsparedrive dgs=x 设置模块磁盘为diskgroup x 的热备盘
$ storcli64 /cx/ex/sx delete hotsparedrive
RAID 配置
RAID 中会有些虚拟化的概念，例如 DG 和 VD。
PD 指的是实打实的 物理磁盘。DG 是 DIsk Group，磁盘组，它由多个 PD 组成。VD 是 Virtual Disk，虚拟磁盘，它构建在 DG 之上。单个 VD 可以构建在多个 DG 至少，单个 DG 也可以承载多个 VD。但为了统一，尽量使用，单 DG 多 VD 或者 单 DG 单 VD。
由于 storcli 不支持创建 DG，所以直接从 VD 起步。这里我们需要先从 storcli64 /c0/eall/sall show 查看 enclosure ID 和 Slot ID。
创建 VD
创建 VD 的同时会创建 DG。
$ storcli64 /c0 add vd help
storcli /cx add vd r[0|1|5|6|00|10|50|60]
[Size=<VD1_Sz>,<VD2_Sz>,..|all] [name=<VDNAME1>,..]
drives=e:s|e:s-x|e:s-x,y,e:s-x,y,z [PDperArray=x][SED]
[pdcache=on|off|default][pi][DimmerSwitch(ds)=default|automatic(auto)|
none|maximum(max)|MaximumWithoutCaching(maxnocache)][WT|WB|AWB][nora|ra]
[direct|cached] [cachevd] [Strip=<8|16|32|64|128|256|512|1024>]
[AfterVd=X] [EmulationType=0|1|2] [Spares = [e:]s|[e:]s-x|[e:]s-x,y]
[force][ExclusiveAccess] [Cbsize=0|1|2 Cbmode=0|1|2|3|4|7]
storcli /cx add vd each r0 [name=<VDNAME1>,..] [drives=e:s|e:s-x|e:s-x,y]
[SED] [pdcache=on|off|default][pi] [DimmerSwitch(ds)=default|
automatic(auto)|none|maximum(max)|MaximumWithoutCaching(maxnocache)]
[WT|WB|AWB] [nora|ra] [direct|cached] [EmulationType=0|1|2]
[Strip=<8|16|32|64|128|256|512|1024>] [ExclusiveAccess]
[Cbsize=0|1|2 Cbmode=0|1|2|3|4|7]
storcli /cx add VD cachecade|nytrocache r[aid][0,1,10, 1EC]
drives = [e:]s|[e:]s-x|[e:]s-x,y [WT|WB] [assignvds = 0,1,2]
[BOOTVOLSIZE=x]
storcli /cx add VD cachecade|nytrocache slice r[aid][0,1]
size=<x> [name=<NytroCacheName>] drives= [e:]s|[e:]s-x|[e:]s-x,y
[WT|WB][assignvds = 0,1,2]
size=[<VD1_Sz>,<VD2_Sz>,..] [name=<VDNAME1>,..] [WT|WB] [nora|ra]
[direct|cached] [CachedBadBBU|NoCachedBadBBU]
[Strip=<8|16|32|64|128|256|512|1024>] [AfterVd=X]
[Spares = [e:]s|[e:]s-x|[e:]s-x,y] [force]
一般使用第一种方式创建就好。
# 假设要全盘创建 RAID0，且只考虑 IO 性能
$ storcli64 /c0 add vd r0 size=all name=just-you-like drives=16:1-24 AWB ra cached strip=256
# 假设要创建 RAID1 系统盘，且准备一个热备盘
$ storcli64 /c0 add vd r1 size=all name=system drivers=16:1-2 WB ra cached strip=256 spares=16:3
查看 DG 信息
$ storcli64 /c0/dall show
查看 VD 信息
$ storcli64 /c0/vall show
设置 VD
$ storcli64 /cx/vx set wrcache=WT|WB|AWB
$ storcli64 /cx/vx set rdcache=RA|NoRA
$ storcli64 /cx/vx set iopolicy=Cached|Direct
$ storcli64 /cx/vx set accesspolicy=RW|RO|Blocked|RmvBlkd
删除 VD
$ storcli64 /cx/vx del [cachecade] [discardcache] [force]
foreign Configuration
RAID 信息不止保存在 RAID 卡上，每个有关联的磁盘上也会有信息，如果 RAID 卡坏了，更换 RAID 卡，就需要配置 foreign Configuration。
查看 foreign Configuration
storcli /cx/fall show [all] [securityKey = xxx]
删除 foreign Configuration
storcli /cx/fall del|delete [securityKey = xxx]
导入 foreign Configuration
storcli /cx/fall import [preview] [securityKey = xxx]