/****************************************************************/ /* Document: A very quick (incomplete) intro on SAN technology */ /* File : very_short_intro_san.txt */ /* version : 0.3 */ /* Date : 09/08/2009 */ /* By : Albert van der Sel */ /****************************************************************/ This very simplistic note is for people who would like a quick intro into basic SAN terminology and technology. =================================================================== 1. How it was in the "old" days: DAS or Directly Attached Storage : =================================================================== Nowadays, in datacentres (and other larger computing facilities), the use of SAN's is actually quite typical, contrary to using "local Server storage", for storing Enterprise data. That does not mean that, "local storage", or "Directly Attached Storage" (on Servers), is totally gone. Sure, many Servers have internal disks (e.g. bootdisk and other disks). But for the critical data, we can say, that this is more likely to be found on SAN's than on some local Server storage. Just as the name suggests, local storage or DAS, means that your Server could have, for example, one or more SCSI cards build in, on which bus, SCSI storage devices are placed. Ofcourse, you already know all this stuff. We are talking about truly local storage here. So there is nothing new here. Server using DAS ------------------------ | build in | | -------| | | |SCSI | | | |card |------ | | | | | | | -------| bus | | | | | | o----------------- | | | | | | | ( ) ( ) ( ) | | disk disk disk | | | -----------------------| fig 1. Just remember, that instead of a "bunch of internal disks", nothing really changes if the disks are located in an external cabinet. This cabinet is still only connected to some local Server system. In fact, the storage itself is quite "private" to the Server, although you can always "export" or "share" filesystems (on that storage) to remote users. Server using DAS ------------------------ external cabinet | | --------------- | -------| | | | | |SCSI | | | ----- disk | | |card |-----------------| ----- disk | | | | | | ----- | | -------| bus | | ----- | | | | ----- | | | | ----- | ------------------------ |-------------| fig 2. Such a collection of disks is sometimes also called JBOD (Just a Bunch Of Disks). In this situation, your Operating System allows you to create partitions on those disks, and create filesystems. That's all pretty standard stuff. And, if your Server Operating System supports some sort of "Logical Volume Manager" (LVM), you can even create a entity, which is mostly called a "Volume Group" (VG), where such a VG can be created from several partitions. disk cabinet |==========================| | part1 | | --|----|--------- disk | | | | part2 | | -----|-----|---- disk | | | | part3 | | |-----|--------- disk | | | |==========================| fig. 3 A LVM allows you to create a Volume Group from several partitions, possibly located on several different disks. In fig 3, I have tried to illustrate that a bit. Here you see three diskpartitions, on three separate disks. The LVM magically "knows" how to create one volume from those individual pieces of storage (in this example part 1, part 2, part 3). Ofcourse, the LVM needs to have some sort of "mapping mechanism" which makes this all possible. Not seldom, certain area's on the physical disks are reserved for those "mappings" and metadata. Then, after the Volume Group has been created, you might create a filesystem (in the usual way) on such an VG, which might be represented by (like for example) "G:" (on Windows Server systems), or (like for example) "/apps" (in Unix-like environments). With the aid of the LVM, you might just create one large "virtual disk" from, say, three separate partitions (but no redundancy here and no protection). Or you might create a RAID 5 disksystem, created from, say, four separate partitions (now with redundancy and protection). Again, after you have created filesystems on such a virtual disk, you might have a "disk" accessible by (for example) "G:" (on Windows), or by (for example) "/appl" (on unix). Such a LVM as discussed above, is sometimes called a "Host based LVM", because it's done by software running on a Server (as opposed to virtualisation at a SAN). Note: more variations are possible. On some architectures, you need to purchase some raid card (feature) before you can implement raid on the disks in that cabinet. Again, this section probably offered nothing new to the reader, but I thought it would be wise to introduce the concept of "DAS", "local storage", LVM etc.. before moving to SAN concepts. =================================================================== 2. Data centric computing and simplification of the Infrastructure : =================================================================== Before we actually move on to SAN implementation in section 3, the following considerations might be of interrest. Some considerations that will lead to the notion of a SAN, are the following: Data centric computing: ----------------------- Ofcourse, using local storage on systems, as we all know, works. But, if you have lots and lots of Servers, all with their own private local storage... could there not be a better way of organizing storage? At a certain point in time, more and more the notion of "Data centric" computing became popular. That is, can we find a way to uncouple direct storage from Server systems, and try to manage it in a centralized way? Or, to put that in other words, can we have a central storage facility, from where we can simply assign storage to Servers, which we can regard as consumers of that storage? And, just as easily, revoke storage from Servers if that would be neccessary? If you would grasp that concept into a simple figure, you would have "Storage and Data" in the centre of the figure, then followed by a ring of Server systems, then followed by all clients on the outer ring, like this: clients clients | clients \ | / SERVER SERVER SERVER \ | / --------- |STORAGE| clients-- SERVER----|& DATA |----SERVER---clients --------- | | SERVER | | clients fig 4. Now, figure 4 will never win a beauty contest, but hopefully the message is clear. Better (safer) infrastructure: ------------------------------ The section above, already shows some of the justification for implementing SAN's. What might be added is the following: -- 1. better management: In a large environment, there is typically a "storage administrator" or a "SAN administrator", or even a "Storage Department". Large central disksystems are managed by those people, using the right tools to do so. So, instead of managing hundreds of individual Servers with their local storage, central management is realized. -- 2. Consolidation / better efficiency Related to point 1, all storage is centrally located. Creating Logical Units (LU's) is much more effective in the sense of using diskspace efficiently, creating redundancy etc.. Just compare it to a situation with hundreds of individual Servers, all with their local storage. Note: for now, just think of a LU (LUN) as some sort of "exported disk" that a Server can use, just in the same way as if it was a local disk. -- 3. Safety, security, redundancy Central storage systems, managed by Storage Administrators, is much more safer and secure than just lots and lots of individual Servers using local storage, managed by lots and lots of local administrators. Also, if the storage people export a "disk" for use by some Server, you can bet that (normally) this "disk" is actually implemented in some RAID form, enhancing redundancy and recoverability. Also, if the business is large enough, it's likely that the data on the SAN used at the production site, is replicted (or remote copied) to some disaster recovery site. -- 4. Central backup facility Usually, a set of "dump disks" are reserved for speedy backups. On the Servers, some sort of backup agent is running, and at scheduled times (or even continuously) backups will run. Those backups could be (temporarily) stored at a set of specific dump disks, or directly stored at tapes under control of some sort of tape robot. Obviously, the larger the organization is, the better this approach is compared to "individual" managed backup runs on local tapedrives mounted on all those individual Servers. There are more considerations to make, but I will stop here. All in all, if you add it all up, the Total Cost of Ownership (TCO) should be reduced as well. =================================================================== 3. What is a SAN? =================================================================== Let's first take a look at some typical characteristics of a SAN: 1. Usually, a Server uses a Fiber Card (FC) that connects to a SAN switch. Note: FC is widely used, but using a networkcard connected to some high-speed ethernet network, can be found as well. Note: The Server might be equipped with two of those FC's, thereby implementing multi-pathing. This will enhance fault-tolerance/redundancy, because if the current FC card fails, the system will switch over in using the other FC card. 2. Basically, your network is "devided" in two parts. - You have a real network on "one side" with clientsystems and Servers, using (normal) networkcards and regular networkswitches. - The Servers are connected (with their FC cards) to SAN switches, which themselves are connected to storage systems. Depicted in a simple figure, it would look like this: clients Servers SAN switches ------ --|\ /| (Note: "o" denotes a client workstation) | ------ / | \/ | o--| | | / | /\ | |----| =--- |/ \|\ | | fc| ------ \ ------------ o--| | | \ | STORAGE | | | =--- ------= | | | fc| \ ------ | | o--| ------ \ |\ /| ------= | | \ | \/ | / | | | \| /\ | / = | |/ \|/ | | Network ------ = | | | fig. 5 ------------ Figure 5, would also not be a candidate for a beauty contest, but the general idea should be clear. 3. The term SAN is usually associated with "block I/O services" in the "SAN network". Indeed, the "storing" function is so central when discussing a SAN. Just as in the situation of local storage, where a Server might use as SCSI card, on which bus storage devices (SCSI disks) are placed, people talk of concepts like HBA, initiator, target, and stuff like that. The Host Bus Adapter can be that SCSI card, which is a controller acting as an initiater. This one usually starts "the conversation" using SCSI commands. The target then, is one of the storage devices (like a SCSI disk) on the bus. When communication is performed between controllers and targets (and thus involving disks), typically the elements in transfer are "block address spaces" and "datablocks". That's why people talk about "block I/O services" when discussing SAN's. So, in the SAN network, the same "block I/O" oriented conversations takes place. Indeed, in the upper layer of the SAN communication protocol, we can position SCSI commands. 4. Let's take a quick look on NAS (Network Attached Storage), and see why it is quite different from a SAN (although there are similarities as well). In NAS, a specialized storage system is connected to the network. The following figure tries to depict that: clients and Servers | o--| | | o--| ----------- | LAN | NAS | --- |---------= Storage | | | | card| | | |----| | | | | | | | --- | | | fig. 6 | ----------- Network Storage people mostly regards SAN and NAS in the following way: SAN's has a typical "storing" function, using block IO. NAS has a typical "filing" function, using SMB or NFS or other Server/Redirector type of protocol. Below SMB or NFS (or other), the "regular" network protocols are used like TCP-IP. 5. The SAN Storage subsystem. To get an idea on what we may expect to find in a SAN Storage device, it typically looks like this: ------------------------------------------------------------- external | | SAN =----| |= ------- disk | network | | ------------ |= ------- disk | ports =----| |Controller| |= ------- disk | (connects | | | | |= ------- | to switches) =----| | =................|= ------- | | |----------| =................|= ------- | | | | |some sort |= ------- | | | | |of internal |= ------- | | | |----------|connection |= ------- | | | implementation | | ---||||||| (sort of fabric) | | cache | | memory | | | fig 7. ------------------------------------------------------------- 6. Virtualization at SAN level. The Storage Administrator has ofcourse a set of tools to manage the storage devices in the SAN device. There could be some tens of disks, or possibly well over 100 disks (or much much more disks), present in such a device (or array of devices). What this Administrator could do, is create Logical Units, from "partitions" of disks, and such a LU has an associated number, which is the LUN. It resembles a bit the situation we had using a LVM on some Host, with which we were able to create "virtual disks". The LUN represents the "access" to the storage space, associated with this LU. Now, when people talk about SAN's, it's often heard that this exposed "virtual disk" is the LUN. It's not too bad, but formally, the LUN "number" is way to access the LU, while the LU represents the virtual disk. Secondly, the Administrator has it's way to asscociate this LUN with a certain external SAN port. That way, a Server which is connected (via a switch) to that particular SAN port, is in principle able to access that LU (or LUN). Often, implemented on that Server card, or implemented in the driver, is a sort of enummeration implemented, which does something like this: "which devices are now present?" But that's often not enough. On the OS level, in many occasions, some sort of config manager needs to run, in order to see and find this new disk on the OS level. From that moment, the Server Administrator can use the "disk" in the usual way. 7. To get a quick impression on the possible used protocols in a SAN, take a look at fig. 8. ---------------------------------------------- | SCSI | |--------------------------------------------- | | Fibre | | Fibre | | ISCSI | | | | Channel| | Channel| ---------- | | ---------- ---------- | TCP | | | | FCIP | ---------- | | ---------- | IP | | | | TCP | ---------- | | | IP | |Ethernet| | | ---------- ---------- | | |Ethernet| | | ---------- | | SAN | fig. 8 ----------------------------------------------