HOWTO: Multi Disk System Tuning <author>Stein Gjoen, <tt/sgjoen@nyx.net/ <date>v0.37 , 2006-04-02 <abstract> <nidx>disk</nidx> <nidx>partitions, disk (see disk)</nidx> This document describes how best to use multiple disks and partitions for a Linux system. Although some of this text is Linux specific the general approach outlined here can be applied to many other multi tasking operating systems. </abstract>  <toc>   <sect>Introduction <nidx>disk!introduction</nidx>       For unclear reasons this brand new release is codenamed    the <bf/Newark/ release. New code names will appear as per industry standard guidelines to emphasize the state-of-the-art-ness of this document. This document was written for two reasons, mainly because I got hold of 3 old SCSI disks to set up my Linux system on and I was pondering how best to utilise the inherent possibilities of parallelizing in a SCSI system. Secondly I hear there is a prize for people who write documents... This is intended to be read in conjunction with the Linux Filesystem Structure Standard (FSSTND). It does not in any way replace it but tries to suggest where physically to place directories detailed in the FSSTND, in terms of drives, partitions, types, RAID, file system (fs), physical sizes and other parameters that should be considered and tuned in a Linux system, ranging from single home systems to large servers on the Internet.  The followup to FSSTND is called the Filesystem Hierarchy Standard (FHS) and covers more than Linux alone. FHS versions 2.0, 2.1 and 2.2 have been released but there are still a few issues to be dealt with. Many recent distributions are now aiming for FHS compliance.  It is also a good idea to read the Linux Installation guides thoroughly and if you are using a PC system, which I guess the majority still does, you can find much relevant and useful information in the FAQs for the newsgroup comp.sys.ibm.pc.hardware especially for storage media. This is also a learning experience for myself and I hope I can start the ball rolling with this HOWTO and that it perhaps can evolve into a larger more detailed and hopefully even more correct HOWTO.  First of all we need a bit of legalese. Recent development shows it is quite important. <sect1>Copyright  This document is Copyright 1996 Stein Gjoen. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. If you have any questions, please contact <{linux-howto@metalab.unc.edu}> <sect1>Disclaimer Use the information in this document at your own risk. I disavow any potential liability for the contents of this document. Use of the concepts, examples, and/or other content of this document is entirely at your own risk. All copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Naming of particular products or brands should not be seen as endorsements. You are strongly recommended to take a backup of your system before major installation and backups at regular intervals. <sect1>News <nidx>disk!news on</nidx> This is a major upgrade featuring a new copyright statement that is intended to be Debian compliant and allow for inclusion in their distribution. A number of mistakes are corrected and new features added such as descriptions of recent ATA features and more.   On the development front people are concentrating their energy towards completing Linux 2.4 and until that is released there is not going to be much news on disk technology for Linux.  Also now the document is available in postscript both for US letter as well as European A4 formats. The latest version number of this document can be gleaned from my plan entry if you   <url url="http://www.mit.edu:8001/finger?sgjoen@nox.nyx.net" name="finger"> my Nyx account. Also, the latest version will be available on my web space on Nyx in a number of formats: <itemize> <item> <url url="http://www.nyx.net/˜sgjoen/disk.html" name="HTML">. <item> <url url="http://www.nyx.net/˜sgjoen/disk.txt" name="plain ASCII text"> (ca. 6200 lines). <item> <url url="http://www.nyx.net/˜sgjoen/disk-US.ps.gz" name="compressed postscript US letter format"> (ca. 90 pages). <item> <url url="http://www.nyx.net/˜sgjoen/disk-A4.ps.gz" name="compressed postscript European A4 format"> (ca. 85 pages). <item> <url url="http://www.nyx.net/˜sgjoen/disk.sgml" name="SGML source"> (ca. 260 KB). </itemize>   <sect1>Credits In this version I have the pleasure of acknowledging even more people who have contributed in one way or another:  <tscreen><verb> ronnej (at ) ucs.orst.edu cm (at) kukuruz.ping.at armbru (at) pond.sub.org R.P.Blake (at) open.ac.uk neuffer (at) goofy.zdv.Uni-Mainz.de sjmudd (at) redestb.es nat (at) nataa.fr.eu.org sundbyk (at) oslo.geco-prakla.slb.com ggjoeen (at) online.no mike (at) i-Connect.Net roth (at) uiuc.edu phall (at) ilap.com szaka (at) mirror.cc.u-szeged.hu CMckeon (at) swcp.com kris (at) koentopp.de edick (at) idcomm.com pot (at) fly.cnuce.cnr.it earl (at) sbox.tu-graz.ac.at ebacon (at) oanet.com vax (at) linkdead.paranoia.com tschenk (at) theoffice.net pjfarley (at) dorsai.org jean (at) stat.ubc.ca johnf (at) whitsunday.net.au clasen (at) unidui.uni-duisburg.de eeslgw (at) ee.surrey.asc.uk adam (at) onshore.com anikolae (at) wega-fddi2.rz.uni-ulm.de cjaeger (at) dwave.net eperezte (at) c2i.net yesteven (at) ms2.hinet.net cj (at) samurajdata.se tbotond (at) netx.hu russel (at) coker.com.au lars (at) iar.se GALLAGS3 (at) labs.wyeth.com morimoto (at) xantia.citroen.org shulegaa (at) gatekeeper.txl.com roman.legat (at) stud.uni-hannover.de ahamish (at) hicks.alien.usr.com hduff2 (at) worldnet.att.net mbaehr (at) email.archlab.tuwien.ac.at adc (at) postoffice.utas.edu.au pjm (at) bofh.asn.au jochen.berg (at) ac.com jpotts (at) us.ibm.com jarry (at) gmx.net LeBlanc (at) mcc.ac.uk masy (at) webmasters.gr.jp karlheg (at) hegbloom.net goeran (at) uddeborg.pp.se wgm (at) telus.net mleisner (at) eng.mc.xerox.com MKC (at) Stowers-Institute.org alanr (at) unix.sh yamadafm (at) gmail.com </verb></tscreen> <sect1>Translations Special thanks go to <tt/nakano (at) apm.seikei.ac.jp/ for doing the <url url="http://www.linux.or.jp/JF/JFdocs/Multi-Disk-HOWTO.html" name="Japanese translation">, general contributions as well as contributing an example of a computer in an academic setting, which is included at the end of this document. There are now many new translations available and special thanks go to the translators for the job and the input they have given: <itemize> <item><url url="http://www.tldp.org/" name="German Translation"> by <tt/chewie (at) nuernberg.netsurf.de/ <item><url url="http://www.swe-doc.linux.nu" name="Swedish Translation "> by <tt/jonah (at) swipnet.se/ <item><url url="http://www.lri.fr/˜loisel/howto/" name="French Translation"> by <tt/Patrick.Loiseleur (at) lri.fr/ <item><url url="http://www.tldp.org/" name="Chinese Translation"> by <tt/yesteven (at ) ms2.hinet.net/ <item><url url="http://www.pluto.linux.it/ildp/HOWTO/Multi-Disk-HOWTO.html" name="Italian Translation"> by <tt/bigpaul (at) flashnet.it/ </itemize> ICP Vortex is gratefully acknowledges for sending in-depth information on their range of RAID controllers. Also DPT is acknowledged for sending me documentation on their controllers as well as permission to quote from the material. These quotes have been approved before appearing here and will be clearly labelled. No quotes as of yet but that is coming. Not many still, so please read through this document, make a contribution and join the elite. If I have forgotten anyone, please let me know.   Any comments or suggestions can be mailed to my mail address on Nyx: <htmlurl url="mailto:sgjoen@mail.nyx.net/" name="sgjoen@mail.nyx.net">. So let's cut to the chase where <tt/swap/ and <tt>/tmp</tt> are racing along hard drive...   <sect>Structure As this type of document is supposed to be as much for learning as a technical reference document I have rearranged the structure to this end. For the designer of a system it is more useful to have the information presented in terms of the goals of this exercise than from the point of view of the logical layer structure of the devices themselves. Nevertheless this document would not be complete without such a layer structure the computer field is so full of, so I will include it here as an introduction to how it works. It is a long time since the <em/mini/ in mini-HOWTO could be defended as proper but I am convinced that this document is as long as it needs to be in order to make the right design decisions, and not longer. <sect1>Logical structure <nidx>disk!structure, I/O subsystem</nidx> This is based on how each layer access each other, traditionally with the application on top and the physical layer on the bottom. It is quite useful to show the interrelationship between each of the layers used in controlling drives. <tscreen><verb> ___________________________________________________________ |__ File structure ( /usr /tmp etc) __| |__ File system (ext2fs, vfat etc) __| |__ Volume management (AFS) __| |__ RAID, concatenation (md) __| |__ Device driver (SCSI, IDE etc) __| |__ Controller (chip, card) __| |__ Connection (cable, network) __| |__ Drive (magnetic, optical etc) __| ----------------------------------------------------------- </verb></tscreen> In the above diagram both volume management and RAID and concatenation are optional layers. The 3 lower layers are in hardware. All parts are discussed at length later on in this document. <sect1>Document structure Most users start out with a given set of hardware and some plans on what they wish to achieve and how big the system should be. This is the point of view I will adopt in this document in presenting the material, starting out with hardware, continuing with design constraints before detailing the design strategy that I have found to work well. I have used this both for my own personal computer at home, a multi purpose server at work and found it worked quite well. In addition my Japanese co-worker in this project have applied the same strategy on a server in an academic setting with similar success. Finally at the end I have detailed some configuration tables for use in your own design. If you have any comments regarding this or notes from your own design work I would like to hear from you so this document can be upgraded. <sect1>Reading plan Although not the biggest HOWTO it is nevertheless rather big already and I have been requested to make a reading plan to make it possible to cut down on the volume <descrip> <tag/Expert/ (aka the elite). If you are familiar with Linux as well as disk drive technologies you will find most of what you need in the appendices. Additionally you are recommended to read the FAQ and the <ref id="bits-n-pieces" name="Bits'n'pieces"> chapter. <tag/Experienced/ (aka Competent). If you are familiar with computers in general you can go straight to the chapters on <ref id="technologies" name="technologies"> and continue from there on. <tag/Newbie/ (mostly harmless). You just have to read the whole thing. Sorry. In addition you are also recommended to read all the other disk related HOWTOs. </descrip> <sect>Drive Technologies <nidx>disk!technologies</nidx> A far more complete discussion on drive technologies for IBM PCs can be found at the home page of <url url="http://thef-nym.sci.kun.nl/˜pieterh/storage.html" name="The Enhanced IDE/Fast-ATA FAQ"> which is also regularly posted on Usenet News. There is also a site dedicated to <url url="http://ata-atapi.com" name="ATA and ATAPI Information and Software">. Here I will just present what is needed to get an understanding of the technology and get you started on your setup. <sect1>Drives <nidx>disk!drives</nidx> This is the physical device where your data lives and although the operating system makes the various types seem rather similar they can in actual fact be very different. An understanding of how it works can be very useful in your design work. Floppy drives fall outside the scope of this document, though should there be a big demand I could perhaps be persuaded to add a little here. <sect1>Geometry <nidx>disk!geometry</nidx> Physically disk drives consists of one or more platters containing data that is read in and out using sensors mounted on movable heads that are fixed with respects to themselves. Data transfers therefore happens across all surfaces simultaneously which defines a cylinder of tracks. The drive is also divided into sectors containing a number of data fields. Drives are therefore often specified in terms of its geometry: the number of Cylinders, Heads and Sectors (CHS). For various reasons there is now a number of translations between <itemize> <item>the physical CHS of the drive itself <item>the logical CHS the drive reports to the BIOS or OS <item>the logical CHS used by the OS </itemize> Basically it is a mess and a source of much confusion. For more information you are strongly recommended to read the Large Disk mini-HOWTO <sect1>Media <nidx>disk!media</nidx> The media technology determines important parameters such as read/write rates, seek times, storage size as well as if it is read/write or read only. <sect2>Magnetic Drives <label id="magnetic-drives"> <nidx>disk!media!magnetic</nidx> This is the typical read-write mass storage medium, and as everything else in the computer world, comes in many flavours with different properties. Usually this is the fastest technology and offers read/write capability. The platter rotates with a constant angular velocity (CAV) with a variable physical sector density for more efficient magnetic media area utilisation. In other words, the number of bits per unit length is kept roughly constant by increasing the number of logical sectors for the outer tracks. Typical values for rotational speeds are   7200 and 10000 RPM, though 15 kRPM disks exist. Seek times are around 10 ms, transfer rates quite variable from one type to another but typically 4-40 MB/s. With the extreme high performance drives you should remember that performance costs more electric power which is dissipated as heat, see the point on <ref id="power-heating" name="Power and Heating">. Fast rotating disks also have a tendency to be more noisy to the point of being too noisy for office use. Note that there are several kinds of transfers going on here, and that these are quoted in different units. First of all there is the platter-to-drive cache transfer, usually quoted in Mbits/s. Typical values here is about 50-250 Mbits/s. The second stage is from the built in drive cache to the adapter, and this is typically quoted in MB/s, and typical quoted values here is 3-40 MB/s. Note, however, that this assumed data is already in the cache and hence for maximum readout speed from the drive the effective transfer rate will decrease dramatically.  <sect2>Optical Drives <nidx>disk!media!optical</nidx> Optical read/write drives exist but are slow and not so common. They were used in the NeXT machine but the low speed was a source for much of the complaints. The low speed is mainly due to the thermal nature of the phase change that represents the data storage. Even when using relatively powerful lasers to induce the phase changes the effects are still slower than the magnetic effect used in magnetic drives. Today many people use CD-ROM drives which, as the name suggests, is read-only. Storage is about 650 MB, transfer speeds are variable, depending on the drive but can exceed 1.5 MB/s. Data is stored on a spiraling single track so it is not useful to talk about geometry for this. Data density is constant so the drive uses constant linear velocity (CLV). Seek is also slower, about 100 ms, partially due to the spiraling track. Recent, high speed drives, use a mix of CLV and CAV in order to maximize performance. This also reduces access time caused by the need to reach correct rotational speed for readout.   DVD offers storage capacity from 4.7 GB up to about 19 GB, depending on number of layers and sides. <sect2>Solid State Drives <nidx>disk!media!solid state</nidx> This is a relatively recent addition to the available technology and has been made popular especially in portable computers as well as in embedded systems. Containing no movable parts they are very fast both in terms of access and transfer rates. The most popular type is flash RAM, but also other types of RAM is used. A few years ago many had great hopes for magnetic bubble memories but it turned out to be relatively expensive and is not that common. In general the use of RAM disks are regarded as a bad idea as it is normally more sensible to add more RAM to the motherboard and let the operating system divide the memory pool into buffers, cache, program and data areas. Only in very special cases, such as real time systems with short time margins, can RAM disks be a sensible solution. Flash RAM is today available in several 10's of megabytes in storage and one might be tempted to use it for fast, temporary storage in a computer. There is however a huge snag with this: flash RAM has a finite life time in terms of the number of times you can rewrite data, so putting <tt>swap</tt>, <tt>/tmp</tt> or <tt>/var/tmp</tt> on such a device will certainly shorten its lifetime dramatically. Instead, using flash RAM for directories that are read often but rarely written to, will be a big performance win. In order to get the optimum life time out of flash RAM you will need to use special drivers that will use the RAM evenly and minimize the number of block erases. This example illustrates the advantages of splitting up your directory structure over several devices. Solid state drives have no real cylinder/head/sector addressing but for compatibility reasons this is simulated by the driver to give a uniform interface to the operating system. <sect1>Interfaces <nidx>disk!interfaces</nidx> There is a plethora of interfaces to chose from widely ranging in price and performance. Most motherboards today include IDE interface which are part of modern chipsets. Many motherboards also include a SCSI interface chip made by Symbios (formerly NCR) and that is connected directly to the PCI bus. Check what you have and what BIOS support you have with it. <sect2>MFM and RLL <nidx>disk!interfaces!MFM</nidx> <nidx>disk!interfaces!RLL</nidx> Once upon a time this was the established technology, a time when 20 MB was awesome, which compared to todays sizes makes you think that dinosaurs roamed the Earth with these drives. Like the dinosaurs these are outdated and are slow and unreliable compared to what we have today. Linux does support this but you are well advised to think twice about what you would put on this. One might argue that an emergency partition with a suitable vintage of DOS might be fitting. <sect2>ESDI <nidx>disk!interfaces!ESDI</nidx>   Actually, ESDI was an adaptation of the very widely used SMD interface used on "big" computers to the cable set used with the ST506 interface, which was more convenient to package than the 60-pin + 26-pin connector pair used with SMD. The ST506 was a "dumb" interface which relied entirely on the controller and host computer to do everything from computing head/cylinder/sector locations and keeping track of the head location, etc. ST506 required the controller to extract clock from the recovered data, and control the physical location of detailed track features on the medium, bit by bit. It had about a 10-year life if you include the use of MFM, RLL, and ERLL/ARLL modulation schemes. ESDI, on the other hand, had intelligence, often using three or four separate microprocessors on a single drive, and high-level commands to format a track, transfer data, perform seeks, and so on. Clock recovery from the data stream was accomplished at the drive, which drove the clock line and presented its data in NRZ, though error correction was still the task of the controller. ESDI allowed the use of variable bit density recording, or, for that matter, any other modulation technique, since it was locally generated and resolved at the drive. Though many of the techniques used in ESDI were later incorporated in IDE, it was the increased popularity of SCSI which led to the demise of ESDI in computers. ESDI had a life of about 10 years, though mostly in servers and otherwise "big" systems rather than PC's. <sect2>IDE and ATA <nidx>disk!interfaces!IDE</nidx> <nidx>disk!interfaces!ATA</nidx> Progress made the drive electronics migrate from the ISA slot card over to the drive itself and Integrated Drive Electronics was borne. It was simple, cheap and reasonably fast so the BIOS designers provided the kind of snag that the computer industry is so full of. A combination of an IDE limitation of 16 heads together with the BIOS limitation of 1024 cylinders gave us the infamous 504 MB limit. Following the computer industry traditions again, the snag was patched with a kludge and we got all sorts of translation schemes and BIOS bodges. This means that you need to read the installation documentation very carefully and check up on what BIOS you have and what date it has as the BIOS has to tell Linux what size drive you have. Fortunately with Linux you can also tell the kernel directly what size drive you have with the drive parameters, check the documentation for LILO and Loadlin, thoroughly. Note also that IDE is equivalent to ATA, AT Attachment. IDE uses CPU-intensive Programmed Input/Output (PIO) to transfer data to and from the drives and has no capability for the more efficient Direct Memory Access (DMA) technology. Highest transfer rate is 8.3 MB/s. <sect2>EIDE, Fast-ATA and ATA-2 <nidx>disk!interfaces!EIDE</nidx> <nidx>disk!interfaces!Fast-ATA</nidx> <nidx>disk!interfaces!ATA-2</nidx> These 3 terms are roughly equivalent, fast-ATA is ATA-2 but EIDE additionally includes ATAPI. ATA-2 is what most use these days which is faster and with DMA. Highest transfer rate is increased to 16.6 MB/s.  <sect2>Ultra-ATA <nidx>disk!interfaces!Ultra-ATA</nidx> A faster DMA mode that is approximately twice the speed of EIDE PIO-Mode 4 (33 MB/s). Disks with and without Ultra-ATA can be mixed on the same cable without speed penalty for the faster adapters. The Ultra-ATA interface is electrically identical with the normal Fast-ATA interface, including the maximum cable length.  The ATA/66 was superceeded by ATA/100 and then ATA/133. While the interface speed has improved dramatically the disks are often limited by platter-to-cache limits which today stands at about 40 MB/s. For more information read up on these overviews and whitepapers from Maxtor: <url url="http://www.maxtor.com/_files/maxtor/en_us/documentation/white_papers_technical/fast_drives_white_papers.pdf" name="Fast Drives Technology"> on the ATA/133 interface and <url url="http://www.maxtor.com/_files/maxtor/en_us/documentation/white_papers_technical/big_drives_white_papers.pdf" name="Big Drives Technology"> on breaking the 137 GB limit. <sect2>Serial-ATA <nidx>disk!interfaces!Serial-ATA</nidx> A new, standard has been agreed upon, the <tt>Serial-ATA</tt> interface, backed by the <url url="http://www.serialata.org/" name="The Serial ATA"> group who made the announcement in August 2001. Advantages are numerous: simple, thin connectors rather than old cumbersome cable mats that also obstructed air flow, higher speeds (about 150 MB/s) and backward compatibility. More recent serial ATA systems now support 300 MB/s. <sect2>ATAPI <nidx>disk!interfaces!ATAPI</nidx> The ATA Packet Interface was designed to support CD-ROM drives using the IDE port and like IDE it is cheap and simple. <sect2>SCSI <nidx>disk!interfaces!SCSI</nidx> The Small Computer System Interface is a multi purpose interface that can be used to connect to everything from drives, disk arrays, printers, scanners and more. The name is a bit of a misnomer as it has traditionally been used by the higher end of the market as well as in work stations since it is well suited for multi tasking environments. The standard interface is 8 bits wide and can address 8 devices. There is a wide version with 16 bit that is twice as fast on the same clock and can address 16 devices. The host adapter always counts as a device and is usually number 7. It is also possible to have 32 bit wide busses but this usually requires a double set of cables to carry all the lines. Data rates have steadily increased <tscreen><verb> Bus Speed (MHz) | 8 16 |Note -------------------------------------------------- 5 | 5 10 | Old 10 | 10 20 | Fast 20 | 20 40 | Fast-20 / Ultra 40 | 40 80 | Ultra 2 40 DDR | 80 160 | Ultra 3 / Ultra-160 80 DDR | 160 320 | Ultra-320 -------------------------------------------------- </verb></tscreen> Before connecting up devices, make sure the voltage is compatible as there is high voltage differential (HVD), normal (single ended) and low voltage differential (LVD). It is possible to plug single ended disks into differential signalled cabling but the result is that the entire chain then reduces speed to that if single ended performance.    The higher performance comes at a cost that is usually higher than for (E)IDE. The importance of correct termination and good quality cables cannot be overemphasized. SCSI drives also often tend to be of a higher quality than IDE drives. Also adding SCSI devices tend to be easier than adding more IDE drives: Often it is only a matter of plugging or unplugging the device; some people do this without powering down the system. This feature is most convenient when you have multiple systems and you can just take the devices from one system to the other should one of them fail for some reason. There is a number of useful documents you should read if you use SCSI, the SCSI HOWTO as well as the SCSI FAQ posted on Usenet News. SCSI also has the advantage you can connect it easily to tape drives for backing up your data, as well as some printers and scanners. It is even possible to use it as a very fast network between computers while simultaneously share SCSI devices on the same bus. Work is under way but due to problems with ensuring cache coherency between the different computers connected, this is a non trivial task. SCSI numbers are also used for arbitration. If several drives request service, the drive with the lowest number is given priority. Note that newer SCSI cards will simultaneously support an array of different types of SCSI devices all at individually optimized speeds. <sect2>Serial Attached SCSI <nidx>disk!interfaces!Serial Attached SCSI</nidx> <nidx>disk!interfaces!SAS</nidx> Also SCSI is turning to serial communications to achieve even higher speeds and more convenient cabling. In fact one of the defined physical connectors is the same as for S-ATA. Note that SAS drives cannot be connected to SATA bus and this is safeguarded using a keyed connector. On the other hand it is possible to connect S-ATA drives to a SAS connector and use it through the Serial Tunneling Protocol (STP). <sect1>Cabling <nidx>disk!cabling</nidx> I do not intend to make too many comments on hardware but I feel I should make a little note on cabling. This might seem like a remarkably low technological piece of equipment, yet sadly it is the source of many frustrating problems. At todays high speeds one should think of the cable more of a an RF device with its inherent demands on impedance matching. If you do not take your precautions you will get a much reduced reliability or total failure. Some SCSI host adapters are more sensitive to this than others. Shielded cables are of course better than unshielded but the price is much higher. With a little care you can get good performance from a cheap unshielded cable. <itemize>  <item>For Fast-ATA and Ultra-ATA, the maximum cable length is specified as 45cm (18"). The data lines of both IDE channels are connected on many boards, though, so they count as <bf/one/ cable. In any case EIDE cables should be as short as possible. If there are mysterious crashes or spontaneous changes of data, it is well worth investigating your cabling. Try a lower PIO mode or disconnect the second channel and see if the problem still occurs. <item>For <tt>Cable Select</tt> (ATA drives) you set the drive jumpers to cable select and use the cable to determine master and slave. This is not much used. <item>Do not have a slave on an ATA controller (primary or secondary) without a master on the same controller, behaviour in these cases is undetermined. <item> Use as short cable as possible, but do not forget the 30 cm minimum separation for ultra SCSI and 60 cm separation for differential SCSI. <item> Avoid long stubs between the cable and the drive, connect the plug on the cable directly to the drive without an extension. <item> SCSI Cabling limitations: <tscreen><verb> Bus Speed (MHz) | Max Length (m) -------------------------------------------------- 5 | 6 10 (fast) | 3 20 (fast-20 / ultra) | 3 (max 4 devices), 1.5 (max 8 devices) xx (differential) | 25 (max 16 devices -------------------------------------------------- </verb></tscreen> <item> Use correct termination for SCSI devices and at the correct positions: both ends of the SCSI chain. Remember the host adapter itself may have on board termination. <item> Do not mix shielded or unshielded cabling, do not wrap cables around metal, try to avoid proximity to metal parts along parts of the cabling. Any such discontinuities can cause impedance mismatching which in turn can cause reflection of signals which increases noise on the cable. This problems gets even more severe in the case of multi channel controllers. Recently someone suggested wrapping bubble plastic around the cables in order to avoid too close proximity to metal, a real problem inside crowded cabinets. </itemize> More information on SCSI cabling and termination can be found at  various web pages around the net. <sect1>Host Adapters <nidx>disk!adapters</nidx> <nidx>disk!host adapters</nidx> This is the other end of the interface from the drive, the part that is connected to a computer bus. The speed of the computer bus and that of the drives should be roughly similar, otherwise you have a bottleneck in your system. Connecting a RAID 0 disk-farm to a ISA card is pointless. These days most computers come with 32 bit PCI bus capable of 132 MB/s transfers which should not represent a bottleneck for most people in the near future. As the drive electronic migrated to the drives the remaining part that became the (E)IDE interface is so small it can easily fit into the PCI chip set. The SCSI host adapter is more complex and often includes a small CPU of its own and is therefore more expensive and not integrated into the PCI chip sets available today. Technological evolution might change this. Some host adapters come with separate caching and intelligence but as this is basically second guessing the operating system the gains are heavily dependent on which operating system is used. Some of the more primitive ones, that shall remain nameless, experience great gains. Linux, on the other hand, have so much smarts of its own that the gains are much smaller. Mike Neuffer, who did the drivers for the DPT controllers, states that the DPT controllers are intelligent enough that given enough cache memory it will give you a big push in performance and suggests that people who have experienced little gains with smart controllers just have not used a sufficiently intelligent caching controller. <sect1>Multi Channel Systems <nidx>disk!multi-channel</nidx> In order to increase throughput it is necessary to identify the most significant bottlenecks and then eliminate them. In some systems, in particular where there are a great number of drives connected, it is advantageous to use several controllers working in parallel, both for SCSI host adapters as well as IDE controllers which usually have 2 channels built in. Linux supports this. Some RAID controllers feature 2 or 3 channels and it pays to spread the disk load across all channels. In other words, if you have two SCSI drives you want to RAID and a two channel controller, you should put each drive on separate channels. <sect1>Multi Board Systems <nidx>disk!multi-board</nidx> In addition to having both a SCSI and an IDE in the same machine it is also possible to have more than one SCSI controller. Check the SCSI-HOWTO on what controllers you can combine. Also you will most likely have to tell the kernel it should probe for more than just a single SCSI or a single IDE controller. This is done using kernel parameters when booting, for instance using LILO. Check the HOWTOs for SCSI and LILO for how to do this. Multi board systems can offer significant speed gains if you configure your disks right, especially for RAID0. Make sure you interleave the controllers as well as the drives, so that you add drives to the md RAID device in the right order. If controller 1 is connected to drives <tt/sda/ and <tt/sdc/ while controller 2 is connected to drives <tt/sdb/ and <tt/sdd/ you will gain more paralellicity by adding in the order of <tt/sda - sdc - sdb - sdd/ rather than <tt/sda - sdb - sdc - sdd/ because a read or write over more than one cluster will be more likely to span two controllers. <label id="drive-names"> The same methods can also be applied to IDE. Most motherboards come with typically 4 IDE ports: <itemize> <item> <tt/hda/ primary master <item> <tt/hdb/ primary slave <item> <tt/hdc/ secondary master <item> <tt/hdd/ secondary slave </itemize> where the two primaries share one flat cable and the secondaries share another cable. Modern chipsets keep these independent. Therefore it is best to RAID in the order <tt/hda - hdc - hdb - hdd/ as this will most likely parallelise both channels. <sect1>Speed Comparison <nidx>disk!speed comparison</nidx> The following tables are given just to indicate what speeds are possible but remember that these are the theoretical maximum speeds. All transfer rates are in MB per second and bus widths are measured in bits. <sect2>Controllers <nidx>disk!speed comparison!controllers</nidx> <tscreen><verb> Modern IDE : 100 - 133 MB/s Modern SCSI : 80 - 320 MB/s </verb></tscreen> <sect2>Bus Types <nidx>disk!speed comparison!bus types</nidx> <tscreen><verb> ISA : 8-12 EISA : 33 VESA : 40 (Sometimes tuned to 50) PCI / PCI-X Bus width (bits) Bus Speed (MHz) | 32 64 -------------------------------------------------- 33 | 133 266 66 PCI 2.2 | 264 533 133 PCI-X | 533 1066 266 PCI-X 2.0 | 1066 2133 533 | 2133 4266 -------------------------------------------------- PCI-Express (also known as PCIe) Name | lanes speed (MB/s) -------------------------------------------------- x1 | 1 250 x4 | 4 1000 x8 | 8 2000 x16 | 16 4000 -------------------------------------------------- </verb></tscreen> <sect1>Benchmarking <nidx>disk!benchmarking</nidx> <nidx>disk!benchmarking!bonnie</nidx> <nidx>disk!benchmarking!iozone</nidx> <nidx>disk!Bonnie Raitt</nidx> This is a very, very difficult topic and I will only make a few cautious comments about this minefield. First of all, it is more difficult to make comparable benchmarks that have any actual meaning. This, however, does not stop people from trying... Instead one can use benchmarking to diagnose your own system, to check it is going as fast as it should, that is, not slowing down. Also you would expect a significant increase when switching from a simple file system to RAID, so a lack of performance gain will tell you something is wrong. When you try to benchmark you should not hack up your own, instead look up <tt/iozone/ and <tt/bonnie/ and read the documentation very carefully. In particular make sure your buffer size is bigger than your RAM size, otherwise you test your RAM rather than your disks which will give you unrealistically high performance. A very simple benchmark can be obtained using <tt/hdparm -tT/ which can be used both on IDE and SCSI drives.  For more information on benchmarking and software for a number of platforms, check out <url url="http://www.acnc.com/benchmarks.html" name="ACNC"> benchmark page as well as this article on   <url url="http://sourcepole.com/sources/reviews/raid/" name="benchmarking results"> and also  <url url="http://www.tldp.org/HOWTO/Benchmarking-HOWTO.html" name="The Benchmarking-HOWTO">. There are also official home pages for <url url="http://www.textuality.com/bonnie/" name="bonnie">, <url url="http://www.coker.com.au/bonnie++/" name="bonnie++"> and <url url="http://www.iozone.org" name="iozone">. Trivia: Bonnie is intended to locate bottlenecks, the name is a tribute to Bonnie Raitt, "who knows how to use one" as the author puts it. <sect1>Comparisons <nidx>disk!comparisons</nidx> SCSI offers more performance than EIDE but at a price. Termination is more complex but expansion not too difficult. Having more than 4 (or in some cases 2) IDE drives can be complicated, with wide SCSI you can have up to 15 per adapter. Some SCSI host adapters have several channels thereby multiplying the number of possible drives even further. For SCSI you have to dedicate one IRQ per host adapter which can control up to 15 drives. With EIDE you need one IRQ for each channel (which can connect up to 2 disks, master and slave) which can cause conflict. RLL and MFM is in general too old, slow and unreliable to be of much use. <sect1>Future Development <nidx>disk!future development</nidx>  SCSI-3 is released and SCSI-3 based devices are appearing, though many still use SCSI-2 mixed with some SCSI-3. Parallel bus interfaces have reached a point where higher transfer rates become complicated and serial busses are now being developed and slowly overtake the parallel equivalents, both for ATA and SCSI. Proticols develop, and also ATA is geining features such as command queueing that was formerly reserved for higher end SCSI systems. Another trend is for larger and larger drives. It is possible to get 500 GB on a single drive though this is rather expensive. Currently the optimum storage for your money is about 300 GB but also this is continuously increasing. The introduction of DVD with nearly 20 GB on a single disk you can have a complete copy of even major FTP sites from around the world. The only thing we can be reasonably sure about the future is that even if it won't get any better, it will definitely be bigger.   A project to encapsulate SCSI over TCP/IP, called  <url url="http://www.ietf.org/rfc/rfc3720.txt" name="iSCSI"> has started, and <url url="http://www.cs.uml.edu/~mbrown/iSCSI/" name="Linux iSCSI implementations"> has appeared. <sect1>Recommendations <label id="recommendations"> <nidx>disk!recommendations</nidx> My personal view is that EIDE or Ultra ATA is the best way to start out on your system, especially if you intend to use DOS as well on your machine. If you plan to expand your system over many years or use it as a server I would strongly recommend you get SCSI drives. Currently wide SCSI is a little more expensive. You are generally more likely to get more for your money with standard width SCSI. There is also differential versions of the SCSI bus which increases maximum length of the cable. The price increase is even more substantial and cannot therefore be recommended for normal users. In addition to disk drives you can also connect some types of scanners and printers and even networks to a SCSI bus. Also keep in mind that as you expand your system you will draw ever more power, so make sure your power supply is rated for the job and that you have sufficient cooling. Many SCSI drives offer the option of sequential spin-up which is a good idea for large systems. See also <ref id="power-heating" name="Power and Heating">.    <sect>File System Structure <nidx>disk!filesystem structure</nidx> Linux has been multi tasking from the very beginning where a number of programs interact and run continuously. It is therefore important to keep a file structure that everyone can agree on so that the system finds data where it expects to. Historically there has been so many different standards that it was confusing and compatibility was maintained using symbolic links which confused the issue even further and the structure ended looking like a maze. <nidx>disk!FSSTND</nidx> In the case of Linux a standard was fortunately agreed on early on called the <em/File Systems Standard/ (FSSTND) which today is used by all main Linux distributions. <nidx>disk!FHS</nidx> Later it was decided to make a successor that should also support operating systems other than just Linux, called the <em/Filesystem Hierarchy Standard/ (FHS) at version 2.2 currently. This standard is under continuous development and will soon be adopted by Linux distributions. I recommend not trying to roll your own structure as a lot of thought has gone into the standards and many software packages comply with the standards. Instead you can read more about this at the <url url="http://www.pathname.com/fhs/" name="FHS home page">. This HOWTO endeavours to comply with FSSTND and will follow FHS when distributions become available. <sect1>File System Features <nidx>disk!filesystem features</nidx> The various parts of FSSTND have different requirements regarding speed, reliability and size, for instance losing root is a pain but can easily be recovered. Losing <tt>/var/spool/mail</tt> is a rather different issue. Here is a quick summary of some essential parts and their properties and requirements. Note that this is just a guide, there can be binaries in <tt>etc</tt> and <tt>lib</tt> directories, libraries in <tt>bin</tt> directories and so on. <sect2>Swap <nidx>disk!swap</nidx> <descrip> <tag/Speed/ Maximum! Though if you rely too much on swap you should consider buying some more RAM. Note, however, that on many old Pentium PC motherboards the cache will not work on RAM above 128 MB. <tag/Size/ Similar as for RAM. Quick and dirty algorithm: just as for tea: 16 MB for the machine and 2 MB for each user. Smallest kernel run in 1 MB but is tight, use 4 MB for general work and light applications, 8 MB for X11 or GCC or 16 MB to be comfortable. (The author is known to brew a rather powerful cuppa tea...) Some suggest that swap space should be 1-2 times the size of the RAM, pointing out that the locality of the programs determines how effective your added swap space is. Note that using the same algorithm as for 4BSD is slightly incorrect as Linux does not allocate space for pages in core. A more thorough approach is to consider swap space plus RAM as your total working set, so if you know how much space you will need at most, you subtract the physical RAM you have and that is the swap space you will need. There is also another reason to be generous when dimensioning your swap space: memory leaks. Ill behaving programs that do not free the memory they allocate for themselves are said to have a memory leak. This allocation remains even after the offending program has stopped so this is a source of memory consumption. Only after the program dies is the memory returned. Once all physical RAM and swap space are exhausted the only solution is to kill the offending processes if possible, or failing that, reboot and start over. Thankfully such programs are not too common but should you come across one you will find that extra swap space will buy you extra time between reboots. Also remember to take into account the type of programs you use. Some programs that have large working sets, such as  image processing software have huge data structures loaded in RAM rather than working explicitly on disk files. Data and computing intensive programs like this will cause excessive swapping if you have less RAM than the requirements. Other types of programs can lock their pages into RAM. This can be for security reasons, preventing copies of data reaching a swap device or for performance reasons such as in a real time module. Either way, locking pages reduces the remaining amount of swappable memory and can cause the system to swap earlier then otherwise expected. In <tt/man 8 mkswap/ it is explained that each swap partition can be a maximum of just under 128 MB in size for 32-bit machines and just under 256 MB for 64-bit machines. This however changed with kernel 2.2.0 after which the limit is 2 GB. The man page has been updated to reflect this change. <tag/Reliability/ Medium. When it fails you know it pretty quickly and failure will cost you some lost work. You save often, don't you? <tag/Note 1/ Linux offers the possibility of interleaved swapping across multiple devices, a feature that can gain you much. Check out "<tt>man 8 swapon</tt>" for more details. However, software raiding <tt>swap</tt> across multiple devices adds more overheads than you gain. Thus the <tt>/etc/fstab</tt> file might look like this: <tscreen><verb> /dev/sda1 swap swap pri=1 0 0 /dev/sdc1 swap swap pri=1 0 0 </verb></tscreen> Remember that the <tt/fstab/ file is <em/very/ sensitive to the formatting used, read the man page carefully and do <em/not/ just cut and paste the lines above. <tag/Note 2/ Some people use a RAM disk for swapping or some other file systems. However, unless you have some very unusual requirements or setups you are unlikely to gain much from this as this cuts into the memory available for caching and buffering. <tag/Note 2b/ There is once exception: on a number of badly designed motherboards the on board cache memory is not able to cache all the RAM that can be addressed. Many older motherboards could accept 128 MB RAM but only cache the lower 64 MB. In such cases it would improve the performance if you used the upper (uncached) 64 MB RAM for RAMdisk based swap or other temporary storage. </descrip> <sect2>Temporary Storage (<tt>/tmp</tt> and <tt>/var/tmp</tt>) <nidx>disk!temporary storage</nidx> <descrip> <tag/Speed/ Very high. On a separate disk/partition this will reduce fragmentation generally, though <tt/ext2fs/ handles fragmentation rather well. <tag/Size/ Hard to tell, small systems are easy to run with just a few MB but these are notorious hiding places for stashing files away from prying eyes and quota enforcement and can grow without control on larger machines. Suggested: small home machine: 8 MB, large home machine: 32 MB, small server: 128 MB, and large machines up to 500 MB (The machine used by the author at work has 1100 users and a 300 MB <tt>/tmp</tt> directory). Keep an eye on these directories, not only for hidden files but also for old files. Also be prepared that these partitions might be the first reason you might have to resize your partitions. <tag/Reliability/ Low. Often programs will warn or fail gracefully when these areas fail or are filled up. Random file errors will of course be more serious, no matter what file area this is. <tag/Files/ Mostly short files but there can be a huge number of them. Normally programs delete their old <tt>tmp</tt> files but if somehow an interruption occurs they could survive. Many distributions have a policy regarding cleaning out <tt>tmp</tt> files at boot time, you might want to check out what your setup is. <tag/Note1/ In FSSTND there is a note about putting <tt>/tmp</tt> on RAM disk. This, however, is not recommended for the same reasons as stated for swap. Also, as noted earlier, do not use flash RAM drives for these directories. One should also keep in mind that some systems are set to automatically clean <tt>tmp</tt> areas on rebooting. <tag/Note2/ Older systems had a <tt>/usr/tmp</tt> but this is no longer recommended and for historical reasons a symbolic link now makes it point to one of the other <tt>tmp</tt> areas. </descrip> (* That was 50 lines, I am home and dry! *) <sect2>Spool Areas (<tt>/var/spool/news</tt> and <tt>/var/spool/mail</tt>) <nidx>disk!spool areas</nidx> <descrip> <tag/Speed/ High, especially on large news servers. News transfer and expiring are disk intensive and will benefit from fast drives. Print spools: low. Consider RAID0 for news. <tag/Size/ For news/mail servers: whatever you can afford. For single user systems a few MB will be sufficient if you read continuously. Joining a list server and taking a holiday is, on the other hand, not a good idea. (Again the machine I use at work has 100 MB reserved for the entire <tt>/var/spool</tt>) <tag/Reliability/ Mail: very high, news: medium, print spool: low. If your mail is very important (isn't it always?) consider RAID for reliability. <tag/Files/ Usually a huge number of files that are around a few KB in size. Files in the print spool can on the other hand be few but quite sizable. <tag/Note/ Some of the news documentation suggests putting all the <tt>.overview</tt> files on a drive separate from the news files, check out all news FAQs for more information. Typical size is about 3-10 percent of total news spool size. </descrip> <sect2>Home Directories (<tt>/home</tt>) <label id="home-dirs"> <nidx>disk!home directories</nidx> <descrip> <tag/Speed/ Medium. Although many programs use <tt>/tmp</tt> for temporary storage, others such as some news readers frequently update files in the home directory which can be noticeable on large multiuser systems. For small systems this is not a critical issue. <tag/Size/ Tricky! On some systems people pay for storage so this is usually then a question of finance. Large systems such as <url url="http://www.nyx.net/" name="Nyx.net"> (which is a free Internet service with mail, news and WWW services) run successfully with a suggested limit of 100 KB per user and 300 KB as enforced maximum. Commercial ISPs offer typically about 5 MB in their standard subscription packages. If however you are writing books or are doing design work the requirements balloon quickly. <tag/Reliability/ Variable. Losing <tt>/home</tt> on a single user machine is annoying but when 2000 users call you to tell you their home directories are gone it is more than just annoying. For some their livelihood relies on what is here. You do regular backups of course? <tag/Files/ Equally tricky. The minimum setup for a single user tends to be a dozen files, 0.5 - 5 KB in size. Project related files can be huge though. <tag/Note1/ You might consider RAID for either speed or reliability. If you want extremely high speed and reliability you might be looking at other operating system and hardware platforms anyway. (Fault tolerance etc.) <tag/Note2/ Web browsers often use a local cache to speed up browsing and this cache can take up a substantial amount of space and cause much disk activity. There are many ways of avoiding this kind of performance hits, for more information see the sections on <ref id="server-home-dirs" name="Home Directories"> and <ref id="www" name="WWW">. <tag/Note3/ Users often tend to use up all available space on the <tt>/home</tt> partition. The Linux Quota subsystem is capable of limiting the number of blocks and the number of inode a single user ID can allocate on a per-filesystem basis. See the <url url="http://www.tldp.org/HOWTO/mini/Quota.html" name="Linux Quota mini-HOWTO"> by Albert M.C. Tam <tt/bertie (at) scn.org/ for details on setup. </descrip> <sect2>Main Binaries ( <tt>/usr/bin</tt> and <tt>/usr/local/bin</tt>)<label id="main-binaries"> <nidx>disk!main binaries</nidx> <descrip> <tag/Speed/ Low. Often data is bigger than the programs which are demand loaded anyway so this is not speed critical. Witness the successes of live file systems on CD ROM. <tag/Size/ The sky is the limit but 200 MB should give you most of what you want for a comprehensive system. A big system, for software development or a multi purpose server should perhaps reserve 500 MB both for installation and for growth. <tag/Reliability/ Low. This is usually mounted under root where all the essentials are collected. Nevertheless losing all the binaries is a pain... <tag/Files/ Variable but usually of the order of 10 - 100 KB. </descrip> <sect2>Libraries ( <tt>/usr/lib</tt> and <tt>/usr/local/lib</tt>) <nidx>disk!libraries</nidx> <descrip> <tag/Speed/ Medium. These are large chunks of data loaded often, ranging from object files to fonts, all susceptible to bloating. Often these are also loaded in their entirety and speed is of some use here. <tag/Size/ Variable. This is for instance where word processors store their immense font files. The few that have given me feedback on this report about 70 MB in their various <tt>lib</tt> directories. A rather complete Debian 1.2 installation can take as much as 250 MB which can be taken as an realistic upper limit. The following ones are some of the largest disk space consumers: GCC, Emacs, TeX/LaTeX, X11 and perl. <tag/Reliability/ Low. See point <ref id="main-binaries" name="Main binaries">. <tag/Files/ Usually large with many of the order of 1 MB in size. <tag/Note/ For historical reasons some programs keep executables in the lib areas. One example is GCC which have some huge binaries in the <tt>/usr/lib/gcc/lib</tt> hierarchy. </descrip> <sect2>Boot <nidx>disk!boot</nidx> <nidx>disk!1023</nidx> <nidx>disk!nuni</nidx> <descrip> <tag/Speed/ Quite low: after all booting doesn't happen that often and loading the kernel is just a tiny fraction of the time it takes to get the system up and running. <tag/Size/ Quite small, a complete image with some extras fit on a single floppy so 5 MB should be plenty. <tag/Reliability/ High. See section below on Root. <tag/Note 1/ The most important part about the Boot partition is that on many systems it <em/must/ reside below cylinder 1023. This is a BIOS limitation that Linux cannot get around. <tag/Note 1a/ The above is not necessarily true for recent IDE systems and not for any SCSI disks. For more information check the latest Large Disk HOWTO.  </descrip> <sect2>Root <nidx>disk!root</nidx> <descrip> <tag/Speed/ Quite low: only the bare minimum is here, much of which is only run at startup time. <tag/Size/ Relatively small. However it is a good idea to keep some essential rescue files and utilities on the root partition and some keep several kernel versions. Feedback suggests about 20 MB would be sufficient. <tag/Reliability/ High. A failure here will possibly cause a fair bit of grief and you might end up spending some time rescuing your boot partition. With some practice you can of course do this in an hour or so, but I would think if you have some practice doing this you are also doing something wrong. Naturally you do have a rescue disk? Of course this is updated since you did your initial installation? There are many ready made rescue disks as well as rescue disk creation tools you might find valuable. Presumably investing some time in this saves you from becoming a root rescue expert. <tag/Note 1/ If you have plenty of drives you might consider putting a spare emergency boot partition on a separate physical drive. It will cost you a little bit of space but if your setup is huge the time saved, should something fail, will be well worth the extra space. <tag/Note 2/ For simplicity and also in case of emergencies it is not advisable to put the root partition on a RAID level 0 system. Also if you use RAID for your boot partition you have to remember to have the <tt/md/ option turned on for your emergency kernel. <tag/Note 3/ For simplicity it is quite common to keep Boot and Root on the same partition. If you do that, then in order to boot from LILO it is important that the essential boot files reside wholly within cylinder 1023. This includes the kernel as well as files found in <tt>/boot</tt>. </descrip> <sect2>DOS etc. <nidx>disk!DOS-related issues</nidx> At the danger of sounding heretical I have included this little section about something many reading this document have strong feelings about. Unfortunately many hardware items come with setup and maintenance tools based around those systems, so here goes. <descrip> <tag/Speed/ Very low. The systems in question are not famed for speed so there is little point in using prime quality drives. Multitasking or multi-threading are not available so the command queueing facility found in SCSI drives will not be taken advantage of. If you have an old IDE drive it should be good enough. The exception is to some degree Win95 and more notably NT which have multi-threading support which should theoretically be able to take advantage of the more advanced features offered by SCSI devices. <tag/Size/ The company behind these operating systems is not famed for writing tight code so you have to be prepared to spend a few tens of MB depending on what version you install of the OS or Windows. With an old version of DOS or Windows you might fit it all in on 50 MB. <tag/Reliability/ Ha-ha. As the chain is no stronger than the weakest link you can use any old drive. Since the OS is more likely to scramble itself than the drive is likely to self destruct you will soon learn the importance of keeping backups here. Put another way: "<it/Your mission, should you choose to accept it, is to keep this partition working. The warranty will self destruct in 10 seconds.../" Recently I was asked to justify my claims here. First of all I am not calling DOS and Windows sorry excuses for operating systems. Secondly there are various legal issues to be taken into account. Saying there is a connection between the last two sentences are merely the ravings of the paranoid. Surely. Instead I shall offer the esteemed reader a few key words: DOS 4.0, DOS 6.x and various drive compression tools that shall remain nameless. </descrip> <sect1>Explanation of Terms <nidx>disk!terms explained</nidx> Naturally the faster the better but often the happy installer of Linux has several disks of varying speed and reliability so even though this document describes performance as 'fast' and 'slow' it is just a rough guide since no finer granularity is feasible. Even so there are a few details that should be kept in mind: <sect2>Speed <label id="speed"> <nidx>disk!terms explained!speed</nidx> This is really a rather woolly mix of several terms: CPU load, transfer setup overhead, disk seek time and transfer rate. It is in the very nature of tuning that there is no fixed optimum, and in most cases price is the dictating factor. CPU load is only significant for IDE systems where the CPU does the transfer itself but is generally low for SCSI, see SCSI documentation for actual numbers. Disk seek time is also small, usually in the millisecond range. This however is not a problem if you use command queueing on SCSI where you then overlap commands keeping the bus busy all the time. News spools are a special case consisting of a huge number of normally small files so in this case seek time can become more significant. There are two main parameters that are of interest here: <descrip> <tag/Seek/ is usually specified in the average time take for the read/write head to seek from one track to another. This parameter is important when dealing with a large number of small files such as found in spool files. There is also the extra seek delay before the desired sector rotates into position under the head. This delay is dependent on the angular velocity of the drive which is why this parameter quite often is quoted for a drive. Common values are   7200 and 10000 RPM (rotations per minute). Higher RPM reduces the seek time but at a substantial cost. Also drives working at high RPM have been known to be noisy and to generate a lot of heat, a factor that should be kept in mind if you are building a large array or "disk farm". Very recently drives working at 15000 RPM has entered the market and here the cooling requirements are even stricter and minimum figures for air flow are given. <tag/Transfer/ is usually specified in megabytes per second. This parameter is important when handling large files that have to be transferred. Library files, dictionaries and image files are examples of this. Drives featuring a high rotation speed also normally have fast transfers as transfer speed is proportional to angular velocity for the same sector density. </descrip> It is therefore important to read the specifications for the drives very carefully, and note that the maximum transfer speed quite often is quoted for transfers out of the on board cache (burst speed) and not directly from the platter (sustained speed). See also section on <ref id="power-heating" name="Power and Heating">. <sect2>Reliability <nidx>disk!terms explained!reliability</nidx> Naturally no-one would want low reliability disks but one might be better off regarding old disks as unreliable. Also for RAID purposes (See the relevant information) it is suggested to use a mixed set of disks so that simultaneous disk crashes become less likely. So far I have had only one report of total file system failure but here unstable hardware seemed to be the cause of the problems. Disks are cheap these days yet people still underestimate the value of the contents of the drives. If you need higher reliability make sure you replace old drives and keep spares. It is not unusual that drives can work more or less continuous for years and years but what often kills a drive in the end is power cycling. <sect2>Files <nidx>disk!terms explained!files</nidx> The average file size is important in order to decide the most suitable drive parameters. A large number of small files makes the average seek time important whereas for big files the transfer speed is more important. The command queueing in SCSI devices is very handy for handling large numbers of small files, but for transfer EIDE is not too far behind SCSI and normally much cheaper than SCSI. <sect>File Systems <nidx>disk!file systems</nidx> Over time the requirements for file systems have increased and the demands for large structures, large files, long file names and more has prompted ever more advanced file systems, the system that accesses and organises the data on mass storage. Today there is a large number of file systems to choose from and this section will describe these in detail. The emphasis is on Linux but with more input I will be happy to add information for a wider audience. <sect1>General Purpose File Systems Most operating systems usually have a general purpose file system for every day use for most kinds of files, reflecting available features in the OS such as permission flags, protection and recovery. <sect2><tt/minix/ <nidx>disk!file system!minix</nidx> This was the original fs for Linux, back in the days Linux was hosted on minix machines. It is simple but limited in features and hardly ever used these days other than in some rescue disks as it is rather compact. <sect2><tt/xiafs/ and <tt/extfs/ <nidx>disk!file system!xiafs</nidx> <nidx>disk!file system!extfs</nidx> These are also old and have fallen in disuse and are no longer recommended. <sect2><tt/ext2fs/ <nidx>disk!file system!ext2fs</nidx> This is the established standard for general purpose in the Linux world. It is fast, efficient and mature and is under continuous development and features such as ACL and transparent compression are on the horizon. For more information check the <url url="http://web.mit.edu/tytso/www/linux/ext2.html" name="ext2fs"> home page. <sect2><tt/ext3fs/ <nidx>disk!file system!ext3fs</nidx> This is the name for the successor to <tt/ext2fs/ available in kernel 2.4 and later. Many features are added to <tt/ext2fs/ but to avoid confusion over the name after such a radical upgrade the name will be changed too.  Metadata, data that describes the structure of the files, is written in a joural to the disk. Note that there are 3 modes of operations for <tt/ext3fs/. <descrip> <tag/data=writeback/ This is metadata-only journalling. File data is written back to the main fs lazily. After a crash and recovery the file integrity is intact but file content can be old. Normally this is the fastest mode. <tag/data=ordered/ Here file data is written before metadata. After a crash and recovery the file integrity is intact and files contain correct, recent data. Normally this is only slightly slower. <tag/data=journal/ All data (as well as to metadata) is written to the journal before it is released to the main fs for writeback. This is a specialised mode for cases where synchronous operations are required, such as for mail spools and synchronous NFS mounts. </descrip> <sect2><tt/ufs/ <nidx>disk!file system!ufs</nidx> This is the fs used by BSD and variants thereof. It is mature but also developed for older types of disk drives where geometries were known. The fs uses a number of tricks to optimise performance but as disk geometries are translated in a number of ways the net effect is no longer so optimal. <sect2><tt/efs/ <nidx>disk!file system!efs</nidx> The Extent File System (efs) is Silicon Graphics' early file system widely used on IRIX before version 6.0 after which xfs has taken over. While migration to xfs is encouraged efs is still supported and much used on CDs. There is a Linux driver available in early beta stage, available at <url url="http://aeschi.ch.eu.org/efs/" name="Linux extent file system"> home page. <sect2><tt/XFS/ <nidx>disk!file system!XFS</nidx> <url url="http://www.sgi.com/" name="Silicon Graphics Inc (sgi)"> has started porting its mainframe grade file system to Linux. Source is not yet available as they are busily cleaning out legal encumbrance but once that is done they will provide the source code under GPL. More information is already available on the  <url url="http://oss.sgi.com/projects/xfs/" name="XFS project page"> at SGI. <sect2><tt/reiserfs/ <nidx>disk!file system!reiserfs</nidx> <nidx>disk!file system!tree based</nidx> As of July, 23th 1997 Hans Reiser <tt/reiser (at) RICOCHET.NET/ has put up the source to his tree based    <url url="http://www.namesys.com" name="reiserfs"> on the web. While his filesystem has some very interesting features and is much faster than <tt/ext2fs/ and is in use by a number of people. It is available in kernel 2.4 and later.  <sect2><tt/enh-fs/ <nidx>disk!file system!enhanced fs</nidx>  The Enhanced File System project is now dead. <sect2><tt/Tux2 fs/ <nidx>disk!file system!Tux2 fs</nidx>  The Tux2 File System project is now dead. <sect1>Microsoft File Systems <nidx>disk!file system!Microsoft</nidx> <nidx>disk!file system!confusion</nidx> This company is responsible for a lot, including a number of filesystems that has at the very least caused confusions. <sect2><tt/fat/ <nidx>disk!file system!fat</nidx> Actually there are 2 <tt/fat/s out there, <tt/fat12/ and <tt/fat16/ depending on the partition size used but fortunately the difference is so minor that the whole issue is transparent. On the plus side these are fast and simple and most OSes understands it and can both read and write this fs. And that is about it. The minus side is limited safety, severely limited permission flags and atrocious scalability. For instance with <tt/fat/ you cannot have partitions larger than 2 GB. <sect2><tt/fat32/ <nidx>disk!file system!fat32</nidx> After about 10 years Microsoft realised <tt/fat/ was about, well, 10 years behind the times and created this fs which scales reasonably well. Permission flags are still limited. NT 4.0 cannot read this file system but Linux can. <sect2><tt/vfat/ <nidx>disk!file system!vfat</nidx> At the same time as Microsoft launched <tt/fat32/ they also added support for long file names, known as <tt/vfat/. Linux reads <tt/vfat/ and <tt/fat32/ partitions by mounting with type <tt/vfat/. <sect2><tt/ntfs/ <nidx>disk!file system!ntfs</nidx> This is the native fs of Win-NT but as complete information is not available there is limited support for other OSes. <sect1>Logging and Journaling File Systems <nidx>disk!file system!logging file systems</nidx> <nidx>disk!file system!journaling file systems</nidx> These take a radically different approach to file updates by logging modifications for files in a log and later at some time checkpointing the logs. Reading is roughly as fast as traditional file systems that always update the files directly. Writing is much faster as only updates are appended to a log. All this is transparent to the user. It is in reliability and particularly in checking file system integrity that these file systems really shine. Since the data before last checkpointing is known to be good only the log has to be checked, and this is much faster than for traditional file systems. Note that while <em/logging/ filesystems keep track of changes made to both data and inodes, <em/journaling/ filesystems keep track only of inode changes. Linux has quite a choice in such file systems but none are yet in production quality. Some are also on hold. <itemize> <item>Adam Richter from Yggdrasil posted some time ago that they have been working on a compressed log file based system but that this project is currently on hold. Nevertheless a non-working version is available on their FTP server. Check out <url url="ftp://ftp.yggdrasil.com/private/adam" name="the Yggdrasil ftp server"> where special patched versions of the kernel can be found. <item>Another project is the  <url url="http://outflux.net/projects/lfs/" name="Linux log-structured Filesystem Project"> which sadly also is on hold. Nevertheless this page contains much information on the topic. <item>Then there is the <url url="http://www.complang.tuwien.ac.at/czezatke/lfs.html" name="LinLogFS -- A Log-Structured Filesystem For Linux"> (formerly known as dtfs) which seems to be going strong. Still in alpha but sufficiently complete to make programs run off this file system <item>Finally there is the <url url="http://developer.axis.com/software/jffs/" name="Journaling Flash File System"> designed for their embedded diskless systems such as their Linux based web camera. </itemize> Note that <tt/ext3fs/, <tt/XFS/ and <tt/reiserfs/ also have features for logging or journaling. <sect1>Read-only File Systems <nidx>disk!file system!read-only file systems</nidx> Read-only media has not escaped the ever increasing complexities seen in more general file systems so again there is a large choice to choose from with corresponding opportunities for exciting mistakes. Note that <tt/ext2fs/ works quite well on a CD-ROM and seems to save space while offering the normal file system features such as long file names and permissions that can be retained when copying files across to read-write media. Also having  <htmlurl url="file:///dev/" name="/dev"> on a CD-ROM is possible. <nidx>disk!file system!CD-ROM</nidx> <nidx>disk!file system!DVD</nidx> <nidx>disk!file system!loopback</nidx> Most of these are used with the CD-ROM media but also the new DVD can be used and you can even use it through the loopback device on a hard disk file for verifying an image before burning a ROM. <nidx>disk!file system!rom file systems</nidx> <nidx>disk!file system!romfs</nidx> There is a read-only <tt/romfs/ for Linux but as that is not disk related nothing more will be said about it here. <sect2><tt/High Sierra/ <nidx>disk!file system!High Sierra</nidx> This was one of the earliest standards for CD-ROM formats, supposedly named after the hotel where the final agreement took place. <tt/High Sierra/ was so limited in features that new extensions simply had to appear and while there has been no end to new formats the original <tt/High Sierra/ remains the common precursor and is therefore still widely supported. <sect2><tt/iso9660/ <nidx>disk!file system!iso9660</nidx> The International Standards Organisation made their extensions and formalised the standard into what we know as the <tt/iso9660/ standard. The Linux iso9660 file system supports both High Sierra as well as <tt/Rock Ridge/ extensions. <sect2><tt/Rock Ridge/ <nidx>disk!file system!Rock Ridge</nidx> Not everyone accepts limits like short filenames and lack of permissions so very soon the <tt/Rock Ridge/ extensions appeared to rectify these shortcomings. <sect2><tt/Joliet/ <nidx>disk!file system!Joliet</nidx> Microsoft, not be be outdone in the standards extension game, decided it should extend CD-ROM formats with some internationalisation features and called it <tt/Joliet/. Linux supports this standards in kernels 2.0.34 or newer. You need to enable NLS in order to use it. <sect2>Trivia <nidx>disk!file system!Trivia</nidx> Joliet is a city outside Chicago; best known for being the site of the prison where Jake was locked up in the movie "Blues Brothers." Rock Ridge (the UNIX extensions to ISO 9660) is named after the (fictional) town in the movie "Blazing Saddles." <sect2><tt/UDF/ <nidx>disk!file system!UDF</nidx> With the arrival of DVD with up to about 17 GB of storage capacity the world seemingly needed another format, this time ambitiously named Universal Disk Format (UDF). This is intended to replace <tt/iso9660/ and will be required for DVD and is available in modern Linux kernels.  <sect1>Networking File Systems <nidx>disk!file system!networking file systems</nidx> There is a large number of networking technologies available that lets you distribute disks throughout a local or even global networks. This is somewhat peripheral to the topic of this HOWTO but as it can be used with local disks I will cover this briefly. It would be best if someone (else) took this into a separate HOWTO... <sect2><tt/NFS/ <nidx>disk!file system!NFS</nidx> This is one of the earliest systems that allows mounting a file space on one machine onto another. There are a number of problems with <tt/NFS/ ranging from performance to security but it has nevertheless become established. <sect2><tt/AFS/ <nidx>disk!file system!AFS</nidx> Also known as Andrew File System, it allows efficient sharing of files across large networks. Starting out as an academic project it is now sold by <url url="http://www-306.ibm.com/software/stormgmt/afs/" name="IBM"> whose home page gives you more details. AFS also branched into open source, more information is available on <url url="http://www.openafs.org" name="OpenAFS home page">. Derek Atkins, of MIT, ported AFS to Linux and has also set up the Linux AFS mailing List ( <htmlurl url="mailto:linux-afs@mit.edu" name="linux-afs@mit.edu">) for this which is open to the public. Requests to join the list should go to <htmlurl url="mailto:linux-afs-request@mit.edu" name="linux-afs-request@mit.edu"> and finally bug reports should be directed to <htmlurl url="mailto:linux-afs-bugs@mit.edu" name="linux-afs-bugs@mit.edu">. Important: as AFS uses encryption it is restricted software and cannot easily be exported from the US. IBM who owns Transarc, has announced the availability of the latest version of client as well as server for Linux. Arla is a free AFS implementation, check the <url url="http://www.stacken.kth.se/projekt/arla/" name="Arla homepage"> for more information as well as documentation. <sect2>Coda <nidx>disk!file system!Coda</nidx>   A networking filesystem similar to <tt/AFS/ is underway and is called <url url="http://coda.cs.cmu.edu/" name="Coda">. This is designed to be more robust and fault tolerant than <tt/AFS/, and supports mobile, disconnected operations. Currently it does not scale very well, and does not really have proper administrative tools, as <tt/AFS/ does and <tt/ARLA/ is beginning to. <sect2><tt/nbd/ <nidx>disk!file system!nbd</nidx> <nidx>disk!device!network block device</nidx> The <url url="http://atrey.karlin.mff.cuni.cz/˜pavel/nbd/nbd.html" name="Network Block Device"> <url url="http://nbd.sourceforge.net/" name="(Sourceforge project pages)"> (<tt/nbd/) is available in Linux kernel 2.2 and later and offers reportedly excellent performance. The interesting thing here is that it can be combined with RAID (see later). <sect2><tt/enbd/ <nidx>disk!file system!enbd</nidx> <nidx>disk!device!enhanced network block device</nidx> The  <url url="http://www.it.uc3m.es/˜ptb/nbd/" name="Enhanced Network Block Device"> (<tt/enbd/) is a project to enhance the <tt/nbd/ with features such as block journaled multi channel communications, internal failover and automatic balancing between channels and more. The intended use is for RAID over the net. <sect2>GFS <nidx>disk!file system!GFS</nidx> <nidx>disk!device!Global File System</nidx> The  <url url="http://www.redhat.com/en_us/USA/home/solutions/gfs/" name="Global File System"> is a file system designed for storage across a wide area network. <sect1>Special File Systems In addition to the general file systems there is also a number of more specific ones, usually to provide higher performance or other features, usually with a tradeoff in other respects. <sect2><tt/tmpfs/ and <tt/swapfs/ <label id="tmpfs"> <nidx>disk!file system!tmpfs</nidx> <nidx>disk!file system!swapfs</nidx> For short term fast file storage SunOS offers <tt/tmpfs/ which is about the same as the <tt/swapfs/ on NeXT. This overcomes the inherent slowness in <tt/ufs/ by caching file data and keeping control information in memory. This means that data on such a file system will be lost when rebooting and is therefore mainly suitable for <tt>/tmp</tt> area but not <tt>/var/tmp</tt> which is where temporary data that must survive a reboot, is placed. SunOS offers very limited tuning for <tt/tmpfs/ and the number of files is even limited by total physical memory of the machine.  Linux now features <tt/tmpfs/ since kernel version 2.4 and is enabled by turning on virtual memory file system support (former <tt/shm fs/ ). Under certain circumstances <tt/tmpfs/ can lock up the system in early kernel versions, make sure you use version 2.4.6 or later. Note that <tt/tmpfs/ is a filesystem which means formatting is not needed. This in contrast to block devices that must be partitioned and formatted before use. <sect2><tt/userfs/ <nidx>disk!file system!userfs</nidx> <nidx>disk!file system!arcfs</nidx> <nidx>disk!file system!docfs</nidx> The user file system (<tt/userfs/) allows a number of extensions to traditional file system use such as FTP based file system, compression (<tt/arcfs/) and fast prototyping and many other features. The <tt/docfs/ is based on this filesystem. Check the <url url="http://www.goop.org/˜jeremy/userfs/" name="userfs homepage"> for more information. <sect2><tt/devfs/ <nidx>disk!file system!devfs</nidx> When disks are added, removed or just fail it is likely that disk device names of the remaining disks will change. For instance if <tt/sdb/ fails then the old <tt/sdc/ becomes <tt/sdb/, the old <tt/sdc/ becomes <tt/sdb/ and so on. Note that in this case <tt/hda/, <tt/hdb/ etc will remain unchanged. Likewise if a new drive is added the reverse may happen. There is no guarantee that SCSI ID 0 becomes <tt/sda/ and that adding disks in increasing ID order will just add a new device name without renaming previous entries, as some SCSI drivers assign from ID 0 and up while others reverse the scanning order. Likewise adding a SCSI host adapter can also cause renaming. Generally device names are assigned in the order they are found. The source of the problem lies in the limited number of bits available for major and minor numbering in the device files used to describe the device itself. You an see these in the  <htmlurl url="file:///dev/" name="/dev"> directory, info on the numbering and allocation can be found in <tt/man MAKEDEV/. Currently there are 2 solutions to this problem in various stages of development: <descrip> <tag/scsidev/ works by creating a database of drives and where they belong, check <em/ man scsifs/ and the <htmlurl url="http://www.garloff.de/kurt/linux/scsidev/" name="scsidev home page"> for more information <tag/devfs/ is a more long term project aimed at getting around the whole business of device numbering by making the  <htmlurl url="file:///dev/" name="/dev"> directory a kernel file system in the same way as  <htmlurl url="file:///proc/" name="/proc"> is. More information will appear as it becomes available. </descrip> <sect2><tt/smugfs/ <nidx>disk!file system!smugfs</nidx> <nidx>disk!file system!huge files</nidx> For a number of reasons it is currently difficult to have files bigger than 2 GB. One file system that tries to overcome this limit is <tt/smugfs/ which is very fast but also simple. For instance there are no directories and the block allocation is simple. It is available as  <url url="ftp://atrey.karlin.mff.cuni.cz/pub/local/mj/linux/" name="compressed tarred source code"> and while it worked with kernel version 2.1.85 it is quite possible some work is required to make it fit into newer kernels. Also the low version number (0.0) suggests extra care is required. <sect1>File System Recommendations There is a jungle of choices but generally it is recommended to use the general file system that comes with your distribution. If you use <tt/ufs/ and have some kind of <tt/tmpfs/ available you should first start off with the general file system to get an idea of the space requirements and if necessary buy more RAM to support the size of <tt/tmpfs/ you need. Otherwise you will end up with mysterious crashes and lost time. If you use dual boot and need to transfer data between the two OSes one of the simplest ways is to use an appropriately sized partition formatted with <tt/fat/ as most systems can reliably read and write this. Remember the limit of 2 GB for <tt/fat/ partitions. For more information of file system interconnectivity you can check out the  <url url="http://students.ceid.upatras.gr/˜gef/fs/oldindex.html" name="file system"> page which has been superseded by <url url="http://www.penguin.cz/˜mhi/fs/" name="file system"> and the article  <url url="http://www.linuxtoday.com/news_story.php3?ltsn=1999-05-02-006-10-NW-SM" name="Kragen's Amazing List of Filesystems">. That guide is being superseded by a HOWTO which is underway and a link will be added when it is ready. To avoid total havoc with device renaming if a drive fails check out the scanning order of your system and try to keep your root system on <tt/hda/ or <tt/sda/ and removable media such as ZIP drives at the end of the scanning order. <sect>Technologies <label id="technologies"> <nidx>disk!technologies</nidx> In order to decide how to get the most of your devices you need to know what technologies are available and their implications. As always there can be some tradeoffs with respect to speed, reliability, power, flexibility, ease of use and complexity. Many of the techniques described below can be stacked in a number of ways to maximise performance and reliability, though at the cost of added complexity. <sect1>RAID<label id="RAID"> <nidx>disk!technologies!RAID</nidx> This is a method of increasing reliability, speed or both by using multiple disks in parallel thereby decreasing access time and increasing transfer speed. A checksum or mirroring system can be used to increase reliability. Large servers can take advantage of such a setup but it might be overkill for a single user system unless you already have a large number of disks available. See other documents and FAQs for more information. For Linux one can set up a RAID system using either software (the <tt>md</tt> module in the kernel), a Linux compatible controller card (PCI-to-SCSI) or a SCSI-to-SCSI controller. Check the documentation for what controllers can be used. A hardware solution is usually faster, and perhaps also safer, but comes at a significant cost. A summary of available hardware RAID solutions for Linux is available at <url url="http://www.Linux-Consulting.com/Raid/Docs/raid_hw.txt" name="Linux Consulting">. <sect2>SCSI-to-SCSI<label id="SCSI-to-SCSI"> <nidx>disk!technologies!RAID!SCSI-to-SCSI</nidx> SCSI-to-SCSI controllers are usually implemented as complete cabinets with drives and a controller that connects to the computer with a second SCSI bus. This makes the entire cabinet of drives look like a single large, fast SCSI drive and requires no special RAID driver. The disadvantage is that the SCSI bus connecting the cabinet to the computer becomes a bottleneck. A significant disadvantage for people with large disk farms is that there is a limit to how many SCSI entries there can be in the  <htmlurl url="file:///dev/" name="/dev"> directory. In these cases using SCSI-to-SCSI will conserve entries. Usually they are configured via the front panel or with a terminal connected to their on-board serial interface. One manufacturer of such systems is <url url="http://www.cmd.com" name="CMD">.  <sect2>PCI-to-SCSI<label id="PCI-to-SCSI"> <nidx>disk!technologies!RAID!PCI-to-SCSI</nidx> PCI-to-SCSI controllers are, as the name suggests, connected to the high speed PCI bus and is therefore not suffering from the same bottleneck as the SCSI-to-SCSI controllers. These controllers require special drivers but you also get the means of controlling the RAID configuration over the network which simplifies management. Currently only a few families of PCI-to-SCSI host adapters are supported under Linux. <descrip> <tag/DPT/ The oldest and most mature is a range of controllers from <url url="http://www.dpt.com" name="DPT"> including SmartCache I/III/IV and SmartRAID I/III/IV controller families. These controllers are supported by the EATA-DMA driver in the standard kernel.  More information from the author of the DPT controller drivers (EATA* drivers) can be found at his pages on  <url url="http://www.staff.uni-mainz.de/neuffer/scsi/" name="SCSI"> and <url url="http://www.staff.uni-mainz.de/neuffer/scsi/dpt/" name="DPT">. These are not the fastest but have a good track record of proven reliability. Note that the maintenance tools for DPT controllers currently run under DOS/Win only so you will need a small DOS/Win partition for some of the software. This also means you have to boot the system into Windows in order to maintain your RAID system. <tag/ICP-Vortex/ A range of controllers is available from <url url="http://www.icp-vortex.com" name="ICP-Vortex"> featuring up to 5 independent channels and very fast hardware based on intelligent controllers. The Linux driver was written by the company itself which shows they support Linux. As ICP-Vortex supplies the maintenance software for Linux it is not necessary with a reboot to other operating systems for the setup and maintenance of your RAID system. This saves you also extra downtime. <tag/Mylex DAC-960/ This is now included in recent kernels. More information is available at <url url="http://dandelion.sourceforge.net/DAC960.html" name="Dandelion Digital's Linux DAC960 Page">. <tag/Compaq Smart-2 PCI Disk Array Controllers/ <url url="http://www.cpqlinux.com/" name="Smart-2"> drivers are available from for Linux. <tag/IBM ServeRAID/ IBM has released their driver as GPL and is now included in recent kernels. </descrip> <sect2>ATARAID<label id="ATARAID"> <nidx>disk!technologies!RAID!ATARAID</nidx> Hardware RAID support for ATA drives are now available from a number of companies, notably Promise and Highpoint. These are supported in kernel 2.4 and later. For more information check out the <url url="http://www.tldp.org/HOWTO/ATA-RAID-HOWTO/" name="Linux ATA RAID HOWTO"> for more information.  <sect2>Software RAID<label id="soft-raid"> <nidx>disk!technologies!RAID!Software RAID</nidx> A number of operating systems offer software RAID using ordinary disks and controllers. Cost is low and performance for raw disk IO can be very high. As this can be very CPU intensive it increases the load noticeably so if the machine is CPU bound in performance rather then IO bound you might be better off with a hardware PCI-to-RAID controller. Real cost, performance and especially reliability of software vs. hardware RAID is a very controversial topic. Reliability on Linux systems have been very good so far. The current software RAID project on Linux is the <tt/md/ system (multiple devices) which offers much more than RAID so it is described in more details later. <sect2>RAID Levels<label id="raid-levels"> <nidx>disk!technologies!RAID!RAID levels</nidx> RAID comes in many levels and flavours which I will give a brief overview of this here. Much has been written about it and the interested reader is recommended to read more about this in the <url url="http://unthought.net/Software-RAID.HOWTO/" name="Software RAID HOWTO">. <itemize> <item>RAID <em/0/ is not redundant at all but offers the best throughput of all levels here. Data is striped across a number of drives so read and write operations take place in parallel across all drives. On the other hand if a single drive fail then everything is lost. Did I mention backups? <item>RAID <em/1/ is the most primitive method of obtaining redundancy by duplicating data across all drives. Naturally this is massively wasteful but you get one substantial advantage which is fast access. The drive that access the data first wins. Transfers are not any faster than for a single drive, even though you might get some faster read transfers by using one track reading per drive. Also if you have only 2 drives this is the only method of achieving redundancy. <item>RAID <em/2/ and <em/4/ are not so common and are not covered here. <item>RAID <em/3/ uses a number of disks (at least 2) to store data in a striped RAID 0 fashion. It also uses an additional redundancy disk to store the XOR sum of the data from the data disks. Should the redundancy disk fail, the system can continue to operate as if nothing happened. Should any single data disk fail the system can compute the data on this disk from the information on the redundancy disk and all remaining disks. Any double fault will bring the whole RAID set off-line. RAID 3 makes sense only with at least 2 data disks (3 disks including the redundancy disk). Theoretically there is no limit for the number of disks in the set, but the probability of a fault increases with the number of disks in the RAID set. Usually the upper limit is 5 to 7 disks in a single RAID set. Since RAID 3 stores all redundancy information on a dedicated disk and since this information has to be updated whenever a write to any data disk occurs, the overall write speed of a RAID 3 set is limited by the write speed of the redundancy disk. This, too, is a limit for the number of disks in a RAID set. The overall read speed of a RAID 3 set with all data disks up and running is that of a RAID 0 set with that number of data disks. If the set has to reconstruct data stored on a failed disk from redundant information, the performance will be severely limited: All disks in the set have to be read and XOR-ed to compute the missing information. <item>RAID <em/5/ is just like RAID 3, but the redundancy information is spread on all disks of the RAID set. This improves write performance, because load is distributed more evenly between all available disks. Parity data is rotated across all disks so total net storage equals all disks minus 1. <item>RAID <em/6/ is similar to RAID 5 except that there is twice the redundancy and the array can survive 2 failed drives. Parity data is also rotated across all disks so total net storage equals all disks minus 2. </itemize> There are also hybrids available based on RAID 0 or 1 and one other level. Many combinations are possible but I have only seen a few referred to. These are more complex than the above mentioned RAID levels. RAID 01 combines striping with duplication as mirrored arrays of striped arrays which gives very high transfers combined with fast seeks as well as redundancy. The disadvantage is high disk consumption as well as the above mentioned complexity. Also a single disk failure turns the array into RAID 0. RAID 1+0 combines striping with duplication as striped arrays of mirrored arrays which gives very high transfers combined with fast seeks as well as redundancy. The disadvantage is high disk consumption as well as the above mentioned complexity. RAID 1/5 combines the speed and redundancy benefits of RAID5 with the fast seek of RAID1. Redundancy is improved compared to RAID 0/1 but disk consumption is still substantial. Implementing such a system would involve typically more than 6 drives, perhaps even several controllers or SCSI channels. <sect1>Volume Management<label id="vol-mgmnt"> <nidx>disk!technologies!volume management</nidx> Volume management is a way of overcoming the constraints of fixed sized partitions and disks while still having a control of where various parts of file space resides. With such a system you can add new disks to your system and add space from this drive to parts of the file space where needed, as well as migrating data out from a disk developing faults to other drives before catastrophic failure occurs. The system developed by <url url="http://www.veritas.com" name="Veritas"> has become the defacto standard for logical volume management. Volume management is for the time being an area where Linux is lacking. One is the virtual partition system project  <url url="http://www-wsg.cso.uiuc.edu/˜roth/" name="VPS"> that will reimplement many of the volume management functions found in IBM's AIX system. Unfortunately this project is currently on hold. Another project is the   <url url="http://sources.redhat.com/lvm2/" name="Logical Volume Manager 2"> that superceeded <url url="http://sources.redhat.com/lvm/" name="Logical Volume Manager"> which was used up to 2.4 kernels but now is no longer being developed. More documentation on LVM2 can be found in the <url url="http://www.tldp.org/HOWTO/LVM-HOWTO/index.html" name="LVM-HOWTO">. Also there is the <url url="http://evms.sourceforge.net/" name="Enterprise Volume Management System"> from IBM. More documentation on EVMS can be found in the <url url="http://www.tldp.org/LDP/EVMSUG/html/index.html" name="EVMS USer Guide">. <sect1>Linux <tt/md/ Kernel Patch <nidx>disk!technologies!md</nidx> <nidx>disk!technologies!raid</nidx> <nidx>disk!technologies!striping</nidx> <nidx>disk!technologies!translucence</nidx> The Linux Multi Disk (md) provides a number of block level features in various stages of development. RAID 0 (striping) and concatenation are very solid and in production quality and also RAID 4 and 5 are quite mature. It is also possible to stack some levels, for instance mirroring (RAID 1) two pairs of drives, each pair set up as striped disks (RAID 0), which offers the speed of RAID 0 combined with the reliability of RAID 1. In addition to RAID this system offers (in alpha stage) block level volume management and soon also translucent file space. Since this is done on the block level it can be used in combination with any file system, even for <tt/fat/ using Wine. Think very carefully what drives you combine so you can operate all drives in parallel, which gives you better performance and less wear. Read more about this in the documentation that comes with <tt/md/.  Unfortunately The Linux software RAID has split into two trees, the old stable versions 0.35 and 0.42 which are documented in the official  <url url="http://www.tldp.org/HOWTO/Software-RAID-0.4x-HOWTO.html" name="Software-RAID HOWTO"> and the newer less stable 0.90 series which is documented in the unofficial <url url="http://ostenfeld.dk/˜jakob/Software-RAID.HOWTO/" name="Software RAID HOWTO"> which is a work in progress. A <url url="http://www-mddsp.enel.ucalgary.ca/People/adilger/online-ext2/" name="patch for online growth of ext2fs"> is available in early stages and related work is taking place at <url url="http://ext2resize.sourceforge.net/" name="the ext2fs resize project"> at Sourceforge.  Hint: if you cannot get it to work properly you have forgotten to set the <tt/persistent-block/ flag. Your best documentation is currently the source code. <sect1>Compression <nidx>disk!technologies!compression</nidx> <nidx>disk!compression!DouBle</nidx> <nidx>disk!compression!Zlibc</nidx> <nidx>disk!compression!dmsdos</nidx> <nidx>disk!compression!e2compr</nidx> Disk compression versus file compression is a hotly debated topic especially regarding the added danger of file corruption. Nevertheless there are several options available for the adventurous administrators. These take on many forms, from kernel modules and patches to extra libraries but note that most suffer various forms of limitations such as being read-only. As development takes place at neck breaking speed the specs have undoubtedly changed by the time you read this. As always: check the latest updates yourself. Here only a few references are given. <itemize> <item>DouBle features file compression with some limitations. <item>Zlibc adds transparent on-the-fly decompression of files as they load. <item>there are many modules available for reading compressed files or partitions that are native to various other operating systems though currently most of these are read-only. <item>  <url url="http://cmp.felk.cvut.cz/˜pisa/dmsdos/" name="dmsdos"> (currently in version 0.9.2.0) offer many of the compression options available for DOS and Windows. It is not yet complete but work is ongoing and new features added regularly. <item><tt/e2compr/ is a package that extends <tt>ext2fs</tt> with compression capabilities. It is still under testing and will therefore mainly be of interest for kernel hackers but should soon gain stability for wider use. Check the   <url url="http://e2compr.sourceforge.net/" name="e2compr homepage"> for more information. I have reports of speed and good stability which is why it is mentioned here. </itemize> <sect1>ACL <nidx>disk!technologies!ACL</nidx> Access Control List (ACL) offers finer control over file access on a user by user basis, rather than the traditional owner, group and others, as seen in directory listings (<tt/drwxr-xr-x/). This is currently not available in Linux but is expected in kernel 2.3 as hooks are already in place in <tt/ext2fs/. <sect1><tt/cachefs/ <nidx>disk!technologies!cachefs</nidx> This uses part of a hard disk to cache slower media such as CD-ROM. It is available under SunOS and under development for Linux. For more information see <url url="http://people.redhat.com/˜dhowells/cachefs/" name="cachefs development page"> at Redhat. <sect1>Translucent or Inheriting File Systems <nidx>disk!technologies!translucent</nidx> <nidx>disk!technologies!inheriting</nidx> This is a copy-on-write system where writes go to a different system than the original source while making it look like an ordinary file space. Thus the file space inherits the original data and the translucent write back buffer can be private to each user. There is a number of applications: <itemize> <item>updating a live file system on CD-ROM, making it flexible, fast while also conserving space, <item>original skeleton files for each new user, saving space since the original data is kept in a single space and shared out, <item>parallel project development prototyping where every user can seemingly modify the system globally while not affecting other users. </itemize> SunOS offers this feature and this is under development for Linux. There was an old project called the Inheriting File Systems (<tt/ifs/) but this project has stopped. One current project is part of the <tt/md/ system and offers block level translucence so it can be applied to any file system.  It should be noted that <url url="http://www-306.ibm.com/software/rational/" name="Clearcase (now owned by IBM)"> pioneered and popularized translucent filesystems for software configuration management by writing their own UNIX filesystem. A <url url="http://translucency.sourceforge.net" name="Translucency project"> has started, more information can be found there. <!-- This is the old section, from which I have moved various parts to other sections. <sect2>General File System Consideration <nidx>disk!technologies!filesystem considerations</nidx> In the Linux world <tt>ext2fs</tt> is well established as a general purpose system. Still for some purposes others can be a better choice. News spools lend themselves to a log file based system whereas high reliability data might need other formats. This is a hotly debated topic and there are currently few choices available but work is underway. Log file systems also have the advantage of very fast file checking. Mail servers in the 100 GB class can suffer file checks taking several days before becoming operational after rebooting. The <tt/Minix/ file system is the oldest one, used in some rescue disk systems but otherwise very little used these days. At one time the <tt/Xiafs/ was a strong contender to the standard for Linux but seems to have fallen behind these days. Adam Richter from Yggdrasil posted recently that they have been working on a compressed log file based system but that this project is currently on hold. Nevertheless a non-working version is available on their FTP server. Check out <url url="ftp://ftp.yggdrasil.com/private/adam" name="the Yggdrasil ftp server"> where special patched versions of the kernel can be found. Hopefully this will be rolled into the mainstream kernel in the near future. An alternative project is the <url url="http://lucien.blight.com/˜c-cook/prof/lfs/" name="Logical Volume Manager"> project. As of July, 23th 1997 <url url="mailto:reiser (at) RICOCHET.NET" name="Hans Reiser"> has put up the source to his tree based <url url="http://idiom.com/˜beverly/reiserfs.html" name="reiserfs"> on the web. While his filesystem has some very interesting features and is much faster than <tt/ext2fs/, it is still very experimental and difficult to integrate with the standard kernel. Expect some interesting developments in the future - this is different from your "average log based file system for Linux" project, because Hans already has working code. There is room for access control lists (ACL) and other unimplemented features in the existing <tt>ext2fs</tt>, stay tuned for future updates. There is also an encrypted file system available but again as this is under export control from the US, make sure you get it from a legal place. Also under development is the <url url="http://www.virtual.net.au/˜rjc/enh-fs.html" name="Enhanced File System"> project. File systems is an active field of academic and industrial research and development, the results of which are quite often freely available. Linux has in many cases been a development tool in such activities so you can expect a lot of continuous work in this field, stay tuned for the latest development. One example of a file system research is <url url="http://www.cs.columbia.edu/˜ezk/research" name="Erez Zadok Research"> page. <sect2>CD-ROM File Systems <nidx>disk!technologies!CD-ROM filesystems</nidx> There has been a number of file systems available for use on CD-ROM systems and one of the earliest one was the <em/High Sierra/ format, supposedly named after the hotel where the final agreement took place. This was the precursor to the <em/ISO 9660/ format which is supported by Linux. Later there were the <em/Rock Ridge/ extensions which added file system features such as long filenames, permissions and more. The Linux iso9660 file system supports both High Sierra as well as Rock Ridge extensions. However, once again Microsoft decided it should create another standard and their latest effort here is called <em/Joliet/ and offers some internationalisation features. This is now available in linux kernel 2.0.34 or newer. You need to enable NLS in order to use it. In a recent Usenet News posting hpa (at) transmeta.com (H. Peter Anvin) writes the following the following interesting piece of trivia: <tscreen><verb> Trivia: Joliet is a city outside Chicago; best known for being the site of the prison where Jake was locked up in the movie "Blues Brothers." Rock Ridge (the UNIX extensions to ISO 9660) is named after the (fictional) town in the movie "Blazing Saddles." </verb></tscreen> Very important note: it was actually Jake who was locked up. Oops. <sect2>Compression <nidx>disk!technologies!compression</nidx> Disk compression versus file compression is a hotly debated topic especially regarding the added danger of file corruption. Nevertheless there are several options available for the adventurous administrators. These take on many forms, from kernel modules and patches to extra libraries but note that most suffer various forms of limitations such as being read-only. As development takes place at neck breaking speed the specs have undoubtedly changed by the time you read this. As always: check the latest updates yourself. Here only a few references are given. <itemize> <item>DouBle features file compression with some limitations. <item>Zlibc adds transparent on-the-fly decompression of files as they load. <item>dmsdos (currently in version 0.9.1.2) offer many of the compression options available for DOS and Windows. It is not yet complete but work is ongoing and new features added regularly. <item><tt/e2compr/ is a package that extends <tt>ext2fs</tt> with compression capabilities. It is still under testing and will therefore mainly be of interest for kernel hackers but should soon gain stability for wider use. Check the <url url="http://netspace.net.au/˜reiter/e2compr.html" name="e2compr homepage"> for more information. I have reports of speed and good stability which is why it is mentioned here. </itemize> <sect2>Other Filesystems <nidx>disk!technologies!filesystems, other</nidx> Also there is the user file system (<tt/userfs/) that allows FTP based file system and some compression (<tt/arcfs/) plus fast prototyping and many other features. The <tt/docfs/ is based on this filesystem. Recent kernels feature the loop or loopback device which can be used to put a complete file system within a file. There are some possibilities for using this for making new file systems with compression, tarring, encryption etc. Note that this device is unrelated to the network loopback device. Very recently a compression package that extends <tt>ext2fs</tt> was announced. It is still under testing and will therefore mainly be of interest for kernel hackers but should soon gain stability for wider