Solaris10 Tech FAQ
From Genunix
SMF - Service Management Facility
Question
Answer
Zones
What are the benefits of vitualization and what are the different types of virtualization techniques available?
Ans:Virtalization dramatically reduces the cost of deploying and maintaining multiple machines and applications.The most common need for virtualization is application consolidation. Many of the larger applications have become so complex that they become a system in themselves - and often they don't play nicely with other applications on the box. So "one app per machine" has become the common paradigm. The second most common need is security, either for your application administrators or your developers. Other reasons certainly exist (rapid test environment deployment, distributed system simulation, etc), but these are the two primary ones.
So what does virtualization buy us? It's all about reducing costs, but there are really two types of cost associated with running a system:
Hardware costs - This includes the cost of the machine, but also the costs associated with running that machine (power, A/C).
Software management costs - This includes the cost of deploying new machines, and upgrading/patching software, and observing software behavior.
As we'll see, different virtualization strategies provide different qualities of the above savings.
One of the most well-established forms of virtualization, the most common examples today are Sun Domains and IBM Logical Partitions. In each case, the hardware is responsible for dividing existing resources in such a way as to present multiple machines to the user. This has the advantage of requiring no software layer, no performance impact, and hardware fault isolation. The downside to this is that it requires specialized hardware that is extremely expensive, and provides zero benefit for reducing software management costs.
Software machine virtualization
This approach is probably the one most commonly associated with the term "virtualization". In this scheme, a software layer is created which allows multiple OS instances to run on the same hardware. The most commercialized versions are VMware and Virtual PC, but other projects exist (such as qemu and PearPC). Typically, they require a "host" operating system as well as multiple "guests" (although VMware ESX server runs a custom kernel as the host). While Xen uses a para-vitualization technique that requires changes to the guest OS, it is still fundamentally a machine virtualization technique. And Usermode Linux takes a radically different approach, but accomplishes the basic same task.
In the end, this approach has similar strengths and weaknesses as the hardware assisted virtualization. You don't have to buy expensive special-purpose hardware, but you give up the hardware fault isolation and often sacrifice performance (Xen's approach lessens this impact, but its still visible). But most importantly, you still don't save any costs associated with software management - administering software on 10 virtual machines is just as expensive as administering 10 separate machines. And you have no visibility into what's happening within the virtual machine - you may be able to tell that Xen is consuming 50% of your CPU, but you can't tell why unless you log into the virtual system itself.
Software application virtualization
On the grand scale of virtualization, this ranks as the "least virtualized". With this approach, the operating system uses various tricks and techniques to present an alternate view of the machine. This can range from simple chroot(1), to BSD Jails, to Solaris Zones. Each of these provide a more complete OS view with varying degrees of isolation. While Zones is the most complete and the most secure, they all use the same fundamental idea of a single operating system presenting an "alternate reality" that appears to be a complete system at the application level. The upcoming Linux Application Environment on OpenSolaris will take this approach by leveraging Zones and emulating Linux at the system call layer.
The most significant downside to this approach is the fact there is a single kernel. You cannot run different operating systems (though LAE will add an interesting twist), and the "guest" environments have limited access to hardware facilities. On the other hand, this approach results in huge savings on the software management front. Because applications are still processes within the host environment, you have total visibility into what is happening within each guest, using standard operating system tools, as well as manage them as you would any other processes, using standard resource management tools. You can deploy, patch, and upgrade software from a single point without having to physically log into each machine. While not all applications will run in such a reduced environment, those that do will be able to benefit from vastly simplified software management. This approach also has the added bonus that it tends to make better use of shared resources. In Zones, for example, the most common configuration includes a shared /usr directory, so that no additional disk space is needed (and only one copy of each library needs to be resident in memory).
What is the performance overhead of Zones?
Ans:CPU overhead of zones is hardly measurable (i.e. <1%) for a few zones or even dozens of zones, depending somewhat on the applications. Inside the kernel, zones are just integers embedded in the relevant data structures.The system still works as it always has in the past, but it has just one more integer comparison to make in certain circumstances.
How are LPARS different from Zones?
Ans: LPARS and Zones address different problems.They are contrasted as below:
- Platform Availability - Zones are supported on SPARC, x86 and x64 whereas, LPARs are extremely vendor specific
- Performance overhead - Zones offer 0 performance overhead for applications, as there is no virtualization layer that apps have to punch through. The overall system overhead for Zones is minimal, due to all the resource sharing.
- Managebability - LPARs do nothing for manageability of the datacenter, all they do is consolidate the hardware footprint. For a large percentage of applications, Zones resolve a large part of the management headache.
- Obserability -If an application in a LPAR is not behaving, there is no way for someone inside that OS instance to see whats going on around itself. You can't call someone up who can check the entire platform to try and diagnose the problem. With Zones, the global zone administrator has full visibility into all the local zones, and into the entire hardware platform, no virtualization.
-LPARs are sharply constrained in number of instances, since each LPAR requires dedicated RAM, rather than being paged in on demand as is possible with processes running in a zone. For example, if you have a machine with 32GB of RAM, you can have at most 8 instances of 4GB each size .So, LPAR memory management is completely static and coarse-grain. Redeploying RAM is rather disruptive. By contrast, hundreds of zones are possible on a machine because of the negligible memory footprint. This is aided by the fact that binaries used by multiple zones (sparse root zones )use the same in-memory copy. So,you could have a hundred copies of your favorite application server orDBMS using the same text segments, with only one copy in RAM and a massive saving in overhead. Not to mention the potential of faster startup times for all but the first person to invoke a binary.
-These capabilities open use cases for zones that are not possible with coarse partitioning schemes like LPARs. For example: you can provide each web developer a zone with a private instance of Apache or Tomcat (or their equivalents) with their own IP address and ports 80 and 443. Or let them have multiple zones - at no cost - so they can build and test N-tier applications on the same machine, simply and effectively.
Another contrast is that you can also more easily manage zone contents - for example, you could configure zone file systems so application home directories are in a loopback file system (see lofiadm) that can be easily saved, backed up, and copied to different servers.
Is Kernel Processing limited to the pool(processor set ) to which the global zone is assigned ? What about interrupt handling for a process that initiated an IO? Will the interrupt get assigned to the processor in a pool of the process that initiated it?
Ans: No, the global zone isn't equal to the kernel.All system calls are performed on the CPU on which the process is running.No interrupts get assigned regardless of pooling. For example we can't tell, which I/O will cause the next interrupt for a SCSI disk as 100s of I/Os may be outstanding for that disk.Neither are the interrupts redirected this way. Only CPUs marked "NOINTR" will get no interrupts.
What parameters affect patching in zones?
Ans: Patching in a zones environment is affected by 3 pkginfo variables
SUNW_PKG_ALLZONES SUNW_PKG_HOLLOW SUNW_PKG_THISZONE
please see man -s 4 pkginfo for detailed information on these variables.
SUNW_PKG_ALLZONES=true means the package can only be installed from the global zone and will apply to the global and all non-global zones. This implies system wide changes.In this case an attempt to apply a patch with ALLZONES=true in a non-global zone will fail. If set to false,patchadd in the global applies the patch to the global and all non-global zone as well, and a patchadd run in a non-global zone will apply the patch to that zone only.
SUNW_PKG_HOLLOW=true means that the contents of the patch is applied to the global zone only, so therefore a patch that contains a package with this set can only be applied from the global zone. The actual patch information is propogated to all non-global zones so that even though the patch contents are not installed in non-global zones, the patch itself is visible to patchadd -p or showrev -p in all non-global zones ( as well as the global zone ). This is mainly for patch packages that patch files that only affect the global zone, i.e. files/binaries that are used to physically boot the system, or actual kernel related files/binaries. They are not used in non-gloabl zones, but in order to satisfy patch dependencies the patch will show as being installed in the non-gloabl zone as well. By default SUNW_PKG_ALLZONES must be true if SUNW_PKG_HOLLOW is also true So a SUNW_PKG_HOLLOW patch can only be applied from the global to the global and all non-global zones as well. A patchadd in the non-global zone will fail.
SUNW_PKG_THISZONE=true, means that the patch will only apply in the current running zone where patchadd/pkgadd is being run. ( be that the global or a non-global zone ). Used mainly for application level software. i.e. Java Enterprise System 4 ( JES 4 ) uses this so that the software is only installed in the zone where the installer is run from. ( if installing JES in the global, we probably don't want it to also install in all non-global zones for instance ).This variable is mutually exclusive of the first two, ie THISZONE cannot be true if ALLZONES=true.This variable also mimics the behaviour of doing a patchadd -G ( or a pkgadd -G ).
How to import raw and block devices into zones using zonecfg ?
Ans:The steps are as follows:
1) Change directories to /usr/tmp. global# cd /usr/tmp 2) Create a new UFS file system. global# mkfile 10m fsfile 3) Attach the file as a block device. global# lofiadm -a âpwdâ/fsfile You will also get the required character device.
4)Import the devices into the zone my-zone. global# zonecfg -z my-zone zonecfg:my-zone> add device zonecfg:my-zone:device> set match=/dev/rlofi/1 zonecfg:my-zone:device> end zonecfg:my-zone> add device zonecfg:my-zone:device> set match=/dev/lofi/1 zonecfg:my-zone:device> end 5) Reboot the zone. global# zoneadm -z my-zone reboot
6) Log in to the zone and verify that the devices were successfully imported. my-zone# ls -l /dev/*lofi/* You will see a display that is similar to this: brw------- 1 root sys 147, 1 Jan 7 11:26 /dev/lofi/1 crw------- 1 root sys 147, 1 Jan 7 11:26 /dev/rlofi/1
7) In the zone my-zone, create a new file system on the disk.
my-zone# newfs /dev/lofi/1
8) Respond yes at the prompt.
newfs: construct a new file system /dev/rlofi/1: (y/n)? y
You will see a display that is similar to this:
/dev/rlofi/1: 20468 sectors in 34 cylinders of 1 tracks, 602 sectors
10.0MB in 3 cyl groups (16 c/g, 4.70MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 9664, 19296,
9) Check the file system for errors.
my-zone# fsck -F ufs /dev/rlofi/1
You will see a display that is similar to this:
** /dev/rlofi/1
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2 files, 9 used, 9320 free (16 frags, 1163 blocks, 0.2% fragmentation)
10) Mount the file system.
my-zone# mount -F ufs /dev/lofi/1 /mnt
11) Verify the mount.
my-zone# grep /mnt /etc/mnttab
You will see a display similar to this:
/dev/lofi/1 /mnt ufs
rw,suid,intr,largefiles,xattr,onerror=panic,zone=foo,dev=24c0001
1073503869
How to save the configuration of a zone in a file and use it to create similar zones
Ans: The steps are as follows:
1) Become the super-user.
2) Print the configuration of the zone to a file.
bash-3.00# zonecfg -z myzone1 export > myzone1.config
3) Specify that myzone.config be used as the zonecfg command file to recreate
similar zones.
bash-3.00# zonecfg -z myzone2 -f myzone1.config
bash-3.00# zonecfg -z myzone2 info
zonepath: /export/myzone1
autoboot: false
pool:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
net:
address: 129.158.233.113/24
physical: bge0
device
match: /dev/lofictl
device
match: /dev/lofictl/*
device
match: /dev/lofi/1
device
match: /dev/rlofi/1
Change the zonepath to /export/myzone2
How to figure out from within a non-global zone whether it is a whole-root zone or a sparse-root zone ?
Ans:There is a command line tool called pkgcond ( will need patch 119254-14 or higher, or Solaris 10 update 1 ). Inside a sparse-root zone, the following output is expected:
# pkgcond is_whole_root_nonglobal_zone
# echo $?
1
# pkgcond is_sparse_root_nonglobal_zone
# echo $?
0
How can we have a read/write directory inside a IPD (Inherited Package Directory) or a readonly mount inside a zone
Often we need to have read/write directories inside the IPDs so that we can install packages in a specific sparse root zone. Consider a package which installs its binaries in /usr/local. Since /usr is readonly inside a sparse root zone /usr/local will also be readonly. But we can have a subdirectory of IPD which is writable, following the steps documented below:
No need for this step if you have a free slice
# mkfile 100m /space/zones/local # lofiadm -a /space/zones/local /dev/lofi/2
Configure zone for adding a writable filesystem inside an IPD:
# zonecfg -z zone2
zonecfg:zone2> add fs
zonecfg:zone2:fs> set dir=/usr/local
zonecfg:zone2:fs> set special=/dev/lofi/2
zonecfg:zone2:fs> set raw=/dev/rlofi/2
zonecfg:zone2:fs> set type=ufs
zonecfg:zone2:fs> end
zonecfg:zone2> commit
zonecfg:zone2> info
zonename: zone2
zonepath: /space/zones/zone2
autoboot: false
bootargs:
pool:
limitpriv:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
fs:
dir: /usr/local
special: /dev/lofi/2
raw: /dev/rlofi/2
type: ufs
options: []
zonecfg:zone2> exit
You need to create a /usr/local since zone utilities wont be able to create it inside the IPD as it is readonly:
# mkdir /usr/local/
# newfs /dev/lofi/2
newfs: construct a new file system /dev/rlofi/2: (y/n)? y
/dev/rlofi/2: 204600 sectors in 341 cylinders of 1 tracks, 600 sectors
99.9MB in 22 cyl groups (16 c/g, 4.69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632
Now Install the zone:
# zoneadm -z zone2 install Preparing to install zone <zone2>. Checking <ufs> file system on device </dev/rlofi/2> to be mounted at </space/zones/zone2/root> Creating list of files to copy from the global zone. Copying <2066> files to the zone. Initializing zone product registry. Determining zone package initialization order. Preparing to initialize <977> packages on the zone. Initialized <977> packages on zone. Zone <zone2> is initialized. The file </space/zones/zone2/root/var/sadm/system/logs/install_log> contains a log of the zone installation.
Checking whether the /usr/local inside is writable:
# zoneadm -z zone2 boot # zlogin zone2 [Connected to zone 'zone2' pts/3] Sun Microsystems Inc. SunOS 5.11 snv_46 October 2007 # cd /usr # touch a touch: a cannot create # cd local # touch a # ls a lost+found
Customize your zone specific package so that it gets installed in /usr/local
Zones and ZFS
Ans:
root@v4u-60m-blr03:zpool create -f mypool c0t0d0s2
root@v4u-60m-blr03:zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool 16.8G 83.5K 16.7G 0% ONLINE -
root@v4u-60m-blr03:zfs create mypool/myzone-data
root@v4u-60m-blr03:zfs set compression=on mypool/myzone-data
root@v4u-60m-blr03:zfs set quota=30m mypool/myzone-data
root@v4u-60m-blr03:zonecfg -z myzone
myzone: No such zone configured Use 'create' to begin configuring a new zone.
zonecfg:myzone> create
zonecfg:myzone> set zonepath=/myzone
zonecfg:myzone> add dataset
zonecfg:myzone:dataset> set name=mypool/myzone-data
zonecfg:myzone:dataset> end
zonecfg:myzone> verify
zonecfg:myzone> commit
zonecfg:myzone> exit
root@v4u-60m-blr03:zoneadm -z myzone install
Preparing to install zone <myzone>.
Creating list of files to copy from the global zone.
Copying <2527> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1022> packages on the zone.
Initialized <1022> packages on zone.
Zone <myzone> is initialized.
Installation of these packages generated warnings: <SUNWcar SUNWkvm SUNWcsl SUNWcsr SUNWcsu SUNWopenssl-libraries SUNWxwplr SUNWcslr SUNWesu SUNWfontconfig-root SUNWcpcu SUNWkrbu SUNWmdu SUNWfmd SUNWtnfc SUNWmdb SUNWtnfd SUNWaudit SUNWrcmdc SUNWscpu SUNWdoc SUNWnfscr SUNWvolu SUNWpcu SUNWscplp SUNWwebminr SUNWapch2r SUNWarcr SUNWnisu SUNWsmapi SUNWzfsu SUNWpool SUNWpapi SUNWippcore SUNWipplr SUNWbindr SUNWpiclu SUNWfruid SUNWfruip SUNWppm SUNWpsu SUNWdhcsu SUNWdhcsb SUNWdmgtu SUNWman SUNWsasnm SUNWncau SUNWntpr SUNWperl-xml-parser SUNWperl584man SUNWpsm-ipp SUNWpsm-lpd SUNWsmcmd SUNWsndmr SUNWsrh SUNWstsfr>
The file </myzone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.
root@v4u-60m-blr03:zoneadm -z myzone boot
root@v4u-60m-blr03:zlogin myzone
[Connected to zone 'myzone' pts/4]
No utmpx entry. You must exec "login" from the lowest level "shell".
[Connection to zone 'myzone' pts/4 closed]
root@v4u-60m-blr03:zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool 104K 16.5G 24.5K /mypool
mypool/myzone-data 24.5K 30.0M 24.5K /mypool/myzone-data
Using ZFS for zone backups/snapshots
Using ZFS for temporary zone backups/snapshots works great
and is quick and easy.
Create a ZFS filesystem for zone1 and set its mountpoint
root@v4u-60m-blr03 zfs create pool1/zones/zone1
root@v4u-60m-blr03 zfs set mountpoint=/zones/zone1 pool1/zones/zone1
Configure the zone and set its zonepath=/zones/zone1
root@v4u-60m-blr03zonecfg -z zone1
root@v4u-60m-blr03zoneadm -z zone1 install
root@v4u-60m-blr03zoneadm -z zone1 boot
root@v4u-60m-blr03 zlogin -C zone1
root@v4u-60m-blr03 zoneadm list -vc
ID NAME STATUS PATH
0 global running /
3 zone1 running /zones/zone1
Take a snapshot of the zone just as it is and call it âtestâ, nothing created inside the zone yet
root@v4u-60m-blr03zfs snapshot pool1/zones/zone1@test
root@v4u-60m-blr03 zlogin zone1
[Connected to zone 'zone1' pts/2]
Last login: Mon Nov 28 14:35:20 on pts/1
Sun Microsystems Inc. SunOS 5.11 snv_27 October 2007
root@v4u-60m-blr03 ls bin etc home mnt opt proc system usr dev export lib net platform sbin tmp var
Make files in the zone, we will revert back to the old without these here in a minute
root@v4u-60m-blr03mkfile 50m testfile root@v4u-60m-blr03mkfile 100m testfile1
root@v4u-60m-blr03 ls bin export mnt platform system tmp dev home net proc testfile usr etc lib opt sbin testfile1 var
root@v4u-60m-blr03 exit
root@v4u-60m-blr03 zonename global
Now back in the global zone, lets rollback to the snapshot where it doesnât have the testfile and testfile1
root@v4u-60m-blr03 zoneadm -z zone1 halt root@v4u-60m-blr03zfs rollback -r pool1/zones/zone1@test root@v4u-60m-blr03zoneadm -z zone1 boot
This whole rollback procedure took about 5 seconds
Lets login to the zone and see if the files are gone and it rolled back successfully:
root@v4u-60m-blr03 zlogin zone1 [Connected to zone 'zone1' pts/2]
Last login: Mon Nov 28 14:35:20 on pts/1
Sun Microsystems Inc. SunOS 5.11 snv_27 October 2007
bin etc home mnt opt proc system usr dev export lib net platform sbin tmp var
Compare Zones on UFS Vs ZFS?
Ans: Zone on top of UFS bash> zonecfg -z test_ufs
test_ufs: No such zone configuredUse 'create' to begin configuring a new
zone.zonecfg:test_ufs> create
zonecfg:test_ufs> set zonepath=/ufstest
zonecfg:test_ufs> verify
zonecfg:test_ufs> commit
zonecfg:test_ufs>
bash> time zoneadm -z test_ufs install
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <2535> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1081> packages on the zone.
Initialized <1081> packages on zone.
Zone is initialized.
Installation of these packages generated warnings:
The file contains a log of the zone installation.
real 22m22.402s
user 5m20.384s
sys 7m58.420s
bash> chmod 700 /data/zone_zfs
bash> time zoneadm -z zone_zfs install
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <2535> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1081> packages on the zone.
Initialized <1081> packages on zone.
Zone is initialized.
Installation of these packages generated warnings:
The file contains a log of the zone installation.
real 16m23.287s
user 5m18.917s
sys 9m28.814s
Enable Compression and put a zone on it.
bash> zfs set compression=on data/zone_zfs2 bash> time zoneadm -z zone_zfs2 install
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <2535> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1081> packages on the zone.
Initialized <1081> packages on zone.
Zone is initialized.
Installation of these packages generated warnings:
The file contains a log of the zone installation.
real 16m3.941s
user 5m19.210s
sys 9m32.384s
Yes ZFS with compression on is even faster.
bash> zfs list
NAME USED AVAIL REFER MOUNTPOINT
data/zone_zfs 93.3M 26.0G 93.3M /data/zone_zfs
data/zone_zfs2 43.4M 26.0G 43.4M /data/zone_zfs2
data/zones 249M 26.0G 249M /data/zones
bash> zfs list -o name,compressratio
NAME RATIO
data/zone_zfs 1.00x
data/zone_zfs2 2.27x
(and even has impressive compression ratio)
