Snap Upgrade Test Plan
From Genunix
TBD for PSARC ID
1 Introduction
1.1 Authors
| Name |
|---|
| Jason.Zhao |
1.2 Test Sponsor
TBD
1.3 Test Plan Approval
| Title | Name | Version | Date |
|---|---|---|---|
| Development Lead | Ethan.Quach | ||
| Development Manager | Eric.Ray | ||
| Test Sponsor |
1.4 Open Source Project
- Is this an open source development project? Yes
1.5 Revision History
| Date | Revision | Comments | Approval/Status |
|---|---|---|---|
| 2007-12-17 | 0.1 | Initial Draft | In Work |
1.6 References
1.7 Staffing
| Name | Role | % Commitment | Duration |
|---|---|---|---|
| Jason Zhao | Test Lead | 80% | 5 months |
<<NOTE: The above table 1.7 Staffing represents the currently allocated head count. Staff will be added as needed to deliver the test results as defined by this test plan to the I-team and C-team for test execution. >>
1.8 Glossary
| Term | Definition |
|---|---|
| libbe | Libbe provides a set of functions to access and manipulate BEs. This library is intended to be used by the installer (as part of initial install and upgrade), the Image Packaging system, the BE utilities, and possibly other applications that require hooks into managing a BE. |
| Be commands | The BE utilities will provide the command line interface to replace the functionality used in Live Upgrade. The BE utilities will call into the BE library to perform its core tasks. |
| GUI installer | This new GUI installer is available for port UFS and ZFS archive feature when could enable users to input UFS backup archive block device. But for May release,Snap will not support regular upgrade through LiveCD but package upgrade by CLI. |
| IPS | The image packaging system, is an attempt to design and implement a software delivery system with interaction with a network repository as its primary design goal. |
| TI | Target Instantiation Service (or TI) will create ZFS for BE pool which provided by libbe, so that Installer can appropriately continue with installation or upgrade process. |
2 Test Program Summary
Our goal is to test and verify the functionality of BE utilities,libbe and package installation upgrade process. We plan to develop automated tests to baseline the operation of the BE utilities and libbe modules. We also expect to test and verify the upgrade process from LiveCD and after the upgrade make sure the correctness from system level.
BE create/upgrade functionality:We plan to automate the majority of the functional testing of the BE utilities and libbe modules. For some features like create/delete BE, it's necessary to write some test suites around BE utilities to test these functions. This is top priority for test automation.
System upgrade correctness: We plan to change the system upgrade correctness after upgrade from LiveCD by making post upgrade verification and error checking. Through checking some system settings,such as user,date and system health and smf services checking to make sure the upgraded system status is correct.
Package upgrade correctness. We plan to automate the functional testing of the package upgrade,to make sure the correctness of the package upgrade. And it could through 2 aspects,one is through post check after upgrade from LiveCD, the other method is to write some test suites around use some IPS checking command to check the upgraded package and make sure the status of packages.
<<Note that GUI installer correctness. In May release,Snap will not support regular upgrade,the only way to make package upgrade is by CLI provided by beadm. So, we will not test regular upgrade from GUI installer in May release. But it will test the later release once Snap supports regular upgrade. In later release, we are going to test migration from UFS to ZFS,upgrade functionality from GUI installer. But we will not test the GUI installer in May release>>
3 Operational Factors
3.1 Assumptions
Assumption #1
Primary testing will be done on supported systems with graphics capabilities and 768mb ram or greater
Assumption #2
Snap Upgrades are only supported to enable package upgrade after initial installation of May release. And it will not support upgrade from Solaris Nevada and Indiana October release. But it might provide a migration tool to enable users to migrate to BE in indiana preview release.
Assumption #3
Snap upgrade will support DomU upgrade in xVM as well as Zones/Containers upgrade. Note that the way to upgrade of DomU is run beadm commands inside DomU, still not support upgrade DomU from Dom0.
3.2 Dependencies
Dependency #1
Test for PKG coverage depends on pkg manauls pkg cmd(5)
Dependency #2
DomU upgrade test will depend on whether the LiveCD in May release will support install on xVM.
3.3 Risks
Risk #1
- Description: We have a limited set of x86 hardware in our test environments. It is a small percentage of the HCL. Currently I estimate about a 3% coverage of the HCL.
- Likelihood of Occurrence:high
- Mitigation and Contingency Plan:The hardware used for this project was selected to provide a good representative sample of platforms and disk controller types. Even a relatively low percentage of the HCL represents a large number of systems.
Risk #2
- Description: The project hasn't been PSARC'd. Design shifts/additions could cause testing delays.
- Likelihood of Occurrence:Medium
- Mitigation and Contingency Plan:
Risk #3
- Description: There is no available test suites for this projects,and some modules need test drivers to make a test.
- Likelihood of Occurrence:High
- Mitigation and Contingency Plan:BE utilities will complete the same functions as current Live upgrade commands,so it could be possible to regard current Live Upgrade Test Suites as a reference to write useful test cases against BE commands. For pkg(5) test cases,we will write some Python test script to test the BE upgrade functionality provided by libbe.
4 Test Development
| Test Name | Automated or Manual | Test Type | Project Component | Description |
|---|---|---|---|---|
| BE commands | Automated | Functional | beadm(5) command | The beadm is the user interface for managing Boot Environments which is using libbe API functions.The BE utility will be implemented with ZFS support only, however a migration path from UFS to ZFS will also be supported butnot in the May release. |
| GUI installer testing | Manual | Functional | Solaris installer test | For May release,we are not going to test it since Snap only support upgrade packages by CLI.But we need to check that the installer creates the "initial" BE correctly upon initial installation. |
| libbe | Automated + manual | Functional | BE function test | Test the functions through BE commands and PKG commands,and also enable manual test by LiveCD or USB stick to guarantee upgrade correctness and post check correctness of whole system. |
| PKG commands | Automated | Functional | pkg(5) upgrade function | Test the upgrade functions through commands of IPS. We are going to write test scripts in Python and Shell to test upgrade functions. |
| libti | manual | Functional | TI library test | this would be tested by running the GUI installer in order to check the module create proper BE correctly on initial installation. |
5 Areas of Testing
5.1 Functional Testing
5.1.1 libbe
- For some features like create/delete BE. It's necessary to write some test suites around BE utilities to test these functions. This is top priority for test automation.
- For API which libbe provide to IPS and libti and may not covered by BE utilities, it needs some test drivers.
5.1.2 BE commands
- For BE commands,some test cases and usability tests and reliability tests are necessary. Depending on beadm manual(5), we are going to deliver some test cases against the beadm commands. And through BE commands,we are going to write some cases to test libbe functions,too.
- The functions of beadm(5) may is written in snapBeUtilDesign.pdf, please see snap upgrade plan documentation.
5.1.3 PKG commands
- We are going to write some test cases against pkg(5) commands,and we will focus on the upgrade functionality of pkg(5) commands.
- Here is pkg command support for BE. And for the pkg(5) upgrade function,please see IPS docs and snap upgrade plan.
5.1.4 libti
- Since libbe will provide API sto libti and libti is going to make changes compared to the one in October release. We will only make libti test by making initial install from LiveCD then check whether the BE that libti setup is correct or not. We will provide some cases in beadm test scripts to check the BE status after initial install to test functions of new libti.
5.1.4 Installer test
- Post upgrade verification and error checking.
- system health
- package upgrade correctness (need to check with package repository?)
- system setting,such as user,date.
- SMF status checking
- zone status check if needed.
5.2 Regression Testing
Since Snap in May release will only provide CLI to make package upgrade,so there is no much regression tests.
5.3 Conformance Testing
NA
5.4 Stress/Robustness Testing
NA
5.5 Performance Testing
N/A
5.6 Memory Leak Testing
Use mdb(1) ::findleaks to detect memory leaks after running stress tests.
5.7 Required Feature Testing
- Zones Testing
- Zones/Containers upgrade are going to get fully support from Snap upgrade projects through beadm(5) and pkg(5). We are going to test Zones/Containers upgrade functions in beadm test scripts. During the test,we could get following scenarios.
- Active BE with Containers
- Active BE with Containers but only upgrade the active BE
- Active BE with Containers but only upgrade a subset of the Containers
- Non-active BE with Containers
- Non-active BE with Containers but only upgrade the active BE
- Non-active BE with Containers but only upgrade a subset of the Containers
- Zones/Containers upgrade are going to get fully support from Snap upgrade projects through beadm(5) and pkg(5). We are going to test Zones/Containers upgrade functions in beadm test scripts. During the test,we could get following scenarios.
- ZFS Testing
- The ZFS testing will involve in the whole function testing
- Trusted Extensions Testing
- N/A
- xVM and LDOM Testing
- xVM are not affected by this project but as part of the system installation/upgrade,it is necessary to make DomU upgradable test, we are going to make xVM upgrade test through beadm(5) and pkg(5) test suites.
- Since it way of upgrade is not upgrade DomU from Dom0, our test appoach will be making initial install first from LiveCD image to DomU, then upgrade itself by beadm commands inside the DomU.
5.8 Interoperability Testing
The upgrade should not destroy co-exist other operating systems,we will verify other instances previously installed on the disk with Solaris will not impact from the upgrade. Like dwarf-installer,we could focus on the following:
- dual boot winXP/Vista and Solaris
- dual boot Linux and Solaris
- triple boot winXP/Vista, Solaris and Linux
- dual boot Solaris
<<Note that, the interoperability testing is based on BE CLI, not through installer since it needs migration from UFS to ZFS>>
5.9 Testing Not Covered
In May release,following tests are not covered.
- Upgrade from Solaris_Nevada
- Regular upgrade by LiveCD,i.e.,GUI installer
- BE with Zones testing
5.10 Documentation Testing
NA
5.11 Internationalization Testing
NA
6 Test Execution
6.1 Hardware Test Configurations
- The following HW platforms will be tested in various combinations:
- Sun x86 hardware and 3rd party x86 desktops and laptops.
- USB storage devices.
- memory configurations of 768mb and above
6.2 Software Test Configurations
- Verify that Solaris co-exists with winXP/winVista and linux on same disks.
- Package upgrade from initial install from Indiana May release.
- Zones/Containers Testing, our test suites will setup certain types of zones on BE to test with test scripts.
- Testing,it is necessary to set up virtual machines with Indiana May release and make upgrade test on the virtual machines.
- Upgrade from Indiana preview October release. Please see Dependency #4. If the dependency could not be satisfied, we will omit the test for May release and will test on later builds.
- Upgrade from UFS to verify the migration to ZFS. <<Note that in Sprint release,Snap project will not support the migration from UFS to ZFS in May release,but will test it on later builds.>>
- LiveCD graphical upgrade to verify GUI work as expected. <<Note that in May release,we are not going to test regular upgrade with LiveCD>>
6.3 Extrapolation Strategy (if applicable)
6.4 Pre-Integration PIT Run
- Is a pre-integration PIT run is required? (REQUIRED, RECOMMENDED, NONE)?NONE
6.5 Test Execution Matrix
| # | Test Name | Dell 270 | w2100z | IBM T61 | x4200 | Comments |
|---|---|---|---|---|---|---|
| 1 | beadm_command_test | |||||
| 2 | pkg_command_test | |||||
| 3 | libti_test | |||||
| 4 | xVM test | |||||
| 5 | interoperability_test |
Here is our HCL we own:
| # | Type | notes |
|---|---|---|
| 1 | Sun lx50 | |
| 2 | Sun v65x | |
| 3 | Sun w1100z | |
| 4 | Sun ultra20 | |
| 5 | Sun Fire v20z | |
| 6 | Dell Optiplex 270 | |
| 7 | Sun Fire x4200 | |
| 8 | Lenovo T61 | 8889-CQ3 & 7663-MJ2 |
7 Schedules and Milestones
| Milestone | Target Date | Actual Date |
|---|---|---|
| Test Plan Approved | 2007-12-17 | |
| Test Plan Review by I-team | ||
| Test specifications against beadm and pkg commands | ||
| Development of beadm utilities test suites Complete | ||
| Development of pkg commands test suites Complete | ||
| Test execution of snap upgrade after initial install from LiveCD and USB stick. | ||
| More test runs for post integration? |
8 Post-Integration Testing Information
8.1 Test Suite Integration Requirements
| Test Suite Name | Source Integration Location | Target Integration Date | Comments |
|---|---|---|---|
| STC2 Gate |
8.2 Test Suite Execution Integration Requirements
| Test Suite Name | Test Execution Group | Comments |
|---|---|---|
