First Edition (June 2000)
Part Number 187244-001
Compaq Computer Corporation
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
Notice
The information in this publication is subject to change without notice.
2000 Compaq Computer Corporation
Compaq, Compaq Insight Manager, SmartStart, ROMPaq, the Compaq logo, ProLiant, ServerNet,
StorageWorks, Registered in U.S. Patent and Trademark Office. ActiveAnswers is a trademark of
Compaq Information Technologies Group, L.P. Microsoft, MS-DOS, Windows, and Windows NT are
trademarks of Microsoft Corporation. Intel, Pentium, and Xeon are trademarks of Intel Corporation.
UNIX is a registered trademark of The Open Group. Oracle, Oracle8 and Oracle8i are trademarks of
Oracle Corporation. All other product names mentioned herein may be trademarks or registered
trademarks of their respective companies.
Compaq shall not be liable for technical or editorial errors or omissions contained herein. The
information in this document is subject to change without notice.
The information in this publication is subject to change without notice and is provided "AS IS"
WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF THE USE OF
THIS INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL COMPAQ BE
LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL, PUNITIVE OR
OTHER DAMAGES WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR
LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION OR LOSS OF BUSINESS
INFORMATION), EVEN IF COMPAQ HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES. THE FOREGOING SHALL APPLY REGARDLESS OF THE NEGLIGENCE OR
OTHER FAULT OF EITHER PARTY AND REGARDLESS OF WHETHER SUCH LIABILITY
SOUNDS IN CONTRACT, NEGLIGENCE, TORT, OR ANY OTHER THEORY OF LEGAL
LIABILITY, AND NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY
LIMITED REMEDY.
The limited warranties for Compaq products are exclusively set forth in the documentation
accompanying such products. Nothing herein should be construed as constituting a further or additional
warranty.
Compaq Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Administrator Guide
First Edition (June 2000)
Part Number 187244-001
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
Contents
About This Guide
Purpose ....................................................................................................................xvii
Audience ..................................................................................................................xvii
Scope ..................................................................................................................... xviii
Referenced Manuals ................................................................................................. xix
Supplemental Documents .......................................................................................... xx
Text Conventions......................................................................................................xxi
Symbols in Text........................................................................................................xxi
Symbols on Equipment............................................................................................xxii
Rack Stability ........................................................................................................ xxiii
Getting Help .......................................................................................................... xxiii
Compaq Technical Support ............................................................................ xxiii
Compaq Website..............................................................................................xxiv
Compaq Authorized Reseller...........................................................................xxiv
Chapter 1
Clustering Overview
Clusters Defined ...................................................................................................... 1-2
Availability .............................................................................................................. 1-3
Scalability ................................................................................................................ 1-4
Compaq Parallel Database Cluster Overview.......................................................... 1-4
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
iv Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Chapter 2
Cluster Hardware and Architecture With the MA8000/EMA12000 Storage
Subsystem
Compaq ProLiant Servers ........................................................................................ 2-2
High-Availability Features of ProLiant Servers ............................................... 2-3
Shared Storage Components .................................................................................... 2-3
MA8000/EMA12000 Storage Subsystem......................................................... 2-4
HSG80 Array Controller................................................................................... 2-4
Storage Switch .................................................................................................. 2-5
Storage Hub ...................................................................................................... 2-6
KGPSA-BC Host Bus Adapter ......................................................................... 2-6
Gigabit Interface Converter-Shortwave ............................................................ 2-6
Fibre Channel Cables........................................................................................ 2-7
Configuring and Cabling the MA8000/EMA12000 Storage Subsystem
Components ............................................................................................................. 2-7
Configuring LUNS for Storagesets................................................................... 2-7
SCSI Cabling Examples.................................................................................... 2-8
UltraSCSI Cables.............................................................................................. 2-9
Using I/O Modules in the Controller Enclosure ............................................... 2-9
Connecting EMUs Between MA8000/EMA12000 Storage Subsystems........ 2-11
I/O Path Configurations in a Fibre Channel Fabric SAN....................................... 2-12
Overview of Fibre Channel Fabric SAN Topology ........................................ 2-12
Redundant Fibre Channel Fabrics................................................................... 2-12
Multiple Redundant Fibre Channel Fabrics .................................................... 2-14
Maximum Distances Between Nodes and Shared Storage Subsystem
Components .................................................................................................... 2-15
I/O Data Paths in a Redundant Fibre Channel Fabric ..................................... 2-16
I/O Path Definitions for a Redundant Fibre Channel Fabric........................... 2-18
I/O Path Configuration Examples for a Redundant Fibre Channel Fabric...... 2-19
Summary of I/O Path Failure and Failover Scenarios for a Redundant Fibre
Channel Fabric................................................................................................ 2-24
I/O Path Configurations in a Fibre Channel Arbitrated Loop SAN ....................... 2-25
Overview of FC-AL SAN Topology .............................................................. 2-25
Redundant Fibre Channel Arbitrated Loops ................................................... 2-25
Multiple Redundant Fibre Channel Arbitrated Loops..................................... 2-27
Maximum Distances Between Nodes and Shared Storage Subsystem
Components .................................................................................................... 2-28
I/O Data Paths in a Redundant FC-AL ........................................................... 2-29
I/O Path Definitions for a Redundant FC-AL ................................................. 2-31
I/O Path Configuration Examples for a Redundant FC-AL ............................ 2-33
Summary of I/O Path Failure and Failover Scenarios for a Redundant
FC-AL ............................................................................................................. 2-37
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide v
Cluster Hardware and Architecture With the MA8000/EMA12000 Storage
Subsystem
continued
Cluster Interconnect Options ................................................................................. 2-38
Redundant Ethernet Cluster Interconnect ....................................................... 2-38
Redundant ServerNet Cluster Interconnect .................................................... 2-40
Local Area Network ....................................................................................... 2-43
Chapter 3
Cluster Software Components for Oracle8 Release 8.0.5/8.0.6
Overview of the Cluster Software............................................................................ 3-1
Microsoft Windows NT Server 4.0.......................................................................... 3-2
Compaq Software .................................................................................................... 3-2
Compaq SmartStart and Support Software....................................................... 3-2
Compaq System Configuration Utility ............................................................. 3-3
Compaq Insight Manager ................................................................................. 3-3
Compaq Insight Manager XE ........................................................................... 3-3
Compaq Options ROMPaq ............................................................................... 3-4
Compaq StorageWorks Command Console ..................................................... 3-4
Compaq StorageWorks Secure Path for Windows NT..................................... 3-4
Compaq Operating System Dependent Modules.............................................. 3-5
Oracle Software ....................................................................................................... 3-5
Oracle8 Server Enterprise Edition Release 8.0.5/8.0.6..................................... 3-6
Oracle8 Server Release 8.0.5/8.0.6................................................................... 3-6
Oracle8 Parallel Server Option Release 8.0.5/8.0.6.......................................... 3-6
Oracle8 Enterprise Manager Release 8.0.5/8.0.6.............................................. 3-7
Oracle8 Certification ........................................................................................ 3-7
Application Failover and Reconnection Software ................................................... 3-7
Chapter 4
Cluster Software Components for Oracle8i Release 8.1.5/8.1.6
Overview of the Cluster Software............................................................................ 4-1
Microsoft Windows NT Server 4.0.......................................................................... 4-2
Compaq Software .................................................................................................... 4-2
Compaq SmartStart and Support Software....................................................... 4-2
Compaq System Configuration Utility ............................................................. 4-3
Compaq Insight Manager ................................................................................. 4-3
Compaq Insight Manager XE ........................................................................... 4-3
Compaq Options ROMPaq ............................................................................... 4-4
Compaq StorageWorks Command Console ..................................................... 4-4
Compaq StorageWorks Secure Path for Windows NT..................................... 4-4
Compaq Operating System Dependent Modules.............................................. 4-5
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
vi Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Cluster Software Components for Oracle8i Release 8.1.5/8.1.6
continued
Oracle Software ....................................................................................................... 4-5
Oracle8i Server Enterprise Edition Release 8.1.5/8.1.6 .................................... 4-5
Oracle8i Server Release 8.1.5/8.1.6.................................................................. 4-6
Oracle8i Parallel Server Option Release 8.1.5/8.1.6......................................... 4-6
Oracle8i Enterprise Manager Release 8.1.5/8.1.6............................................. 4-6
Oracle8i Certification........................................................................................ 4-7
Application Failover and Reconnection Software.................................................... 4-7
Chapter 5
Cluster Planning With the MA8000/EMA12000 Storage Subsystem
Site Planning ............................................................................................................ 5-1
Capacity Planning for Cluster Hardware ................................................................. 5-2
Compaq ProLiant Servers ................................................................................. 5-2
Planning Shared Storage Subsystem Components............................................ 5-3
Planning Cluster Interconnect and Client LAN Components ........................... 5-8
Planning Cluster Configurations for Fibre Channel Fabrics .................................... 5-9
Planning Dual Redundancy Configurations.................................................... 5-10
Planning Quad Redundancy Configurations................................................... 5-13
Planning Cluster Configurations for Fibre Channel Arbitrated Loops................... 5-15
Planning Dual Redundancy Configurations.................................................... 5-15
Planning Quad Redundancy Configurations................................................... 5-18
RAID Planning for the MA8000/EMA12000 Storage Subsystem......................... 5-20
Supported RAID Levels.................................................................................. 5-21
Raw Data Storage and Database Size ............................................................. 5-22
Selecting the Appropriate RAID Level........................................................... 5-22
Planning the Grouping of Physical Disk Storage Space ........................................ 5-24
Disk Drive Planning............................................................................................... 5-25
Nonshared Disk Drives ................................................................................... 5-25
Shared Disk Drives ......................................................................................... 5-26
Network Planning .................................................................................................. 5-26
Windows NT Server Host Files for the Ethernet Cluster Interconnect ........... 5-26
Windows NT Server Hosts Files for the ServerNet Cluster Interconnect....... 5-27
Client LAN ..................................................................................................... 5-28
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide vii
Chapter 6
Installation and Configuration for Oracle8 Release 8.0.5/8.0.6
Reference Materials for Installation......................................................................... 6-1
Installation Overview............................................................................................... 6-2
Installing the Hardware............................................................................................ 6-4
Setting Up the Nodes........................................................................................ 6-4
Installing the KGPSA-BC Host Bus Adapters.................................................. 6-4
Cabling the Host Bus Adapters to Storage Switches or Storage Hubs ............. 6-5
Installing the Cluster Interconnect Adapters..................................................... 6-6
Installing the Client LAN Adapters .................................................................. 6-6
Cabling the Cluster Interconnect ...................................................................... 6-6
Cabling the Client LAN.................................................................................... 6-8
Installing Hardware Into an MA8000/EMA12000 Storage Subsystem............ 6-9
Cabling the Controller Enclosure to Disk Enclosures ...................................... 6-9
Cabling EMUs to Each Other ......................................................................... 6-13
Cabling Array Controllers to Storage Switches.............................................. 6-13
Cabling Array Controllers to Storage Hubs.................................................... 6-15
Installing Operating System Software ................................................................... 6-17
Guidelines for Clusters ................................................................................... 6-17
Automated Installation Using SmartStart ....................................................... 6-18
Setting Up and Configuring an MA8000/EMA12000 Storage Subsystem............ 6-21
Designating a Server as a Maintenance Terminal........................................... 6-21
Powering On the MA8000/EMA12000 Storage Subsystem........................... 6-21
Installing the StorageWorks Command Console (SWCC) Client .................. 6-22
Configuring a Storage Subsystem for Secure Path Operation ........................ 6-22
Verifying Array Controller Properties............................................................ 6-27
Configuring a Storageset ................................................................................ 6-29
Installing Secure Path for Windows NT ................................................................ 6-32
Overview ........................................................................................................ 6-32
Installing the Host Bus Adapter Drivers......................................................... 6-33
Installing the Fibre Channel Software Setup (FCSS) Utility.......................... 6-34
Installing the Secure Path Drivers, Secure Path Agent, and Secure Path
Manager .......................................................................................................... 6-34
Specifying the Preferred_Path for Storage Units............................................ 6-36
Powering Up All Other Storage Switches or Storage Hubs............................ 6-37
Creating Partitions .......................................................................................... 6-37
Installing Oracle Software ..................................................................................... 6-38
Verifying Cluster Communications ....................................................................... 6-39
Installing the Compaq OSDs ................................................................................. 6-40
OSD Installation Steps.................................................................................... 6-40
Running the NodeList Configurator ............................................................... 6-44
Installing Object Link Manager ............................................................................. 6-46
Configuring Oracle Software................................................................................. 6-46
Additional Notes on Configuring Oracle Software ........................................ 6-47
Start the Database in Parallel Mode................................................................ 6-48
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
viii Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Installation and Configuration for Oracle8 Release 8.0.5/8.0.6
continued
Verifying the Hardware and Software Installation................................................. 6-48
Cluster Communications................................................................................. 6-48
Access to Shared Storage from All Nodes...................................................... 6-49
OSDs ............................................................................................................... 6-49
Other Verification Tasks................................................................................. 6-49
Power Distribution and Power Sequencing Guidelines.......................................... 6-50
Overview......................................................................................................... 6-50
Server Power Distribution............................................................................... 6-50
Storage Subsystem Power Distribution........................................................... 6-51
Power Sequencing........................................................................................... 6-51
Chapter 7
Installation and Configuration for Oracle8i Release 8.1.5/8.1.6
Reference Materials for Installation......................................................................... 7-1
Installation Overview............................................................................................... 7-2
Installing the Hardware............................................................................................ 7-3
Setting Up the Nodes ........................................................................................ 7-3
Installing the KGPSA-BC Host Bus Adapters.................................................. 7-4
Cabling the Host Bus Adapters to Storage Switches or Storage Hubs.............. 7-5
Installing the Cluster Interconnect Adapters..................................................... 7-6
Installing the Client LAN Adapters .................................................................. 7-6
Cabling the Cluster Interconnect....................................................................... 7-7
Cabling the Client LAN .................................................................................... 7-9
Installing Hardware Into an MA8000/EMA12000 Storage Subsystem .......... 7-10
Cabling the Controller Enclosure to Disk Enclosures..................................... 7-10
Cabling EMUs to Each Other ......................................................................... 7-13
Cabling Array Controllers to Storage Switches .............................................. 7-13
Cabling Array Controllers to Storage Hubs .................................................... 7-15
Installing Operating System Software.................................................................... 7-17
Guidelines for Clusters ................................................................................... 7-17
Automated Installation Using SmartStart ....................................................... 7-18
Setting Up and Configuring an MA8000/EMA12000 Storage Subsystem ............ 7-21
Designating a Server as a Maintenance Terminal........................................... 7-21
Powering On the MA8000/EMA12000 Storage Subsystem........................... 7-21
Installing the StorageWorks Command Console (SWCC) Client................... 7-22
Configuring a Storage Subsystem for Secure Path Operation ........................ 7-22
Verifying Array Controller Properties ............................................................ 7-27
Configuring a Storageset................................................................................. 7-29
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide ix
Installation and Configuration for Oracle8i Release 8.1.5/8.1.6
continued
Installing Secure Path for Windows NT ................................................................ 7-32
Overview ........................................................................................................ 7-32
Installing the Host Bus Adapter Drivers......................................................... 7-33
Installing the Fibre Channel Software Setup (FCSS) Utility.......................... 7-33
Installing the Secure Path Drivers, Secure Path Agent, and Secure Path
Manager .......................................................................................................... 7-34
Specifying the Preferred_Path for Storage Units............................................ 7-35
Powering Up All Other Storage Switches or Storage Hubs............................ 7-37
Creating Partitions .......................................................................................... 7-37
Installing Compaq OSDs ....................................................................................... 7-38
Verifying Installation of the SNMP Service................................................... 7-39
Verifying Cluster Communications ................................................................ 7-40
Mounting Remote Drives and Verifying Administrator Privileges ................ 7-41
Installing Ethernet OSDs ................................................................................ 7-42
Installing ServerNet OSDs, Drivers, and SNMP Agents................................ 7-52
Verifying the ServerNet Cluster Interconnect ................................................ 7-63
Installing Oracle Software ..................................................................................... 7-64
Configuring Oracle Software.......................................................................... 7-64
Installing Object Link Manager...................................................................... 7-65
Verifying the Hardware and Software Installation ......................................... 7-66
Power Distribution and Power Sequencing Guidelines ......................................... 7-68
Overview ........................................................................................................ 7-68
Server Power Distribution .............................................................................. 7-68
Storage Subsystem Power Distribution .......................................................... 7-69
Power Sequencing .......................................................................................... 7-69
Chapter 8
Cluster Management With the MA8000/EMA12000 Storage Subsystem for Oracle8
Release 8.0.5/8.0.6
Cluster Management Concepts ................................................................................ 8-1
Powering Off a Node Without Interrupting Cluster Services ........................... 8-1
Managing a Cluster in a Degraded Condition................................................... 8-2
Managing Network Clients Connected to a Cluster ......................................... 8-2
Cluster Events................................................................................................... 8-3
Management Applications ....................................................................................... 8-3
Monitoring Server and Network Hardware ...................................................... 8-3
Monitoring Storage Subsystem Hardware........................................................ 8-4
Managing Shared Storage................................................................................. 8-4
Monitoring the Database .................................................................................. 8-6
Remotely Managing a Cluster .......................................................................... 8-6
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
x Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Cluster Management With the MA8000/EMA12000 Storage Subsystem for
Oracle8 Release 8.0.5/8.0.6
continued
Software Maintenance.............................................................................................. 8-7
Deinstalling the OSDs....................................................................................... 8-7
Upgrading Oracle8 Server ................................................................................ 8-7
Upgrading the OSDs ......................................................................................... 8-7
Managing Changes to Shared Storage ..................................................................... 8-8
Adding New Storagesets to Increase Shared Storage Capacity ........................ 8-8
Replacing a Failed Drive in a Storage Subsystem ............................................ 8-9
Replacing a Host Bus Adapter ........................................................................ 8-10
Adding a Shared Storage Subsystem .............................................................. 8-12
Replacing a Cluster Node ...................................................................................... 8-13
Removing the Node ........................................................................................ 8-13
Adding the Replacement Node ....................................................................... 8-14
Adding a Cluster Node........................................................................................... 8-17
Preparing the New Node................................................................................. 8-17
Preparing the Existing Cluster Nodes ............................................................. 8-19
Installing the Cluster Software........................................................................ 8-19
Monitoring Cluster Performance............................................................................ 8-20
Tools Overview............................................................................................... 8-20
Using Secure Path Manager............................................................................ 8-21
Uninstalling Secure Path................................................................................. 8-24
Chapter 9
Cluster Management With the MA8000/EMA12000 Storage Subsystem for Oracle8i
Release 8.1.5/8.1.6
Cluster Management Concepts ................................................................................ 9-1
Powering Off a Node Without Interrupting Cluster Services ........................... 9-1
Managing a Cluster in a Degraded Condition................................................... 9-2
Managing Network Clients Connected to a Cluster.......................................... 9-2
Cluster Events ................................................................................................... 9-3
Management Applications ....................................................................................... 9-3
Monitoring Server and Network Hardware....................................................... 9-3
Monitoring Storage Subsystem Hardware ........................................................ 9-4
Managing Shared Storage ................................................................................. 9-4
Monitoring the Database................................................................................... 9-5
Remotely Managing a Cluster .......................................................................... 9-6
Software Maintenance.............................................................................................. 9-6
Deinstalling the OSDs....................................................................................... 9-6
Upgrading Oracle8i Server ............................................................................... 9-9
Upgrading the OSDs ....................................................................................... 9-10
Deinstalling a Partial OSD Installation ........................................................... 9-13
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide xi
Cluster Management With the MA8000/EMA12000 Storage Subsystem for
Oracle8i Release 8.1.5/8.1.6
continued
Managing Changes to Shared Storage ................................................................... 9-14
Adding New Storagesets to Increase Shared Storage Capacity ...................... 9-14
Replacing a Failed Drive in a Storage Subsystem .......................................... 9-15
Replacing a Host Bus Adapter........................................................................ 9-16
Adding a Shared Storage Subsystem .............................................................. 9-18
Replacing a Cluster Node ...................................................................................... 9-19
Removing the Node ........................................................................................ 9-19
Adding the Replacement Node....................................................................... 9-20
Adding a Cluster Node .......................................................................................... 9-23
Preparing the New Node................................................................................. 9-24
Preparing the Existing Cluster Nodes............................................................. 9-25
Installing the Cluster Software ....................................................................... 9-26
Monitoring Cluster Performance ........................................................................... 9-27
Tools Overview .............................................................................................. 9-27
Using Secure Path Manager............................................................................ 9-28
Uninstalling Secure Path ................................................................................ 9-30
Chapter 10
Cluster Troubleshooting With the MA8000/EMA12000 Storage Subsystem for
Oracle8 Release 8.0.5/8.0.6
Basic Troubleshooting Tips ................................................................................... 10-1
Power .............................................................................................................. 10-1
Physical Connections...................................................................................... 10-2
Access to Cluster Components ....................................................................... 10-2
Software Revisions ......................................................................................... 10-3
Firmware Revisions........................................................................................ 10-3
Troubleshooting Oracle 8.0.5/8.0.6 and OSD Installation Problems and Error
Messages ................................................................................................................ 10-4
While Running the NodeList Configurator, A Dialog Box Reports Inability
to Connect to Remote Nodes .......................................................................... 10-4
Error Message: Unable to Start Cluster Manager (CMSRVR.EXE) .............. 10-5
Error Message: Unable to Start Oracle Service .............................................. 10-5
Error Message: "Dependent Service Has Not Started" Received While
Attempting to Start the Oracle Service ........................................................... 10-6
Error Message: Unable to Configure Oracle Using OPSCONF Utility.......... 10-6
Error Message: Error In Creating Oracle Instance ......................................... 10-6
Error Message: Initialization of the Dynamic Link Library CM.DLL
Failed. The Process Is Terminating Abnormally ............................................ 10-7
Unable to Start the Database........................................................................... 10-7
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
xii Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Cluster Troubleshooting With the MA8000/EMA12000 Storage Subsystem for
Oracle8 Release 8.0.5/8.0.6
continued
Troubleshooting Node-to-Node Connectivity Problems........................................ 10-8
Nodes Are Unable to Communicate With Each Other ................................... 10-8
Unable to Ping the Cluster Interconnect or the Client LAN ........................... 10-8
Node or Nodes Unable to Rejoin the Cluster.................................................. 10-9
Troubleshooting Client-to-Cluster Connectivity Problems.................................. 10-10
A Network Client Cannot Communicate With the Cluster........................... 10-10
Troubleshooting Shared Storage Subsystem Problems........................................ 10-10
Verifying Host Bus Adapter Device Driver Installation ............................... 10-10
Verifying KGPSA-BC Device Driver Initialization ..................................... 10-11
Verifying Connectivity to a Redundant Fibre Channel Fabric...................... 10-12
Verifying Connectivity to a Redundant Fibre Channel Arbitrated Loop ...... 10-13
A Cluster Node Cannot Connect to the Shared Drives ................................. 10-15
Windows NT Disk Administrator Shows Storagesets With the Same Label
(Dual Image) ................................................................................................. 10-15
Device or Devices Were Not Found by KGPSA-BC Device Driver ............ 10-16
Devices on One I/O Connection Path Cannot Be Seen by the Cluster
Nodes ............................................................................................................ 10-17
Troubleshooting Secure Path ............................................................................... 10-18
Secure Path Manager Shows Reversed Location for Top and Bottom Array
Controllers .................................................................................................... 10-18
Chapter 11
Cluster Troubleshooting With the MA8000/EMA12000 Storage Subsystem for
Oracle8i Release 8.1.5/8.1.6
Basic Troubleshooting Tips ................................................................................... 11-1
Power .............................................................................................................. 11-1
Physical Connections ...................................................................................... 11-2
Access to Cluster Components ....................................................................... 11-2
Software Revisions ......................................................................................... 11-3
Firmware Revisions ........................................................................................ 11-3
Troubleshooting Oracle8i and OSD Installation Problems and Error Messages ... 11-4
Potential Difficulties Installing the OSDs With the Oracle Universal
Installer ........................................................................................................... 11-4
Unable to Start OracleCMService................................................................... 11-5
Unable to Start OracleNMService .................................................................. 11-5
Unable to Start the Database........................................................................... 11-6
Initialization of the Dynamic Link Library NM.DLLFailed........................... 11-6
Troubleshooting Node-to-Node Connectivity Problems........................................ 11-7
Nodes Are Unable to Communicate With Each Other ................................... 11-7
viping Does Not Complete Successfully......................................................... 11-8
Unable to Ping the Cluster Interconnect or the Client LAN ........................... 11-9
Node or Nodes Unable to Rejoin the Cluster................................................ 11-10
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide xiii
Cluster Troubleshooting With the MA8000/EMA12000 Storage Subsystem for
Oracle8i Release 8.1.5/8.1.6
continued
Troubleshooting Client-to-Cluster Connectivity Problems ..................................11-11
A Network Client Cannot Communicate With the Cluster............................11-11
Troubleshooting Shared Storage Subsystem Problems.........................................11-12
Verifying Host Bus Adapter Device Driver Installation................................11-12
Verifying KGPSA-BC Device Driver Initialization ......................................11-12
Verifying Connectivity to a Redundant Fibre Channel Fabric ......................11-13
Verifying Connectivity to a Redundant Fibre Channel Arbitrated Loop.......11-15
A Cluster Node Cannot Connect to the Shared Drives..................................11-16
Windows NT Disk Administrator Shows Storagesets With the Same Label
(Dual Image)..................................................................................................11-17
Device or Devices Were Not Found by KGPSA-BC Device Driver.............11-17
Devices on One I/O Connection Path Cannot Be Seen by the Cluster
Nodes .............................................................................................................11-18
Troubleshooting Secure Path ................................................................................11-20
Secure Path Manager Shows Reversed Location for Top and Bottom Array
Controllers .....................................................................................................11-20
Appendix A
viping Utility
Syntax and Option Summary...................................................................................A-1
Example ............................................................................................................A-2
Error Diagnostics .....................................................................................................A-2
Appendix B
Diagnosing and Resolving Shared Disk Problems With the MA8000/EMA12000
Storage Subsystem
Introduction ............................................................................................................. B-1
Run Object Link Manager on All Nodes ................................................................. B-3
Restart All Affected Nodes in the Cluster ............................................................... B-4
Rerun and Validate Object Link Manager On All Affected Nodes ......................... B-4
Run and Validate Secure Path Manager On All Nodes ........................................... B-5
Run Windows NT Server Disk Administrator On All Nodes..................................B-6
Run and Validate the StorageWorks Command Console From All Storage
Subsystems .............................................................................................................. B-6
Perform Cluster Software and Firmware Checks .................................................... B-7
Perform Cluster Hardware Checks .......................................................................... B-7
Contact Your Compaq Support Representative ....................................................... B-8
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
xiv Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Glossary
Index
List of Figures
Figure 1-1. Example of a two-node PDC/O5000 .................................................... 1-3
Figure 2-1. SCSI bus numbers for I/O modules in the controller enclosure ........... 2-9
Figure 2-2. UltraSCSI cabling between a controller enclosure and three
disk enclosures ................................................................................................. 2-10
Figure 2-3. UltraSCSI cabling between a controller enclosure and six disk
enclosures ......................................................................................................... 2-11
Figure 2-4. Two-node PDC/O5000 with a two-fabric redundant Fibre
Channel Fabric ................................................................................................. 2-13
Figure 2-5. Two-node PDC/O5000 with two redundant Fibre Channel
Fabrics .............................................................................................................. 2-15
Figure 2-6. Maximum distances between PDC/O5000 cluster nodes and
shared storage subsystem components ............................................................. 2-16
Figure 2-7. Host bus adapter-to-storage switch data paths.................................... 2-17
Figure 2-8. Storage switch-to-MA8000/EMA12000 Storage Subsystem
data paths.......................................................................................................... 2-18
Figure 2-9. Dual redundancy configuration for a redundant Fibre Channel
Fabric, option 1 ................................................................................................ 2-21
Figure 2-10. Dual redundancy configuration for a redundant Fibre Channel
Fabric, option 2 ................................................................................................ 2-22
Figure 2-11. Quad redundancy configuration for a redundant Fibre Channel
Fabric................................................................................................................ 2-23
Figure 2-12. Two-node PDC/O5000 with a two-loop redundant Fibre
Channel Arbitrated Loop.................................................................................. 2-26
Figure 2-13. Two-node PDC/O5000 with two redundant Fibre Channel
Arbitrated Loops .............................................................................................. 2-28
Figure 2-14. Maximum distances between PDC/O5000 cluster nodes and
shared storage subsystem components ............................................................. 2-29
Figure 2-15. Host bus adapter-to-storage hub data paths ...................................... 2-30
Figure 2-16. Storage hub-to-MA8000/EMA12000 Storage Subsystem data
paths ................................................................................................................. 2-31
Figure 2-17. Dual redundancy configuration for a redundant FC-AL, option
1 ........................................................................................................................ 2-34
Figure 2-18. Dual redundancy configuration for a redundant FC-AL, option
2 ........................................................................................................................ 2-35
Figure 2-19. Quad redundancy configuration for a redundant FC-AL.................. 2-36
Figure 2-20. Redundant Ethernet cluster interconnect components in a
two-node PDC/O5000 ...................................................................................... 2-40
Figure 2-21. Redundant ServerNet cluster interconnect components in a
two-node PDC/O5000 ...................................................................................... 2-42
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide xv
Figure 2-22. Redundant ServerNet cluster interconnect components in a
four-node PDC/O5000 ..................................................................................... 2-43
Figure 5-1. Dual redundancy configuration for Fibre Channel Fabric,
option 1 ............................................................................................................ 5-11
Figure 5-2. Dual redundancy configuration for Fibre Channel Fabric,
option 2 ............................................................................................................ 5-12
Figure 5-3. Quad redundancy configuration for Fibre Channel Fabric................. 5-14
Figure 5-4. Dual redundancy configuration for FC-AL SAN, option 1................ 5-16
Figure 5-5. Dual redundancy configuration for FC-AL SAN, option 2................ 5-17
Figure 5-6. Quad redundancy configuration for FC-AL SAN .............................. 5-19
Figure 5-7 MA8000/EMA12000 Storage Subsystem disk grouping for a
PDC/O5000 cluster .......................................................................................... 5-24
Figure 6-1. Connecting host bus adapters to storage switches or storage
hubs .................................................................................................................... 6-5
Figure 6-2. Redundant client LAN and Ethernet cluster interconnect .................... 6-7
Figure 6-3. SCSI bus numbers for I/O modules in the controller enclosure ......... 6-10
Figure 6-4. UltraSCSI cabling between a controller enclosure and three
disk enclosures ................................................................................................. 6-11
Figure 6-5. UltraSCSI cabling between a controller enclosure and six disk
enclosures......................................................................................................... 6-12
Figure 6-6. Dual redundancy configuration for a redundant Fibre Channel
Fabric ............................................................................................................... 6-14
Figure 6-7. Dual redundancy configuration for a redundant FC-AL .................... 6-16
Figure 6-8. Server power distribution in a three-node cluster............................... 6-51
Figure 7-1. Connecting host bus adapters to storage switches or storage
hubs .................................................................................................................... 7-5
Figure 7-2. Redundant client LAN and Ethernet cluster interconnect .................... 7-8
Figure 7-3. SCSI bus numbers for I/O modules in the controller enclosure ......... 7-11
Figure 7-4. UltraSCSI cabling between a controller enclosure and three
disk enclosures ................................................................................................. 7-12
Figure 7-5. UltraSCSI cabling between a controller enclosure and six disk
enclosures......................................................................................................... 7-13
Figure 7-6. Dual redundancy configuration for a redundant Fibre Channel
Fabric ............................................................................................................... 7-15
Figure 7-7. Dual redundancy configuration for a redundant FC-AL .................... 7-16
Figure 7-8. Server power distribution in a three-node cluster............................... 7-68
Figure B-1. Tasks for diagnosing and resolving shared storage problems ............. B-2
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
xvi Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
List of Tables
Table 2-1 High-Availability Components of ProLiant Servers................................ 2-3
Table 2-2 SCSI bus address ID assignments for the MA8000/EMA12000
Storage Subsystem ............................................................................................. 2-8
Table 2-3 I/O Path Failure and Failover Scenarios for a Redundant Fibre
Channel Fabric ................................................................................................. 2-24
Table 2-4 I/O Path Failure and Failover Scenarios for a Redundant FC-AL ......... 2-37
Table 5-1 .................................................................................................................. 5-4
Storage Switches Required for Dual Redundancy Configurations .......................... 5-4
Table 5-2 .................................................................................................................. 5-5
Storage Switches Required for Quad Redundancy Configurations.......................... 5-5
Table 5-3 .................................................................................................................. 5-7
Fibre Channel Storage Hubs Required for Dual-Redundancy Configurations ........ 5-7
Table 5-4 .................................................................................................................. 5-7
Fibre Channel Storage Hubs Required for Quad-Redundancy
Configurations .................................................................................................... 5-7
Table 6-1 Controller Properties.............................................................................. 6-28
Table 7-1 Controller Properties.............................................................................. 7-28
Table A-1 viping Errors .......................................................................................... A-2
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
About This Guide
Purpose
This administrator guide provides information about the planning, installation,
configuration, implementation, management, and troubleshooting of the
Compaq Parallel Database Cluster Model PDC/O5000.
Audience
The expected audience of this guide consists primarily of MIS professionals
whose jobs include designing, installing, configuring, and maintaining
Compaq Parallel Database Clusters.
The audience of this guide must have a working knowledge of Microsoft
Windows NT Server and of Oracle databases or have the assistance of a
database administrator.
This guide contains information for network administrators, database
administrators, installation technicians, systems integrators, and other
technical personnel in the enterprise environment for the purpose of cluster
planning, installation, implementation, and maintenance.
IMPORTANT: This guide contains installation, configuration, and maintenance
information that can be valuable for a variety of users. If you are installing the PDC/O5000
but will not be administering the cluster on a daily basis, please make this guide available
to the person or persons who will be responsible for the clustered servers after you have
completed the installation.
Compaq Confidential Need to Know Required
Writer: Vaughn Hasslein Project: Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6 Comments:
Part Number: 187244-001 File Name: a-frnt.doc Last Saved On: 6/28/00 9:20 PM
xviii Parallel Database Cluster Model PDC/O5000 for Oracle 8.0.5/8.0.6 and 8.1.5/8.1.6
Scope
This guide offers significant background information about clusters as well as
basic concepts associated with designing clusters. It also contains detailed
product descriptions and installation steps.
This administrator guide is designed to assist you in the following objectives:
I Understanding basic concepts of clustering technology
I Recognizing and using the high-availability features of the PDC/O5000
I Planning and designing a PDC/O5000 cluster configuration to meet your
business needs
I Installing and configuring PDC/O5000 hardware and software
I Managing the PDC/O5000
I Troubleshooting the PDC/O5000
The following summarizes the contents of this guide:
I Chapter 1, "Clustering Overview," provides an introduction to
clustering technology features and benefits.
I Chapter 2, "Cluster Hardware and Architecture With the
MA8000/EMA12000 Storage Subsystem," describes the hardware
components of the PDC/O5000 and provides detailed I/O path
configuration information.
I Chapter 3, "Cluster Software Components for Oracle8 Release
8.0.5/8.0.6," describes software components used with the PDC/O5000
for Oracle8 Release 8.0.5/8.0.6.
I Chapter 4, "Cluster Software Components for Oracle8i Release
8.1.5/8.1.6," describes software components used with the PDC/O5000
for Oracle8i Release 8.1.5/8.1.6.
I Chapter 5, "Cluster Planning With the MA8000/EMA12000 Storage
Subsystem," outlines an approach to planning and designing cluster
configurations that meet your business needs.
I Chapter 6, "Installation and Configuration for Oracle8 Release
8.0.5/8.0.6," outlines the steps you will take to install and configure the
PDC/O5000 hardware and software for Oracle8 Release 8.0.5/8.0.6.
I Chapter 7, "Installation and Configuration for Oracle8i Release
8.1.5/8.1.6," outlines the steps you will take to install and configure the
PDC/O5000 hardware and software for Oracle8i Release 8.1.5/8.1.6.
| 187244-001
|
