First Edition (June 1999)
Part Number 104286-001
Compaq Computer Corporation
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
Notice
The information in this publication is subject to change without notice.
COMPAQ COMPUTER CORPORATION SHALL NOT BE LIABLE FOR TECHNICAL OR
EDITORIAL ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR
CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR
USE OF THIS MATERIAL. THIS INFORMATION IS PROVIDED "AS IS" AND COMPAQ
COMPUTER CORPORATION DISCLAIMS ANY WARRANTIES, EXPRESS, IMPLIED OR
STATUTORY AND EXPRESSLY DISCLAIMS THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR PARTICULAR PURPOSE, GOOD TITLE AND AGAINST
INFRINGEMENT.
This publication contains information protected by copyright. No part of this publication may be
photocopied or reproduced in any form without prior written consent from Compaq Computer
Corporation.
1999 Compaq Computer Corporation.
All rights reserved. Printed in the U.S.A.
The software described in this guide is furnished under a license agreement or nondisclosure agreement.
The software may be used or copied only in accordance with the terms of the agreement.
Compaq, Deskpro, Fastart, Compaq Insight Manager, Systempro, Systempro/LT, ProLiant, ROMPaq,
QVision, SmartStart, NetFlex, QuickFind, PaqFax, ProSignia, registered United States Patent and
Trademark Office.
Neoserver, Netelligent, Systempro/XL, SoftPaq, QuickBlank, QuickLock are trademarks and/or service
marks of Compaq Computer Corporation.
Microsoft, MS-DOS, Windows, and Windows NT are registered trademarks of Microsoft Corporation.
Pentium is a registered trademark and Xeon is a trademark of Intel Corporation.
Oracle is a registered trademark and Oracle8i is a trademark of Oracle Corporation.
Other product names mentioned herein may be trademarks and/or registered trademarks of their
respective companies.
Compaq Parallel Database Cluster Model PDC/ 1000 for Oracle8i Release 8.1.5 Administrator
Guide
First Edition (June 1999)
Part Number 104286-001
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
Contents
About This Guide
Purpose......................................................................................................................... ix
Audience ...................................................................................................................... ix
Scope ............................................................................................................................. x
Referenced Manuals .................................................................................................... xi
Supplemental Documents ...........................................................................................xii
Text Conventions...................................................................................................... xiii
Symbols in Text ......................................................................................................... xiv
Symbols on Equipment...............................................................................................xv
Getting Help...............................................................................................................xvi
Compaq Technical Support ................................................................................ xvi
Compaq Website.................................................................................................xvi
Compaq Authorized Reseller ............................................................................xvii
Chapter 1
Clustering Overview
Clusters Defined ....................................................................................................... 1-2
Availability ............................................................................................................... 1-3
Scalability ................................................................................................................. 1-3
Compaq Parallel Database Cluster Overview ......................................................... 1-4
Chapter 2
Architecture of the Compaq Parallel Database Cluster
Compaq ProLiant Servers......................................................................................... 2-2
High Availability Features of ProLiant Servers ................................................ 2-3
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
iv Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Architecture of the Compaq Parallel Database Cluster
continued
Shared Storage Components.............................................................................................2-4
Shared Storage Clustering ..................................................................................2-4
Availability Features of the RA4000 External Storage Subsystem ...................2-5
Maximum Distances Between Cluster Nodes and Shared Storage
Subsystem Components .....................................................................................2-6
Using Multiple Fibre Channel Loops in a PDC/ 1000 Cluster ........................2-6
RA4000 External Storage Subsystem Components ...........................................2-8
Fibre Channel Data Paths .................................................................................2-11
Cluster Interconnect Options ...................................................................................2-14
Ethernet Cluster Interconnect...........................................................................2-15
ServerNet Cluster Interconnect ........................................................................2-21
Local Area Network .........................................................................................2-24
Chapter 3
Cluster Software Components
Overview of the Cluster Software .............................................................................3-1
Microsoft Windows NT Server 4.0 ...........................................................................3-2
Compaq Software ......................................................................................................3-2
Compaq SmartStart and Support Software ........................................................3-2
Compaq System Configuration Utility...............................................................3-3
Compaq Support Software Diskette for Windows NT .....................................3-3
Compaq Insight Manager ...................................................................................3-3
Compaq Options ROMPaq.................................................................................3-4
Compaq Operating System Dependent Modules ...............................................3-4
Oracle Software .........................................................................................................3-5
Oracle8i Server Enterprise Edition Release 8.1.5 ..............................................3-5
Oracle8i Server Release 8.1.5 ............................................................................3-5
Oracle8i Parallel Server Option Release 8.1.5...................................................3-5
Oracle8i Enterprise Manager Release 8.1.5 .......................................................3-6
Client Reconnect Software.................................................................................3-6
Oracle Certification ............................................................................................3-7
Chapter 4
Planning
Site Planning..............................................................................................................4-2
Capacity Planning for Cluster Hardware...................................................................4-2
Compaq ProLiant Servers ..................................................................................4-2
Planning Shared Storage Components ...............................................................4-3
Planning Cluster Interconnect and Client LAN Components ............................4-5
Reference Material for Hardware Sizing ...........................................................4-6
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide v
Planning
continued
Planning the Cluster Configuration.............................................................................. 4-7
Sample Midsize Cluster Configuration ............................................................. 4-7
Sample Large Cluster Configuration................................................................. 4-8
RAID Planning ......................................................................................................... 4-9
Supported RAID Levels .................................................................................. 4-10
Raw Data Storage and Database Size.............................................................. 4-11
Selecting RAID Levels.................................................................................... 4-12
Planning the Grouping of Physical Disk Storage Space......................................... 4-13
Disk Drive Planning ............................................................................................... 4-14
Non-Shared Disk Drives.................................................................................. 4-14
Shared Disk Drives.......................................................................................... 4-15
Network Planning ................................................................................................... 4-15
Windows NT Server Hosts Files for the Ethernet Cluster Interconnect ......... 4-15
Windows NT Server Hosts Files for the ServerNet Cluster Interconnect....... 4-16
Client LAN ...................................................................................................... 4-17
Chapter 5
Installation and Configuration
Installation Overview................................................................................................ 5-2
Installing the Hardware............................................................................................. 5-3
Setting Up the Nodes......................................................................................... 5-3
Installing the Fibre Host Adapter ...................................................................... 5-3
Installing the Cluster Interconnect Adapters ..................................................... 5-4
Installing the Client LAN Adapters................................................................... 5-4
Setting Up and Configuring the RA4000 External Storage Subsystem............ 5-5
Cabling the Cluster Interconnect ....................................................................... 5-6
Cabling the Client LAN................................................................................... 5-11
Power Distribution and Power Sequencing..................................................... 5-12
Installing the Operating System Software .............................................................. 5-14
Using SmartStart.............................................................................................. 5-15
Compaq OSD Installation....................................................................................... 5-18
Verifying Installation of the SNMP Service ................................................... 5-19
Verifying Cluster Communications................................................................. 5-20
Mounting Remote Drives and Verifying Administrator Privileges ................ 5-21
Installing Ethernet OSDs ................................................................................. 5-22
Installing ServerNet OSDs, Drivers, and SNMP Agents ............................... 5-31
Verifying the ServerNet Cluster Interconnect................................................. 5-42
Oracle Installation................................................................................................... 5-42
Configuring Oracle Software.................................................................................. 5-43
Additional Notes on Configuring Oracle Software......................................... 5-43
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
vi Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Installation and Configuration
continued
Installing Object Link Manager...............................................................................5-44
Verifying the Hardware and Software Installation..................................................5-45
Cluster Communications ..................................................................................5-45
Access to Shared Storage from All Nodes .......................................................5-45
OSDs.................................................................................................................5-45
Chapter 6
Cluster Management
Cluster Management Concepts..................................................................................6-2
Powering Off a Node Without Interrupting Cluster Services ............................6-2
Managing Network Clients Connected to a Cluster...........................................6-2
Managing a Cluster in a Degraded Condition....................................................6-3
Cluster Events.....................................................................................................6-3
Management Applications.........................................................................................6-4
Monitoring Cluster Hardware ............................................................................6-4
Managing Shared Drives ....................................................................................6-4
Monitoring the Database ....................................................................................6-5
Remotely Managing a Cluster............................................................................6-5
Software Maintenance ...............................................................................................6-6
Deinstalling the Operating System Dependent Modules ..................................6-6
Upgrading Oracle8i Server.................................................................................6-9
Replacing/Adding A Cluster Node..........................................................................6-10
Replacing A Cluster Node................................................................................6-10
Adding a Cluster Node .....................................................................................6-15
Managing Changes to Shared Storage.....................................................................6-18
Replacing a Failed Disk ...................................................................................6-18
Adding a Disk to Increase Storage Capacity....................................................6-19
Adding a Shared Storage Array........................................................................6-20
Monitoring Cluster Performance .............................................................................6-22
Chapter 7
Troubleshooting
Basic Troubleshooting Tips.......................................................................................7-2
Power ..................................................................................................................7-2
Physical Connections .........................................................................................7-2
Accessibility .......................................................................................................7-3
Software Revisions.............................................................................................7-3
Firmware Revisions............................................................................................7-4
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide vii
Troubleshooting
continued
Oracle and OSD Installation Problems and Error Messages.................................... 7-5
Difficulties installing the OSDs with the Oracle Universal Installer ................ 7-5
Unable to Start OracleCMService ..................................................................... 7-6
Unable to Start OracleService ........................................................................... 7-6
Unable to Start the Database ............................................................................. 7-7
Initialization of the Dynamic Link Library NM.DLL Failed........................... 7-7
Troubleshooting Node-to-Node Connectivity Problems.......................................... 7-8
Nodes Are Unable to Communicate with Each Other ...................................... 7-8
Viping Does Not Complete Successfully .......................................................... 7-9
Unable to Ping the Cluster Interconnect or the Client LAN .......................... 7-10
Node or Nodes Unable to Rejoin the Cluster .................................................. 7-11
Ping Shows the Cluster Interconnect IP Address
Instead of Client LAN IP Address................................................................... 7-11
Troubleshooting Client-to-Cluster Connectivity Problems.................................... 7-12
A Network Client Cannot Communicate with the Cluster............................. 7-12
Troubleshooting Shared Storage Problems ............................................................ 7-13
Node Cannot Connect to the Shared Drives.................................................... 7-13
Drives in the RA4000 Array Are Not Recognized.......................................... 7-14
Troubleshooting Other Potential Problems ............................................................ 7-15
NT Blue Screen With AFD.SYS Failure Displayed ....................................... 7-15
Appendix A
viping Utility
Glossary
Index
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide
Purpose
This Administrator Guide provides information about the planning,
installation, configuration, and implementation of the Compaq Parallel
Database Cluster.
Audience
The expected audience of this guide consists primarily of MIS professionals
whose jobs include designing, installing, configuring, and maintaining
Compaq Parallel Database Clusters.
The audience of this guide must have a working knowledge of Microsoft
Windows NT Server and of Oracle databases or have the assistance of a
database administrator.
This guide contains information for network administrators, database
administrators, installation technicians, systems integrators, and other
technical personnel in the enterprise environment for the purpose of cluster
planning, installation, implementation, and maintenance.
IMPORTANT: This guide contains installation, configuration, and maintenance
information that can be valuable for a variety of users. If you are installing the Compaq
Parallel Database Cluster but will not be administering the cluster on a daily basis, please
make this guide available to the person or persons who will be responsible for the
clustered servers after you have completed the installation.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
x Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Scope
This guide offers significant background information about clusters as well as
basic concepts associated with designing clusters. It also contains detailed
product descriptions and installation steps.
This administrator guide is designed to assist you in the following objectives:
3 Understanding basic concepts of clustering technology
3 Recognizing and using the high availability features of the Compaq
Parallel Database Cluster Model PDC/ 1000
3 Planning and designing your Compaq Parallel Database Cluster
configuration to meet your business needs
3 Installing and configuring your PDC/ 1000 hardware and software
3 Managing your PDC/ 1000 cluster
The contents of this guide are outlined below:
3 Chapter 1, "Clustering Overview," provides an introduction to
clustering technology features and benefits.
3 Chapter 2, "Architecture of the Compaq Parallel Database Cluster"
describes the hardware components of the PDC/ 1000.
3 Chapter 3, "Cluster Software Components" describes software
components used with the PDC/ 1000.
3 Chapter 4, "Planning," outlines an approach to planning and designing
cluster configurations that meet your business needs
3 Chapter 5, "Installation and Configuration" outlines the steps you will
take to install and configure PDC/ 1000 hardware and software.
3 Chapter 6, "Cluster Management," includes techniques for managing
and maintaining your PDC/ 1000 cluster.
3 Chapter 7, "Troubleshooting," contains troubleshooting information for
the PDC/ 1000.
3 Appendix A, "viping Utility," documents the use of the viping utility to
test the ServerNet cluster interconnect.
3 The Glossary contains definitions of many terms used in this guide.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide xi
Some clustering topics are mentioned, but not detailed, in this guide. For
example, this guide does not describe how to install and configure Oracle8i on
a cluster.
Referenced Manuals
For additional information, refer to documentation related to the specific
hardware and software components of your Compaq Parallel Database Cluster.
These include but are not limited to:
3 Documentation related to the ProLiant servers you are clustering (for
example, guides, posters, Performance and Tuning guides)
3 Compaq ServerNet documentation
Y ServerNet PCI Adapter Installation Guide
Y ServerNet Switch Installation Guide
3 Compaq StorageWorks documentation
Y Compaq StorageWorks RAID Array 4000 User Guide
Y Compaq StorageWorks Fibre Channel Host Adapter Installation
Guide
Y Compaq StorageWorks Fibre Channel Storage Hub 7 Installation
Guide
Y Compaq StorageWorks Fibre Channel Storage Hub 12 Installation
Guide
Y Compaq Fibre Channel Troubleshooting Guide
3 Microsoft Windows NT Server documentation
Y Microsoft Windows NT Server Administrator's Guide
Y Microsoft Windows NT Server/Enterprise Edition Administrator's
Guide
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
xii Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
3 Oracle8i Release 8.1.5 documentation
Y Oracle8i Parallel Server Setup and Configuration Guide Release
8.1.5
Y Oracle8i Enterprise Edition Installation Release 8.1.5 for Windows
NT and Windows 95
Y Oracle8i Enterprise Edition for Windows NT and Windows 95/98,
Release 8.1.5
Supplemental Documents
The following technical documents contain important supplement information
for the Compaq Parallel Database Cluster Model PDC/ 1000:
3 Supported Ethernet Interconnects for Compaq Parallel Database
Clusters Using Oracle Parallel Server (ECG062/0299), located at
www.compaq.com/highavailability/
3 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i
Parallel Server Release 8.1.5 Certification Matrix, located at
www.compaq.com/highavailability/
3 Configuring Compaq RAID Technology for Database Servers, technote
# 1842061 located at www.compaq.com/highavailability/
3 Various technical whitepapers on Oracle8i and cluster sizing, which are
available from Compaq's activeAnswers website at
www.compaq/activeanswers/
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide xiii
Text Conventions
This document uses the following conventions to distinguish elements of text:
Keys Keys appear in boldface. A plus sign (+) between
two keys indicates that they should be pressed
simultaneously.
USER INPUT User input appears in a different typeface and in
uppercase.
FILENAMES File names appear in uppercase italics.
Menu Options, These elements appear in initial capital letters.
Command Names,
Dialog Box Names
COMMANDS, These elements appear in uppercase.
DIRECTORY NAMES,
and DRIVE NAMES
Type When you are instructed to type information, type
the information without pressing the Enter key.
Enter When you are instructed to enter information, type
the information and then press the Enter key.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
xiv Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Symbols in Text
These symbols may be found in the text of this guide. They have the following
meanings.
WARNING: Text set off in this manner indicates that failure to follow directions
in the warning could result in bodily harm or loss of life.
CAUTION: Text set off in this manner indicates that failure to follow directions
could result in damage to equipment or loss of information.
IMPORTANT: Text set off in this manner presents clarifying information or specific
instructions.
NOTE: Text set off in this manner presents commentary, sidelights, or interesting points
of information.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide xv
Symbols on Equipment
These icons may be located on equipment in areas where hazardous conditions
may exist.
Any surface or area of the equipment marked with these symbols
indicates the presence of electrical shock hazards. Enclosed area
contains no operator serviceable parts.
WARNING: To reduce the risk of injury from electrical shock hazards,
do not open this enclosure.
Any RJ-45 receptacle marked with these symbols indicates a Network
Interface Connection.
WARNING: To reduce the risk of electrical shock, fire, or damage to
the equipment, do not plug telephone or telecommunications
connectors into this receptacle.
Any surface or area of the equipment marked with these symbols
indicates the presence of a hot surface or hot component. If this
surface is contacted, the potential for injury exists.
WARNING: To reduce the risk of injury from a hot component, allow
the surface to cool before touching.
Power Supplies or Systems marked with these symbols
indicate the equipment is supplied by multiple sources of
power.
WARNING: To reduce the risk of injury from electrical shock,
remove all power cords to completely disconnect power from
the system.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
xvi Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Getting Help
If you have a problem and have exhausted the information in this guide, you
can get further information and other help in the following locations.
Compaq Technical Support
You are entitled to free hardware technical telephone support for your product
for as long you own the product. A technical support specialist will help you
diagnose the problem or guide you to the next step in the warranty process.
In North America, call the Compaq Technical Phone Support Center at
1
1-800-OK-COMPAQ . This service is available 24 hours a day, 7 days a week.
Outside North America, call the nearest Compaq Technical Support Phone
Center. Telephone numbers for world wide Technical Support Centers are
listed on the Compaq website. Access the Compaq website at
www.compaq.com.
Be sure to have the following information available before you call Compaq:
3 Technical support registration number (if applicable)
3 Product serial number (s)
3 Product model name(s) and numbers(s)
3 Applicable error messages
3 Add-on boards or hardware
3 Third-party hardware or software
3 Operating system type and revision level
3 Detailed, specific questions
Compaq Website
The Compaq website has information on this product as well as the latest
drivers and Flash ROM images. You can access the Compaq website by
logging on to the Internet at www.compaq.com.
1
For continuous quality improvement, calls may be recorded or monitored.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
About This Guide xvii
Compaq Authorized Reseller
For the name of your nearest Compaq Authorized Reseller:
3 In the United States, call 1-800-345-1518.
3 In Canada, call 1-800-263-5868.
3 Elsewhere, see the Compaq website for locations and telephone
numbers.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: a-frnt.doc Last Saved On: 5/18/99 4:18 PM
1
Chapter
Clustering Overview
For many years, companies have depended on clustered computer systems to
fulfill two key requirements: to ensure users can access and process
information that is critical to the ongoing operation of their business, and to
increase the performance and throughput of their computer systems at minimal
cost. These requirements are known as availability and scalability,
respectively.
Historically, these requirements have been fulfilled with clustered systems
built on proprietary technology. Over the years, open systems have
progressively and aggressively moved proprietary technologies into
industry-standard products. Clustering is no exception. Its primary features,
availability and scalability, have been moving into client/server products for
the last few years.
The absorption of clustering technologies into open systems products is
creating less expensive, non-proprietary solutions that deliver levels of
functionality commonly found in traditional clusters. While some uses of the
proprietary solutions will always exist--such as those controlling stock
exchange trading floors and aerospace mission controls--many critical
applications can reach the desired levels of availability and scalability with
non-proprietary client/server-based clustering.
These new clustering solutions use industry-standard hardware and software,
thereby providing key clustering features at a lower price than proprietary
clustering systems. Before examining the features and benefits of the Compaq
Parallel Database Cluster Model PDC/ 1000 (PDC/ 1000), it is helpful to
understand the concepts and terminology of clustered systems.
Compaq Confidential Need to Know Required
Writer: Linda Waterhouse Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Comments: 104286-001
Part Number: 104286-001 File Name: b-ch1 Clustering Overview.doc Last Saved On: 5/6/99 4:08 PM
1-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Clusters Defined
A cluster is an integration of software and hardware products that enables a set
of loosely coupled servers and shared storage subsystem components to
present a single system image to clients and to operate as a single system. As a
cluster, the group of servers and shared storage subsystem components offers a
level of availability and scalability far exceeding that obtained if each cluster
node operated as a standalone server.
The PDC/ 1000 uses Oracle8i Parallel Server, which is a parallel database
that can distribute its workload among the cluster nodes.
Figure 1-1 shows an example of a PDC/ 1000, including six nodes, one
Compaq StorageWorks RAID Array 4000 (RA4000 Array), one Compaq
StorageWorks Fibre Channel Storage Hub (Storage Hub), and one Ethernet or
ServerNet Switch for the cluster interconnect. Clients access the database
through the client local area network (LAN).
RA4000 Array
Node 2 Node 4 Node 6
Node 5 Node 3 Node 1
Storage Hub
Switch
(Cluster Interconnect)
Client LAN
Clients
Figure 1-1. Example of a Compaq Parallel Database Cluster
Model PDC/ 1000
Compaq Confidential Need to Know Required
Writer: Linda Waterhouse Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Comments: 104286-001
Part Number: 104286-001 File Name: b-ch1 Clustering Overview.doc Last Saved On: 5/6/99 4:08 PM
Clustering Overview 1-3
Availability
When computer systems experience outages, the amount of time the system is
unavailable is referred to as downtime. Downtime has several primary causes:
hardware faults, software faults, planned service, operator error, and
environmental factors. Minimizing downtime is a primary goal of a cluster.
Simply defined, availability is the measure of how well a computer system can
continuously deliver services to clients.
Availability is a system-wide endeavor. The hardware, the operating system,
and the applications must be designed for availability. Clustering requires
stability in these components, then couples them in such a way that failure of
one item does not render the system unusable. By using redundant components
and mechanisms that detect and recover from faults, clusters can greatly
increase the availability of applications critical to business operations.
Scalability
Simply defined, scalability is a computer system characteristic that enables
improved performance or throughput when supplementary hardware resources
are added. Scalable systems allow increased throughput by adding components
to an existing system without the expense of adding a new system.
In a stand-alone server configuration, scalable systems allow increased
throughput by adding processors or more memory. In a cluster configuration,
this result is usually obtained by adding cluster nodes.
Not only must the hardware benefit from additional components, but also
software must be constructed in such a way as to take advantage of the
additional processing power. The PDC/ 1000 uses Oracle8i Parallel Server,
which is a parallel database that can distribute its workload among the cluster
nodes. As more nodes are brought into the cluster, cluster-aware applications
can use the parallel features of Oracle8i Parallel Server to distribute workload
among more servers, thereby obtaining greater throughput.
Compaq Confidential Need to Know Required
Writer: Linda Waterhouse Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Comments: 104286-001
Part Number: 104286-001 File Name: b-ch1 Clustering Overview.doc Last Saved On: 5/6/99 4:08 PM
1-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Compaq Parallel Database
Cluster Overview
As traditional clustering technology has moved into the open systems of
client/server computing, Compaq has provided innovative, customer-focused
solutions. The PDC/ 1000 moves client/server computing one step closer to
the capabilities found in expensive, proprietary cluster solutions, at a fraction
of the cost.
The PDC/ 1000 combines the popular Microsoft Windows NT Server
operating system and the industry-leading Oracle8i Parallel Server with
award-winning Compaq ProLiant servers and shared storage subsystems.
Together, these hardware and software components provide improved
performance through a truly scalable parallel application and improved
availability using clustering software that rapidly recovers from detectable
faults. These components also provide improved availability through
concurrent multinode database access using Oracle8i Parallel Server.
Compaq Confidential Need to Know Required
Writer: Linda Waterhouse Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Comments: 104286-001
Part Number: 104286-001 File Name: b-ch1 Clustering Overview.doc Last Saved On: 5/6/99 4:08 PM
2
Chapter
Architecture of the Compaq Parallel
Database Cluster
The Compaq Parallel Database Cluster Model PDC/ 1000 (PDC/ 1000) is
an integration of a number of different hardware and software products. This
chapter discusses how each of these products plays a role in bringing a
complete clustering solution to your computing environment.
The hardware products include:
3 Compaq ProLiant servers
3 Shared storage subsystem components
Y Compaq StorageWorks RAID Array 4000 (RA4000 Array)
Y Compaq StorageWorks RAID Array 4000 Array Controller
(RA4000 Array Controller)
Y Compaq StorageWorks Storage Hub (Storage Hub)
Y Compaq StorageWorks Fibre Channel Host Bus Adapter (Fibre Host
Adapter)
3 Cluster interconnect components
Y NIC adapters (Ethernet or ServerNet)
Y cables (Ethernet or ServerNet)
Y switches (Ethernet or ServerNet)
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
2-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
The software products include:
3 Microsoft Windows NT Server 4.0 with Service Pack 3 or 4
3 Compaq drivers and utilities
3 Oracle8i Enterprise Edition with the Oracle8i Parallel Server Option
IMPORTANT: Windows NT Server 4.0 with Service Pack 4 is recommended for proper
operation of the redundant Ethernet cluster interconnect. However, because certain
applications may not work with Service Pack 4, you may need to use Service Pack 3 with
an approved Microsoft hotfix to support a redundant Ethernet cluster interconnect. Consult
with your software expert to confirm that your applications can run with Service Pack 4.
Refer to Chapter 3, "Cluster Software Components," for a description of the
software products used with the PDC/ 1000.
Compaq ProLiant Servers
A primary component of any cluster is the server. Each PDC/ 1000 cluster
consists of nodes, where each node is a Compaq ProLiant server.
All nodes in a PDC/ 1000 cluster must be identical in model. In addition, all
components common to all nodes in a cluster, such as memory, number of
CPUs, and the interconnect adapters, must be identical and identically
configured.
NOTE: For an up-to-date list of Compaq Parallel Database Cluster Model
PDC/O1000-certified servers and detailed information about minimum and maximum
cluster configurations, refer to the Compaq Parallel Database Cluster Model PDC/ 1000
for Oracle8i Parallel Server Release 8.1.5 Certification Matrix. This document is available
on the Compaq website at www.compaq.com/highavailability/
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
Architecture of the Compaq Parallel Database Cluster 2-3
Figure 2-1 shows the major hardware components contained in a six-node
PDC/ 1000 cluster.
RA4000 Array
Node 2 Node 4 Node 6
Node 5 Node 3 Node 1
Storage Hub
Switch
(Cluster Interconnect)
Client LAN
Clients
Figure 2-1. Diagram of a six-node PDC/ 1000 cluster
High Availability Features of ProLiant Servers
ProLiant servers include many reliability features that provide a solid
foundation for effective clustered server solutions. The PDC/ 1000 cluster is
based on ProLiant servers, most of which offer excellent reliability through
redundant power supplies, redundant cooling fans, and Error Checking and
Correcting (ECC) memory. Table 2-1 lists the high-availability features found
in many ProLiant servers.
Table 2-1
High Availability Components of ProLiant Servers
Hot-Pluggable Hard Drives Redundant Power Supplies
Digital Linear Tape (DLT) Array (optional) ECC-Protected Processor-Memory Bus
Uninterruptible Power Supplies (optional) Redundant Processor Power Modules
ECC Memory Hot Plug PCI slots (in some servers)
Offline Backup Processor Redundant Cooling Fans
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
2-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Shared Storage Components
Shared Storage Clustering
NOTE: In this guide, the term "Fibre Channel Storage System" has been replaced by the
term "RA4000 External Storage Subsystem" when referring to the shared storage
subsystem components (RA400 Arrays, RA4000 Array Controllers, and Storage Hubs)
used in the PDC/ 1000 cluster.
The PDC/ 1000 cluster is based on a cluster architecture known as Shared
Storage Clustering, in which clustered servers share access to a common set of
hard drives. All clustered (shared) data is stored in an external storage
subsystem.
The RA4000 External Storage Subsystem is the shared storage solution for the
PDC/ 1000 cluster. The RA4000 External Storage Subsystem brings these
advantages to clustering:
3 Throughput of up to 100 MB/sec
3 Server-to-storage distances of up to 1000 meters
3 Increased connectivity and ease of use
3 Hot-pluggable drives
3 Hot-pluggable fan assembly
3 Hot-pluggable power supply (in redundant power supply configurations)
3 Manageability through Compaq Insight Manager
Throughout this guide, you will see references to the RA4000 External Storage
Subsystem for the PDC/ 1000 cluster. When this term is used, it refers to all
components that make up the storage system itself. The three primary
components of the RA4000 External Storage Subsystem are:
3 Storage Hub 7 or Storage Hub 12
3 RA4000 Array
3 RA4000 Array Controller
A fourth component of Compaq Fibre Channel technology is the Fibre Host
Adapter, an adapter card that resides in each server. While the Fibre Host
Adapter is not part of the RA4000 External Storage Subsystem per se, it plays
a key role in getting data to and from the servers in a cluster.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
Architecture of the Compaq Parallel Database Cluster 2-5
Each of these components is further discussed in the following sections of this
chapter. For more detailed information, refer to these Compaq guides:
3 Compaq StorageWorks RAID Array 4000 User Guide
3 Compaq StorageWorks Fibre Channel Host Adapter Installation Guide
3 Compaq StorageWorks Fibre Channel Hub 7 Installation Guide
3 Compaq StorageWorks Fibre Channel Hub 12 Installation Guide
Availability Features of the RA4000 External
Storage Subsystem
An important part of a high-availability system is the ability to improve data
availability, traditionally accomplished by implementing RAID technology.
Hardware RAID is an important part of the RA4000 External Storage
Subsystem. RAID is implemented on the RA4000 Array Controller, which is
located in the RA4000 Array. The RA4000 Array also accepts redundant,
hot-pluggable power supplies and a hot-pluggable fan module.
The RA4000 Array Controller supports pre-failure notification on hard drives
and provides an Array Accelerator made with ECC memory. The Array
Accelerator is backed with onboard rechargeable batteries, ensuring that the
data temporarily held (cached) is safe even with equipment failure or power
outage. For a complete list of features and accompanying descriptions, refer to
the Compaq StorageWorks RAID Array 4000 User Guide.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
2-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Maximum Distances Between Cluster Nodes
and Shared Storage Subsystem Components
By using standard short-wave Fibre Channel cables, the RA4000 Array can be
placed up to 500 meters from the Storage Hub, and the Storage Hub can be
placed up to 500 meters from the cluster nodes. See Figure 2-2.
RA4000 Array
500 m
500 m 500 m
Node 2 Node 4
Node 3 Node 1
500 m 500 m
Storage Hub
Figure 2-2. Maximum distances between cluster nodes and shared
storage components
Using Multiple Fibre Channel Loops
in a PDC/ 1000 Cluster
The PDC/ 1000 cluster supports the use of multiple Fibre Channel loops
within a cluster. You would install additional Fibre Channel loops to a
PDC/ 1000 cluster to:
3 Increase the amount of shared storage available to the cluster's servers
when your first Storage Hub 12 is filled to capacity. With just one Fibre
Channel loop present, your shared storage resources are restricted by the
number of ports available on the Storage Hub.
3 Increase the PDC/ 1000 cluster's I/O performance.
Consider the original RA4000 External Storage Subsystem in the cluster to be
the first Fibre Channel loop. Adding one or more additional Fibre Channel
loops involves duplicating the hardware used in the first loop. Note, however,
that each additional Fibre Channel loop can contain fewer RA4000 Arrays
than the first Fibre Channel loop.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
Architecture of the Compaq Parallel Database Cluster 2-7
For the PDC/ 1000 cluster, the Fibre Channel loop hardware includes a Fibre
Host Adapter in each server, a Storage Hub, one or more RA4000 Arrays with
one RA4000 Array Controller each, and Fibre Channel cables. Up to four
Fibre Channel loops can be installed in a single PDC/ 1000 cluster.
Figure 2-3 shows a four-node PDC/ 1000 cluster with two Fibre Channel
loops. Each Fibre Channel loop has its own Fibre Host Adapter in each server,
Storage Hub, and RA4000 Arrays. In the diagram, the components that
constitute the second Fibre Channel loop are shaded.
RA4000
Arrays (8)
Fibre Fibre
Storage Hubs
Host Adapters Host Adapters
RA4000
Arrays (4)
Figure 2-3. PDC/ 1000 cluster with two Fibre Channel loops
In Figure 2-3, the original Fibre Channel loop contains eight RA4000 Arrays.
The second Fibre Channel loop contains four RA4000 Arrays.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
2-8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
RA4000 External Storage Subsystem Components
RA4000 Array
The RA4000 Array is the storage enclosure that contains the disk drives,
power supply, and one RA4000 Array Controller. The RA4000 Array can hold
twelve 1-inch or eight 1.6-inch Wide-Ultra SCSI drives. The Compaq RA4000
Array comes in either a rack-mountable or a tower model.
A PDC/ 1000 cluster must have at least one RA4000 Array set up as external
shared storage. The maximum number of RA4000 Arrays you can install in
one Fibre Channel loop of a PDC/ 1000 cluster depends on:
3 How many ports are present on the Storage Hub (Storage Hub 7 or
Storage Hub 12)
3 How many nodes are present in the cluster.
For example, if you used a Storage Hub 12 in a four-node PDC/ 1000 cluster,
you could install a maximum of eight RA4000 Arrays.
Using the 1-inch high, 9-GB drives provides a maximum storage capacity of
108 GB per RA4000 Array (twelve 9-GB drives). Using the 1.6-inch high,
18-GB drives provides a maximum storage capacity of 144 GB per RA4000
Array (eight 18 GB drives).
NOTE: This release of the PDC/ 1000 cluster supports multiple Fibre Channel loops.
Physically, this means that within each node in a cluster, multiple Fibre Host Adapters can
be used to connect the nodes to a different RA4000 External Storage Subsystem. See
"Using Multiple Fibre Channel Loops in a PDC/ 1000 Cluster" for more information.
For more information about the RA4000 Array, refer to the Compaq
StorageWorks RAID Array 4000 User Guide.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
Architecture of the Compaq Parallel Database Cluster 2-9
Figure 2-4 shows three Compaq RA4000 Arrays present in a two-node
PDC/ 1000 cluster.
RA4000 Array
RA4000 Array RA4000 Array
Node 2
Node 1
Fibre Fibre
Host Adapter Host Adapter
ProLiant ProLiant
Server Server
Storage Hub
Switch
(Cluster Interconnect)
Client LAN
Figure 2-4. Three RA4000 Arrays in a two-node PDC/ 1000 cluster
RA4000 Array Controller
A single RA4000 Array Controller, which resides in the RA4000 Array, is
included with your RA4000 External Storage Subsystem. This controller
provides RAID and caching capabilities to the RA4000 External Storage
Subsystem. The cable connection from the Storage Hub is made to this
controller.
For more information about the RA4000 Array Controller, refer to the
Compaq StorageWorks RAID Array 4000 User Guide.
Storage Hub
A Storage Hub is required to create a Fibre Channel path from one or more
RA4000 Arrays to each server in the PDC/ 1000 cluster. Each Fibre Channel
loop in a PDC/ 1000 cluster requires its own Storage Hub.
The Storage Hub comes in two models: Storage Hub 7 and Storage Hub 12.
Either type can be used in a PDC/ 1000 cluster. However, the combined
number of nodes and RA4000 Arrays cannot exceed the number of ports on
the Storage Hub.
Each node in the cluster connects to one port on the Storage Hub, as does each
RA4000 Array. For example, in a three-node cluster with two RA4000 Arrays,
three ports on the Storage Hub are used by the cluster nodes and two are used
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
2-10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
by the RA4000 Arrays. The remaining unoccupied ports can be used for to
accommodate new cluster nodes or RA4000 Arrays.
Refer to the Compaq StorageWorks Fibre Channel Storage Hub 7 Installation
Guide and the Compaq StorageWorks Fibre Channel Storage Hub 12
Installation Guide for further information about these products.
Figure 2-5 shows a Storage Hub connected to shared storage in a two-node
PDC/ 1000 cluster.
RA4000 Array RA4000 Array
RA4000 Array
Node 2
Node 1
Fibre Fibre
Host Adapter Host Adapter
ProLiant ProLiant
Server Server
Storage Hub
Switch
(Cluster Interconnect)
Client LAN
Figure 2-5. Storage Hub connects the shared storage to servers in a
two-node PDC/ 1000 cluster
IMPORTANT: Currently, the Storage Hub cannot be cascaded (you cannot connect the
port of one Storage Hub to a port on another Storage Hub).
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
Architecture of the Compaq Parallel Database Cluster 2-11
Fibre Host Adapter
Fibre Host Adapters are the interface between the servers and the RA4000
External Storage Subsystem. Each node in a PDC/ 1000 cluster requires one
Fibre Host Adapter to interface with the Storage Hub. A Fibre Channel cable
runs from each Fibre Host Adapter to a port on the Storage Hub
(see Figure 2-6).
RA4000 Array RA4000 Array
RA4000 Array
Node 2
Node 1
Fibre Fibre
Host Adapter Host Adapter
ProLiant ProLiant
Server Server
Storage Hub
Switch
(Cluster Interconnect)
Client LAN
Figure 2-6. Fibre Host Adapters connect the servers to the Storage Hub in a
two-node PDC/ 1000 cluster
For more information about the Fibre Host Adapter, refer to the Compaq
StorageWorks Fibre Channel Host Adapter Installation Guide.
Fibre Channel Data Paths
The RA4000 External Storage Subsystem has two distinct data paths,
separated by the Storage Hub:
3 One data path runs from the Fibre Host Adapter in the servers to the
Storage Hub.
3 Another data path runs from the Storage Hub to each RA4000 Array.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
2-12 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide
Fibre Host Adapter-to-Storage Hub Data Path
Multiple connections occur in this data path. Each server has a single
connection to the Storage Hub. See Figure 2-7.
Client LAN
Fibre Fibre
Host Adapters Host Adapters
Switch
(Cluster Interconnect)
ProLiant ProLiant
Servers Servers
Storage Hub
RA4000 Array
Figure 2-7. Fibre Host Adapter-to-Storage Hub data path
If one of these connections experiences a fault, the connections from the other
servers ensure continued access to the database. The fault results in the
eviction of the cluster node with the failed connection. All network clients
accessing the database through that node must reconnect through another
cluster node. The effect of this failure is relatively minor. It affects only those
users who are connected to the database through the affected node. The
duration of downtime includes the time to detect the failure, the time to
reconfigure from the failure, and the time required for the network clients to
reconnect to the database through another node.
Note that Compaq Insight Manager monitors the health of the RA4000
External Storage Subsystem. If any part of the Fibre Channel data path
disrupts a server's access to a RA4000 Array, the Fibre Array Controller status
changes to "Failed" and the condition is red. The red condition is reported to
higher-level Insight Manager screens, and eventually to the device list. Refer
to the Compaq Insight Manager Guide for details.
Compaq Confidential Need to Know Required
Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i Release 8.1.5 Administrator Guide Comments:
Part Number: 104286-001 File Name: c-ch2 Architecture.doc Last Saved On: 5/18/99 2:16 PM
Architecture of the Compaq Parallel Database Cluster 2-13
Storage Hub-to-RA4000 Array Data Path
| 104286-001
|
