First Edition (November 1998)
Part Number 386751-001
Compaq Computer Corporation
Notice
The information in this publication is subject to change without notice.
COMPAQ COMPUTER CORPORATION SHALL NOT BE LIABLE FOR TECHNICAL OR
EDITORIAL ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR
CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR
USE OF THIS MATERIAL. THIS INFORMATION IS PROVIDED "AS IS" AND COMPAQ
COMPUTER CORPORATION DISCLAIMS ANY WARRANTIES, EXPRESS, IMPLIED OR
STATUTORY AND EXPRESSLY DISCLAIMS THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR PARTICULAR PURPOSE, GOOD TITLE AND AGAINST
INFRINGEMENT.
This publication contains information protected by copyright. No part of this publication may be
photocopied or reproduced in any form without prior written consent from Compaq Computer
Corporation.
1998 Compaq Computer Corporation.
All rights reserved. Printed in the U.S.A.
The software described in this guide is furnished under a license agreement or nondisclosure agreement.
The software may be used or copied only in accordance with the terms of the agreement.
Compaq, Deskpro, Fastart, Compaq Insight Manager, Systempro, Systempro/LT, ProLiant, ROMPaq,
QVision, SmartStart, NetFlex, QuickFind, PaqFax, ProSignia, registered United States Patent and
Trademark Office.
Netelligent, Systempro/XL, SoftPaq, QuickBlank, QuickLock are trademarks and/or service marks of
Compaq Computer Corporation.
Microsoft, MS-DOS, Windows, and Windows NT are registered trademarks of Microsoft Corporation.
Oracle is a registered trademark and Oracle8 is a trademark of Oracle Corporation.
Other product names mentioned herein may be trademarks and/or registered trademarks of their
respective companies.
Compaq Parallel Database Cluster Model PDC/1000
First Edition (November 1998)
Part Number 386751-001
iii
Contents
About This Guide
P u r p o s e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . v i i
Audience.................................................................................................................................... vii
Scope ........................................................................................................................................ viii
Additional Resources .................................................................................................................. x
Text Conventions ....................................................................................................................... xi
Symbols in Text ........................................................................................................................ xii
Symbols on Equipment ............................................................................................................. xii
Rack Stability ........................................................................................................................... xiii
Getting Help ............................................................................................................................. xiv
Compaq Website ............................................................................................................... xiv
Telephone Numbers .......................................................................................................... xiv
PART I: Introduction
Chapter 1
Clustering Overview
Clusters Defined ....................................................................................................................... 1-2
Availability ............................................................................................................................... 1-2
Scalability ................................................................................................................................. 1-3
Compaq Parallel Database Cluster ........................................................................................... 1-4
Chapter 2
Architecture of the Compaq Parallel Database Cluster
Overview of the Compaq Parallel Database Cluster Components ........................................... 2-1
Hardware Overview ................................................................................................................. 2-2
Compaq ProLiant Servers ................................................................................................. 2-2
High Availability Features of ProLiant Servers ................................................................ 2-3
Clustering Shared Storage.................................................................................................2-3
Availability Features of Shared Storage............................................................................2-5
Compaq Fibre Channel Storage System............................................................................2-6
Fibre Channel Host Controller .......................................................................................... 2-8
Fibre Channel Data Paths .................................................................................................. 2-9
Host Controller-to-Fibre Channel Storage Hub Data Path .............................................. 2-10
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
iv
Architecture of the Compaq Parallel Database Cluster continued
Fibre Channel Storage Hub-to-Fibre Channel Array Data Path......................................2-11
SCSI Disks ...................................................................................................................... 2-12
Cluster Interconnect ........................................................................................................ 2-12
Local Area Network ........................................................................................................ 2-15
Software Overview.................................................................................................................2-16
Microsoft Software..........................................................................................................2-16
Compaq Software............................................................................................................2-17
Oracle Software...............................................................................................................2-20
Client Reconnect Software..............................................................................................2-21
Oracle Certification ................................................................................................................ 2-22
PART II: Planning, Installation, and Configuration
Chapter 3
Planning
Site Planning ............................................................................................................................ 3-1
Capacity Planning.....................................................................................................................3-1
Cluster Hardware Components ......................................................................................... 3-2
Cluster Software Components...........................................................................................3-3
Sample Maximum Configuration......................................................................................3-4
RAID Planning ......................................................................................................................... 3-5
Calculating Raw Data Storage and Database Size ............................................................ 3-5
Selecting RAID Levels......................................................................................................3-6
Fibre Channel Disk Grouping .................................................................................................. 3-7
Network Considerations ........................................................................................................... 3-9
Windows NT Server Hosts files ........................................................................................ 3-9
Client LAN........................................................................................................................3-9
Disk Drive Considerations ..................................................................................................... 3-10
Non-Shared Disk Drives ................................................................................................. 3-10
Shared Disk Drives..........................................................................................................3-10
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
v
Chapter 4
Installation and Configuration
Installation Overview ............................................................................................................... 4-2
Installing the Hardware ............................................................................................................ 4-3
Setting Up the Nodes.........................................................................................................4-3
Setting Up and Configuring the Compaq Fibre Channel Storage System.........................4-5
Setting up the Cluster Interconnect and Client LAN.........................................................4-7
Power Distribution and Power Sequencing.......................................................................4-8
Installing the Software............................................................................................................4-10
Using SmartStart ............................................................................................................. 4-11
Oracle Installation ........................................................................................................... 4-14
Compaq OSD Installation ............................................................................................... 4-15
Configuring Oracle Software ................................................................................................. 4-18
Notes on Configuring Oracle Software ........................................................................... 4-19
Verifying the Hardware and Software Installation.................................................................4-21
Cluster Communications ................................................................................................. 4-21
Access to Shared Storage from All Nodes ...................................................................... 4-22
OSDs ............................................................................................................................... 4-22
Client Access to the OPS Database ................................................................................. 4-22
PART III: Cluster Management
Chapter 5
Cluster Management
Cluster Management Concepts.................................................................................................5-1
Managing a Cluster Without Interrupting Cluster Services .............................................. 5-1
Managing a Cluster in a Degraded Condition ................................................................... 5-2
Managing Network Clients Connected to a Cluster .......................................................... 5-3
Cluster Events ................................................................................................................... 5-3
Management Applications........................................................................................................5-4
Monitoring Cluster Hardware ........................................................................................... 5-4
Monitoring the Database ................................................................................................... 5-5
Remotely Managing a Cluster...........................................................................................5-5
Software Maintenance .............................................................................................................. 5-6
Uninstalling the Operating System Dependent Modules (OSDs) ..................................... 5-6
Upgrading Oracle8 Server.................................................................................................5-6
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
vi
Cluster Management continued
Modifying Physical Cluster Resources.....................................................................................5-7
Managing Changes to Drive Ordering .............................................................................. 5-7
Replacing a Failed Drive...................................................................................................5-9
Replacing A Cluster Node.................................................................................................5-9
Adding a Cluster Node....................................................................................................5-14
Chapter 6
Troubleshooting
Troubleshooting Cluster Connectivity ..................................................................................... 6-1
You are Unable to Ping the Cluster Interconnect or the Client LAN................................6-1
A Network Client Cannot Communicate with the Cluster ................................................ 6-2
Nodes Unable to Communicate with other Nodes in the Cluster......................................6-2
Node or Nodes Unable to Rejoin the Cluster .................................................................... 6-3
Ping Shows Cluster Interconnect IP Address Instead of Client
LAN IP Address ................................................................................................................ 6-3
Troubleshooting Oracle and OSD Installation Problems ......................................................... 6-4
While Running the NodeList Configurator, a Dialog Box Appears Indicating
Inability to Connect to Remote Nodes .............................................................................. 6-4
Error Message Received Indicating Inability to Start Cluster Manager
(CMSRVR.EXE) ............................................................................................................... 6-4
Error Message "Dependent Service Has Not Started" Received While Attempting to
Start the Oracle Service.....................................................................................................6-5
Error Message Received Indicating Inability to Start Oracle Service ............................... 6-5
Error Message Received Indicating Inability to Configure Oracle Using
OPSCONF Utility..............................................................................................................6-5
The Oracle PGMS Service Could Not Be Started When Trying to Create an Oracle
Instance ....................................................................................................................... . . . . . . 6 - 6
Unable to Start the Database ............................................................................................. 6-6
Initialization of the Dynamic Link Library CM.DLL Failed ............................................ 6-7
Troubleshooting Fibre Channel Problems................................................................................6-7
Drives in the Fibre Channel Array are not Recognized .................................................... 6-7
Troubleshooting Other Potential Problems .............................................................................. 6-9
NT Blue Screen With AFD.SYS Failure Displayed..........................................................6-9
Appendix G
Glossary
Index
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
vii
About This Guide
Purpose
This Administrator Guide provides information about the planning, installation,
configuration, and implementation of the Compaq Parallel Database Cluster.
Audience
The expected audience of this guide consists primarily of MIS professionals
whose jobs include designing, installing, configuring, and maintaining Compaq
Parallel Database Clusters.
The audience of this guide must have a working knowledge of Microsoft
Windows NT Server and of Oracle databases, or have the assistance of a
database administrator.
This guide contains information for network administrators, database
administrators, installation technicians, systems integrators, and other technical
personnel in the enterprise environment for the purpose of cluster planning,
installation, implementation, and maintenance.
IMPORTANT: This Guide contains installation, configuration, and
maintenance information that can be valuable for a variety of users. If you are
installing the Compaq Parallel Database Cluster but will not be administering
the cluster on a daily basis, please make this guide available for the person(s)
who will be responsible for the clustered servers when you have completed
the installation.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
viii About This Guide
Scope
This guide offers significant background information about clusters as well as
basic concepts associated with designing clusters. It also contains more
traditional information, such as detailed product descriptions and installation
steps.
Some clustering topics are mentioned, but not detailed, in this guide. For
example, this guide does not describe how to install and configure specific
applications on a cluster. However, Compaq provides several TechNotes that
provide this information for industry-leading applications (see "Additional
Resources" below).
This Administrator Guide is designed to assist you in attaining the following
objectives:
Understanding basic concepts of clustering technology
s
Recognizing and using the high availability features of the Compaq
s
Parallel Database Cluster
Planning and designing your Compaq Parallel Database Cluster
s
configuration to meet your business needs
Installing and configuring your Compaq Parallel Database Cluster
s
hardware and software
Managing your Compaq Parallel Database Cluster
s
The contents of this guide are outlined below:
Part I - "Introduction"
s
Chapter 1, "Clustering Overview," provides an introduction to
t
clustering technology features and benefits.
Chapter 2, "Architecture of the Compaq Parallel Database Cluster,"
t
describes the hardware and software components of the Compaq
Parallel Database Cluster.
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
ix
Part II - "Planning, Installation, and Configuration"
s
Chapter 3, "Planning," outlines an approach to planning and
t
designing a cluster configuration that meets your business needs
Chapter 4, "Installation and Configuration," outlines the steps you
t
will take to install and configure Compaq Parallel Database Cluster
hardware and software.
Part III - "Cluster Management and Troubleshooting"
s
Chapter 6, "Cluster Management," includes techniques for
t
managing and maintaining your Compaq Parallel Database Cluster.
Chapter 7, "Troubleshooting," contains Compaq Parallel Database
t
Cluster troubleshooting information.
The Glossary contains definitions of many terms used in this guide.
s
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
x About This Guide
Additional Resources
For additional information, refer to documentation related to specific hardware
and software components of your Compaq Parallel Database Cluster. These
include but are not limited to:
Documentation related to the ProLiant servers you are clustering (for
s
example, manuals, posters, Performance and Tuning guides)
Compaq Fibre Channel Storage documentation
s
Compaq Fibre Channel Storage System User Guide
t
Compaq Fibre Channel Host Controller Installation Guide
t
Microsoft Windows NT Server documentation
s
Microsoft Windows NT Server Administrator's Guide
t
Microsoft Windows NT Server/Enterprise Edition Administrator's
t
Guide
Oracle documentation
s
Oracle8 Enterprise Edition Getting Started Release 8.0.5 for
t
Windows NT
Oracle Parallel Management User's Guide
t
TechNotes and other documents available from the Compaq website
s
(www.compaq.com)
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
xi
Text Conventions
This document uses the following conventions to distinguish elements of text:
Keys appear in boldface. A plus sign (+) between two
Keys
keys indicates that they should be pressed
simultaneously.
USER INPUT User input appears in a different typeface and in
uppercase.
File names appear in uppercase italics.
FILENAMES
Menu Options, These appear in initial capital letters.
Command Names,
Dialog Box Names
COMMANDS, These always appear in uppercase.
DIRECTORY NAMES,
and DRIVE NAMES
Type When you are instructed to type information, type the
information without pressing the Enter key.
Enter When you are instructed to enter information, type the
information and then press the Enter key.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
xii About This Guide
Symbols in Text
These symbols may be found in the text of this guide. They have the
following meanings.
WARNING: Indicates that failure to follow directions in the warning could
! result in bodily harm or loss of life.
CAUTION: Indicates that failure to follow directions could result in
damage to equipment or loss of information.
IMPORTANT: Presents clarifying information or specific instructions.
NOTE : Presents commentary, sidelights, or interesting points of information.
Symbols on Equipment
These icons may be located on equipment in areas where hazardous conditions
may exist.
Any surface or area of the equipment marked with these
symbols indicates the presence of electrical shock hazards.
Enclosed area contains no operator serviceable parts.
WARNING: To reduce risk of injury from electrical shock
hazards, do not open this enclosure.
Any RJ-45 receptacle marked with these symbols indicates a
Network Interface Connection.
WARNING: To reduce risk of electrical shock, fire, or damage
to the equipment, do not plug telephone or
telecommunications connectors into this receptacle.
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
xiii
Any surface or area of the equipment marked with these
symbols indicates the presence of a hot surface or hot
component. If this surface is contacted, the potential for injury
exists.
WARNING: To reduce the risk of injury from a hot component,
allow the surface to cool before touching.
Power Supplies or Systems marked with these symbols
indicate the equipment is supplied by multiple sources
of power.
WARNING: To reduce the risk of injury from electrical
shock, remove all power cords to completely disconnect
power from the system.
Rack Stability
WARNING: To reduce the risk of personal injury or damage to the
! equipment, be sure that:
s The leveling jacks are extended to the floor.
s The full weight of the rack rests on the leveling jacks.
s The stabilizing feet are attached to the rack if it is a single rack
installations.
s The racks are coupled together in multiple rack installations.
s A rack may become unstable if more than one component is extended
for any reason. Extend only one component at a time.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
xiv About This Guide
Getting Help
If you have a problem and have exhausted the information in this guide, you
can get further information and other help in the following locations.
Compaq Website
The Compaq Website has information on this product as well as the latest
drivers and Flash ROM images. You can access the Compaq Website by
logging on to the Internet at www.compaq.com.
Telephone Numbers
For the name of your nearest Compaq Authorized Reseller:
In the United States, call 1-800-345-1518
In Canada, call 1-800-263-5868
For Compaq technical support:
In the United States and Canada, call 1-800-OK COMPAQ (For contiuous
quality improvement, calls may be monitored or recorded.)
For Compaq technical support phone numbers outside the United States and
Canada, visit the Compaq Website at:
www.compaq.com
Writer: Linda Waterhouse Bob Schierman John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: A-FRNT.DOC Last Saved On: 11/3/98 11:15 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
PART I
Introduction
Writer: Linda Waterhouse/Bob Schierman/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: 386751-001
File Name: B-PT01.DOC Last Saved On: 10/21/98 6:23 PM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
1-1
Chapter 1
Clustering Overview
For many years, companies have depended on clustered computer systems to
fulfill two key requirements: to ensure employees can access and process
information that is critical to the ongoing operation of their business, and to
increase the performance and throughput of their computer systems at minimal
cost. These requirements are known as availability and scalability, respectively.
Historically, these requirements have been fulfilled with clustered systems built
on proprietary technology. Over the years, open systems have progressively and
aggressively moved proprietary technologies into industry-standard products.
Clustering is no exception. Its primary features, availability and scalability,
have been moving into client/server products for the last few years.
The absorption of clustering technologies into open systems products is
creating less-expensive, non-proprietary solutions that deliver levels of
functionality commonly found in traditional clusters. While some uses of the
proprietary solutions will always exist--such as those controlling stock
exchange trading floors and aerospace mission controls--many critical
applications can reach the desired levels of availability and scalability with
client/server-based clustering.
These new clustering solutions use industry-standard hardware and software,
thereby providing key clustering features at a lower price than proprietary
clustering systems. Before examining the features and benefits of the Compaq
Parallel Database Cluster, it is helpful to understand the concepts and
terminology of clustered systems.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: C-CH01.DOC Last Saved On: 10/23/98 10:54 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
1-2 Clustering Overview
Clusters Defined
A cluster is an integration of software and hardware technologies that enables a
set of loosely coupled servers and storage to present a single system image to
clients and to operate as a single system. As a cluster, the group of servers and
storage offers a level of availability and scalability far exceeding that obtained
if each cluster node operated as a standalone server. This translates into
increased performance and data availability.
Fibre Fibre
Fibre
Channel Array Channel Array
Channel Array
Node 2 Node 4
Node 3 Node 1
Fibre Channel Hub
Ethernet Switch
(Cluster Interconnect)
Client LAN
Figure 1-1. Diagram of a four-node cluster with three storage arrays
Availability
When computer systems experience outages, the amount of time the system is
unavailable is referred to as downtime. Downtime has several primary causes:
hardware faults, software faults, planned service, operator error, and
environmental factors. Minimizing the effects of the causes of downtime is a
primary goal of a cluster.
Simply defined, availability is the measure of how well a computer system can
deliver services to clients continuously.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: C-CH01.DOC Last Saved On: 10/23/98 10:54 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
1-3
Availability is a system-wide endeavor. The hardware, the operating system,
and the applications must be designed for availability. Clustering requires
stability in these components, then couples them in such a way that failure of
one item does not render the system unusable. By using redundant components
and mechanisms that detect and recover from faults, clusters can greatly
increase the availability of applications critical to business operations.
Scalability
Simply defined, scalability is a computer system characteristic that enables
improved performance or throughput when supplementary hardware resources
are added. Scalable systems increase the systems' power and throughput by
adding components to an existing system without the expense of adding a new
system.
In a stand-alone server configuration, scalability is usually obtained by adding
processors or more memory. In a cluster configuration, scalability is usually
obtained by adding cluster nodes.
Not only must the hardware benefit from additional components, but also
software must be constructed in such a way as to take advantage of the
additional processing power. The Compaq Parallel Database Cluster uses
Oracle8 Parallel Server, which is a parallel database that can distribute its
workload among the cluster nodes. As more nodes are brought into the cluster,
cluster-aware applications can use the parallel features of Oracle8 Parallel
Server to distribute workload among more servers, thereby obtaining greater
performance and/or throughput.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: C-CH01.DOC Last Saved On: 10/23/98 10:54 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
1-4 Clustering Overview
Compaq Parallel Database Cluster
As traditional clustering technology has moved into the open systems of
client/server computing, Compaq has provided innovative, customer-focused
solutions. The Compaq Parallel Database Cluster moves client/server
computing one step closer to offering comparable functionality found in
expensive, proprietary cluster solutions, at a fraction of the cost.
The Compaq Parallel Database Cluster combines the popular Microsoft
Windows NT Server operating system and the industry-leading Oracle8 Parallel
Server with award winning Compaq ProLiant Servers and storage systems.
Together, these hardware and software components provide improved
performance through a truly scalable parallel application and improved
availability using clustering software that rapidly recovers from detectable
faults.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: C-CH01.DOC Last Saved On: 10/23/98 10:54 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-1
Chapter 2
Architecture of the Compaq
Parallel Database Cluster
Overview of the Compaq Parallel
Database Cluster Components
The Compaq Parallel Database Cluster is an integration of a number of
different hardware and software products. This chapter discusses how each of
these products plays a role in bringing a complete clustering solution to your
computing environment.
The hardware products are:
Compaq ProLiant Servers
s
Compaq Fibre Channel Storage System
s
Compaq Interconnect Adapters
s
The software products are:
Microsoft Windows NT Server 4.0
s
Compaq Drivers and Utilities
s
Oracle8 Enterprise Edition, with the Oracle8 Parallel Server option
s
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-2 Architecture of the Compaq Parallel Database Cluster
Hardware Overview
Fibre Channel Array
Node 2 Node 4 Node 6
Node 5 Node 3 Node 1
Fibre Channel Hub
Ethernet Switch
(Cluster Interconnect)
Client LAN
Clients
Figure 2-1. Diagram of a six-node cluster
Compaq ProLiant Servers
A primary component of any cluster is a server. The initial release of the
Compaq Parallel Database Cluster supports from two to six cluster nodes, in
which each node is a server. All nodes must be identical in model. In addition,
all components common to all cluster nodes, such as memory, number of CPUs,
and the interconnect adapters, must be identical and identically configured.
ProLiant 1850R
s
ProLiant 6500
s
ProLiant 6500 with Intel Pentium II Xeon processor
s
NOTE: To obtain an up-to-date list of Compaq Parallel Database Cluster-
certified servers check the Compaq website at
www.compaq.com/highavailability/
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-3
High Availability Features of ProLiant Servers
In addition to the increased application and data availability enabled by
clustering, ProLiant servers include many reliability features that provide a
solid foundation for effective clustered server solutions. The Compaq Parallel
Database Cluster is based on ProLiant servers, most of which offer excellent
reliability through redundant power supplies, redundant cooling fans, and Error
Checking and Correcting (ECC) memory. The high-availability features of
ProLiant servers are a critical foundation of Compaq clustering products. The
following table lists all the high availability features found in many ProLiant
servers.
Table 2-1
High Availability Components of ProLiant Servers
Hot-Pluggable Hard Drives Redundant Power Supplies
DLT Tape Array ECC-Protected Processor-Memory Bus
Uninterruptible Power Supplies Redundant Processor Power Modules
ECC Memory Hot Plug PCI slots (in some servers)
Offline Backup Processor Redundant Cooling Fans
Clustering Shared Storage
The Compaq Parallel Database Cluster is based on a cluster architecture known
as Shared Storage Clustering, in which clustered servers share access to a
common set of hard drives. The Compaq Parallel Database Cluster requires all
clustered (shared) data to be stored in an external storage system.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-4 Architecture of the Compaq Parallel Database Cluster
Compaq has chosen its Fibre Channel Storage System option as the clustering
shared storage for this product. The Compaq Fibre Channel Storage System
was chosen for the Compaq Parallel Database Cluster because it brings
significant advantages to clustering, including:
Throughput of up to 100 MB/sec
s
Server-to-storage distances of up to 1000 meters
s
Increased connectivity and ease of use
s
Hot-pluggable drives
s
Hot-pluggable fan assembly
s
Hot-pluggable power supply (in redundant power supply configurations)
s
Manageability through Compaq Insight Manager
s
Throughout this guide, you will see references to the Compaq Fibre Channel
Storage System. When this term is used, it refers to all components that make
up the storage system itself. The three primary components of the Compaq
Fibre Channel Storage System are:
Fibre Channel Array
s
Fibre Channel Storage Hub (either 7-port or 12-port)
s
Fibre Channel Array Controller
s
A fourth component of Compaq Fibre Channel technology is the Fibre Channel
Host Controller, an adapter card that resides in a server. While the Fibre
Channel Host Controller is not part of the term "Fibre Channel Storage
System," it plays a key role in getting data to and from the server(s).
Each of these components is discussed below. For detailed information, refer to
the Compaq Fibre Channel Storage System User Guide and the Compaq Fibre
Channel Host Controller Installation Guide.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-5
Availability Features of Shared Storage
An important part of a high-availability system is the ability to improve data
availability, traditionally accomplished by implementing RAID technology.
Hardware RAID is an important part of the Compaq Fibre Channel Storage
System that is used for shared storage in the Compaq Parallel Database Cluster.
RAID is implemented on the Fibre Channel Array Controller, which is located
in the Fibre Channel Array. The Fibre Channel Array also allows redundant,
hot-pluggable power supplies and a hot-pluggable fan module.
The Fibre Channel Array Controller supports RAID levels 0, 1, 4, and 5. It
supports pre-failure notification on hard drives and provides a Fibre Array
Accelerator made with ECC memory. The Array Accelerator is backed with
onboard rechargeable batteries, ensuring that the data temporarily held (cached)
is safe even with equipment failure or power outage. For a complete list of
features and accompanying descriptions, refer to the Compaq Fibre Channel
Storage System User Guide.
By using standard short-wave Fibre Channel, the Fibre Channel Storage System
can be placed up to 500 meters from the Fibre Channel Storage Hub, and the
Fibre Channel Storage Hub can be placed up to 500 meters from the cluster
nodes (see Figure 2-2).
Fibre
500 m
Channel Array
500 m 500 m
Node 2 Node 4
Node 3 Node 1
500 m 500 m
Fibre Channel Hub
Ethernet Switch
(Cluster Interconnect)
Figure 2-2. Maximum distances between cluster nodes and shared storage
components
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-6 Architecture of the Compaq Parallel Database Cluster
Compaq Fibre Channel Storage System
Compaq Fibre Channel Array
The Fibre Channel Array is the storage cabinet that primarily contains the disk
drives, power supply, and Fibre Channel Array Controller. The Fibre Channel
Array can hold twelve (12) 1-inch or eight (8) 1.6-inch Wide-Ultra SCSI drives.
The Compaq Fibre Channel Array comes in either a rack-mountable or a tower
model.
Each Compaq Parallel Database Cluster must have at least one Fibre Channel
Array set up as external shared storage. The maximum number of Fibre
Channel Arrays available in a Compaq Parallel Database Cluster depends on
which Fibre Channel Hub is used and how many nodes are in the cluster. For
example, with a 12-port hub in a four-node cluster configuration, a maximum
of eight storage arrays could be used. With the 1-inch high, 9-GB drives, this
means a maximum storage capacity is approximately 864 GB (12 drives x 9 GB
x 8 arrays). With the 1.6-inch high, 18-GB drives, this means a maximum
storage capacity is approximately 1152 GB (8 drives x 18 GB 28 arrays).
Fibre
Fibre Fibre
Channel Array
Channel Array Channel Array
Node 2
Node 1
Fibre Channel Fibre Channel
Host Controller Host Controller
ProLiant ProLiant
Server Server
Fibre Channel Hub
Ethernet Switch
(Cluster Interconnect)
Client Lan
Figure 2-3. Fibre Channel Arrays provide storage in a two-node Compaq
Parallel Database Cluster
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-7
Compaq Fibre Channel Storage Hub
A Fibre Channel Storage Hub is required to create a data path from one or more
Fibre Channel Arrays to one or more Compaq servers. The Compaq Parallel
Database Cluster requires at least one Fibre Channel Storage Hub.
The Fibre Channel Storage Hub comes in two sizes: 7-port and 12-port. Both
are supported by the Compaq Parallel Database Cluster.
Each node in the cluster uses one port on the hub, as does each Fibre Channel
Array. For example, in a three-node cluster using two Fibre Channel Arrays,
three ports on the Fibre Channel Storage Hub are used by the cluster nodes and
two are used by the storage arrays. The remaining unused ports can be used for
future expansion of cluster nodes or storage arrays.
Fibre
Fibre Fibre
Channel Array
Channel Array Channel Array
Node 2
Node 1
Fibre Channel Fibre Channel
Host Controller Host Controller
ProLiant ProLiant
Server Server
Fibre Channel Hub
Ethernet Switch
(Cluster Interconnect)
Client Lan
Figure 2-4. Fibre Channel hub connects storage to servers in a two-node
Compaq Parallel Database Cluster
IMPORTANT: Currently, the Fibre Channel Storage Hub cannot be
"cascaded" (you cannot connect the port of one Fibre Channel Storage Hub
to the port of another Fibre Channel Storage Hub).
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-8 Architecture of the Compaq Parallel Database Cluster
Fibre Channel Array Controller
A single Fibre Channel Array Controller, which resides in the Fibre Channel
Array, is included with your Fibre Channel Storage System. This controller
provides RAID and caching capabilities to the Fibre Channel Storage System.
The cable connection from the Fibre Channel Storage Hub is made to this
controller.
Fibre Channel Host Controller
The controller adapters are the interface between the server and the Fibre
Channel Storage System. Each node requires a PCI Fibre Channel Host
Controller to be able to interface with the shared Fibre Channel Storage Hub. A
cable runs from each controller to a port in the Fibre Channel Storage Hub.
Fibre
Fibre Fibre
Channel Array
Channel Array Channel Array
Node 2
Node 1
Fibre Channel Fibre Channel
Host Controller Host Controller
ProLiant ProLiant
Server Server
Fibre Channel Hub
Ethernet Switch
(Cluster Interconnect)
Client Lan
Figure 2-5. Fibre Channel Host Controllers connect the servers to the Fibre
Channel Hub in a two-node Compaq Parallel Database Cluster
NOTE: While the Compaq Parallel Database Cluster supports multiple Fibre
Channel Arrays as part of the shared storage system, the initial release of the
product supports only a single fibre channel loop. Physically, this means that in
each cluster node only one Fibre Channel Host Controller can be connected to
shared storage.
For more information about this product, refer to the Compaq Fibre Channel
Host Controller Installation Guide.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-9
Fibre Channel Data Paths
The Fibre Channel Storage System provides shared storage for the Compaq
Parallel Database Cluster. As stated previously, the storage system consists of a
host controller in each server, a fibre channel hub, a storage array controller,
and a storage array containing SCSI disks.
The Fibre Channel Storage System has two distinct data paths, separated by the
Fibre Channel Storage Hub:
The first data path runs from the Fibre Channel Host Controller in the
s
servers to the Fibre Channel Storage Hub.
The second data path runs from the Fibre Channel Storage Hub to the
s
Fibre Channel Array.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-10 Architecture of the Compaq Parallel Database Cluster
Host Controller-to-Fibre Channel Storage Hub
Data Path
Multiple connections occur in this data path. Each server has a single
connection to the storage hub.
Client LAN
Fibre Channel Fibre Channel
Host Controllers Host Controllers
Ethernet Switch
(Cluster Interconnect)
ProLiant
ProLiant
Servers
Servers
Fibre Channel Hub
Fibre Channel Array
Figure 2-6. Fibre Host Controller to Fibre Channel Storage Hub data path
If one of these connections experience a fault, the connections from the other
servers ensures continued access to the database. The failed connection results
in the eviction of the cluster node that contains the host controller. All network
clients accessing the database through that node must reconnect through
another cluster node. The effect of this failure relatively minor. It affects only
those users who are connected to the database through the affected node. The
duration of downtime is approximately the time to detect the failure plus the
time for the network clients to reconnect to the database through another node.
Note that Compaq Insight Manager monitors the health of the Fibre Channel
Storage System. If any part of the Fibre Channel data path disrupts a server's
access to the storage array, the Fibre Array Controller status changes to
"Failed" and the condition is red. The red condition bubbles up to higher level
Insight Manager screens, and eventually to the device list.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-11
Fibre Channel Storage Hub-to-Fibre Channel
Array Data Path
Only one connection exists in the Fibre Channel Storage Hub-to-Fibre Channel
Array Data Path. There is one connection between the storage hub and the
array.
Client LAN
Fibre Channel
Fibre Channel
Host Controllers
Host Controllers
Ethernet Switch
(Cluster Interconnect)
ProLiant ProLiant
Servers Servers
Fibre Channel Hub
Fibre Channel Array
Figure 2-7. Fibre Channel Storage Hub to Fibre Channel Array data path
If this one connection experiences a fault, the database cannot be accessed from
any nodes. Because the nodes do not have access to shared storage, users
cannot reach the database. The data however, is unharmed, and remains safely
stored on the physical disks inside the Fibre Channel Array.
As with the Server-to-Fibre Channel Storage Hub data path, Compaq Insight
Manager detects this fault, changes the Fibre Channel Array status to "Failed,"
and changes its condition to red.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-12 Architecture of the Compaq Parallel Database Cluster
SCSI Disks
The Fibre Channel Array uses standard Wide-Ultra SCSI disks, ensuring
investment protection for existing SCSI users. The Fibre Channel Array can
hold as many as twelve (12) 1-inch or eight (8) 1.6-inch Wide-Ultra SCSI
drives. Due to the variety of drive sizes and storage capacities, the choice of
SCSI drives to use in the external Fibre Channel Array is left to the system
administrator. In a Shared Disk Clustering scheme, the SCSI drives must be
100% compatible with the clustering software. Compaq has worked diligently
to ensure Compaq SCSI drives meet the stringent needs of clustering. It is
recommended you use only Compaq SCSI drives.
Cluster Interconnect
The cluster interconnect is a data path over which nodes of a cluster
communicate. This type of communication is termed intra-cluster
communication. At a minimum, the interconnect consists of two network
adapters (one in each server), a network switch, and cables joining each adapter
to the switch. The cluster nodes use the interconnect data path to:
Communicate individual resource and overall cluster status
s
Send and receive heartbeat signals
s
Coordinate database locks through the Oracle Integrated Distributed
s
Lock Manager
IMPORTANT: The cluster management software in the Compaq Parallel
Database Cluster requires the use of TCP/IP. When configuring the cluster
interconnect, be sure to enable TCP/IP.
NOTE: Several terms for cluster interconnect are used throughout the
industry. Others are: private LAN, private interconnect, system area network
(SAN), and private network. Throughout this document the term cluster
interconnect is be used.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-13
Interconnect Adapters
Each of the supported servers come equipped with Compaq Ethernet adapters
capable of 100 Mbit/sec transfer rates. Though some adapters may be capable
of 10 Mbit/sec and 100 Mbit/sec, it is required that the interconnect run at 100
Mbit/sec.
The ProLiant 1850R is equipped with a single port Netelligent 10/100 TX
embedded UTP controller. It is recommended this controller be used for the
client LAN and a second Compaq Ethernet controller be used for the
interconnect.
The ProLiant 6500 is equipped with a dual port Netelligent Dual 100 TX PCI
UTP Controller. It is recommended that one of the ports be used for the
interconnect and the other port for the client LAN.
The Compaq ProLiant 6500 with Intel Pentium II Xeon processor is equipped
with a NC3122 PCI Dual 10/100. It is recommended that one of the ports be
used for the interconnect and the other port for the client LAN.
The adapters must have passed Windows NT Server 4.0 HCT certification.
Connecting the Adapters
The Compaq Parallel Database Cluster requires the use of a 100 Mbit/sec
Ethernet switch for the interconnect path. The 100 Mbit/sec Ethernet switch
handles higher network loads, which is essential to the uninterrupted operation
of the Compaq Parallel Database Cluster. An Ethernet hub is not supported.
NOTE: Although a two-node configuration can conceivably use an Ethernet
crossover cable to directly connect the nodes' interconnect adapters, it is not
supported in the Compaq Parallel Database Cluster.
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-14 Architecture of the Compaq Parallel Database Cluster
The cluster interconnect requires:
One Ethernet adapter in each cluster node
s
One standard Ethernet cable for each cluster node
s
One 100 Mbit/sec Ethernet switch
s
IMPORTANT: The switch used for the cluster interconnect must be
dedicated to the interconnect. It cannot be connected to the client network
(LAN) or to servers that are not part of the Compaq Parallel Database Cluster.
Fibre
Channel Array
Fibre Channel Hub
Ethernet Adapter
Ethernet Adapter Ethernet Switch
(Interconnect)
(Interconnect) (Cluster Interconnect)
Ethernet
Node 1 Node 2
Client LAN
Figure 2-8. Ethernet Adapters connected to a dedicated Ethernet switch form
the interconnect in a two-node Compaq Parallel Database Cluster
For detailed information about installing the server interconnect see Chapter 4,
"Installation and Configuration."
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-15
Local Area Network
Every client/server application requires a local area network, or LAN, over
which client machines and servers communicate. In the case of a cluster, the
hardware components of the client LAN are no different than in a standalone
server configuration.
The software components used by network clients should have the ability to
detect node failures and automatically reconnect the client to another cluster
node. For example, Net8, Oracle Call Interface (OCI) and Transaction Process
Monitors can be used to address this issue.
For complete information on how to ensure client auto-reconnect in an Oracle8
Parallel Server environment, refer to the Oracle white paper, "Oracle Parallel
Server on Windows NT Clusters: Implementing Client Failover."
Compaq Parallel Database Cluster Model PDC/O1000 Administrator Guide
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
2-16 Architecture of the Compaq Parallel Database Cluster
Software Overview
The Compaq Parallel Database Cluster combines software from several leading
computer vendors. The integration of these components creates a stable cluster
management environment upon which the Oracle database server can operate.
The only supported operating system is Windows NT Server. The cluster
management software is a combination of Compaq Operating System
Dependent Modules (OSDs) and Oracle 8.0.5 Enterprise Edition with the
Parallel Server option.
Microsoft Software
Microsoft Windows NT Server 4.0
The Compaq Parallel Database Cluster supports only version 4.0 of Windows
NT Server.
Both Windows NT Server 4.0 Standard Edition and Windows NT Server 4.0
Enterprise Edition are supported. Windows NT Server 4.0 Service Pack 3 is
required.
The Compaq Parallel Database Cluster does not work in conjunction with
Microsoft Cluster Server. Do not install Cluster Server on any of the cluster
nodes.
Writer: Linda Waterhouse/Bob Schiermann/John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 Administrator
Guide Comments: p/n 386751-001
File Name: D-CH02.DOC Last Saved On: 10/23/98 11:20 AM
COMPAQ CONFIDENTIAL - NEED TO KNOW REQUIRED
| 386751-001
|
