High-Reliability Technology of Mission-Critical IA Server

284 FUJITSU Sci. Tech. J., 41,3,p.284-290(October 2005)

High-Reliability Technology ofMission-Critical IA Server PRIMEQUEST

V Ohsai Hamada(Manuscript received May 20, 2005)

The mission-critical IA server PRIMEQUEST has various Reliability, Availability, andServiceability (RAS) functions — from the Application Specific Integrated Circuit (ASIC)and unit levels up to the system level. These RAS functions provide the high reliabilityand high availability required for mission-critical operations on these servers. Thispaper gives an overview of PRIMEQUEST. It then discusses the scalability and thespeed-enhancement, high-reliability, and high-availability technologies of PRIMEQUEST,focusing on the high-reliability technology implemented in the system hierarchies.

1. IntroductionThe new PRIMEQUEST IA servers for

mission-critical tasks incorporate applicationspecific integrated circuits (ASICs) developed byFujitsu and Intel’s 64-bit Itanium 2 processors.

This paper gives an overview of PRIME-QUEST’s hardware. It then discusses PRIME-QUEST’s speed-enhancement, high-reliability,and high-availability technologies and its scalabil-ity and high-maintainability design.

This paper also outlines the following.1) The crossbar system, which interconnects the

main units of the system.2) The partitioning function used to partition

the system.3) The Flexible I/O, which enables flexible

combination of the CPU/memory and I/Oresources.

4) The System Mirror function, which keeps thesystem running when hardware failuresoccur.

5) The cableless design, which improves thereliability and maintainability.

6) The various redundancy and hot-swappingfunctions that improve system availability.

2. Hardware outlineThe PRIMEQUEST models are the PRIME-

QUEST 480, which can mount up to 32 CPUs, andthe PRIMEQUEST 440, which can mount up to16 CPUs. The following OSs are supported.1) Red Hat Enterprise Linux AS (v.4 for Itani-

um)2) Novell SUSE LINUX Enterprise Server 9 for

Itanium Processor Family3) Windows Server 2003, Enterprise Edition for

Itanium-Based Systems4) Windows Server 2003, Datacenter Edition for

Itanium-Based SystemsFigure 1 shows the physical layout of

PRIMEQUEST 440 and 480, and Figure 2 showstheir block diagram.

PRIMEQUEST 440 and 480 are structurallyidentical. The only difference between these twomodels is the maximum number of system boards(SBs), which contain the CPUs and memories, andthe number of I/O units (IOUs), which contain thehard disk drives (HDDs) and PCI slots. PRIME-QUEST 440 can have up to four SBs and fourIOUs, and PRIMEQUEST 480 can have up toeight SBs and eight IOUs.

285FUJITSU Sci. Tech. J., 41,3,(October 2005)

O. Hamada: High-Reliability Technology of Mission-Critical IA Server PRIMEQUEST

As shown in Figure 2, PRIMEQUEST isconfigured using a crossbar that interconnects theSBs and IOUs. The crossbar is constructed on amidplane (described later) and printed circuitboard assemblies containing the ASICs usedfor path control and transfer of address/datainformation.

The midplane is mounted in the center of thecabinet and can accommodate units on both sides.The various high-speed interfaces, including thecrossbar, are wired on the midplane to achieve a

cableless design.In addition to the SBs and IOUs, a server man-

agement unit called the Management Board (MMB)that handles system management within the cab-inet is mounted on the midplane. Firmware isinstalled on the MMB for system-managementtasks such as controlling the power supply of eachunit in the cabinet and monitoring the tempera-ture and voltage within the cabinet and otherenvironmental factors. The system administra-tor manages the system using a Web server thatprovides operation views. The Web server isoperated using a Web browser running on ageneral-purpose PC that is connected via a LAN.

For details about system management, seethe paper, “Operation Management of Mission-Critical IA Server PRIMEQUEST for TCOReduction,” presented elsewhere in this specialedition.

Moreover, an optional Gigabit switchboard(GSWB) can be installed internally. The LANinterface from the Gigabit Ethernet (GbE) portsinstalled on each IOU is connected to the GSWBvia the midplane and output externally from thecabinet.

Table 1 lists PRIMEQUEST’s specifications.

(b) Rear(a) Front

Power supplyunit

System board× 8

I/O unit× 4

Crossbar

Servermanagementunit

Gigabitswitchboard

I/O unit × 4

1100 mm

1800

mm

738 mm

GSWB#0 GSWB#1

IOU#1

IOU#2

IOU#3

IOU#4

IOU#5

IOU#6

IOU#7

PCIcard

IOU#0

HDD×4

MMB#0

SB: System board IOU: I/O unitMMB: Server management unit MEM: MemoryNB: North Bridge (ASIC) GSWB: Gigabit switchboard

Crossbar

MMB#1SB#0 SB

#1SB#2

SB#3

SB#4

SB#5

SB#6

SB#7

CPU

NBMEM

Figure 1Physical layout of PRIMEQUEST.

Figure 2Block diagram of PRIMEQUEST.

286 FUJITSU Sci. Tech. J., 41,3,(October 2005)


3. Dual Synchronous SystemArchitectureA new architecture called the Dual Synchro-

nous System Architecture has been developed forPRIMEQUEST. “Dual Synchronous SystemArchitecture” is the generic name of a technologythat mirrors the system by configuring a high-speed synchronous parallel bus and enablesconstruction of a high-performance, large-scalemultiprocessor system. (The System Mirrorfunction duplicates the hardware components ofthe server and enables synchronous operations.)

The Dual Synchronous System Architectureenables mirror operations to be applied to aPRIMEQUEST-class, large-scale multiprocessorsystem for the first time in the world.

4. Speed-enhancementtechnologyAs described above, PRIMEQUEST uses a

midplane structure in which the units are dense-ly packed on both sides of the midplane and thehigh-speed synchronous parallel bus that config-ures the crossbar is wired on the midplane.

4.1 High-speed synchronous parallel busPRIMEQUEST’s main bus is a specially

developed high-speed synchronous parallel bus.This bus operates at a very fast clock speed of800 MHz or 1.33 GHz, which makes it the world’sfastest synchronous parallel bus. In addition, thebus employs a single-ended transmission meth-od. Compared to the differential transmissionmethod, this method requires only half the sig-naling wires to transfer the same number of databits. As a result, when printed circuit boards ofthe same cost are used, a bus having double the

Model

CPU

CPU count

Memory (*1)

System boards

I/O units

Crossbar

Disks

PCI slots

GbE interface

SCSI interface

Partitions

Redundant configuration

Basic cabinet dimensions (mm)

Weight (kg)

OS (support schedule, etc.)

PRIMEQUEST 480

Itanium 2 processor (1.60 GHz/9 MB L3 cache, 1.50 GHz/4 MB L3 cache)

Up to 32 CPUs (4 CPU/SB × 8)

Up to 512 GB (64 GB/SB × 8)

Up to 8

Up to 8

Up to 102.4 GB/s

Up to 147 GB × 32 units

Up to 128 slots

Up to 32 ports

Up to 16 ports

Up to 8

Disks, power supplies, fans, server management units (MMBs), gigabit switchboard (*2), crossbar (*2), and memories (*2)

738 W × 1100 D × 1800 H

720

Red Hat Enterprise Linux AS（v. 4 for Itanium） (End of June 2005)Novell SUSE LINUX Enterprise Server 9 for Itanium Processor Family (End of September 2005, for overseas markets)Windows Server 2003, Enterprise Edition for Itanium-Based Systems (End of September 2005)Windows Server 2003, Datacenter Edition for Itanium-Based Systems (End of September 2005)

Up to 16 CPUs (4 CPU/SB × 4)

Up to 256 GB (64 GB/SB × 4)

Up to 4

Up to 4

Up to 51.2 GB/s

Up to 147 GB × 16 units

Up to 64 slots

Up to 16 ports

Up to 8 ports

Up to 4

600

*1: Standard. In mirror mode, up to 256 GB × duplication for PRIMEQUEST 480, up to 128 GB × duplication for PRIMEQUEST 440*2: Option

PRIMEQUEST 440

Table 1PRIMEQUEST 480/440 specifications.



bit width can be installed. The high transmis-sion speed and wide bit-width of this bus give thePRIMEQUEST models the highest bandwidth inthe world. Fujitsu’s state-of-the-art CS101 semi-conductor technology is used for the ultrahigh-speed I/O interface circuits that make thisbus possible. For details about the ASICs devel-oped for PRIMEQUEST, see the paper, “Fujitsu’sChipset Development for High-Performance,High-Reliability Mission-Critical IA ServersPRIMEQUEST,” presented elsewhere in thisspecial edition.

4.2 CrossbarA point-to-point crossbar is used to intercon-

nect the SBs and IOUs. Dedicated ports for eachunit are provided to connect up to eight SBs andeight IOUs. For the SB ports, a bandwidth from12.8 to 21.3 GB/s (= 16 × 0.8 GB/s to 16 × 1.33 GB/s)is achieved. Because there are eight ports, thetotal bandwidth is from 102.4 to 170 GB/s (= 8 ×12.8 GB/s to 8 × 21.3 GB/s).

5. Flexible operation and highscalabilityTo increase the flexibility of system opera-

tion, PRIMEQUEST provides a partitioningfunction and a Flexible I/O. In addition, PRIME-QUEST supports high scalability to handleincreases in the workload after the server is in-stalled. This section describes these functions andhigh scalability.

5.1 Partitioning functionPRIMEQUEST supports a partitioning func-

tion that can divide the system installed in onecabinet into multiple independent systems. Eachof these independent systems is called a partition.The minimum components of a partition are oneSB and one IOU. The PRIMEQUEST 480 modelcan have up to eight partitions, and the PRIME-QUEST 440 model can have up to four partitions.

The partitioning function enables varioustypes of uses, for example, mixed use of different

OSs or mixed use of partitions for executingproduction tasks and partitions for executingdevelopment tasks. As such, the partitioningfunction enables the construction of flexiblesystems. In addition, because the partitionconfiguration can be changed on a daily basis, onlythe minimum number of SBs and IOUs need beprepared. That is, only the required resources areneeded. Moreover, by using the Flexible I/O, whichis the main feature of PRIMEQUEST, the SB andIOU resources can be assigned to the partitionsin any combination.

5.2 Flexible I/OPRIMEQUEST provides a function called the

Flexible I/O that enables partitions to beconstructed from arbitrary SBs and IOUs. Fordetails about the Flexible I/O, see the paper, “Flex-ible I/O Improves Flexibility and Reliability ofMission-Critical IA Server PRIMEQUEST,” pre-sented elsewhere in this special edition.

5.3 ScalabilityTo handle the ever-growing mission-critical

operations, up to 16 CPUs and 256 GB of memorycan be mounted in the 440 model, and up to 32CPUs and 512 GB of memory can be mounted inthe 480 model. In addition, up to 16 or 32 PCIslots can be mounted in the cabinets of the 440and 480 models, respectively. When more PCIslots are needed, external PCI boxes can beconnected so up to 64 or 128 PCI slots can be add-ed for the 440 and 480 models, respectively.Moreover, four Gigabit Ethernet ports areinstalled for each IOU, so up to 16 or 32 GigabitEthernet ports can be installed for the 440 and480 models, respectively.

In addition, scenarios of scale-out and scale-up can be realized by using the partitioningfunction described above.

Figure 3 shows an example of scale-out andscale-up.

When applied to the server of the AP layer(application layer), any increase in the workload



of the AP layer can be handled by increasing(scaling-out) the number of small partitions allo-cated to the server of the AP layer. When appliedto the server of the DB layer (database layer),increases in workload can be handled by addingSBs and IOUs to increase (scale-up) the partitionsizes.

6. High-reliability and high-availability technologiesHigh-reliability and high-availability are

essential for mission-critical systems that mustoperate 24 hours a day, 365 days a year. ForPRIMEQUEST, Fujitsu has used the various tech-nologies it developed for its mainframes and UNIXservers and has developed zero-failure systemsthat guarantee data integrity.

6.1 Data integrityTo ensure that no data is corrupted during

processing, PRIMEQUEST adds an Error Correct-ing Code (ECC) or parity protection to the RAMdata and data bus. Some examples are givenbelow. An ECC or parity is added to the embed-ded RAM data of each ASIC. When the RAM datais read, the ECC or parity is checked to ensure itsintegrity.

When transmitting data within an ASIC orbetween ASICs, ECC or parity is also added toensure data integrity during transmission. Inparticular, for data transmission lines that include

the crossbar, an ECC or parity check is performedfor each stage of the line to ensure data integrity.If a data error occurs, extremely precise erroranalysis is performed so the location of the errorcan be identified.

6.2 System Mirror functionPRIMEQUEST offers a System Mirror func-

tion as an option. This function duplicates mostof the units and implements synchronous opera-tion. If a failure occurs on one side of theduplicated units, the System Mirror function en-ables the other side to continue operation. Fordetails about the System Mirror function, see thepaper, “Highly Reliable System Mirror Functionof Mission-Critical IA Server PRIMEQUEST,”presented elsewhere in this special edition.

6.3 Redundancy functionsPRIMEQUEST supports redundant config-

uration of most of its components, including thepower supplies, fans, hard disk drives, and MMBs.1) Power supply system

The standard power-supply configuration isN + 1 redundancy. Therefore, if one of the powersupply units fails, power is still supplied so oper-ation can continue. In addition, the AC powersupply source can be duplicated by installing theoptional dual AC power feed function.2) Cooling system

The cabinet is divided internally into sever-al cooling zones with N + 1 redundancy of thecooling fans for each zone. If a cooling fan fails,the remaining cooling fan will continue to oper-ate and the failed fan can be hot swapped withoutstopping the system.3) Server management unit

As standard, two MMBs are installed forredundancy. One of the MMBs is active, whilethe other is on hot standby. The MMBs check eachother to ensure they are operating normally. Whenthe hot standby MMB detects an error in theactive MMB, it automatically takes over as theactive MMB. The failed MMB can then be hot

CPU

I/O

PRIMEQUEST

16 CPU 32 CPU

CPUs can be added tohandle any increase in theamount of data processing.

• Large-scale DB server• Large-scale ERP server

CPU

I/O

PRIMEQUEST

Workload optimization isenabled.

• Server consolidation

Scale up Scale out

AP server DB server

Figure 3Scale-out and scale-up of PRIMEQUEST.



swapped without stopping execution of the taskson each partition.4) Clock circuits

Two system clock oscillators are installed inthe cabinet. When the system is powered, initialdiagnosis checks that the selected system clockoscillator is operating normally. If an error is de-tected in this oscillator, the system switches tothe other oscillator to start operation.

6.4 Hot-swapping functionPRIMEQUEST supports a hot plug function

for the PCI cards, hard disk drives, and coolingfans.

In addition, the SBs, IOUs, and GSWBs canbe hot swapped under specific conditions. There-fore, because the main units can be hot swapped,faulty units can be replaced without stopping thewhole system.

6.5 Cableless design for highmaintainabilityA complete plug-in method is employed by

which all high-speed interfaces are installed onthe midplane in the center of the cabinet and allunits are plugged in on both sides of the midplane.The following interfaces are installed on the mid-plane without the use of cables.1) Crossbars between the SBs and IOUs2) A Gigabit Ethernet interface between the

IOUs and GSWBs3) A management LAN (100 megabit Ethernet

interface) between the IOUs and MMBs4) Video, keyboard, and mouse interfaces be-

tween the IOUs and KVM (Keyboard, Video,and Mouse) interface units

5) Various management interfaces between theMMBs and units

6) Power linesBy making all wiring cableless, maintenance

work such as installing the servers and addingoptional units can be done more easily and withhigher reliability. Human errors such as incor-rect cable connections can also be avoided. In

addition, by reducing the cabling costs incurredwhen adding and changing the cable wiring andachieving more reliable signal connections, thereliability and maintainability are improved.

The following gives an outline of the majorinterfaces installed on the midplane.1) Built-in GSWB

Using a GSWB provides several benefits. Itsignificantly reduces the amount of work neededto connect the cables, reduces the operation costs,improves reliability because of the reducednumber of cable connections, and improves main-tainability and operability.

Changes to the partition configurationwithin the cabinet, including additions of newpartitions, can be done simply by changing theVirtual LAN (VLAN) definitions of the GSWB.There is no need to physically change or add anyLAN cable connections, which reduces the opera-tion costs.

The GSWB significantly reduces the numberof required external cables, which greatly im-proves reliability.2) Management LAN and built-in LAN switch

In addition to the wiring of the gigabit LAN,which is mainly used for tasks, the wiring of themanagement LAN used for operation manage-ment is also arranged on the midplane. Moreover,the LAN switch of the management LAN is built-in. Therefore, the LAN can immediately be usedas a management LAN after the server isinstalled. This not only reduces the number ofsetup processes at the installation site, but canalso reduce the number of connections andsettings, which helps to reduce the Total Costof Ownership (TCO). For example, when a newSB or partition is added, only the IP address ofthe Ethernet interface within the partitionconnected to the management LAN need bechanged. As a result, the amount of work requiredwhen expanding the system can be kept to aminimum.3) KVM interface unit

The video interface and three USB interfac-



Ohsai Hamada received the B.S. andM.S. degrees in Electronics Engineer-ing from Hokkaido University, Sapporo,Japan in 1981 and 1983, respectively.He joined Fujitsu Ltd., Kawasaki, Japanin 1983, where he has been developingserver systems, including mainframesand IA servers. He is a member of theInstitute of Electronics, Information andCommunication Engineers (IEICE) ofJapan and the IEEE.

es installed on each IOU are connected to the KVMinterface unit via the midplane. MMB firmwarecontrols selection of the video and USB interfacesof one of the IOU’s interfaces for connection to thedisplay unit, keyboard, and mouse via the exter-nal connector of the KVM interface unit.

One of the USB interfaces is connected to theDVD-ROM drive installed in the cabinet. Thisinterface is also controlled by the MMB firmware.The interface can be switched to a USB interfaceof one of the IOUs and shared from theindividual IOUs.

7. ConclusionThis paper discussed the speed-enhancement

technology, scalability, high-reliability technolo-gy, and high-availability technology ofPRIMEQUEST.

To handle mission-critical operations in thecurrent ubiquitous computing era, in which near-ly every business operation involves the use of acomputer network, we will continue our efforts toexpand the high-reliability technology and speed-enhancement technology of the PRIMEQUESTseries to develop and deliver servers that meetour customers’ requirements.

This research has been partially funded bythe Ministry of Economy, Trade and Industry(METI) and the New Energy and IndustrialTechnology Development Organization (NEDO).

Documents

High-Reliability Technology of Mission-Critical IA Server