Qualification, Manufacturing, and Reliability Testing Methodologies

Qualification, Manufacturing, and Reliability Testing Methodologies for Deploying High-Reliability Solar Modules

by D. DeGraaff, S. Caldwell, R. Lacerda, G. Bunea, A. Terao and D. Rose2010

2

EXECUTIVE SUMMARY

E X E C U T I V E S U M M A R Y

With over a decade of experience fielding and maintaining solar fields – built with a wide variety of PV

modules from more than a dozen different manufacturers – SunPower has broad experience with reliability.

Combining this field data with the manufacturing rigor inherited from roots in the semiconductor industry has

lead to the development of a robust methodology for deploying high-reliability solar modules. This paper

shares SunPower’s methodology, which spans qualification testing, supplier quality, manufacturing quality

and ongoing reliability testing. For each of these areas, the test development or process development

processes is articulated, along with the specific testing protocols used by SunPower. Also mentioned are

problems that can occur in each of these areas, with examples taken from both SunPower modules and

non-SunPower modules. The most important result of this research is that it requires considerable care and

analysis to design and manufacture high-reliability PV modules, with an emphasis on testing to determine

the physics of the failure modes. This requirement goes well beyond what is required by the UL or IEC

certification standards.

3S E C T I O N 1 INTRODUCT ION

Figure 1: End-to-end process

By publishing these methodologies, along with many examples of lessons learned, it is hoped that industry practitioners may find aspects of this process which they want to integrate into their own processes and testing methods; SunPower believes a high prevalence of robust solar modules will benefit the entire solar industry.

INTRODUCTION

SunPower has focused for many years on the challenge of delivering high-reliability products, and has developed

the robust process illustrated in Figure 1. This paper discusses each of these boxes in turn, although it should be

noted that the process is not entirely linear, since new data causes revisions to the FMEA and thus revised test plans.

S E C T I O N 1

4S E C T I O N 2 FA I LURE MODES AND EFFECTS ANALYS IS

Building the FMEAThe “Failure Modes and Effects Analysis” (FMEA) is the foundation of the Design for Reliability process, since it provides a rank-ordered list of all the known failure modes and their causal stresses. It underlies the qualification tests which will determine if the design of a new product, or a product modification, meets the functional and reliability requirements.

In order to build an FMEA, one must start with a design concept, and a set of clearly articulated functional and reliability specifications. The latter lists all the functions that the product must deliver. For example, a few lines for a PV module backsheet are:

• Must provide electrical isolation between the back of the module and solar cells at 1000V

• Must reflect more than 80% of irradiance incident on the cell-side surface

• Must maintain color for 30 years in the field for each climate

• Must shrink less than 0.5% during the lamination process along any dimension

Reliability specifications are a subset of the functional specifications, with detailed reliability requirements. Continuing the backsheet example:

• Must maintain a continuous electrical barrier (no cracks) with 99% likelihood after 25 years in each climate with cells that are at 1000V.

• Must not bubble or delaminate with 99% likelihood after 25 years in each climate with cells that are at or below 110C during times of peak irradiance.

Given the list of specifications, and a design, the FMEA can be developed. A group of subject matter experts brainstorm to create a prioritized list of all the possible ways a product might fail. Note that a specific design concept is necessary for this step, so the requirements can be assessed against a concrete product design – for example, a backsheet with well specified layers and adhesives so interlayer adhesion can be examined. This brainstorm is captured on an FMEA chart; Table I shows a snippet example for a new backsheet (which lists 35 different known failure modes). Note the scoring used to create the prioritized list: the Severity is the economic and safety impact when the failure does occur, the Frequency is chance that the failure actually occurs, and the Detectability is the chance that the failure mode can be averted by detection and prevention before it reaches the customer (low score means high detectability). The product of those values constitutes the Risk Priority Number which is used to prioritize what must be tested to qualify the product.

S E C T I O N 2

FAILURE MODES AND EFFECTS ANALYSIS


Table I: Backsheet FMEA snippet

TC – thermal cycling (5 per day between 90C and -40C), HF – humidity-freeze cycles (1 cycle per day of 85C with 85% relative humidity, with an hour at -40C), DH – hours of damp heat (85C with 85% relative humidity)

Functional Req’tFailure Mode

Effects of failure

Causes / Mechanisms of failure

Current design: Prevent / Detect

Electrical isolation

Jbox detaches from backsheet

Loss in power, increasedleakage

current, safety issue

10RTV loses adhesion to back-

sheet due to surface properties2

Test J-box and backsheet interlayer adhesion under hot spot condi-tion on cell on top of JBox with

and without weights; test Jbox and backsheet adhesion in high temp

and humidity environments

2 40

Dielectric degradation

Leakagecurrent, safety

issue10

Hot-spot creates burns on backsheet

(insufficient temp resistance)2

Heat soak at varioustemperatures to determine

activation energy40

Moisture degrades insulating layer (hydrolytic embrittlement)

3Damp heat testing; check for

cracks and increased wetleakage current

30

Growth of pinholes or increased conduction through

pin holes1

Wet leakage current test and opti-cal inspection of backsheet surface as laminated, and post accelerated

tests (e.g. DH, HF, TC)

10

Acetic acid from EVA attacks backsheet

1Test for various acid resistance of

backsheet layers10

Etc.

Dielectric separation (backsheet

layersseparate)

Leakag current, safety

issue10

Propagation of slit inside jbox where ribbons exit

2 DH2000, TC200 1 20

Internal physical puncture 4Wet leakage current test post vari-

ous accelerated tests1 40

UV attacks individual layers or adhesives

1Wet leakage current test post UV exposure (equivalent dose for 25

years)1 10

Bubbles, loss of interlayer adhesion

5High temperature test with and

without hanging weight1 50

Reflectivity Color changes Power loss 3UV attacks individual layers 3

UV exposure, Combwined UV/DH exposure, outdoor exposure

2 18

Etc.

S E C T I O N 2

Seve

rity

Occ

urre

nce

Dete

ctab

ility

RPN


To enable a thorough Qualification Test Plan, the FMEA must have thorough coverage over the possible field failure mechanisms, and the combinations of stresses that cause them. The three primary approaches used by the subject matter experts are discussed in the following three sections: Field Experience, Highly Accelerated Life Testing, and Theoretical Understanding.

Field Experience Field experience is a critical element for identifying real design failure modes. Solar modules have been in the field for many years, so most of the failure mechanisms have been identified [1]. The pictures in Figures 2 and 3 show examples from the field which informed the FMEA: j-box-to-backsheet adhesion failure, and backsheet interlayer adhesion failure. Note that failure analysis must be done on each type of field failure to determine what combination of stresses caused the failure, so a test can be devised to ensure the new product does not have the same weaknesses.

Highly Accelerated Life Testing (HALT) The second primary input for defining the FMEA is Highly Accelerated Life Testing. This is a method of discovering product weaknesses by applying stress combinations which are relevant to field deployment, but that are considerably beyond the normal operating environment. It is not a pass/fail test, but rather a test designed to activate the same failure modes one would encounter after many years in the field, in just a few days or weeks. It is difficult or impossible to obtain information about how things fail in the absence of actual failures, and HALT is a remarkably effective way to reveal unanticipated failure modes, and thus to identify and then toughen up any weak points in the product design.

Figure 2: Front-contact-cell module J-box separation

Figure 3: Front-contact-cell module backsheet interlayer adhesion failure

Because SunPower builds more projects than it can supply with its own back-contact cells, modules from more than a dozen different manufacturers are part of SunPower projects. In addition to long-term testing, HALT plays a significant role in qualifying these modules, so some results of that work are presented here to illustrate the efficacy of this methodology.

Table II summarizes the HALT protocol typically used to find design weaknesses in modules. These tests involve intermediate data reads to have data on relative rates of failures, and also have some continuously monitored

S E C T I O N 2

1. Quintana MA, King DL, McMahon TJ, Osterwald CR., “Commonly observed degradation in field-aged photovoltaic modules,” Photovoltaic Specialists Conference, 2002. Conference Record of the 29th IEEE, 19–24 (May 2002) 1436–1439.

7

S E C T I O N 2

S E C T I O N 2 FA I LURE MODES AND EFFECTS ANALYS IS

parameters (e.g. leakage current in the damp heat with voltage bias test, and cell temperatures in the Solar Simulator test). Note that for non-SunPower modules, 5 kWh/m2 of outdoor exposure is done before the test commences, to accomplish light-induced degradation (LID). SunPower back-contact cells use N-type silicon wafers and so they do not exhibit LID.

Table II: Module HALT

The first test listed in Table II is the Solar Simulator, which is able to concurrently apply most of the stress factors seen by modules in the field, and with exaggerated intensity: 1) temperature, 2) humidity, 3) visible light, 4) UV light, and 5) various electrical load conditions. This test has proven very effective for exposing module design weaknesses, both in power degradation and in visually-identified defects, e.g.,

melting of certain laminate materials, breakdown of adhesives, corrosion, discoloration, out-gassing, extremely hot cells, local hot spots, etc.

Figure 4: Comparison of efficiency degradation for front-contact-cell modules from different manufacturers in the Solar Simulator. Four SunPower back-contact-cell modules are solid black lines (3 samples fall on the same -0.1% line).

The HALT methodology consists of pushing the design to the point where failures occur and weaknesses are exposed. Completely eliminating these weaknesses is not always necessary as it can lead to overdesign and excessive cost. However, going beyond what is strictly necessary often offers tremendous paybacks: the design is more robust against contamination and other process issues, and deviations from target processes are more easily detected before they result in field failure. A defect like front delamination in Figure 5 due to solder flux residue points to a manufacturing process control problem, or a material compatibility problem, and must be solved. Backside bubbling is

Test Primary failure modes

5 days Solar Simulator• short-circuited modules• 1000 W/m2 of solar spectrum• An additional 150 W/m2 of UV

spectrum• 55% relative humidity and 60C

environment

Delamination along the front interconnect ribbons Backsheet browning and

bubbling

Encapsulant browning

Maximum temperatures of reverse-bias cells

5-days Oven• 130C

Backsheet bubbling in certain areas (typically near

lay-up tapes or extremely hot cells)

600 hrs damp heat with bias• 85% relative humidity, 85C• 1000V bias between the cells and

the grounded frame

Series resistance increase due to electrolytic corrosion of metal lines or degrada-tion of Si-metal interface

1,000 alternating cycles of +/-2400Pa, then 4 temp. cycles -40 to 60C

Cell cracking

8

2. UL 1703: Standard for Flat-Plate Photovoltaic Modules and Panels, 3rd Edition, March 15 (2002) ISBN 0-7629-0760-6.

S E C T I O N 2 FA I LURE MODES AND EFFECTS ANALYS IS

S E C T I O N 2

sometimes less clear; bubbles that form under an extremely hot cell might be deemed acceptable, as the hottest cells in this test can exceed 170C, which far exceeds normal field conditions.

Figure 5: Front-contact-cell modules after Solar Simulator test: solder flux residue along the interconnect ribbon caused delamination bubbles (left side), and lay-up tape out-gassing cause backsheet bubbling (right side)

The maximum cell temperature achieved during the test is itself of significant interest, since it is predictive of how hot reverse-bias cells can get in the field. This temperature is determined by both cell and circuit characteristics. PV cells are divided into two types according to their reverse breakdown characteristics [2]. Type A cells have a breakdown voltage that is higher than the sum of the forward voltages of all the other cells in the same string protected by a bypass diode. They are said to be voltage limited. Type B cells are said to be current limited because they can absorb all the current from forward-biased cells without any bypass diode protecting them. This can be either because they have a low reverse breakdown voltage

or because they have a high leakage current due to reduced shunt resistance.

Figure 6: Electroluminescence images of a front-contact-cell module before testing (left side), and EL image after 600 hours of damp heat with +1000V bias (right side)

The Solar Simulator test can be run with and without diodes for modules from different manufacturers to explore worst-case reverse-bias conditions. Type A cells with a high reverse breakdown voltage and high shunt resistance require bypass diodes in suitable number per module to protect against over-heating. Cells with a high reverse breakdown voltage that are of Type B because of a low shunt resistance are at risk of local overheating (hot spots).

Cells of Type B that have a low reverse breakdown voltage are intrinsically protected against over-heating if their breakdown is controlled such that it happens uniformly across the whole cell. That is the case for


SunPower cells, so the temperature in reverse bias does not depend on the number or even the presence of bypass diodes.

The oven test in Table II is a faster and lower-cost test that stimulates similar delamination and bubbling defects as the Solar Simulator, although it uses only one of the five stress factors available in the Solar Simulator. This decouples the reverse bias characteristics of the cells from the temperature resistance of module components.

The third test on Table II, damp heat with 1000V between the frame and the cells, exaggerates an aspect of fielded modules that is frequently overlooked: there is usually a high voltage between the cells and the frame. Figure 6 shows electroluminescence images confirming that cells closer to the grounded frame are impacted more than cells near the center of the module.

The power degradation in Figure 7 is significantly accelerated over damp heat testing that is done without the electro-chemical acceleration from the voltage bias. For standard front-contact cells, the module leads are connected to +1000V relative to the grounded frame (negative grounding), because this is normal installation practice, and because these modules degrade even faster with a -1000V bias, as demonstrated by NREL testing [3]. The SunPower module leads are connected to -1000V since the installation directions require positive grounding, so some differences in degradation modes are expected.

Similar testing was done as part the NREL Test-to-Failure Protocol, for 2000 hours of damp heat with +/-600V bias [3]. Results from that study are plotted in Figure 8, and show very similar trends.

S E C T I O N 2

3. P. Hacke, K. Terwilliger, S. Glick, D. Trudell, N. Bosco, S. Johnston, S. Kurtz, “Test-to-Failure of Crystalline Silicon Modules,” 35th IEEE PVSC (2010) 0-66

Figure 7: Comparison of efficiency degradation for modules from different manufacturers in damp heat with bias. SunPower back-contact-cell modules are solid black lines.

Figure 8: NREL Test-to-Failure Protocol data for modules that have completed 2 rounds of testing (selected from top 20 manufacturers). SunPower back-contact-cell module is the solid black line, and had a -600V bias. The other modules were at +600V bias.


The left picture in Figure 9 shows the corrosion visible on the metal of front-contact cells that is responsible for the increase in series resistance and decrease in efficiency. Data from NREL also suggest that the degradation rate correlates to the leakage current. This is only true, however, when comparing results for modules with similar cells, materials, and construction. In particular, the SunPower back-contact cells appear significantly more resistant to electrolytic corrosion. This is because almost the entire back surface of the cell is plated with copper “finger” conductors (see Figure 12), and the thickness of that plated copper is 40 microns. This provides a large safety-factor against corrosion, with an order of magnitude more cross-sectional area of copper relative to the metal fingers on the front of a front-contact cell.

Figure 9: Following 600 hours of damp heat with voltage bias for front-contact-cell modules, corrosion can be seen on the front-contact metal (left side). The right-side picture shows the view looking down the edge of the aluminum frame that surrounds the laminate glass.

The right side of Figure 9 shows a white powder (consisting primarily of Al2O3), which commonly forms along the edge of the frame where it contacts the glass. This has also been seen in the field on modules with a high leakage current and corroding Aluminum frames.

The combination of cyclic loading and temperature cycles in Table II is very effective for investigating cell cracking [4], and whether the stress inherent in the interconnect-ribbon-to-silicon wafer solder bond is robust [5]. For front-contact cells, the copper interconnect ribbons must be soldered onto the silicon wafer at elevated temperatures, and because of the coefficient of thermal expansion mismatch between the copper and the silicon, thermal stresses are built into the interface. If not created with well controlled processes and materials, the solder bond can be too strong (causing “cratering” where small cracks form in the cell underneath the ribbon), or too weak (causing the ribbon to tear off the cell).

Figure 10 shows typical before and after images of front-contact polycrystalline-cell modules before and after the dynamic load HALT stress. The dark areas show 7 cells with cracks that caused areas of these cells to be isolated from the electrical circuit, with a consequent 2.1% drop in the measured power production.

Figure 11 shows the same test run on SunPower back-contact cells, which generated one cracked cell near the center of the module. The toughness of these cells is due primarily to the thick copper plated fingers on the back of the cell.

S E C T I O N 2

4. J.H. Wohlgemuth, et al. “Using Accelerated Tests and Field Data to Predict Module Reliability and Lifetime,” Proc. 23rd European PVSEC, Valencia, Spain (2008) # 4EP1.2, 2663-2669.; 5. A. M. Gabor et al., “Soldering Induced Damage to Thin Si Solar Cells and Detection of Cracked Cells in Modules”, 21st EPSEC, Dresden (2006) 2042–2047.


Figure 10: Electroluminescence images of a front-contact-cell module before (left), and after (right), 1000 cycles of +/-2400 Pa (equivalent to maximum wind load) of alternating pressure load on the glass. Power loss was 2.1%.

Once cracked, the mechanism for power loss between a front and back-contact cell is different. In a front-contact cell, areas of the wafer with cracks that cut the front-contact metal from the interconnect ribbons isolate areas of the cell. These areas no longer contribute charge carriers, and show up as dark regions in the right picture of Figure 10. For the back-contact cell, copper fingers cover essentially the entire back of the cell. Those fingers are ductile and do not break, so that regions where the silicon wafer breaks remain connected to the circuit regardless of the cracks. The only efficiency loss is due to the recombination that occurs at the recombination sites within the crack itself, as shown by the black lines in the electroluminescence image in Figure 11. This has a relatively small impact, and in fact no power loss was measured for the modules shown in Figure 11, which is typical.

Figure 11: Electroluminescence images of a SunPower back-contact-cell module before (left), and after (right), 1000 cycles of +/-2400 Pa (equivalent to maximum wind load) of alternating pressure load on the glass. No measurable power loss.

A final example of a HALT is the “autoclave” test on bare cells, which SunPower does for cell product development. During the design of some of the first solar cells made by SunPower, this test provided a critical measure of the effectiveness of the moisture-barrier characteristics of the outermost layer of the cells. The comparisons ruled out Titanium Oxide in favor of Silicon Nitride, and also enabled the selection of the optimal Silicon Nitride deposition technique. This test is discussed more fully in Section 6.

It is worth underlining that the purpose of HALT is not to quantitatively estimate the module useful lifetime, but to discover failure modes and causal stresses that must be accounted for in the FMEA. It is also an effective and fast tool for comparing different technologies or products. Lastly, it provides some of the groundwork

S E C T I O N 2


for later testing programs that will operate continuously during manufacturing (Section 6).

Theoretical Understanding The final means of developing a thorough FMEA is through a theoretical understanding of the failure mechanisms. This is especially critical for capturing slow-moving failure modes, which are unlikely to show up in the field for many years, and may not even be encountered during HALT. The only way to capture these failure modes is with lab testing to understand the physics of the breakdown, guided by theoretical modeling.

Good examples of critically important, but slow-moving, failure modes occur in the metal interconnect and solder bonds that electrically connect the cells in a module. Metal-fatigue-failure of interconnects had been observed historically as one of the top two failure modes in the field for traditional front-contact-cell modules. SunPower used this information in the design of the interconnect for back-contact cells in 2004. A design with in-plane strain relief was chosen, with validation including 10,000 mechanical cycles of the interconnect itself (with no failures) and thermal cycling of coupons and modules to 200 cycles (with a power degradation of less than 3%).

However, subsequent testing revealed that solder bond failure could occur in extended thermal cycling, leading to a loss in fill-factor of ~8% at ~500 cycles. The failure mechanism was found to be cracking as the result of heterogeneous coarsening of the solder microstructure [6].

As a result, this interconnect was redesigned to reduce stress on the solder bonds by increasing the compliance of the interconnect, and work was done to model the failure mechanism. This effort included both the SnPb solder which was used in 2004 and the SnAg solder which was under development for improved environmental stewardship. Industry literature indicated SnAg solder should be less susceptible to solder-fatigue failure, which was validated by the thermal cycling. The combination of the higher compliance interconnect (shown in Figure 12) and the SnAg solder resulted in fill-factor loss of < 2% in 700 thermal cycles, and production was switched accordingly [6].

Additional information was then sought to quantify the acceleration factors of the solder joints to be able to predict product life. It was found that grain-boundary-creep and matrix creep are different for the different types of solder, and that different tests and different dwell times and temperatures in cycling could be used to derive the strain energy behavior and model performance [7]. Tests were done with direct monitoring of solder bond resistance of individual solder bonds in coupons through thousands of temperature cycles of varying amplitudes [8], which confirmed very high fatigue-failure resistance for the combination of the interconnect shown in Figure 12 and SnAg solder. There were 0 failed joints in 2300 thermal cycles, significantly more than required for 25 years in a harsh climate. Based on the modeling, extended thermal cycling with close monitoring of fill-factor is used as a standard test for any new product or any change which could influence solder stress or quality.

S E C T I O N 2

6. A. M. Gabor et al., “Soldering Induced Damage to Thin Si Solar Cells and Detection of Cracked Cells in Modules”, 21st EPSEC, Dresden (2006) 2042–2047. 7. J. P. Clech, Proceedings SMTA International Conference (Sept 2004) 776-802. 8. Y. Meydbray, K. Wilson, E. Brambila, A. Terao, S. Daroczi, “Solder Joint Degradation in High


S E C T I O N 2

Figure 12: Strain-relieved cell interconnect between two SunPower back-contact cells. Note the three solder bonds on each cell, combined with the plated copper connections between those bonds on the cell, create redundancy.

14S E C T I O N 3 QUAL IF ICAT ION TEST ING

Example qualification test: backsheetOnly after a solid FMEA is completed (and of course it must be regularly updated as more is learned from ongoing testing and field results), can a test plan be developed to ensure that the product design has high reliability. For example, the first line of the backsheet FMEA indicates the detachment of the junction-box from the backsheet due to a loss of adhesion of the RTV or backsheet interlayer adhesion (Table I). Given an RPN number high enough to make this failure mode of concern, a test must be derived to determine whether the product meets the specifications. To test that particular failure mode, a series of modules and coupons are built with junction-boxes attached to backsheets with RTV, and put into field testing with the cells above the j-boxes shaded to induce operating temperatures from hot cells. Additionally, oven tests at different temperatures up to 120C, with 1 kg weights hanging from the junction-boxes, must not cause any delamination in order to pass.

These qualification tests can be pass/fail if the relevant acceleration factors are known, or they can require an equivalent or better performance compared to a known high-reliability baseline. In most cases, the tests are extended to produce failures, to learn how that happens. Table III is a summary of the testing done to approve a new backsheet for use in the SunPower modules – each test is motivated by one or more failure modes in the FMEA.

Note that these tests are primarily focused on quantifying the targeted failure modes and verifying they will not cause a failure during the expected

lifetime of the product. It is also critical to get modules started early in long-term testing to try to catch any unexpected failure modes.

Long-term testing

Certification-type tests are a good place to start for long-term testing, just by extending the tests to when failure happens. Figure 13 shows SunPower modules in the standard certification tests, and indicates no run-away failure modes, only slow degradation, given these stresses. Note that these tests are similar to HALT, but the stresses are gentler. In the context of qualification testing, it’s important to run the tests until failures occur These extended tests are also repeated periodically as part of the ongoing reliability testing (ORT) for manufacturing.

Field testing

Field testing should always be started immediately, and done simultaneously in different climates (hot and humid, hot and dry, temperate, urban, rural, coastal, etc.) on full size modules. These deployments must be monitored and visited regularly to look for any new developments. The problem with field testing is that the stresses are not accelerated; the benefit is that the tests are inherently realistic combined stresses, and many more samples can be fielded relative to what can be sampled with artificial stress testing. Of course when something does go wrong in the field testing, it means the product has serious flaws.

A good example of a field test catching a problem happened during a failed qualification attempt for a new encapsulant. Coupon-size prototypes had shown

QUALIFICATION TESTING

S E C T I O N 3

15

no problems in lab tests, and as soon as the supplier had the ability to produce material that enabled manufacturing of full size modules, the modules were put into lab tests and deployed to the field in various climates. The field tests were first to show delamination at the encapsulant/glass interface – defects that did not occur during coupon testing (although they also did occur in damp heat testing of full size modules). No changes to the design of encapsulant were made according to the supplier, so this was due to the change in scale of the modules, or to unknown manufacturing volume ramp-up problems for the supplier. When possible it is recommended to start field testing on full size modules rather than small scale prototypes.

There are also some experiments for which field deployment is much more conducive, such as the effect of various anti-reflective glass coatings on soiling rate.

T

Figure 13: Extended testing of SunPower back-contact-cell modules, which pass insulation resistance testing and are less than 10% degraded in power after more than 7x the certification standard in DH, 11x in TC, and 26x in HF.Table III: Backsheet qualification tests (abbreviations

listed below Table V)

S E C T I O N 3 QUAL IF ICAT ION TEST ING

Sample Tests

Backsheet sample: 6”x6”

Moisture Vapor Transmission RatePartial dischargeOut-gassing post heat soak

Surface morphology analysis (initial and post HF10)

Shrinkage (at lamination conditions and during accelerated tests)

Interlayer adhesion (as received, post lamination and HF10)

UV exposure

3-cell laminated coupons

Adhesion tests (to encapsulant, interlay-ers; J-box, labels, framing materials)

ACL168 UVHF40DH2000TC200Sequence B (UV/TC50/HF10)HF/UV combinationCut test (pre- and post-HF, checking for crack formation)high temperature voltage stabilityfield testschemical resistance (various chemicals found in environment)

Full modules

High temperature heat soak (check for out-gassing due to interaction with other module components)

J-box with weights tests for creepHALT (see Table II)HF40DH2000TC200

field test (in various environments, and some with additional weights on J-box)

S E C T I O N 3

16S E C T I O N 3 QUAL IF ICAT ION TEST ING

S E C T I O N 3

Qualification Testing ConclusionThe product can spend considerable time in qualification testing, which typically involves detailed failure analysis and then design changes and re-testing. This is time well-spent, however, as the design stage is by far the most cost-effective time to discover any reliability weaknesses and fix them.

One might notice that the certification tests are not generally sufficient for qualifying a product design. This is because certification tests are designed to indicate a reasonable level of initial product quality and safety, and are not meant to indicate product lifetime [9]. It is worth noting that all commercial modules which have had a problem in the field were of a design which had previously passed certification testing. Product qualification approaches that are based solely on successfully completing certification tests, or simply multiples of the certification tests, are not following good Design for Reliability methodology and are likely to miss important failure modes that are unrelated to the particular stresses utilized by the certification tests.

17S E C T I O N 4 SUPPL IER QUAL ITY

Four stage processOnce the design is qualified, that gives confidence the product will be successful if it is manufactured according to the design – which requires that the input materials are well specified and held within those specifications continuously. To do that, SunPower uses a Supplier Qualification Program, and a Supplier Continuous Monitoring Program, which is defined in 4 stages:

Stage 1 is internal to SunPower, where requirements are clarified and a sourcing strategy is determined. Key requirements are part of the output of the qualification process.

Stage 2 involves contacting likely suppliers, setting expectations and developing joint plans.

Stage 3 is where the primary suppliers go through a “PSC audit” to evaluate how they are doing along the three most important dimensions:

• Prevention: Extent of employee training, usage of Statistical Process Control, FMEAs and a formal CAPA (Corrective And Preventive Actions) methodology, as well as their own Reliability and Supplier Quality Programs.

• Standardized/Simplified/Scalable: Evaluation of business processes to make sure they are of high quality, in particular, the change notification process.

• Customer Satisfaction: customer surveys, responsiveness to customer issues, etc.

Stage 4 is where a strong partnership is executing to deliver high quality products at a large scale. Regular self-assessments with periodic reviews and validation, as well as improvement plans, are put into place. This rigor is absolutely necessary for ensuring a consistently high-quality product.

As part of the periodic review the supplier also gets a STARS score (“Supplier Total Achievement Rating System”), and must have at least an 80% to be an approved supplier, based on:

• Quality: customer issues, reliability, compliance to SunPower change requests, PSC Audit performance, problem recurrence rate.

• Cost and cost-reduction plan

• Availability: on-time delivery, lead time, etc.

• Technology: how does the product roadmap dovetail with SunPower’s plans

Supplier materials auditThis testing is generally accomplished with a mix of vendor tests (with shared reports), and periodic inspections of incoming material. Table IV shows a partial list of the incoming materials testing that SunPower does as part of a regular sampling program.

It’s critically important to have the strong requirement that absolutely no formulation or design changes are to be made to the qualified components without communication and re-qualification. This was recently underlined when a j-box manufacturer changed the formulation of the j-box lid to a stronger type of plastic. Although it appeared to be a desirable change, the

SUPPLIER QUALITY

S E C T I O N 4

18S E C T I O N 4 SUPPL IER QUAL ITY

interaction between components created a new failure mode. During damp-heat testing, significant de-polymerization was found on the inside of the j-box lid, as shown in Figure 14, which proved to be due to a reaction between the new plastic and a chemical out-gassing from the double-sided tape that is used to hold the j-box in place while the RTV cures.

Table IV: Incoming materials audit

One of the more subtle areas to pay attention to is the qualification of the vendor’s different manufacturing sites, even when the component being purchased is the same part number. For example, a particular backsheet supplier was bringing up a new manufacturing facility, and although the pre-ramp-up material passed accelerated testing results that exceeded the IEC61215 testing requirements [10], and also passed SunPower’s internal qualification, the fully-ramped-up material was found to have a defect. Incoming inspection tests showed the backsheet from the supplier’s new manufacturing facility had weaker interlayer adhesion as compared to material received during the qualification testing. This pointed to an inconsistency of the material quality from the new manufacturing facility that the supplier was then able to correct.

Figure 14: De-polymerization damage on the interior of the j-box lid due to chemical interaction found during damp heat testing

Material Material

Encapsulant Gel test, pull test, temp soak testBacksheet Peel test, temperature soak testDiodes Leakage current test

Connectors Production test (wiring fitness, crimping, pull test)

Frame Mechanical load test, frame pull testLabel Tape test, IPA test

S E C T I O N 4

19S E C T I O N 5 MANUFACTUR ING QUAL ITY

SystemConsistency in manufacture is an essential part of achieving and maintaining the quality necessary to meet the reliability requirements of PV cells and modules. Statistical Process Control (SPC), Total Quality Management (TQM), and Six Sigma are among the general systems which can help attain and maintain quality in manufacturing. All these methodologies emphasize one main point: quality must be built into the product, and cannot be audited into the product.

The semiconductor industry has been a leader in the development of manufacturing quality procedures. This was necessitated by manufacturing processes that involve hundreds of steps, and 30 to 60 days of work-in-process. Given those conditions, very rigorous Quality and Change Control methodology were required, since if a problem was not discovered until the end of the line, millions of dollars of product must be scrapped. With the 25 year warranty period of PV modules, minimizing problems in manufacturing which would not be discovered until later is also essential.

SunPower had the good fortune of a close relationship with Cypress Semiconductor as a way to learn advanced manufacturing quality techniques. Cypress invested in SunPower in 2002, completed its acquisition in 2004, and concluded its ownership of SunPower shares in 2008. Cypress offered tremendous support developing SunPower’s quality programs and discipline during the pilot line testing and the deployment of the first full-scale cell manufacturing facility.

The system developed by SunPower and Cypress was built on the foundation from Cypress, and then refined to better match PV manufacturing. Key elements of the

system include (1) systematized business processes for all developments and changes, (2) coordinated product, process, and equipment development procedures with six levels of control, (3) quality and continuous improvement mentality, and (4) variability and waste reduction.

Process development and implementationThe development of the product and the manufacturing processes must result in process control (including all interactions, material variability, etc.) which is sufficient to consistently deliver a high-quality, reliable product. SunPower uses a six-level approach which manages the interactions of product characteristics, process variation, and equipment variation. The process portion of each level is as follows:

Level 1 determines what to control. At this stage, the key variables of product, process, and equipment which can affect the final product are identified.

Level 2 determines how to control. Response Surface Maps (RSMs) are used to understand the process space, and Coefficients of Variance (COVs) are used to determine sampling plans. Without RSMs, it is possible that the process is statistically robust (e.g., a Cpk > 1.3), but near a portion of the process space that would result in significant movement out of the process window in the event of a shift of one of the input variables.

Level 3 establishes control. Online SPC charts, as shown in Figures 15 and 16, are used to verify that sufficient control has been achieved and to provide the ability to react to a change in the process (not just to a

MANUFACTURING QUALITY

S E C T I O N 5

20

change that results in an out-of-specification condition). Product is also run with the extremes of process windows of other steps in manufacturing (called “running the corners”) to determine if the product is reliable with the largest possible interaction between steps.

Level 4 verifies control. Implementation of out-of-control action plans (OCAPs) are an important part of this level because they ensure that manufacturing appropriately responds to any changes.

Level 5 establishes ownership of the process, which includes active root-cause-corrective-action (RCCA) and tool matching (comparing the performance and sensitivities of similar tools) to further improve the process.

Level 6 drives continuous improvement. Structured business processes to manage any changes are very important for carefully introducing improvements.

It is essential that process development be done in coordination with product development and equipment implementation. Interactive improvement of the product specification is needed to both drive the process development, and to benefit from the work done in process development.

Example SPC chartsFigure 15 and 16 provide 2 of the more than 100 SPC charts that SunPower uses in the manufacturing quality program. They illustrate that control limits are not the same as specification limits. As seen in Figure

15, the guarantee of adhesion for the material shown is 46 N/cm (lower spec. limit), but the control limits force action if the mean changes from the 177 N/cm target (XBB) by more than 42 N/cm in either direction (upper and lower spec. limits). Even an unusual shift of the average or standard deviation with all points still within the control limits requires determination of the cause of the change and whether or not it can have any product quality implications.

Figure 15: Daily gel-test measurements from the SPC chart of one laminator. USL/LSL – upper and lower specification limits, UCL/LCL – upper and lower control limits, XBB – target.

S E C T I O N 5 MANUFACTUR ING QUAL ITY

S E C T I O N 5

21S E C T I O N 5 MANUFACTUR ING QUAL ITY

Figure 16: Daily pull-test measurements from the SPC chart of one laminator

Quality AuditThe Quality audit is an important part of achieving and maintaining the consistent manufacture of a high reliability product. This is most effective if done by people who are outside of a manufacturing unit, such as within an independent Quality organization, since they are independent and more able to effectively identify areas for improvement.

The audit of workers’ adherence to the manufacturing specifications serves both to instill the proper mentality on the line and to catch breeches. The mentality must be “follow the spec, or change the spec” (with any proposed change to the specification being done only through the structured business process).

As part of the Quality Audit, SunPower regularly does an “Out-of-box audit.” This is a random sampling of products that have been packed for shipment, to check the quality of the packaging, documentation, product cleanliness and visual defects, and electrical performance as compared to recorded performance.

S E C T I O N 5

22S E C T I O N 6 TEST ING CONTINUOUS MFG REL IAB I L I TY

S E C T I O N 6

Two methodologiesproduct does not change over time. The testing plan derives directly from the (regularly updated) FMEA, and falls into two categories:

• Highly Accelerated Stress Audit (HASA) - to detect any material defects or manufacturing problems which have escaped the supplier quality and manufacturing SPC hurdles.

• Ongoing Reliability Testing (ORT) - to continue to validate the baseline reliability of the product. These are longer-duration tests with milder stress levels.

Highly Accelerated Stress Audit (HASA)These tests must have a fast turn-around time and must be very well structured with clear pass/fail criteria that non-experts can interpret unambiguously. Note that the pass/fail criteria should be set to show problematic manufacturing variations, and must be closely tied to Out-of-Control-Action-Plans (OCAPs) to resolve any failed tests quickly. In many cases, there are gradations of “fail,” ranging from a mild failure that requires a high-priority engineering review and failure analysis, to serious failure requiring a full line shutdown. For HASA testing, a “fail” does not necessarily mean a serious product problem, but serves as a warning that something is drifting from the target process. Unlike HASA in the electronics industry, even passing samples used for these tests are not sold to customers, since they are stressed to the point where they show degradation so they must be scrapped. This has the advantage, however, that higher stress levels can be used to attain an improved testing turn-around time.

The HASA tests are built to exercise the important degradation mechanisms determined from the earlier steps in the product design process (HALT, historical data, theoretical understanding, and FMEAs from the product design and the process designs). The list of degradation mechanisms is a guide for the stresses required to degrade the product which in turn helps to define the tests. It is also important to determine the appropriate pass/fail metric, which in most cases comes from a performance test (IV curve trace with the associated efficiency, fill-factor, Imp, Vmp, etc.). For some degradation mechanisms, however, performance testing is not the appropriate metric, e.g., if the failure mode is delamination of the encapsulant, then it makes more sense to do an adhesion measurement. Table V lists the failure modes and degradation mechanisms captured by SunPower’s current HASA tests. Note that the HASA test battery is not necessarily comprehensive, but is designed to complement the incoming materials inspections and manufacturing SPC.

SunPower does HASA testing on bare cells, 3-cell coupons, and full sized modules, depending on the T – temperature, UV – ultraviolet light, TC – thermal cycling, RH – relative humidity, ACL168 – autoclave (168 hours in 100% relative humidity, 121C at 2 atmospheres of pressure), HF – humidity-freeze cycles degradation mechanism being tested. For instance, to measure the sensitivity of the cells to moisture and temperature degradation, there is no need to encapsulate the cell in a module. Bare cells can be placed in a high temperature (121C) and high humidity (100% relative humidity) environment, and the rate of oxidation of the Silicon Nitride moisture-barrier layer can be inferred from the Vmp drop after 7 days (168

TESTING CONTINUOUS MFG RELIABILITY


S E C T I O N 6

Failure Mode Degradation Possible Defects Stress Test Measurement

Cell Impdegradation (or both Imp and Vmp)

Front-surface passivation degradation

Surface contamination,non-uniformities or pinholes in moisture barrier

T, RH Cells - ACL168 PerformanceUV Cells – UV test Performance

Voltage Bias Coupons – Oven with voltage bias Performance

Metaldelamination

Metal-to-contact misalignment, residual oxide, contamination TC Cells – ACL168 Performance

Corrosion Contamination T, RH Cells – ACL168 Performance, Metal pull test

Cell cracking Overly-thick metalization,surface irregularities

Mechanical loading, TC Coupons – HF10 Performance,

EL imaging

Cell Vmpdegradation

Back-surface passivation degradation


T, RH Cells - ACL168 Performance

Moduledelamination

Reduction in surface bonds

Contamination on cell or glass, over curing, EVA voids or bubbles, shift in lamination process

T, RH, UV, Bias Modules – DH with voltage bias

Visual InspectionPerformance Backsheet pullHigh pot. test

EVAtransmission loss EVA Browning Incoming material variation

(variation in UV absorber) R, RH, UV Coupons – UV test Visual inspectionPerformance

Backsheetdegradation

Cracking, inter-layer adhesion failure, Insula-tion de-polymer-ization

Backsheet wrinkling, incoming material variation T, RH Backsheet sample –

ACL168 Visual inspection

hours) in this harsh environment – this is the “autoclave” test, called “ACL168” in the top row of Table V.

The efficacy of this approach was demonstrated during the ramp-up of SunPower’s second back-contact cell fabrication facility. Some of the first-runs of cells were found to have an increased oxidation rate after the autoclave test, suggesting they were not as robust as normal cells, even though the modules were passing the standard ORT tests. Failure analysis revealed a potassium surface contamination, and this was traced to a design problem in the rinsing process on a new manufacturing tool. This was corrected, and performance of the cells in the HASA autoclave test returned to normal. 6.3 Ongoing Reliability Testing (ORT)

Table V: HASA test matrix


S E C T I O N 6

These tests are longer with lower stresses, smaller sampling sizes, and provide a general indication that the baseline reliability of the product is not changing. 1000 hours of damp heat (85C and 85% relative humidity), 200 thermal cycles (5 per day between 90C and -40C), and 40 cycles of humidity-freeze (1 cycle per day of 85C with 85% relative humidity, with an hour at -40C), are good ongoing reliability tests because they are the standard tests for qualification testing.

For all ORT tests, the performance of the module is monitored using mid-reads at predetermined intervals to provide an early indication if there is a problem. In addition to the performance measurements, the samples undergo visual inspection, insulation resistance tests and measurement of electroluminescence during the mid-reads to look for defects that do not clearly impact the performance measurements in the short term. As part of the ORT testing, it’s also worthwhile to periodically do very long-term tests-to-failure with new product samples, similar to what is shown in Figure 13. This improves the chances that a failure mode that is not activated by the standard ORT tests has not been accidentally introduced. This is done periodically as an extension of the standard ORT tests.

Table VI list the SunPower ORT tests as well as the applicable failure modes and degradation mechanisms. For these tests, the measurements taken are all approximately the same: visual inspection (discoloration, delamination, etc.), flash test (efficiency, Imp, Vmp, fill-factor, etc.), insulation resistance test (leakage current), electro-luminescence imaging (relative recombination), and J-box pull tests.

Table VI: ORT test matrix

Sample-size determinationDetermining the best sample size for HALT and ORT is a challenging balance of resources and test coverage. In general, the stakes of having a problem released into the field are high due to the high cost of replacement, and so a significant sampling rate is required to minimize the risk. Increasing the sample size decreases the expected amount of time it takes to detect a problem, since larger sample sets means it is more likely that at least one of the test samples has the defect and will fail the test. Thus, increasing the sample size will decrease the number of defective products that escape manufacturing. On the other hand, increasing the sample size requires testing personnel and equipment, and reduces output.

Test Failure Mode Degradation Mech.

DH1000

Glass AR-coat-ing deg.


Adhesion loss Metal-to-contact misalignment, residual oxide, contamination

EVA trans. loss Contamination

Backsheetdegradation

Overly-thick metalization,surface irregularities

TC200 Solder bond failure


HF40

DelaminationContamination on cell or glass, over curing, EVA voids or bub-bles, shift in lamination process

J-box failure Embrittlement, internal metal cor-rosion, attachment to backsheet

Connector degradation Embrittlement

25

hfghfhj

S E C T I O N 6 TEST ING CONTINUOUS MFG REL IAB I L I TY

S E C T I O N 6

The sampling rates also depend on the maturity of the manufacturing line, and its historical performance. SunPower has taken a conservative approach; for example, mature manufacturing lines have a sampling rate of approximately 1 cell per 20,000 for the HASA ACL168 test (Table V), and 1 module per 5,000 for the ORT tests (Table VI). Each company must make its own determination of the right balance.

26

Glass-surface contaminationThis is a good example of a seemingly insignificant change in manufacturing process having a surprising effect. For a short period in 2007, a modification was made to the cooling racks for SunPower modules coming out of the laminator that used a different insulating cloth to protect the hot modules from being scratched while they cooled. This new cloth, however, turned out to contain a silicone oil which caused the glass to have a different contact angle with water. This was invisible in the factory, but creating uneven soiling when deployed to the field.

Although this problem was discovered fairly quickly, it still required significant investment in a 9-step cerium oxide cleaning process to bring all the customer installations up to the normal look of new modules.

This episode shows the value of fully vetting every process change. SunPower now uses an outgoing inspection for non-washable stains on glass by using water spray. This matches the field methodology of

using water for washing, whereas previously these tests in the factory were conducted with isopropyl alcohol. Additionally, all manufacturing accessories that have any physical contact with modules are now included in the list of materials to be controlled as part of production.

Figure 17: Fielded modules with silicone contamination on glass surface show 4 stripes.Loose Junction-box terminals

S E C T I O N 7 CLOSED- LOOP LEARNING FROM THE F IE LD

S E C T I O N 7

CLOSED-LOOP LEARNING FROM THE FIELD

From the perspective of reliability, field failures present a tremendously valuable opportunity for closed-loop learning about real product degradation and the causal stresses. Every new failure should have a thorough and thoughtful failure analysis done, and then an update to the FMEA and associated testing. Below is a selection of examples of failure modes that were learned from fielding modules, which now advise many of the test plans currently in use.

27S E C T I O N 7 CLOSED- LOOP LEARNING FROM THE F IE LD

During a short period in 2006, SunPower determined that some screws securing output wires into an old-style j-box were insufficiently tightened during production. The effects of this issue manifested as a high-resistance connection in the j-box, which in turn caused low power modules and heating of the terminal blocks.

Figure 18: J-box showing a loose screw connector (right side screw shows thermal damage)

The root cause was calibration drift for the torque wrenches used during assembly. SunPower reworked all the deployed modules in situ, and implemented a manufacturing process requiring validation of the torque wrench calibration at every shift to address the root cause.

This issue is a good reminder that production facilities must closely monitor and control use of every tool used in module production – not only the cell manufacturing processes, but also the more mundane processes,

like the use of torque wrenches to assemble frames or connect wires. To avoid this failure mode entirely, SunPower eliminated screw connections inside j-boxes.

Connectorized cable insulation resistanceThis example did not quite make it to the field, but it does illustrate the same process of closing the loop from final testing to new standard test plans and updated FMEAs. In 2009, the final qualification test for a new wire/connector pair was humidity-freeze, and a wet insulation resistance failure occurred. This was traced to a sizing mismatch between the cable and connector that resulted in a broken sealing ring in the connector.

Continued investigation revealed that even dimensionally compatible cables also sometimes had failures, which lead to a new qualification test to compare lifetimes of a population of cable/connector pairs with cycles of humidity-freeze.

Figure 19: Wet-insulation resistance reliability vs. number of Humidity-Freeze cycles for 4 different cable/connector pairs.

HF cycles

Reliability

0 10020 40 60 800.0

1.0

0.2

0.4

0.6

0.8

S E C T I O N 7

28CLOSED- LOOP LEARNING FROM THE F IE LD S E C T I O N 7

From the perspective of reliability, field failures present a tremendously valuable opportunity for closed-loop learning about real product degradation and the causal stresses. Every new failure should have a thorough and thoughtful failure analysis done, and then an update to the FMEA and associated testing. Below is a selection of examples of failure modes that were learned from fielding modules, which now advise many of the test plans currently in use.

S E C T I O N 7

29S E C T I O N 8 CONCLUS IONS

S E C T I O N 8

CONCLUSIONS

A complete design-for-reliability process has been presented to illustrate a methodology for deploying high-reliability modules, even in the face of the rapid materials and process changes required to decrease product costs while still delivering on the industry-standard 25 year warranty.

This starts with a thorough FMEA based on field experience, HALT, and theoretical understanding – the output of which determines the qualification testing needed to deliver a high-reliability product design, as well as an understanding of the product’s failure modes and causal stresses. Supplier quality is maintained with a supplier qualification program, with continuing audits of incoming materials. Manufacturing builds quality into the product through Statistical Process Control of all the important variables, and ongoing sample testing of the final products; HASA and ORT assures the continued high-reliability of the product.

Figure 20: The primary methodologies discussed in each process area

It is important that many of these tests are “tests to failure” since the goal is to understand and then eliminate failure modes in the product design and manufacturing processes. This is a broader goal than the standard certification pass/fail tests, since those are designed as a baseline assurance of initial product quality and safety, and are not meant to be indicative of product lifetime.

This material is based in part upon work supported by the Department of Energy under Award Number DE-FC36-07GO17043.

29

SUNPOWER CORPORATION3939 North 1st StreetSan Jose, CA 95134 1.800.SUNPOWER (1.800.786.7693)sunpowercorp.com

SUNPOWER and the SUNPOWER logo are trademarks or registered trademarks of SunPower Corporation.© February 2011 SunPower Corporation. All rights reserved.

30

Documents

Qualification, Manufacturing, and Reliability Testing Methodologies