SEB ProcessSEB Process

Chris WhyteChris Whyte

NASA GSFC SEB ManagerNASA GSFC SEB Manager

Feb 22, 2010Feb 22, 2010

Getting Started SEB Appointment Memo and OGE 450s should be complete. Inform legal of

changes between anticipated offerors and actual offerors after proposal receipt, including significant subcontractors.

See William Brown (SEB IT Support) for demonstration of AES software evaluation tool, if needed. If not using AES, develop your own evaluation form/tool.

Coordinate Keys/Keycards with Candace Carlsen (SEB Assistant) All SEB members should review the RFP Instructions and Evaluation Criteria

(RFP Sections L&M). CO should coordinate proposal receipt with shipping/receiving. If possible, CO or

designated team member should be at shipping receiving approximately 15 minutes before/after due date/time.

Designate initial proposal compliance review team to review the following: Compliance with page limits Offeror signature on proposal cover form and acknowledgement of amendments Completion of offeror fill-ins, section K, etc Receipt of all required proposal volumes and attachments Business size status, if applicable Any prime or subcontractor deviations/exceptions to RFP terms and conditions –

contact SEB Mgr and Legal Request or follow-up on DCAA Audit ASAP

11

Key Roles & Responsibilities Source Selection Authority (SSA)

Appoints SEB Members, as soon as possible in the acquisition Makes selection at completion of evaluation phase Should not receive communication during evaluations, except for schedule delays

or significant issues that may require amendment or competitive range SEB Chairperson

Principal Operating Executive of the SEB Ensures proposals are evaluated fairly Manages team efficiently Ensures schedules are met Resolves procedural matters Ensures proposals and SEB Information are safeguarded

Consultants Individuals have specialized knowledge or experience, provide findings to SEB, do not

attend SEB deliberations Cost Team

Individuals have specialized resources experience to verify/validate realism of cost proposal and consistency with technical proposal

Ex-Officios Center Management Officials, advisory role, not part of rating process, generally only

involved to support resolution of unanticipated problems/issues SEB Recorder

Administrative Assistant to SEB22

Key Roles & Responsibilities

Voting Members

Include SEB Chairperson, Senior Key Technical Representative, Experienced Procurement Representative, Senior Safety and Mission Assurance Representative (as appropriate)

Limited to a maximum of seven Should be assigned on a full-time basis Assist Source Selection Authority (SSA) Provide expert analysis of proposals Prepare and present findings Avoid trade-off judgments among offerors or evaluation factors Work closely with Cost Team members to ensure consistency between the

technical proposal and cost proposal Do not make recommendation for selection

33

Solicitation Phase Approved SEB Board Members

Financial Disclosure Statement per SEB Documentation, Evaluation Tree, and Training for AES NLT 2 weeks before

proposal receipt 2 methods for using AES (1 recorder versus individual evaluators input data)

Legal, Security, SEB Briefings required SEB Access

Keys, Keycards, Badges

Proposal must be received in Building 16W and coordinated with the logistics services contractor personnel Ensures timely/untimely receipt of proposals is documented Ensures logging of proposal volumes actually received Ensures secure control of volumes until secure transport to evaluation site(s) is

coordinated(Remember to inform Shipping and Receiving and SEB facility personnel of your

proposal due date (and extensions) and date/time of transport from Bldg. 16W to Bldg. 25)

44

Solicitation Phase Inform Legal Counsel of offerors received

Legal Counsel will notify CO whether any conflict of interest is present based upon review of SEB members financial disclosure statements

If no conflict of interest is evident, SEB may proceed with evaluation

Document Control Establish log to track all proposal volumes

Must insure all are accounted for when evaluation phase is completed

Distribute only to authorized team members Proposals must be marked “Contractor Bid or Proposal Information

See FAR 3.104”

55

Evaluation Phase Proposals must be protected at ALL times

Locked up whenever NOT under direct, personal control of evaluator No discussions outside of controlled areas No information shared with non-team personnel No writing on the proposals Telephone communications should be on the secure phone system (STU-

III)(see SEB Manager for exceptions) Proposals should be evaluated serially

Read proposal by proposal After each proposal evaluated in series, then horizontal and vertical

consistency check Proposals are compared to RFP evaluation factors and

subfactors—NOT compared against each other

66

Evaluation Phase Preliminary Evaluation

Identification of unacceptable proposals It does not represent a reasonable initial effort to address the essential

requirements of the RFP or clearly demonstrates that the offeror does not understand the requirements

In R&D acquisitions, a substantial design drawback is evident in the proposal, and sufficient correction or improvement to consider the proposal acceptable would require virtually an entire new technical proposal

It contains major deficiencies or omissions or out-of-line costs which discussions with offeror could not reasonably be expected to cure

CO/Chair shall consult with SEB Manager and Legal Counsel before proceeding

CO shall document the rationale for discontinuance of initial evaluation

77

Evaluation Phase Exchanges with Offerors after receipt of proposals

Clarifications – Limited Exchanges (relevance of past performance, adverse past performance, resolve minor or clerical errors). Award without discussions is contemplated as cited in RFP

Communications (before establishment of CR)– Exchanges leading to establishment of competitive range. Held only with (1) offerors whose past performance is the determining factor preventing them from being placed in CR and (2) offerors whose exclusion from, or inclusion in , the CR is uncertain

Enhance understanding, allow reasonable interpretation, facilitate evaluation process Not used to cure proposal deficiencies or material omissions, alter technical or cost

elements, revise proposal Used for the purpose of addressing issues that must be explored to determine whether

a proposal should be placed in CR Communications (after establishment of CR)– Negotiations are exchanges with

intent of allowing offeror to revise proposal. Deficiencies, significant weaknesses, adverse past performance, cost/price elements that do not appear to be justified are revealed. Final Proposal Revision (FPR) is page limited.

88

Evaluation Phase

Initial Evaluations

Competitive Range/Selection on Initial Proposals

Final Proposal Revision

Final Evaluations Proposals re-scored based on FPR evaluations Scoring changes between initial and FPRs shall be clearly

traceable

99

Evaluation PhaseDocumenting Evaluation Findings Identify link between RFP requirement and proposal Classify findings as strengths, weaknesses, significant strengths, significant

weaknesses, and deficiencies Include words from definitions in each finding

Weakness – a flaw in the proposal that increases the risk of unsuccessful contract performance

Significant Weakness – a proposal flaw that appreciably increases the risk of unsuccessful contract performance

Deficiency – a material failure of a proposal to meet a Government requirement or a combination of significant weaknesses in a proposal that increases the risk of unsuccessful contract performance to an unacceptable level

Strength a proposal area that enhances the potential for successful performance or contributes toward exceeding the contract requirements in a manner that provides additional value to the government (this could be associated with a process, technical approach, materials, facilities, etc.).

Significant Strength a proposal area that greatly enhances the potential for successful performance or contributes significantly toward exceeding the contract requirements in a manner that provides additional value to the government.

1010

Evaluation PhaseDocumenting Evaluation Findings Minor irregularities or apparent clerical mistakes are not

weaknesses and are identified to offerors during clarifications Combine findings when appropriate

Several findings should be considered for accumulation as a significant finding, when related and appropriate

Document with confidence and courtesy Findings will be disclosed during debriefings and protests Do not retain technical notes, documents, early drafts. Retain latest

version of evaluator findings and ultimately, the final report/presentation (and all documents presented to the SSA)

Do retain backup documentation related to cost realism analysis and past performance evaluation details

Evaluators work as team to consolidate findings as consensus findings

1111

Evaluation PhaseDocumenting Evaluation Findings Findings shall be clear and concise with supporting rationale (avoid simple conclusory statements)

Structure of a finding: Conclusion (1 to 2 sentences) – A broad statement based on many facts.

Ex. Demonstrates an excellent understanding of……; Proposed design exceeds the requirement of/for….

Facts (most lengthy) – specific examples, relevant to conclusion and direct from proposal Impact Statement (1 to 2 sentences) – relates directly to risk/benefit to the Government

(technical, schedule, cost, safety). Evaluations must consider the probability of success and the impact of failure and must identify associated risks or benefits with an identified weakness or strength

Ex. Proposed approach requires repeated critical testing resulting in a substantial delay for delivery; High employee turnover rate would adversely impact effective/efficient operations

Risks which impact cost must be considered in the Cost/Price evaluation (Cost Realism Adjustment)

Include words from findings definitions in each finding

1212

Evaluation PhaseMission Suitability Factor Evaluation

Evaluate proposal thoroughly against evaluation criteria in RFP and develop findings Remember to complete proposal evaluations serially,

not concurrently Horizontal consistency check will involve some

comparison only in terms of ensuring accuracy and fairness across proposals

Helps avoid comparing proposals against the “best” proposal versus the RFP

All team members (voting) must read entire proposal Insures full participation and all relevant findings will

be identified (many pairs of eyes)

1313

Evaluation PhaseMission Suitability Factor Evaluation

Identification and documentation of proposal strengths and weaknesses (evaluation findings) is vital Basis for initial selection or determination of competitive range Framework for discussions with offerors Basis for trade-offs for SSA to justify cost differentials

between proposals Framework for debriefing of unsuccessful offerors

After consensus on findings, the next steps are the adjective ratings for each subfactor, followed by numerical scoring (if applicable)

1414

Evaluation PhaseCost/Price Factor

Cost Realism evaluates elements of proposed cost estimate Are costs realistic for the work to be performed as required by the contract? Do costs reflect a clear understanding of contract requirements? Are costs consistent with offeror’s technical proposal?

Under cost reimbursement competitions, offerors may hope to improve chance of selection by proposing unrealistically low costs Compromises the integrity of the competition Compromises contractor cost performance after award

Cost Realism Adjustments Verification/Validation of proposal Verification/Validation of DCAA audit report (Cognizant ACO); Request for

DCAA audit report early in process (NASA Form 1434) ), be clear in request and follow-up regularly with DCAA on status

Technical Adjustments Indicate level of confidence in probable cost assessment for each proposal “Resource Realism” findings may be in Mission Suitability

Procurement Policy POC for Cost/Price expertise is available for support1515

Evaluation PhasePast Performance FactorPast Performance assesses offeror’s record of performing services similar to

those in the relevant procurement Team determines appropriate adjectival rating but no point score Negative past performance evaluations must be shared with offerors

(potential clarifications or discussions) Findings (not classified as strengths or weaknesses) documented and

consensus achieved Evaluation considers many sources of information

Must consider both prime and subcontractor data (depending on RFP Evaluation Criteria)

All Information should be updated after “Discussions”, if applicable Offeror’s listed contracts and performance descriptions Questionnaires submitted by offeror’s references Past Performance data bases (NASA PPDB, PPIRS) Award Fee Ratings See SEB Manager for Senior Management Input Process

1616

Evaluation Phase

SEB Scoring

Identified in RFP Mission Suitability - Subfactors adjectival rating and

numerical weights Cost/Price – proposed cost and probable cost (with

confidence assessment) Past Performance – adjectival rating (level of confidence)

Scoring consistency (consistently applied) Subfactor Strengths and Weaknesses Definition of adjective ratings Numerical scoring philosophy

1717

Evaluation Phase – Mission Suitability Subfactors Only

.

1818

A comprehensive and thorough proposal of exceptional merit with one or more significant strengths. No deficiency or significant weakness exists.

AdjectiveRating

NumericalEquivalent (%) Definition

Excellent 91-100

Very Good 71-90 A proposal having no deficiency and which demonstrates over-all competence. One or more significant strengths have been found, and strengths outbalance any weaknesses that exist.

Fair

Poor

51-70

31-50

0-30

A proposal having no deficiency and which shows a reasonably sound response. There may be strengths or weaknesses, or both. As a whole, weaknesses not off-set by strengths do not significantly detract from the offeror's response.

A proposal having no deficiency and which has one or more weaknesses. Weaknesses outbalance any strengths

A proposal that has one or more deficiencies or significant weaknesses that demonstrate a lack of overall competence or would require a major proposal revision to correct.

Good

Evaluation Phase

SEB Scoring Process

Consensus on adjectival rating must be achieved Consensus on percentile score within adjectival rating required

(averaging individual percentages not permitted) Subfactor point scores are calculated by multiplying subfactor

weight by consensus percentile Subfactor scores are totaled to establish Mission Suitability point

score – no adjective rating for total score

1919

Recent Protest Issues Past Performance: PAAC and SCNS

Failure to adequately consider and document size relevance to justify Excellent Ratings

GAO Guidance from SCNS indicates prime contracts must be approximately 30% or larger in comparison to the proposed effort to be at least somewhat relevant in size, although this is not a strict rule

Mission Suitability: Failure to evaluate in accordance with RFP Lack of consistency between offerors for similar content/findings Unreasonable weakness for incumbent capture rates based on Government data

that was unclear Cost/Price:

Inaccurate Calculations and/or erroneous data

Selection/Award Phase Written SEB Report, Charts, Selection Presentation

CO to coordinate with Code 210 Assoc. Division Chief re: a request to the PO to waive the review and approval of contract award dependent upon the nature and extent of changes included in the model contract.

EEO Preaward Clearances Required for contracts and subcontracts of $10M or more (excluding construction) CO shall request clearance from OFCCP regional office before award of any contract, unless:

Proposed contractor is listed in OFCCP’s National Preaward Registry via Internet (PIC 99-2); Projected award date is within 24 months of the proposed contractor’s Notice of Compliance

completion date in the Registry; and CO documents the Registry review in the contract file

Central Contractor Registration (CCR) issued Cage Codes Effective December 31, 2001, a contractor must supply a Cage Code in order to be awarded a

contract. Cage Codes need to be specific to the contracting entity by name and address. CCR may be accessed via Procurement Library website to verify Cage Code Exemption apply per NFS

VETS – 100 Report (Federal Contractor Veterans’ Employment Report) Applicable to awards of $25K or more (N/A for commercial items or contracts that do not exceed simplified

acquisition threshold CO can not obligate or expend funds with a contractor that has not submitted required annual Form VETS-100

Performance Evaluation Plan - For award fee contracts, Performance Evaluation Plans must be completed/approved prior to contract award

Need a Funded PR (or PRs if you also have a phase-in PO) – If you were previously working from a planning PR with no funds, you need to get funding prior to award!

2020

Selection/Award Phase

Source Selection Statement (SSS) NASA Source Selection Statement Development Guide – PIC 04-10 The signed SSS is released to all offerors, and is not redacted. The

SSS is posted to NAIS after contract award. The Contracting Officer drafts the SSS, Legal Counsel and the SEB

Coordinator provides input, assistance, and review The SEB Technical Members should review the draft SSS for accuracy

and to ensure that no proprietary information is included See GSFC Procurement Circular 98-2 for official review requirements

2121

Selection/Award Phase Actions Requiring Administrator Notification (change from Administrator's

Notice of Significant Contract Action (ANOSCA) and NASA Public Announcement Public Announcement required over $5M ($25M for options), ANOSCA required over

$25M – Must inform Center Director First via SEB Manager! CO should work with HQ Office of Procurement for unique actions that are below these

threshold levels, but may still be appropriate for Public Announcement and/or ANOSCA Single template used for HQ Public Announcement and/or ANOSCA.

http://procurement.nasa.gov/cgi-bin/vpo/vpo_matrix.cgi

Template is coordinated through HQ Office of Procurement via encrypted e-mail (PKI) as soon as possible after the selection meeting and at least 5 workdays prior to planned public announcement. The SEB Manager and Assistant have PKI capability, if needed.

For GSFC (Not 210.H) Public Announcements over $25M, a Draft press release should be coordinated through the installation PAO (Cynthia O’Carroll)1-3 days prior to the final selection meeting, but only after the successful dry run for the selection meeting.

Press release draft shall include general procurement information, but shall not include information on any anticipated successful offeror(s). Successful offeror information shall be provided to the PAO after the selection meeting/decision.

Coordination of 210.H press release information will occur after the template has been submitted to the HQ Office of Procurement.

See NASA Procurement Notice 04-08 and the GSFC SEB Manager for more detailed information

2222

Selection/Award Phase

Notifications to Unsuccessful Offerors Preaward Notices

Exclusion from competitive range Small Business Programs (SB Set-Aside, SDB

benefit, HUBZone) See FAR 15.503(2) for required pre-award selection

notice 5 days prior to set-aside awards Postaward Notices

Within 3 days after contract award to those in competitive range and not selected

Notification to successful offeror(s) and unsuccessful offerors via phone calls from SSA must be coordinated along with Administrator Notification and Public Announcement

2323

Selection/Award Phase Debriefings Process

(NASA Procurement Debriefing Guide – PIC 04-11) GSFC does not release point scores (see SEB Manager for exceptions) GSFC has “debriefing official” position at least one level higher than the CO, and equal to

the solicitation approval level, unless delegated For GSFC, legal counsel and the SEB Manager do not typically attend, although it is

considered on a case-by-case basis Consult with SEB Manager prior to debriefings to review the debriefing process and

schedule the debriefing preparation meeting Facilitate open, appropriate and meaningful information exchanges that reduce

misunderstandings and protest Preaward

Written request submitted within 3 days after receipt of the notice of exclusion from competition Limited information disclosed

Postaward Written request submitted within 3 days after the date offeror has rec’d notification of contract

award Substantial information disclosed

2424

Selection/Award Phase

Debriefings Continued Modified Practices Effective 1/2003

Redacted charts, agenda, source selection statement provided in advance of debriefing meeting

Redacted charts contain general procurement information (excluding Cost Estimate/budget, schedule, etc) and specific evaluation information on offeror being debriefed. Total probable cost/price inclusive of fee/profit and total contract value inclusive of fee/profit is provided. Mission Suitability information is provided without point scores. Written relevant questions accepted in advanced and answered orally at debriefing.

Protest Shall be filed 10 calendar days after contract award or within 5 days after debriefing date,

whichever is later

2525

Selection/Award Phase

Transition Project Team Dedicated to coordinating and leading all aspects of contract transition Team members include home organization management, contracting officer (CO),

Contracting Officer’s Technical Representative (COTR), Resource Analyst, IT representative, Security representative, and representatives from major user organizations

Role is to establish, publish, and maintain a comprehensive and coordinated schedule of all events and activities necessary to contract transition

A Transition Handbook has been developed

Transition Challenges Keeping track of all technical equipment, intellectual property (software and data) and end

items required from the incumbent contractor Considering the potential impact of computers/data for transition of follow-on service

support contracts. Government software and individual personal data cannot be taken on contractor computers. These issues need to be coordinated with the CIO Office and legal counsel.

Coordinating with Security for badges and keys Identifying property to be transferred to the new contractor

2626

Milestone Chart – Provide to SEB Manager and Procurement Associate

Evaluation Milestones Start Date Completion Date # of work days

       

Technical Evaluation      

- Complete Contractor 1 Draft Findings      

- Complete Contractor 2 Draft Findings      

- Complete Contractor 3 Draft Findings      

- Complete Contractor 4 Draft Findings      

Past Performance Evaluation      

Cost/Price Evaluation      

Horizontal/Vertical Review of Findings      

Probable Cost Adjustments      

Technical Adjective Ratings/Scores      

Horizontal Review of Ratings/Scores      

Selection Report and SSA Slide Prep      

Reviews (SEB Mgr, Legal, Proc Assoc)     5

SSA Presentation Dry Run     1

Revisions and Presentation to SSA     5

SSA Selection     1

Award Review, ANOSCA, Press Release     10

2727