Survey Results & Analysis - Transportation.orgsp.scopt.transportation.org/Documents/SMR Workshop Evaluation... · Survey Results & Analysis . for SMR Workshop Evaluation Tuesday,

Survey Results & Analysis

for

SMR Workshop Evaluation

Tuesday, May 08, 2012 Powered by Vovici EFM

www.vovici.com

Executive Summary

This report contains a detailed statistical analysis of the results to the survey titled SMR Workshop Evaluation. The results analysis includes answers from all respondents who took the survey in the 37 day period from Tuesday, March 13, 2012 to Wednesday, April 18, 2012. 25 completed responses were received to the survey during this time.

P:\MTAP\State Management Reviews\Survey Results May 8 2012 SLE Report.doc Page 1 of 14

http://www.vovici.com/

Survey: SMR Workshop Evaluation Author: AASHTO MTAP Filter: Responses Received: 25

Agency Date of SMR Workshop

you attended Location of SMR

Workshop you attended

New Mexico DOT February 2012 Atlanta, GA

Mississippi Dept of Transportation February 2012 Atlanta, GA

Oregon Department of Transportation March 2012 Chicago, IL

WisDOT March 2012 Chicago, IL

Wisconsin Dept. of Transportation March 2012 Chicago, IL

WI DOT March 2012 Chicago, IL

WisDOT Transit Section March 2012 Chicago, IL

DOT March 2012 Chicago, IL

Illinois Department of Transportation March 2012 Chicago, IL

Michigan Department of Transportation March 2012 Chicago, IL

Minnesota Department of Transportation March 2012 Chicago, IL

Utah Department of Transportation March 2012 Denver, CO

Wyoming DOT March 2012 Denver, CO

Idaho Transportation Department March 2012 Denver, CO

Wyoming Department of Transportation March 2012 Denver, CO

Consultant to UDOT March 2012 Denver, CO

Ks. Dept. of Transportation March 2012 Kansas City, MO

Nebraska Dept. of Roads March 2012 Kansas City, MO

MS Dept of Transportation February 2012 Atlanta, Georgia

NYSDOT ‐ Public Transportation Bureau March 2012 New York City

AZ Department of Transportation March 2012 New York City

NYS Dept of Transportation March 2012 New York City

NHDOT March 2012 New York City

Maryland Transit Administration February 2012 Washington, DC

R.I. Department of Transportation February 2012 Washington, DC


Have You Personnally Partipated in an FTA SMR in the Last Three Years?

15

10

0

2

4

6

8

10

12

14

16

YES NO


SMR PROCESS: How well did this workshop prepare you for the next SMR of your agency in terms of the process of the SMR?

By process, we mean knowing what to expect in terms of the what will happen before, during and after the SMR and what actions your agency will have to take to effectively participate in the process?

SMR CONTENT: How well did this workshop prepare you for the next SMR in terms of the content of the SMR?

By content, we meaning knowing what processes, practices, and procedures of your agency will be reviewed?

How Well Did the Workshop Prepare you for the Next SMR in Terms of Process and Content of the SMR?

Process, 15

Process, 10

Process, 0

Content, 16

Content, 9

Content, 00

2

4

6

8

10

12

14

16

18

Feel Very Well Prepared Feel Somewhat Prepared Do Not Feel Prepared


For PROCESS, why did you give this rating?

I personally have not participated in the entire process of a SMR. I have submitted information when requested, but have not fully participated in the entire process.

The presenters were very informed and delivered the information in a simple format that was easily understood. I also liked the question and answer time.

As far as the process, good information was given.

They were very detailed in what would be expected and gave examples.

I believe the workshop covered each step thoroughly.

I answered very well prepared as they give you each question they will be asking as well as suggested corrective action with an explanation if you need assistance.

Learned a lot of helpful information

This was the first SMR workshop using the new review workbook (it was still in draft form). The consultants were trying out the format of the workshop to see how the schedule fit the time allowed for review.

Trainers had great experience. Reassured me that the process continues as prior years.

Would have been better prepared with questions had the workbook been available online for review prior to training.

The focus was more about what constitutes compliance rather than how to prepare for the review.

I was one of the only ones in the course who hadn't been through an SMR course before and/or hadn't taken the course before, so it glossed over many of the details.

The training instructors discussed the SMR process from beginning to end.

As a relatively new transit employee, I'm still unsure what to expect. I'm sure the training will be very useful to me as we go through this process, but I don't feel "very well" prepared.

I'm well prepared to know what's expected of us. We have since received the questions for our next SMR in May. Our training in Chicago was a 'just in time' training.

I'm new to the position I find any information to be helpful. In addition it was good to meet the good folks that I will be working with during the review.

I don't necessarily know what's going to be important to regional office staff or the reviewers during the reviews.

The book was very informational and explained things pretty well. The instructors were great about highlighting the areas that were being stressed and the new questions.

They basically stated that you need to do everything for every compliance category, but did not give us any sample documents to refer to. They just kept referring to the circulars.


For PROCESS, why did you give this rating?

The content was covered well. The process appears to be a moving target depending on what the hot topics of the day are. This was the first time that we saw the new manual for the SMR since they are not available online yet.

The consultants were willing to make sure all questions were understood and what to expect during the review.

The presentation covered the process.

As a new person seeing the questions that will be asked at least gives me the idea of what steps to prepare for.

It was a very good review of what is required.

For CONTENT, why did you give this rating?

There were a few areas that were questionable...like exactly which requirements pertain to what grant funding program.

The presenters delivered the information in a very understandable manner.

Information given was a little spotty. In the effort to consolidated review areas and questions, it seems that the there some intermingling of triennial subgrantee issues and state issues which at time did not make sense to states.

They were very detailed in what would be expected and gave examples. They also went through the Review Guide which was helpful.

I believe the workshop covered each step thoroughly.

They provide you a list of assessment areas, program specific reviews and project level reviews. They also provided a review sample package as well as resources to assist you.

Content presented in an easy manner to understand

The consultants are behind schedule in preparing for our SMR (June 5 ‐7, 2012). They haven't sent the review package requesting the info we have to submit ahead of the review yet. According to their schedule, they want the info returned to them 4 weeks ahead of their site visit.

FTA presence in workshop was too strong. Intimidated the participants. Not as much frank Q&A as I wished. Some trainers and FTA staff had hard time remembering that it is SMR, not Triennial review. Keep answering questions as if Triennial.

Can't cover all details in 2 days!

As a new employee, I'm still learning about the State's programs. However, I do have a good sense of what is required. This was clear from the presentations.

The training instructors went through each of the review areas thoroughly and described what is to be expected in each area.

The presentation and materials were very detailed and thorough regarding the areas that will be focused on during the review.


For CONTENT, why did you give this rating?

The training covered in great detail the content of a SMR.

Because I am new to the position, I have nothing to compare it to, but I feel I have a good general outline of what to expect.

Workbook is much clearer than previous versions.

Again, the instructors highlighted the areas that are being stressed this year and the questions that are new.

Same as above.

The team did a good job explaining the various topics and the changes in the procedures.

Again, the consultants were willing to make sure all questions were understood and what to expect during the review.

The workshop covered lots of pertinent information.

I am new enough at this that some of the internal processes will have to be researched to see if the comply or if I have additional questions

We are reviewing our State Management Plan to make sure it is consistent with all federal requirements.


SMR AS TECHNICAL ASSISTANCE: Did this workshop serve as technical assistance?

Specifically, did this workshop help you understand the specific FTA expectations and requirements for your agency such that you could you use what you learned from this workshop to go back to your office and make specific changes in the processes, practices, and procedures of your agency so that they will better comply with federal regulations or FTA expectations?

Did this Workshop Work Well as Technical Assistance?

1312

00

2

4

6

8

10

12

14

Worked Well Worked Somewhat Did Not Work

If this workshop did not work well as technical assistance, how could the workshop have been changed to provide you technical assistance?

Again, there were some gray areas but overall, the information provided will be helpful.

Clearer assistance. Since one size does not fit all, there are unique interfering issues with almost every scenario.

It did on some issues but most I was already aware of.


If this workshop did not work well as technical assistance, how could the workshop have been changed to provide you technical assistance?

Step by step examples of processes that are being used by other states that have been successful.

I would have viewed the Technical Assistance aspect higher if there were more time before our actual review. We'll be using all of our time just compiling documentation and responding to questions.

The instructors were great, but again, the course assumed a working knowledge of the process. All I can think of is to break the course into a 3‐4 day "initial" course and a 2‐day "refresher" course. Also, if I had been given specific pre‐homework to do to research how the state manages aspects of certain topic areas; it would have provided a working knowledge for some of the areas.

The training was relatively compressed due to time. There was not the ability to devote much time to digging into a particular compliance area.

Again, as a new employee to the program the material presented can help me shape the processes of my program.

Technical assistance only works as well as the consistency in application once we embark on the SMR.

Did not give concrete examples or samples of what is expected.

All involved were knowledgeable and they offered plenty of opportunity for discussion among the group.

The workshop provided good t/a.

I especially enjoyed the very thorough knowledge of the people there, and meeting the people who will actually come to the state.

If this workshop did not work well as technical assistance, what other methods should FTA use to provide you technical assistance?

I think more samples of forms would have been helpful; i.e. checklists, etc.

Quarterly FTA Regional Office Conference Calls that provide information on an emerging issue. Webinars should be held on review issues.

Would like to have had this training 4 to 6 months before the review. Our review package was sent to us on the final day of training.

I'd like monthly webinars that pick apart and provide best practices for each module. For instance, FTA could pick actual examples from a state (with the state's blessing) to highlight its streamlined methods of compliance in a topic area, or simply run through the NTD, etc.

Again, gave me a good framework. You may get more useful information from the staff with more experience.

Webinars on specific topics as long as sample documents are provided.


UNDERSTANDING OF SPECIFIC TOPIC AREAS: Based on what you learned at the workshop identify which of each of the following areas do you have a high level of understanding/high level of confidence in knowing what content will be covered in the SMR and what the expectations are of your agency and identify which of these your understanding level and confidence level is still low.

You do not need to provide a rating for each one, just those that you would rate as high or low

Based on What you Learned at the Workshop What is Your Level of Understanding/Confidence in the Following Areas?

95.7% 91.7%

62.5%69.6% 66.7%

83.3% 83.3%

70.8%

29.2%

62.5%

92.0%79.2%

69.6% 75.0%

83.3%

70.8% 70.8%

75.0%

76.0%

83.3% 83.3%

64.0% 60.0%

45.8%

0%

20%

40%

60%

80%

100%

Program Management

Grant Administration

Selection and Eligibility

Financial Management

Intercity Bus

RTAP

Satisfactory Continuing Control

Maintenance

Procurement

DBE

Buy America

Debarment and Suspension

Planning and Coordination

Title VI

ADA

Charter Bus

School Bus

Drug Free Workplace

Drug and Alcohol

EEO

NTD

Fare Increases and M

ajor Service Reductions

Safety and Security

ARRA

No Rating/No Response Low Understanding/Confidence Level High Understanding/Confidence Level


Do you have any comments regarding why you selected a high or low rating for the areas above?

The training was very informative and the presenters did an excellent job on explaining each topic.

Some areas just don't apply to me directly so other staff members have a higher understanding. While some topics are handled by the transit agency not the DOT.

Some of these subjects were no longer in the SMR training.

The consultants and the FTA staff in attendance didn't always agree on responses to questions from attendees.

Some areas were covered too quickly. Or with impossible expectations.

Answers related to the program areas in which I specialize.

Poor delivery of the Drug & Alcohol portion of the training.

Again, I just don't have the experience to come out confident after a two‐day course. I could have used another two days at least!

I answered with high ratings, because I could follow the trainers’ comments during the training. I do feel that I understand these areas and would know where to look in the federal circulars or FTA's website if I needed further direction.

Overall I think I received a good understanding of the various aspects of the programs. At the very least I don't expect to be surprised by any part of the review.

Do not have financial management responsibilities so I didn't pay attention. “RTAP” and “Fare increases” weren't discussed much. ARRA is nearly over; we got the timelines.

I'm still somewhat new and haven't fully gotten into everything on this list.

It is one thing to have an understanding of what is expected ‐ it is entirely different to have the required resources available to have confidence that our agency can meet these requirements.

I was not able to attend the entire meeting.


CHANGES TO THE SMR PROCESS: FTA has indicated that the SMR content/process has changed in 2012 and this workshop reflects those changes. How well do you understand how the SMR process has changed?

How Well Do You Understand how the SMR Process has Changed?

8%

40%

52%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Have a Good Sense Have Somewhat of a Sense Do not have a Sense

Why did you choose your rating?

The presenters did an excellent job of explaining the information.

Differences between 5307 subgrantees and States got lost in the consolidation.

They made an effort to point out the changes.

I have been to 2 workshops before the changes and I have been through a SMR.

I understand SMR process was changed from 27 review areas to 13 review areas, but I do not know all of the review that were removed.

Understandable information ‐ well presented

I know how they said it had changed, but I'm not sure I can believe it until we have the review,

Although the trainers talked about more technical assistance and fewer questions the training itself seemed to add detail and increase expectations. Instead of areas eliminated or simplified, areas seem just folded together so not less expectation, but fewer, but more complex questions.


Why did you choose your rating?

Doesn't matter as I was not familiar with the SMR until this training.

The instructor(s) made it clear that there was an effort made to simplify and consolidate the oversight efforts.

This was discussed. Condensed sections together (i.e. procurement and DBE)

This is my first SMR and so I don't have a good understanding of what the process/content was.

I was more focused on what the federal auditors will home in on for our coming audit instead of 'what used to be done' in the past.

Very good considering I'm a new person.

Despite the clarity of the workshop there is no assurance that the regional staff and the reviewers will have the same focus,

My instructors were good about noting the changes and the new things in the manual and in the process. Not having been through it before, I have a basic understanding of the changes.

They seemed to make a big deal about how things were streamlined. It appears that nothing was really streamlined, they just combined compliance areas. In addition, they added new 5 year planning and capital needs requirements.

This was covered in the review thoroughly.

Even though the process has changed the content is still there. FTA just moved it around.

This was explained at the workshop.

I thought it was very clear what was dropped or switched to a new field.

Overall – Any Other Questions, Comments or Concerns?

Very disappointed that the Workshop Manual was not posted on the FTA Website nor did we received a disk copy of it.

I came away feeling really discouraged. It seemed like the state role is impossible. It would cost so much to provide oversight to the level of labor and documentation that is suggested.

I told FTA that these classes should be open to our rural grantees so they get an understanding of what is required of the state and sub‐recipients. This would allow our grantees to see what is required of all of us.

For the past three years I have been preparing files and procedures based upon the 2009 & 2010 SMR workbook. I would be much better prepared for the SMR workshop had I prior knowledge of the new methodology and had time to formulate related questions. My existing documents need to be reviewed and revised to address the SMR as given to us. To assist other agencies involved in our SMR, an electronic copy would be helpful, especially if it was available online.

I'll reiterate how well I think the instructors did. My lack of knowledge/confidence has to do with



Overall – Any Other Questions, Comments or Concerns?

my lack of experience (and competency?!). I think a more thorough course for FTA "newbies" might help such people be more confident in going back to their home state and suggesting actual corrections to processes, etc.

Two out of the three presenters were very strong.

The only issue I had was the room temperature was unexpectedly quite cold, but I was better prepared after our 1st break. The training itself was very good and exceeded expectations.

Overall a very nice job, the presenters did a good job facilitating the group and keeping everyone involved and engaged. Personally I appreciated meeting the staff members.

The circular still has the "old" A thru U categories, yet the new plan covers 13 areas. Looking forward to seeing the circular revised so that there is a better match.

Generated: 5/8/2012 10:50:30 AM