TATC File No. H-4207-27
MoT File No. 2802-306927
TRANSPORTATION APPEAL TRIBUNAL OF CANADA
Patrick John Vaughan, Applicant
- and -
Minister of Transport, Respondent
Aeronautics Act, R.S.C., 1985, c. A-2, paragraph 6.71(1)(b)
Decision: February 16, 2017
Citation: Vaughan v. Canada (Minister of Transport), 2017, TATCE 04 (Review)
Heard in: Montréal, Quebec, on November 30, 2016
REVIEW DETERMINATION AND REASONS
Held: The Minister has proven, on the balance of probabilities, that the applicant did not fulfil the conditions necessary for the issuance of a Canadian aviation document as per paragraph 6.71(1)(b) of the Aeronautics Act.
 On February 1, 2016, the Minister of Transport (Minister) issued a Notice of Refusal to Issue or Amend a Canadian Aviation Document (Notice) to the applicant, Patrick John Vaughan, pursuant to section 6.71(1)(b) of the Aeronautics Act, R.S.C., 1985, c. A‑2. According to the Notice, the refusal was based on the fact that Mr. Vaughan did not meet the qualifications or conditions necessary for the issuance of a Line Operational Evaluation (LOE) on an Airbus 320 aircraft. Specifically, Mr. Vaughan's flight test of January 28, 2016 was assessed as unsatisfactory due to unsuccessful initial and repeat attempts of one of the LOE's event sets.
 On February 10, 2016, the applicant requested a review of the Minister's decision by the Transportation Appeal Tribunal of Canada (Tribunal).
II. STATUTES, REGULATIONS AND RELEVANT STANDARDS
 Subsection 6.71(1) of the Aeronautics Act reads in part as follows:
6.71(1) The Minister may refuse to issue or amend a Canadian aviation document on the grounds that
(b) the applicant or any aircraft, aerodrome, airport or other facility in respect of which the application is made does not meet the qualifications or fulfil the conditions necessary for the issuance or amendment of the document; […]
 Subsection 705.106(1) of the Canadian Aviation Regulations, SOR/96‑433 (CARs), reads in part as follows:
705.106 (1) Subject to subsection (3), no air operator shall permit a person to act and no person shall act as the pilot-in-command, second-in-command or cruise relief pilot of an aircraft unless the person
(e) has fulfilled the requirements of the air operator's training program.
 The following excerpts from Transport Canada's Advanced Qualification Program Evaluator Manual, TP 14672, are relevant to the case:
CHAPTER 1 – AQP EVALUATOR PROGRAM (AQPEP)
1.1 PROGRAM DESCRIPTION
1.1.1 The AQP Evaluator Program (AQPEP) allows an air operator the opportunity to develop and maintain a program of AQP Validations and Evaluations independent of the availability of Transport Canada Civil Aviation Inspectors (hereafter referred to as TC Inspectors).
1.1.2 The AQPEP consists of AQP Evaluators (hereafter referred to as “evaluators”), who have been delegated the authority to conduct AQP Validations (hereafter referred to as “validations”) and/or AQP Evaluations (hereafter referred to as “evaluations”) on behalf of the Minister. […]
2.3 TYPE E EVALUATOR
2.3.1 A Type E Evaluator is a person who is authorized by the Minister to administer and conduct Line Operational Evaluations (LOE), Manoeuvres Validations (MV) First-Look Manoeuvres (FLM) and Online Evaluations (OE).
8.10 LINE OPERATIONAL EVALUATION (LOE)
8.10.1 The LOE is the primary mode of proficiency evaluation. […]
8.10.4 The LOE addresses the individual's ability to demonstrate technical and CRM skills appropriate to fulfilling job requirements in a full mission scenario environment. The intent of a LOE is to evaluate and verify that an individual's job knowledge, technical skills, and CRM skills are commensurate with AQP qualification standards. For the Qualification Curriculum (QC), the LOE is also used to verify that the individual is qualified to begin the Initial Operating Experience (IOE) portion of the Qualification Course.
8.10.5 LOEs are graded at the event set level. A LOE consists of a minimum of 8 events sets. During the LOE, an individual event set graded as unsatisfactory may be repeated. Two repeats are allowed for each candidate. No single event set can be repeated more than once. […]
8.10.7 The LOE is considered a jeopardy event and a failure is reported to TC. In the event of a failure, the entire copy of the candidate's LOE report is faxed to Transport Canada for licensing action (i.e. suspension).
8.10.9 A LOE shall be conducted according to the following protocol:
(b) The evaluator shall conduct a briefing in accordance with the procedures and protocols established in the air operator's AQP and PADB documentation, as applicable;
(c) A LOE is normally comprised of 8 to 11 (with a minimum of 8) event sets and usually starts at the flight planning/dispatch stage and ends at the gate after the parking checklist is completed;
(d) Both technical and CRM topics are evaluated during each event set;
(e) PNF duties will be validated in conjunction with PF duties; […]
9.3 FLIGHT CREW CONCEPT
9.3.1 Validations and evaluations on multi-crew aircraft shall be conducted under the flight crew concept and not on an individual basis. (This does not apply to SKVs, which are individual assessments of knowledge.)
9.3.2 During a validation or evaluation, a manoeuvre or event set may involve duties and/or responsibilities for crewmembers other than the pilot flying (PF). A sequence that is graded as “unsatisfactory” for the PF may, due to inappropriate action on the part of other crewmembers (i.e., the pilot not flying [PNF]), be rated as “unsatisfactory” for the PNF also. In such a case, it is possible that an assessment of “unsatisfactory” may be given to more than one crewmember involved in the same flight sequence.
9.5 PRE-FLIGHT BRIEFING – VALIDATION OR EVALUATION CONDUCTED IN A SIMULATOR
9.5.2 The briefing for a validation/evaluation conducted in a simulator shall include or state:
(g) the requirement for the candidate to demonstrate any normal or emergency procedure applicable to the aircraft and that the candidate's technical performance will be assessed in accordance with the air operator's approved qualification standards […]
(j) that normal crew coordination is expected in accordance with the aircraft AOM/AFM or company SOPs as applicable, and that an emergency situation caused by an incorrect or inappropriate action or response on the part of the candidate will not be corrected by the evaluator;
(k) that multiple, unrelated failures will not be required, but the candidate must be prepared to take corrective action on related failures (ex. loss of hydraulics or electrical supply due to a failed engine);
9.8 GENERAL ASSESSMENT “FAILED”
9.8.1 A LOE will receive a General Assessment of "Failed", if:
a) the candidate's initial attempt and repeat of any event set are both unsuccessful; or
b) the candidate is unsuccessful on the initial attempt of three separate event sets.
10.7 VALIDATIONS/EVALUATIONS - GENERAL
10.7.1 To evaluate the overall technical proficiency, communications skills, leadership and situational awareness of pilots with respect to normal and abnormal procedures, evaluators must observe the performance of each crew closely. To evaluate specific items listed in the Qualification Standards, the applicable validation/evaluation shall be conducted in a manner that enables the pilots to demonstrate knowledge and skill with respect to such things as aircraft automation including FMS/RNAV programming, auto flight systems and flight mode awareness, pilot not flying (PNF) duties, crew coordination and pilot decision making.
10.7.2 When assessing normal procedures, the evaluator must ensure the crew demonstrates adequate knowledge of the company SOPs and aircraft systems to confirm their ability to use installed equipment properly. […]
10.39.2 Multiple, unrelated failures that have a cumulative effect on the operation of the aircraft must not be planned as part of a validation or evaluation scenario. […]
(1) Michel Paré
 Michel Paré has been employed by Transport Canada in various roles over the last 14 years, including in the Standards and Surveillance departments. His present role is one of oversight in Quality Assurance and this includes oversight responsibilities with large air operators such as Air Canada.
 One of his initial responsibilities within Transport Canada dates back to 2005, when his department completed the regulatory framework for the Advanced Qualification Program (AQP). The witness went on to clarify that although the AQP has not yet been introduced in the CARs, it was introduced as a regulatory exception program and implemented and used by Air Canada. No other operator has this exemption at this time.
 Mr. Paré testified that the AQP is an alternative training and checking program that is used to train and qualify pilots and examiners. He explained that as opposed to the traditional pilot proficiency check (PPC) program, the AQP allows the operator to develop a training program that reflects more realistically its own operational issues and concerns and helps it improve crew performance and safety by measuring both technical and non-technical skills such as workload management, communication, leadership and decision making.
 Mr. Paré also testified that the progressive stages of training and checking under the AQP approach culminate in a Line Operational Evaluation (LOE). LOEs are highly scripted to a very precise detail, allowing the crew a number of predetermined outcomes to each scenario.
 The witness clarified that Transport Canada, as the aviation regulator, sets the standards, guiding documents and policies governing the AQP program. It also validates the program through surveillance and monitoring and delegates the examiners that conduct the various checks and evaluation events for Air Canada. Any unsatisfactory performance by a pilot must be reported to Transport Canada using form 26-0249, Flight Test Report Pilot Proficiency Check (Exhibit M‑1). This form then results in a Notice of Refusal to Issue or Amend a Canadian Aviation Document being sent to the affected pilot (Exhibit M‑2).
 Finally, Mr. Paré introduced the Advanced Qualification Program Evaluator Manual, TP 14672, a Transport Canada publication that is publicly available (Exhibit M‑3).
 On cross-examination, Mr. Paré was asked if he was involved in the development and monitoring of the AQP program at Air Canada. He responded that at the time he had been involved. He was asked whether the check pilots are tested or questioned as to their understanding and knowledge of the publication being used. He responded that they should be well versed in the policies of the AQP.
 When asked if the AQP evaluators are evaluated, tested or trained annually, Mr. Paré responded that he had not personally conducted enough of these tasks to properly answer the question. When asked if he was involved in the development of the scripts used by AQP evaluators, he responded that the extent of Transport Canada's involvement in that regard is to conduct quality assurance as part of its oversight function.
 Mr. Paré was asked to provide specific examples of what quality assurance would consist of in relation to a script. Again re-iterating that he was not part of the oversight team assigned to Air Canada, he explained that the role of Transport Canada is to ensure the script follows the guidelines in the AQP manual. For that to be the case, among other requirements, a script must be realistic, allow the evaluation of technical and non-technical skills and contain the required number of events.
 He also testified that there is a policy document that explains how to implement a script. Mr. Paré was asked specifically whether a check pilot who believes the script “does not make sense” would have to adhere to the script provided or could deviate or apply another one. He responded that the check pilot is allowed very little discretion. He went on to clarify that if the AQP evaluator believes the script to be flawed or unrealistic, then this should be reported to the company. He emphasized that this should not happen as scripts are developed based on data analysis by the Air Canada team and then passed on to Transport Canada for review and acceptance.
 Mr. Paré was asked if he was familiar with the script that was used during the LOE in the case at hand and the requirement for a script's scenario not to have multiple unrelated failures. He responded that he was not familiar with the script used on that particular evaluation day but did add that operators are encouraged to script scenarios as much as possible.
(2) Kryn Riemersma
 Mr. Riemersma is an approved AQP evaluator and line captain for Air Canada. On behalf of Transport Canada, he is authorized to exercise Type E AQP evaluator functions in the conduct of several types of evaluations, including LOEs (Exhibit M‑4). He has had this endorsement with respect to Airbus 320 aircraft since June of 2009.
 He explained that he also has a Type O delegation, which entitles him to conduct online evaluations (OEs) as a crewmember from either seat onboard the aircraft. As a line pilot, he explained the he is subject to all the standard recurrent training and checking, including simulator training and evaluation. He added that he is also subject to an annual course called the Inter-Rater Reliability, which is meant to ensure all evaluators are grading to the same AQP standard. Finally, he undergoes annual monitoring by Transport Canada while he is conducting an evaluation ride in order to maintain his delegation.
 Mr. Riemersma testified that he had undergone one such Evaluator Monitor by Transport Canada (Exhibit M‑4) on October 27, 2015, which was graded as satisfactory. In total he estimated that, as an evaluator, he had conducted 215 LOEs in full flight simulators as well as close to 300 OEs online in aircraft. He went on to explain that he had conducted an OE on Mr. Vaughan in June of 2015, which had been assessed as satisfactory.
 He also defined for the Tribunal what a LOE is. In short, he testified that a LOE is a scripted simulator session, conducted in such a fashion so as to emulate an actual everyday flight from start to end. This allows the evaluator to properly assess the candidates or candidate on their technical knowledge as well as their crew resource management (CRM) abilities in ensuring the safe completion of the flight. The crew is met with at the beginning of the session, documents are verified, the flight plan is provided and the briefing explaining the session is given.
 The actual LOE is composed of a series of scripted events, anywhere from eight to eleven. Twice during the evaluation, an event can be repeated if the grade is unsatisfactory. At the end of the session, the crew is advised if the LOE was satisfactory or unsatisfactory, unless it was terminated prior to reaching the end. This can occur for various reasons such as a crash of the simulator, a repeated event being assessed as unsatisfactory, a third initial unsatisfactory score or an assessment by the evaluator that the safety of the flight was jeopardized.
 In the event of an unsatisfactory grade, a 0249 form is required to be filled out with an explanation and a reason code. This is sent to the air operator's chief pilot, who then forwards it to Transport Canada.
 Mr. Riemersma introduced Script 23 REV 1.8, and testified that this was the script used during Mr. Vaughan's evaluation check ride of January 28, 2016 (Exhibit M‑5). He detailed the top section of the script, pointing out that this mentions it is approved for various crew complements: either captain/first officer, captain/captain or first officer/first officer. The script follows a daytime, summer scenario in which the crew conducts a flight between Toronto and Montréal, with a stop in Ottawa. The first leg of the flight is operated by one pilot as the pilot flying and the other as the pilot not flying (or “pilot monitoring”) and then the roles are reversed for the second leg.
 The script contains a Minimum Equipment List (MEL) Release item that is given to the crew prior to departure. This specifies that FMGC 1 (the aircraft's primary flight management and guidance computer) is inoperative but the crew is allowed to dispatch. In the scripted scenario, the second system, FMCG 2, fails at approximately 15,000 ft. At that time the crew is expected to deal with the situation in accordance with the Aircraft Operating Manual and the Flight Operations Manual.
 On January 28, 2016, during the pre-simulator briefing, the crew was asked if they had any questions and, to Mr. Riemersma's recollection, none were raised. He briefed the crew on the importance of error mitigation, flight crew concept, CRM and communication between pilots. He did recall the crew mentioning that the crew composition of captain/captain was not ideal but he testified that he had briefed clearly that the captain sitting in the right seat as pilot monitoring would not be evaluated on his ability to fly as a normal first officer but rather on his pilot monitoring abilities.
 He further explained that on events number 1 through 6, the crew was evaluated as satisfactory. At event number 7, FMGC 2 failed, as per the script. The crew was quite surprised by the resulting malfunctions and the pilot flying had to resort to manually flying the aircraft. This was done poorly with the pilot flying over-controlling the aircraft. The pilot monitoring seemed disconcerted with this. Without consulting the pilot flying, Mr. Vaughan, as the pilot monitoring, interrupted the electronic centralized aircraft monitor (ECAM), made a “PAN, PAN, PAN” call and requested a level-off at 21,000 ft. from air traffic control.
 A dual FMGC failure results in the loss of various systems, yet the communication between the pilots was very minimal with regard to what was occurring and why. No considerations were discussed regarding the failure of the second FMGC or its ramifications for the approach into Ottawa. Poor situational awareness and poor CRM skills were evident as far as ensuring the aircraft was properly backed up.
 The pilot flying over-controlled the aircraft during the approach. The pilot monitoring, Mr. Vaughan, made no calls with regard to the final approach fix or identification of crossings, or any cross-checks on altitudes as required by the Flight Operations Manual and the Aircraft Operating Manual. Mr. Riemersma did recall that the pilot monitoring made some calls in relation to the glide slope and speed. At approximately 500 ft., the pilot flying recognized that no approach speeds were set up or calculated so the crew randomly made one up. As they continued to approximately 150 ft., the pilot flying continued to have difficulty in maintaining any stable approach criteria and called a go-around. At no time did the pilot monitoring make any of the required “Unstabilized” calls.
 Mr. Riemersma explained that on the second approach the pilot flying asked the pilot monitoring to calculate the approach speed but he was unable to. The pilot flying finally calculated it himself. Overall he observed that on the second approach the pilot monitoring made minimal calls, did not identify the final approach fix or make any cross-check calls. He made calls to the effect of “watch vertical speed”, “watch your glide slope” but did not make the proper call of “Unstabilized” which is required with an unstabilized approach. Finally, with the pilot flying continuing to fly erratically, the pilot monitoring threw up his arms in a frustrated manner and no further calls were made until the aircraft was on the runway.
 Mr. Riemersma explained that even during the rollout, the pilot monitoring made no calls as to the aircraft landing long, using heavy braking and the autobrake not being available. In his opinion, safety of flight was jeopardized, and the Aircraft Operating Manual and Flight Operations Manual requirements were not met. He advised the crew at that time that the LOE was being terminated and it was being assessed as unsatisfactory.
 He went on to explain that prior to meeting the crew for debriefing he encountered Mr. Jeff Oliver of Transport Canada in the facility. He informed him that he had just assessed a LOE as unsatisfactory and needed his advice as to what and how the paperwork should be filled out, wanting to make sure he was relying on the correct terminal proficiency objective terminology as per the Precision Approach Qualification Standard (Exhibit M‑6). The comments were then added to the 0249 evaluation form (Exhibit M‑1).
 Mr. Riemersma explained the requirements from the Flight Operations Manual (Exhibit M‑7) and Aircraft Operating Manual (Exhibit M‑8) as to the need for the crew to use the correct checklists, procedures, standard calls and feedback to ensure performance targets are met. This includes the procedures to be followed by the crew, and the pilot monitoring in particular, at the final approach fix crossing.
 On cross-examination, Mr. Riemersma was asked to confirm that the licensing event involved two captains and that they were informed that this would be a jeopardy event for both should they fail. He responded that both pilots were told that this was a licensing event and that the captain sitting in the right seat would be assessed and evaluated on pilot monitoring duties and not on the actual flying of the aircraft.
 The applicant's representative, Mr. Marc Delorme, asked the witness if he was familiar with the provision in the Air Canada pilots' collective agreement that mentions that if a captain/captain or first officer/first officer pairing occurs in training, the event is seen as a non-jeopardy event. Mr. Riemersma responded that he was unfamiliar with this clause. Furthermore, he stated that to a Transport Canada-delegated AQP evaluator, this is a non-relevant issue as it pertains to the relationship between the pilots and the employer.
 The witness again reiterated in questioning by Mr. Delorme that the crewmembers were properly advised that this was a licensing event. He was asked whether a defective intercom switch in the simulator at the time of the ride (Exhibit A‑1, tab 5, p. 79) could have had an effect on the communication between pilots. He responded that the intercom switch would have had no effect because communications could be heard from all headsets including his own, which is considered standard procedure in the simulator environment.
 Specifically concerning the script used that day, Mr. Riemersma was asked if the scenario was valid as per the AQP manual with regard to possible errors such as having unrelated multiple failures. He stated that the script used was presented to him as being already approved and he saw no issue with using it.
 A discussion ensued regarding a letter dated March 2016 by the witness addressed to Mr. Tom Kuilder of Transport Canada. Mr. Riemersma explained that the letter contained details from his notes of the unsatisfactory check ride he had conducted on Mr. Vaughan (Exhibit A‑1, p. 68–72). He stated that even several months after the event, he recalled the ride as the crew's performance was, in his words, “shocking”. He also confirmed that he had personally flown with Mr. Vaughan previously during an OE and that his performance then had been satisfactory.
 On continued cross-examination, Mr. Riemersma was asked to explain his markings and evaluation grades of various events during the ride (Exhibit A‑1, p. 61–64), in particular with regard to point 6, pertaining to Monitoring and Feedback, for which he had changed the grade from a 3 to a 2. He explained that he had initially graded the event as a 3 but at a certain point as the event unfolded and he observed the entire event, he had elected to grade it as a 2. He acknowledged that he did intervene during an event in which Mr. Vaughan had become agitated with regard to attempting to tune a navigational aid. He decided that this could lead to a possible distraction with the candidate wondering whether it was a simulator-induced fault or not. He had told Mr. Vaughan to disregard the issue in order to maintain the integrity of the LOE.
 Mr. Riemersma was asked if Mr. Vaughan, as right-seat pilot monitoring, in a situation where the other pilot was overwhelmed, might have been frustrated because he was not trained in the position and simply trying to prevent the other pilot from crashing the aircraft. He replied that Mr. Vaughan was hampered by a lack of knowledge in being able to tune a navigational aid and that he had provided minimal feedback to the captain flying and no monitoring.
 On continued cross-examination, Mr. Riemersma was asked to explain what happened after the unsatisfactory check ride and elaborate on the encounter he had with an inspector from Transport Canada immediately following the termination of the LOE. He testified that his encounter with Mr. Oliver of Transport Canada was coincidental and he had simply asked for his advice as to the correct terminology to use in one part of the flight test report. He had then proceeded to the debrief room and explained to the crew the unsatisfactory grade for event number 7. He had offered them a copy of the 0249 form but they had both chosen to receive it by mail.
 In re-examination, Mr. Riemersma was asked if he was involved in developing the scripts used under the AQP program. He responded that he was not.
 Three witnesses provided evidence on behalf of the applicant: Mr. Gary James, an expert witness on the AQP program; the applicant himself; and Mr. Daniel Comeau, who provided the Air Canada Pilots Association's perspective on the collective agreement.
(1) Gary James
 Mr. James is an Air Canada pilot with an extensive training background that includes the Centre québécois de formation aéronautique college program, Transport Canada and the military. He has experience as a chief pilot, a training pilot and a check airman. His credentials include involvement as an inspector for Transport Canada, and he has conducted numerous PPCs as a check airman for Air Canada. He was also involved in the development and implementation of the AQP transition program at Air Canada. This included developing a training program in which conventional approved check pilots would transition to evaluators, validators and instructors under the AQP program. Both the Tribunal and Transport Canada accepted his status as an expert witness.
 Mr. James introduced various reference materials to the Tribunal. His role in explaining the main differences between conventional PPCs and LOEs to the Tribunal was important. An AQP program's main objective is to expose crews to scenarios that emulate real-world experiences in simulator training and evaluation modules. In essence, a LOE takes a realistic uninterrupted flight scenario and allows the crew to handle events, mitigate the issues that are introduced and make the decisions that ensure a safe outcome. The witness pointed out that a LOE is similar to a line-oriented flight training (LOFT) scenario but that a LOFT is a non-jeopardy event while a LOE is a jeopardy event.
 He emphasized that Transport Canada did not invent the AQP program but rather relied on the United States Federal Aviation Administration Advisory Circular for guidance and reference material. In this documentation, little is available to address improper crew complements such as captain/captain. The witness further explained that a key function of the LOE under the AQP is to have a proper crew complement in order to emulate normal line flying.
 Mr. James explained in more detail the various events that take place during a standard AQP recurrent training and evaluation session. This involves a Manoeuvres Training Validation (MTV) on day one followed by the LOE on day two, both of which are jeopardy events. As to the scripts used during the LOE, he stated that these were created in-house at Air Canada and then sent to Transport Canada for oversight. Although Transport Canada does not approve them per se, they ensure that they meet the program criteria before sending them back to Air Canada. The evaluator then receives a couple of versions of the script and can choose between one or another but does not modify the content in any way.
 Finally, Mr. James clarified that the form being used during the LOE is the 0249 form, which dates back several years and is the same one used for conventional PPCs. It is used for the LOE as a stopgap measure until a proper form is developed by Transport Canada. He concluded by saying that the responsibility of the AQP evaluator and that of a traditional approved check pilot are no different. Both must understand the content of their respective manuals and ensure pilots “train as they fly and fly as they train”.
(2) Patrick Vaughan
 Mr. Vaughan is an Airbus 320 captain and has been on this aircraft since 2007 after having transitioned from the DC-9 and Being 767. He was also a training captain for four or five years on the Airbus 320 aircraft. He testified that from 1989 through January 28, 2016, he had no training failures on record (Exhibits A‑1 and A‑2, tab 2).
 Introducing his original complaint letter addressed to Transport Canada, dated February 26, 2016 (Exhibit A‑4, tab 1, p. 3–4), the witness explained that although some colleagues were sometimes paired captain/captain, he had never himself been paired with another captain in training. His normal crew complement consisted of a captain and a first officer. He testified that his understanding of the captain/captain crewing, as on the day of the LOE, was that he would be acting as a seat filler only as per the collective agreement in effect at Air Canada (Exhibit A‑5, Tab 3). His only time in a right seat role was as a training captain and that would be after an extensive training program.
 Mr. Vaughan testified that the evaluator, Mr. Riemersma, had never explained to him during the briefing that he would be placed in jeopardy from the right seat. He again reiterated that he had not been trained for his role in the right seat and would never have agreed to do the ride had he known this was a jeopardy event for him from this crew seat.
 As to the events on the day of the LOE, he was unaware that he had failed. He stated that it was only a few days later, when he called Mr. Scott Fines, a standards check pilot at the company, that he realized that the LOE was deemed to be unsatisfactory. Even then, Mr. Fines had explained to him that the unsatisfactory grade was an internal issue and that the results would not be sent to Transport Canada.
 Commenting on an email exchange between Mr. Riemersma and Mr. Kuilder of Transport Canada (Exhibit A‑6, tab 5), Mr. Vaughan stated that during the LOE, a double FMGC failure resulted in the aircraft basically flying with little automation or guidance. This resulted in the left-seat captain being overwhelmed and just trying to keep the aircraft from crashing. Mr. Vaughan explained that his primary goal on the first approach was simply to try to help him do that. On the second approach, the left-seat captain did a better job of controlling the aircraft and according to Mr. Vaughan, “at 200 ft. above the ground…the ride is basically done”.
 Mr. Vaughan stated that the 0249 form was never provided to him and that he had to request it from Air Canada. He again testified that the evaluator should have been aware that under the collective agreement this was a non-jeopardy ride for him from the right seat as he was not trained to occupy this seat. Lastly, he reiterated that his primary goal during the LOE was to keep the left-seat pilot from crashing the aircraft.
 On cross-examination, Mr. Vaughan was asked if a LOE was a jeopardy event. He stated that it was, but under a proper crew complement. He had not asked any questions during the briefing since this had been his assumption. He was also asked if during the previous day's event, the MTV, he had been paired with the other captain, and if this was a jeopardy event. He responded to both questions in the affirmative.
 When asked to explain the difference between the two events, Mr. Vaughan explained that the MTV was equivalent to a PPC while the LOE was more of a full-mission scenario.
 On continued cross-examination, Mr. Vaughan was asked to explain the content of an email he had sent on February 1, 2016 to several individuals in which he voiced concerns in regards to the LOE. Mr. Vaughan did recall the email. He was asked why this email only made reference to the issue of the crew complement and not of his role as a seat filler. He again responded that when he had sent the email, he was still under the belief it was a non-jeopardy event. Only when he received the Notice from Transport Canada did he realize that this was a jeopardy event.
 Mr. Vaughan was asked if he had been properly briefed regarding his role as pilot monitoring and advised his CRM skills would be evaluated during the LOE. He responded that the briefing given was a standard briefing and that his role was one of aiding his left-seat colleague during his portion of the LOE. He agreed that the captain in the left seat did require extra training. He went on to state that the event was frustrating to him as he was occupying a seat that he was not qualified for. His role as he saw it was basically to let the left seat pilot initiate calls. His primary role as pilot monitoring became one of trying to prevent the left-seat pilot from crashing the simulator.
 The Minster's representative asked Mr. Vaughan why as pilot monitoring he had not made any of the mandatory calls. He responded that the pilot flying should have been making the initial calls, such as the final approach fix call. Since the pilot flying was not making any calls and was challenged just to fly the aircraft under the condition it was in, his own role had become simply to monitor the flight path. In his view, this would supersede all Standard Operating Procedures (SOPs).
(3) Daniel Comeau
 Mr. Comeau is the Chair of the Air Canada Pilots Association Local of Montreal. He testified that as far as the union was concerned, Mr. Vaughan's role during the failed check ride was that of a seat filler and under the collective agreement this was to have been a non-jeopardy event for him. Therefore, Transport Canada should not have been notified.
 The Minister's representative submitted that the issue between Air Canada and the Pilots Association relating to the wording in the collective agreement as to whether the LOE at issue was a jeopardy event or not is irrelevant to Transport Canada since it is not bound by any agreement between an employer and employee with regard to a licensing event.
 The Minister's representative argued that if a candidate is undergoing a PPC or LOE evaluation, Transport Canada must be advised of any failure to demonstrate the standard required to maintain the licence. In respect to event number 7 in the LOE script, the evidence presented clearly demonstrated that Mr. Vaughan did not meet the standard. Furthermore, no evidence from the applicant was heard to dispute this.
 As to the crew pairing, the script is approved for captain/captain, captain/first officer and first officer/first officer pairings. The AQP Evaluator Manual (Exhibit M‑3) states that a LOE is a jeopardy event and that a failure must be reported to Transport Canada. The same manual also allows the evaluator to intervene if required in order to allow the candidate the opportunity to demonstrate his abilities.
 The applicant's representative argued that this LOE was not a jeopardy event under the collective agreement and in view of the fact that Mr. Vaughan was occupying a seat that he was not trained on. He further submitted that based on the evidence and testimony presented, the script used that day was invalid as it contained multiple unrelated failures, which goes against the AQP guidelines.
 He argued that the check pilot, as the Minister's delegate, should have exercised discretion in regard to the evaluation he was conducting and opted not to go ahead with the script as it did not follow the program's requirements. Finally, the applicant's representative submitted that the script was not reviewed properly by Transport Canada given that it contains multiple unrelated failures.
 Various statements made by Mr. Vaughan should be dismissed at the outset. He suggested that by 200 feet prior to landing, the evaluation ride is “basically done”. This is erroneous as the evaluation assesses behaviour up to, during, and after the landing. He also provided that, at a certain point, his role was to monitor the flight path, which supersedes all SOPs. Regardless, the role of pilot monitoring includes making the required calls, which he failed to do sufficiently.
 The Minister refused to issue a Canadian aviation document to Mr. Vaughan on the grounds that he did not meet the required standard for an LOE due to two failed attempts at a scripted event during an evaluation ride. The standard of proof imposed on the Minister as per the Transportation Appeal Tribunal of Canada Act, S.C. 2001, c. 29, subsection 15(5) is “proof on the balance of probabilities”. I believe the Minister has met this obligation. To reach this conclusion, however, three specific questions needed to be answered in order to determine whether the applicant fulfilled the requirements for holding a Canadian aviation document.
1- Was the LOE event conducted on January 28, 2016 a jeopardy event for Mr. Vaughan?
2- Could Mr. Vaughan be evaluated on his pilot monitoring abilities from the right seat during the LOE event?
3- Did the script used in the LOE meet the required standards and approvals as per the AQP Evaluator Manual and guidelines?
(1) Question 1
 The critical matter of whether the LOE of January 28, 2016 was a jeopardy event for Mr. Vaughan lies at the center of the review. Throughout the hearing, the applicant's representative and Mr. Vaughan argued that the LOE was a non-jeopardy event for him based on two factors. First, the applicant invoked a provision of the collective agreement, and second, his alleged role that day was that of a seat filler and hence, as per Transport Canada's policy outlined in ACP/AQP Bulletin 01/12 on the status of seat substitutes during PPCs and LOEs (Exhibit A‑7, tab 9), it was not a jeopardy event. Both these points must be addressed, but before going any further, the LOE must be properly defined.
 The AQP Evaluator Manual (Exhibit M‑3) provides the following definition:
LINE OPERATIONAL EVALUATION (LOE): A proficiency evaluation conducted by a qualified evaluator in an approved simulation device that addresses an individual's ability to demonstrate technical and Crew Resource Management (CRM) skills appropriate to the job requirements in a full mission scenario environment.
 Mr. Riemersma was a qualified evaluator (Exhibit M‑4) conducting the LOE in an approved simulator (simulator number 471, as per Exhibit M‑1). An AQP evaluator, acting as a delegate of the Minister as per subsection 4.3(1) of the Aeronautics Act, must apply the same regulatory criteria to candidates.
 The foreword to the AQP Evaluator Manual states:
This manual contains the policies, procedures and guidelines that pertain to the Advanced Qualification Program (AQP) Evaluators. It is published for use by Transport Canada Inspectors, air operators and AQP Evaluators.
AQP Evaluators are authorized to conduct AQP Validations and Evaluations on behalf of Transport Canada. They receive their authority and are approved by the Regional Managers, Commercial and Business Aviation (RMCBA) or the Chief, Airline Inspection.
When performing their duties, AQP Evaluators are first and foremost acting as delegates of the Minister according to subsection 4.3(1) of the Aeronautics Act thus it is imperative that the policies and procedures specified in this manual be adhered to.
 Testimony heard from Mr. Riemersma was credible. A briefing did occur prior to the event explaining that this was a jeopardy event. Mr. Vaughan did not contradict this fact, although he reported that some questions were raised as to the crew complement. Mr. Vaughan also testified that he had not asked about the jeopardy factor for the ride as he had assumed it to be non-jeopardy. The AQP Evaluator Manual at section 8.10.7 clearly states that a LOE is a jeopardy event and that a failure will be reported to Transport Canada. Section 8.10.9(5) iterates that both pilot not flying and pilot flying duties will be validated.
 The Transport Canada document entitled Development and Implementation of an Advanced Qualification Program (Exhibit A‑8, tab 15), at page 86, section D, again clearly spells out that the LOE event is considered a jeopardy event with any failure having to be reported to Transport Canada.
 Finally, Mr. James, testifying as an expert witness for the applicant on the AQP program also confirmed the fact that a LOE is a jeopardy event. There is little doubt that this was a jeopardy event for Mr. Vaughan, as was the event held on the previous day. Testimony heard by both Mr. Vaughan and Mr. Riemersma confirmed that the LOE was being conducted as part of the standard AQP recurrent training. Mr. Vaughan testified that on the previous day he had undergone the first phase of the AQP and had successfully completed the Manoeuvres Training Validation (MTV), which was a jeopardy event.
 The applicant's argument that the LOE was not a jeopardy event for Mr. Vaughan centered on a clause of the collective agreement between Air Canada and the Pilots Association. Based on testimony from Mr. James and on the evidence submitted, in particular Exhibits M‑3 and A‑8, the AQP clearly falls under the policies and guidelines of Transport Canada and any agreement between employer and employee does not supersede licensing requirements. The LOE held on January 28, 2016 was without doubt a licencing event conducted by an approved evaluator using an approved script in an approved simulator device. The licence and its privileges are issued, maintained and governed by Transport Canada and do not belong to any air operator, union or individual.
 The collective agreement regulates the terms and conditions of employees in their workplace, and in return the duties of the employer. It cannot determine whether a LOE is a jeopardy event because the pilot licensing authority belongs exclusively to Transport Canada with respect to issuing and renewing licence privileges. As is clearly stated in each pilot's aviation document, “…all licences, permits and medical certificates are issued under the authority of the Aeronautics Act and the Canadian Aviation Regulations”.
 The second factor raised was the understanding or assumption by Mr. Vaughan that his role on the LOE that day was that of a seat filler. If this were accurate, then I would agree with him that this would be a non-jeopardy event. I note, however, that the evidence provided by Mr. Riemersma and to a certain extent that of Mr. Vaughan contradicts this position. The LOE session conducted on January 28, 2016 was a continuation of a recurrent training schedule for both pilots involved.
 The AQP Evaluator Manual defines a seat filler as “a qualified crewmember who substitutes for a candidate who is unable to attend an evaluation session, thus allowing the rest of that candidate's crew to complete their evaluation with a full crew complement”. Was this Mr. Vaughan's role on January 28, 2016?
 Mr. Vaughan testified that he was undergoing his recurrent training session at the time. He had already completed his MTV event successfully on the previous day (Exhibit A‑3, tab 2) and was scheduled to complete the recurrent session with a LOE. On that day, he was not acting as a substitute but as one of two crewmembers scheduled to complete their respective LOE. The fact that the crew was composed of two captains was permissible and allowed. As explained by Mr. Riemersma, one captain would be evaluated as the pilot flying while the other captain would be evaluated as the pilot not flying (or pilot monitoring) during the first leg and the roles would be switched for the second leg. This constitutes a standard training practice and an approved crewing with respect to the particular script used that day.
 Consequently, I find that the LOE event held on January 28, 2016 was a jeopardy event for both pilots, properly administered and sanctioned under the AQP program. I also note that Mr. Vaughan did not meet the definition of a seat filler on that occasion.
(2) Question 2
 The applicant argued that he was not trained to occupy his right seat position during the LOE event. Leaving aside the issue of the pilot being seated right or left, I find that the critical factor here is to properly understand the role of the pilot monitoring. The operative words used in all documentation relating to AQP are “Crew” and “CRM”. There is no dispute that a LOE is based on the ideal crew concept scenario of a captain/first officer pairing.
 However, the issue is whether the crew composition that day was an approved crew complement and one that would meet the criteria applicable to a LOE. I find that the oral and documentary evidence submitted support that a captain/captain crew would meet these criteria. The LOE session of January 28, 2016 was conducted using Script 23 REV 1.8. The various crew complements accepted for this script by both Air Canada and Transport Canada are captain/first officer, captain/captain and first officer/first officer.
 It is clearly understood that a first officer/first officer crew complement would not operate in an actual flight, but would a captain/captain crew operate in a normal line flight and thus require the right-seat captain to function as a pilot monitoring? Based on oral testimony from both the applicant and the AQP evaluator, this is a crew complement that does operate in actual line operations.
 Testimony heard from Mr. Riemersma mentioned that he had conducted approximately 300 Online Evaluations (OEs), from either seat position. In fact, he had completed an OE on Mr. Vaughan recently. The goal of an OE is to evaluate flight crewmembers under normal flight operations and grade them on their proficiency in their respective duty positions, which includes the skills and ability to operate as a crew. The evaluator can occupy a crew seat during these evaluations (Exhibit M‑3). This would lead me to conclude that a captain/captain pairing is and can be used during line operational training and evaluations, which would require both pilots to be able to function as the pilot flying (PF) or pilot not flying (PNF).
 I would agree with the Minister's position that Mr. Vaughan was capable and trained to be seated in the right seat in a pilot monitoring role and must be appraised during the LOE to meet the standard prescribed in the AQP program. Use of proper SOP callouts, checklists, and flight path monitoring calls are required skills that need to be evaluated.
 The role of the pilot monitoring or PNF is a critical component of CRM, and even more so in abnormal situations such as the one encountered during the LOE. The AQP Evaluator Manual states that PNF duties will be validated along with PF duties (sections 8.10.9(e) and 9.3.2). Adherence to normal crew coordination is expected in accordance with the Aircraft Operating Manual, Flight Operations Manual and company SOPs. Mr. Vaughan, in a normal line flight, can be designated as PF or PNF and is qualified and trained to be a pilot monitoring crewmember. Therefore, I believe that Mr. Vaughan could be evaluated to company SOPs and expected to use proper callouts and procedures as a PNF or pilot monitoring from the right seat. This is evident in the fact that he had been recommended to undergo the LOE.
 As such, discussion by the witnesses of Mr. Vaughan's performance as a pilot monitoring from the right seat leaves me to conclude that it was unsatisfactory. In his testimony, the applicant did not dispute his performance and the lack of standard calls or adherence to the SOPs in the right seat. Instead, he clearly indicated that these omissions were intentional since the captain was having a difficult time flying the aircraft and his role had become simply to try to get the captain on the ground without crashing. Also taking into consideration the evaluator's description and assessment of Mr. Vaughan's performance on that day, I find that the Minister has proven this element of the failure on the part of the applicant to meet the required standard.
(3) Question 3
 The applicant's representative argued that Script 23 REV 1.8 did not meet the guidelines and criteria as outlined in Transport Canada's document on AQP development and implementation (Exhibit A‑8) because it contained multiple unrelated malfunctions. The point raised is valid and needs to be addressed.
 Both Transport Canada guidance documents (Exhibits M‑3 and A‑8) state that scripts should focus on technical and CRM topics in a realistic line operation scenario. The AQP Evaluator Manual (Exhibit M‑3) explicitly states at section 10.39.2 that multiple unrelated failures that have a cumulative effect must not be planned in a scenario. In Script 23, the pre-flight briefing and flight plan contain a MEL dispatch provision according to which one FMGC unit is unserviceable. I noted during testimony that the crew was aware of this item failure, yet based on testimony, minimal discussion as to the ramifications of operating on a single FMGC or of the potential failure of the second unit was had by the crew prior to departure.
 Once FMGC unit number two failed, the crew found itself in a high workload environment, but in my opinion, this cannot be regarded as constituting a case of multiple unrelated failures. The loss of the second FMGC indeed added to the complexity of the approach but it did not have a cumulative effect on an unrelated system or add to an already excessive workload for the crew. Testimony from Mr. Riemersma indicated that the crew displayed little knowledge of certain system anomalies resulting from a failed FMGC and little to no discussion took place with regard to the possibility of the second and last unit failing. In my opinion, departing with a single FMGC should have triggered the crew to review all applicable systems that would be affected and discuss the possible issues and concerns that may arise should the second unit fail.
 Finally, Mr. Riemersma, Mr. Paré and Mr. James all testified that the script used was approved and vetted through both the Air Canada AQP program team and the oversight officer at Transport Canada, and it had been in use since November 7, 2015 with no issue raised prior to this event. Therefore, I cannot conclude that the script used for the applicant's LOE was invalid.
 The Minister has proven, on the balance of probabilities, that the applicant did not fulfil the conditions necessary for the issuance of a Canadian aviation document as per paragraph 6.71(1)(b) of the Aeronautics Act.
February 16, 2017
- Date modified: