Keywords: Flight proficiency check, Standards, Disapproval rates, Examiner, Inspector, Qualification, Oversight

The contracting States of the Convention on International Civil Aviation (1944) shall ensure the validity of

TABLE OF FIGURES

THE INTERSHIP'S SUBJECT

Problematics

There is an anecdotal perception among the Civil Aviation Inspectors that the Brazilian Civil Aviation Authority -ANAC lacks safety culture, standardization, oversight, and Inspectors' pilot training that could be associated to low flight proficiency check's disapproval rates among pilots.

To investigate that premise, this study focused on answering the following research questions:

1. What is the status of the Brazilian pilot flight proficiency check's rates? 2. Is there a statistical difference between Examiners' rates and Inspectors' rates?

3. How different are the Brazilian pilot flight proficiency check's rates compared to the American rates?

In order to answer the research questions, this study compared the disapproval rates of pilots in flight proficiency checks applied by Examiners and Inspectors from the ANAC (Brazil) from 2014 to 2019, and compared them to the FAA's rates (United States of America) from the same period, focusing on initial checks for the issuance of the following: PPLprivate pilot license, CPLcommercial pilot license, ATPLair transport pilot license, Instrument and Flight Instructor ratings. I declare that this thesis was written solely by myself and that it has not been submitted, in whole or in part, in any previous application for a degree. Except otherwise stated by reference or acknowledgment, the work presented is entirely my own. It reflects my voluntary personal opinion on aviation safety data and safety information derived from, both protected in accordance with Section 5. The Annex 1 to the Convention on International Civil Aviation, on its Chapter 2, subsection A, section 1.2 General rules concerning licences, paragraph (a), has divided, among others, the flight crew into the following licences:

•

private pilot;

• commercial pilot;

• multi-crew pilot;

• Airline transport pilot;

• remote pilot; to be included as of 3 November 2022 (ICAO, 2018) The Annex 1 to the Convention on International Civil Aviation, on its Chapter 2, subsection A., paragraph 2.1.1 General rules concerning pilot licences and ratings (ICAO, 2018), has divided, among others, the aircraftcategory1 into:

• Aeroplane;

• Helicopter;

Among the SARPs, the contracting States shall ensure the validity of issued licences.

Therefore, flight proficiency checks are used to grant its privileges, to maintain the holder's competency2 , and shall be conducted by personnel authorized by the Licensing Authority3 .

(ICAO, 2018)

The issuance of licences and certificates, the permanent character of licences, and the validity of certificates are provisioned by the Brazilian Federal Law 7. (The Federative Republic of Brazil, 1986) are divided into fine, suspension, and revocation.

-The context description of pilot flight proficiency checks

The SARPs -Standards and Recommended Practices for Personnel Licensing were first adopted as minimum standards in 1948 and designated as Annex 1 to the Convention.

Among these standards, the contracting States shall ensure the validity of issued licences.

Therefore, a flight proficiency check is used to grant its privileges, to maintain the holder's competency, and shall be conducted by personnel authorized by the Licensing Authority. In other words, these professionals are also known as:

• Civil Aviation Inspector (ANAC's employee);

• Accredited Examiner (ANAC's outsourced);

• ASI -Aviation Safety Inspector (FAA's employee); , 2020, pp. 4, 11).

In order to be designated by the ANAC as an According to Riley's future study's recommendations (Riley, 2004)

-The student's position during the internship

The student put in practice the knowledge of the Advanced Master's academic phase acting in the management of aviation safety data regarding the pilot flight proficiency check's activities through knowledge and skills on project management, data analyses, and risk assessment.

-The student's career experience

During the last twelve years, the student has been working full-time at the Brazilian National Civil Aviation Agency -ANAC as an Aviation Regulations Specialist, as well as an Chapter 3 -The Project Plan's timeline and the Work Plan

-The initial scheduled Project Plan

The study was structured in order to follow 6 monthly steps, which initial project plan's timeline was according to the following diagram:

Each step was detailed in Section 3.2 -.

-The Work Plan

Diagram 3.1 -Project Plan's timeline Source: own elaboration.

-Data extraction

For the purpose of this scientific research, data privacy has been guaranteed by Article 31, § 3º, II of the Brazilian Federal Law 12.527/2011(The Federative Republic of Brazil, 2011). In order to properly compare the ANAC's data with the FAA's detailed data, and to better define the research scope, both data would range from 2014 to 2019.

-Data classification

The data mentioned above contained For proper comparison with the FAA's data, the ANAC's data were classified, whenever possible, using the template of the published U.S. Civil Airmen Statistics, without the prejudice of the in-depth analyses of the comprehensive ANAC's data.

-Data analysis

At first, the ANAC's classified data allowed us to analyze the general flight proficiency check status compared to the FAA's.

Then, ANAC's data, which were split into two sample groups (Examiners and Inspectors), allowed us to analyze the difference of disapproval rates and shares.

Finally, the subdivision into licences allowed us to analyze the Inspectors' direct participation in the personal licensing for pilots and further risk assessments.

-Literature research

The

-Enhancements proposals

The enhancements proposals aimed to allow ANAC to strengthen the defences, present in the James Reason's Model, with technology, training, and regulations, present in Snook's Theory, to avoid accidents related to the practical drift of pilot flight proficiency checks' oversight.

-Proficiency Check Dashboard

The creation of the Proficiency Check Dashboard was proposed by the student, using the dataset from the SACI -Civil Aviation Information Integrated System, as an outcome of this project, in order to better present the results, providing a widened view on the subject, and to make viable future oversights.

Chapter 4 -In-depth analysis of the pilot flight proficiency check's disapproval rates Point of HR and due to the error in the SACI, identified by the student. The system did not register their ANAC's code (7 Inspectors were found and there are many other unknown

Examiners), so their data were grouped into the Others category.

Among the total of 130.243 pilot flight proficiency checks, the database permitted to classify those required for initial personnel licensing purposes, which represents the population N= 24.307, including PPLprivate pilot licence, CPLcommercial pilot licence, ATPLair transport pilot licence, Instrument and Flight Instructor ratings.

-General rates

The

-Personnel Licensing rates

The pilot flight proficiency checks for personnel licensing purposes considered herein were only those required for the initial licensing or rating and do not include the recurrent flight checks considered in the previous section General rates.

Therefore, the pilot flight proficiency check's histogram for licensing was represented by the following figure: The following table represented the results of pilot flight proficiency check for licensing: The following figure represents the pilot flight proficiency check's disapproval histogram for licensing: Every disapproval rate shown in the figure above is higher than the general disapproval rate, which includes several thousand recurrent checks, as expected for being an initial check.

-The ANAC's Airplane PPLprivate pilot licence

The ANAC's Airplane PPLprivate pilot licence rates are demonstrated, as follows:

-The ANAC's Airplane CPLcommercial pilot licence

The ANAC's Airplane CPLcommercial pilot licence rates are as follows:

-The ANAC's Airplane ATPL -Airline transport pilot licence

The ANAC's Airplane ATPL -Airline transport pilot licence rates are demonstrated, as follows:

-The rates of ANAC's Instrument rating

The rates of ANAC's Instrument Rating are demonstrated, as follows:

-The rates of ANAC's Flight Instructor rating

The rates of ANAC's Flight Instructor rating are demonstrated, as follows: The figure above showed a large increase of disapprovals by Inspectors in 2018 and a slight increase tendency since 2014, contrasting to the opposite large decrease of Examiners' rates.

-The Null Hypotheses

-ANAC's Null Hypothesis of the general pilot flight proficiency check

Considering the ANAC's Null Hypothesis H0, i.e., no differences between the ANAC's samples n1 -Accredited Examiners and n2 -Civil Aviation Inspectors (H0:  - =0), it was necessary to analyze the data using quantitative descriptive analyses and the statistical Two Proportions z -Test. Analyzing the descriptive statistics of ANAC's Civil Aviation Inspectors' general pilot flight proficiency checks, the Count is the number of checks (sample size) n2= 10348; the Sum is the number of Disapproved checks (statistical success events) Σ X2= 201; the mean is 𝑥̅ 2 = 0.019424; the Standard Deviation is s2= 0.138017; the Sample Variance is 𝑠 2 2 = 0.019049; and the Confidence Level for the mean is 95%.

Highlighting both ANAC's samples n1= Examiners and n2= Inspectors disapproval means, there is the following graph: Analyzing the ANAC's tables and figure above, from 2014 to 2019, the Examiners' general disapproval mean 𝑥̅ 1 was 0.53%, while the Inspectors' 𝑥̅ 2 was 1.94%, the equivalent to 366% of the Examiners' mean.

Analyzing the dataset of both ANAC's samples n1 = Examiners and n2= Inspectors using the Two Proportions z -Test with a significance level α of 5%, and considering that the dataset of approval or disapproval results as a binomial distribution, the assumption of normality was assessed using the CLT -Central Limit Theorem, which the mean of any random variable will be approximately normally distributed as sample size increases according to the CLT.

Therefore, with a sufficiently large sample size (n > 50), deviations from normality will have little effect on the results [START_REF] Pituch | [END_REF]. Therefore, it was possible to calculate, with the formula below, the following values: Considering that the calculated z value is -10.29064864 and the z Critical two-tail value is 1,959964, i.e., |z| > |zCrit| and the p-value less than 0,001, then the Null hypothesis (H0) for general pilot flight proficiency check could be rejected, and there is a statistically highly significant difference between the ANAC's Accredited Examiners and Civil Aviation Inspectors proportions for general pilot flight proficiency check.

0,53% 1,94% 0,64% 0,00% 0,50% 1,00% 1,50% 2,00% 2,50% Examiners' Inspectors' Disapproval ANAC's mean 𝑧 = (𝑥̅ 1 -𝑥̅ 2 ) -(𝜇 1 -𝜇 2 ) √ 𝜎 1 2 𝑛 1 + 𝜎 2 2 𝑛 2

-ANAC's Null Hypothesis of pilot flight proficiency check for licensing

Considering the ANAC's Null Hypothesis of pilot flight proficiency check for licensing H0, i.e., no differences between the ANAC's samples n3 -Accredited Examiners and n4 -Civil Aviation Inspectors (H0:  - =0), it was necessary to analyze the data using quantitative descriptive analyses and the statistical Two Proportions z -Test.

The ANAC's descriptive statistics on the disapproval rate of pilot flight proficiency check for licensing from 2014 to 2019 are demonstrated, as follows: Analyzing the descriptive statistics of ANAC's Civil Aviation Inspectors' pilot flight proficiency checks for licensing, the Count is the number of checks (sample size) n4= 1047; the Sum is the number of Disapproved checks (statistical success events) Σ X4= 29; the mean is 𝑥̅ 4 = 0.02769819; the Standard Deviation is s4= 0.16418509; the Sample Variance is 𝑠 4 2 = 0.02695674; and the Confidence Level for the mean is 95%.

Highlighting both ANAC's samples n3= Examiners and n4= Inspectors disapproval means for licensing, there is the following graph: Analyzing the figure above, from 2014 to 2019, the Examiners' disapproval mean for licensing was 0.75%, while the Inspectors' was 2.77%, the equivalent to 369% of the Examiners' mean.

Analyzing the dataset of both ANAC's samples n3 = Examiners and n4= Inspectors using the Two Proportions z -Test with a significance level α of 5%, and considering that the dataset of approval or disapproval results as a binomial distribution, the assumption of normality was assessed using the CLT -Central Limit Theorem, which the mean of any random variable will be approximately normally distributed as sample size increases according to the CLT.

Therefore, with a sufficiently large sample size (n > 50), deviations from normality will have little effect on the results [START_REF] Pituch | [END_REF]. Therefore, it was possible to calculate, with the formula below, the following values: Considering that the calculated z value is -3,951395028 and the z Critical two-tail value is 1,959964, i.e.,, |z| > |zCrit| and the p-value less than 0,001, then the Null hypothesis for licensing (H0) could be rejected, and there is a statistically significant difference between the ANAC's Accredited Examiners and Civil Aviation Inspectors proportions of pilot flight proficiency check for licensing.

𝑧 = (𝑥̅ 3 -𝑥̅ 4 ) -(𝜇 3 -𝜇 4 ) √ 𝜎 3 2 𝑛 3 + 𝜎 4 2 𝑛 4 0,75% 2,77% 0,84% 0,00% 0,50% 1,00%

-ANAC's Accredited Examiners' and Civil Aviation Inspectors' Licensing rates comparison

In order to test the Null Hypothesis, this study used the Two Proportions z-Test to examine whether there was a significant difference between the Examiners' and Inspectors' disapproval proportions of ANAC's pilot flight proficiency checks for licensing -PPL, CPL, andATPL (2014 -2019).

Considering the dataset of approval or disapproval results as a binomial distribution, and that the assumption of normality was assessed using the CLT -Central Limit Theorem, in which the mean of any random variable will be approximately normally distributed as sample size increases according to the CLT. Therefore, with a sufficiently large sample size (n > 50), deviations from normality will have little effect on the results [START_REF] Pituch | [END_REF].

In order to compare the pilot flight proficiency checks for licensing applied by Accredited Source: own elaboration.

The total Disapproval events were considered the statistical success (k) and resulted in 204 flight checks for licensing.

The Examiners' sample size (n3) was 23.260 flight checks for licensing, while the Disapproval events were considered the statistical success (k1) and resulted in 175 flight checks for licensing.

The Inspectors' sample size (n4) was 1047 flight checks for licensing, while the Disapproval events were considered the statistical success (k2) and resulted in 29 flight checks for licensing.

The probabilities of being checked by Examiners or Inspectors were demonstrated as follows: From the above tables, it was possible to conclude the following:

• the ANAC has outsourced 95,69% and insourced 4,31% of the pilot flight proficiency checks for licensing -PPL, CPL, and ATPL (2014 -2019);

• the differences between the Examiners' and Inspectors' disapproval rates (0,75%, 2,77%) which represents a difference of 369%;

• From the 95,69% of the checks, the Examiners approved 95,78%;

• From the 4,31% of the checks, the Inspectors approved 4,22%;

• Even with 95,69% of the checks, the Examiners were only responsible for 85,78% of disapprovals;

• With only 4,31% of the checks, the Inspectors were responsible for 14,22% of disapprovals;

• Considering the Examiners and Inspectors approval rates, the expectation seems consistent with the Null Hypothesis H0 because the Examiners' and Inspectors' approval rates were closer to the share of performed checks, 0.001%, and 2.1%, respectively.

• Considering the Examiners' and Inspectors' disapproval rates, the expectation does not seem consistent with the Null Hypothesis H0 because the Examiners' and Inspectors' disapproval rates were not closer to the share of performed checks, 11.6% and 69.7%, respectively.

Therefore, the data let us infer that Inspectors were 369% harsher while performing flight checks and, even performing only 4.31% of the licensing flight checks, they were responsible for 14.22% of the disapprovals, better assuring that the standards and safety were met. The total issued FAA's Instrument rating was published in Analyzing the descriptive statistics of FAA's Inspectors' pilot flight proficiency checks for licensing, the Count is the number of checks (sample size) n6= 6167; the Sum is the number of Disapproved checks (statistical success events) Σ X6= 1503; the mean is 𝑥̅ 6 = 0.2437166; the Standard Deviation is s6= 0.4293585; the Sample Variance is 𝑠 6 2 = 0.1843487; and the Confidence Level for the mean is 95%.

-Other

Analyzing the FAA's tables above, from 2014 to 2019, unlike ANAC's samples means, the FAA's Inspectors' disapproval mean 𝑥̅ 6 is only 5.5% higher than the Examiners' mean 𝑥̅ 5 .

Analyzing the dataset of both FAA's samples n5 = Examiners and n6 = Inspectors using the Two Proportions z -Test with a significance level α of 5%, and considering that the dataset of approval or disapproval results as a binomial distribution, the assumption of normality was assessed using the CLT -Central Limit Theorem, which the mean of any random variable will be approximately normally distributed as sample size increases according to the CLT.

Therefore, with a sufficiently large sample size (n > 50), deviations from normality will have little effect on the results [START_REF] Pituch | [END_REF]. Therefore, it was possible to calculate, with the formula below, the following values: Considering that the calculated z value is -2,302847 and that the z Critical two-tail value is 1,959964, i.e., |z| > |zCrit| and the p-value is 0,0212875, then the Null hypothesis (H0) could be rejected, and there is a statistical difference between the FAA's Designated Pilot Examiners and Aviation Safety Inspectors proportions of pilot flight proficiency check for licensing.

𝑧 = (𝑥̅ 5 -𝑥̅ 6 ) -(𝜇 5 -𝜇 6 ) √ 𝜎 5 2 𝑛 5 + 𝜎 6 2 𝑛 6
The FAA's pilot flight proficiency checks rates for licensing (2014 -2019) were: The FAA's Examiners' pilot flight proficiency checks' rates for licensing (2014 -2019) were: The FAA's Inspectors' pilot flight proficiency checks' rates for licensing (2014 -2019) were:

-The Commonwealth of Australia's rates

In order to provide another source of data for comparison, electronic mails were exchanged by the student and the Australian CASA -Civil Aviation Safety Authority from September 14, 2020 through November 25, 2020, but without success in receiving the data due to the specificities of the Australian Authority's oversight activities, summarized in the last correspondence as: (Cameron 2020, pers. comm., 25 Nov) Unfortunately, CASA was unable to provide data for the intended comparison.

-The Federative Republic of Brazil and The United States of America comparative analyses

In order to compare the Brazilian and the American licensing rates of the pilot flight proficiency checks, it was necessary to straighten the scope to the 2014 to 2019 range, combine the FAA's yearly published data into the following licences and ratings: PPLprivate pilot licence , CPLcommercial pilot licence, ATPLair transport pilot licence, Instrument and Flight Instructor ratings, using the same criteria, represented by the Table 4.38 and Table 4.39 on the following page: Therefore, with a sufficiently large sample size (n > 50), deviations from normality will have little effect on the results [START_REF] Pituch | [END_REF]. Therefore, it was possible to calculate, with the formula below, the following values: Considering that the calculated z value is -229.928 and the z Critical two-tail value is 1,959964, i.e., |z| > |zCrit| and the p-value less than 0,001, then the Null hypothesis (H0) could be rejected, and there is a statistically highly significant difference between the ANAC's Accredited Examiners and FAA's Designated Pilot Examiners.

𝑧 = (𝑥̅ 3 -𝑥̅ 5 ) -(𝜇 3 -𝜇 5 ) √ 𝜎 3 2 𝑛 3 + 𝜎 5 2 𝑛 5
Analyzing the dataset of both ANAC's Inspectors n4 and FAA's Inspectors n6 using the Two Proportions z -Test with a significance level α of 5%, and considering that the dataset of approval or disapproval results as a binomial distribution, the assumption of normality was assessed using the CLT -Central Limit Theorem, which the mean of any random variable will be approximately normally distributed as sample size increases according to the CLT.

Therefore, with a sufficiently large sample size (n > 50), deviations from normality will have little effect on the results [START_REF] Pituch | [END_REF]. Therefore, it was possible to calculate, with the formula below, the following values: Considering that the calculated z value is -28.96007 and the z Critical two-tail value is 1,959964, i.e., |z| > |zCrit| and the p-value less than 0,001, then the Null hypothesis (H0) could be rejected, and there is a statistically highly significant difference between the FAA's and ANAC's Examiners and FAA's and ANAC's Inspectors.

𝑧 = (𝑥̅ 4 -𝑥̅ 6 ) -(𝜇 4 -𝜇 6 ) √ 𝜎 4 2 𝑛 4 + 𝜎 6 2 𝑛 6
Therefore, comparing both Two Proportions z -Test from FAA's and ANAC's Examiners and FAA's and ANAC's Inspectors, i.e., z scores -229.928 and -28.96007, the large difference between the categories relatively shows how many standard deviations they are from their means.

Once again, there is evidence to reject that the Null hypothesis (H0), meaning no differences between the ANAC's n3 -Accredited Examiners and n4 -Civil Aviation Inspectors of pilot flight proficiency check for licensing (H0:  - =0) could be rejected, and there is a statistically significant difference between ANAC's Accredited Examiners and Civil Aviation Inspectors.

4.9 -FAA's initiatives

-Oversight

The FAA's Order 8000.95 CHG 5 contains the oversight policy of Designees, as follows:

2. General.

a. Oversight Principles.

(1) Promote Safety. A primary responsibility of the FAA is to promote safety through systematic oversight of industry stakeholders, including designees. Information generated from oversight programs permits the FAA to identify safety hazards, mitigate risk, and enhance aviation safety. In order for oversight programs to be effective, they must be carefully planned and executed during the conduct of specific inspection activities.

(2) Evaluate Performance. The objective of an oversight program is to ensure that the designee performs to the standards and expectations set forth by the FAA in its policies and regulations.

Oversight is not merely an isolated event or series of activities. Oversight results should be considered in total to provide a high-level perspective of a designee's performance over time. (FAA, 2020)

[our emphasis]

The FAA considers that the designee oversight is not an isolated event; on the contrary, it is a high-level perspective of a designee's performance over time that promotes safety and permits safety hazards identification, risk mitigation, and safety enhancements.

The The FAA uses a data-driven approach to support a comprehensive designee oversight system, focusing on higher risk indicators.

The SEED Inspections are planned considering the following:

SEED Inspections.
SEED inspections concentrate resources on five primary areas, which are explained in detail later in this section. These include:

1) Pre-surveillance analysis of airman testing data from internal FAA sources and publicly accessible data regarding the designee's activities.

2) Comprehensive review of the designee's responsible Flight Standards office records and past certification data.

3) Structured interviews of recently tested airmen.

4) Observation of a test performed by the designee.

5) Information gathering from members of the aviation community and/or other sources. (FAA, 2019) [our emphasis]

The SEED team members eligibility requirements are the followings: The FAA uses qualified Operations Inspector, which means they hold at least Commercial Pilot Certificate, appropriate class or type rating, and, for two-place aircraft, current in the aircraft.

-Analysis of the difficulties

-ANAC's regulation

The following ANAC's regulations are intended to demonstrate the difficulties for the Inspectors and Examiners that perform pilot flight proficiency checks, and the major difficulty is that ANAC does not abide by their own regulations regarding the Inspectors' compliance, as required.

Instead, it is common that the regulations that contained these requirements are revoked and, sometimes, replaced by others that do not mention the need for Inspectors' pilot technical training, such as: Therefore, the following regulations were chosen due to their scarcely mentions to the Inspectors' pilot flight proficiency check activity and its associated outsourced activity performed by Examiners, which have better described licence and rating requirements. According to Item 5.2.3.17 of the IS -Supplementary Instruction 00-002 A and B, the Inspector may use the unoccupied pilot's seat, if rated, or, if not rated, a rated instructor must occupy the pilot's seat. (ANAC, 2011) (ANAC, 2012). However, the ANAC does not maintain the Inspectors rated.

In fact, due to the lack of CMA and ratings, the ANAC forces the Inspectors to act as merely observers, usually with very limited view of the pilot's seat or the aircraft's instrument panel, due to occupying a passenger's seat or crouching between the pilots' seat (without ergonomics and seat belt), compromising their safety and relying on an unknown pilot presented by the examinee.

Another problem of Inspectors lacking ratings is that the initial flight check for an examinee without a valid rating on a two-place aircraft have to be delegated to Examiners, thus reducing this sensitive licensing and oversight activities that must be done by the Inspectors.

According Unfortunately, one hypothesis for the current disapproval rates and the associated difficulties might be the lack of an Examinations branch, and technically qualified and licensed key personnel, as detailed, not making judgements in ANAC's PEL Office.

-Accident causation

In order to continue the difficulties analyses, it is important to briefly explain the Accident

-The latent conditions

Most of the difficulties explained in the previous section were considered latent conditions present in the system in a dormant state since ANAC's foundation. The purpose of this research was also to highlight these latent conditions and make mitigations possible, before an operational, organizational, or total system accident occur.

-ANAC's Safety Culture

For the first time, the ANAC's research on Safety Culture done in 2020 (ANAC, 2020), based on the SM ICG -Safety Management International Collaboration Group tool, pointed that 18% of the active human resources responded, i.e., 239 out of 1.342 civil servants.

This might be seen as low participation, worsen by the result of the respondents with:

• 12% stating that they do not act upon safety;

• 33% that they indirectly act upon safety;

• 55% directly act upon safety; in contrast to the 77% being in the target activity safety. According to the Paragraph 3.2.5.1 of the Doc 9859 -Safety Management Manual, Fourth Edition:

SSPs and SMSs are sustained by safety data and safety information that is necessary to address existing and potential safety deficiencies and hazards, including safety issues identified by personnel. The success of a reporting system depends entirely on the continuous flow of information from, and feedback to, organizations and individuals. [our emphasis]

At ANAC's it is not uncommon that the safety perceived and practiced by the personnel seems jeopardized by the management, which acts towards the opposite direction of its own personnel, giving way to the Management Dilemma expressed in the following figure: Chapter 5 -Presentation of the different solutions to the identified problem

-Presentation of the possible solutions and justifications

As seen in In-depth analysis of the pilot flight proficiency check's disapproval rates, there was a statistically highly significant difference between ANAC's Examiners and Inspectors regarding their test results, in particular the disapproval rates, which pieces of evidence could reject the null hypothesis H0:  - =0, that assumed no differences between the ANAC's Accredited Examiners and Civil Aviation Inspectors.

The main possible solutions identified to solve this situation lies in standardization and surveillance, which unveils other shades of problems discussed in this chapter.

-Standardization

Regarding standardization, the ANAC has periodically improved the pilot flight proficiency check standards through the issuance of the IS -Supplementary Instruction 00-002, according to the following table: Unlike the FAA, all the obsolete revisions of the regulations are not kept in the ANAC's regulations' portal as a history, which represents difficulty to the oversight and enforcement of past events.

The publication of obsolete revisions and the use of an Amendments' record as an editorial practice would avoid the above-mentioned difficulties and would cope with the currency of Examiners and Inspectors in the regulations.

Note: A designee must not exercise his or her designation privileges unless his or her training is

current and up to date.

Knowledge and Skill. The determination that the designee's knowledge of and application of current procedures and standards are adequate for renewal of the designation may be based on oversight of the designee and the designee's attendance at a meeting or seminar conducted for the purpose of standardization. (FAA, 2018) [our emphasis]

It is worth mentioning that, unlike Inspectors' recurrent training requirement every three years, there is no recurrent training requirement for maintaining the Examiners' accreditation.

The certification of Accredited Examiners, according to the SEI! -Electronic Information System, is divided into the following activities: Therefore, even with objective standards and balanced training to become Examiners, as mentioned above, the Examiners' disapproval rate was still significantly shorter than the Inspectors' rates, although it might seem an improvement from the 2017's 0.07% to 2018's 0.14% and 2019's 0.12% of disapproval rate for licensing, it barely avoided its downward movement towards 0%, while, in contrast, the spread from the Inspectors' rates were increasing, aggravated by its own increase.

-Surveillance

The continued surveillance of operators is a State obligation to confirm that they continue to meet the initial certification requirements and is functioning satisfactorily, as provisioned by Annex 6 and Annex 19 to the Convention on International Civil Aviation.

Concerning surveillance, according to the SEI! -Electronic Information System, the surveillance of pilots' Accredited Examiners are divided into the following activities: The following table represents the proportion of oversight per certification activities: Considering the rates above, the second approach for the solution, an active surveillance following the model used in the FAA's SEED -Special Emphasis Evaluation of a Designee, seems to be more promising, since the past efforts were focused on Accredited

Examiner's certification, creating a huge gap between certification and oversight. Inspectors -First Edition:

CE-5 Technical guidance, tools and provision of safety-critical information

The State shall provide appropriate facilities, comprehensive and up-to-date technical guidance material and procedures, safety-critical information, tools and equipment

The State must establish conditions that allow CASIs to discharge their duties. This requires the

State to delegate specific powers of authority to CASIs supporting the accomplishment of their tasks. Some States codify the CASI's power of authority and degree of accountability into aviation law to enable CASIs to effectively carry out their assignments. Creating the conditions that allow

CASIs to discharge their duties also requires the DGCA to:

• establish clear and comprehensive workplace policies and procedures;

• provide a safe working environment; and Most of these aircraft were lent to nonprofit flying clubs prior to ANAC's foundation without due. Astonishingly, a few were received brand-new by ANAC from the manufacturer and shuttled to an airport which shortly became their graveyard. A similar situation happened to many other aircraft that were lent after the use of the aircrafts' available hours, mainly due to the cost of TBO -Time between overhauls which the flying clubs claim they cannot afford to pay.

The most important information about the ANAC's aircraft: none of them were used for Inspectors' training or currency maintenance, despite the easiness of its use.

To summarize the easiness of the aircraft use, the ANAC could be in charge of refueling and flying if it acts as a lessor of its owned aircraft, while the operator acts as a lessee, operating and maintaining the aircraft airworthiness whenever established in the lease term.

The usage of ANAC's owned aircraft in the Inspectors' training should be incentivized in order to comply with the above-mentioned SAPRs as a complementary in-company approach to the approved ANAC's Flight Inspectors Training Programme (ANAC, 2019).

-Technical guidance, tools, and provision of safety-critical information -(CE-5)

-Digitalization of Job aids -Pilot flight proficiency check forms

According to Item 7.2.4 of ICAO's DOC 9941 (ICAO, 2011), the job aids have to be prepared to assist the user to attain the performance objective or to replace training partially or completely. There are also several fundamental points when preparing a job aid, such as accessibility, conciseness, accuracy, durability and modifiability.

Therefore, the digitalization of check forms is recommended and could help Inspectors to better apply the standards and register results, preflight, in-flight, and postflight, compared to a paper form that requires a pen, a clipboard and is subject to adverse operational environment.

The electronic information gathered could be easily used to monitor both the Examiner and examinee performances, instead of being in a paper in which only the approved or disapproved information is commonly used while all the other written precious data is put in the second place.

-Preflight Operational Risk Management Form

It is recommended the creation of a Preflight Operational Risk Management Form, such as the FAA's Preflight Operational Risk Management Form (NON-AEG or AEG Flights Only), contained in Notice 8900.442 -Flight Standards Aviation Safety Inspector (Operations) Risk

Management (FAA, 2017), in order to provide a safe working environment to the Inspectors.

Considering the difficulties detailed in Chapter 4 -, the Form should also include risks associated to the lack of Inspectors' rating, currency, medical certificate and technical training while existent, for proper risk assessment in ANAC's work environment.

-Unifying systems

In order to reduce unnecessary rework, it is recommended that the different systems work together as a unified system, avoiding filling the same data repeatedly.

The current status, from the beginning to the end of the process, the Inspector's assignment is done via the SisCheck -Roster's Flight Check Management System, which contains the candidates' data and the destination; then, the Inspector shall fill itinerary data in the SICOS -Service Order System, which sends data to someone else, who have to insert data again in the SCDP -Allowance and Fare System after the pilot flight proficiency check, in which is produced a handwritten paper form, the Inspector shall fill the simplified Electronic PEF -Pilot Evaluation Form in the SACI -Civil Aviation Information Integrated System, and shall report travel administrative accountability via e-mail, which was already pre-approved in three different systems: SisCheck, SICOS, and SCDP, before the activity, even if no changes were required.

-Mobile APPapplication

The use of the mobile APP as a tool to register the flight data brings many benefits, such as the enhancement of the flight proficiency check through pictorial debriefing, the possibility to conduct off-site flight checks' oversight, and to analyze the data in order to improve the ANAC's flight activities.

It is worth mentioning that the SDEA -Santos Dumont English Assessment (ICAO English level) requires the upload of massive size of video files in order to conduct off-site oversight, according to Item 5.8.2 of the Supplementary Instruction 183-001D (ANAC, 2018), which is not the proposal, because the data file size is just several kBkilobytes.

Unlike GNSS -Global Navigation Satellite System purposes, the proposed tracker's accuracy, integrity, continuity and availability is not relevant per se, because its intended for post-flight use, assisting during debriefings, and for off-site oversight activities which might accept deviation.

-Licensing

(ICAO, 2018) [our emphasis]

According to Section 1.2 of Doc 10070, if the Inspectors do not acquire the necessary competencies, safety will be impacted. Also, the Inspectors should be valued by being a highlytrained team, which acquisition and maintenance of a balanced team should be funded by the State to ensure that safety levels are not compromised:

While it is appreciated that some States with limited aviation activity have difficulty finding suitable CASI candidates, not acquiring the necessary competencies within the CAA Inspectorate impacts safety.

The DGCA should champion the acquisition and maintenance of a well-balanced CAA Inspectorate

suitably funded by the State to ensure aviation safety levels are not compromised.

…

An additional recruitment strategy would be to highlight the benefits of belonging to a team of highlytrained professionals working towards creating a safer and more efficient air transportation system for the country and its citizens and the opportunity of further developing themselves professionally (ICAO, 2016)

[our emphasis]

The implementation of the PCIV -ANAC's Flight Inspectors Training Programme is a huge step toward acquiring the necessary competencies for providing the safety duties, and the enabler could be the use of a Civil Aviation Inspector ratings according to the similarities established in the PCIV, i.e.:

1. single engine;

2. multi-engine;

3. normal category jet 1;

4. normal category jet 2;

5. transport category multi-engine;

6. transport category jet 1;

7. transport category jet 2;

8. transport category jet 3; 9. transport category heavy jet 1; 10. transport category heavy jet 2;

11. single engine helicopter;

12. turboshaft helicopter 1;

13. turboshaft helicopter 2;

14. multi-engine helicopter 1;

15. multi-engine helicopter 2;

Analyzing the 2018's internal surveys made by and for the flight Inspectors, during the PCIV project, there were 51 Civil Aviation Inspectors that could be eligible for conducting pilot flight proficiency checks, as soon as properly provided recurrent training, instead of being misapplied in nonspecialized tasks.

It is also possible to rescue 7 Civil Aviation Inspectors that once were eligible for conducting pilot flight proficiency checks, but who lost interest, mainly due to the lack of training, i.e., a 14% increase in the specialized staff.

The major Inspectorate capacity improvement lies in the use of 22 potential candidates that meet the entry-level requirements for becoming Civil Aviation Inspectors eligible for conducting safety oversight activities, including pilot flight proficiency checks, i.e., a 43% increase in the specialized staff.

In other words, the better use of the available specialized staff (flight Inspectors) could be improved by at least 57% compared to the current use. There would be plenty of space for other huge improvements if the flight Inspectors were assigned to oversight activities that require their competencies and KSAknowledge, skills, and attitudes, instead of being misused or underused in nonspecialized activities.

-Surveillance obligations -(CE-7)

Once provided with qualified personnel from CE-4, the appropriate technical guidance, and tools from CE-5, the certification obligations from CE-6, it is time to focus on the Therefore, the Examiners must be closely supervised, their conduction of proficiency checks should be observed, and their performance monitored by the Inspectors because the final responsibility remains with the ANAC. In order to accomplish the above-mentioned supervision, there must be issued requirements for minimum annual numbers of examinations and periodic observation of ratings and proficiency checks conducted by the designated Examiner.

-Surveillance of Accredited Examiners

As mentioned in Presentation of the different solutions to the identified problem, there were some oversight activities focused on the Accredited Examiners. The oversight must be intensified through the use of appropriate specialized personnel, i.e., flight Inspectors, and be planned and accomplished by the use of a data-driven approach.

One of the requirements emphasized during the oversight is the requirement to fill the electronic PEF -Pilot Evaluation Form provisioned by Item 12.2 of the IS 00-002F: The dataset gathered from the above-mentioned requirement made possible the analyses contained in Chapter 4 and has proven its potential towards a data-driven safety oversight.

Nonetheless, through the data collected, it is clear that several Accredited Examiners were not abiding by the requirement provisioned by Item 12.2 of the IS 00-002F, for instance, the Airlines' Accredited Examiners, which are not exempted from the filling of the electronic PEF, even when using other systems such as SISHAB -Rating System.

The above-mentioned lack of conformity to the IS 00-002F reduces the proposed datadriven oversight capabilities and should be widely corrected.

-Regulation improvement

The proposal of regulation improvement regarding pilot flight proficiency checks to require recurrent training as explained inIn-depth analysis of the pilot flight proficiency check's disapproval rates; and establish a limit of two checks per day (initial, retests, discontinuances, or continuances) or one initial flight instructor check, similarly to the provisions of Section 2, Item14, a., of the Order 8900.2C, as follows: It is worth mentioning that the FAA has published the Notice 8900.485 -Removal of Designated Pilot Examiner Geographic Limitations and Other Restrictions, which allowed a temporary increase to up to three checks per day, but that was canceled in a one-year timeframe. Therefore, the following text is no longer valid:

(3) Number of Tests Allowed Daily. Managing offices will notify their DPEs that they are allowed to conduct up to three tests per day without additional approval and that there will be no limit on the number of retests that can be conducted per day. (FAA, 2018) The proposed limitation to two checks per day aims to guarantee time availability to focus on the quality of the evaluation. Exceptions to this rule were incentivized considering the specifics of each pilot flight proficiency check, such as reducing to 1 flight instructor or

Examiner checks or increasing to up to 3 when the operations environment permits.

In contrast to the FAA's reality, the lack of a limitation on the number of flight checks per days led to the following ANAC's scenario: Surprisingly, the data showed that the number of flight checks per day varied from 1 to 24.

The majority of flight checks (60%) is performed in the categories of 1 and 2 per day.

The supermajority of flight checks (77%) is performed in the categories of 1, 2, and 3 per day.

In order to identify the outliers regarding the number of flight checks per day, the IQR -Interquartile Range was used. The first quartile (Q1) was 1 flight check per day, the third quartile (Q3) was 3 flight checks per day. Using the IQR formula 𝐼𝑄𝑅 = 𝑄3 -𝑄2, the result was 2 flight checks per day. The outliers were found using two formulas:

𝑙𝑜𝑤𝑒𝑟 𝑜𝑢𝑡𝑙𝑖𝑒𝑟𝑠 = 𝑄1 -1.5 𝐼𝑄𝑅 𝑢𝑝𝑝𝑒𝑟 𝑜𝑢𝑡𝑙𝑖𝑒𝑟𝑠 = 𝑄3 + 1.5 𝐼𝑄𝑅 There were no lower outliers in the data, but the upper outliers bound were at 6 flight checks per day. In other words, more than 6 flight checks per day were considered outliers, which represents 4% of the flight checks. The outliers should be mandatorily investigated for enforcement purposes or accreditation termination based on the premise that they might be a threat to safety. To avoid extreme quantities of flight checks per day and to assure, at least, time availability that permits a minimum of quality during the flight checks, the regulation urges for a limitation on the number of flight checks, and the study's proposal lies in the IQR -Interquartile Range of 2 flight checks per day, which was exactly the same as the FAA's Order 8900.2C and compatible to the empirically perception of the Inspectorate experience that it should not be more than 4 flight checks per day.

As a matter of comparison, there is a limitation on the number of the exams for the SDEA -Santos Dumont English Assessment (ICAO English level) to up to four, preferably two in the morning and two in the afternoon, based on item 5.8.1, d), of the Supplementary Instruction 183-001D (ANAC, 2018). It is worth mentioning that these four exams are performed in an office, with air conditioning and noise isolation, unlike the harsh operations field conditions found in the equatorial and tropical regions of Brazil.

-Justification of the options which were chosen

The above-mentioned solutions were justified in order to abide by the ICAO SARPs, to identify the safety threats, to increase the defenses presented by the James Reason's Model, and to avoid the current Snook's practical drift on the pilot flight proficiency checks.

-Results obtained.

The

-Implementation of Oversight Intensification

The oversight intensification could be immediately implemented through the prioritization of flight Inspectors' assignments for the activities that require their KSAknowledge, skills, and attitudes, such as the oversight of Accredited Examiners and pilot flight proficiency checks.

However, annually, the ANAC's middle-management proposes the next year's work plan to the top-management, who decides its implementation. That being said, the implementation requires the convincement of a few management levels before it is put into practice during the next year.

Ideally, in order to obtain the best results of the oversight intensification, the Inspectors shall receive a prior qualification, especially considering the last decade's gap and the extreme differences in the ANAC's and the FAA's disapproval rates.

Meanwhile, an immediate oversight intensification could still benefit from the 369% harsher Inspectors' disapproval rate, compared to the Examiners' rate, as mitigation.

A data-driven approach through the use of the Proficiency Check Dashboard, delivered by this study, could easily show the number of active Examiners, as well as their key performance indicators, in order to better plan the oversight activities. Examiners with a yearly budget of R$ 149.296,00, in order to accomplish the surveillance of all Examiners in a 5-year cycle, as well as planning off-site activities and determining several accreditation terminations based on data-driven analysis that shows potential safety concerns.

-Implementation of Standardization

In order to implement Standardization, it was necessary to understand the current situation. Therefore, an online survey was conducted among ANAC's experts, exclusively sent The survey was distributed on 3 rd February 2021, a reminder e-mail was sent on 8 th February 2021, and data collection was terminated on 12 th February 2021.

There were 50 responses to the survey comprised of 63 ANAC's experts, which resulted in the following expertise profile: ...

(5) hora de voo em instrução: todas as horas de instrução de voo registradas como horas de instrução, seja de voo visual, de voo por instrumentos ou em dispositivos de treinamento por voo simulado, devem ser certificadas pelo instrutor de voo que tenha ministrado a referida instrução:

... The lack of flight hours log dates back to 2009, when a remarkable welcome meeting to newly hired pilots (with valid ratings and medical certificates as entry requirements), held by a former superintendent and latter director, who told them that they were not pilots from that time onward, nor must they act as pilot in command or occupy a pilot's seat, be rated or log their flight hours, subjected to conflict of interest and Office for Internal Affairs inquiries. Similar meetings were held periodically with the same superior understanding while often challenged by the Inspectorate.

Also, these pilots often took around three years to be assigned and to conclude their training as Flight Inspectors, even being experienced such as international B767 captains and B777 first officers, domestic B737 and A320 captains and first officers, and Air Carrier, SAE -Specialized Air Service, General Aviation or Flight instructors pilots, and, in the meantime, performing nonspecialized tasks and being misused or underused.

It is worth mentioning that the FAA considered each proficiency check as 5 hours of Source: own elaboration using (The jamovi project, 2020) (R Core Team, 2019) [START_REF] Revelle | ANNEX 1 -Inspectors' Survey's statements -Inspectors' factors TRN1 -The Flight Inspectors receive equipment Ground School from ANAC[END_REF] The survey's Internal Consistency was measured by the Cronbach's Alpha α, for a questionnaire X, and κ number of items, σ i 2 is the variance of item i, and σ X 2 is the total variance of the questionnaire. The Cronbach's Alpha α were calculated using the following formula and represented in Table 7.3, Table 7.4, and Table 7.5: Source: own elaboration using (The jamovi project, 2020) [START_REF] Core | R: A Language and environment for statistical computing[END_REF] [START_REF] Revelle | ANNEX 1 -Inspectors' Survey's statements -Inspectors' factors TRN1 -The Flight Inspectors receive equipment Ground School from ANAC[END_REF] Source: own elaboration using (The jamovi project, 2020) (R Core Team, 2019) [START_REF] Revelle | ANNEX 1 -Inspectors' Survey's statements -Inspectors' factors TRN1 -The Flight Inspectors receive equipment Ground School from ANAC[END_REF] Figure 7.6 -The ANAC's disapproval rate histogram for licensing (2014 -2019) Source: own elaboration

𝛼 = 𝜅 𝜅 -1 (1 - Σ 𝑖=1 𝜅 𝜎 𝑖 2 𝜎 𝑋 2 )
The above-mentioned figure represents an increase of 117% in 2018's disapproval rate for licensing, after an oversight intensification held in 2017 (with 25 inspections), which also reverted its downward tendency.

The evaluation of the proposed solution might expect a similar result, although intensified by the enhancements proposals and by the in-depth knowledge of the pilot flight proficiency check status, acquired during this research and available through the Proficiency Check Dashboard, which will provide valuable information such as the following figures:

-Difficulties and constraints

The access to data or information must be prioritized in the ANAC because they seem to be compartmentalized in the many sectors and different systems as they were not intended to serve a common goal, i.e., safety, which basically possesses the same structure, despite being a pilot flight proficiency check based on the licensing requirements of RBAC 61 or operations requirements of RBAC 90, RBAC 91, RBAC 121, RBAC 135 or RBAC 137.

In order to further compare the pilot's practical and theoretical disapprovals' rates, the data directly from the database were requested several times to the respective managers, without success. Therefore, only the pilot's practical rates were used from the SACI -Civil Aviation Information Integrated System, available for the Civil Aviation Inspectors, and with the Operations Standards Superintendent's authorization.

Unfortunately, the pilot's theoretical disapprovals' rates data were not shared, if existent at all, but certainly would have been higher than the practical rates, would have shown a significant difference, and would have emphasized the need to enhance the pilot flight proficiency checks, empirically-based among ANAC's Experts.

A similar difficulty was faced with the Australian CASA when trying to provide another data source for comparison. Despite the fact that their outsourcing policies seem to be emphasized, they stated that oversight focuses on Flight Examiners' flight test and proficiency checks, being a very small sample size, unable to provide data as requested.

The existence of several different and dissociated subsystems within SACI, such as PEL and SISHAB, or outside, such as SISCHECK, many sectorized spreadsheets and SharePoint pages owned by its managers not easily accessible to third parties, add difficulties to see the pilot flight proficiency check system as a whole. Some examples lie in the absence Programme Continuous Monitoring Approach, not only while implementing a SSP -State Safety Programme, but whenever overseeing the aviation activities, such as the pilot flight proficiency check is still worthwhile.

-Experience gained

The main academic experiences gained during this research were the use of a

Methodology and the Statistics knowledge, acquired from scratch, which were applied to the aviation data, rejecting the null hypothesis, reinforcing the previous Inspectorate's anecdotal sense, which was very pleasant to me.

While the main professional experiences gained were the management knowledge, including data-driven management to plan, execute, deliver and assess an entire project related to aviation safety, which might reach several operators and a huge quantity of operations, improving safety broadly.

Another experience that was reinforced during this study was my leadership skills, which were put to the test on every difficulty faced during the project or when the wind blew in the opposite direction of the intended safety.

Therefore, the z-score is the measurement, in units of the standard deviation, of the difference between them, considering they should not have differences.

Comparing both Two Proportions z-Test from ANAC's and FAA's Examiners and ANAC's and FAA 's Inspectors,, the large difference between the categories relatively shows how many standard deviations they are from their means.

In order to clearly show the differences between ANAC's and FAA's Examiners and ANAC's and FAA's Inspectors for each studied license and rating, refer to Table 4.38 and Table 4.39.

Although the differences between Brazilian and American Examiners and Inspectors were astonishing highly significant, the reasons were not the scope of this study and should be investigated in future studies. For the time being, they will be hanging in the air, with a sense that ANAC has room for improvement.

Chapter 10 -CONCLUSION

This research aimed at academically investigating an Inspectors' anecdotal perception that the Brazilian Civil Aviation Authority -ANAC lacks safety culture, standardization, oversight, and Inspectors' pilot training that could be correlated to low flight proficiency check disapproval rates among pilots.

To investigate that premise, this study focused on answering the following research questions:

1. What is the status of the Brazilian pilot flight proficiency check's rates?

2. Is there a statistical difference between Examiners' rates and Inspectors' rates?

3. How different are the Brazilian pilot flight proficiency check's rates compared to the American rates?

The study answered the research questions by the descriptive statistics and the comparative quantitative method Two Proportions z-Test used to identify differences among Brazilian and American Examiners and Inspectors on their pilot flight proficiency checks' results. The Null Hypotheses were Examiners and Inspectors have no differences in their results, even when comparing Brazilians with Americans.

As detailed in Chapter 1and Chapter 8 -, the project's plan was followed accordingly (100%), except to the continuity of the predicted sequence of months due to a 5-month interruption caused by the 2020's Pandemic's professional and personal constraints.

In fact, the student delivered more than previously planned by conducting an Inspectors' Survey, detailed in Chapter 7 -, which served to qualitatively better understand the current ANAC's Inspectorate perceptions on Standards, Technical Training and Surveillance.

Another attempt, which was to deliver more than previously planned by linking the Proficiency Check Dashboard to the updated database, was frustrated by the Technology Information Department, which did not allow the access to the student.

As detailed in Chapter 8 -, the access to data or information must be prioritized in the ANAC, because they seem to be compartmentalized in many sectors and in different systems, subjected to managers' will, as they were not intended to serve a common goal, i.e. safety, and should be inspired according to CENIPA's philosophy that states that in accident prevention there is no secrets, nor flags, present in Section 1.5 of NSCA 3-3 (CENIPA, 2013).

Due to the lack of proper register of the pilot flight proficiency checks by the Air Carriers and Airlines in the database used in this research, even being a mandatory requirement expressed in the IS 00-002F, this study had to be limited to general and licensing rates. Programme is capable of fulfilling its needs, once properly and fully implemented in its 5-year cycle.

Inappropriately, the PCIV's first round for single engine and multi-engine aircraft did not fulfill the rating requirements, though, the quality was praised by the participant Inspectors.

  9.1 -ANAC's General rates ............................................................................................ 9.2 -ANAC's Licensing rates ......................................................................................... 9.3 -FAA's Licensing rates ............................................................................................ 9.4 -ANAC's and FAA's Comparison ............................................................................. Chapter 10 -CONCLUSION .............................................................................................. 10.1 -Main conclusions ................................................................................................. GLOSSARY ....................................................................................................................... BIBLIOGRAPHY................................................................................................................. TABLE OF ANNEXES ........................................................................................................ IX TABLE OF DIAGRAMS Diagram 1.1 -Tree diagram of pilot flight proficiency checks ................................................ 5 Diagram 2.1 -ANAC's Organizational Chart ......................................................................... 7 Diagram 2.2 -Example of the organizational structure of a State civil aviation system ......... 8 Diagram 3.1 -Project Plan's timeline ...................................................................................11 Diagram 7.1 -Flight Inspector Training Honeycomb -ASEL, AMEL, and Turboprop ..........

  (2014-2019) .............................................................................................................................................18 Table 4.4 -Results of ANAC's PPLprivate pilot licence approval by Inspectors (2014-2019) .............................................................................................................................................19 Table 4.5 -Results of ANAC's CPLcommercial pilot licence approval by Examiners (2014-2019) ....................................................................................................................................19 Table 4.6 -Results of ANAC's CPLcommercial pilot licence approval by Inspectors (2014-2019) ....................................................................................................................................19 Table 4.7 -Results of ANAC's ATPLairline transport pilot licence by Examiners (2014-2019) .............................................................................................................................................19 Table 4.8 -Results of ANAC's ATPLairline transport pilot licence by Inspectors (2014-2019) .............................................................................................................................................19 Table 4.9 -Results of ANAC's Rotorcraft -PPLprivate pilot licence, CPLcommercial pilot licence, ATPLair transport pilot licence rates by Examiners (2014-2019) ..........................20 Table 4.10 -Results of ANAC's Rotorcraft -PPLprivate pilot licence, CPLcommercial pilot licence, ATPLair transport pilot licence rates by Inspectors (2014-2019)...................20

  Examiners (2014-2019) ........................................................................................................22 Table 4.17 -ANAC's general pilot flight proficiency checks performed by Civil Aviation Inspectors (2014 -2019) ......................................................................................................23 Table 4.18 -ANAC's PPLprivate pilot licence check by Inspectors -2016's increase and 2019's decrease ...................................................................................................................24 XII Table 4.19 -ANAC's descriptive statisticsgeneral pilot flight proficiency checks (2014 -2019) ....................................................................................................................................26 Table 4.20 -ANAC's Accredited Examiners' descriptive statistics of general pilot flight proficiency checks (2014 -2019) .........................................................................................27Table 4.21 -ANAC's Civil Aviation Inspectors' descriptive statistics of general pilot flight proficiency checks (2014 -2019) .........................................................................................27 Table 4.22 -ANAC's Two Proportions z -Test n1 -Accredited Examiners and n2 -Civil Aviation Inspectors general pilot flight proficiency checks (2014 -2019) ...........................................29 Table 4.23 -ANAC's descriptive statistics of pilot flight proficiency checks for licensing (2014 -2019) .................................................................................................................................30 Table 4.24 -ANAC's Examiners' descriptive statistics of pilot flight proficiency checks for licensing (2014 -2019) ........................................................................................................30 Table 4.25 -ANAC's Inspectors' descriptive statistics of pilot flight proficiency checks for licensing (2014 -2019) ........................................................................................................31 Table 4.26 -ANAC's Two Proportions z -Test n3 -Accredited Examiners and n4 -Civil Aviation Inspectors pilot flight proficiency checks for licensing (2014 -2019) ....................................33 Table 4.27 -ANAC's frequency of pilot flight proficiency checks for licensing -PPL, CPL, and ATPL (2014 -2019) .............................................................................................................34 Table 4.28-ANAC's prior probabilities of being checked by Examiners or Inspectors for licensing -PPL, CPL, and ATPL (2014 -2019) ...................................................................34 Table 4.29 -ANAC's conditional probabilities of being checked by Examiners or Inspectors for licensing -PPL, CPL, and ATPL (2014 -2019) ...................................................................35 Table 4.30 -ANAC's posterior probabilities of being checked by Examiners or Inspectors for licensing -PPL, CPL, and ATPL (2014 -2019) ...................................................................35 Table 4.31 -FAA's descriptive statistics pilot flight proficiency checks for licensing (2014 -2019) ....................................................................................................................................38 Table 4.32 -FAA's Examiners' descriptive statistics pilot flight proficiency checks for licensing (2014 -2019) .......................................................................................................................39 Table 4.33 -FAA's Inspectors' descriptive statistics pilot flight proficiency checks for licensing (2014 -2019) .......................................................................................................................39 Table 4.34 -FAA's Two Proportions z-Test n5 -Examiners and n6 -Inspectors pilot flight proficiency checks for licensing (2014 -2019) .....................................................................40 XIII Table 4.35 -FAA's pilot flight proficiency checks' rates for licensing (2014 -2019) .............41 Table 4.36 -FAA's Examiners and Inspectors pilot flight proficiency checks rates for licensing (2014 -2019) .......................................................................................................................41 Table 4.37 -FAA's Examiners and Inspectors pilot flight proficiency checks rates for licensing (2014 -2019) .......................................................................................................................41 Table 4.38 -FAA's pilot flight proficiency checks for licensing (2014-2019) .........................43 Table 4.39 -ANAC's pilot flight proficiency checks for licensing (2014-2019) ......................43 Table 4.40 -ANAC's Examiners n3 and FAA's Examiners n5 Two Proportions z-Test of pilot flight proficiency checks for licensing (2014 -2019) .............................................................44Table 4.41 -ANAC's Inspectors n4 and FAA's Inspectors n6 Two Proportions z -Test of pilot flight proficiency checks for licensing (2014 -2019) .............................................................45 Table 5.1 -IS -Supplementary Instruction 00-002 history ...................................................59 Table 5.2 -Accredited Examiners' Certification (2016 -2019) .............................................61 Table 5.3 -Accredited Examiners' Oversight Inspections (2016 -2019) .............................62 Table 5.4 -Accredited Examiner's oversight over certification proportion (2016 -2019) .....62 Table 6.1 -Quantity of aircraft owned by ANAC...................................................................69 Table 7.1 -Examiners' oversight activities:transport and DSAdaily subsistence allowance costs .....................................................................................................................................86 Table 7.2 -PCA -principal component analysis...................................................................95 Table 7.3 -Reliability Analysis -Component 1 ....................................................................98 Table 7.4 -Reliability Analysis -Component 2 ....................................................................99 Table 7.5 -Reliability Analysis -Component 3 .................................................................. 100 Table 7.6 -Paired Samples T-Test -Inspectors' Survey ................................................... 101 Table 8.1 -SACI's dataset compared to Internship's last week's shared dataset. .............. 111

Figure 4

 4 Figure 4.1 -ANAC's general pilot flight proficiency checks's histogram (2014 -2019) ........16 Figure 4.2 -ANAC's general pilot flight proficiency checks' disapproval histogram (2014 -2019) ....................................................................................................................................17

Figure 4

 4 Figure 4.3 -ANAC's pilot flight proficiency checks's histogram for licensing(2014 -2019) .17

Figure 4

 4 Figure 4.4 -ANAC's pilot flight proficiency checks' disapproval histogram for licensing (2014 -2019) .................................................................................................................................18 Figure 4.5 -ANAC's general pilot flight proficiency checks performed by Accredited Examiners (2014 -2019) .......................................................................................................................21 Figure 4.6 -ANAC's general pilot flight proficiency checks Accredited Examiners' disapproval (2014 -2019) .......................................................................................................................22 Figure 4.7 -ANAC's general pilot flight proficiency checks performed by Civil Aviation Inspectors (2014 -2019) ......................................................................................................24 Figure 4.8 -ANAC's general pilot flight proficiency checks Inspectors' disapproval (2014 -2019) ....................................................................................................................................25 Figure 4.9 -Comparison of ANAC's Examiners' and Inspectors' disapproval means for general pilot flight proficiency checks (2014 -2019) .........................................................................28 Figure 4.10 -Comparison of ANAC's Examiners' and Inspectors' disapproval means for pilot flight proficiency checks for licensing (2014 -2019) .............................................................32 Figure 4.11 -FAA's pilot licensing proficient flight checks histogram (2014-2019) ...............37 Figure 4.12 -FAA's Instrument rating histogram (2014-2019)..............................................37 Figure 4.13 -SPL -Department of Civil Aviation Personnel organizational structure ..........52 Figure 4.14 -The organizational structure of a typical large PEL Office and its relation to other CAA directorates and offices ................................................................................................53 Figure 4.15 -Concept of accident causation ........................................................................55 Figure 4.16 -Concept of practical drift ................................................................................56 Figure 4.17 -Concept of a safety space ..............................................................................58 Figure 6.1 -The eight critical elements of a State safety oversight system ..........................68

  3 of Annex 19 to the Convention on International Civil Aviation, not associated to ANAC's official opinion. context description of pilot licensing Since the Convention on International Civil Aviation, the contracting States agreed on certain international civil aviation principles at Chicago in 1944. The provision of international standards and procedures was present in Article 37, and the provision of certificates of competency and licenses for pilots was present in Article 32 (Convention on International Civil Aviation, 1944). The provision of certificates of competency and licenses for pilots were present in Article 32 of the Convention (Convention on International Civil Aviation, 1944), and the SARPs -Standards and Recommended Practices for Personnel Licensing were first adopted as minimum standards in 1948 and designated as Annex 1 to the Convention on International Civil Aviation (ICAO, 2018).

(

  ICAO, 2018) The authorized personnel to conduct flight proficiency checks are Examiners (Designated) or Inspectors (Authority). In order to license civil aviation personnel, ANAC follows the Standards and Recommended Practices provided by Annex 1 to the Convention on International Civil Aviation, ensuring confidence in safe air navigation. The flight proficiency checks are provisioned by the Brazilian Federal Law 7.565/1986 -Brazilian Aeronautics Code Article 197 (The Federative Republic of Brazil, 1986) and the RBAC 61, RBAC 90, RBAC 91, RBAC 121, RBAC 135, RBAC 137, RBAC 141, and RBAC 142, according to the specific purpose or operation. To conduct flight proficiency checks, one shall be authorized by the Licensing Authority or designated by the training organization following criteria approved by Authority (ICAO, 2018).

•

  DPE -Designated Pilot Examiner (FAA's outsourced); • Flight Operations CASI -Civil Aviation Safety Inspector (ICAO's nomenclature) (ICAO, 2016); Hereinafter, they will be classified as Inspectors or Examiners. The main difference between them lies in that Inspectors are part of the CAA -Civil Aviation Authority staff, supporting an Effective Inspectorate while performing their safety inspection, certification, and on-going surveillance duties, authorized by the State. In contrast, Examiners are outsourced within the industry to perform very specific duties on behalf of the CAA, e.g., pilot flight proficiency checks. The tree diagram of pilot flight proficiency check is represented by the following diagram: Diagram 1.1 -Tree diagram of pilot flight proficiency checks

  , a comparison could be made to determine similarities and differences among the failure rates of DPEs as compared to OI -Operations Inspectors. Riley's research found that 34% of initial applicants fail the CFI Certified Flight Instructor practical exam. This recommendation was exactly the objective of the current study, by comparing the disapproval rates of pilots in flight proficiency checks among Examiners and Inspectors from ANAC (Brazil) and FAA (United States of America) from 2014 to 2019, focusing on initial exams for the issuance of the following pilot licenses: PPL, CPL, ATPL, Instrument and Flight Instructor ratings. Unlike Riley's motivation based on a high percentage of failure, i.e., 65% (Riley, 2004, p. 1 apud Shain 1998), this research was based on the inspectorate anecdotal perception that the lack of some Authority's aspects such as safety culture, standardization, oversight, and Inspectors' pilot training could be associated to low disapproval rates in the flight proficiency checks. of the ANAC -National Civil Aviation Agency The ANAC is the Brazilian Civil Aviation Authority responsible for regulating, inspecting, licensing the civil aviation personnel and certifying aircraft, training organizations, air transport operators, manufacturers, maintenance organizations, and aerodromes in Brazil. This project's work environment was limited to the SPL -Department of Civil Aviation Personnel, particularly working with the proficiency checks data of the following pilot licences: PPL, CPL, ATPL, as well as instrument and flight instructor ratings. The ANAC's Organizational Chart is presented below: Diagram 2.1 -ANAC's Organizational Chart Source: (ANAC, 2020) [our emphasis] The ICAO has provided an example of the organizational structure of a State in Doc 9734 -Safety Oversight Manual -Part A -Second Edition (ICAO, 2006, pp. 3-9), as described in the following organizational structure: Diagram 2.2 -Example of the organizational structure of a State civil aviation system Source: (Figure 3-1,ICAO, 2006, p. 3-9) The ANAC followed the above-mentioned structure with slight differences, considering that its Personnel Licensing and Airworthiness Departments were directly linked to the Board of Directors, and the Department of Flight Safety Standards is responsible for Aircraft Operations. It is worth mentioning that the Brazilian airspace control is DECEA's -Department of responsibility, while regulating, inspecting, and managing the airspace, overseeing technical and operational organizations, licencing air navigation personnel, and providing Aeronautical Information Service, Air Traffic Management, Search and Rescue, Aeronautical Cartography, Aeronautical Meteorology, Air Navigations Aids, Aeronautical Telecommunication and its Flight Inspections (DECEA, 2020).

  Inspector in the following areas: personnel licensing and training (PEL), including flight proficiency checks, flight crew licensing, ICAO English level assessment, and aircraft operations (OPS), including air transport operator and training organization certification. Through specialized training in ANAC's aviation regulations, familiarity with FAA and EASA, and the implementation to respond to USOAP -Universal Safety Oversight Audit Programme, especially the CEcritical elements CE-6, CE-7, and CE-8, the student have been able to manage and oversee the safety of a significant amount of air transport operators and training organization by the usage of SMS -Safety Management System, CCscompliance checklists, technical audits, findings, and CAPscorrective action plans. This technical training, his graduation in Civil Aviation (Aeronautics), his prior experience as an ATCo -Air Traffic Controller and as Commercial Pilot of airplane (with ATP and helicopter Commercial Pilot theoretical knowledge) led him up to the accomplishment of the assigned tasks. As a subject matter expert, team leader, and Inspector-Instructor, the student has been adopting and implementing ICAO -International Civil Aviation Organization's provisions and SARPs nationwide, particularly those from Annexes 1, 6, 10, 18, and 19 to the Convention on International Civil Aviation.

  Since 2012, the ANAC's data about flight proficiency checks come from SACI -Civil Aviation Information Integrated System available for the Civil Aviation Inspectors. The Examiners and Inspectors constantly feed this database after every flight proficiency check or up to 5 days later as provisioned by the IS 00-002F (ANAC, 2020). Due to the large size of the data (130.243 flight proficiency checks), the extraction had to be done by semesters and then combined. Otherwise, the data would get corrupted.The FAA publishes a comprehensive annual study on the U.S. Civil Airmen Statistics to meet its demands and the demands of other government agencies and the industry. The data range is from 2005 to 2019, and it became more detailed since 2014.

  following Figure represents ANAC's general pilot flight proficiency checks' histogram from 2014 to 2019:

Figure 4

 4 Figure 4.1 -ANAC's general pilot flight proficiency checks's histogram (2014 -2019)

Figure 4

 4 Figure 4.3 -ANAC's pilot flight proficiency checks's histogram for licensing(2014 -2019)

Figure 4

 4 Figure 4.4 -ANAC's pilot flight proficiency checks' disapproval histogram for licensing(2014 -2019)

Figure 4

 4 Figure 4.7 -ANAC's general pilot flight proficiency checks performed by Civil Aviation Inspectors(2014 -2019)

Figure 4

 4 Figure 4.8 -ANAC's general pilot flight proficiency checksInspectors' disapproval (2014Inspectors' disapproval ( -2019) )

  Analyzing the descriptive statistics of ANAC's general pilot flight proficiency checks, the Count is the number of checks (population size) N= 130243; the Sum is the number of Disapproved checks (statistical success events) Σ xi= 836; the mean is μ= 0.006419; the Standard Deviation is σ= 0,07986, and the Population Variance is σ 2 = 0,006378, and the Confidence Level for mean is 95%. The ANAC's Accredited Examiners' descriptive statistics on the disapproval rate of general pilot flight proficiency check from 2014 to 2019 are as follows:

  Analyzing the descriptive statistics of ANAC's Accredited Examiners' general pilot flight proficiency checks, the Count is the number of checks (sample size) n1= 119895; the Sum is the number of Disapproved checks (statistical success events) Σ X1= 635; the mean is 𝑥̅ 1 = 0.006419; the Standard Deviation is s1= 0,072583; and the Sample Variance is 𝑠 1 2 = 0,005268;and the Confidence Level for mean is 95%.The ANAC's Civil Aviation Inspectors' descriptive statistics on the disapproval rate of general pilot flight proficiency check from 2014 to 2019 are demonstrated, as follows:

Figure 4 . 9 -

 49 Figure 4.9 -Comparison of ANAC's Examiners' and Inspectors' disapproval means for general pilot flight proficiency checks(2014 -2019)

  Analyzing the descriptive statistics of ANAC's pilot flight proficiency checks for licensing, the Count is the number of checks (population size) N= 24307; the Sum is the number of Disapproved checks (statistical success events) Σ xi= 204; the mean is μ= 0.008392644; the Standard Deviation is σ= 0,091228011, and the Population Variance is σ 2 = 0,00832255, and the Confidence Level for mean is 95%. The ANAC's Accredited Examiners' descriptive statistics on the disapproval rate of pilot flight proficiency check for licensing from 2014 to 2019 are as follows:

Figure 4 .

 4 Figure 4.10 -Comparison of ANAC's Examiners' and Inspectors' disapproval means for pilot flight proficiency checks for licensing(2014 -2019)

  Examiners with those applied by Civil Aviation Inspectors from 2014 to 2019, this study identified among the population N = 24.307 the frequency of events classified into Examiners or Inspectors and Approved or Disapproved, and applied the Bayes Theorem as the following tables:

Figure 4 .

 4 Figure 4.11 -FAA's pilot licensing proficient flight checks histogram(2014)(2015)(2016)(2017)(2018)(2019)

For

  DPE SEEDs, at least one team member must observe a practical test being administered by the DPE to an actual applicant. This observation must be an exam that has progressed to the flight portion of the test. Unless the observation is being conducted from a required crewmember station, any qualified Operations ASI may conduct the complete evaluation (both ground and flight). NOTE: For the purposes of this section, qualified means that the ASI holds at least a Commercial Pilot Certificate (and the appropriate type rating in the case of an aircraft which requires a type rating) in the same aircraft category and class as the aircraft in which the test is being administered. It is not necessary for the ASI to be current in the aircraft, nor to be a participant in the Flight Standards Flight Program. For ASIs conducting evaluations for DPEs who conduct the majority of their tests in two-place aircraft, and the ASI is acting as the applicant for the flight portion of the evaluation, the ASI must be current in the aircraft per Volume 1, Chapter 3, Section 6. (FAA, 2019) [our emphasis]

  4.10.1.1 -RBAC -Brazilian Civil Aviation Regulation 01 EMDamendment 08 According to Section 01.1 of the RBAC -Brazilian Civil Aviation Regulationamendment 08, the definition of Designated Servant is the accredited person by the Civil Aviation Authority to perform the civil aviation oversight activities, previously denominated INSPAC -Civil Aviation Inspector. 4.10.1.2 -RBAC -Brazilian Civil Aviation Regulation 183 EMDamendment 01 According to Section 183.11 of the RBAC 183 Amendment 1, the ANAC might accredit qualified professionals for executing flight proficiency checks. (ANAC, 2018) According to Section 183.75 (a) of the RBAC 183 Amendment 1, in order to apply the ANAC required checks, the societies may only use people that meet the following minimal requirements: … (2) appropriate CMA -Medical Certificate; and (3) compatible licences and ratings of applied checks (ANAC, 2018). According to Section 183.77, (b) of the RBAC 183 Amendment 1, the control and expiry date of the CMA and ratings is the societies' responsibility, as well as forbid the Examiners' prerogatives after the expiry date of the CMA and ratings (ANAC, 2018). It is worth mentioning that the Examiner's definition is any person that is accredited or designated by ANAC to conduct pilot proficiency checks, according to Item 4.1 of IS 00-002F (ANAC, 2020), which includes the Inspectors. Although the ANAC requires that the Accredited Examiners hold valid CMA and ratings, the same ANAC does not maintain valid its own Inspectors' CMAs and ratings, since its foundation, back in 2005. On the contrary, the ANAC revoked, by the Resolution 422 of 2 nd May 2017, regulations that previously established INSPAC licences and ratings and that the Flight Inspectors must keep flight currency at least every 30 days, while recommended every 15 days (IAC -Civil Aviation Instruction 3201-0198) (DAC, 1998). 4.10.1.3 -RBAC -Brazilian Civil Aviation Regulation 61 EMDamendment 13 According to Section 61.2 of the RBAC 61 Amendment 13, the ANAC defines Rating as an authorization associated with a licence or certificate, where qualifications and expiry date are specified. (ANAC, 2020) According to Section 61.3, (i), (2), of the RBAC 61 Amendment 13, there is the provision of the Accredited Examiners and Civil Aviation Inspectors (INSPAC) being properly rated and qualified. (ANAC, 2020) According to Section 61.17 of the RBAC 61 Amendment 13, the pilot's licence is permanent, and the holder's prerogative may only be exercised when the following requirements are met: (1) valid CMA; (2) valid ratings; (3) recent flight experience (90 days). (ANAC, 2020) According to Section 61.31, (c), (5), (ii), the hours of pilot flight proficiency check must be registered by the Civil Aviation Inspector (INSPAC) or the Accredited Examiner responsible for the check in the CIV -Logbook and CIV Digital -Electronic Logbook. (ANAC, 2020) 4.10.1.4 -RBAC -Brazilian Civil Aviation Regulation 67 EMD -Amendment 4 According to Section 67.15, (a), (b), and (c) of RBAC 67 Amendment 4, the Medical Certificate must obey the expressed validities, must take into account the requirements compliance within RBAC 67 and the Medical Certificate holder must report to ANAC any psychophysical diminishment which could impede the exercise of licenses or ratings prerogatives without affecting safety. (ANAC, 2020) 4.10.1.5 -IS -Supplementary Instruction 00-002 According to Item 5.2.3.2 of the IS -Supplementary Instruction 00-002 A and B, the checks are conducted in a manner to allow the Inspector and Examiner to analyze all the technical aspects, considered as a minimum to the exercise of the licences prerogatives, in the manners of the ratings qualifications. (ANAC, 2011) (ANAC, 2012)

  ANAC's organization restructure happened in August 2020 and according to Resolution 581 -ANAC's Internal Regulation. The PEL Department were created including the areas: Technical Advisory (GTAS), Technical Regulations (GTNO), Personnel Licensing (GCEP) and Approved Training Organization (GCOI).

Figure 4 .

 4 Figure 4.13 -SPL -Department of Civil Aviation Personnel organizational structure

  Figure 4.16 -Concept of practical drift

  Thus, this research seems to corroborate with the sense of lack of safety culture, especially whilst combined with the safety research results of general disagreement level at 38,8%, which seems to contrast with the 95.58% of EI -Effective Implementation of USOAP-CMA -Continuous Monitoring Approach of ICAO. According to the Paragraph 3.1.1 of the Doc 9859 -Safety Management Manual, Fourth Edition: Safety Culture has been described as 'how people behave in relation to safety and risk when no one is watching'. It is an expression of how safety is perceived, valued and prioritized by management and employees in an organization, and is reflected in the extent to which individuals and groups are: a) aware of the risks and known hazards faced by the organization and its activities; b) continuously behaving to preserve and enhance safety; c) able to access the resources required for safe operations; d) willing and able to adapt when facing safety issues; e) willing to communicate safety issues; and f) consistently assessing the safety related behaviours throughout the organization. (ICAO, 2018) [our emphasis] According to the Paragraph 3.2.4 of the Doc 9859 -Safety Management Manual, Fourth Edition:3.2.4 A positive safety culture relies on a high degree of trust and respect between personnel and management.Time and effort are needed to build a positive safety culture, which can be easily damaged by management decisions and actions, or inactions. Continuous effort and reinforcement are needed.When leadership actively endorses safe practices, it becomes the normal way of doing things. The ideal situation is a fully implemented and effective SSP/SMS and a positive safety culture. Hence, an organization's safety culture is often seen as a reflection of the maturity of its SSP/SMS. Effective safety management empowers a positive safety culture and a positive safety culture empowers effective safety management.(ICAO, 2018) [our emphasis]

Figure 4 .

 4 Figure 4.17 -Concept of a safety space

  Regarding Examiners' training (supposedly initial), the ANAC has trained 261 Examiners in 2018 and 288 Examiners in 2019, with approval rates of 68.2% and 71.8%, respectively, according to the information available at the ANAC's TRAINAIR PLUS Training Center portal (ANAC, 2020). According to Section 141.91 and 141.93 of RBAC 141, the published requirements demand 500 hours of instruction for airplane (or 250 hours for helicopter), an initial training, a flight proficiency check and subsequent certification administrative process and flight proficiency check every two years, contrasting with the recurrent training requested by the FAA in the Order 8900.2C: 3. Recurrent Training. Each designee is encouraged to attend safety meetings, aviation seminars, and other programs that contribute to the technical skills required for the designation. The designee is responsible for ensuring their training currency. Only after successful completion of the initial training (or initial standardization seminar) can the designee register for the recurrent training seminar on the DRS. Attendance at a recurrent training seminar is mandatory. … The designee must forward a copy of the Certificate of Completion to the managing FAA office. Note: Designees may complete the Designated Pilot Examiner Initial Standardization seminar, conducted by the Designee Standardization Branch, in lieu of the recurrent training seminar to meet the recurrent requirement set forth in paragraph 3 above.

  Figure 6.1 -The eight critical elements of a State safety oversight system

  , and transportation means, as applicable, to the technical personnel to enable them to perform their safety oversight functions effectively and in accordance with established procedures in a standardized manner. (ICAO, 2016) [our emphasis] According to Paragraph 1.1.2 CAA OBLIGATIONS TO SUPPORT AN EFFECTIVE INSPECTORATE of Doc 10070 -Manual on the Competencies of Civil Aviation Safety

  the Civil Aviation Inspector's flight duties According to Section 6 of Appendix 1 of Annex 19 to the Convention on International Civil Aviation, the State shall ensure that individuals meet the requirements to exercise the privileges of a licence: States shall implement documented processes and procedures to ensure that individuals and organizations performing an aviation activity meet the established requirements before they are allowed to exercise the privileges of a licence, certificate, authorization or approval to conduct the relevant aviation activity. (ICAO, 2016) [our emphasis] According to Chapter 10 of Annex 6 to the Convention on International Civil Aviation, the flight operations officer shall be licensed in accordance with the provisions of Annex 1 to the Convention on International Civil Aviation: When the State of the Operator requires that a flight operations officer/flight dispatcher, employed in conjunction with an approved method of control and supervision of flight operations, be licensed, that flight operations officer/flight dispatcher shall be licensed in accordance with the provisions of Annex 1.

  Surveillance obligations -(CE-7).The State shall proactively assure that the requirements are continuously met by the aviation personnel, which includes the surveillance of the designated personnel to perform safety oversight functions on its behalf, as provisioned by Annex 1, 6, and Annex 19 to the Convention on International Civil Aviation. According to Section 8 of Annex 1 -Personnel Licensing -Twelfth Edition, States shall maintain an effective oversight programme: Contracting States shall maintain an effective oversight programme of the approved training organization to ensure continuing compliance with the approval requirements. (ICAO, 2018) [our emphasis]According to Section 1.2 of Annex 1 -Personnel Licensing -Twelfth Edition, the Contracting State shall ensure the validity of issued licences. Therefore, flight proficiency checks are used to grant its privileges, to maintain the holder's competency, and shall be conducted by personnel authorized by the Licensing Authority: 1.2.5 Validity of licences 1.2.5.1 A Contracting State, having issued a licence, shall ensure that the privileges granted by that licence, or by related ratings, are not exercised unless the holder maintains competency and meets the requirements for recent experience established by that State. … Note 1.-Until 2 November 2022, the maintenance of competency of flight crew or remote flight crew members, engaged in commercial air transport operations, may be satisfactorily established by demonstration of skill during proficiency flight checks completed in accordance with Annex 6. (ICAO, 2018) [our emphasis] According to Section 9 of Annex 1 -Personnel Licensing -Twelfth Edition, with the purpose of conducting flight proficiency checks, one shall be authorized by the Licensing Authority or designated by the training organization in accordance with criteria approved by the Licensing Authority: 9. Evaluation and checking When a State has authorized an approved training organization to conduct the testing required for the issuance of a licence or rating, the testing shall be conducted by personnel authorized by the Licensing Authority or designated by the training organization in accordance with criteria approved by the Licensing Authority. (ICAO, 2018) [our emphasis] It is worth mentioning that the required flight proficiency checks face several constraints caused by the relationship between ANAC and DECEA -Department of Airspace Control, which publishes NOTAM -Notices to Airmen or AIP -Aeronautical Information Publication remark that prohibit ANAC's pilot flight proficiency checks. DECEA's position jeopardizes aviation safety when it prohibits and interferes in the required use of certain navigation aids or operational maneuvers during the pilot flight proficiency checks, as the following examples:

Figure 6 . 2 -

 62 Figure 6.2 -Example of NOTAM -Notices to Airmen prohibiting training and ANAC's pilot flight proficiency check

…

  The Examiner shall also, within 5 days of the check accomplishment, fill the electronic PEF -Pilot Evaluation Form (FAP) at the website https://sistemas.anac.gov.br/saci/.(ANAC, 2020)[our emphasis]

Figure 6

 6 Figure 6.5 -ANAC's pilot flight proficiency check per day(2014)(2015)(2016)(2017)(2018)(2019)

  results obtained came from the assessment of CE -Critical Elements provisioned by Annex 19 to the Convention on International Civil Aviation: • Qualified technical personnel -(CE-4) • Technical guidance, tools and provision of safety-critical information -(CE-5) • Licensing, certification, authorization and approval obligations -(CE-6) • Surveillance obligations -(CE-7) (ICAO, 2016) The requirements for technical personnel qualification were detailed, based on the Annexes 1, 6, and 19 to the Convention on International Civil Aviation and several SARPs present in the ICAO Docs. The need for Inspectors' qualification, rating, and recent experience was emphasized to the point of demanding urgent implementation of the PCIV -ANAC's Flight Inspectors Training Programme. Through the implementation of the proposed training, the ANAC would increase the flight Inspectors' human resources by at least 57%, without any additional hiring. The pilot flight proficiency check and oversight capacities would increase largely if the required oversight was put in practice or the outsourcing policy was challenged by this research findings.The enhancement of tools used by the flight Inspectors was proposed by the digitalization of job aids such as the PEF -Pilot Evaluation Form, which helps them on-site and, additionally, would collect comprehensive data to feed the system and improve oversight capabilities.The licensing and rating of the flight Inspectors was proposed and justified based on the requirements and operational needs to properly perform flight activities, especially those that require the use of a two-seat aircraft.The Proficiency Check Dashboard was proposed as a tool to assist the managers in their decision-making processes as well as allowing subsequent analysis to identify possible safety threats or enhancements by the technical personnel and for the data-driven oversight and standardization planning. The oversight intensification proposed was possible and justifiable based on the collected pieces of evidence of the statistical differences between ANAC's Examiners and Inspectors, while their disapproval rates were extremely different from the FAA's disapproval rates, meaning that both Examiners and Inspectors need improvements towards the assessment of the pilot flight proficiency checks. It is worth mentioning that according to Section 61.197, (a), (2), (ii) of 14 CFR PART 61, the flight instructor who has served as a company check pilot, chief flight instructor, company check airman, or flight instructor in a part 121 or part 135 operation, or in a position involving the regular evaluation of pilots are granted certificate renewal without passing a practical test (FAA, 2020): §61.197 Renewal requirements for flight instructor certification. (a) A person who holds a flight instructor certificate that has not expired may renew that flight instructor certificate by-(1) Passing a practical test for-(i) One of the ratings listed on the current flight instructor certificate; or (ii) An additional flight instructor rating; or (2) Submitting a completed and signed application with the FAA and satisfactorily completing one of the following renewal requirements-(i) A record of training students showing that, during the preceding 24 calendar months, the flight instructor has endorsed at least 5 students for a practical test for a certificate or rating and at least 80 percent of those students passed that test on the first attempt. (ii) A record showing that, within the preceding 24 calendar months, the flight instructor has served as a company check pilot, chief flight instructor, company check airman, or flight instructor in a part 121 or part 135 operation, or in a position involving the regular evaluation of pilots. (FAA, 2020) [our emphasis] The alternative presented above, although not by any manner of means recommended by the author, could mitigate the lack of check and ratings among ANAC's Inspectors, by at least being verified and granted by the Authority. Fortunately, the PCIV's first round of technical training for a normal category jet was held at a Training Center as ground school and as flight training using a FFS -Full Flight Simulator, which does fulfill the rating requirements of proficiency flight check in a Training Center, contained in Sections 61.213 and 61.215 of RBAC 61 EMD13. As an enhancement proposal to the implementation of the PCIV -ANAC's Flight Inspectors Training Programme and further Inspectorate's training planning and control using it as a heatmap, the following diagram was presented, containing recommended additional training such as Emerging Technology, e.g. avionics (FMS -Flight Management System, HUD -Head-up Display, EFVS -Enhanced Flight Vision Systems) or Performance-Based Navigation, as required to the authority by the ICAO's Doc 10070, in order to support the accomplishment of the Inspectors' tasks, and recommended for the operators according to ATTACHMENT 2.B of Annex 6 -Part II: Diagram 7.1 -Flight Inspector Training Honeycomb 10 -ASEL, AMEL, and Turboprop 1.4 PREPARING CASIs FOR THE FUTURE In order to comply with their oversight obligations, States need to ensure that: a) they have enough CASIs (tools are available on website xxx to support States in their CASI manpower planning process); and b) their CASIs are qualified and competent. This manual focuses on the latter in order to enhance the effectiveness of a State's oversight activities today and to better prepare for their oversight needs of the future. (ICAO, 2016) [our emphasis] Unconsciously, the Diagram 7.1, which recommended additional flight training, resembles the shape of a butterfly, a perfect metaphor for the Inspectors' pilot training change of course, related to the training gap during the last 15 years of the ANAC's existence, which represented the chrysalis stage, waiting to expand its wings. Considering that the required organizational readiness is not yet fulfilled for Technical Training and subsequent Personnel Licensing, the proposed solution will be focused on Oversight Intensification and Standardization. 10 ASES -Airplane Single Engine Sea, ASEL -Airplane Single Engine Land, AMEL -Airplane Multi-Engine Land, VFR -Visual Flight Rules, IFR -Instrument Flight Rules, PBN -Performance-based Navigation, W&B -Weight and Balance, OPS 91 -General Aviation Operations, OPS 141 -Training Organization Operations, OPS 135 -Air Carrier Operations, GS -Ground School.

  Considering 2018 and 2019, there were 373 and 392 active Examiners, respectively, as well as 34 active Inspectors during both years. In 2019, there were 41 oversight activities focusing on the training organizations' Examiners, which represents roughly 10%. The Examiners' oversight activities are divided into planned and on-demand, and the 2020's Annual Work Plan took into account an average of transport cost and DSAdaily subsistence allowance, from SCDP -Allowance and Fare System, as the following table:

  to their professional e-mails. In spite of having only 51 Inspectors in the studied dataset, or 34 active Inspectors in 2019, the population that met the criteria for being Flight Inspector was 63, including those who decided to stop conducting pilot flight proficiency check, mainly for safety reasons, and excluding those who are not currently employed by the Civil Aviation Agency.Therefore, the survey asked the ANAC's Inspectors, based on their professional experience, their perception related to the Inspectors and those related to the Examiners, though identical in content, except for the additional questions related to surveillance. The final survey was comprised of 48 items, including 1 Multi select Multiple Choice Question for Operations Specifications, 38 5-point Likert Scales for the level of agreement on statements, 2 open boxes for statement inclusion and their associated 5-point Likert Scales for the level of agreement, 3 Single select Multiple Choice Questions for professional experience validity and 2 Multi select Multiple Choice Questions for enhancement proposals to fill the perceived safety gap.

Figure 7 . 1 -

 71 Figure 7.1 -Years of professional experience as pilot.

Figure 7 Figure 7

 77 Figure 7.2 -Quantity of total flight hours.

TRN2-

  The Flight Inspectors receive equipment Flight Training from ANAC. TRN3 -The Flight Inspectors receive Differences Training from ANAC. OPS1 -The Flight Inspectors possess valid rating in the equipment. OPS2 -The Flight Inspectors possess currency (E.g.: 90 days -Day, Night, Tailwheel; 6 months -IFR). OPS3 -The Flight Inspectors possess Medical Certificate for real flight. OPS4 -The Flight Inspectors operate in PBN -Performance Based Navigation. OPS5 -The Flight Inspectors have difficulties to visualize the panel flight instruments due to aircraft's ergonomics. OPS6 -The Flight Inspectors have difficulties to perform the required flight check procedures due to ATC -Air Traffic Control restrictions (E.g.: NOTAM -… CAP1 -The Flight Inspectors receive CRM -Crew Resource Management training from ANAC. CAP2 -The Flight Inspectors receive PBN -Performance Based Navigation training from ANAC CAP3 -The Flight Inspectors receive 3 (three) OJT -On-the-job training from ANAC. CAP4 -The Flight Inspectors receive advanced training from ANAC (Ex.: Workshop, Specialization, Master, Doctorate) STD1 -The objective criteria for flight check failing are adequate. STD2 -The subjective criteria for flight check failing are adequate. STD3 -The flight check procedures are standardized.

Figure 7

 7 Figure 7.10 -Checks per day by CANAC.

8. 5 -

 5 What are the remaining actions to achieve that would be useful to do in order to complete the study?The benefits of the use of qualified technical personnel in charge of the intensified oversight duties rely on the proper implementation of the PCIV -ANAC's Flight Inspectors Training Programme. Also, the mandatory filing of the Electronic PEF -Pilot Evaluation Form by the RBAC 121 and 135 operators, as provisioned by Item 12.2 of the IS 00-002F, would allow a comprehensive analysis of the pilot flight proficiency check of all operators.8.6 -How could this work be reused?Although this work was a 2014 to 2019 pilot flight proficiency check rates' screenshot from the ANAC and the FAA, not only it could be reused, but also its deliverables could be reused to better plan, technical training, standardization, oversight or accreditation activities, using updated data contained in the Proficiency Check Dashboard delivered by this study.Another incentivized reuse of this work could be to manage all the other categories of Accredited Examiners such as Aviation Medical Examiners, Mechanic Examiners, Aircraft Dispatch Examiners, Aircrew Examiners, Manufacturing Examiners, Project Examiners, Airworthiness Examiners, and English Proficiency Examiners.

  It is highly recommended similar studies, and implementation of the proposed solutions to the Air Carriers' and Airlines' Examiners, based on the brief observations of the SISHAB's SAE -Exam Authorization Request reprovals due to low pilot flight proficiency check's disapproval, not focused in this research because of the above-mentioned regulation noncompliance and data access difficulty. Another incentivized reuse of this work could be to manage all the other categories of Accredited Examiners such as Aviation Medical Examiners, Mechanic Examiners, Aircraft Dispatch Examiners, Aircrew Examiners, Manufacturing Examiners, Project Examiners, Airworthiness Examiners, and English Proficiency Examiners, focusing on data-driven analysis, standardization, and surveillance, or better, developing and implementing an Accredited Examiner management system such as the FAA's DMS -Designee Management System. Although this work was a 2014 to 2019 pilot flight proficiency check's rates' screenshot from the ANAC and the FAA, not only it could be reused, but also its deliverables could be reused to better plan standardization, oversight or accreditation activities, by using updated data direct from the database and the Proficiency Check Dashboard delivered by this study. Once the Brazilian and the American pilot flight proficiency checks' status and the Examiners' and Inspectors' rates were known, this study researched and analysed the difficulties in Section 4.10 -, and proposed three possible solutions to solve them: Technical Training, Standardization and Surveillance. Regarding Technical Training, the PCIV -ANAC's Flight Inspectors Training

Table 4 .

 4 1 -Results of ANAC's general pilot flight proficiency checks (2014 -2019) ............16 Table 4.2 -Results of ANAC's pilot flight proficiency checks for licensing (2014 -2019) .....18 Table 4.3 -Results of ANAC's PPLprivate pilot licence approval by Examiners

Table 4 .

 4 11 -Results of ANAC's Instrument rating by Examiners (2014-2019) .....................20

Table 4 .

 4 12 -Results of ANAC's Instrument rating by Inspectors (2014-2019) ......................20

Table 4 .

 4 13 -Results of ANAC's Flight Instructor rating by Examiners (2014-2019) ..............20

Table 4 .

 4 14 -Results of ANAC's Flight Instructor rating by Inspectors (2014-2019) ..............21

Table 4 .

 4 15 -ANAC's PPLprivate pilot licence check by Examiners -2016's decrease ...21

Table 4 .

 4 16 -Results of ANAC's general pilot flight proficiency checks performed by Accredited

  The checks for license and ratings can be applied by Examiners outsourced from the industry such as: Approved Training Organisations (RBAC 141 and RBAC 142) and Air Operators(RBAC 90, RBAC 91, RBAC 121, RBAC 135 and RBAC 137), or by Inspectors,

	restricted by their experience level according to internal procedures, contained in MPR/SPO-
	002-R02 (ANAC, 2018). The exams result could be Approved, Disapproved or Discontinued,
	according to Items 6.3 and 11.4 of IS 00-002F (ANAC

  literature research was based on the Convention on International Civil Aviation and

	its Annex 1, Annex 6, Annex 19, ICAO's Doc Series such as Doc 7300, Doc 8335, Doc 9374,
	Doc 9379, Doc 9859, Doc 9941 and Doc 10070, national and foreign regulations, RBAC 01,
	RBAC 61, RBAC 141, RBAC 183, IN 137, IS 183-001D, IS 00-002 (A to F), 14 CFR PART 61,
	14 CFR PART 183, ORDER 8000.95, ORDER 8900.1, ORDER 8900.2B, ORDER 8900.2C,
	Notice 8900.442, and Notice 8900.485.

4.1 -The pilot flight proficiency check's rates analysis

	This study used quantitative analysis for data collection of ANAC's pilot flight
	proficiency check from 2014 to 2019. The data were collected from the SACI -Civil Aviation
	Information Integrated System and represents a total of 130.243 pilot flight proficiency checks,
	most of them from the general aviation (RBAC 91) and for licensing or rating purposes, both
	initial or recurrent (RBAC 141 and RBAC 142), hereinafter called as general.
	The purpose of this collection was to determine the general flight proficiency check's
	disapproval rate of the Brazilian applicants, to provide situational awareness of this critical
	task, to build a foundation in order to assist the Managers in their decision-making processes
	as well as allowing subsequent analysis to identify possible safety threats or possible
	enhancements by the technical personnel.
	The pilot flight proficiency check's general database had no distinction between
	Accredited Examiners and Civil Aviation Inspectors; thus, in order to divide into these two
	groups, it was necessary to individually link the Inspectors' names contained in a few different
	civil servant training lists from the Operations Focal Point of HR -Human Resources to their
	respective ANAC's code. This process made it possible to merge lists containing only names
	into a list of the Civil Aviation Inspectors' operations information, useful for the data extraction
	and division into two groups.
	Therefore, the two groups were populated by 698 Examiners and 51 Inspectors. Due
	to a system error, there is another category (Others) included in the Examiners' group as 1
	Examiner (with a total of 1.986 checks).
	It is worth mentioning that the classification into Civil Aviation Inspectors or Accredited
	Examiners might not be exhaustive due to the criteria established by the Operations Focal

Table 4

 4

	.1 -Results of ANAC's general pilot flight proficiency checks (2014 -2019)
	Approved	Percent	Disapproved	Percent	Total
	129407	99,36%	836	0,64%	130243
			Source: own elaboration.
	The disapproval rate of 0.64% confirmed the Inspectorate's sense of presumed low
	disapproval rate that motivated this study, among other issues.
	The ANAC's general pilot flight proficiency checks' disapproval rate from 2014 to 2019
	is expressed according to the following table, and its disapproval mean dropped year after
	year, until it reached 0.41% in 2019, as follows:

Table 4

 4

	.2 -Results of ANAC's pilot flight proficiency checks for licensing (2014 -2019)
	Approved	Percent	Disapproved	Percent	Total
	24103	99,16%	204	0,84%	24307
			Source: own elaboration.

Table 4

 4

	.4 -Results of ANAC's PPL -private pilot licence approval by Inspectors (2014-2019)
	Inspectors Approved	Inspectors Disapproved	Inspectors Total
	33	1	34
	97.06%	2.94%
		Source: own elaboration.

Table 4

 4

	.7 -Results of ANAC's ATPL -airline transport pilot licence by Examiners (2014-2019)
	Examiner Approved	Examiner Disapproved	Examiner Total
	552	14	566
	97.53%	2.47%
		Source: own elaboration.
	Table 4.8 -Results of ANAC's ATPL -airline transport pilot licence by Inspectors (2014-2019)
	Inspectors Approved	Inspectors Disapproved	Inspectors Total
	810	22	832
	97.36%	2.64%
		Source: own elaboration.
	4.2

.4 -The ANAC's Rotorcraft rates (PPL -private pilot licence , CPL - commercial pilot licence, ATPL -air transport pilot licence)

	The ANAC's Rotorcraft -PPL -private pilot licence, CPL -commercial pilot licence,
	ATPL -air transport pilot licence rates are demonstrated, as follows:

Table 4

 4

	.9 -Results of ANAC's Rotorcraft -PPL -private pilot licence, CPL -commercial pilot licence,
	ATPL -air transport pilot licence rates by Examiners (2014-2019)
	Examiner Approved	Examiner Disapproved	Examiner Total
	4160	30	4190
	99.28%	0.72%
		Source: own elaboration.
	Table 4.10 -Results of ANAC's Rotorcraft -PPL -private pilot licence, CPL -commercial pilot
	licence, ATPL -air transport pilot licence rates by Inspectors (2014-2019)
	Inspectors Approved	Inspectors Disapproved	Inspectors Total
	25	1	26
	96.15%	3.85%
		Source: own elaboration.

Table 4

 4

	.11 -Results of ANAC's Instrument rating by Examiners (2014-2019)
	Examiner Approved	Examiner Disapproved	Examiner Total
	9103	36	9139
	99.61%	0.39%
		Source: own elaboration.
	Table 4.12 -Results of ANAC's Instrument rating by Inspectors (2014-2019)
	Inspectors Approved	Inspectors Disapproved	Inspectors Total
	425	8	433
	98.15%	1.85%
		Source: own elaboration.

Table 4

 4

	.14 -Results of ANAC's Flight Instructor rating by Inspectors (2014-2019)
	Inspectors Approved	Inspectors Disapproved	Inspectors Total
	534	4	538
	99.13%	0.87%
		Source: own elaboration.
	4.3

-The ANAC's Accredited Examiners' rate

	The Accredited Examiners (sample n1), outsourced by ANAC as explained in Chapter
	1 -, performed the pilot flight proficiency checks, from 2014 to 2019, as the following:
	Figure 4.5 -ANAC's general pilot flight proficiency checks performed by Accredited Examiners (2014
	-2019)
	Source: own elaboration.
	Analyzing Figure 4.5, there was a decrease in the Examiners' flight check in 2016,
	larger than the expected decrease of all flight checks for that year, as shown in the table below:

Table 4

 4

	.15 -ANAC's PPL -private pilot licence check by Examiners -2016's decrease
		2015	2016
	Checked by Examiners	23465	17305
	Percent	91%	86%
	Source: own elaboration.

-The ANAC's Civil Aviation Inspectors' rate

	The table above clearly shown a decreasing tendency in the Examiners' disapproval
	rate since 2014, which approximately kept stable at 0,35%.
	4.4 The Brazilian Civil Aviation Inspector is a professional, usually a civil servant,
	accredited by the CAA -Civil Aviation Authority as provisioned by the Article 197 of the
	Table 4.16 -Results of ANAC's general pilot flight proficiency checks performed by Accredited Brazilian Federal Law 7.565/1986 -Brazilian Aeronautics Code (The Federative Republic of
	Examiners (2014-2019) Brazil, 1986) and by the Article 1, Sections XIX and XX of the Brazilian Federal Law
	Examiner Approved (Percent) 10.871/2004 (The Federative Republic of Brazil, 2004), which performs safety oversight duties Examiner Examiner Disapproved (Percent) Total provisioned by the Article 66 of the Brazilian Federal Law 7.565/1986 -Brazilian Aeronautics
	Code (The Federative Republic of Brazil, 1986), including the pilot flight proficiency checks,
	119260 and defined as aircraft crew by the Article 156 of the Brazilian Federal Law 7.565/1986 -635 119895
	99.33% Brazilian Aeronautics Code (The Federative Republic of Brazil, 1986) while providing the 0.67%
	Source: own elaboration. specialized air services provision by Section VIII of the Article 201 of the Brazilian Federal
	Law 7.565/1986 -Brazilian Aeronautics Code (The Federative Republic of Brazil, 1986) and
	The table above shown that Examiners approved 119.260 candidates (99.33%), while as provisioned by the Article 5, Section IV of the Brazilian Federal Law 13.475/2017 -Aircrew
	disapproved 635 (0.67%). (The Federative Republic of Brazil, 2017), requiring licence, rating and medical certificate to
	The ANAC's general pilot flight proficiency checks disapproved by Examiners were be accredited to their duties as provisioned by the Article 159 of the Brazilian Federal Law
	represented, as follows: 7.565/1986 -Brazilian Aeronautics Code (The Federative Republic of Brazil, 1986), in order
	to abide to International Treaties, Conventions and Acts as provisioned by the Article 1 of the
	Figure 4.6 -ANAC's general pilot flight proficiency checks Accredited Examiners' disapproval (2014 -2019) Brazilian Federal Law 7.565/1986 -Brazilian Aeronautics Code (The Federative Republic of
	Brazil, 1986).
	1,20%	1,12%
	1,00%
	0,80%
	0,60%		0,52%
	0,40%			0,35%	0,33%	0,36%	0,35%
	0,20%
	0,00%
		2014	2015	2016	2017	2018	2019
			Examiners' disapproval	Cumulative Moving Average
			Source: own elaboration.

From 2014 to 2019, the general pilot flight proficiency checks performed by ANAC's Civil Aviation Inspectors were according to the following figure:

Table 4

 4

	Inspector Approved (Percent)	Inspector Disapproved (Percent)	Inspector Total
	10147	201	10348
	(98.1%)	(1.9%)

.17 -ANAC's general pilot flight proficiency checks performed by Civil Aviation Inspectors

(2014 -2019)

Source: own elaboration.

The table above showed that Inspectors approved 10.147 candidates (98.1%), while disapproved 201 (1.9%). The ANAC's general pilot flight proficiency checks performed by

ANAC's Civil Aviation Inspectors (2014 -2019)

were as follows:

Table 4

 4 Section 17.13 of the RBHA -Brazilian Requirements for Aeronautical Certification 17, revoked by the Resolution 472 of 6 th June 2018.Additionally to lack of Inspectorate's technical training, there is lack of rating, medical certificate, and safe work environment, which represents that ANAC's safety culture might be hazardous to aviation and might threaten its safety.

		2015	2016	2019
	Checked by Inspectors	2243	2838	497
	Percent	8.7%	14.1%	2.6%

.18 -ANAC's PPLprivate pilot licence check by

Inspectors -2016's increase and 2019's decrease

Source: own elaboration.

The Figure and

Table above showed the peak of the pilot flight proficiency check performed by the Inspectors in 2016, reaching the largest percentage of 14.1% and the lowest point in 2019, reaching the smallest percentage of 2.6%, which only represents 17.5% of the 2016's number of checks.

This drastic decrease is due to ANAC's efforts to outsource the pilot flight proficiency checks. Many Inspectors also avoided performing pilot flight proficiency checks due to the Safety Recommendations A -134/CENIPA/2014 -05 and A -134/CENIPA/2014 -06 proposed by CENIPA -Accident Investigation Authority (CENIPA, 2014) who pointed out the Inspectorate's lack of training for conducting pilot flight proficiency check, even being

  The ANAC's descriptive statistics on the disapproval rate of general pilot flight proficiency check from 2014 to 2019 are demonstrated, as follows:

	3,50%
						3,08%
	3,00%
	2,50%				2,10%		2,41%
	2,00%	1,76%	1,83%	1,59%
	1,50%
	1,00%
	0,50%
	0,00%
		2014	2015	2016	2017	2018	2019
			Inspectors' disapproval	Cumulative Moving Average

Table 4 .

 4 19 -ANAC's descriptive statisticsgeneral pilot flight proficiency checks(2014 -2019)

	Mean	0,006419
	Standard Error	0,000221
	Median	0
	Mode	0
	Standard Deviation	0,07986
	Population Variance	0,006378
	Kurtosis	150,8054
	Skewness	12,36135
	Range	1
	Minimum	0
	Maximum	1
	Sum	836
	Count	130243
	Confidence Level (95,0%) 0,000434

Source: own elaboration.

Table 4 .

 4 20 -ANAC's Accredited Examiners' descriptive statistics of general pilot flight proficiency checks(2014 -2019)

	Mean	0,005296
	Standard Error	0,00021
	Median	0
	Mode	0
	Standard Deviation	0,072583
	Sample Variance	0,005268
	Kurtosis	183,8241
	Skewness	13,63162
	Range	1
	Minimum	0
	Maximum	1
	Sum	635
	Count	119895
	Confidence Level (95,0%) 0,000411

Source: own elaboration.

Table 4

 4

	Mean	0,019424
	Standard Error	0,001357
	Median	0
	Mode	0
	Standard Deviation	0,138017
	Sample Variance	0,019049
	Kurtosis	46,52545
	Skewness	6,965376
	Range	1
	Minimum	0
	Maximum	1
	Sum	201
	Count	10348
	Confidence Level (95,0%) 0,00266

.21 -ANAC's Civil Aviation Inspectors' descriptive statistics of general pilot flight proficiency checks

(2014 -2019)

Source: own elaboration.

Table 4

 4

	.22 -ANAC's Two Proportions z -Test n1 -Accredited Examiners and n2 -Civil Aviation
	Inspectors general pilot flight proficiency checks (2014 -2019)
		ANAC's	ANAC's
		Examiners	Inspectors
	Mean	0,005296301	0,019424043
	Known Variance	0,005268	0,019049
	Observations	119895	10348
	Hypothesized Mean Difference	0
	z	-10,29064864
	P(Z<=z) one-tail	0,0000000000
	z Critical one-tail	1,644853627
	P(Z<=z) two-tail	0,0000000000
	z Critical two-tail	1,959963985

Source: own elaboration.

Table 4 .

 4 23 -ANAC's descriptive statistics of pilot flight proficiency checks for licensing(2014 -2019)

	Mean	0,008392644
	Standard Error	0,000585144
	Median	0
	Mode	0
	Standard Deviation	0,091228011
	Population Variance	0,00832255
	Kurtosis	114,1841579
	Skewness	10,77843972
	Range	1
	Minimum	0
	Maximum	1
	Sum	204
	Count	24307
	Confidence Level (95,0%) 0,001146918

Source: own elaboration.

Table 4 .

 4 24 -ANAC's Examiners' descriptive statistics of pilot flight proficiency checks for licensing (2014 -2019)The ANAC's Civil Aviation Inspectors' descriptive statistics on the disapproval rate of pilot flight proficiency check for licensing from 2014 to 2019 are as follows:

	Mean	0,007523646
	Standard Error	0,000566603
	Median	0
	Mode	0
	Standard Deviation	0,086413897
	Sample Variance	0,007467362
	Kurtosis	127,9496269
	Skewness	11,39906247
	Range	1
	Minimum	0
	Maximum	1
	Sum	175
	Count	23260
	Confidence Level (95,0%) 0,001110579

Source: own elaboration.

Table 4 .

 4 25 -ANAC's Inspectors' descriptive statistics of pilot flight proficiency checks for licensing(2014 -2019)

	Mean	0,02769819
	Standard Error	0,00507412
	Median	0
	Mode	0
	Standard Deviation	0,16418509
	Sample Variance	0,02695674
	Kurtosis	31,286873
	Skewness	5,7642963
	Range	1
	Minimum	0
	Maximum	1
	Sum	29
	Count	1047
	Confidence Level (95,0%) 0,0099566

Source: own elaboration.

Table 4 .

 4 26 -ANAC's Two Proportions z -Test n3 -Accredited Examiners and n4 -Civil AviationInspectors pilot flight proficiency checks for licensing(2014 -2019)

	3,00%
	2,50%
	2,00%
	1,50%
	Examiners'	Inspectors'
	Disapproval	ANAC's mean for licencing

Source: own elaboration.

Table 4 .

 4 27 -ANAC's frequency of pilot flight proficiency checks for licensing-PPL, CPL, and ATPL (2014 -2019)

	Event	Approved Disapproved Row Total
	Examiners' check	23085	175	23260
	Inspectors' check	1018	29	1047
	Column Total	24103	204	24307

Table 4

 4

				Row
	Event	Approved Disapproved	Total
	Examiners'
	check	94,97%	0,72%	95,69%
	Inspectors'
	check	4,19%	0,12%	4,31%
	Column Total	99,16%	0,84% 100,00%
		Source: own elaboration.
	The prior probability of the pilot flight proficiency checks for licensing being assigned to
	Examiners was 95.69%, while the prior probability to Inspectors was only 4.31%.
	The ANAC's disapproval rate of pilot flight proficiency check for licensing regardless of
	the division between Examiners or Inspectors was expressed at 0.84%.

.28-ANAC's prior probabilities of being checked by Examiners or Inspectors for licensing -PPL,

CPL, and ATPL (2014 -2019)

The conditional probabilities of Approval or Disapproval divided by Examiners or Inspectors are as follows:

Table 4 .

 4 29 -ANAC's conditional probabilities of being checked by Examiners or Inspectors for licensing -PPL,CPL, and ATPL (2014 -2019)

		P[I|E] = p[E,I] / P [E]	Row Total
	P[I|E]	Approved	Disapproved
	Examiners' check	99,25%	0,75%	100,00%
	Inspectors' check	97,23%	2,77%	100,00%
		Source: own elaboration.
	The Examiners' disapproval probability was 0.75%, while the Inspectors' was 2.77%
	The probabilities of Approval or Disapproval based on the entire population N of the
	ANAC's pilot flight proficiency checks for licensing -PPL, CPL, and ATPL (2014 -2019) were
	demonstrated as follows:

Table 4

 4

	.30 -ANAC's posterior probabilities of being checked by Examiners or Inspectors for licensing
	-PPL, CPL, and ATPL (2014 -2019)
		Approved/Total	Disapproved/Total
	P[E|I]	Approved	Disapproved
	Examiners' check	95,78%	85,78%
	Inspectors' check	4,22%	14,22%
	Column Total	100,00%	100,00%

Source: own elaboration.

FAA's Null Hypothesis of pilot flight proficiency check for licensing

	4.7.2 -With the purpose of also testing the FAA's Null Hypothesis for licensing pilot flight
	41718 proficiency check H0, i.e.,, no differences between the FAA's samples n5 -Designated Pilot 38802 50530 44647 43434 38547 Examiners and n6 -Aviation Safety Inspectors, it was necessary to extract the data from Table
	19 -ORIGINAL AIRMEN CERTIFICATES APPROVED/DISAPPROVED BY CATEGORY
	AND CONDUCTOR of the FAA's -Federal Aviation Administration annual reports on the U.S.
	Civil Airmen Statistics (FAA, 2014 to 2019), transform them into individually approved and
	disapproved data classified into Examiners' and Inspectors' and analyze these data using
	quantitative descriptive analyses and the statistical Two Proportions z -Test.
	2014	2015	2016	2017	2018	2019
		Flight Checks		Cumulative Moving Average
	Figure 4.12 -FAA's Instrument rating histogram (2014-2019)
						30744
					26813
	21533	20173	21806	22815
	2014	2015	2016	2017	2018	2019
		Source: own elaboration.

Table 21 of the ORIGINAL AIRMEN CERTIFICATES ISSUED BY CATEGORY (FAA, 2014 to 2019), as follows:

Table 4

 4

	.31 -FAA's descriptive statistics pilot flight proficiency checks for licensing (2014 -
	2019)
	Mean	0,2312648
	Standard Error	0,0007816
	Median	0
	Mode	0
	Standard Deviation	0,421642
	Population Variance	0,177782
	Kurtosis	-0,3751
	Skewness	1,2747165
	Range	1
	Minimum	0
	Maximum	1
	Sum	67305
	Count	291030
	Confidence Level (95,0%)	0,0015319
	Disapproved checks (statistical success events) Σ xi= 67305; the mean is μ= 0.2312648; the
	Standard Deviation is σ= 0,421642, and the Population Variance is σ 2 = 0,177782, and the
	Confidence Level for mean is 95%.
	The Examiners' descriptive statistics on the disapproval rate of pilot flight proficiency
	check from 2014 to 2019 are demonstrated, as follows:

Source: own elaboration.

Analyzing the descriptive statistics of FAA's pilot flight proficiency checks for licensing, the Count is the number of checks (population size) N= 291030; the Sum is the number of

Table 4

 4 Analyzing the descriptive statistics of FAA's pilot flight proficiency checks for licensing, the Count is the number of checks (population size) n5= 284863; the Sum is the number of Disapproved checks (statistical success events) Σ X5= 65802; the mean is 𝑥̅ 5 = 0.2309953; the Standard Deviation is s5= 0.4214701; and the Population Variance is 𝑠 5

	.32 -FAA's Examiners' descriptive statistics pilot flight proficiency checks for licensing (2014 -
	2019)
	Mean	0,2309953
	Standard Error	0,0007897
	Median	0
	Mode	0
	Standard Deviation	0,4214701
	Sample Variance	0,1776371
	Kurtosis	-0,37051
	Skewness	1,2765158
	Range	1
	Minimum	0
	Maximum	1
	Sum	65802
	Count	284863
	Confidence Level (95,0%) 0,0015477
	Source: own elaboration.

2 = 0.1776371; and the Confidence Level for the mean is 95%. The Inspectors' descriptive statistics on the disapproval rate of pilot flight proficiency check from 2014 to 2019 are demonstrated, as follows:

Table 4

 4

	.33 -FAA's Inspectors' descriptive statistics pilot flight proficiency checks for licensing (2014 -
	2019)
	Mean	0,2437166
	Standard Error	0,0054674
	Median	0
	Mode	0
	Standard Deviation	0,4293585
	Sample Variance	0,1843487
	Kurtosis	-0,57411
	Skewness	1,1941844
	Range	1
	Minimum	0
	Maximum	1
	zSum	1503
	Count	6167
	Confidence Level (95,0%) 0,0107181

Source: own elaboration.

Table 4

 4

	.34 -FAA's Two Proportions z-Test n5 -Examiners and n6 -Inspectors pilot flight proficiency
	checks for licensing (2014 -2019)
		FAA's Examiners FAA's Inspectors
	Mean	0,2309953	0,2437166
	Known Variance	0,177637	0,184349
	Observations	284863	6167
	Hypothesized Mean
	Difference	0
	z	-2,302847
	P(Z<=z) one-tail	0,0106437
	z Critical one-tail	1,6448536
	P(Z<=z) two-tail	0,0212875
	z Critical two-tail	1,959964

Source: own elaboration.

Table 4 .

 4 35 -FAA's pilot flight proficiency checks' rates for licensing(2014 -2019)

	Approved	Percent	Disapproved	Percent	Total
	223725	76.87%	67305	23.13%	291030

Source: own elaboration.

Table 4

 4

	.36 -FAA's Examiners and Inspectors pilot flight proficiency checks rates for licensing (2014 -
	2019)
	Approved	Percent	Disapproved	Percent	Total
	219061	76,90%	65802	23,10%	291030

Source: own elaboration.

Table 4

 4

	Approved	Percent	Disapproved	Percent	Total
	4664	75,63%	1503	24,37%	291030

.37 -FAA's Examiners and Inspectors pilot flight proficiency checks rates for licensing

(2014 - 2019)

Source: own elaboration.

may not be the ideal data source for your research, given the manner in which we oversight these activities. Generally

	speaking, CASA
	does not ordinarily perform 'every day' flight tests and proficiency checks that industry Flight Examiner
	Rating holders ordinarily perform. Rather,

After thorough consultation I have been advised that Australia

our oversight focus is on Flight Examiner Rating (FER) flight test and proficiency checks

-which industry do not ordinarily perform. Where we do perform other 'every day' flight tests and proficiency checks, the

sample size is very small and is not CASA's core or usual business. As such CASA is unable to provided data which you have requested.

Table 4 .

 4 38 -FAA's pilot flight proficiency checks for licensing(2014)(2015)(2016)(2017)(2018)(2019) the above tables, there is an enormous difference between the FAA's and ANAC's approval rates. While the FAA's Examiners' approval mean is 76.90%, the ANAC's Examiners' approval mean is 99.47%. A similar difference is present in the FAA's Inspectors' Analyzing the dataset of both ANAC's Examiners n3 and FAA's Examiners n5 using the Two Proportions z -Test with a significance level α of 5%, and considering that the dataset of approval or disapproval results as a binomial distribution, the assumption of normality was assessed using the CLT -Central Limit Theorem, which the mean of any random variable will be approximately normally distributed as sample size increases according to the CLT.

	Category of Certificates	Examiner Approved	Examiner Disapproved	Examiner Total	Examiner Percent Approved	Inspector Approved	Inspector Disapproved	Inspector Total	Inspector Percent Approved
	Private	99853	35426	135279	73,81%	331	188	519	63,78%
	Commercial	48208	15287	63495	75,92%	245	78	323	75,85%
	Airline Transport	39147	4899	44046	88,88%	1336	267	1603	83,34%
	Rotorcraft (only) 4	10970	1237	12207	89,87%	192	14	206	93,20%
	Flight Instructor Certificates	20883	8953	29836	69.99%	2560	956	3516	72,81%
	Instrument rating 5 7 8	143884 5	-	-	-	143884 5	-	-	-
	Licensing Total 6	219061	65802	284863	76.90%	4664	1503	6167	75.63%
	Category of Certificates	Examiner Approved	Examiner Disapproved	Examiner Total	Examiner Percent Approved	Inspector Approved	Inspector Disapproved	Inspector Total	Inspector Percent Approved
	Private	11594	87	11681	99,26%	33	1	34	97,06%
	Commercial	6779	44	6823	99,36%	150	5	155	96,77%
	Airline Transport	552	14	566	97,53%	810	22	832	97,36%
	Rotorcraft (only) 4	4160	30	4190	99,28%	25	1	26	96,15%
	Flight Instructor rating 7 8	18228	59	18287	99,68%	534	4	538	99,26%
	Instrument rating (only) 7 8	9103	36	9139	99,61%	425	8	433	98,15%
	Instrument rating (additional) 7 8	28059	188	28247	99.33%	5747	112	5859	98.09%
	Licensing Total 9	23085	175	23260	99,47%	1018	29	1047	98,06%

Source: own elaboration. Table

4

.39 -ANAC's pilot flight proficiency checks for licensing

(2014)(2015)(2016)(2017)(2018)(2019)

Source: own elaboration.

4

Helicopters' PPL, CPL, and ATPL.

5

Original and additional ratings, due to FAA' unclassified data of Examiners' and Inspectors' approved/disapproved totals.

6

Licensing Total, including airplane's and helicopter's PPL, CPL, ATPL and Flight Instructor Certificates. 7 Airplane's and Helicopter's.

8

Not included in Licensing Total due to unclassified data. 9 Licensing Total, including airplane's and helicopter's PPL, CPL, ATPL.

Comparing

Table 4

 4

	.40 -ANAC's Examiners n3 and FAA's Examiners n5 Two Proportions z-Test of pilot flight
	proficiency checks for licensing (2014 -2019)
		ANAC's	FAA's
		Examiners	Examiners
	Mean	0,0075236	0,2309953
	Known Variance	0,0074674	0,1776371
	Observations	23260	284863
	Hypothesized Mean Difference	0
	z	-229,928
	P(Z<=z) one-tail	0
	z Critical one-tail	1,6448536
	P(Z<=z) two-tail	0
	z Critical two-tail	1,959964

Source: own elaboration.

Table 4

 4

		ANAC's Inspectors FAA's Inspectors
	Mean	0,0276982	0,2437166
	Known Variance	0,0269567	0,1843487
	Observations	1047	6167
	Hypothesized Mean Difference	0
	z	-28,96007
	P(Z<=z) one-tail	0
	z Critical one-tail	1,6448536
	P(Z<=z) two-tail	0
	z Critical two-tail	1,959964

.41 -ANAC's Inspectors n4 and FAA's Inspectors n6 Two Proportions z -Test of pilot flight proficiency checks for licensing

(2014 -2019)

Source: own elaboration.

AVS-level policy that looks holistically at oversight and management of designees.

  FAA has a DMS -Designee Management System for AVS -Aviation Safety, which includes several designee categories such as Pilot Examiners, Aviation Medical Examiners, for managing all aspects of the lifecycle of individual representatives of the Administrator, otherwise referred to as designees… to consolidate independent designee management policies into one

	…
	application for designation; selection of an individual for appointment; appointment and orientation of new
	designees; oversight activities; suspension and termination. For some organizations, DMS
	Mechanic Examiners, Aircraft Dispatch Examiners, Aircrew Examiners, Manufacturing
	Examiners, Engineering Representatives, Airworthiness Representatives-Maintenance, and
	Parachute Rigger Examiners. The DSM's oversight activities allow designee suspension, and

termination, and provide key information to better visibility into designee activity. …system

will also provide key information leading to better visibility into designee activity at all levels

	. (FAA, 2018)
	(FAA, 2018) [our emphasis]
	4.9.2 -The SEED Program
	The FAA's SEED -Special Emphasis Evaluation of a Designee is contained in the
	Order 8900.1 CHG -change 669 V 13 CH 5 and express:

The SEED Program. A SEED is a team-focused, indepth evaluation of a part 183 designee. The SEED program was developed as a tool to support a comprehensive designee oversight system. Using a data-driven approach, this program focuses FAA resources on designees who exhibit indicators that potentially pose a higher risk to FAA certification processes.

(FAA, 2019) [our emphasis]

134/CENIPA/2014 -05 Review the criteria for the definition of similarity, which

  for conducting pilot flight proficiency checks, even being provisioned by Section 17.13 of the RBHA -Brazilian Requirements for Aeronautical Certification 17. Additionally to the lack of Inspectorate's technical training, there is the lack of rating, medical certificate, safe work environment, which represents that the ANAC's safety culture might be hazardous to aviation and might threaten its safety. enable Civil Aviation Flight Inspectors and accredited examiners to

	•	RBHA -Brazilian Requirements for Aeronautical Certification 17, revoked by
		the Resolution 472 of 6 th June 2018;
	•	IAC -Civil Aviation Instruction 3201-0198, revoked by Resolution 422 of 2 nd
		May 2017

The above-mentioned revocation took place a few years after the CENIPA -Accident Investigation Authority's Safety Recommendations A -134/CENIPA/2014 -05 and A -134/CENIPA/2014 -06

(CENIPA, 2014, pp. 157, 158)

, who pointed out the Inspectorate's lack of training A -

conduct evaluations of pilots in various types of aircraft, taking into account the compatibility of the avionics and airborne systems. A -134/CENIPA/2014 -06 Make sure that all Civil Aviation Flight Inspectors and accredited examiners are kept up-to-date on the aircraft utilized for the evaluation of pilots

.

[START_REF] Cenipa ; Collingridge | The quality of qualitative research[END_REF]

  Considering that the Inspectors have access to or occupy the pilot's seat as mentioned above, the Inspector's lack of Ratings and Medical Certificate poses a threat to their own safety and the safety of others.

	Regrettably, there is no consensus on the official position regarding the lack of Ratings
	and Medical Certificate and the risk assessment lies on the Inspector's accountability,
	sometimes unpleasantly challenged by the Office for Internal Affairs or the Justice.
	According to Item 5.3.1 of IS -Supplementary Instruction 00-002 C, D, E and F, the
	Examiner must act with professionalism, high level of knowledge and maintain his technical
	integrity even under pressure. (ANAC, 2017) (ANAC, 2018) (ANAC, 2018) (ANAC, 2020)
	According to Item 5.3.3, a), b), ANAC require that the Examiners maintain outstanding
	professional standards, integrity and ethics and abide by this Supplementary Instruction and
	any applicable regulation. It is worth reminding that Section 61.3, (i), (2), of RBAC 61 requires
	that Accredited Examiners and Civil Aviation Inspectors (INSPAC) be properly rated and
	qualified.
	to Item 5.2.3.18 of the IS -Supplementary Instruction 00-002 A and B, the
	Inspector must occupy the student's pilot seat during the Flight Instructor exam. (ANAC, 2011)
	(ANAC, 2012)
	According to Item 9.1, d), of the IS -Supplementary Instruction 00-002 C, D, E and F
	the PIC -Pilot in Command has to be defined and could be the candidate, the Examiner or a
	third pilot, if one comply with the requirements for the function, i.e. valid medical certificate,
	valid ratings, endorsements and currency. (ANAC, 2017) (ANAC, 2018) (ANAC, 2018) (ANAC,
	2020)

Table 5

 5

	.1 -IS -Supplementary Instruction 00-002 history
	IS -Supplementary Instruction	Order	Year
	IS 00-002A	2.177/SSO	2011
	IS 00-002B	1085/SSO	2012
	IS 00-002C	63/SPO	2017
	IS 00-002D	180/SPO	2018
	IS 00-002E	1.450/SPO	2018
	IS 00-002F	1.527/SPO	2020

Source: own elaboration.

Table 5

 5

	.2 -Accredited Examiners' Certification (2016 -2019)
	RBAC	2016	2017	2018	2019
	141	225	673	470	257
	142	44	203	69	36
	135	57	222	355	209
	121	58	278	190	309

Source: own elaboration.

Table 5

 5

	.3 -Accredited Examiners' Oversight Inspections (2016 -2019)
	RBAC	2016	2017	2018	2019	Total
	141	0	25	6	41	72
	142	3	7	1	0	11
	135	1	7	3	0	11
	121	14	72	79	90	255

Source: own elaboration.

Table 5

 5

	.4 -Accredited Examiner's oversight over certification proportion (2016 -2019)
	RBAC	2016	2017	2018	2019	Mean
	141	0,0%	3,7%	1,3%	16,0%	5.25%
	142	6,8%	3,4%	1,4%	0,0%	2.9%
	135	1,8%	3,2%	0,8%	0,0%	1.45%
	121	24,1%	25,9%	41,6%	29,1%	30,17%
			Source: own elaboration.
	The ANAC's surveillance of Accredited Examiners' mean, from 2016 to 2019, was
	considered insufficient because its activities varied from 1.45% (RBAC 135 -Air Carriers),
	2.9% (RBAC 142 -Training Centers), 5.25% (RBAC 141 -Approved Training Organisations)
	to 30,17% (RBAC 121-Airlines) of the accreditation activities.

Table 6

 6

	.1 -Quantity of aircraft owned by ANAC
	Legal Entity Name	CNPJ -National Register of Legal Entities Quantity
	National Civil Aviation Agency	07.947.821/0002-60	94
	ANAC
	Federal Government -ANAC	07.947.821/0001-89	90
	Federal Government ANAC	07.947.821/0013-12	8
	Federal Government ANAC	00.394.429/0043-60	372
		Total	564

Source: own elaboration.

of Practical Tests Per Day. A DPE may not conduct more than two practical tests in 1 calendar-day

  (i.e., midnight to midnight), unless that DPE has received specific written permission from the managing FAA office. Initial tests, retests, discontinuances, or continuations of practical tests are all considered practical tests, and the DPE must have received specific written permission from the managing FAA office to do more than two practical tests in any 1 calendar-day.

	14. Structure of the Practical Test. The practical test consists of a demonstration of aeronautical knowledge
	and skill. The two demonstrations are not intended to be separate tests. They should be conducted
	concurrently.
	a.	Number

In the case of a designee administering an initial flight instructor practical test, the designee will not conduct more than one test per day.

  If the initial flight instructor test is not a complete practical test (i.e., a retest or continuance),

	the designee may conduct an additional test on the same calendar-day (FAA, 2018, pp. 7-27) [our
	emphasis]

Table 7

 7

	.1 -Examiners' oversight activities: transport and DSA -daily subsistence allowance costs
	2020's Annual Work Plan	Transport cost	Transport quantity	DSA cost	DSA quantity	Total
	Planned	R$ 671,00	2	R$ 197,00	2	R$ 1.736,00
	On-demand	R$ 743,00	2	R$ 199,00	2	R$ 1.884,00

Less than or equal to 2 10% 3 to 5 26% 6 to 8 22% 9 to 11 30% 12 to 14 8% 15 to 17 4% Greater than or equal to 18 0% (ii) as horas realizadas no exame de proficiência devem ser registradas pelo INSPAC ou examinador credenciado responsável

  por tal exame, que informará se o aluno piloto foi aprovado ou

	reprovado, seu CANAC, nome e assinatura;
	(iii) as pessoas citadas nos parágrafos (c)(5)(i) e (c)(5)(ii) desta seção que

preencherem ou endossarem um lançamento na CIV ou CIV Digital com informações ou dados inexatos ou adulterados ficam sujeitas às providências administrativas previstas

	na Lei nº 7.565/86 -Código
	Brasileiro de Aeronáutica (CBA) ou na legislação complementar, além das sanções penais e civis cabíveis,
	uma vez que tal fato poderá comprometer a segurança de voo; e (Redação dada pela Resolução nº
	378, de 18.03.2016) (ANAC, 2020) [our emphasis]
	According to Paragraph 2.1.1.1 of Annex 6 -Part II, the pilot-in-command shall comply
	with the laws, regulations and procedures of those States in which operations are conducted
	(ICAO, 2018).
	According to Paragraph 2.7.2.1 of Annex 6 -Part II, the pilot-in-command shall ensure
	flight crew member holds valid licence, are rated, and maintain competency:

a) ensure that each flight crew member holds a valid licence issued by the State of Registry, or if issued by another Contracting State, rendered valid by the State of Registry; b) ensure that flight crew members are properly rated; and c) be satisfied that flight crew members have maintained competency. (ICAO, 2018) [our emphasis]

Aviation Safety Inspectors (ASI) … may credit former ASIs (Operations) 5 hours of PIC currency for certain tests/checks administered.

  This recent PIC credit may be offered under the

	following three conditions:
	…
	(3) …
	(a) For each proficiency check that was administered under 14 CFR parts 121, 125, or 135, 5 hours of
	PIC currency within the preceding 12 calendar-months.
	(b) For each proficiency check, 5 hours of PIC currency that was administered to a chief instructor,
	assistant chief instructor, or check instructor under 14 CFR part 141 within the preceding 12 calendar-
	months.
	... (FAA, 2016)
	g. Former
	PIC -Pilot in Command currency experience, according to Order 8900.2B (FAA, 2016),
	superseded by Order 8900.2C (FAA, 2018), which requires 10 checks for currency
	requirement:

g. Former

Aviation Safety Inspectors (ASI) … for PIC experience requirements in the past 12 calendar-months

…

(2) ASI (

Operations) must have conducted a combination of at least 10 tests or checks

  Unfortunately, the Examiners are not subject to the same Capacity Development(CAP) requirements, applicable to the Inspectors, i.e., receive 3 (three) OJT -On-the-job training from ANAC; and receive theoretical recurrent capacity development from ANAC. If both Capacity Development were also applied to the Examiners, the standardization and the disapproval rates could converge to the Inspectors' higher values. Also, according to the level of agreement on the Surveillance (SUR) statements, the Examiners are not exempted from conflict of interest and seldom subjected to surveillance during the accreditation validity. The possibility of a conflict of interest, in addition to the lack of surveillance and to the Inspectorate's status quo, worsen by the current management decisions on outsourcing pilot flight proficiency checks to private individuals, not even linked to a certified operator, nor surveilled, certainly endangers the aviation system and must be thoroughly regulated, standardized and supervised.

	Figure 7.4 -Inspectors' survey on Inspectors' factors.
	TRN1 -The Flight Inspectors receive equipment Ground School from ANAC.	52%	46%				2%	0%
		72%		28%			0%
		90%			10%	0%
		86%			10%		4%	0%
		98%			2%		0%
		70%			12%		6%	8%	4%
		82%			4%		10%	4%	0%
			6%	14%		32%	40%	8%
			8%	20%	10%	12%	50%
		68%			16%	8%	8%	0%
		60%		24%		8%	4%	4%
				12%	16%		10%	18%	44%
		38%	34%			16%	12%	0%
			26% 10%		10%	44%	10%
			14%	28%			14%	28%	16%
		42%			14%		14%	30%	0%
	The survey's statements, attached to ANNEX 1 and ANNEX 2, were based on Pilot 40% 16%				28%	10%	6%
	Technical Training (TRN), Operations (OPS), Capacity Development (CAP), Standards (STD), -50 -40 -30 -20	-10		0		10	20	30	40	50
	and Surveillance (SUR), with the level of agreement expressed in Figure 7.4 -Inspectors'
	survey on Inspectors' factors. and Figure 7.5 -Inspectors' survey on Accredited Examiners'
	factors.
	The largest disparities, on the one hand, were related to the lack of Inspectors' Pilot
	Technical Training (TRN), Operations (OPS) requirements compliance, and Capacity
	Development (CAP), while, on the other hand, were related to the lack of Accredited
	Examiners' Capacity Development (CAP), Standards (STD) and Surveillance (SUR).
	Fortunately, the PCIV -ANAC's Flight Inspectors Training Programme, if properly
	implemented, will provide the required Technical Training (TRN) and Capacity Development
	(CAP), which could be enough to comply with the Operations (OPS) requirements, if

… (FAA, 2018) [our emphasis]

According to the ANAC's Inspectors' level of agreement on the surveyed statements, it was possible to perceive clear disparities between Inspectors and Examiners, when both professional categories were subject to the same regulations and to perform standardized pilot flight proficiency checks.

appropriately accepted by the Personnel Licensing (PEL) Department of the CAA, and should be periodically reassessed through similar surveys to keep the Programme operationally and technologically up-to-date.

  Accredited Examiners and Civil Aviation Inspectors (H0:  - =0).The Face validity was assessed by two non-experts (an Engineer and a Lawyer), by two ANAC's experts (an Air Traffic Controller and an Accident Investigator and Journalist, both Commercial Pilots), and a former ANAC's expert (an Airline Transport Pilot). The three ANAC's experts also validated the content, which reflected the information gathered from the literature reviewed in the previous chapters. Through their contribution, it was possible to spot understanding problems as well as leading statements during the Pilot Test, which were corrected prior to the distribution.

	Figure 7.5 -Inspectors' survey on Accredited Examiners' factors.
	7.3.1 -Inspectors' survey's validation
	TRN1E -The Accredited Examiners receive equipment Ground School.			10%	16%			10%	26%	38%
	TRN2E -The Accredited Examiners receive equipment Flight Training. The above-mentioned survey was based on Dave Collingridge's six-step validation	18%	8%	10%	26%	38%
	22% method, which includes Face validation, Pilot Test of the Questionnaire, Clean Collected Data, TRN3E -The Accredited Examiners receive Differences training.	8%			38%	6%	26%
	OPS1E -The Accredited Examiners possess valid rating in the equipment. PCA -Principal Component Analysis, Cronbach's Alpha, and Question revision if needed	20%	2%	10% 10%	58%
	OPS2E -The Accredited Examiners possess currency (E.g.: 90 days -Day,… (Aithal & Aithal, 2020, p. 11, apud Collingridge & Gantt, 2008).				20%		24%	12%	44%
	OPS3E -The Accredited Examiners possess Medical Certificate for real flight.				16%	4%		10%	6%	64%
	12% The survey's construct was aligned to the Null Hypothesis of no differences between 8% OPS4E -The Accredited Examiners operate in PBN -Performance Based…			48%	10%	22%
	OPS5E -The Accredited Examiners have difficulties to visualize the panel's… the ANAC's		10%	38%				34%	18%	0%
	OPS6E -The Accredited Examiners have difficulties to perform the required…			12%	32%			2%	24%	30%
	OPS7E -The Accredited Examiners have enough flight experience.				4%			34%	28%	34%
	CAP1E -The Flight Inspectors receive CRM -Crew Resource Management…			24%	14%		22%	8%	32%
	CAP2E -The Accredited Examiners receive PBN -Performance Based…			20%	8%		42%	10%	20%
	CAP3E -The Accredited Examiners receive 3 (three) OJT -On-the-job training…		36%		8%	16%	12%	28%
	CAP4E -The Accredited Examiners receive advanced training (Ex.:…	38%		32%			26%	2%	2%
	CAP5E* -The Accredited Examiners must receive theoretical recurrent…				8%	4%	4%	8%	76%
	STD1E -The objective criteria for flight check failing are adequate.		22%	20%	22%	32%	4%
	STD2E -The subjective criteria for flight check failing are adequate.		28%		22%	20%	10%	20%
	STD3E -The flight check procedures are standardized.		32%	20%			12%	30%	6%
	STD4E -It is permitted to the the Accredited Examiners the equipment…		30%	8%	26%	14%	22%
	SUR1E* -The Accredited Examiners are exempted of conflict of interest.	64%			28%		4%	4%	0%
	SUR2E* -The Accredited Examiners are surveilled by ANAC during the…	30%	36%			18%	8%	8%
	STD4 -The equipment evaluation by similarity is safe (E.g.: ASEL, AMEL,
	Executive Jet, Turboprop, Narrow-Body or Wide-Body Group)	-50	-40	-30	-20	-10				0	10	20	30	40	50
	Strongly disagree	Partially disagree	Neither agree nor disagree	Partially agree			Strongly agree
	Strongly disagree	Partially disagree	Neither agree nor disagree	Partially agree			Strongly agree
	94

The survey's Construct Validity (Convergent and Discriminant Validities) could be verified by means of factor analysis conducted using PCA -principal component analysis with varimax rotation method as described below:

Table 7

 7

	.2 -PCA -principal component analysis
	Component Loadings
		Component
	1	2	3	Uniqueness
	TRN1		0.374	0.817
	TRN2		0.503	0.708
	OPS3		0.663	0.543
	OPS4	-0.312	0.304	0.736
	OPS5		0.439	0.732

  CAP3E', 'CAP5E', 'OPS5E', and 'OPS6E' correlate negatively with the total scale and probably should be reversed

	Scale Reliability Statistics
	Cronbach's α
	scale	0.840
	Note. items 'Correlation Heatmap

Table 7 .

 7 4 -Reliability Analysis -Component 2

	Scale Reliability Statistics
		Cronbach's α
	scale	0.876
	Correlation Heatmap

  Unlike the FAA, the ANAC does not use qualified Operations Inspector, which means they don't hold appropriate class or type rating, nor they are current in the aircraft, not even for two-place aircraft, which usually have only the unrated candidate and the Inspector (or Flight Inspectors Training Programme PEF -Pilot Evaluation Form (FAP -Ficha de Avaliação de Piloto) PIC -Pilot in Command PPLprivate pilot licence RBAC -Brazilian Civil Aviation Regulation (Regulamentos Brasileiros da Aviação Civil) SACI -Civil Aviation Information Integrated System (Sistema Integrado de Informações da Aviação Civil) SACI -Sistema Integrado de Informações da Aviação Civil (Civil Aviation Information Integrated System) SAE -Exam authorization request SAE -Specialized Air Service SARPs -Standards and Recommended Practices SCDP -Allowance and Fare System SDEA -Santos Dumont English Assessment SEED -Special Emphasis Evaluation of a Designee SEI! -Electronic Information System SICOS -Service Order System SisCheck -Roster's Flight Check Management System SISHAB -Rating System (Sistema de Habilitações) SM ICG -Safety Management International Collaboration Group SMS -Safety Management System SPL -Department of Civil Aviation Personnel SSP -State Safety Programme USOAP -Universal Safety Oversight Audit Programme VFR -Visual Flight Rules W&B -Weight and Balance

	OI -Operations Inspectors
	Operations, OPS 135 -Air Carrier Operations
	OPS 141 -Training Organization Operations
	OPS 91 -General Aviation Operations
	PBN -Performance-based Navigation
	PCA -principal component analysis
	PCIV -ANAC's
	Examiner).

Aircraftcategory. Classification of aircraft according to specified basic characteristics, e.g. aeroplane, helicopter, glider, free balloon. (ICAO,

2018)

Competency. A combination of skills, knowledge and attitudes required to perform a task to the prescribed standard.(ICAO, 2018)

Licensing Authority. The Authority designated by a Contracting State as responsible for the licensing of personnel.(ICAO, 2018)

Page intentionally left blank

Flight ChecksCumulative Moving Average

Disapproval (PEL)Cumulative Moving Average

Source: own elaboration.

Examiners' flight checksCumulative Moving Average

Inspectors' flight checksCumulative Moving Average

Table 7.5 -Reliability Analysis -Component 3

Source:(FAA, 2017)

ACKNOWLEDGEMENTS

I want to express my gratitude to my supervisor, Dr. Adriano Canolla, and his invaluable assistance to guide this project, even when facing the adversities of the 2020's Pandemic and his professional and personal priorities. I wish to thank the Operations Standards Superintendent, MSc. João Garcia, for allowing the use of the pilot flight proficiency check's dataset during this research. I want to pay my special regards to the ANAC's Flight Inspectors, in particular to Mr. Adriano Baumgartner and Mr. Adriano Sousa, and many other ANAC's colleagues such as Mr. Luiz Beltrão and Mr. Rafhael Santos, as well as managers, especially Mr. João Hirae, superintendents and directors directly involved in the creation of the PCIV -ANAC's Flight Inspectors Training Programme, for the motivation of its accomplishment and the possibilities of its implementation toward aviation safety. I am grateful for my family's support, who provided me with the means that kept me working on this project. IV V DMS -Designee Management System DPE -Designated Pilot Examiner DSAdaily subsistence allowance EMDamendment FAA -Federal Aviation Administration FFS -Full Flight Simulator GS -Ground School HR -Human Resources IAC -Civil Aviation Instruction ICAO -International Civil Aviation Organization IFR -Instrument Flight Rules INSPAC -Civil Aviation Inspector IQR -Interquartile Range IS -Supplementary Instruction kBkilobyte KSAknowledge, skills, and attitudes NOTAM -Notices to Airmen ODA -Organization Designation Authorization

AMEL -Airplane Multi-Engine Land APPapplication Approved Training Organisation ASEL -Airplane Single Engine Land ASES -Airplane Single Engine Sea ASI -Aviation Safety Inspector ATCoair traffic controller ATPLair transport pilot licence AVS -Aviation Safety CAA -Civil Aviation Authority CANAC -ANAC's Code (Código ANAC) CAPscorrective action plans CASA -Civil Aviation Safety Authority CASI -Civil Aviation Safety Inspector CCscompliance checklists CEcritical elements CENIPA -Accident Investigation Authority (Centro de Investigação e Prevenção de Acidentes Aeronáuticos) CHGchange CIV -Logbook CIV Digital -Electronic Logbook CLT -Central Limit Theorem 45, CMA -Medical Certificate CMA -Continuous Monitoring Approach CPLcommercial pilot licence

According to Paragraphs 2.3.1.1 and 2.3.1.2 of Doc 9379 -Manual of Procedures for Establishment and Management of a State's Personnel Licensing System, a PEL Office must be suitably organized and staffed with qualified personnel and a typical organizational structure were represented as follows: and Management of a State's Personnel Licensing System, the granting of licence privileges cannot be done satisfactorily if the qualifications of the person making the judgements are not at least equal to those required of the licence applicant:

The granting of licence privileges to participants in the civil aviation system cannot be done satisfactorily if the qualifications of the person making the judgements are not at least equal to those required of the licence applicant. … However, the minimum qualification of the key personnel in all PEL Offices should be: Breaches in safety defences can be a delayed consequence of decisions made at the higher levels of the organization, which may remain dormant until their effects or damaging potential are activated by certain operating conditions (known as latent conditions). (ICAO, 2018) [our emphasis]

According to the Reason's Model, accidents involve the successive breaches of multiple defences, such as those mentioned in this research, i.e., lack of training, lack of medical certificate, lack of rating, lack of safety culture, and lack of oversight, as well as actions, inactions, active disapprovals, and latent conditions, represented as the holes in the slices, consequently, the slices represents the defences, mainly regulations, training, and technology, as follows: (ICAO 2018, p. 2-7) According to the Paragraph 2.3.8.1 of the Doc 9859 -Safety Management Manual, Fourth Edition:

Scott A. Snook's theory of practical drift is used to understand how performance of any

system drifts away from its original design. Tasks, procedures, and equipment are often initially designed and planned in a theoretical environment, under ideal conditions, with an implicit assumption that nearly everything can be predicted and controlled, and where everything functions as expected. This is usually based on three fundamental assumptions that the: a) technology needed to achieve the system production goals is available; b) personnel are trained, competent and motivated to properly operate the technology as intended; and c) policy and procedures will dictate system and human behaviour.

-Implementation of Technical Training

Regarding the Technical Training, the PCIV -ANAC's Flight Inspectors Training

Programme is capable of fulfilling its needs, once fully implemented in its 5-year cycle. While this research was being performed, the initial implementation of the PCIV has started with resource allocation with an estimate of R$ 1.743.806,00 for aircraft-related training (ground school, initial and recurrent) for at least 15 categories of aircraft and for 51 Inspectors.

However, in order to fulfill the ICAO's recommendation of Inspectors being "at least as qualified as the personnel to be inspected or supervised" (ICAO, 2010, pp. I-6-1) Source: own elaboration using (The jamovi project, 2020) (R Core Team, 2019) [START_REF] Revelle | ANNEX 1 -Inspectors' Survey's statements -Inspectors' factors TRN1 -The Flight Inspectors receive equipment Ground School from ANAC[END_REF] In order to test the Null Hypothesis of no differences between the ANAC's Accredited

Examiners and Civil Aviation Inspectors (H0:  - =0) into the Inspectors' Survey's data, it was used the Paired Samples T-Test as follows: Source: own elaboration using (The jamovi project, 2020) (R Core Team, 2019) [START_REF] Revelle | ANNEX 1 -Inspectors' Survey's statements -Inspectors' factors TRN1 -The Flight Inspectors receive equipment Ground School from ANAC[END_REF] Almost all factors have delivered a p-value less than 0.05, then the Null hypothesis (H0) could be rejected, and there is a statistically significant difference between the ANAC's Accredited Examiners and ANAC's Inspectors, according to the data gathered from the Inspectors' survey's. The exceptions are mainly related to standardization factors, which is positive, although may require improvements.

The detailed comparison is contained in the ANNEX 3 -Inspectors' Survey -Paired Samples T-test -Descriptive Plots.

-Validation of the solutions against the initial needs

As a critical look at the oversight intensification and standardization proposed by this research against the initial needs, the short-term results might not even be noticed if there is no change in the current rotten safety culture. However, considering that each one of the 51

Inspectors available (or 80 Inspectors if proper training is provided) manage to supervise and standardize 2 (or 1) of the 698 Examiners in order to disapprove a single examinee due to a bad performance compared to the standards, we will see an increase greater than 100% based on the 2019's disapproval rate, and this scenario is quite feasible, without even mentioning the Inspectors' own rates increase. It is worth mentioning that to reach the FAA's mean disapproval rate for licensing it is required an increase of 2825,67%, but the intention is not to pursue or establish a rate.

The oversight activities aimed at the Examiners from the RBAC 141 demonstrated in Table 5.3 -Accredited Examiners' Oversight Inspections (2016 -2019)Table 5.3 and in Table 5.4 represent an oversight intensification conducted in 2019 (with 41 inspections), motivated by the discussions between the student and his manager before this research was started and represented how easily it could be intensified if the managers were convinced of its needs.

The following figure represents the ANAC's disapproval rate histogram for licensing (2014 -2019): To conclude the data access difficulties, while the student was trying to step-up and deliver more than initially planned, by linking the Proficiency Check Dashboard directly to the daily up-dated database, during the internship's last week, an up-dated dataset was shared, containing slightly different quantities of pilot flight proficiency checks for the 2014-2019 period. Although it was not possible to understand the above-mentioned difference between datasets, there was no significant difference in the results of this research.

-How have I managed to succeed?

Through an effortful quest to analyze the pilot flight proficiency check's data under an academic method using a statistical perspective, started from scratch, while researching within regulations and ICAO's provisions and SARPs to provide based arguments to break the status quo bias' inertia.

Developing leadership and management skills to fulfill the project's timeline by gathering pieces of information about the pilot flight proficiency check's environment during several meetings with ANAC's personnel.

-What is still to be done?

The inner use of more proactive conduct to enable information-driven, risk-based, and result-oriented approach, such as the ICAO's USOAP CMA -Universal Safety Oversight Audit

Chapter 9 -Discussion

-ANAC's General rates

The analysis results of the ANAC's 130.243 general pilot flight proficiency checks were 99.36% approved and 0.64% disapproved.

The analysis results of the general pilot flight proficiency checks of ANAC's Examiners' were 99.47% approved and 0.53% disapproved.

The analysis results of the general pilot flight proficiency checks of ANAC's Inspectors' were 98.06% approved and 1.94% disapproved, summarized by Figure 9.1: The result of the Two Proportions z-Test showed that ANAC's Examiners and Inspectors have a statistically highly significant difference due to the z-score of -10.29 and pvalue less than 0.001, rejecting the Null hypothesis, as shown in Table 4.22. Therefore, considering that the z-score is the measurement of the difference between them, in units of the standard deviation, they did not perform the same. For this reason, standardization and oversight were recommended.

-ANAC's Licensing rates

The analysis results of the ANAC's 24.307 pilot flight proficiency checks for licensing were 99.16% approved and 0.84% disapproved.

The analysis results of the pilot flight proficiency checks for licensing of ANAC's Examiners' were 99.25% approved and 0.75% disapproved.

The analysis results of the pilot flight proficiency checks for licensing of ANAC's Inspectors' were 97.23% approved and 2.77% disapproved, summarized by the following The result of the Two Proportions z-Test showed that ANAC's Examiners and

Inspectors have a statistically highly significant difference due to the z-score -3.95 and p-value less than 0.001, rejecting the Null hypothesis, as shown in Table 4.26. Therefore, considering that the z-score is the measurement of the difference between them, in units of the standard deviation, they did not perform the same. The Inspectors were 369% harsher while performing flight checks and, even performing only 4.31% of the licensing flight checks, they were responsible for 14.22% of the disapprovals, better assuring that the standards and safety were met.

In order to correct noncompliances to the SARPs and regulations, Inspector's pilot training, regulation revisions, standardization, and oversight prioritization and intensification were recommended.

-FAA's Licensing rates

The analysis results of the FAA's 291.030 pilot flight proficiency checks for licensing were 76.87% approved and 23.13% disapproved.

The analysis results of the pilot flight proficiency checks for licensing of ANAC's Examiners' were 76.90% approved and 23.10% disapproved.

The analysis results of the pilot flight proficiency checks for licensing of ANAC's Inspectors' were 75.63% approved and 24.37% disapproved, summarized by the following figure: 4.34. Therefore, considering that the z-score is the measurement of the difference between them, in units of the standard deviation, they did not perform the same, though with a smaller standard deviation than ANAC's.

-ANAC's and FAA's Comparison

The FAA's Examiners disapprove 22.57 percentage points more than ANAC's Examiners, based on the descriptive statistics.

The FAA's Inspectors disapprove 22.43 percentage points more than ANAC's Inspectors, based on the descriptive statistics.

The result of the Two Proportions z-Test showed that ANAC's Examiners and the FAA's

Examiners have a statistically significant difference for licensing due to the result of the z-score -229.93 and p-value less than 0.001, rejecting the Null hypothesis, as shown in Table 4.40.

Therefore, the z-score is the measurement, in units of the standard deviation, of the difference between them, considering they should not have differences.

The result of the Two Proportions z-Test showed that ANAC's Inspectors and the FAA's

Inspectors have a statistically significant difference for licensing due to the result of the z-score -28.96 and p-value less than 0.001, rejecting the Null hypothesis, as shown in Table 4.41.

Fortunately, the PCIV's first round of a normal category jet did fulfill the rating requirements.

Additionally, as an enhancement proposal to the implementation of the PCIV and purposes. Also, there were recommendation to establish a limit of two checks per day (initial, retests, discontinuances, or continuances) or one initial flight instructor check, to guarantee time availability to focus on the quality of the evaluation, , similarly to the FAA's provisions of Section 2, Item 14, a., of the Order 8900.2C.

If both Capacity Development were also applied to the Examiners, the standardization and the disapproval rates could converge to the Inspectors' higher values. Therefore, a revision on the RBAC -Brazilian Civil Aviation Regulation 183 -Personnel Accreditation is recommended for that purpose.

Considering that the Inspectors have access to or occupy the pilot's seat, the Inspector's lack of ratings, currency and medical certificate poses a threat to their own safety and the safety of others.

Regrettably, there is no consensus on the official position regarding the lack of ratings, currency and medical certificate, and the risk assessment lies on the Inspector's accountability, sometimes unpleasantly challenged by the Office for Internal Affairs or the Justice.

In Source: (FAA, 2017)