Organizations are increasingly considering the use of Artificial Intelligence (“AI”) tools to enhance internal business processes. Applying AI to human resource operations is no exception. The use of AI tools to help determine who to hire, whom to fire, and who should be promoted raises a host of labor and employment issues, and implicates new state and municipal privacy statutes including, in particular, NYC Local Law 144 (“NYC 144”) which takes effect on July 5, 2023. NYC 144, and the regulations implementing the statute, require that employers that use an automated employment decision tool or “AEDT” to assist with employment decisions confirm that such tools have undergone a “bias audit.” This article provides an overview of the new law and its requirements.

1. To whom does the law apply?

NYC 144 applies to employees residing in New York City. So long as a resident of New York City is applying for a job which uses AEDTs, any employer or employment agency subject to the provision must ensure proper disclosures and procedures for compliance of NYC 144.

2. What is an AEDT?

AEDTs are defined by NYC 144 as “any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons.”^[1] The definition does not include tools that do not automate, support, substantially assist, or replace discretionary decision-making processes and that does not materially impact natural persons, such as a junk email filter, firewall, antivirus software, spreadsheet, or other compilation of data.^[2] The regulations implementing the local law also added a definition to the terms “machine learning, statistical modeling, data analytics, or artificial intelligence” in order to clarify which tools and processes are in scope.^[3]

3. What is a Bias Audit?

Employers must ensure that a bias audit was performed on the AEDT within one year of using it.^[4] This is required even if the AEDT is not being used to make the final hiring decision. For example, a bias audit must be performed if an AEDT will be used as a screening mechanism at an early point in the application process or where it is being used to score candidates for employees being considered for promotion.

Per the Final Rules, the bias audit must calculate the rate at which individuals in a category are either selected to move forward in the hiring process, or assigned a classification by an AEDT (“selection rate”) based on the categories required to be reported on pursuant to the U.S. Equal Employment Opportunity Commission’s (“EEOC”) EEO Component 1 report.^[5] The audit’s selection rates must be compared against the selection rate of the most selected category to determine an impact ratio of sex categories, race/ethnicity categories, and intersectional categories of sex, ethnicity, and race.^[6] The Final Rules provide examples on how the selection rate and impact ratio for the three categories relate.^[7]

Historical data from the AEDT must be used to perform the bias audit.^[8] Test data or historical data of other employers or employment agencies may be used if insufficient historical data is available to conduct a statistically significant bias audit.^[9] If an employer wishes to rely on a bias audit that uses the historical data of other employers or employment agencies, it must have either never used the AEDT before, or it must provide its own set of historical data from using the tool previously so that it may be considered by the independent auditor.^[10] An independent auditor may also exclude a category that represents less than 2% of the data being used for the bias audit from the required calculations for impact ratio, provided that the auditor includes a justification for the category’s exclusion.^[11]

Bias audit calculations within a given data set are calculated in the following manner:

1. For each protected category, the selection rate of each group should be calculated by dividing the number selected to proceed by the number of applicants.

2. For each group, its selection rate should be divided by the highest selection rate in Step 1 to obtain the group’s impact ratio.

The following is an example of a bias audit calculation for an AEDT based on a protected category:

In evaluating candidates for an interview, an AEDT is used to evaluate 500 male and 400 female applicants, with 300 males and 200 females selected to proceed.

i. The selection rate of each group would be calculated as follows:

4. Do the results of a bias audit need to be published?

Once the audit is complete for all protected categories, results must be publicly available on the employment section of the employer or employment agency’s website in a clear and conspicuous manner or hyperlink, containing the date of the most recent bias audit of the AEDT, the date the employer or employment agency began using the specific AEDT (“distribution date”), and a summary of the results containing the following information:

• The source and explanation of the data used to conduct the bias audit;

• the number of individuals the AEDT assessed that fall within an unknown category;

• the number of applicants or candidates;

• the selection or scoring rates, as applicable; and

• the impact ratios for all categories.^[12]

If the auditor excluded a category that represented less than 2% of the data being used for the bias audit, the summary must include the justification for the exclusion, as well as the number of applicants and scoring rate or selection rate for the excluded category.^[13] The results must be publicly available at least six months after the latest use of the AEDT for an employment decision.^[14]

5. Do employers need to tell employees and applicants that they intend to use an AEDT?

The employer or employment agency must provide notices to employees and job candidates on the use of the AEDT.^[15] Pursuant to the Final Rules, the notice required by NYC 144 § 20-871(b)(1) must include instructions for how an individual can request an alternative selection process or a reasonable accommodation under other laws, if available. At least ten days before the use of an AEDT on a candidate for employment, notice must be provided either on the employment section of the website, in a job posting, or via mail or email to potential candidates.^[16] For the use of an AEDT for promotion decision purposes on a current employee, notice may be given in a written policy or procedure provided to employees, in a job posting, or via mail or email, in all cases at least ten days before use of the AEDT.^[17]

In addition to disclosures to specific candidates or employees subject to an AEDT, at all times, an employer or employment agency must also provide information on the employment section of its website including its AEDT data retention policy, the type of data collected for the AEDT, and the source of the data.^[18] If the employer or employment agency does not disclose such information on its website, it must make such information available upon written request for such information (which must be provided within thirty days of the requests). In the event that a written request for the information is received, but disclosure of the information would violate local, state, or federal law, or interfere with a law enforcement investigation, an explanation must be provided to an employee or candidate for promotion for why disclosure of such information is a violation.^[19]

Interestingly, the Final Rules clarify that an employer or employment agency is not required to provide an alternative selection process. However, those who do not have alternative selection processes must ensure that their bias audit results are therefore up to date.

6. What are the penalties for violating NYC 144?

Violations of NYC 144 include civil penalties of not more than $500 for a first violation and each additional violation occurring on the same day as the first violation, and not less than $500 nor more than $1,500 for each subsequent violation.^[20]

FOOTNOTES

[1] 2021 N.Y.C. Local Law No. 144, N.Y.C. Admin. Code. § 20-870.

[2] Id.

[3] “Machine learning, statistical modeling, data analytics, or artificial intelligence” means a group of mathematical, computer-based techniques:

1. that generate a prediction, meaning an expected outcome for an observation, such as an assessment of a candidate’s fit or likelihood of success, or that generate a classification, meaning an assignment of an observation to a group, such as categorizations based on skill sets or aptitude; and

2. for which a computer at least in part identifies the inputs, the relative importance placed on those inputs, and, if applicable, other parameters for the models in order to improve the accuracy of the prediction or classification.

Rules of City of New York Department of Consumer and Worker Protection (6 RCNY) § 5-300

[4] Id. at 6 RCNY § 5-301(a).

[5] Id. at 6 RCNY § 5-301(b).

[6] Id. at 6 RCNY § 5-301.

[7] Id.

[8] Id. at 6 RCNY § 5-302(a).

[9] Id. at 6 RCNY § 5-302(b).

[10] Id. at 6 RCNY § 5-302(a).

[11] Id. at 6 RCNY § 5-301(d).

[12] Id. at 6 RCNY § 5-303.

[13] Id. at 6 RCNY § 5-301(d).

[14] Id. at 6 RCNY § 5-303(c).

[15] Id. at 6 RCNY § 5-304; 2021 N.Y.C. Local Law No. 144, N.Y.C. Admin. Code. § 20-871(b).

[16] Id.

[17] Id.

[18] Id.

[19] Id.

[20] 2021 N.Y.C. Local Law No. 144, N.Y.C. Admin. Code. § 20-872.