City Hall – City Council Member Peter Koo today introduced legislation that would bring greater transparency and understanding to the automated decision systems used by city agencies to make mission-critical decisions affecting the daily lives of New Yorkers.
Intro 1806 would require annual reporting on Automated Decision Systems (ADS), or the computerized implementation of one or more algorithms by city agencies, including those derived from machine learning or other data processing or artificial intelligence techniques, which is used to make or assist in making decisions.
Council Member Peter Koo, who previously served as Chair of the Council’s Committee on Technology, stated, “This bill looks to give legislators as well as the public a better understanding on how the city uses algorithms. Automated equations are essential to the creation of the statistics and systems that are responsible for everything from school zoning resource allocations, and it is imperative that this important information is vetted with full transparency to ensure its accountability.”
The legislation builds off of Local Law 49 of 2018 (Open Algorithms Law), which was spearheaded by former Council Member and Chair of the Technology Committee, Jimmy Vacca. LL49 required the Mayor’s Office of Operations to create an ADS Task Force to provide recommendations on how information on agency ADS may be shared with the public and how agencies may address instances where people are harmed by agency ADS.
The bill would give guidelines for what each agency considers to be ADS and require specific reporting on:
- The name/brief descriptor of such system;
- What such system is intended to measure or reveal;
- A description of how the information collected will be used;
- The name of the entity that developed the system;
- When the system was first used.
Former Council Member James Vacca, who developed the algorithm legislation mandating the Mayor’s Task Force report, said “Councilman Koo’s legislation is an important measure to assure the intent of my legislation is fully realized. New Yorkers have a right to know much more than they now know about algorithms and how data is used as it’s that data that is used to make day to day decisions impacting every resident of our city. I would hope the administration is committed not just to a bureaucratic shuffling of the deck when it comes to algorithm systems but much more transparency about exactly what each formula is and how it was developed and evolved into its current state.”
Rashida Richardson, Director of Policy Research at AI Now Institute, stated, “Despite repeated demands by advocates and experts for increased transparency around the use of automated decision systems in New York, the ADS Task Force failed to provide any accounting of the ways in which the city is using ADS. The legislation introduced today by Council Member Koo, which requires city agencies disclose their ADS use, is an important step towards accountability around the use of invisible, yet potentially harmful, tools in our city. We urge the City Council to vote to swiftly pass this important piece of legislation.”
“We should not fear the future by making sure we have a clear and unobstructed view of how automated decision making systems are used to deliver government services in our digital era. Those systems should be as accountable as the people who program or purchase them,” said Noel Hidalgo, BetaNYC’s Executive Director. “We support CM Koo’s bill to inventory the City’s use of automated decision making systems. This law is a no brainer. Before the City’s transformative open data law was passed in 2012, the City published a similar inventory and catalog of datasets in 1993.”