Between 2017 and 2021 the city of Rotterdam used an automated system to assign a risk score for welfare recipients. The scores were meant to act as an indicator of potential fraudulent activity, and those who received the highest or “riskiest” scores were subject to investigations.

Lighthouse Reports, in collaboration with WIRED, used freedom of information laws to gain access to the code, training data, and machine learning model that powers the system. In their investigation they discovered that, like many such attempts to automate criminal justice, Rotterdam’s model is rife with problems. Notably, the model appears to discriminate based on gender and language skill (which can be a proxy for ethnicity) in its score determinations. You can read Lighthouse’s full methodology and findings here.

The risk calculator is programmed in R and uses gradient boosted decision trees to determine scores (for a fuller explanation of gradient boosted machines see Lighthouse’s methodology). The system takes in 315 variables, including age, number of children, neighborhood, and so on, and passes those values through 500 decision trees. Each individual tree outputs a numerical score.

Here’s an example of a single tree from the model:

Each box or node in the tree represents a piece of data about a particular welfare recipient. The lines connecting the nodes create a path for the data to flow through. In this example a hypothetical subject had a child when they were younger than 17.5 years old, they lived at their address of less than 4550 days, they had less than 4.5 “appointment writeups with unique results” and so on, leading to a score of 0.0066 for this tree.

The individual scores from all 500 trees are combined to produce the final risk score, a number between 0.0 and 1.0, where 1.0 is “riskiest” and 0.0 is “least risky.”

Working with Lighthouse, the Center for Artistic Inquiry and Reporting, and Eyebeam’s Center for the Future of Journalism, and with assistance from Amy Qu, I was tasked with creating an interface for exploring the risk assessment model. It’s quite unusual to gain access to a trained model that’s actually been in use like this one has, so I decided to make something that would let users run the model in real time.

I was less interested in creating something purely explanatory, and more in giving users full and unmediated access to the system itself. I wanted to highlight the model’s complexity and opaqueness rather than trying to hide it or simplifying it. Can a system such as this one ever be satisfactorily explained?

You can see the results embedded below, or at https://rotterdam.lav.io (note that it works best on desktop):

The form presents the entirety of the model’s 315 input fields, translated from Dutch. As you will likely notice, some fields are more difficult to understand than others, a result of difficult-to-translate terms as well as general bureaucratic impenetrability.

Clicking the “Run Model” button actually runs the model in R, in real time, and then shows how the data runs through each of the 500 trees that constitute the system. The trees can be viewed visually (as in the example above), or in text, as a kind of bureaucratic risk-scoring poem (my preferred mode of experiencing the arbitrary, chaotic and capricious quality of these types of automated decision-making systems).

The project is also available in Dutch at https://rotterdam.lav.io/?lang=nl.

Because Number of registered addresses is more then 7.5, and Because Number of postal addresses is more then 0.5, and Because District Feijenoord is more then 1.5, and Because Number of days at current address is less then 21300.5 and is more then 2732, and Because Neighborhood Groot Ijsselmonde is True, and Because Neighborhood other is True, and Because Lives inside Rotterdam is False, and Because District Charlois is True, and Because Number of home addresses entered manually is True, and Because Lives in the Noord district is True, and Because Lives in district 'other; is True, and Because District Prins Alexander is True, and Because Number of words written by social worker is more then 170.5, and Because Number of appointments about making an appointment plan is more then 0.5, and Because No penalty for failing language requirement is less then 0.5, and Because Number of meetings last year about an exemption from Dutch langauge requirement is more then 0.5, and Because Number of appointments last year because of a social worker tip is less then 1.5, and Because Number of appointments last year: progress and search for work is more then 1.5 and is less then 6.5, and Because Number of messages about a penalty is less then 0.5, and Because Number of total appointments: application completed from web application. is more then 2.5, and Because Appointment Discuss End Looking For Galo Interview is more then 0.5, and Because Number of appointments about work effort is more then 0.5 and is less then 3.5, and Because Number of words in appointment writeups last year is more then 0.5, and Because Number of appointment writeups with unique results last year is more then 3.5, and Because Number of appointments: other is more then 5.5, and Because Number of meetings about a follow up penalty is more then 0.5, and Because Number of appointment writeups with unique results is less then 4.5 and is more then 2.5, and Because Number of appointments because of an employer tip is more then 5.5, and Because Number of meetings because of social worker tip is more then 5.5, and Because appointment add UWVWB Registration is more then 0.5, and Because Number of appointments about not complying with Dutch langauge requirement is less then 0.5, and Because Number of days with financial problems is more then 64, and Because Days with physical problems is more then 445.5, and Because Days with psychological struggles is more then 171.5, and Because Struggles financially is True, and Because Addiction problems is more then 0.5, and Because Obstacle ind Hist is False, and Because Obstacle Psychological problems is False, and Because Obstacle residential situation is True, and Because Abnormal avalibility because of medical conditions is more then 0.5, and Because Availability current deviation due to medical conditions is False, and Because availability recently deviating due to social situation is True, and Because Competence Control is more then 0.5, and Because Lacks skill: analyzing is more then 0.5, and Because Lacks skill: drive and ambition is less then 0.5, and Because Number of outgoing phone calls last year is more then 0.5, and Because Competence convince and influence is more then 0.5, and Because Lacks skill: planning and organizing is less then 0.5, and Because Lacks skill: profesional work experience is more then 0.5, and Because Contacts subject work motivation is more then 5.5, and Because Number of messages about a work intake is more then 0.5, and Because Number of messages about Dutch proficiency assessments is more then 1.5, and Because Contacts Subject Boolean Motivation is True, and Because Contacts topic documents is less then 2.5, and Because Contacts Subject Document Type CV is more then 2.5, and Because Number of agreements is more then 1.5, and Because Number of messages about financial situation is more then 1.5, and Because Contacts Subject group meeting is more then 3.5, and Because Number of messages about income is more then 0.5, and Because Project work and reintegration is more then 1.5, and Because Number of messages about a considered penalty is more then 0.5, and Because Number of messages about change is more then 0.5, and Because Number of missed meerings is more then 0.5, and Because Different types of contact with subject 'other' is more then 24.5, and Because Number of meetings with a financial counselor is more then 0.5 and is less then 2.5, and Because Contacts Subject Quickscan is more then 0.5, and Because Contacts Subject screening is more then 1.5, and Because Number of messages about benefits payback is more then 3.5 and is less then 6.5, and Because Number of messages about work trajecotry is more then 4.5, and Because Contacts Subject Invitation is more then 5.5, and Because Contacts subject work intake did not appear is less then 1.5, and Because Contacts subject Sick or cancellation is more then 0.5, and Because Number of messages health is more then 6.5, and Because Different types of contact with subject 'other' last year is more then 3.5, and Because Number of outgoung documents sent last year is more then 4.5, and Because Number of incoming emails last year is more then 2.5, and Because Number of outgoing emails last year is more then 0.5, and Because Number of messages about an in-person interview is more then 1.5, and Because Number of incoming phone calls last year is more then 0.5, and Because Lacks skill: delivering quality is more then 2.5, and Because Number of messages with subject 'other' is more then 4.5, and Because Number of incoming documents is more then 13.5, and Because Number of outgoing documents is more then 6.5, and Because Number of incoming emails is more then 5.5, and Because Number of outgoing emails is more then 11.5, and Because Number of messages about an in-person meeting is more then 0.5, and Because Contacts Type Group meetings is more then 1.5, and Because Contacts Type Report Participation is more then 2.5, and Because Contacts Type RIB Report is less then 4.5, and Because Number of incoming phone calls is more then 9.5, and Because Number of outgoing phone calls is more then 25.5, and Because Number of unique projects is more then 0.5, and Because Has not started back-to-work plan projects is more then 7.5, and Because Participation Act Reintegration Ladder Supporting Instruments is less then 7.5 and is more then 0.5, and Because Number of messages about the job search effort is more then 19.5, and Because instrument number last year is less then 4.5, and Because Has received labor reintegration support is more then 1.5, and Because Instrument Ladder History Other is less then 1.5, and Because Receives labor reintegration support is more then 0.5 and is less then 1.5, and Because Instrument terminated does not achieve reintegration is more then 0.5, and Because instrument reason Under termination History objective does not reach reintegration is more then 1.5, and Because instrument reason termination history not successful is less then 1.5, and Because Instrument Reason termination History Other is more then 1.5 and is less then 3.5, and Because Instrument Reason termination History Transfer to MO is more then 0.5, and Because Instrument Reason termination History Transfer to regular team IWPM is more then 0.5, and Because Labor reintegration succuesful is more then 0.5, and Because Number of rejected job offers is more then 0.5, and Because Exemption current is False, and Because Average number of exemption days is more then 506.25 and is less then 2144.25, and Because Number of exemption days because of a medical condtion is more then 2137.5, and Because Exemption in history is False, and Because Number of exemptions from finding temporary work is more then 0.5, and Because Gender is False, and Because Age is more then 47.5 and is less then 64.5, and Because Personal characteristics communication is False, and Because Days since personal qualities updated is more then 644.5 and is less then 1311, and Because Days since language requirement is more then 162 and is less then 655.5, and Because Personal characteristics perseverance is False, and Because Personal qualities NL writing false is False, and Because Personal characteristics NL speak2 is False, and Because Personal characteristics disposition remark is True, and Because Personal qualities Other comments is True, and Because Spoken language is more then 0.5, and Because Language requirement completed is more then 0.5, and Because Personal characteristics external care remark is False, and Because Self estimate to find work is less then 5.5, and Because Social worker estimate to find work is less then 2.5, and Because PLA Current PLA Category Objective 9 is False, and Because PLA End of objective reached is more then 0.5, and Because Back-to-work plan objective reached is less then 0.5, and Because PLA End objective not achieved is more then 0.5, and Because Back-to-work plan: has not achieved objective is more then 4.5, and Because PLA End of planned end date exceeded without further notice is more then 0.5, and Because Back-to-work plan objective is not finding work is more then 0.5, and Because PLA HIST PLA Category Objective 1 is less then 3.5 and is more then 0.5, and Because PLA HIST PLA Category Objective 11 is more then 1.5, and Because Back-to-work plan objective: volunteer work is more then 0.5, and Because Back-to-work plan objective: participation in society is more then 0.5, and Because PLA HIST PLA Category Objective 3 is more then 4.5 and is less then 5.5, and Because PLA HIST PLA Category Objective 4 is more then 2.5 and is less then 3.5, and Because PLA HIST PLA Category Objective 5 is more then 2.5, and Because PLA HIST PLA Category Objective Other is more then 0.5, and Because PLA History Social Exercise is True, and Because Back-to-work plan: development is False, and Because PLA History Work is False, and Because PLA currently signed is more then 0.5, and Because Number of signed Back-to-work plans is more then 0.5 and is less then 5.5, and Because relationship child primary school child is more then 0.5, and Because Number of children is more then 0.5, and Because Number of adult children is more then 2.5 and is less then 3.5, and Because Age when had first child is more then 17.5, and Because relationship child teen is more then 0.5, and Because relationship child adult is less then 1.5, and Because Number of current costsharers is more then 0.5 and is less then 9.5, and Because Number of relationships of type 'other' is more then 0.5, and Because Number of administrators is False, and Because Number of roommates is more then 0.5, and Because Number of authorized representatives is more then 0.5, and Because Number of past costsharers is more then 3.5, and Because Relationship Other History Form maintenance person is less then 2.5, and Because Number of costsharers is False, and Because Number of days with a partner is more then 29, and Because Number of days with assigned customer type is more then 3391.5 The Risk Score Is 0.6661.