Published:
Tuesday, July 11, 2023
In recent years, touting fairness and consistency, local governments have begun turning to predictive analytics tools to aid in child welfare agencies’ decision-making. However, these systems raise a lot of questions about their purpose and effectiveness, as well as the unintended consequences they create. By exploring how these systems have been designed and used, their flaws become more apparent. When it comes to the welfare of children, the risks of automated systems like predictive analytics are worthy of consideration.
The purpose of child welfare predictive analytics:
While child welfare predictive analytics tools are described as predicting “risk” or generating “risk scores,” indicating the likelihood of child maltreatment, they are actually doing something very different. Rather than estimating the likelihood that the same child will be the subject of a future child maltreatment allegation or, more common, the likelihood a child might be harmed, most of the tools currently in use are designed to predict the likelihood that the child welfare agency will remove a child from their family and place him or her in foster care (“child removal”).
So, these analytics tools are deployed with the goal of preventing harm to children but are actually designed for another purpose. Probing further, how close are the proxies of maltreatment reports or child removal to the thing agencies are actually trying to guard against, namely, harm to a child?
Reports of neglect or abuse have been time and again proven to be poor indicators of actual child maltreatment. The most recent federal data collection shows that, in 2021, approximately 61% of allegations (or 60,529 out of 98,934) received by Washington child welfare authorities were dismissed outright, and no agency action taken.[i] Of the children that authorities did follow up on, 8% (or 3,487 of 43,474) were found to be victims of maltreatment, and the majority of these substantiated allegations were for neglect (as opposed to abuse).[ii]
Meanwhile, child removal is several steps, and actors, removed from the original conduct that prompted an investigation in the first place. For instance, in Washington, child removal happens only after:
How past disparities impact future decisions:
For decades, women and children who are Indigenous, Black, or experiencing poverty have been disproportionately placed under child welfare’s scrutiny. Once there , Indigenous and Black families fare worse than their white counterparts at nearly every critical step. These disparities are partly the legacy of past social practices and government policies that sought to tear apart Indigenous and Black families. But the disparities are also the result of the continued policing of women in recent years through child welfare practices, public benefits laws, the failed war on drugs, and other criminal justice policies that punish women who fail to conform to particular conceptions of “fit mothers.”
Thus, by using historical records as the building blocks of today’s predictive analytic tools, the discriminatory outcomes of the past are replicated through an algorithm, which gives the veneer of objective, mathematically proven “truths” but, in reality, is flagging certain communities for regulation and intervention simply because the jurisdiction had regulated and intervened in those families in the past.
How these tools are currently created and used:
While the tools more commonly in use or in development today can be divided into the four types described below, they are typically created in a similar manner. Tool designers pull an agency's historical cases and, either through human review or statistical modelling, identify recurring characteristics in those cases. Generally, these characteristics – which are called variables or features – are things associated with people described in the allegations (e.g., a family’s or individual’s prior use of public benefits, involvement with the foster care system, criminal history, or housing instability) or with the neighborhood in which the alleged incidents took place (e.g., high violent crime rate, proximity to condemned or foreclosed properties). Then the designers identify which of those variables are associated with different outcomes, assign mathematical weights based on how strong the association is, and devise a formula for calculating the likelihood of those outcomes occurring. The most common types of tools are:
While these tools are marketed as improving decision-making and reducing bias, in practice, they can harm decision-making and increase bias. In 2017, Illinois stopped using a tool marketed by a Florida-based social service provider, Eckerd Connects, after it both failed to predict several high-profile child fatalities and flagged an inordinate number of children as having a 90% or greater risk of death or injury. This avalanche of red flags made the truly dangerous cases hard to find and, upon closer review, reflected the tool’s inability to meaningfully distinguish between families in need and families at risk.
A few years prior, Los Angeles County decided not to implement a model after an evaluation showed its inutility. The model accurately identified 76% of cases that resulted in death or severe injury, but also flagged another 3,829 cases as falling in the same high-risk category, 95% of which did not result in such harm.
While Allegheny County, PA and the designers of its Family Screening Tool (AFST) claimed a reduction in disparities between Black and white children’s case opening rates, other serious problems persisted. As their own impact evaluation report notes, the screen-in rates for Black and white children alike were declining before the tool was first introduced. And upon implementation, that pre-existing decline “largely halted” through at least 2021. In addition, while the tool was in place from February 2019 through June 2021, the majority of allegations that were screened-in were investigated but no need for any services or further intervention was deemed necessary for the well-being of any children in those households. In other words, even with introduction of the AFST, many families that did not raise any legitimate child maltreatment concerns continued to be investigated . The harms of investigation, even in the absence of any family separation have been documented by research and first-hand accounts of parents and children. Families describe child welfare investigations as frightening, invasive, stigmatizing, and rife with bias. This is not surprising since CPS caseworkers are responsible not just for identifying services that might help a family, but also for, first, investigating whether a parent has endangered their child and, later, monitoring the parent to document signs of parental unfitness. The number and scope of unnecessary CPS investigations harms children and families and wastes resources.
Demonstrable harm. Questionable benefit:
Despite predictive analytics tools being fraught with biases and exacerbating existing harms, agencies are using these systems often without any transparency or accountability. In the family regulation space, individuals who are assessed by these tools do not often know these systems are in use. Additionally, the majority, if not all, jurisdictions using such automated risk assessments do not inform parents or children who are subjected to investigation what the tool concluded about them and how that conclusion impacted CPS workers’ decision-making. As a result, families have no opportunity to contest, explain, or refute at least some of the grounds on which CPS involvement was justified. So, for instance, a tool that considers prior child welfare involvement, multiple home addresses, and a single-parent household as characteristics that, to varying degrees, might indicate future “risk,” a parent cannot point out that the prior CPS investigation was prompted by an inaccurate medical diagnosis, the housing instability did not in and itself result in poor parenting, or that they left their former spouse because of intimate partner violence. Yet, all of these factors, even if buried in code, are among the reasons why a screening worker decided their parenting abilities were questionable.
It is critical that these systems are subject to transparency and accountability standards that at a minimum, would allow people to know how decisions about them are being made and seek remedies for harms. In Washington state, we have introduced a first-of-its kind algorithmic accountability bill (SB 5356) that would prohibit discrimination via algorithm and require government agencies to make transparent the automated decision systems they use. We will continue to advocate for legislation to hold systems like predictive analytics tools accountable.
The purpose of child welfare predictive analytics:
While child welfare predictive analytics tools are described as predicting “risk” or generating “risk scores,” indicating the likelihood of child maltreatment, they are actually doing something very different. Rather than estimating the likelihood that the same child will be the subject of a future child maltreatment allegation or, more common, the likelihood a child might be harmed, most of the tools currently in use are designed to predict the likelihood that the child welfare agency will remove a child from their family and place him or her in foster care (“child removal”).
So, these analytics tools are deployed with the goal of preventing harm to children but are actually designed for another purpose. Probing further, how close are the proxies of maltreatment reports or child removal to the thing agencies are actually trying to guard against, namely, harm to a child?
Reports of neglect or abuse have been time and again proven to be poor indicators of actual child maltreatment. The most recent federal data collection shows that, in 2021, approximately 61% of allegations (or 60,529 out of 98,934) received by Washington child welfare authorities were dismissed outright, and no agency action taken.[i] Of the children that authorities did follow up on, 8% (or 3,487 of 43,474) were found to be victims of maltreatment, and the majority of these substantiated allegations were for neglect (as opposed to abuse).[ii]
Meanwhile, child removal is several steps, and actors, removed from the original conduct that prompted an investigation in the first place. For instance, in Washington, child removal happens only after:
- a screening worker decides that allegations warrant some action by Child Protective Services (CPS);
- a CPS caseworker conducts an investigation;
- the state Attorney General’s office files in court seeking removal and makes factual and legal arguments;
- a guardian ad litem is potentially appointed for their own investigation and recommendation, representing the “best interests” of the child;
- and a judge who makes the removal decision.
How past disparities impact future decisions:
For decades, women and children who are Indigenous, Black, or experiencing poverty have been disproportionately placed under child welfare’s scrutiny. Once there , Indigenous and Black families fare worse than their white counterparts at nearly every critical step. These disparities are partly the legacy of past social practices and government policies that sought to tear apart Indigenous and Black families. But the disparities are also the result of the continued policing of women in recent years through child welfare practices, public benefits laws, the failed war on drugs, and other criminal justice policies that punish women who fail to conform to particular conceptions of “fit mothers.”
Thus, by using historical records as the building blocks of today’s predictive analytic tools, the discriminatory outcomes of the past are replicated through an algorithm, which gives the veneer of objective, mathematically proven “truths” but, in reality, is flagging certain communities for regulation and intervention simply because the jurisdiction had regulated and intervened in those families in the past.
How these tools are currently created and used:
While the tools more commonly in use or in development today can be divided into the four types described below, they are typically created in a similar manner. Tool designers pull an agency's historical cases and, either through human review or statistical modelling, identify recurring characteristics in those cases. Generally, these characteristics – which are called variables or features – are things associated with people described in the allegations (e.g., a family’s or individual’s prior use of public benefits, involvement with the foster care system, criminal history, or housing instability) or with the neighborhood in which the alleged incidents took place (e.g., high violent crime rate, proximity to condemned or foreclosed properties). Then the designers identify which of those variables are associated with different outcomes, assign mathematical weights based on how strong the association is, and devise a formula for calculating the likelihood of those outcomes occurring. The most common types of tools are:
- Screening: Used at intake. This type of tool estimates the likelihood of a child maltreatment-proxy event occurring in the future and either outputs a “risk score” or recommended action based on its estimation. These become one factor used by screening staff in deciding whether to refer the report in for further agency action or close it at the outset.
- Service-matching: Used when deciding what type of intervention is appropriate for a given child/family. This tool type categorizes or ranks the child/family in one of several levels of need or “risk,” which the agency uses to identify which services or programs to provide the child/family.
- Open-case review: Used when matter is already under investigation up until when matter might be closed. A tool of this type generates a “risk” score or classification for each matter and, those with scores beyond a certain threshold or in certain classifications are flagged for closer scrutiny or assignment to more experienced caseworkers.
- Geo-spatial risk models: Not used in connection with specific reports but more generally. This tool identifies “risk” associated with different geographic areas, rather than with families or children.
While these tools are marketed as improving decision-making and reducing bias, in practice, they can harm decision-making and increase bias. In 2017, Illinois stopped using a tool marketed by a Florida-based social service provider, Eckerd Connects, after it both failed to predict several high-profile child fatalities and flagged an inordinate number of children as having a 90% or greater risk of death or injury. This avalanche of red flags made the truly dangerous cases hard to find and, upon closer review, reflected the tool’s inability to meaningfully distinguish between families in need and families at risk.
A few years prior, Los Angeles County decided not to implement a model after an evaluation showed its inutility. The model accurately identified 76% of cases that resulted in death or severe injury, but also flagged another 3,829 cases as falling in the same high-risk category, 95% of which did not result in such harm.
While Allegheny County, PA and the designers of its Family Screening Tool (AFST) claimed a reduction in disparities between Black and white children’s case opening rates, other serious problems persisted. As their own impact evaluation report notes, the screen-in rates for Black and white children alike were declining before the tool was first introduced. And upon implementation, that pre-existing decline “largely halted” through at least 2021. In addition, while the tool was in place from February 2019 through June 2021, the majority of allegations that were screened-in were investigated but no need for any services or further intervention was deemed necessary for the well-being of any children in those households. In other words, even with introduction of the AFST, many families that did not raise any legitimate child maltreatment concerns continued to be investigated . The harms of investigation, even in the absence of any family separation have been documented by research and first-hand accounts of parents and children. Families describe child welfare investigations as frightening, invasive, stigmatizing, and rife with bias. This is not surprising since CPS caseworkers are responsible not just for identifying services that might help a family, but also for, first, investigating whether a parent has endangered their child and, later, monitoring the parent to document signs of parental unfitness. The number and scope of unnecessary CPS investigations harms children and families and wastes resources.
Demonstrable harm. Questionable benefit:
Despite predictive analytics tools being fraught with biases and exacerbating existing harms, agencies are using these systems often without any transparency or accountability. In the family regulation space, individuals who are assessed by these tools do not often know these systems are in use. Additionally, the majority, if not all, jurisdictions using such automated risk assessments do not inform parents or children who are subjected to investigation what the tool concluded about them and how that conclusion impacted CPS workers’ decision-making. As a result, families have no opportunity to contest, explain, or refute at least some of the grounds on which CPS involvement was justified. So, for instance, a tool that considers prior child welfare involvement, multiple home addresses, and a single-parent household as characteristics that, to varying degrees, might indicate future “risk,” a parent cannot point out that the prior CPS investigation was prompted by an inaccurate medical diagnosis, the housing instability did not in and itself result in poor parenting, or that they left their former spouse because of intimate partner violence. Yet, all of these factors, even if buried in code, are among the reasons why a screening worker decided their parenting abilities were questionable.
It is critical that these systems are subject to transparency and accountability standards that at a minimum, would allow people to know how decisions about them are being made and seek remedies for harms. In Washington state, we have introduced a first-of-its kind algorithmic accountability bill (SB 5356) that would prohibit discrimination via algorithm and require government agencies to make transparent the automated decision systems they use. We will continue to advocate for legislation to hold systems like predictive analytics tools accountable.
[i] See Child Maltreatment Report 2021, at 13 (available at https://www.acf.hhs.gov/sites/default/files/documents/cb/cm2021.pdf).
[ii] See Child Maltreatment Report 2021, at 44-45 (available at https://www.acf.hhs.gov/sites/default/files/documents/cb/cm2021.pdf).
