Does every security problem really need an AI fix?

It’s not predicting, it’s extrapolating

CAS divided neighbourhoods into 125-by-125-metre grids. During the morning briefing, an officer might be told that certain red grids had a higher risk of burglary. Officers would then decide whether to patrol those areas more frequently. As a result, some neighbourhoods received more policing than they would have without the system.

According to Schuilenburg, CAS had numerous fundamental problems. “Traditional crime, such as burglary, has been falling sharply for years. The system therefore had little data to learn from. A city like Los Angeles, where this type of software originates, has far more data than, for example, Dordrecht or Vlaardingen, where CAS was also deployed.” Moreover, CAS did not actually make predictions. “It’s not predicting, it’s extrapolating,” he says. The system simply extends existing trends.

According to Schuilenburg, the system would not have automatically improved its accuracy beyond those two correct predictions in a hundred. “The police are struggling with a shortage of staff. Sending officers to a ‘red’ area means taking them away from elsewhere, and the net benefit of that remains unknown. So a system like this is never separate from the practical realities surrounding it. Furthermore, CAS focused on traditional crime, whereas it is now primarily cybercrime and AI-related crime that are on the rise.”

Is it allowed? A nuance regarding discrimination

Whether something is ‘allowed’ is not that easy to determine and is not the only question that needs answering. The audit by the National Audit Office reveals that, amongst other things, CAS lacked checks for potential bias, and that historical data may be skewed – for example, if certain neighbourhoods were subject to more intensive checks in the past, which could lead to inaccurate predictions. The police’s internal advisory committee reached a similar conclusion and rated the risk of discrimination posed by CAS in 2025 as ‘high’, meaning that, in the committee’s view, the risk of undesirable, discriminatory outcomes for certain groups or neighbourhoods was significant.

Schuilenburg adds a nuance to this. According to him, the police did attempt to develop CAS ethically and removed data that could lead to indirect discrimination, such as personal data and the average income or education level of a neighbourhood. In this respect, he argues, CAS differs from the well-known predictive systems in the United States.

Do we want it? First, ask whether AI is necessary

Schuilenburg poses three questions. “The whole debate surrounding AI and security still revolves largely around the question: is it possible? That is a technical question. I am far more interested in the question: is it permissible? That is a legal question. And do we want it? That is the ethical and social question.”

That final question involves an initial check which he calls ‘Target’, the first of his four Ts (Target, Tracked, Talked and Tested). This test asks whether the problem actually requires AI. According to him, this question is rarely asked, because we increasingly believe that technology can solve every problem. “Do we actually need AI?” Even if the answer is ‘yes’, technology never stands alone. “There is no society without technology. But there is also no technology without society.” This makes everything socio-technological, and according to Schuilenburg, this starting point must be taken into account when researching and evaluating any AI system. “Officers in the Intelligence department tend to have desk-based roles, analysing data and passing on leads to the investigation team. Whereas officers on the beat rely much more on their experience, intuition and contact with local residents. So you also need to look at how and by whom a system is used in practice.”

Impact: from criticism to collaboration

Criticism of CAS had been around for a long time. In 2016, the Police Academy found no evidence that the system reduced crime, and in 2022 the National Audit Office highlighted shortcomings. Yet the system continued to operate for years. Schuilenburg describes it as an open secret that CAS has hardly been used at all in recent years. FTM reports that nobody knew exactly how often teams used it. He sees two realities. “What is actually used in practice is often far less than what is on paper or what is publicised with great fanfare.”

According to Schuilenburg, this highlights the shortcomings of retrospective criticism. Evaluating AI and subjecting it to critical scrutiny remains necessary, but all too often little or nothing happens afterwards. The tool remains in use or is repurposed. “Once the system is in place, as a scientist you’re actually always too late. It rarely goes away.” He has therefore deliberately shifted his own approach in a different direction in recent years. In his view, criticism from a distance had little effect, so he and his group of PhD students are now trying to exert influence at an early stage – for example, during the development phase of a new technology. He believes this is a completely different and new way of working, because a researcher who takes on a partly guiding role also loses some of the traditional academic detachment. “But if you dare to step beyond that, you can really make a difference.” That is why he wants to be involved at an early stage, “because that is when the decisions are made”. For instance, he is a member of the Police Scientific Advisory Council (WARP), which advises the Chief Constable, and he contributes to the police’s ethics forums, where technicians, ethicists, experts by experience and policymakers discuss new tools before they are deployed.

That role requires a balance. Schuilenburg describes himself as a ‘critical friend’. Anyone who offers input runs the risk of an organisation claiming that scientific advice has already been sought. He refers to this as ‘scientist washing’: the risk that an organisation will use a scientist’s involvement to legitimise a new AI system. In addition, he regularly collaborates with investigative journalists and asks his PhD students to write not only a scientific article meeting all the relevant requirements, but also a concise policy recommendation that translates the research into practical terms. For the ‘Harmful Policing’ conference, which he is organising with his PhD students in November, impact is being created in a different way, including through a documentary and an exhibition at the Nieuwe Instituut in Rotterdam. In this way, he aims both to advance the scientific evidence base and to exert a direct influence on practice.

What comes after CAS?

Schuilenburg believes the police deserve credit for discontinuing CAS. “The Netherlands was the first country in the world to introduce predictive policing on a national scale, and our neighbouring countries are watching closely. The thinking is that if the Netherlands stops using it, why should they carry on?” At the same time, he believes this should have happened sooner, as it has been known for some time that many predictive systems are inherently problematic.

In recent years, the police have discontinued several predictive systems, such as RTI-G – a violence detection tool in The Hague – following criticism from Schuilenburg, and now CAS. At the same time, according to Schuilenburg, the police are investing more in tools that operate in real time or analyse past data. Examples include cameras that recognise faces, the detection of pistol and rifle shots, the recognition of human emotions, online data collection and the searching of seized data. He believes that as the number of real-time tools increases, so do the concerns. “With CAS, an officer could still decide not to intervene. With real-time systems, there is far less time to think things through. Whether this will make the future a happier, better and more ethical place, I dare to doubt.”

For Schuilenburg, the lesson is not that predictive technology should be definitively banned at neighbourhood level, as advocated by the human rights organisation Amnesty International. He is convinced that an AI system can only be used responsibly if it is clear in advance what its added value is, whether everything is being done in an ethically, legally and socially responsible manner, and whether there is ongoing evaluation to ensure it actually delivers on its promises. “Until that happens, AI remains a promise without proof.”