Looking at discussions of Regulating Artificial Intelligence it struck me that a lot isn’t new, and a lot isn’t specific to AI. Jisc already has a slightly formal Pathway document to help you identify issues with activities that might involve AI. But here are some topics that seem to often come up in those discussions. Thinking about these, or even realizing you already have thought about them, might reassure you that just because something has the marketing label “AI”, it might not be either as new or as uncertain, as you thought.

Context . Rather than the technology, think about the situation and process in which you are proposing to use it. Is it a situation where human empathy is critical, or is it more important that actions and decisions reflect what the data and statistics tell us? Make sure systems and processes bring components together in an appropriate way.

Bias . If a situation does involve data, do you understand the characteristics of what you have, and the effects of how you might use it? Biased data and processes may be most obvious when they result in discrimination, but data quality and meaning can also be affected by different learning or teaching styles, access to systems or equipment. That may not be a bad thing (focused actions may be what we want), so long as we understand what those effects are and can justify and account for them. But if data or actions exclude certain groups or situations, this should be deliberate, not accidental.

Where data relate to individuals , the Information Commissioner has already published comprehensive guidance on issues likely to arise with “AI” tools and approaches .

The term “Artificial Intelligence” creates a high risk of different kinds of (self-) deception . Just because something can communicate in natural language doesn’t mean it is human, has any other human attributes or understands the sequence of letters it produces; just because something looks like a photograph or video doesn’t mean it actually happened. Think whether the context around your technology is likely to encourage this kind of misunderstanding: most AI Principles require that technology must declare itself, but that doesn’t always seem to be effective.

Finally, an area that does need new thinking is where technology replaces a human that has a particular legal role, presumptions and responsibilities. Non-human “drivers”, “authors”, “performers”, etc. leave gaps in existing legal frameworks that could produce a nasty surprise. Rather than grand “ AI laws ”, however, these typically need specific solutions, maybe in the form of interpretive guidance (“authors’ legal rights pass to X”) rather than laws. The EU’s proposal on AI liability is an interesting approach: essentially suggesting a starting point for discussions of where displaced liabilities might land.

Looking at the contents of the Government’s new Bill suggests it may be more about Digital Information than Data Protection:

• Personal Data Processing (1-23)
• National Security & Intelligence Services (24-6)
• Information Commissioner’s Role etc. (27-43)
• Miscellaneous (44-5)
• Digital Verification Services (46-60)
• Customer & Business Data (a general framework for services like Open Banking) (61-77)
• Privacy and Electronic Communications (78-86)
• Trust services (87-91)
• Public service data sharing etc (92-99)
• Information Commission Governance (100-106)

Even those first 23 clauses, which are about the day-to-day processing of personal data, are largely clarifications or re-phrasings of existing (UK) GDPR and Data Protection Act provisions, so seem unlikely to result in organisations changing their existing processes.

The Act is presented as a series of amendments to existing laws, which makes it hard to interpret, but things I spotted include:

• New (narrower) definition of personal data (c1). The original GDPR definition (Art.4(1)) depends on the phrase “identifiable natural person” but didn’t answer the question “identifiable by whom?”. For a while there were two conflicting German cases, one saying “by the data controller”, the other “by anyone in the world”. The DPDIB goes for the former. As far as I know this was never resolved at EU level: perhaps because, when designing processes, it doesn’t really matter. Any collection of data is likely to include some people who the data controller can identify – customers, staff, those who have self-identified, … – so you have to have a process for people “identifiable by me” anyway. And if you have such a process it seems like a lot of work to develop a parallel process to work out which records are “not identifiable by me” and handle those differently.
• Research and Statistical Purposes (c2,3,9&22). These sections largely consist of text copied or redrafted from the GDPR’s many Recitals (notably 156-162) and Articles (principally 89) on “scientific research”. It has been suggested that the new definition in c2 expands such research to cover both commercial and non-commercial activity, but I can’t see anything in the original GDPR (several different language versions) that currently limits those provisions to “non-commercial” activities. Indeed the current ICO guidance says “ It can also include research carried out in commercial settings, and technological development, innovation and demonstration ”.
• Recognised legitimate interests (c5 & Annex 1). These seem to be a list of purposes that could have been covered by the existing “public interest” and “legitimate interest” justifications. Worryingly, although the category name still mentions “legitimate interest”, there doesn’t seem to be any requirement to consider the interests or fundamental rights and freedoms of individuals, as under the old Article 6(1)(f) Legitimate Interests. We have found that balancing test really useful to reassure both ourselves and our users that we are applying appropriate safeguards, for example when processing to ensure system, network and data security .
• Automated decision-making (ADM) (c11). This seems to answer another long-standing (since 1995!) open question: whether a “right not to be subject” to ADM is a preemptive ban on making such decisions, or creates a right of human review of individual decisions. In the UK, under the DPDIB, the answer seems to be a ban on ADM using special-category data (unless “necessary for reasons of substantial public interest”, in Art.9(2)(g)) but a right of review otherwise.
• Data Protection Impact Assessments (c17&18). These have been relabelled as Assessments of High Risk Processing, and several sources of guidance have been removed, but the basic idea remains: when planning processing the controller must think about potential risks to individuals’ rights and freedoms and take documented measures to ensure any high risks are mitigated. Jisc has found this a very useful tool in planning our own services and reassuring their users (see for example our published DPIAs on network security and learning analytics ).

In going through the new (2023) Data Protection and Digital Information (No.2) Bill I noticed that it does actually make a change to UK law on cookies: according to clause 79(2A), consent will no longer be needed to store or access information in the user’s terminal equipment if this is

• done by the person who operates the website, and
• the sole purpose is to collect statistical information to improve either the website or the service it offers, and
• that information is not shared with any other person for any other purpose.

Otherwise the new clause 79 pretty much reproduces the existing rules dating back to 2009. And this new exemption (colloquially known as first-party analytics) was actually proposed by European regulators in late 2012. To be fair, European legislators didn’t get around to proposing the change till 2017 and their law still hasn’t passed . So although both legislations are still declaring this a solution to “consent fatigue”, it doesn’t seem as if there’s much enthusiasm for it.

I think there might be some broader lessons here for the capabilities and limitations of “regulation”, whether at organisational, national or international scale.

Have we already got a regulation that could cover this harm? Back in 2009, the concern wasn’t primarily the storage of cookies, but the privacy invasions enabled by tracking individual users. Cookies were the main way that was then done, but many other technologies can be, and are, now used. Cross-site tracking was considered particularly harmful, hence the long-standing distinction between first- and third-party analytics. Privacy harms are, of course, the remit of privacy and data protection laws, and (as I discussed in a journal paper ) European data protection law already contained a framework that could have been used to develop a limited, and technology-neutral, framework for website improvement (an obvious “legitimate interest”) in ways that safeguarded users’ rights and freedoms. But, instead, a solution was sought in new provisions on storage and access which, by some accounts, were actually intended to deal with spyware. Rather than spending the last decade (and counting) discussing what is and is not acceptable behaviour by websites, we’ve been producing ever less relevant technical distinctions.

Is the harm controlled by someone responsive to our regulation anyway? That 2009 “spyware” provision had the simple idea that users should be free to accept or refuse the addition of additional software they hadn’t asked for. But, as the market developed, it quickly became apparent that how/whether that choice was offered to users depended on the website, the browser and, particularly, plugins and add-ons to both. It’s still uncommon for consent interfaces to give equal prominence to “accept all cookies” and “reject all cookies”, even though this has been a clear requirement of European law since 2018 (“It shall be as easy to withdraw as to give consent” (GDPR Art 7(3)). It seems the providers of that software are more responsive to other pressures.

Will the reaction to regulation actually deliver what we want? Where cookie banners have responded to changes in law, this typically involves making them larger, more frequent and more intrusive. The term “consent fatigue” quickly emerged. I can’t believe this was the intention of the regulators, but I think it could have been foreseen. When proposing a change to law or policy or any other kind of “rules”, it’s worth role-playing how people and organisations might respond. If that doesn’t help the original problem, maybe it’s worth considering another – maybe even an existing – approach?

Whether you refer to your technology as “data-driven”, “machine learning” or “artificial intelligence”, questions about “algorithmic transparency” are likely to come up. The finest example is perhaps the ICO’s heroic analysis of different statistical techniques . But it seems to me that there’s a more fruitful aspect of transparency earlier in the adoption process: why was a particular mix of technology, theory and human skill chosen, and what contribution does each of these make to a successful process? Thinking about that might help both deployers of technology, and those it is intended to help, to find better approaches.

Where a process draws insights from existing data there’s also a question about why that particular aspect of the past was considered informative. This doesn’t have to be as fundamental as concerns over ChatGPT’s selection of source material , but can be a helpful reminder of likely limits. If a target measure of student engagement was derived from text-based courses, it’s worth checking whether that measure is also appropriate for more practical activities. Does it still reflect the desired balance of participation and autonomous learning? Or, if our aim is to improve a process, does it make sense to still use data from an older, pre-improved, version of that process to inform our activities?

This sort of transparency seems to add value to another popular idea: “ AI registers ”. A public explanation of why an organisation decided to use automation in its delivery of services would help me – even as a lapsed mathematician – much more than a statement that it uses “random forest” algorithms. And I’d hope that writing that explanation would help the organisation build confidence in its choices, too.

A few weeks ago I presented on “ORCID and GDPR” at a UK Consortium event. I hope this was reassuring: I’ve always been very impressed with ORCID’s approach to Data Protection (in the European sense of “managed processing”, not the more limited one of “security”), but take it from the German Consortium’s lawyers , back in 2018:

The data protection assessment of ORCID has not been able to identify any serious deficiencies. On the contrary, with its privacy functionalities, the system supports users in exercising their right to informational self-determination and at times has a role model in this regard

The one circumstance where “a risk-free forecast cannot be made” – a remarkably high standard – was where individual researchers could not freely consent to processing of their ORCID IDs: for example where this was required by employers or funders.

Here, it’s important to recall that researchers’ personal data is already being processed by institutions, funders, publishers. And usually much more of it than is required for a functional ORCID record. Those data controllers ought to have identified a GDPR lawful basis for that processing, so the simplest approach is to consider the same lawful basis for ORCID IDs. As the Germans noted, Consent is unlikely to be valid, but there are at least three other possibilities:

• Necessary for (employment) Contract: in the sense that the substance of the contract can’t be achieved with any less processing;
• Necessary for Public Task;
• Necessary for organisation’s Legitimate Interest.

Each of those includes requirements to reduce both risk and – because they all include the word “necessary” – processing, and it may well be that an “ORCIDised” (sorry!) version of the process can deliver both of those. To check that, and to reassure individuals and regulators, I’d suggest following and documenting the following steps:

1. What is the purpose of processing?
2. Is that purpose legitimate ?
3. Can the purpose be achieved less intrusively (for example, can we let researchers choose whether or not to populate/release some fields in their records, using ORCID’s fine-grained controls?)?
4. What organisational and technical safeguards can we apply?
5. Does the remaining risk to the individual override the benefit of the purpose?

Those familiar with data protection will recognise this as the Article 6(1)(f) Legitimate Interest Assessment (which is effectively a superset of the requirements for the other lawful bases) and indeed an LIA or Data Protection Impact Assessment ( DPIA ) might be good ways to document this thinking.

This approach should also highlight opportunities to use ORCID itself as a safeguard: an ORCID ID already has the technical characteristics of a pseudonym (GDPR Art 4(5)). Using ORCID in your systems should also help with organisational safeguards, for example by reducing the need for re-typing, and the risk of confusing different researchers with similar names.

The Home Office consultation on Computer Misuse Act (CMA) reform raises the possibility of a new offence of “possessing or using illegally obtained data”. This is presumably in response to the growing complexity of cyber-crime supply chains. It’s good to see immediate recognition that this will need “appropriate safeguards”. This post looks at why someone in possession of information obtained through crime may not be a criminal and, indeed, may be engaged in activities that society (and victims) should encourage.

Information obtained through cyber-attacks is often left, or made, publicly available. Most simply, the criminal may not wish to send information direct from the victim system to their home base, which could leave a direct trail to their identity. Instead, information is often exfiltrated to third-party storage: either deliberately or accidentally this often has no access restrictions. At a later stage, various ways of converting information into money require the criminal to demonstrate publicly the quality of what they have, either to strengthen a blackmail demand or to demonstrate to potential buyers that they have something of current value.

Incident response teams often seek out these public collections of “illegally obtained data”, to obtain early warning of successful attacks, to directly help victims reduce damage (for example by changing passwords or cancelling credit cards) and because it may help to determine how and when the attack occurred. Obtaining and sharing information “to contain the effects of incidents and recover more efficiently” is recognised by the NIS2 Directive (Recitals 119 and following) as something to be encouraged, so clear safeguards are essential to ensure there is no fear that it might be challenged under any new law. Indeed it is not clear that such a law is needed: there are already criminal offences under s3A of the current CMA (for “making, supplying or obtaining articles for use in offence”, which was justified at the time as covering lists of passwords and credit card numbers) and s170 of the Data Protection Act 2018 (for “unlawful obtaining etc. of personal data”, which includes “retaining”), which may well be sufficient to address the harms identified without creating any new perverse incentives. With so few cases under these provisions being reported, it’s hard to know whether what’s needed is more laws or more investigation and enforcement. In the meantime, it’s essential not to discourage the protective mechanisms we do have.

When the Internet first came to legislators’ notice, there was a tendency to propose all-encompassing “laws of internet” for this apparently new domain. A celebrated paper by Frank Easterbrook argued that (my summary) there wasn’t a separate body of new harms to address and that existing laws might well prove sufficiently flexible to deal with many of them. The title pointed out that studying (or creating) the “law of the horse” would ignore a lot of the legal and social principles that are already widely established. Looking at proposals for “AI laws”, I wonder whether we might be back in similar territory?

The proposed EU AI Act doesn’t seem self-confident. First, it has to define “AI”, then it declares that most of that definition doesn’t need regulating anyway and, for the rest, proposes something that looks a lot like a traditional product safety law. The Act is already being criticized for an over-simplified view of supply chains . Perhaps starting with a scope that encompasses everything from speech recognition to probation recommendations was too ambitious? Lack of an AI law doesn’t seem to have hindered courts, which have applied everything from data protection to discrimination laws to reach apparently satisfactory conclusions to harms caused by AI. A very different approach is taken by the proposed EU AI Liability Directive : rather than creating new laws this suggests how existing ones might be applied in complex AI supply chains.

So, for both legislators and developers of new technologies, the message seems to be to check how existing laws will apply. If that doesn’t seem right, try to work out an interpretation that fills the gap (or addresses any genuinely new harms) in the spirit and objectives of the existing rules. A recent review of “The Law of the Horse” considers this in more detail. For those developing or applying “AI”, make sure you understand how existing laws on personal data, discrimination and safety will apply to your idea. You may well find more guidance there than you expect.

The recent rash of ransomware incidents has been linked to the availability of crypto-currencies – as a way that victims can pay ransoms to anonymous attackers – so Trend Micro reviewed the economic models for ransomware and, among many other aspects, whether changes in the crypto-currency world might have knock-on effects. Their conclusions are mixed: successful intrusions can be monetised in other ways, but defences that focus on initial access and lateral movement should help against those too.

Crypto-currencies have been in the news themselves: some collapsing for internal reasons, others being proposed for regulation. Some crypto-currency and ransomware groups have been made subject to sanctions. However Trend see these as long-term developments that may, at most, increase the costs to cyber-criminals who continue to use ransoms to monetise their access to organisations’ systems. In any case, extortion is not the only way that profits can be extracted: fake invoices and bank instructions are much more plausible if sent from the organisation’s own systems, for public companies there are signs of intruders using “inside information” to distort share prices in profitable ways.

The good news for defenders is that these other monetisation techniques still depend on initial access plus lateral movement/privilege escalation to reach sensitive information and systems. So preventing, detecting and eliminating either of these earlier stages should continue to be effective even if the eventual monetisation technique changes.

Over the past few months there has been a lot of discussion of the impact of the Government’s Online Safety Bill on large providers. Ofcom’s July 2022 Implementation Roadmap (p5) estimates that there are 30-40 of those, to be covered by Categories 1, 2a and 2b. However the roadmap mentions a further 25000 UK services that will be in scope of the Bill: “Broadly speaking, services where users may encounter content (such as messages, images, videos and comments) that has been generated, uploaded or shared by other users will be in scope of the online safety regime” (p11). There are some exemptions in the Bill but, for example, none of those seem to apply to the comment feature on this blog. What might the Bill require here?

Although the Bill has been subject to considerable change, two types of content have been a consistent focus: “illegal” and “harmful to children”. In each case it’s envisaged that there will be a list of specific kinds of harm: service operators will need to first assess the risk of each kind of material appearing, then apply appropriate safeguards to that risk. Whether the children list needs to be considered depends on each service’s assessment of “whether children are likely to access their service or part of their service” (p16). The categories considered “harmful to children” will be defined in a future statutory instrument; those considered “illegal” are currently in Schedules 5 (Terrorism), 6 (Child Sexual Exploitation and Abuse) and 7 (Assisting Suicide, Threats to Kill, Public Order Offences, Drugs and Psychoactive Substances, Firearms and Other Weapons, Illegal Immigration, Sexual Exploitation, Sexual Images, Proceeds of Crime, Fraud, Financial Services crimes) though this may change.

All services will need to implement processes to receive reports of content in their relevant categories, to take down (at least) illegal content, and to deal with complaints about these processes (p14). There will also be duties on “review and record keeping” (p13) including – according to clause 29 of the current Bill – “every risk assessment”, “any measures taken” and regular compliance reviews.

For small sites, the amount of work will depend heavily on the required risk assessments and safeguards. The Bill seems to require that these are done separately for each kind of harm (current clause 8(5)(b)(i) ), but details of how to assess and what protection is required are left to Ofcom. For illegal content, their Roadmap suggests:

“This must assess, amongst other things, the risk of individuals encountering illegal content on a service, the risk of harm presented by illegal content and how the operations and functionalities of a service may reduce or increase these risks” (p14)

and

“All services will need to put in place proportionate measures to effectively mitigate and manage the risks of harm from illegal content.” (p14)

There are similar requirements for the “harmful to children” categories.

A lot will depend on those words “proportionate” and “effectively”. Will it be sufficient, for example, to say that all comments to this site are already checked and approved by humans before they are published? I can’t think what we could do that would further reduce any (I hope, low) risk of encountering illegal or harmful content. Ofcom do note that large services have “capabilities and resources that vastly outstrip those of most in-scope services” (p8) and “each service’s chosen approach should reflect its characteristics and the risks it faces” (p5). But the Bill applies the same risk management framework to everyone, so their flexibility may be limited.

The Bill was significantly changed in December 2022, and Ofcom’s Roadmap refers to an earlier version. I have concentrated here on areas which were not affected. However the Bill is yet to go to the House of Lords (expected Jan/Feb 2023) and both government and opposition have declared their intention to make further changes there. Other obligations may appear or disappear. But if it is to become law, the Bill needs to be agreed before the summer. Ofcom’s powers will commence two months after that happens, and the Roadmap envisages a consultation on draft guidance on illegal content shortly thereafter, with a final version a year later (p7). Categories harmful to children need to be defined in further legislation, so that guidance is likely to appear later following a similar process.

The final text of the revised European Network and Information Security Directive (NIS 2 Directive) has now been published. This doesn’t formally apply in the UK, but does have some helpful comments on using data protection law to support network and information security. I’ve blogged about these previously but, since the final version significantly changes the draft numbering, I thought it was worth posting a revised index to those posts:

CSIRT (international) Information Sharing : Draft Recital 69, which encouraged incident response and information sharing, is now split across Recitals 120 and 121. The former is now even more explicit that “entities should be encouraged and assisted by Member States to collectively leverage their individual knowledge and practical experience at strategic, tactical and operational levels with a view to enhancing their capabilities to adequately prevent, detect, respond to or recover from incidents or to mitigate their impact”. The societal importance of this is still in Recital 3.

CSIRT Collaboration: Helpfully, the Directive separates “reporting obligations” (Article 23) of various kinds of regulated entities from more general “exchange on a voluntary basis” (Article 29, formerly 27), which should  involve anyone with relevant information and skills to improve the security of networks, systems and data. The latter might include “information relating to cyber threats, near misses, vulnerabilities, techniques and procedures, indicators of compromise, adversarial tactics, threat-actor-specific information, cybersecurity alerts and recommendations regarding configuration of cybersecurity tools to detect cyberattacks”, so long as the aim is “to prevent, detect, respond to or recover from incidents or to mitigate their impact” with the effect of “enhanc[ing] the level of cybersecurity”, again with an extensive range of examples: “raising awareness in relation to cyber threats, limiting or impeding the ability of such threats to spread, supporting a range of defensive capabilities, vulnerability remediation and disclosure, threat detection, containment and prevention techniques, mitigation strategies, or response and recovery stages or promoting collaborative cyber threat research between public and private entities”.

Lots here to support our activities.

The European legislative process on Artificial Intelligence has moved on one step with the Council of Ministers (representatives of national governments) agreeing on their response to the text proposed by the European Commission last year. The main focus of the proposed law is makers of products that use “AI”: where these are designed for a specified list of “high-risk” purposes, the products must be designed and documented according to set rules. Those rules – covering things like risk and quality management, transparency, human oversight and interpretation, logging, accuracy, robustness and security – seem valuable for any AI: the question is when they should be formal, rather than informal, requirements.

The Commission identified education as a field that might contain high-risk applications. Their proposed scope has typically been summarised as “high-stakes assessment”, though the formal specification ( para 3 of Annex III ) is a bit longer:

Education and vocational training:
(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and vocational training institutions;
(b) AI systems intended to be used for the purpose of assessing students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to educational institutions.

The Council’s text is pretty similar on point (a), but seems to be significantly different in (b):

Education and vocational training:
(a) AI systems intended to be used to determine access, admission or to assign natural persons to educational and vocational training institutions or programmes at all levels;
(b) AI systems intended to be used to evaluate learning outcomes, including when those outcomes are used to steer the learning process of natural persons in educational and vocational training institutions or programmes at all levels.

Here “assessing students” has been replaced by “evaluate learning outcomes”, with an illustrative example of “steer[ing] the learning process of natural persons”. This feels a lot more like something that would take place during a course, not just at the start or end. Many examples of personalised learning seem quite close to this definition, for example consider an online language course that identifies a student as having difficulty with the past tense and “steers” their revision exercises to focus on that.

Under the Council’s proposal, fitting the Annex III definition isn’t the sole determinant of whether an application needs to demonstrate formal compliance: they have added a final per-application test “of the significance of the output of the AI system in respect of the relevant action or a decision to be taken”. My language tutor might be ruled “purely accessory in respect of the relevant action or decision to be taken and is not therefore likely to lead to a significant risk to the health, safety or fundamental rights” (Article 6(3)). But if the Council’s broadening of scope is intended as I’m reading it, it might be  interesting to consider which processes and decisions within a course might create such risks.

The European Parliament is expected to produce its version of the text in the first quarter of 2023; the three bodies then agree on a final version, which can take months or years. This won’t apply directly in the UK, but if AI products we use are also designed for the European market, we may see the results of the required design processes and documentation.

One promising application of Machine Learning in education is marking support. Colleagues in Jisc’s National Centre for AI have identified several products that implement a similar process, where a program “watches” a human marking assessments, learns in real time, and suggests how the quality and consistency of marking can be maintained or even improved. This seems an attractive human/machine collaboration, with each partner doing what it does best.

The approach actually involves two stages of Machine Learning (ML):

• the first is trained by the vendor to extract sufficient information from students’ input to be able to recognise similar sections in different submissions. This may involve domain-specific knowledge, for example to interpret hand-written mathematical or scientific equations, but is likely to be the same for every purchaser of the system;
• the second stage of training is performed by a human marker as they work on each assessment: typically by highlighting parts of the submission, marking them as positive (a relevant step in a calculation, for example) or negative, and attaching feedback. The ML can then apply its pre-learned idea of “similarity” to point out when another submission contains a similar point and suggest attaching the same mark and comment. The human can agree or disagree with the suggestion, in either case providing more information for the ML’s learning about that particular assessment.

This combination of human and machine offers advantages for both markers and students. Once the machine is making appropriate suggestions for points that appear in most submissions, the marker can quickly approve those. This lets the human focus on less common insights or misunderstandings, with more time to provide relevant feedback on those. Students should get more consistent marks and better feedback. Furthermore, most systems record the structure of feedback as well the content, so markers can review how often each piece of feedback was referenced and, for example, expand those relating to common misunderstandings. All students benefit from this enhanced feedback, not just those marked after the need for it was noticed.

In terms of AI regulation, this two-stage collaborative process has several attractions. The marker remains very much a human-in-the-loop, with both marks and feedback individually approved. The link between the human’s actions and the machine’s interpretation of them is quick and direct: well suited for what is referred to as “human oversight” and “correction”. Those are provided by humans who are experts in the domain where the AI is operating, not in AI, which insights from safety-critical systems suggest is a desirable feature.

The process should also provide clear signals (through rejected suggestions) when either stage of the ML isn’t working: this might indicate either that the marker isn’t highlighting enough of the submission for the ML to be able to recognise common features, or that the ML’s original training isn’t extracting sufficient meaning from the submissions. The draft EU AI Act concentrates on information flow from providers of AI systems to their users, but here there seems to be value in the provider inviting reports in the other direction: that “your system isn’t performing well in these circumstances” and either supporting the users with better instructions or improving the performance of the first stage ML.

The recent increased awareness of federated social networks has produced some discussion about their status under new “platform regulation” laws, such as the UK Online Safety Bill . Most of this has focussed on whether federated instances might be covered by legislation and, if so, what their operators’ responsibilities are.

But this post uses them as a way to look at content regulation in general. In particular, are these laws about controlling what we post, or what we read? On a centralised platform such as Facebook or Twitter, there’s no difference: the platform operator controls both what its users post and what they can see. But in a federated system, each instance has its own community of people who can both post and read, but members of that community can also choose to read content posted on other instances by people who have no relationship with the local instance or its policies. What light does that difference shed on how we think about regulation?

Posting is fundamental to the definitions in the Online Safety Bill: a service that doesn’t allow posting (clause 2 says “generated”, “uploaded”, “shared”) isn’t a user-to-user service, so immediately falls out of scope. Services that allow interaction, but limit this posting to “expressing a view” (via likes, votes, etc.) on provider content are also exempt (see Schedule 1 clause 4). Posting is also at the heart of the model of different federated instances having different policies : these may be pre-defined by an instance operator and those who find them welcoming can join, or an existing community may decide on its preferred rules and create an instance to implement them. Perhaps the strongest community link is an instance for employees , where contracts may already contain a policy on acceptable posting. This is very different to a centralised social network where a single policy covers all posters and readers, no matter how (un)comfortable they may be with it.

Reading isn’t as deeply embedded in the Bill, though groups of readers are likely to be a consideration in the required risk assessments. Two features of current federated systems support group-appropriate reading. As above, federated instances are expected to set up and enforce different rules for what is posted locally, and members of an instance can choose what (if any) content they see from outside that instance. Such choices are more effective in the currently normal situation where federated instances don’t use algorithms to select or promote extra content to individual users. An individual reader can start from their local timeline (which should follow the instance’s policies) and use controls to narrow or widen their personal policies by blocking, following, searching or accessing a broader timeline. Instance operators can block whole external instances, typically because of incompatible policy or practice, but readers who want to read content from a blocked instance can still do so, either by joining it, or by reading its public feed. Both of these are outside the control or even visibility of the blocking instance operator.

Federated social networks offer an alternative way to think about platform regulation. It will be interesting to see whether Parliament or OFCOM incorporate this additional perspective as they develop and implement the UK legislation.

European Data Protection Regulators have been expressing their concerns for nearly twenty years about public records of domain name ownership (commonly referred to as WHOIS data ). A recent case (C37-20) on public records of company ownership (required under money-laundering legislation) suggests that the European Court of Justice would have similar doubts. But its comments on how access to such records might be made lawful could provide a useful framework for Incident Response Teams or registries wishing to obtain or provide access to WHOIS data as well.

Interestingly the Court contrasts a situation where records of ownership are public with an earlier one where such records were available to “any person or organization capable of demonstrating a legitimate interest”. The latter is the rule currently applied by many Domain Name Registries. So how might such a legitimate interest be established? First, the Court dismisses (72) the claim that the difficulty of providing a detailed definition is a justification for dropping the requirement. Instead, those wishing access need to demonstrate:

1. That their need for access relates to an “objective of general interest” (46);
2. That the interference with individuals’ privacy and other rights is necessary for that objective (in the usual EU sense of “could not be achieved by other means less prejudicial”) (66);
3. That that interference is proportionate, in particular “capable of being offset by any benefits which might result” (85);
4. And that there are “sufficient safeguards enabling data subjects to protect their personal data effectively against risks of abuse” (86). For example company ownership data “enables a potentially unlimited number of persons to find out about the material and financial situation of a beneficial owner” (42) and “the potential consequences for the data subjects resulting from possible abuse of their personal data are exacerbated by the fact that, once those data have been made available to the general public, they can not only be freely consulted, but also retained and disseminated” (43).

Incident response teams that analyse WHOIS data to detect and even prevent security incidents shouldn’t find it too hard to meet these requirements. Doing so, using the structure from the case, should reassure regulators and registries, as well as system and network users. Reducing incidents and their impact is identified as a general interest in both data protection and network security laws. CSIRTs have been applying “ necessary and proportionate ” tests to their activities for many years: the benefit to individuals of their data, systems and networks being secure helps to support this case. The purpose of incident response itself requires strong safeguards against information being misused or inappropriately disseminated (it would help attackers greatly if they could find out how much of their activities had been detected); though the case also highlights the need for safeguards on registers to ensure that only authorised individuals can access data.

Discussions of student wellbeing tend to focus on providing individual support for those who are struggling to cope. That’s great, but likely to demand a lot of skilled staff time. A few years ago Bangor University investigated whether the university might be contributing to stress through excessive or spiky workloads. Addressing causes of stress would, of course, benefit many students at once. And quite possibly staff, too…

The Bangor researchers considered a department that had a single catalogue of modules and assignments. From that, timelines of student workload could be extracted in a consistent fashion. When I heard the work presented, they were planning to model how different student behaviours would affect the timing and intensity of workloads: the student who works steadily as soon as an assignment is set may have a different experience to the one who leaves everything to the last minute. Tutors could then be helped to adjust their assignments or schedules to avoid creating excessive peaks across the cohort.

Expanding and reproducing that idea at institution scale would require a central source of information about modules and assignments. Depending on institutional practice and technology, that might be available from a VLE or the Jisc Learning Analytics Service. Refinements could include distinguishing formative and summative assignments and types, and adding exams, but even partial data can generate ‘heat maps’ of assessments and dates across courses or faculties that suggest where pinch points may exist.

If that “demand-side” information isn’t available, then perhaps there are “supply-side” proxies that could be used instead? A colleague pointed out that the act of submitting assignments also produces records, and that that might be a more consistent source of cross-institution data. Logs from submission or checking systems should at least show how many assessments were completed each week, revealing peaks and troughs. Additional details such as number of submission attempts and proximity to submission date might reveal common strategies that may need support.

Doing this at assessment, module or programme level shouldn’t require any personal data: just counts. Determining whether particular combinations result in high workloads probably does require linking submissions by the same individual (“students doing Intellectual Property and Data Protection made three submissions that week”), but should be possible using strong pseudonyms that don’t identify who the students are. The same is, I think, true of the approach using “demand-side” data: either can be done in a privacy-protecting way. The aim here isn’t to identify “steady workers” and “last minutes”, but to adjust our demands so as to make life tolerable for both.

I’ve read two documents this week – one academic paper and one guide from the Information Commissioner – pointing out that just because someone chooses to participate in an activity doesn’t mean that Consent is the appropriate legal basis for processing their personal data. There might be several reasons for that…

First, if the nature of the activity for which the individual volunteers requires a longer-term commitment then frequent flips in and out – which must be provided if consent is used – may well make their participation a waste of everyone’s time. For example a research study is likely to benefit from a continuous sequence of information, not one with arbitrary gaps. Participants in a Covid-tracking programme who absorb resources by joining but then withdraw their consent to processing rather than report a positive test are likely to degrade the quality of everyone else’s statistics. Joining an activity of this kind may be a free choice but that choice ought to involve more consideration and commitment than the Consent framework, with its stress on ease of changing your mind, can really support.

Second, Consent relies on individuals making good choices. If they don’t do so, either because they have insufficient information, or because the consequences of the processing are too hard to predict, then confidence in the whole system may be put at risk. It’s often better for the data controller to take responsibility and ensure that the processing is, and continues to be, safe, at least for participants within a clearly defined range of characteristics. “Necessary” legal bases, such as public task and (particularly) legitimate interests, provide more support and guidance, and may be more appropriate for this . Both still allow a volunteer participant to change their minds if unforeseen risks emerge: formally the data controller can refuse a “ right to object ” if the individual’s risks are not significantly different to what was anticipated, but the law also allows them to be generous in granting opt-outs where that is appropriate.

It seems to me there might even be an argument that volunteers are psychologically more at risk of making inappropriate consent choices. If an activity is something I am passionate about, then it might be particularly important that someone else continues to keep an eye on my safety while I am doing it. In GDPR terms, the freedom whether to participate then becomes an additional safeguard for processing that already has a sound legal basis.

When GDPR was first proposed, one of the stated aims was to address “the overuse of consent”. That hasn’t always worked out, but it’s good to have situations highlighted where an alternative does indeed provide a better approach. For both data subjects and data controllers.

One of the major causes of disruption on the Internet is Distributed Denial of Service (DDoS) attacks. Unlike “hacking”, these don’t require there to be any security weakness in the target system: they simply aim to overload it with more traffic than it (or its network connection) can handle. Often such attacks are launched from multiple sources at once (hence “distributed”), with many or all of the sources being innocent machines that are being controlled, but not owned, by the attacker.

From the defender’s point of view that creates a new challenge as, in principle at least, the attack packets can be identical to legitimate ones. We could simply block all packets in an over-large flow, but that does the attacker’s work for them. Fortunately there are often patterns that can be used to (mostly) distinguish the malicious packets from the genuine ones. These are commonly identified, and sometimes implemented, using automated systems.

Applying my generic model of a security automat , here are some thoughts…

Levers : DDoS protection systems typically consist of two layers. The first selects a portion of the packets on the network, based on header characteristics (source address, port/service, etc.) and re-routes these to an inspection system. Here the re-routed packets are examined more closely: those that appear harmless are routed on to their original destination, the rest are judged to be part of the attack and typically dropped. Thus the outcomes may be:

• Packet follows normal route to destination
• Packet is (slightly) delayed through re-routing, but identified as harmless and passed to destination following inspection
• Packet is dropped after failing both stages of check.

It is worth noting that without DDoS protection an attack would likely cause significant packet loss for the target site, and possibly the network, in any case. So a DDoS protection system doesn’t have to perfectly classify every packet: a sizeable reduction in bad traffic is what we are looking for. If a few good packets get blocked, they may well be re-transmitted by their (legitimate) origins anyway: if a few bad ones get through, the target system should be able to deal with those without overload.

Data : The first stage check is likely to use only packet header data; its re-routing algorithm may also take account of current context, e.g. any recent unusual flows to/from the same destination. Second stage check may inspect any accessible portion of the packet, including unencrypted content. It seems unlikely that either decryption or flow re-assembly will be worth the required processing cycles in a situation where the aim is to “make things less bad”, rather than “achieve perfection”.

Malice : A malicious actor may try to persuade the automat that a DDoS attack is in progress, to cause it to block either a particular source or destination, or a particular application. These outcomes could, of course, be generated by actually creating a DDoS attack, but deceiving the automat into mis-applying its levers is likely to be cheaper than renting a “stresser” service, and may be harder to trace to its origin.

Controls : The human operator of the DDoS service may wish to intervene at two different levels:

• Correct the automat’s mis-identification of a flow as being (part of) a DDoS attack. Particularly on research networks, large unexpected flows may be the successful outcome of new systems or experiments (for example novel file-synchronisation protocols have triggered many alarms in the past). In this case the operator is likely to want to withdraw the automat’s proposed intervention entirely. Or
• Refine the automat’s classification of a DDoS flow, either by amending the rules for identifying packets for re-routing or the rules for analysing and disposing of them.

Depending on context, each of these options may be required both before a new rule is introduced (human approval of proposed blocking) and/or afterwards (human review). Some operators may also wish to pro-actively list some flows as exempt (typically identified by source/destination/port) from redirection or blocking if their nature (e.g. DNS responses) means that any interruption by the “protection” system would effectively deny service to the receiving site anyway.

Signals : The operator is likely to want to know when a new rule(set) is proposed or introduced, either to approve it before implementation, or to review it afterwards. Statistical information may also be required about the current status and activity of the protection system (e.g. how many/which destinations are under attack, what proportion of traffic is being forwarded after cleaning, etc.). Since attacks and campaigns are typically short-lived, the operator may want to know which rules are no longer matching traffic, so they can be disabled to save space, network and processing capacity.

Historic information might also be useful to assess effectiveness: how much has the addition of the rule changed the traffic being delivered compared to what it looked like before the attack? Perfection would be “no change at all”, if there is a significant difference this might be an indication that the rules need reviewing, or that the attack is having an impact on systems or networks elsewhere.

The Proposal for a Regulation on Cybersecurity Requirements, recently published by the European Commission, significantly raises the profile of software vulnerabilities and processes for dealing with them after a product is delivered. The Regulation on Digital Resilience in the Financial Sector (DORA) , proposed in 2020 and likely to become law shortly, does require organisations to “have appropriate and comprehensive policies for patches and updates” (Art.8(4)(f)). But that’s limited to the financial sector.

The new proposal covers all manufacturers/vendors of software that is capable of being “connected” to anything else “via hardware interfaces … network sockets, pipes, files, application programming interfaces or any other types of software interface” (Recital 7), with only a very limited exemption if software is not “supplied in the course of a commercial activity” (Recital 23). Criticality of software or application doesn’t change the requirements (it does make a difference to how compliance needs to be demonstrated – see Annex III), because “even hardware and software considered as less critical can facilitate the initial compromise of a device or network, enabling malicious actors to gain privileged access to a system or move laterally across systems” (Recital 7).

The requirement that all such products be “designed, developed and produced in such a way that they ensure an appropriate level of cybersecurity” (Annex I 1(1)) is an obvious read-across from existing laws on physical product quality and safety. Annex I 1(3) provides an extensive list of cybersecurity issues that should be considered in design: secure by default, access controls, data confidentiality/integrity, logs, etc. And “Products with digital elements shall be delivered without any known exploitable vulnerabilities” (Annex I 1(2)).

What’s new is the recognition that a digital product – and its manufacturer’s responsibility – isn’t finished when it is delivered to the customer. The last bullet of Annex I 1 introduces an obligation to: “ensure that vulnerabilities can be addressed through security updates, including, where applicable, through automatic updates and the notification of available updates to users”. And there follows a lot more detail in Annex I 2 (my summaries):

1. Provide a Software Bill of Materials (SBoM) listing third-party components
2. Address and remediate vulnerabilities without delay, including by providing security updates
3. Regular tests and reviews of product security
4. Public disclosure of information about fixed vulnerabilities, including impact, severity and how to remediate
5. Coordinated Vulnerability Disclosure (CVD) policy
6. Contact details for reporting vulnerabilities
7. Mechanisms for secure distribution of updates
8. Dissemination “free and without charge” of patches and information about mitigation

That’s a pretty state-of-the-art vulnerability handling process! And, according to Article 10(6), it must be maintained for at least five years or, if shorter, the expected product lifetime. Some software producers may already meet the requirements, but many will need to improve.

Once the law is passed – which could take a while – manufacturers will have two years to comply, so there won’t a sudden improvement in the availability of patches. But it’s a striking statement of what European legislators feel is needed to secure the digital society.

A few weeks ago I was invited to contribute to Team Cymru’s Future of Cyber Risk podcast . As I hope is apparent from the resulting recording, it was a fun conversation about working with regulators and how apparently different risks often turn out to be the same after all.

The latest draft part of the ICOs guidance on data protection technologies covers Privacy Enhancing Technologies (PETs) . This is a useful return to a topic covered in a very early factsheet, informed both by technical developments and a better understanding of how technologies can (and cannot) contribute to data protection.

Perhaps the most important message is in the very first section. All the technologies can help to reduce risk – both to data subjects and data controllers – but very few will change personal data into anonymous data. Data Protection law still applies, both to the application of PETs and to their results. Thoughtfully used, PETs can contribute, in particular, to data minimisation, security and risk reduction, making existing processing safer and, sometimes, permitting processing that would otherwise involve too high a risk.

Conversely, PETs can increase risk if used inappropriately. In particular, most privacy-enhancing technologies rely on the privacy-enhancing organisational measures and processes that surround them. Weaknesses (or misplaced trust) in these organisational measures can undermine the protection provided by the technology, or even increase the privacy risk if they increase the scope or duration of access to personal data. This makes PETs hard to add retrospectively – they are best incorporated at the design stage, where tools such as Data Protection by Design and Data Protection Impact Assessments can provide the breadth of analysis required.

Unlike the earlier guidance, the discussion of specific PETs assumes that basic security and minimisation measures have already been applied. There is no discussion of encrypted storage and transmission or pseudonymisation for example: these should now be routine considerations for all data controllers. The division of PETs into three classes provides a useful framework:

• Deriving or generating data which reduces or removes identifiability (the latter the only group whose product might fall outside the definition of personal data);
• Hiding or shielding data;
• Splitting or controlling access to certain parts of data.

Familiar technologies (e.g. statistics, encryption and key-coding, respectively) can contribute to all of these, and should be used where possible.

The remainder of the guidance considers individual technologies in each of these categories. Most are still active topics of computer science research, so likely to be suitable only for exploration by technologically advanced data controllers. Oddly the sequence in which they are presented – Homomorphic Encryption, Secure Multi-Party Computation, Private Set Intersection, Federated Learning, Trusted Execution Environments, Zero-Knowledge Proofs, Differential Privacy and Synthetic Data – doesn’t seem to match either the document’s own categories or my impression of how close to production use they are. Synthetic data and differential privacy are the ones I’d expect to be considering first.

The document is a draft for consultation: feedback to the ICO is welcome .

Display Names are often how we are represented online. Michael might choose to appear as “MusicFan”, “Mikey”, “Florence” or “Andrew”. Does that establish a good tone for discussion? Or does it risk misleading readers, perhaps making them act on the basis of a mistaken identity? Platforms that use display names can and, I think, should consider what intuitions their users may have, and choose policies and practices that help to establish a shared understanding.

If the nature of a platform is such that mistaken identity is a serious risk, then maybe the platform shouldn’t allow display names at all? Or only display them to people (such as platform operators) who ought to remember their (lack of) significance? There are many complaints, however, that the random user identities everyone else sees are unfriendly, hard to recognise and use.

Or a platform might decide only to accept users whose display names can be vouched for by a trusted third party. This will exclude all those who can’t provide a verified name and may discourage those who could. Depending on the purpose of the platform and its intended community, exclusion may be more or less of a problem: for some platforms, it could entirely defeat their purpose. Note, incidentally, that even “official names” aren’t unique guarantees of identity: there are several “Andrew Cormack” in the blogosphere.

Or a platform might try to guide users to an appropriate balance of those risks. The platform can control how display names are presented: I’ve put mine in quotes above, for example, an even less subtle approach would be to append “(not verified)”; suggested pseudonyms might encourage informality, readers might be less sure of “Andrew Cormack (Jisc)” when surrounded by the likes of “Prickly Hedgehog”; user interface experts will have many and much better suggestions.

The key point, I think, is that the appropriate answer will differ between platforms, so is something that each platform should consider. As a platform operator, what expectations do you need your users to have about display names, and what can you do to encourage that to happen?

By the way, I can be found on Twitter as …

Andrew Cormack (five vax, mask, one infection, one OK) @Janet_LegReg

… or so I self-assert. What do you believe? And does it matter?

Sophos have recently released a tool that uses Machine Learning to propose simple rules that can be used to identify malware. The output from YaraML has many potential uses, but here I’m considering it as an example of how automation might help end devices identify hostile files in storage ( a use-case described by Sophos ) and also in emails. As usual, I’m structuring my thoughts using my generic model of a security automat (Levers, Data, Malice, Controls, Signals), and hoping the results are applicable to a general class of automation applications, not just the one that happened to catch my eye…

Levers . In Sophos’ system, the Machine Learning component doesn’t have any levers: it just creates a list of rules. The levers belong to whatever software that ruleset is fed into. If that’s a scanner that examines files in storage then presumably it will move any file that matches into a quarantine directory: fine if the match is correct, but potentially damaging or making the device unusable if there’s a false match when examining, for example, a critical operating system or software component. Typical actions when scanning emails – such as marking or filing a message – are easier to remedy when they are mistakenly applied to legitimate content. The most extreme response might be to block a particular organisation, website or IP address that is the source of content considered malicious. A false positive here will be inconvenient, though usually remediable, though I have come across examples of automats blocking critical services such as DNS resolvers…

Data . The machine learning component of the system takes as input two directories: one containing files considered malicious and one containing a similar number considered good. Based on these, Machine Learning identifies text fragments (“substring features” according to the documentation) that seem to be more common in good or bad files; YaraML’s output is a list of these fragments with weights indicating how strong an indication of good/badness they represent. Even a low-power end device should be able to search a new file for these fragments, calculate the weighted sum, and check whether it exceeds threshold. The quality of the rules clearly depends heavily on the quality of the input datasets; finding the necessary quantity of correctly classified samples might be a challenge, as the article suggests that 10,000 of each would be ideal. Statistical models can always misclassify: smaller training data sets might increase this probability, making Signals and Controls particularly important to detect and remedy when that happens.

Malice . The obvious way for a malicious person to affect the process is by way of the training data sets. If I can insert enough examples of my malware into the “good” collection (or even swap a significant number from your “bad” to “good”) then the resulting rules might provide false reassurance to the end devices. This stresses the need for secure and reliable sourcing of training datasets, and for security during the training and deployment processes. An interesting aspect of making the software available as open source is that different organisations might use different training sets to generate rules for the same malware. At least this limits the scope of any interference: subsequent (careful, to avoid cross-pollution!) comparison of rulesets might also help to detect this kind of interference.

Controls . Thinking about how an organisation might respond if it discovered it had deployed a rogue ruleset – either through malice or accident – the obvious control is to be able to un-deploy it from all end devices. This depends on the facilities provided by the application: anti-virus software is typically designed to add new rulesets quickly in response to new threats, but I’d want to check it could also remove those that turned out to be significantly harmful, or at least change the levers available. Thinking specifically about the risk of quarantining an operating system component, it occurred to me that it would be good to have a list of files that should be treated with extra care. It turns out that someone involved in Yara development had an even better idea: “ YARA-CI helps you to detect poorly designed rules by scanning a corpus of more than 1 million files extracted from the National Reference Software Library , a collection of well-known, traceable files maintained by the U.S. Department of Homeland Security”. So destructive false positives against those files should be detected even before the rules are deployed. Nice!

Signals . An obvious desirable signal once a ruleset has been deployed is how many times it has been triggered. That’s relevant for both true positives (“how much badness do we have?”) and false positives (“how accurate is my rule?”). That needs some sort of feedback mechanism from the end devices to the deployers; for applications like email scanning it would also be useful to learn how often users disagree with the rule’s classification, for example by moving a quarantined message back out of the spam/malware folder. For file quarantining, an equivalent signal might come from helpdesk reports of “my machine stopped working”. But the Yara-CI idea suggests that it’s not just raw numbers that matter. A positive match in a folder belonging to the operating system is more significant than one in a user folder, whether it’s true or false. If true, then malicious code has managed to install itself into a particularly dangerous location: if false then there’s an increased risk that the mistaken quarantining action might have harmful consequences.

I’m hoping my generic model of a security automat (Levers, Data, Malice, Controls, Signals) will help me think about how tools can contribute to network security and operations. It produces the ideas I’d expect when applied to areas that I already know about, but the acid test is what happens when I use it to think about new applications. So here are some thoughts prompted by an automat that helps identify and debug wifi (and other) network problems: Juniper’s Mist AI . This was chosen purely because a colleague mentioned their Networkshop presentation: I wasn’t there, so the following is based on the published documentation.

Levers . Mist AI seems to produce two distinct kinds of output (noticing that got me thinking about different modes of automation ). In a few cases – upgrading firmware the main one highlighted in the product videos – it offers to actually perform the task for the human operator. But more often it suggests to the human what the root cause of a problem might be – “this DHCP server seems to be linked to a lot of other failures” – and relies on human expertise to diagnose and fix the problem. So humans make, or at least approve, all changes. There are interesting comments in one of the videos that more automation would be possible: “it’s not a technical issue, more one of building trust” and “we want no false positives”. The latter seems a very high bar, since most automats will be based on statistics and correlations, but the trust point is really important. There’s a challenging balance in any new source of “authority” (see the introduction of printing!) to make humans trust it enough that it does save time/improve quality, but not so much that they follow it blindly even when its recommendations – for example in a situation it wasn’t designed for – don’t make sense.

Data . Mist AI takes data from a huge range of sources: pretty much any compatible network equipment. The one obvious gap is the computer/laptop/phone/printer/thing that is trying to connect to and use the network. Given the challenges of installing agents on such devices, that’s probably sensible. But it probably is something that needs to be flagged to the system’s operators: when it says “most likely cause is a misconfigured device, but I don’t know”, that’s a design choice, not a system failure. Otherwise that huge range of input data enables another mode of human/machine partnership: helping humans visualise and navigate. The interface provides timelines and other visual representations to help humans spot patterns, as well as the ability to “pivot” through different sources – “show me what else is going on at that time/that device/etc.” – that has been used in many of the most impressive examples of security diagnosis I’ve seen.

Malice . The few fully-automated actions envisaged for Mist AI seem to give an attacker little opportunity, though anything with administrator access needs to be designed carefully. There’s a steady trickle of attacks that use auto-update functions to either downgrade security or install entirely new software. Perhaps more interesting is whether an automat could detect malicious activity at the network level: sending requests with odd parameters to get more than your fair share of bandwidth, or to disconnect others; or perhaps trying to confuse or overload network devices to redirect traffic or hide malicious activity in a storm of noise. This probably isn’t something you’d want to respond automatically to (that could easily provide a disruptive attacker with a useful tool) but an automat that alerts when things simply don’t make sense could be a useful function.

Controls . The executive powers of the current Mist AI system are very limited, and human operators have to approve individual actions (e.g. a firmware upgrade). One reason that works is that the system is addressing the kind of problems that users often assume will “go away”: network slow, took longer than usual to connect to wifi, etc. Indeed one stated aim of Mist is to help operators fix problems (or contact the user) before they are reported. On that time-scale, human-in-the-loop is fine, and worth it for the confidence it adds. Given the aim to fix problems before they are noticed, I wondered whether the system could check that the fix did actually reduce the problem: if not, the operator might want to automatically revert an approved change, or go back to a “last known good” state. Perhaps undoing a change is more readily automatable than doing it? And, since there is “machine learning” going on, maybe invoking those controls could be input to the next learning cycle (“OK, that didn’t work out”)?

Signals . One of the interesting displays I spotted on the video was strength-of-correlation: “100% of DHCP failures involve this server” suggests we have found the root cause, 30% and I might do a bit more digging. Confidence is a useful signal in pretty much any recommendation system, not least to highlight occasions when the machine, programmed to find something it can recommend, has searched ever more widely and ended up with a suggestion that’s little better than a random guess. Any fully-automated system should probably have a lower confidence threshold beyond which it will still seek human confirmation. And, as in Controls above, the timeline displays in Mist AI got me thinking whether we could use automated before/after comparisons to evaluate the effectiveness of changes: did the original problem reduce? Did anything else start happening? And I like the option to “list clients experiencing problems”: sometimes talking to a person will be the quickest way to work out what is going on; and to show we aren’t completely mechanical, too!

A couple of recent discussions have mentioned “trade-offs” between risks. But I wonder whether that might sometimes be a misleading phrase: concealing dangers and perhaps even hiding opportunities? “Trade-off” makes me think of a see-saw – one end down, other up – which has a couple of implications. First, the two ends are in opposition; and second, we can always change our minds, change the weights, and things will go back to where they were.

But think about a real-world example from 2014: care.data . Here a risk was identified: that medical research would be limited by shortage of data from real patients. I’ve no idea if the proposed solution was thought of as a “trade-off”. But making patient data from their family doctors available to a central research service was seen by many people as increasing the risk to their personal privacy. Individuals could ask their doctors not to transfer their data. But doctors identified another risk: that patients would say less about their symptoms if they thought they might go beyond the consulting room, so treatment would be less well informed. To mitigate that risk, some doctors stated publicly that they would not be participating in the scheme. Confidence fell resulting, according to 2022’s Goldacre Review , in “very large numbers of patients opting out of their records ever being shared outside of their GP practice (approximately three million by the end of 2021) with opt-outs now at a scale that will compromise the usefulness of the data” (p88). To put it another way, because of linked risks, the attempt to reduce the risk of insufficient research data actually made the research data risk worse. So the see-saw image was wrong on two counts: the risks to research and patient privacy weren’t actually opposed, but linked in a way that created feedback; and that feedback actually changed the environment so the seesaw couldn’t (easily) be returned to its original position.

Thinking instead of a spiral (OK, technically a helix ) explains better what happened: the linked risks took the system around a loop (data => individual => doctor => data) but when, some months later, it returned to the original position things had deteriorated and the original situation could not be recovered.

But spirals can go both ways… Can we use linked risks to make a situation better? Regulators suggest a couple. According to the Information Commissioner (specifically referring to consent, but the point is general):

• “Getting this right should be seen as essential to good customer service: it will put people at the centre of the relationship, and can help build confidence and trust. This can enhance your reputation, improve levels of engagement and encourage use of new services and products.”

But:

• “Handling personal data badly … can erode trust in your organisation and damage your reputation. Individuals won’t want to engage with you if they think they cannot trust you with their data; you do things with it that they don’t understand, want or expect; or you make it difficult for them to control how it is used or shared”

Here are clear statements of the link between the risks of insufficient data for business and risk of privacy invasion for customers, and the possible spirals. The first is a spiral of improvement: if a business uses personal data in ways that also mitigate customers’ risks then those customers may be willing to volunteer more information which can then be used for mutual benefit. Now we have gone around the helix, but arrived at a better place for everyone. Similarly, but at sector or societal level, the European Commission’s Recitals to the NIS2 Directive suggest that appropriate sharing of information to improve the security of systems and data could increase confidence and make individuals more willing to transact through digital systems, with benefits for individuals, organisations, “economy and society”.

Considering whether risks might be mutually reinforcing (in either a positive or negative direction) rather than a trade-off, might help us find positive opportunities or, at least, highlight the risk of downward spirals before they do serious damage.

Earlier in the year, Networkshop included a presentation on Juniper’s Mist AI system for managing wifi networks. I was going to look at it – as an application I don’t know – as a test for my model for thinking about network/security automation. That may still happen, but first it has taken me down an interesting diversion…

The product video shows two use cases: first, identifying when an access point needs a firmware upgrade and (once approved by a human) doing it; second, tracing from an intermittent client connectivity failure through to a possible root cause in DHCP pool management. In terms of the human/automat relationship, those seem to represent different reasons for automating. The first is, perhaps, the traditional application of automation in the physical world, where humans are doing a repetitive task that could be scripted. Not necessarily a simple task – one of the benefits of automation is that a long sequence of actions can be performed consistently – but a repeated one. The second seems to challenge human capability from the opposite direction – a one-off situation with so many possible resolutions that working out which are most likely to be right involves more data, more knowledge and, perhaps, more linkages than most humans can keep track of. Here automation helps by refining down the whole solution space to a small enough number that a human can examine each one, work out its consequences and at least plan the implementation of the one they choose.

Which takes me back to my picture, but at a higher level, of how automation works and how it might evolve .

The simple tasks spend most of their time in the left-hand (“automated”) loop, but we need an automat that can detect and alert us when something about the task has changed so the original automation “script” may no longer be appropriate. Then the human can step in, perhaps redefine the boundaries and/or update the script for the new circumstances.

The complicated tasks spend most of their time in the right-hand (“human”) loop, but an automat may be able to spot patterns – either in the technological context or in how humans respond to it – that help it make occasional suggestions. This could be anywhere between a traditional expert system approach (“if connection failures, look at DHCP”) and something more like data-driven machine learning (“which kinds of logfile entry are correlated with this kind of problem report?”). Some humans can do that unassisted – I remember an amazing presentation at a FIRST conference many years ago where a human analyst pivoted through a series of data sources, including image hashes, to link together apparently isolated malware incidents. But most of us could probably use the occasional suggestion from an automat that can look at far more datapoints and possible connections than we can. Ideally we’d involve both the human “I remember one of these…” and the machine “oooh, data…” approaches. And, by providing (cross-)feedback on the results of collaboration, maybe even improve both?

For a final squeeze on the diagram, could we use it to explore moving some (sub-)problems from one side to the other? If part of a simple problem evolves in a way that we need to human-approve those solutions, can we define which part, and help the automat send the right situations for approval? Or, coming the other way, identify some parts of complex problems that we understand sufficiently well (at least, for now) that they can transition at least from human-advice to human-approval, and maybe even to fully automated?

Throughout the time I’ve been working for Janet, the possibility of using technology to block undesirable activity on networks and computers keeps coming up. Here are four questions I use to think about whether and how technology is likely to be effective in reducing a particular kind of activity:

Where is the list?

Any technology needs a set of instructions. In the case of blocking, we need to tell it how to distinguish things that should be blocked from things that should be allowed. Typically, that’s a list of Internet locations. One day machine learning may get closer to understanding content or intention, but we’ll still need to provide it with a good/bad model.

So, can we get that list from someone else, or do we have to create and maintain it ourselves? Maintained lists of different categories of activity may be available, either free or as part of commercial services or appliances. If we have to create a new list, do we have the skills, resources and permission (in some cases including legal) to do that? How will we keep it up to date, and handle any challenges to our decisions to include or exclude particular actions or content?

Where is the technology?

Internet technologies typically give us four different ways to specify things to be blocked: network (IP) addresses, domain names (DNS), application identifiers such as URLs and email addresses, and content inspection (e.g. keywords or hash values). Each of these gives a different precision, depending on the nature of the unwanted activity, so we should choose the one that most accurately defines what it is we want to block. Errors are likely in both directions – over-blocking that prevents legitimate activity: under-blocking that allows some unwanted – but choosing the right blocking mechanism should minimise these. Modern technologies such as cloud hosting and Content Delivery Networks (CDNs) involve a lot of sharing of both domain names and IP addresses, so those rarely offer good precision. Application identifiers are usually the most precise but extracting and checking them adds delay and privacy issues. Content inspection is unreliable outside a narrow set of applications, such as detecting repeat appearances of known illegal images.

Whatever technology layer we choose for blocking, we need some equipment to implement the block, and some way to ensure that network traffic goes through that equipment. Depending on the approach chosen, existing routers (IP), resolvers (DNS) or proxies (identifiers and content) may offer relevant functions: otherwise new equipment will be needed. Note that forcing traffic through blocking equipment is likely to create a single point of failure. Blocking and resilience are very hard to reconcile.

Who are the users?

A few kinds of activity – notably, active threats to connected computers – can be blocked for every user of the network. More often institutions will want to choose which blocks to apply and to whom, so should opt-in to the blocking, rather than having it imposed. If institutions need to make local changes to make blocking effective, imposing it before they are ready will have unpredictable results, possibly undermining existing protection measures. To assess the effectiveness of blocking, or to use the blocked content in research or teaching, particular individuals or locations will need to be exempted from the block.

These issues have implications for where the blocking equipment is located, who configures it and has access to logs. Equipment should be placed where it will have access to as much of the traffic to be checked as possible and (because most technologies add delay) as little other traffic as can be arranged. Where fine-grained per-user or per-location control is needed, this must be managed by the organisation that can identify the individuals and locations that should be (temporarily) exempted: typically their institution. Note that such fine-grained control is technically complex to implement for IP and DNS blocks. Where access to logs is required – for example to provide help to those who may have tried to undertake prohibited activities – this should also be at institutional level.

How will people react?

Technical blocks can always be circumvented, so are most effective against activity that no one should want to encounter. Even if recipients welcome the block, we still need to consider how malicious actors will respond: they might simply change location so we have to update lists more frequently; but they may also move activity closer to legitimate services to make over-blocking more likely and more disruptive.

Attempting to block activity that users desire gives them an incentive to circumvent the block. They can use different connectivity (home or mobile), but there are many technical ways to evade blocks without changing network. The activity may then continue but be invisible to those operating the network. Worse, most evasion technologies circumvent all blocks, including those for unwanted activity such as viruses, ransomware and other threats to devices and individuals. As our Guide to Filtering on Janet explains, it is particularly important that technical measures against desired content are part of a wider awareness, behaviour and support process: information and warnings may help reduce deliberate circumvention.

Examples

Two examples show how the questions can help explore the use of technology against different types of unwanted activity.

Distributed Denial of Service (DDoS)

• Where is the list? DDoS attacks against Janet and its customers are usually identified and blocked using a combination of source IP addresses and packet characteristics. Live information is available from commercial sources as well as Jisc’s own threat analysts.
• Where is the technology? Although some attacks can be blocked using existing routers it is more efficient (and less disruptive to the routers’ intended function) to redirect suspect traffic to a dedicated cleaning service where malicious traffic can be identified and blocked and legitimate traffic from the same sources (which are usually compromised computers) forwarded to its intended direction.
• Who are the users? DDoS attacks can target any institution or service. Since blocks are typically temporary (the average attack duration in mid-2022 was one hour, the maximum six) and precise, they can be applied to protect all users of the Janet network.
• How will people react? Targets of blocked attacks should welcome the protection provided. Attackers can, and do, switch both the sources and targets of their attacks when blocked. Hence it is essential that blocks reflect live information from the network, as well as from external sources.

Terrorism

• Where is the list? Lists of content, including some regulated by Terrorism laws, are available through filtering services.
• Where is the technology? Terrorist content is frequently published through otherwise legitimate social media and hosting services. Unwanted content therefore needs to be defined at URL level, suitable for application proxies able to make these distinctions.
• Who are the users? UniversitiesUK has guidance on how to provide researchers with the access they need to security-sensitive material . Such access must be managed by the institution that can vet requests for access, verify the identities of authorised researchers, and provide appropriate access control facilities.
• How will people react? The Home Office Prevent Duty Guidance warns that some individuals may be drawn in by this kind of material. These may quickly adopt technologies to evade any blocks, so institutions’ Prevent strategies should aim to provide appropriate advice and support to anyone showing early signs of interest.

Victoria Baines closed the FIRST conference with a challenge to improve our image ( video ). Try searching for “cyber security” and you’ll see why: lots of ones, zeroes, padlocks, and faceless figures in hoodies. Some of the latter look a lot like the grim reaper , which makes the task seem hopeless: in fact, cyber badguys can be resisted. And you don’t need to read binary or work in a datacentre to do it.

[Image by Eduardo Vianna Eduardo from Pixabay ]

What’s especially odd is that similar images and phrases are often used for defenders, too. Mystique may make us feel good, but it doesn’t help with recruitment and retention. We need a much wider range of skills, personalities and people to defend the online world. And referring to them as superheroes doesn’t help either: everyone can, and should, contribute to their own and others’ security. Of course superheroes save the world: that’s what they do. What we need to celebrate is the ordinary people who save the world by their choices and actions: reporting odd-looking websites or double-checking when the CEO asks them to buy gift vouchers.

Victoria’s research traces this hyperbole back two thousand years. If you are trying to draw attention to a threat, overstate it, whether it is a “ Cybercrime Tsunami ” in 2003, or flying chamberpots in the streets of 1st Century Rome (though Mary Beard thinks Juvenal may not have been overstating much!). Sadly, Victoria’s recent book on the Rhetoric of Insecurity is priced for academic libraries, but her Gresham College lecture series , starting this autumn, will be free in person, online or on YouTube.

So what’s the alternative? We need a message and images that encourage everyone to do their bit. Maybe it’s worth returning to the idea of (individual) cyber-hygiene: not so much “coughs and sneezes spread diseases”, but “catch it, bin it, kill it” might have promise.

But if you are thinking “cyber-pandemic”, please don’t!

Knowledge Management (KM) isn’t a topic I remember being presented at a FIRST conference before, but Rebecca Taylor ( video ) made a good case for its relevance. Security and incident response use and produce a lot of information – a Knowledge Management approach could help us use it better. Most teams quickly recognise the benefits of having knowledge recorded, rather than just in individuals’ heads, so most will have contacts list, processes and playbooks. Many are also asked to provide statistics. But KM could also help with things like internal and external knowledge bases, from tips for effective forensic investigations to threat intelligence or customer Frequently Asked Questions.

The first step in making this information useful is to know where it is; then attach metadata, such as when it should be used, when it was last checked/updated, etc. Just establishing these single points of truth can build confidence in the information and make the team’s work more effective. But KM seems to call for a more dynamic approach – it’s “knowledge” management, not just “document” management – where those who use information participate in improving it. So the knowledge system also needs to help users and authors communicate and collaborate, both to mark “good document: still relevant” and “I had problems interpreting this bit”. Somewhere around here, we should be moving from recorded information to shared knowledge, I think.

Systems need to support this way of working – for example change control must balance ease of updating and maintaining accuracy – but we also need to promote the right culture. Staff should be encouraged to identify opportunities and problems: those who help to improve knowledge should be recognised and rewarded. One way to do this is to use a KM approach to look at known pain points or inefficiencies: for example rapid sharing Indicators of Compromise between teams working on different engagements.

KM can even help with future planning. Looking at which information is being actively used (and, conversely, sought but not found) can help us make that easier to find and/or justify effort to improve it. If those using a process are discussing changes, can we anticipate, and pre-approve, any variations of policy or mission that may be needed? There’s a link here, I think, to Vilius Benetis’ talk on CSIRT improvement . A proactive review, ideally every few months, should check: does this still work, could it work better?

While many companies offer “knowledge management” software, that’s not the only option. Rebecca’s talk included effective examples of both a customised commercial system and one (for forensic practitioners, including a Knowledge Base, processes and templates) using Microsoft Teams. Starting small/focussed is definitely the way to go. Identify an area where there’s an obvious need – whether in a particular team or subject area or for management or funders – and use a KM approach to make their work easier. When that succeeds, you’ll have champions to support your work on the next area. Above all, treat KM as a tool to help make improvements, not a thing that should be “done”.

Incident Response Teams are, as the name indicates, responsive. Often they will try to provide whatever services their constituency asks for, or seems to need. However over time that can result in a mismatch between what the team offers and what its resources, capabilities and authority can actually deliver. That leads frustration, both among disappointed customers and among team members who know they are not delivering the best they could. And, as Vilius Benetis asked at the FIRST conference “do their eyes shine with passion?”.

He was presenting ( video ) a report by ENISA that, although titled “ How to set up CSIRT and SOC ”, can also help existing teams move to a more consistent and satisfying state. Critically, this adds a feedback loop to the design/implement/operate sequence that many teams – more or less formally – adopt. An “improve” stage considers the results of “operate” and how “design” might be changed to deliver better outcomes for the team and its constituency. This might involve changes to the CSIRT’s mandate; the services it offers; its processes and workflows; skills and training; facilities; technologies, including automation; cooperation; information security management plan; or implementation requirements. Budgets and other resources may mean it’s only possible to deliver a subset of these ideas, but those selected should be developed into improvement initiatives and detailed design changes. If resources are limited, this might include reducing the range of services offered by the team, to improve the performance of those that are most important.

These feedback reviews should take place regularly, ideally annually: developing relevant metrics for CSIRT performance will ensure consistent reviews as well as guiding operational activities. The presentation identified several sources that can be used, including:

• SIM3 model : for assessing/benchmarking the current maturity of your CSIRT and the required future status
• CSIRT services framework : for discussions of key services relevant to the constituency
• (draft) CSIRT roles and competences : for discussions of what will be needed to deliver those services

The objective of this process is to improve satisfaction, both within the team and among its constituents. So communicating and celebrating improvement is an important part of that. Shiny-eyed customers may be too much to hope for, but at least we should be enthusing our team members.

Tony Kirtley’s FIRST conference talk ( video ) explored how the Kubler-Ross model of grieving can help understand the emotional effects of a ransomware attack , both to avoid negative consequences and, where possible, to use natural emotions to support positive responses:

Denial : in a ransomware attack, denial should be short-lived, as the nature of the problem will quickly be clear and undeniable. However there is a danger that individuals at this stage will take unplanned actions, such as changing passwords or rebuilding systems, that are at best a waste of time (while the bad actor still has access to the system) and at worst may destroy information needed for recovery. A related possibility is misplaced (mis)trust in systems, data or people whose reliability isn’t yet known.

Anger : depending how it is directed, anger can be either destructive – if channelled into finding someone to blame – or constructive – if used to bond and inspire those involved in recovery. “We are all in this lousy situation together, let’s combine our energy to get out of it” can be positive, but needs care, because…

Depression : individuals may naturally believe the situation is their fault, even if there was no way their actions could have changed the course of events. Leaders must provide constant reassurance, otherwise a feeling of hopelessness can easily spread through the organisation.

Bargaining : here the risk is of being too successful in the previous stages, leading individuals to over-commit to the recovery process. Ransomware incidents take a long time to repair – anything from two weeks to four months was suggested – which is too long for anyone to work in “emergency” mode. The impact of burnout is amplified because not only the individual’s effort is lost, so is their detailed knowledge and understanding of the affected system. Here external support can help by taking on the “commodity” recovery actions, allowing local staff to focus their knowledge, skills and efforts on the locally-unique aspects.

Acceptance : this is essential to plan and perform the recovery process. Leaders need to establish and enforce a tempo that will sustain the required level of work without risking burnout, plan a recovery process, and ensure it is trusted by the whole organisation. Earlier emotions may recur, in particular anger and depression, so everyone must ensure the shared, no-blame approach is maintained. Here external support can help emotionally as well as practically: people who are less directly engaged are better placed to manage their own emotions and can spread confidence – “we’ve done this before, with a successful outcome” – among those who are going through a thoroughly unpleasant experience for the first time.

Tony suggested a sixth stage, not in the original Kubler-Ross model:

Meaning : sometimes referred to as “never let a good crisis go to waste”. Once an organisation has successfully recovered from an incident, it should always review what lessons can be learned and implement measures that make a repetition less likely. This still needs care to manage emotions: a successful review will identify improvements to processes, systems and guidance; one that descends into blaming is unlikely to help the organisational situation and may even make it worse.

The theme of this year’s FIRST conference is “ Strength Together ”. Since I first attended the conference in 1999, we’ve always said the basis for working together was “trust”. However that’s a notoriously slippery word – lawyers, computer scientists and psychologists mean very different things from common language – and I wonder whether security and incident response would benefit from a different framing.

When I joined the global incident response community I tried to observe behaviour, so I could fit in without causing offence. My conclusion was that relationships were actually established by “I will spend some time on you: if that makes my life better then I will spend more time on you”. Trust may develop as part of that collaboration, but the actual basis for it is mutual benefit. The hour I take out of my primary job of protecting my customers will be more than justified if your actions save me two hours in future.

This may seem like semantics, but I think it’s more important. As Wendy Nather’s keynote explored, my next security catastrophe may well originate in an entity I’ve never heard of: whether an obscure software library, an organisation deep in my (security!) supply chain, or a data processor engaged by an apparently peripheral organisational function. In a world where global service providers can be disabled by insecure webcams , “strength together” needs to extend far beyond those we have established trust relations with. In an emergency, “are we trusted?” may be too high a bar, “are we recognised?” (by others and by the claimed constituency) may be where we need to start.

And, in tough economic times, invoking “trust” and “social responsibility” may underplay the importance of working together. It’s often said that trust is hard to gain, easy to lose. When working together is business-critical, we simply can’t afford to lose the basis for it. A panel session suggested “socially responsible” as a motivation for information sharing, but if that’s the best we can do then we shouldn’t be surprised when its budget gets cut. Again, we need to frame working together as essential, not optional.

As the European Commission’s draft NIS2 Directive recognised, effective cyber-security collaboration is now critical for individuals, organisations, the economy and society. The converse of “strength together” is “weakness apart”: unless we recognise the necessity of working with others to improve the whole digital environment then it may not be long before that environment becomes intolerable for all of us.

Wendy Nather’s keynote at the FIRST conference ( video ) considered the security poverty line, and why it should concern those above it at least as much as those below. To secure our systems and data requires resources (tools and people); expertise to apply those effectively; and capability, including sufficient influence to overcome blocking situations or logistics.

But most current guidance, tools and practice are designed for those above the poverty line, not below. That’s a problem, because insecurity now affects everyone in the digital environment. Pollution is a better metaphor than escaping hungry bears: “there’s more than enough bear for everyone”. Even organisations whose own security is excellent can be hit by breaches in software or services they didn’t know they were dependent on, or devices with which they have no relationship at all. In a digital world where global retailers can be taken offline by insecure webcams , helping improve others’ security may be as important as improving your own.

To do that we need to move beyond talking about “awareness” and do what we can to increase “capability”. Small organisations, or those in sectors with low profit margins, can’t afford state-of-the-art security software or people. Dashboards that give security experts visibility of everything that is going on may be less useful to a part-time system administrator who just needs to identify and fix a problem. Open-source software is great, but it’s not free when you include the costs of the skilled people to install, configure and run it. A survey of security experts asked “what is the minimum set of tools?” came up with lists from four to thirty-one. The baseline looked a lot like PCI-DSS, but even that may be beyond the capability of a small business using off-the-shelf security tools.

Legacy systems are a major risk factor: organisations that proactively refresh their technology experience much better security outcomes. It may even have wider benefits: recruitment is likely to be easier for organisations that offer a modern infrastructure experience. So what can we do to help others move at least non-core business systems (for example email and payroll) to cloud-based services where many of the security issues are looked after by the provider? When we work with our own providers, can we encourage them to make essential security functions, such as multi-factor authentication, part of the basic product rather than an add-on? Instead of bare lists of tools, could industry sectors develop their own reference architectures, fitting business and cultural constraints, to help those with less capability implement systems that are easier to operate securely, improve interoperability, and reduce vendor lock-in? And can they work together to discover services that represent a common dependency, and to help them reduce the shared risk?

The pollution metaphor suggests a shared reputational risk as well as a security one. If individuals lose confidence in digital systems and services then we all suffer, not just those directly causing the problem. Over the past decade, Governments have started to help with “ordinary” internet security threats not just advanced, state-level, ones. If you are fortunate enough to be above the security poverty line then consider how you can contribute: help others reduce incidents, respond to those that happen, and learn from them, to improve security and confidence for all of us.

My first reaction to Mehmet Surmeli’s FIRST Conference presentation on Incident Response in the Cloud ( video ) was “here we go again”. So much seemed awfully familiar from my early days of on-premises incident investigations more than twenty years ago: incomplete logs, tools not designed for security, opaque corners of the target infrastructure, even the dreaded “didn’t we tell you that…?” call from the victim organisation.

But the response and lessons learned were different, and more positive. Maybe the next cloud incident can be different…

It turns out that, although they are often turned off by default, cloud platforms do have logging facilities, and it often requires just a couple of clicks to enable them. Bear in mind, however, that logs kept within the cloud container may be lost when the load scales up or down. Instead it’s better to use the cloud service to build your own (virtual) logging infrastructure, gathering logs from transient virtual machines into a persistent central storage location where you can use cloud facilities to process and explore them. Twenty years ago we knew we ought to have separate infrastructure for gathering, storing and processing logs: cloud systems might actually make that feasible for most organisations to implement.

Keeping incident response within the cloud fits the technical and economic models, too: avoiding limits or costs on exporting large volumes of data and, instead using cloud facilities for their intended purpose of analysing large datasets. As with local incident response, things will be much easier if you prepare tools in advance and use separate accounts and access controls to move data to secure places where intruders can’t follow. As with compromised physical machines, don’t investigate on a system that the badguy can access. Once you’ve established an incident response toolkit on each (major) platform your organisation uses, you can quickly bring new activities within its scope and add new tools as you find them useful. Once you have a working incident response infrastructure and toolkit, consider how you might use cloud tools for real-time monitoring: it should be possible to investigate what intruders are doing as they do it.

Some key principles:

• Get logs out of their default locations: cloud dashboards and tools are not designed for incident response;
• Default logging is not enough: use the cloud to build the logging infrastructure you need;
• Tag and map your assets: don’t make the incident response team reverse engineer what your cloud deployment is supposed to look like;
• Establish incident responder accounts, with sufficient privileges to monitor production systems, but no more.

Wout Debaenst’s FIRST talk ( video ) described the preparatory steps an adversary must take before conducting a targeted phishing campaign, and the opportunities each of these presents for defenders to detect and prevent the attack before it happens. The talk was supposed to be accompanied by live demos, but these were sufficiently realistic that the hosting provider blocked them the night before the presentation!

1. Create the domain name(s) to be used to create (misplaced) confidence in the phishing emails. I was familiar with the term ‘typosquatting’ for domains that transpose two letters, or use look-alike characters, but ‘combosquatting’ – adding something plausible like “shop” to a genuine domain name – and ‘doppelganger’ – where the malicious domain is created by removing punctuation from the real one – were new, useful, descriptions. Tools exist, including the free DNStwister , to check for domains that may be suspiciously close to your own. Another technique is to register something that looks like a genuine platform service, then add the target company as a subdomain: here certificate transparency reports can be a useful source to check for your company name appearing in unexpected places, but wildcard certificates mean that individual subdomains will not be reported.
2. Prepare the infrastructure that will be used to send emails, host webpages and so on. This involves considerable effort, so phishers will often reuse the same infrastructure across multiple attacks. This creates an opportunity for detection: if your company name appears in association with an IP address, WHOIS information or even website images and templates that have previously been reported in relation to phishing attacks, then it’s likely that bad things are being prepared for you. Tools such as Virustotal and Brandefense map these known associations. Intruders can evade these by setting up new infrastructure for each attack, or mingle their activities with genuine traffic by using a CDN, but this increases the cost of the attack, hopefully beyond its likely benefit.
3. Send phishing emails. This provides an additional source of information whose reputation can be checked by the receiving organisation. Email content is often checked, but there are opportunities also to check the originating domain, IP address, mailserver and other information. These defender checks complement those at the previous stage, because freshly-created domains may themselves look suspicious.
4. Execute malicious code. Although phishing can be conducted using only plaintext, executable components are more common and can often be blocked by disabling unnecessary features, such as macros or filetypes, in endpoint clients and devices.

None of these techniques can prevent phishing by a sufficiently determined attacker, but they increase the cost of a successful attack, both in terms of required preparation and risk of discovery. For many organisations, that should put off sufficient threat actors to significantly reduce the risk.

My post about automating incident response prompted a fascinating chat with a long-standing friend-colleague who knows far more about Incident Response technology than I ever did. With many thanks to Aaron Kaplan (AK), here’s a summary of our discussion…

Developments in automated defence

AK: Using Machine Learning (“AI”) in cyber-defence will be a gradual journey. So, in practice in the next years, we won’t even notice that ML is there. I don’t see it as a turnkey/switch on magic. It will appear first for a few well defined sub-fields such as fighting spam – already here for many years (spam assassin, Bayes statistics, etc), never 100% eliminated the spam problem but we can’t live without ML for spam classification anymore – or weeding out False Positive alerts in a Security Operations Centre….

AC: Since you mentioned “false positives” I have to mention my favourite talk from FIRST 2019: Desiree Sacher on learning (mostly about your own organisation’s processes and systems) from those…

AK: Or – this is already being worked on, for example by CIRCL – we will see very simple classifiers which try to answer simple questions such as “what kind of IP address is this?” (residential CPE, server, datacentre, …). Or questions such as “What’s the probability that this is a genuine webpage on that URL? Or is it rather a simple web server’s default page (in all its variations up till landing pages of domain grabbers)?”. IMHO these tasks are well suited to ML. And all that it will give us for Incident Response is that we get another “opinion” from some system. So the “opinion”/probability for decision support is probably going to come first.

AC: So this is providing incident responders with better information for their decision-making. Indeed often automating routine searches and giving statistical evidence for gut-feel decisions that, if they had time, they’d hope to do manually?

AK: It also is a path towards fully automatic decisions based on (possibly biased) simpler ML models (such as the ones mentioned above). But it will be a gradual, very long journey. First, we will discover just how hard it is to get a good ML based system running and how hard it is to eliminate bias. And since it’s a long journey, we won’t even notice we are on this journey. We plan and learn as we go. One example I am looking into right now: do GPT-3 based systems sufficiently summarise APT threat intelligence reports? It’s *really* fun to play around with…

Developments in automated attack

AC: You mentioned that the badguy might be able to interfere with my robot in even simpler ways?

AK: Yes. Here is a pretty good paper on how a 1-pixel change tricks the classifier into believing that a cat is a bear . Paper by no other one than Adi Shamir: a crypto legend looking into Deep Learning 🙂

AC: I guess they are both about the impact of tiny non-randomnesses…

AK: Another thought which came to mind after seeing the good guy – bad guy infinity-eight picture : this is basically reinforcement learning from the attacker’s point of view. And if the defender also deployed ML on his side of the infinity-eight game, we would end up with essentially something like Generative Adversarial Networks .

ANC: Hmmm. When you use a GAN in the lab, there’s a referee to make sure the good guy wins…

Conclusions

ANC: I think you’ve reassured me that both defenders and attackers are on a long journey to use this stuff. And, actually, maybe it doesn’t hugely tilt the balance? On the defender side, we can become more efficient by using ML decision-support tools to free up analysts’ and incident responders’ time to do the sort of things that humans will always be best at. Meanwhile explore what aspects of active defences can be automated.

Attackers will get new tools, too, but for most sites those are going to be mass attacks and, I think, pretty noisy? One of the few things I’ve always taken reassurance from is that a mass attack ought to be detectable simply because it is mass. It might take us a while to work out what it is, but so long as we share information, that doesn’t seem impossible. That’s how spam detection continues to work, and I’d settle for that level of prevention for other types of attack!

Some organisations will, by their nature, be specific targets of particularly well-funded attackers who may use ML for precision. Those organisations need equivalent skills in their defenders. But for most of us our defences need to be good – say, a bit better than good practice – but probably not elite. Thanks.

Threat hunting is perhaps the least mechanical of security activities: according to Joe Slowik’s FIRST presentation ( video ) the whole point is to find things that made it past our automated defences. But that doesn’t mean it should rely entirely on human intuition. Our hunting will be much more effective if we think first about which threats it will be most beneficial to find and how we are most likely to find them.

Thoughtful threat hunting requires an understanding of likely adversaries; telemetry and data sources; and the ability to search and query them. Rather than randomly searching for signs of intrusion, threat hunting provides most benefit if it concentrates on the kinds of threat that would cause most harm to the particular organisation. Thinking about how those actors are likely to operate, and what their goals might be, should guide us to the services and systems they are most likely to use. Then we can consider what traces they might leave, and what records we might need to find them. If those don’t exist, then we can fill the gaps either by increasing activity logging in specific areas (but not so far that we overload ourselves) or by considering alternative sources that already exist.

For example a frequent blind spot, mentioned in a number of different talks, is network activity within the organization. Perimeter systems such as firewalls should give good visibility of ingress and egress traffic, but multi-stage threats such as ransomware are more easily detected by their unusual lateral movement between organisational systems. But for organisations that identify email fraud as a significant risk, email headers are more likely to be a relevant source.

Even with a focus on specific threats and data sources, threat hunters are likely to have a “needle in the haystack” challenge: data sources are too big for humans alone to analyse. So we need tools to explore individual data sources and, particularly, patterns (or their suspicious absence) across sources. Flexible, exploratory tools are likely to be harder to use effectively than single-purpose searches, so threat hunters need more time to plan and develop their skills. Again, focusing on particular threats can guide this learning to where it will most benefit the organisation.

Finally, when a threat is discovered we should “codify the success”. Having discovered the signs of an successful intrusion, try to update the rules that it bypassed to make the same technique less likely to succeed in future. Repeated hunting for the same threat is frustrating for the hunter and a waste of precious resource for the organisation.

Following my Networkshop talk on logfiles, I was asked at what point logfiles can be treated as “anonymous” under data protection law. Since the GDPR covers all kinds of re-identification, as well as data that can “single out” an individual even without knowing their name, that’s a good CompSci/law question: the work of Paul Ohm and others suggests it may take a very long time. But when designing processes I wonder if we should approach from a different angle?

When we were looking at GDPR for Jisc’s (then) 130+ services, we concluded that much the best place to start was identifying the purpose of the processing and the lawful basis applicable to that . Once we understood those, most of the requirements – including on transparency, safeguards and user rights – could “simply” be looked up in the law.

I’m now wondering whether we can get a similarly helpful collection of guidance by treating “processing that can use anonymous data” as a sort-of seventh lawful basis? Like the guidance we derived from the other six , that should deliver:

• Clear definition (and limitation) of purpose , without which we’re unlikely to get the right kind and level of anonymisation. We might even conclude that the purpose is better achieved using personal data within the GDPR;
• Transparency both about how we can produce the anonymised data we need and – since the anonymisation step involves processing of personal data – how we explain that to data subjects;
• Safeguards , in particular how we ensure that our data are, and remain, anonymised. These will depend on whether we are actually destroying the input personal data (for example aggregating log records to get usage statistics over time), or just removing sufficient information to make re-identification or singling out unlikely. As the UK Anonymisation Network explain, at least the latter approach requires an ongoing risk-management process not dissimilar to the legitimate interests balancing test when using personal data;
• Individual rights , at least those applicable to whatever lawful basis we are using for the anonymisation process, and perhaps also something similar to a “right to object” if the data subject can point out a particular reason why that process doesn’t effectively anonymise them. We should at least learn from the GDPR right to object and use any data subject concerns as prompts to review what we are doing and how we are explaining it , to understand why that isn’t sufficiently reassuring.

The term “anonymous” is used with sufficient variety of meaning that I find it worrying more often than reassuring. Has the speaker actually implemented a process to produce “personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable” (GDPR Rec. 26), or just blanked the “Personally Identifying Information” (the dreaded US term)? Something like the above approach would give me a lot more confidence.

Two common concerns in incident response are (a) not having the data needed to investigate an incident and (b) not being able to find signs of incidents in a mass of other data. My Networkshop talk (see “ Making IT Safer… Safely ”) looked at how the GDPR principles might help us to get it, like Goldilocks’ porridge, “just right”.

• First, write your incident response (IR) process(es). How am I going to detect a (possible) incident? What will I do when that happens? This helps towards the GDPR principle that processing should be Lawful and Fair: and also means that your documented IR process will be quick;
• Then think what data and sources those processes will need, make sure you know where those are and that you can access them quickly when needed. For the GDPR Data Minimisation principle that tells you which data and sources you don’t need: on the IR side it means the ones you do need should be ready to hand;
• Then think how long after an incident it is realistically possible and worthwhile to conduct an investigation, rather than simply wiping the affected machine and restoring it to a more secure state. You’ll often need external information to interpret your logfiles: what was the network and system configuration? What vulnerabilities existed and were being exploited? Once those data no longer exist an investigation is likely to be hard work. Also, once an intruder has had access to a system for months, they’ve had ample time to do damage and conceal their traces. How much of the logs can be trusted? Getting rid of logfiles that are too old to be worth investigating addresses the Storage Limitation principle: it should also reduce the size of the Incident Detection haystack;
• Keeping logs secure should be obvious, but the same needs to apply to the tools used to access them. Both are a goldmine for any intruder. Security is both a GDPR principle and an IR obligation;
• Automating at least the early stages of incident detection is another win-win. Data Protection law has long recognised that having a computer, rather than a human, look at personal data can contribute to Process Minimisation and reduce intrusiveness: from the IR side, having a machine group logfile “events” into security “alerts” reduces tedium and lets humans get on with what they are best at. Incidentally, I’ve also been finding some guidance on automation in draft AI laws .
• One of the legal advantages of incident response is that most of the identifiers we deal with (IP and MAC addresses in particular) are what GDPR refers to as pseudonyms: they don’t directly identify an individual but require an additional step (such as looking at DHCP and authentication logs) to do so. Postponing that lookup process until we are pretty sure that an incident has happened and that there are victims in need of help, contributes to GDPR Purpose Limitation and delivers the benefits of IR to those who need it most.

Once you’ve developed your process, it may be worth checking it by conducting a more or less formal Data Protection Impact Assessment . We are doing these regularly for the Janet Security Operations Centre, who are finding them a really useful way to think about their work (so much so that they asked to do them more often!). We also publish the results , which we hope will reassure users of the network that, yes, we do process a lot of personal data, but that’s necessary to protect you and we work hard to ensure we do it as safely as we can.

In response to my posts about the relevance of the draft EU AI Act to automated network management one concern was raised: would falling within scope of this law slow down our response to attacks? From the text of the Act, I was pretty sure it wouldn’t, so I’m grateful to Lilian Edwards for the light-bulb moment that not only it won’t, in practice, but it can’t, in principle.

That’s because the AI Act follows in a long tradition of European product liability/safety laws. These regulate how products are developed but say almost nothing about how they may be used after sale or supply. In the case of the AI Act , Articles 9 to 19 apply to the designers and providers of high-risk AI systems, but only Article 29 applies to users, and simply requires them to use the system in accordance with the instructions.

So the AI Act does suggest how to think about AI during its development. As I’ve suggested in a previous post, those are exactly the kinds of thought we should be having anyway, to reduce the risk of our automation going rogue (perhaps encouraged by a badguy). By insisting, in Article 14, that the system provides all the human-machine tools that the user might need to enable effective oversight and control of operations, the Act should even increase the flexibility and speed with which we can deploy and use automation. Might we want the possibility of fully-automated defence? If so, Article 14 reminds us to think, during software development, about the tools we will need to do that safely. Do we need the option to step in, audit and debug the automat’s behaviour? If so, then Article 12 reminds us to build in the controls and records those actions and processes will need.

Where a literal interpretation of law might be a problem is Article 22 of the GDPR, whose opaque wording (a source of confusion since at least 2001) now seems to be interpreted as a ban – rather than a right to request human review – on “decision[s] based solely on automated processing … which produce[] legal effects … or similarly significantly affect[] him or her”. For a full discussion, see chapter 6 of my paper on incident detection . The problem is that, unlike the corresponding text in Recital 38 of the Law Enforcement Directive , the GDPR omits the  word “adverse”. So an over-literal reading could read this as prohibiting automated processing with significant beneficial effects. Say, for example, protecting an individual’s data and rights against spam & malware (approved by the Article 29 Working Party in 2006), distributed denial of service attacks or, the most recent application, ransomware …

Does the GDPR really compel us to wait for individual human approval for automated defensive measures that Regulators have supported for fifteen years? Fortunately, the approach to interpretating European law is “purposive”, so a legally valid response to such a suggestion would be “that’s nuts”.

To help me think about automated systems in network and security management , I’ve put what seem to be the key points into a picture. In the middle is my automated network management or security robot: to the left are the systems the robot can observe and control, to the right its human partner and the things they need.

Taking those in turn, to the left:

• The robot has certain levers it can pull. In network management, those might block traffic flows, throttle or redirect them; in spam detection they might send a message to the inbox, the spambox, or direct to the bin. First thing to think about is how those powers could go wrong, now or in future: in my examples they could delete all mail or block all traffic. If that’s not OK, we need to think about additional measures to prevent it , or at least make it less likely (15).
• Then there’s the question of what data the robot can see, to inform its decisions on how to manipulate the levers. Can it see content, or just traffic data (former is probably needed for spam, latter probably sufficient for at least some network management)? Does it need more, or less, information, for example historic data or information from other components of the digital system? If it needs training, where can we obtain that data, and how often does it need updating? (10).
• Finally, we can’t assume that the world the robot is operating in is friendly, or even neutral. Could a malicious actor compromise the robot, or just create real or fake data to make it operate its levers in destructive ways? Why generate a huge DDoS flow if I can persuade an automated network defender to disconnect the organisation for me? Can the actor test how the robot responds to changes, and thereby discover non-public information about our operations? Ultimately, an attacker could use their own automation to probe ours (15).

And, having identified how things could go wrong, on the right-hand side:

• What controls does the human partner need to be able to act effectively when unexpected things happen? Do they need to approve the robot’s suggestions before they are implemented (known as human-in-the-loop), or have the option to correct them soon afterwards? If the robot is approaching the limits of its capabilities, does the human need to take over, or enable a simpler algorithm or more detailed logging so that the event can be debugged or reviewed later? (14).
• And, what signals does the human need, to know when and how to operate their controls. This could include individual decisions, capacity warnings or metrics, alerts of unusual situations, etc. What logs are needed for subsequent review and debugging (12, 13)?

Having applied these questions to the case of email filtering, they seem to be a helpful guide to achieving the most effective machine/human partnership. Also, and encouragingly, answering them seems to address most of the topics required for high-risk Artificial Intelligence in the draft EU AI Act (the numbers in the bulleted list are the relevant Articles and my visualisation). Whether or not these systems are formally covered by the final legislation, it’s always good when two completely different approaches get to similar answers.

I’m delighted to announce that the Journal of Learning Analytics has published our paper on why and how we developed the Jisc Wellbeing Analytics Code of Practice . If you want to know the context that prompted our interest in data-supported wellbeing, or how we mined the GDPR for all possible safeguards, then have a look at

Cormack & Reeve (2022) Developing a Code of Practice for Using Data in Wellbeing Support

Thanks to all those who – knowingly or unknowingly – contributed to our thinking on this important area.

Decisions whether or not to use Artificial Intelligence (AI) should involve considering several factors, including the institution’s objectives, purpose and culture, readiness, and issues relating to the particular application. Jisc’s Pathway Towards Responsible, Ethical AI is designed to help you with that detailed investigation and decision-making.

But I wondered whether there might be a check that can be done in a few minutes, to get an initial feel for whether a particular use of AI is likely to be a good institutional fit. So here’s a proposed scale of “AI Terrain Roughness”, using objective factors that should be easy to determine from the documentation of a candidate product or service. It only covers some of the relevant factors, but if a glance at this “Terrain” seems to fit your level of experience and comfort with data and AI, then it’s worth moving on to the more detailed investigation.

The scale tries to capture the complexity that’s likely to be involved in using AI for a particular purpose, at all levels from technical and legal to organisational consultation and communications. I’ve chosen a hiking metaphor: for even a simple exploration you should know the route and weather forecast; whereas a complex AI project will require good preparation, skills and equipment, take significant time and teamwork, probably involve setbacks and some (institutional) discomfort. On deeper investigation, you may conclude that the benefits of a particular application do justify that complexity, and experience may give you confidence that you can deal with any problems. But a three-mountain application probably shouldn’t be your first venture into AI.

The factors considered are

• the kinds of data used, or generated, by the AI;
• the technology used, or purpose for which it is deployed;
• the degree of prior control over how the AI is configured for, or learns about, its environment; and
• the need to integrate it with other existing systems and data.

For each factor the scale is based on either legal ( data & purpose ) or technical ( learning & integration ) distinctions that have been identified as making AI more or less challenging. If you really want to reduce this to a single number, choose the highest of the four.

Data	This factor records the most sensitive type of data that the system appears to use or generate. Definitions are taken from the General Data Protection Regulation .
	No use or creation of personal data.
	Uses or creates personal data (“data relating to an identified or [directly or indirectly] identifiable natural person … in particular by reference to an identifier such as a name, identification number, location data, online identifier or one or more factors specific to … that natural person” – GDPR Art.4(1). Note that this is much broader than the US concept of “personally identifiable information”).
	Uses or creates special category personal data (racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, biometric, health and sexual – GDPR Art.9).
Purpose	This factor reflects how regulators assess the risk of using AI with a particular technology or for a particular purpose. Sources are primarily the EU draft Regulation on AI , UK and EU case law, and statements by national data protection regulators, particularly where these relate to education.
	Technology/purpose is regarded as low-risk (i.e., typically, not mentioned) in law, regulation and cases.
	Technology/purpose is regarded as high-risk in law, regulation and cases (including determining access to education or course of someone’s life, e.g. assessing students ’ performance; automated decision-making ).
	Technology has been banned for some purposes or contexts (e.g. behavioural manipulation or profiling likely to cause harm or unjustified disadvantage; face recognition and other forms of remote biometrics ; potentially discriminatory data sources ).
Learning	This factor looks at how the AI is programmed or “learns” about its local environment. As discussed in the AI Pathway , this includes the degree of control over the inputs from which the AI learns, the method by which it learns, and the range of outputs it can produce. Less control lets the system behave in unexpected ways, which may be desirable in some contexts but not in others. Managing the risk of unexpected behaviour adds complexity.
	Learning/programming by well-understood methods to map from a known set of input data to a defined set of outputs (risk can be managed by design and comprehensive testing).
	Learning where outputs are constrained (e.g. to a pre-defined set of decisions or categories), but either the input data or method are unconstrained, e.g. supervised or goal-based learning (risk of undetected or inexplicable learning failures, including bias).
	Real-time learning, feedback loops or uncontrolled outputs (risk of emergent inappropriate behaviour).
Integration	This factor considers whether the AI can be, or must be, integrated with other systems. Such integration increases the technical/effort requirement, but also creates risks that must be managed. AI that consumes data from other systems can amplify problems (such as data quality or understanding of process): AI outputs consumed by other systems may lose essential qualifiers or caveats. Either may produce unexpected or harmful results.
	AI system is designed to operate standalone (e.g. the natural language processing in an informational chatbot).
	AI system can operate standalone, or be integrated with other data and services (e.g. adding transactional ability to a chatbot).
	AI system can only function if integrated with other data and services (e.g. a guided learning system that needs information from the virtual learning environment).

Mountain icon made by Freepik from www.flaticon.com

Legal cases aren’t often a source for guidance on system management but, thanks to the cooperation of the victims of a ransomware attack, a recent Monetary Penalty Notice (MPN) from the Information Commissioner (ICO) is an exception. Vulnerability management was mentioned in previous MPNs (e.g. Carphone Warehouse , Cathay Pacific , and DSG ), but they don’t go much beyond “do it”, and “unpatched for more than four years is not acceptable”. Now we have information on factors to include in patch prioritisation, steps that a vulnerability management process may include, and the features of data protection law that should encourage vulnerability management. The last is particularly interesting, as those features aren’t just in UK law, but are common to privacy laws across the globe.

In this case, intruders stole documents containing personal data from an archive server, before encrypting it and demanding ransom. It was suspected that access was obtained via an unpatched vulnerability in software, though this wasn’t proven. The vulnerability had been announced by the software vendor in December 2019 and mitigations provided, a patch was made available in January 2020, but the system was not patched until June (para 51). The ICO stresses that:

• The original announcement “strongly urges affected customers to immediately upgrade to a fixed build OR apply the provided mitigation” (52);
• The UK National Cyber Security Centre (NCSC) published an alert in late January warning “that malicious actors were exploiting” the vulnerability, with a tool to detect the vulnerability (53);
• The NCSC published a further joint advisory with the US Department of Homeland Security in April (54);
• The vulnerability was given a Common Vulnerability Scoring System (CVSS) Base score of 9.8, “Critical” (57).

There are clear messages here that organisations should monitor security announcements by vendors of software and operating systems, as well as the NCSC and similar (inter-)national bodies; also that CVSS scores and reports of active exploitation should be considered when prioritising remedial action.

Not actually explicit in the MPN, but strongly implied, is that we should be prioritising among vulnerabilities. It seems unlikely that this was the only vulnerability announced during the relevant period: the MPN’s focus on factors creating urgency implies that resources should have been allocated first to the high-severity, actively-exploited one. However, even for this urgent vulnerability, the ICO recognises that immediate patching may not be the appropriate response: there is explicit mention (59) of “test[ing] the patch prior to deployment” and reference (56) to ISO27002’s “suggestion that organisations should define a timeline to react to notifications of potentially relevant technical vulnerabilities, and once a vulnerability has been identified, associated risks should be identified and actions taken, such as patching the system to remove the vulnerability”. Application of the patch should be “prompt” (59) and the Cyber Essentials timescale of 14 days is referenced (57).

The MPN therefore appears to be a strong endorsement of risk-based vulnerability management, using tools such as vendor and NCSC alerts , CVSS scores and likelihood of exploit tools such as Forum of Incident Response and Security Teams’ Exploit Prediction Scoring System (EPSS) and the CISA known exploited vulnerabilities catalogue . The MPN also discusses multi-factor authentication and encryption, so could be a useful reference for those, too. Four features of data protection law should provide a particularly effective incentive to adopt such good practice: the law applies to behaviour, of an individual organisation (the ICO has explicitly rejected comparisons with “industry practice” in previous MPNs), expects risk to be considered (through words such as “appropriate measures”), and can apply sanctions irrespective of harm or causation. I’ve found these features in laws from six continents so, wherever you are, demonstrating how security measures relate to a law of this kind should be a good way to communicate their importance between technical, compliance and regulatory spheres.

Reading the Machine Learning literature, you could get the impression that the aim is to develop a perfect model of the real world. That may be true when you are trying to distinguish between dogs and muffins , but for a lot of applications in education, I suspect that a model that achieved perfection would be a sign of failure.

That’s because our models are often part of a process designed to change the real world. We use analytics to understand how to teach and learn better: students should be enabled and encouraged to beat the model. Even applications like the Graide feedback tool should, over time, result in students needing different feedback, by helping tutors explain hard concepts better the first time around. At the simplest level, that means we should be updating our models frequently, and limiting the age of data we include in them as it will – by design – go out of date.

But the safety-critical world may provide higher-level guidance. Here change is considered inevitable, and something that systems must plan for. An intriguing example from a paper on using safety-critical thinking to inform AI design is that operators will always find different ways to use systems from how their designers intended. That’s not a bug, it’s a feature. The operators’ approach may well work better in the real world; the real world may itself have changed from the design. Systems – technical, organizational, and human – within which Artificial Intelligence sits must detect those changes, ensure that they cannot produce unsafe outcomes , and work out what can be learned from them. If a change makes the system better, we should ensure it is widely adopted: if it highlights a problem with the system (for example that use as designed is inefficient or inconvenient) then the system needs to be improved.

Above all, the reality of change must be part of the culture around AI. A human finding a different way to use it must not be considered a “problem” or “at fault”. They are using human creativity to identify either an opportunity or a risk: both should be welcomed, even encouraged. A system that doesn’t cope with change is a problem.

I’ve been reading a fascinating paper on “ System Safety and Artificial Intelligence ”, applying ways of thinking about safety-critical software to Artificial Intelligence (AI). Following is very much my interpretation: I hope it’s accurate but do read the paper as there’s lots more to think about.

AI is a world of probabilities, statistics and data. That means that anything that could possibly happen, might. We can adjust AI to make particular behaviours unlikely, but the statistical nature of the approach means we can’t make them impossible. This contrasts with or, the paper suggests, is complementary to, the approach taken in safety-critical systems, which declares some outcomes to be prohibited and uses limits and controls to prevent them happening; or, at least, to require human confirmation before they do. The key point seems to be that you don’t rely on a single system to restrict its own behaviour, you wrap independent controls around it.

In the AI world, even in applications where safety isn’t an issue, it strikes me that this kind of approach might be preferable to trying to incorporate all the limits we want into the AI itself. If we need to restrict what impacts the AI can have on the outside world, consider whether that can be done by way of a wrapper around it. Swaddling a baby keeps it warm, but also restricts its movement and cushions its surroundings against unexpected behaviour. And, once the AI is wrapped up, its design and training can focus on achieving the best statistical interpretation of its inputs; any low-probability quirks may be more tolerable if we know there’s an external wrapper to catch those that would cause harm. And, incidentally, to draw designers’ attention to the fact that something considered unlikely did actually happen.

In my visualisation of the draft EU AI Act , such a wrapper would probably invoke Human Oversight, but fit mainly in the Risk Management, Quality Management and Lifecycle areas. As those headings suggest, outside contexts where the use of AI is entirely prohibited, the Act itself aims at managing risk, rather than making certain outcomes impossible. But there are a few hints at hard external limits on the AI: “ mitigation and control measures in relation to risks that cannot be eliminated” (Art.9(4)(b)); “Training, validation and testing data sets … particular to the specific geographical, behavioural or functional setting ” (Art.10(4)), and “shall use such systems in accordance with the instructions of use accompanying the systems” (Art.29(1)). Technical and procedural controls that stop the AI going beyond its intended setting or operating outside instructions might be good candidates for an external wrapper.

This approach naturally focuses attention on the links the AI has to the outside world: these are the points where the AI approach and the safety-critical one meet. What information can the AI measure? What levers can it pull? How could that could go wrong, either through accident or deliberate, including malicious, action? Are there inputs – for example a situation outside the one the AI was trained for, or data such as signs deliberately modified to mislead – where exceptional action is required: warnings, changing algorithm to something simpler or more explicable, ignoring the particular input, or returning control to a human? On the output side, are there actions that the AI is capable of triggering that we need to prevent from taking effect?

This reminded me of a situation, many years ago, where a “smart firewall” decided its network was under attack on UDP port 53. In accordance with its design, it blocked the “hostile” traffic. Unfortunately blocking responses to DNS requests turned out to be a very effective way to make the Internet unusable for everyone behind that firewall. This does seem like an example where we would want the AI wrapper to intervene, probably by asking a human to confirm whether this particularly significant port number should be treated according to the normal blocking rules.

And that, in turn, suggests what I think may be a common rule: that the wrapper around the AI needs to be designed by those with expertise in the particular domain where the AI will operate. A data scientist might reasonably assume that port 53 is no different to ports 52 or 54; a network manager will immediately know its significance. Having identified these unusual situations, the domain experts need to work with AI experts to identify how they might be detected and responded to. Are there relevant confidence levels that the AI can use to warn human operators, to increase logging, or change to a different algorithm? What information or signals could it generate to help operators understand what is happening? What alternative processes can we fall back to if it’s no longer safe to rely on the AI?

Considering those questions before deploying AI should significantly reduce the number of nasty surprises after it starts operating.

I’m hoping to use the EU’s draft AI Act as a way to think about how we can safely use Artificial Intelligence. The Commission’s draft sets a number of obligations on both providers and users of AI; formally these only apply when AI is used in “high-risk” contexts, but they seem like a useful “have I thought about…?” checklist in any case.

The text can be found below, but I’ve been using this visualisation to explain to myself what’s going on. Article numbers are at what I think is the relevant point on the diagram. Comments and suggestions very welcome!

[11/4/22: Added arrows to show that Training and Application have links from/to the outside world]

What the draft Act says (first sentence of each of the requirement Articles):

• Article 9: A risk management system shall be established, implemented, documented and maintained in relation to high-risk AI systems
• Article 10: High-risk AI systems which make use of techniques involving the training of models with data shall be developed on the basis of training, validation and testing data sets that meet the quality criteria…
• Article 11: The technical documentation of a high-risk AI system shall be drawn up before that system is placed on the market or put into service and shall be kept up-to date (further detail in Article 18).
• Article 12: High-risk AI systems shall be designed and developed with capabilities enabling the automatic recording of events (‘logs’) while the high-risk AI systems is operating. Those logging capabilities shall conform to recognised standards or common specifications (obligations on retention in Article 20).
• Article 13: High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system’s output and use it appropriately.
• Article 14: High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which the AI system is in use.
• Article 15: High-risk AI systems shall be designed and developed in such a way that they achieve, in the light of their intended purpose, an appropriate level of accuracy, robustness and cybersecurity, and perform consistently in those respects throughout their lifecycle
• Article 17: Providers of high-risk AI systems shall put a quality management system in place that ensures compliance with this Regulation.
• Article 19: Providers of high-risk AI systems shall ensure that their systems undergo the relevant conformity assessment procedure in accordance with Article 43, prior to their placing on the market or putting into service.
• Article 29: Users of high-risk AI systems shall use such systems in accordance with the instructions of use accompanying the systems.

A really interesting series of talks on how to gather and share information about the performance of networks at today’s GEANT Telemetry and Data Workshop . One of the most positive things was a clear awareness that this information can be sensitive both to individuals and to connected organisations. So, as the last speaker, I decided I didn’t need to present “be careful” and talked instead about “how to reassure lawyers, partners and users”.

Key point, in my experience of being on both sides of that question, is not to ask “Can we do this?”: that immediately gets people envisaging how wrong it could go. Instead, explain “This is what we need to do, here’s how we propose to make it safe: are you comfortable with that?”. They may still be able to suggest improvements, which is great: now we’re working together, rather than in opposition.

As usual on this blog, the GDPR provides some useful pointers to how to make things safe. Remember that its subject is “the protection of natural persons … and the free movement of data”. So how might we do both of those with information about networks? Some ideas:

• Active measurements, especially between devices deployed for that purpose such as perfSONAR , are likely to contain less extraneous, potentially-sensitive, information than monitoring real user traffic. For many of the things you want to know about a network, they are a more reliable source anyway.
• In GDPR terms, data about a device such as a router or server that has a large number of indistinguishable users is much easier to work with than data about something that might be a single-user’s endpoint. Again, measuring performance between institutions and/or key network nodes is often more relevant than individual flows, too.
• If you do need to get closer to the edge of the network and measure user experience, then statistics will often provide relevant information. Data protection law doesn’t have a fixed threshold, but if you can show that you are combining data from at least 10-20 individual users then both users and lawyers will usually be more relaxed.
• Finally, there are situations where you do need to look at individual flows: primarily to debug them (if the individual involved has reported a problem) or to investigate a possible security problem (if someone else has). In the first case it should be natural, and uncontentious, to explain that to debug a network issue you need to look at the individual’s network traffic. Professional codes, such as our System Administrator’s Charter (which also works for network admins) should add reassurance. For security problems, I’ve written and presented a lot on how the GDPR actually encourages necessary use and sharing of network and system data .

Each of these kinds of safeguards can benefit from a bit of simple (definitely not lawyerly!) documentation: how and where active measurement works; how you select devices that are privacy-safe to monitor; how your statistical techniques irreversibly anonymise general data; and the safeguards you apply when investigating faults or security issues. And, above all, how these activities benefit all users of networks and systems. With those in hand you should be able to discuss your planned network telemetry activities with confidence.

GDPR Article 21 provides a “right to object” whenever personal data are processed based on either Legitimate Interests or Public Interests. In both cases, an individual can highlight “grounds relating to his or her personal situation” and require the data controller to consider whether there remain “compelling legitimate grounds for the processing which override the interests, rights and freedoms” of that individual. If there are no such grounds then processing must cease.

Responding to an objection therefore requires the data controller to analyse both the “grounds for processing” and the “interests, rights and freedoms” of the individual. However the different origins of “Legitimate” and “Public” interests mean the data controller’s ability to do so is likely to be very different depending on which basis is used for processing.

For Legitimate Interests there should be little difficulty. Before processing can start, Article 6(1)(f) requires the controller to define the interests served, Article 13(1)(d) requires them to inform data subject of those interests; Article 6(1)(f) requires them to consider what “interests or fundamental rights and freedoms” might be affected by the processing. On receipt of an Article 21 objection, the controller simply has to re-assess the risks to that particular individual’s interests, rights and freedoms and determine whether or not those change the balance with the interests that have already been defined and declared. If they do, the objection must be granted and the individual excluded from processing.

But for Public Interest (Article 6(1)(e)), the data controller’s initial duty is merely – according to Article 6(3) – to identify an applicable law that identifies a “task carried out in the public interest” for which the processing is necessary or else assigns them “official authority”. Responsibility for the content of the law is given to the legislator: it must “meet an objective of public interest and be proportionate to the legitimate aim pursued”. However there is no obligation on the legislator to declare either what that “objective of public interest” was, or what risks were considered when ensuring that it is “proportionate”. This leaves a challenging task for any Public Interest data controller required to “demonstrate compelling legitimate grounds … which override the interests, rights and freedoms”. How can they assess whether the objecting individual’s “particular situation” changes that balance, when they may have no idea how the balance was assessed in the first place?

A possible approach might be to focus on the risk side. Arguably the data controller should have done some sort of risk assessment when determining whether their chosen means of processing was “necessary” for the legally-defined task, or if a different approach would be less intrusive. Having done such a general assessment, it should be easier to determine whether a given individual’s “particular situation” changes that assessment significantly. If the individual is exposed to a risk that was not previously considered then there must be doubt – without additional information – whether the legislator’s original proportionality test would still be satisfied. If there are no new risks, but an apparently increased exposure, then it’s worth considering whether additional safeguards could be applied to bring the individual within the original (presumably acceptable to the legislator) range. As in my other Right to Object post, identifying new safeguards that can make processing safer for everyone is a highly positive outcome of the Objection process. If, however, no suitable safeguards can be found for an increased risk, it seems unlikely that the controller will be able to demonstrate the required “compelling legitimate grounds” to continue processing after an objection.

Finally, it’s worth noting that starting processing for a Legitimate Interest requires “legitimate grounds”; but continuing after an objection has the stronger requirement that those grounds be “compelling”. This makes sense for both Legitimate and Public Interests, since overruling an objection is, itself, likely to leave the individual feeling harmed. When assessing an objection, data controllers should seek reassurance that the risks are safely within the range originally contemplated, not on the borderline.

I was invited to contribute to a seminar on the Right to Object (RtO). Normally this GDPR provision is seen as a way to prevent harm to a particular individual because of their special circumstances. But I wondered whether data controllers could also use the RtO process as an opportunity to review whether their processing can be made safer for everyone.

The diagram shows how this might work. Across the top we have the normal “Accountability” design process whereby a controller identifies the appropriate lawful basis for their processing; applies the relevant tests of benefits, risks and safeguards; and publishes information describing the processing and demonstrating their accountability. Ideally, this should result in processing, safeguards and documentation that makes everyone content.

For the purposes of this talk I’ve divided up the six lawful bases into three groups:

• Those (contract, legal obligation, vital interests) that allow processing to be mandatory for everyone in a group;
• Those (legitimate interest, public interest) that – subject to specific safeguards – allow processing of those who don’t object, in other words where it’s desirable to have broad coverage; and
• Consent, which only allows processing of those who actively opt in; and, incidentally, lets them change their mind as often as they like.

The Right to Object only applies when using a lawful basis from the “desirable” group. Formally it isn’t an automatic right to opt out: the data controller can continue processing if it can demonstrate “compelling legitimate grounds for the processing which override the interests, rights and freedoms of the data subject” (Art.16(1)). Doing so is likely to further upset the individual, so this is a risky approach compared to treating an Objection as a simple opt-out.

But the important point for this analysis is that the arrival of an Objection indicates that at least one data subject was not reassured by the results of our Accountability process. GDPR requires the data controller to address the concerns of that individual, but it may be possible to get more value from the process by considering the Accountability results more generally. Looking at the stages of the diagram in turn:

• It’s possible that an Objection might identify that the wrong lawful basis was chosen, for example if an individual feels they ought to be able to opt-out from processing that we previously identified as mandatory. Changing lawful basis is a significant step – probably involving deleting all data and starting again – so it’s worth quite a lot of effort to get this right first time.
• More likely, an Objection may highlight new risks or safeguards that weren’t identified by the original Accountability process. This sort of change is positively encouraged by GDPR, so it’s well worth considering whether risks can be addressed, or safeguards incorporated, in our normal processing.
• Or the Objection may show that some aspect of the processing – for example its benefits or safeguards – could be better communicated. Both can be explained through a Legitimate Interests Assessment (LIA) or Data Protection Impact Assessment (DPIA); publishing those documents is a good way to reassure data subjects.

Most data controllers would hope that Objections will be rare if they get their Accountability right. So it’s well worth getting the maximum benefit when they do arrive.

When the Government first announced plans to regulate online discussion platforms I wondered whether small organisations would be able to outsource the compliance burden to a provider better equipped to deliver rapid and effective response. Clause 180(2) of the Online Safety Bill suggests the answer is yes:

The provider of a user-to-user service is to be treated as being the entity that has control over who can use the user-to-user part of the service (and that entity alone).

The first thing to note is that this implements the Impact Assessment statement (para 67) that any exemption for educational institutions “also includes platforms like intranets and cloud storage systems, but also ‘edtech’ platforms”. Whether the platform is operated in-house or by a third party, the institution is likely to “control who can use” it, so it will be the provider and the platform’s status will reflect that of the institution.

Conversely, if an organisation wants to outsource the compliance duties for a discussion platform, it must outsource “control over who can use” it. The first such platform I came across was Disqus, with three “ login options ” used here to illustrate how responsibility might work:

• Universal Disqus: a single user account that works across all Disqus instances, so appears to be controlled by Disqus and make them responsible for Online Safety Bill compliance;
• Social Login: including Facebook, Google and Twitter; although it doesn’t manage credentials for these users, or have any control over who they are issued to, Disqus does seem to be able to block trouble-makers, which could be an argument that they “control who can use”;
• Single Sign-On (SSO): assuming the individual organisation “controls” who has accounts in its user management system, it seems likely to be the provider of the discussion service for the purpose of the Bill, responsible for any compliance obligations. Note, however, that “Internal Business Services” – an obvious application for SSO – may be covered by the exemption in Schedule 1 para 7 .

[21/6: Added more examples of public engagement]

[22/3: Updated analysis of why read-only access fits within the para 8 exemption]

The Government has now published its Online Safety Bill : the text that will be debated, and no doubt amended, in Parliament. Compared to last summer’s draft , this is somewhat clearer on whether platforms operated by educational organisations are within scope. The Impact Assessment is clear, according to paragraph 67 the following are excluded from the Bill:

Online services managed by educational institutions, including early years, schools, and further and higher education providers. This includes platforms used by teachers, students, parents and alumni to communicate and collaborate. It also includes platforms like intranets and cloud storage systems, but also “edtech” platforms.

And the text of the draft Bill (unlike last summer) does now match this, mostly. Schedule 1 Part 2 (paras 12-24) has a list of “persons providing education and childcare”: childminding, nursery schools, schools, independent training providers, and further education for persons under 19. This is a welcome clarification over the previous “public authorities”. But Higher Education isn’t there, indeed HE institutions that also provide Further Education are explicitly removed from the exemption.

So HEIs need to look at the exemptions by service type. Here there is another welcome clarification, that Schedule 1 Paragraph 7’s “internal business services … for the purpose of any activities of the business” include “educational institution” (para 7(3) – which does include HE) and that authorised persons (allowed to comment without bringing the service into scope) include “in the case of an educational organisation, pupils or students”.

So, as far as I can see, the main situation where an HEI might fall within scope is if it wants to receive and publish comments from those outside the “closed group” of staff, pupils, students (para 7(2)(c)). This might include outreach activities such as festivals of ideas; inviting public comments or responses to research or community activities; supporting professional practice communities (including cross-institution) or user groups.

Here there seem to be three possible approaches, though none is entirely clear:

1. Create personal accounts for external individuals who wish to make comments thus, perhaps, “authorising” them within the meaning of para 7(3)(iv); this would seem to be the only way to bring in the “parents and alumni” mentioned in the Impact Assessment, so perhaps other interested individuals could be included as well. Para 8 seems to allow content and comments on exempt platforms to be visible to those who aren’t yet “authorised” (adding read-only access to the internal service doesn’t “enable” any new user-generated content, in terms of para 8(1)(b)), so they can decide if they want to request permission; or
2. Rely on the distinction in Schedule 1 para 4 between comments “relating to provider content”, which keep the service out of scope, and comments (other than likes, emojis & votes) on other people’s comments, which bring it in. As in my previous post , turning off threading is the only technical measure I can think of that might possibly support that distinction; or
3. Combine these two ideas, allow non-registered users to like, vote, or post emojis (definitely OK under Schedule 1 para 4); but only registered ones can add text comments (maybe OK under para 7(3)(iv)).

Comments very welcome if you spot a legal or technical possibility that I’ve missed…

[Script for a presentation at a recent Westminster Education Forum event…]

Back in February 2020 we knew what assessment looked like. Jisc had just published “ The Future of Assessment ”, setting five targets – Authentic, Accessible, Appropriately Automated, Continuous, and Secure – to aim for by 2025. Then COVID made us all look at assessment through a new, and much more urgent, lens

It seems to me there were three responses, invoking different roles for technology

First: technology as defender of the existing process. This saw the lockdown requirement for remote assessment as a threat, with technology as the solution. In its most extreme form, this produced e-proctoring: which makes assessment even more stressful, even less Authentic, and even less Accessible for those who don’t have their own private space or technology, or without bandwidth for a live video connection throughout the assessment. These – I hope – are temporary measures…

Then there’s technology as facilitator . Can it help us adopt different forms of assessment that address both the problems we knew about in 2020 and those of the pandemic? Here I’m thinking of things like open book – probably a more Authentic preparation for the workplace – and, if we actually want to test memory, rapid-fire multiple choice. Technology should create opportunities for things like advance downloading and on-device management of resources and timings, so network connectivity is no longer critical; or assessment of multiple-choice responses; or supporting markers by highlighting or grouping desired features in long-form essays; or suggesting when they may be marking inconsistently. Incidentally this takes a very different approach to assessment security: redefining “misconduct” as a feature, rather than a bug. If someone can select appropriate quotes and examples from a whole library, rather than just their memorized notes, or look up multiple choice answers within tight time limits, then they probably know the subject pretty well.

Finally, something to bear in mind as we contemplate a “return to…”. Can technology be an enabler for kinds of assessment that are otherwise impractical? Could it give us a better perspective on assessing group work – or at least an early indication of things going wrong – by looking at patterns of communication among the students? Having technology inspect discussion content may be too intrusive; or maybe not, if the alternative is accusations of “letting the team down”, breaking personal relationships and individual confidence? Or could we use technology to move from assessment of learning, through assessment for learning, to assessment as learning, where the assessment activities themselves are relevant and productive learning experiences. For example students could learn by providing anonymised feedback – with tutor guidance – on one anothers’ work? These exciting ideas, which are already being tested, were presented at a EUNIS workshop in November 2021.

So, will technology be a defender, a facilitator, or an enabler of our post-pandemic assessment system? Could pandemic-enforced changes move us towards better assessments?

Informed by experiences of the pandemic, Jisc has now published revised principles . Assessment should:

• Help learners understand what good looks like;
• Support the personalized needs of learners;
• Foster active learning;
• Develop autonomous learners;
• Manage staff and learner workload effectively;
• Foster a motivated learning community;
• Promote learner employability .

We will need humans and technology to work together to deliver those.

We’ve been talking to computers for a surprisingly long time. Can you even remember when a phone menu first misunderstand your accent? Obviously there have been visible (and audible) advances in technology since then: voice assistants are increasingly embedded parts of our lives. A talk by Joseph Turow to the Privacy and Identity Lab (a group of university researchers in the Netherlands) explored some of the less obvious developments that might be coming; some may even be present already. As well as what we say, might voice technology analyse how we say it? Could a voice assistant recognise our speech to connect us to our accounts? Could it sense our mood, and suggest appropriate comfort or celebratory food? Could it recognise signs of COVID, or other diseases? Could it recognise angry customers and pass them to an appropriately trained human? Could it detect when we are doubtful about a product, and offer a discount? Or when we are excited, and see an opportunity for upselling? Should it?

Turow’s book is about the advertising industry, but the discussion got me thinking about the tools that might exist to regulate the use of these ideas more generally. Could contraints help us explore what seem acceptable applications and technologies with more confidence that they won’t later slip into unacceptable territory? I’ve come up with four, but there may be more:

• Law : Turow suggests that some applications of voice technology in advertising should simply be banned. This sounds like a strong lever, but competition authorities discovered long ago that if a prohibited technology practice was sufficiently profitable then legal costs and fines would simply be absorbed as a cost of doing business;
• Economics (and, remember, I’m a mathematician, not an economist or marketer): Researching, developing and operating these technologies costs money. There ought to be a limit on advertising budgets: if the profit from selling one more item is less than the cost of the advanced tech required to make the sale, I can’t see the logic in using the technology;
• Business : many of the major players in this space have multiple roles, which may have different business incentives. For example, as browser providers, both Google and Apple are changing their rules on third-party cookies in ways that significantly affect existing ad.tech. solutions. Technology can be a gatekeeper as well as an enabler;
• Social : some technologies simply turn out to be socially unacceptable, for example it appears that when Google realised that Glass users were being treated as outcasts, the consumer product was withdrawn. Here the lack of visibility of what voice processing is actually taking place could cut both ways. If buyers, visitors and employers can’t tell how their voices are actually being processed by a particular device, they may become nervous about a whole class of products, even ones that don’t actually involve the offensive practice;

Following on from this last point, there was a fascinating contribution from a linguist who is researching potential uses of voice processing to help care for people living in their own homes. Sensing, triaging and responding to mood, emotion, even medical conditions might have much greater social benefit in these contexts than for advertising. Unfortunately none of my tools seem to cope well with that level of nuance.

Using and sharing information can create benefits, but can also cause harm. Trust can be an amplifier in both directions: with potential to increase benefit and to increase harm. If your data, purposes and systems are trusted – by individuals, partners and society – then you are likely to be offered more data. By choosing and using that effectively, you build further confidence that your innovations will be safe and beneficial to others. Those who have data that adds further value are likely to see sharing with you as low risk, those who want access to your data and services may consider that sufficiently valuable to commit to enhanced standards of practice. But if you lose trust, then you’ll get less data; any innovations – even uses that aren’t particularly innovative – are likely to be closely scrutinised. Spirals lead both upward and downward.

But we do need to step onto the spiral. Doing the absolute minimum with data might be “safe”, but it also provides the absolute minimum benefit. Gradually, those who do a little bit more will be more effective, provide more benefit and become more trusted. Those who do a lot more may do even better in the short term, of course, but if they go beyond what their trust or capability will bear they will flip to the downward arm of the spiral and rapidly become the partner and service that no one wants to be involved with. Occasional mistakes may be tolerated: an effective response to them may even enhance trust in the medium and long term. But intentional deception, much less so.

Trust is an essential foundation: without it, data will only be shared when there’s a legal obligation. Those who share data must trust one another; they must also work to make sure their community is trusted by those who do not participate but may influence wider sentiment and attitudes. But, according to a study for the Open Data Institute on the Economic Impact of Trust , to achieve optimal levels of sharing needs more than just trust. We need to be able to find those with whom trusted sharing would increase value, and to establish permanent infrastructures – in the broad sense: all the way from technical, organisational, legal to cultural – to enable the sharing.

But this move from ad hoc to systematic, even automated, sharing is tricky. Fundamentally, it’s likely to involve moving the basis of what we do from inter-personal trust – which we can all understand, but rarely rationalise – to inter-organisational trust – based on agreed standards, as rational as you need, but always likely to be less intuitive or instinctive. That change has benefits, notably when individuals with strong personal trust networks leave. But unless we take care to show how the new system is even more trustworthy than the old, the transition can be a place where doubts arise.

Externally enforced rules and visible sanctions for breaking them can help here. In an interaction sometimes summarised as “you can drive faster if you have good brakes” strong regulation can actually increase confidence and trust in the regulated. If individuals and groups can see innovation being effectively scrutinised by an independent regulator that champions their rights, they may be more willing to trust the innovators. If they can rely on a regulator correcting – or, if necessary, closing down – an unsafe processing ecosystem, individuals are less likely to worry about, or withdraw, their personal participation. Such a regime benefits regulated organisations both by warning if they approach the downward arm of the trust spiral and by providing more stable public sentiment towards information sharing.

However the UK’s proposals for post-Brexit Data Protection law reform have been criticised as involving both broader permissions for industry and weaker regulation . The analysis above should explain the apparent paradox that this might reduce the amount of data made available for innovation, rather than increase it.

My LLM dissertation (published ($$) in 2016 as “ Is the Subject Access Right Now Too Great a Threat to Privacy? ”) discussed the challenge of reliably identifying a data subject who you only know through pseudonymous digital channels or identifiers. Others have conducted practical experiments, finding that it would, indeed, be relatively easy to use GDPR Subject Access protocols to persuade data controllers to release someone else’s personal data . So it’s good to see new draft guidance from the European Data Protection Board that recognises the problem, provides suggestions on how to handle digital SARs safely, and explicitly states that there may be circumstances where a SAR can lawfully be refused because it’s impossible to sufficiently verify the requester’s identity.

Many on-line services work with just an account name (often an email address) and don’t need strong, or any, proof of the customer’s actual identity. This is good for satisfying the GDPR’s data minimisation principle, but makes it tricky to deal with individuals who contact you outside those existing channels. If you’ve never previously needed to link the human to the account, is it possible to do so retrospectively, or is the risk of accidental or deliberate misidentification – resulting in personal information being disclosed to the wrong person – just too high?

The EDPB guidance (starting on page 23) strongly recommends using existing, established digital channels wherever possible (paragraphs 63&71). To access your personal data, login to your account and view or download it. Alternatively, it may be possible to link the existing account to a new communication channel (never the other way around!), for example sending a one-time access code to mobile phone known to belong to the account-holder (para 66) or requiring someone to demonstrate knowledge of a shared digital secret, such as a previously shared unique cookie, to prove that they are entitled to view the associated data (para 67).

Additional information may sometimes increase confidence that requester is the data subject, but the guidance points out (para 69) that this information must be no more than necessary or proportionate and – less often discussed – that the information requested must actually be useful. The guidance is clear about the common practice of demanding “a national identity document”: first, those are far from secret as hotels, banks and car rental companies routinely request and take copies of them; second that transmitting or storing an image of such a document involves a high risk; and, third, “that identification by means of an identity card does not necessarily help in the online context (e.g. with the use of pseudonyms) if the person concerned cannot contribute any other evidence, e.g. further  characteristics matching to the user account” (para 73). A photographic ID card is excellent proof of the legal name of a person standing in front of you: it may be useless when trying to connect one digital persona (for example an email address) to another (for example a user account). Such proof might be useful for high-risk disclosures: the guidance mentions special category data or processing whose disclosure may present a risk to the individual (para 77). But strong security precautions must be implemented: further copying or storage of the image of the document is “likely to amount to an infringement” of both GDPR and national law (para 78) so only a record that “ID document was checked” should be kept.

Finally, and for the first time I’m aware of, the guidance explicitly states in paragraph 68 that in some cases it may be “impossible” – even with additional information – to identify the requester with sufficient confidence to safely release the requested data. Here the data controller may lawfully refuse the request but, if they can, must inform the requester of the “demonstrable impossibility”.

Most of our digital infrastructures rely on automation to function smoothly. Cloud services adjust automatically to changes in demand; firewalls detect when networks are under attack and automatically try to pick out good traffic from bad. Automation adjusts faster and on a broader scale than humans. That has advantages: when Jisc’s CSIRT responded manually to denial of service attacks it took us about thirty minutes – remarkably quick by industry standards – to mitigate the damage, now we often do it before the target site even notices the attack. But automation can also amplify mistakes: years ago I remember working with one site whose anti-virus update deleted chunks of the PC operating system, and another whose firewall had decided that responses to DNS queries were hostile and to be discarded. Automation has power, in both directions!

So it’s interesting to see suggestions that a future European law on Artificial Intelligence might be further broadened to classify digital infrastructure operations as a “high risk” use for AI, on a par with industrial and public safety measures. [UPDATE: it’s been pointed out that page 48 of the UK’s Cyber Security Strategy also mentions “ Use of AI to secure systems. Many AI algorithms are complex and opaque, with the potential to introduce new classes of vulnerability. Understanding and mitigating these emerging threats is a priority”]. That definitely doesn’t mean AI is banned, but that its power is expected to be wrapped in a lot of human thought. Leaving an AI to just get on with it isn’t appropriate. But nor should a human be approving every decision: that throws away much of automation’s potential benefit.

Instead the model (in Articles 9-20 & 29 of the original Commission draft ) seems to require a close and continuing partnership between human and AI capabilities. Humans must design appropriate contexts and limits within which AI can work, taking account of both the risks of too much automation and too little; inputs and training data must be appropriately chosen and updated; there must be documentation of how the system was intended to behave, and how it actually did; live information, monitoring, authority and fallback plans must enable humans to take over quickly and effectively when the AI is no longer behaving appropriately. There are repeated reminders that systems may be working in actively hostile environments. As with my old DNS example, attackers may try to deceive or confuse the AI and turn its power to their advantage. Humans will definitely be needed to identify these possibilities, design precautions against them, and respond when the AI is recruited to the dark side. Providing excellent digital infrastructures will need both excellent AI and excellent people, working together.

I’ve been reading about Slow Computing and the need for ‘digital forgetting’. But, unlike the GDPR Right to Erasure, human forgetting isn’t clean: more often involving uncertainty rather than simple elimination. That leaves our database in a different state: whereas digital erasure has no effect on the records that remain, much of our human memory is still present but of uncertain quality. We don’t know which of those memories are accurate, so should be wary of placing too much reliance on them.

So what would happen if there were a Right to Data Decay? We might imagine databases inhabited by chaos monkeys that randomly alter a small percentage of field values each time they ran. To mimic human forgetting, the likelihood of alteration could relate to the age of the record or the length of time since it was last used. Over time, old or unused records would become increasingly unreliable, and relying on them increasingly hazardous. Provided the database holder bears the cost of that unreliability, they might well be motivated to introduce more frequent re-validation processes and/or to simply dispose of data once it approached the age of unreliability.

Those seem like incentives that benefit both the data controller and the data subjects. So do we need such a right? Probably not, once you realise that the real world already develops in ways that make data go out of date. The university still sending glossy alumni brochures to the person who sold our house a decade ago is wasting money through not recognising that; the online book-seller still recommending children’s books for relatives who are now grown up is devaluing its other recommendations to me and, presumably, “people like me” too; data science models that incorporate too-old data may well lose accuracy, especially where the purpose of the model is to change the reality that it reflects.

Recognise data decay, save money, improve services!

Feedback and performance review are routine parts of many employment relationships. So it’s surprising to find that they take us into obscure corners of data protection law. Regulators have been clear for more than a decade that an opinion about someone is personal data , but there has been much less exploration of the fact that it’s likely to be information about two people – the one who is the subject of the opinion and the opinion holder. And – arguably unlike the situation of social network “friends” – those two people have very different relationships to the same data: for one it may be no more than an impression, for the other it may have significant financial or reputational impact. Any processing of that data must deal with two sets of rights and responsibilities, which are inextricably entangled.

Entangled data present an immediate challenge to the normal triad of data protection safeguards: information, individual rights, and data controller accountability. GDPR Article 13 offers the data subject information and individual rights at the point where data are collected or observed. But the subject of an opinion isn’t involved in the collection of their personal data, so processes that assume the data subject is present won’t work. This makes the data controller’s exercise of accountability (for example ensuring compliance with the Article 5 Principles and an Article 6/9 legal basis) even more important.

Uncertain data protection requirements may matter less for traditional human feedback and review processes, which are also covered by a great deal of good practice guidance (e.g. from the Chartered Institute for Personnel and Development ) and, ultimately, employment law. However a new trend for using data, rather than human sources, to generate “opinions” may require new thinking. For example, would it be acceptable to use student sentiment, attendance or performance data as part of staff reviews? It’s worth reviewing which GDPR sections might help us think about situations where the same personal data relates to two different people. Although the following uses review/feedback for illustration, the aim is to sketch a possible approach to entangled data in general.

The Article 5 Principles and Accountability are always a good starting point. The Art.5(1)(b) requirement for purposes to be explicit means those who provide information must know who else it may be linked to and, because Art.5(1)(a) requires fairness to both, that knowledge must not distort the providers’ own behaviour or relationship with the processes through which data are collected. This links to Art.5(1)(c): if introducing a second purpose affects the quality or meaning of the data, then it is doubtful whether it will still be “adequate and relevant” for either purpose. The Art.5(1)(d) requirement that data be “accurate” highlights a specific challenge for entangled data: what if an opinion is accurate in the holder’s mind but not the subject’s? Do the two sides (and processes) share, or need, the same definition of “accurate”? Some of the less familiar Individual Rights may help to resolve these situations, as discussed below.

Article 5(1)(a) requires that every action to process data must be covered by one of the Article 6 lawful bases . Where the same personal data relates to two different people those bases don’t have to be the same but, if they are not, then particular care will be needed to ensure that the right conditions and safeguards apply to all the relevant processing. For example, Jisc’s standard model for analytics uses the Legitimate Interests of the institution for finding patterns in observed behaviour. Those interests must not (by Art.6(1)(f)) be over-ridden by the “interests, fundamental rights and freedoms of the data subject”: where there are multiple data subjects, this means that every one’s interests, rights and freedoms (not just to privacy) must be considered. By contrast, using data for performance review is most likely to be justified as “necessary for the performance of [the employment] contract” (Art.6(1)(b)), which involves a different set of data protection and employment law safeguards.

If different lawful bases apply to the different data subjects, it’s likely that their processing will be for different purposes. Art.5(1)(b) permits multiple purposes, so long as they are “compatible”. Incompatible purposes can only be added with consent: a problem for entangled data because – even if their contexts permit valid consent – it must be obtained from both data subjects. Although Article 6(4) is most often used to assess compatibility when different purposes affect the same data subject, its factors for Purpose Compatibility should also be useful in assessing which data and purposes are compatible across different groups of data subjects. Highlighting links between processes, context of collection, nature of data, possible consequences, and safeguards suggests that compatibility is most likely when the two groups have a similar relationship with the data, and when the processing has similar levels of impact on both. Using data collected in an informal context from one group to influence formal consequences for the other is unlikely to be compatible. We should also beware of situations where the meaning and significance of the information change significantly when viewed from the two sides.

As mentioned above, the normal Individual Rights safeguards of information (Art.13) and subject access (Art.15) are less effective when the same personal data relates to multiple data subjects. Information cannot be provided at the point of collection if a data subject is not present; subject access rights of one data subject may need to be limited to protect the privacy and rights of the others (Art.15(4)). It may be more appropriate to use Article 14’s “Information to be provided when personal data have not been obtained from the data subject”, but Art.14(2)(f) highlights that disclosing “from which source the personal data originate” may, again, breach the privacy of the individual source. Normally these information and access rights combine to help data subjects, controllers and sources identify inaccurate data and correct it. If necessary, Article 16 gives the data subject a legal Right to Rectification. However, where entangled privacy rights and disputes about the meaning of “accurate” hinder this approach, it may be better to combine the Right of Rectification with the Article 18 Right to Restriction. Such a process would let a data subject contest the accuracy of personal data and exclude it from further processing until/unless its meaning can be agreed.

This review suggests that data protection law can guide appropriate use of multi-party entangled personal data, but that this may involve considering some less familiar sections and perspectives.

Under the GDPR’s breach notification rules, it’s essential to be able to quickly assess the level of risk that a security breach presents to individual data subjects. Any breach that is likely to result in a risk to the rights and freedoms of natural persons must be reported to the relevant data protection authority, with at least initial notification within 72 hours. Where the risk is high, affected individuals must also be notified. Where there is unlikely to be a risk, only internal documentation is required. The Article 29 Working Party published general guidance on breach notification in 2017, which was subsequently adopted by the European Data Protection Board. However the EDPB has now published a supplement, specifically on the question of assessing risk .

This takes the very helpful approach of looking at clusters of similar breaches, and explaining the factors and differences that may lead to different risk assessments. Clusters cover ransomware, data exfiltration, internal human risk (both deliberate and accidental), lost/stolen devices/documents, postal errors; finishing with a couple of examples of social engineering. In many cases the guidance suggests appropriate mitigation measures, as well as what notifications are required.

This guidance should be helpful in reducing the amount of thinking required in the immediate stressful aftermath of detecting a security breach. Check if your incident matches one of these patterns and follow the relevant instructions for initial notification (Note that the earlier guidance explicitly allows update notifications, whether to provide more information, revise the risk assessment, or even declare a false alarm).

Even better if you can use it to review your systems and datasets before any breach occurs , when you can take time to assess the likely risk that would be created by a future confidentiality, integrity or availability breach. With that kind of specific preparation, a quick check of whether the actual breach was significantly different to the anticipated one should be all you need before initiating the relevant notification and response processes.

More than a decade ago, European data protection regulators identified the problem of “consent fatigue”, where website users were overwhelmed with multiple requests to give consent for processing of their personal data. In theory, responding to those requests let individuals exercise control but, in practice, it seemed more likely that they were just clicking whatever was needed to get the content they wanted.

Despite the Article 29 Working Party’s 2009 comment (not specific to websites) that “ the complexity of data collection practices, business models, vendor relationships and technological applications in many cases outstrips the individual’s ability or willingness to make decisions to control the use and sharing of information through active choice ” and the UK Information Commissioner’s concern “ that the creation and sharing of personal data profiles about people, to the scale we’ve seen, feels disproportionate, intrusive and unfair, particularly when people are often unaware it is happening ”, regulators’ and legislators’ response to the problem has largely been to look at legal and technical formalities, not to question whether the data processing behind many commercial websites was simply too complex for individuals to meaningfully control.

Thus the latest enforcement action by the French regulator against concerns the widespread practice of offering a choice between “accept” and “configure”: the latter typically leading to pages of detailed settings that the user must refuse individually. The Regulator’s conclusion that “it is not as easy to refuse cookies as to accept them” – as required by GDPR Art.7(3) – is hardly surprising. But given earlier enforcement actions demanding that data controllers give more detail and granularity, providers might be tempted to think “you asked for it, you got it”. As with previous enforcement, the result seems more likely to be an adjustment of practice towards the regulator’s rulings, rather than a major change of approach.

Ironically, the opportunity for that change may now come from the technical side, where browser creators (including Google) are proposing new technologies for Internet advertising that may not obviously relate to existing legal provisions and rulings. The Information Commissioner has responded with a set of privacy expectations for such developments : these still call for “User Choice”, but alongside “Data Protection by Design”, “Accountability”, “Purpose”, and “Reducing Harm”. Whether this will result in a new approach, or just a new front in the battle of formalities, we will have to wait and see.

Terminology matters. OK, you’d expect me to say that, as a sometime mathematician, engineer and lawyer. But the importance to all of us is highlighted by a confusing tangle of terminology that has grown out of Ann Cavoukian’s original idea of “ Privacy by Design ”.

That phrase was introduced in 1995 – just too late to make it into the European Data Protection Directive – but was taken up by many Regulators as a requirement that “ has always been an implicit requirement of data protection ”, according to the UK Information Commissioner. While the general idea seems good the phrase itself is open to criticism, notably from engineers required to actually turn it into code: that lawyers and philosophers have been arguing about the meaning of “privacy” for at least a couple of millennia, making it tricky to understand what the design requirement actually is.

In 2016, the drafters of the General Data Protection Regulation avoided the phrase and, instead, created an obligation of “Data Protection by Design” (DPbD) in Article 25(1). According to the European Commission that requires “ implement[ing] technical and organisational measures, at the earliest stages of the design of the processing operations, in such a way that safeguards … data protection principles right from the start ”. So far, so good: the GDPR does define “data protection principles” (in Article 5) and most of them have reasonably obvious translations to technical requirements. Unfortunately, this change was made so late in the process that many privacy regulators simply retitled their existing “Privacy by Design” documentation, making it unclear whether Data Protection by Design is actually a new, clear, requirement, or just a rebrand of the old, unclear, one. For example the European Data Protection Supervisor’s Preliminary Opinion on Privacy by Design describes itself as “contributing to the successful impact of the new obligation of ‘ data protection by design and by default’ as set forth by Article 25 of the General Data Protection Regulation”.

If that wasn’t enough, Article 25(2) added a third phrase and requirement: “data protection by default”, defined as “by default, only personal data which are necessary for each specific purpose of the processing are processed”. And, perhaps inevitably, the set was soon completed by authors referring to this as “privacy by default”.

So does it matter that we have “Privacy by Design”, “Privacy by Default”, “Data Protection by Design” and “Data Protection by Default”, all used pretty much interchangeably? I think it does:

• “Privacy by Design” is a good slogan and provided valuable direction for twenty years. But it now seems to be cited – worryingly often – as “unclear, so whatever I have done must be OK”. As a privacy-sensitive individual, and engineer trying to do the right thing, that’s a distraction I’d rather not have floating around;
• “ Data Protection by Design ” is, I think, much to be preferred. As in my new paper “ Thinking with GDPR ” it contains about a dozen specific obligations (the Principles, the Individual Rights, and the obligations of whichever Lawful Basis is chosen), most of which translate into clear engineering requirements;
• “Data Protection by Default”, even though it’s the phrase chosen by the GDPR, I find, frankly, scary. A default is something each individual can choose to turn off. Again, there are already too many claims that user behaviour can “opt out” of data protection, for example by publishing personal data (“it’s on the web, so I can use it how I like”) or, more often implicitly, by “consent”. Our terminology shouldn’t be giving support to such ideas;
• “ Privacy by Default ” – ironically, the phrase that doesn’t actually have official blessing – I find a much more helpful engineering requirement. Indeed looking at the explanation in Article 25(2) it seems a much more accurate description of what’s required: wherever we offer individuals a choice, the do-nothing option should be the one that provides the best privacy protection. That doesn’t require a philosophical debate about the global meaning of “privacy”, just that for each decision we must be able to locate the options on a linear scale. If we can’t do that, maybe the choice isn’t one that can be communicated in a “concise, transparent, intelligible and easily accessible form, using clear and plain language” (Article 12(1)), and we should do more work on our Data Protection by Design to come up with choice(s) that can.

So: four phrases, two of which positively support engineering that respects personal data, one of which obscures that goal, and one that actively undermines it. I think the choice does matter…

Last year, I was invited to give a talk “on GDPR” to NISO , an organisation that develops standards for managing digital information. While most of my thinking and writing has looked at applying data protection law to existing systems, this seemed like a good opportunity to think about how you might use it at an earlier stage, when designing a protocol, system or software. Flipping the usual phrase, rather than “data protection by design”, can we do “design by data protection”?

The resulting ten-minute talk got a good response from both the conference audience and organisers; and an offer to publish a paper if I would like to write an expanded version of the ideas. The result – “ Thinking with GDPR: A guide to better system design ” – has now been published.

It starts by pointing out three common, but false, assumptions about the law: that it’s about preventing processing of personal data, that it’s most relevant to individuals, and that it’s mainly about consent. Then moves on to how organisations can use the law – in particular the Principles, Lawful Bases and Individual Rights – to design their systems, demonstrate accountability in their approach to personal data, and build trust. Then gives three practical examples – student voter registration, federated access management, and data analytics – of how Jisc and the wider research network community are using the approach to design and develop systems that are innovative and world-leading both in the functions they provide and in their built-in respect for personal data.

A few years ago I wrote a post on how the GDPR copes with situations when there was a conflict between the obligation to prevent, detect and investigate incidents and the obligation to inform all those whose personal data you process. Do you, for example, need to inform someone who is attacking your systems that their hostile activity has been detected?

GDPR Article 14(5) provides a general tool for resolving that conflict: you don’t need to inform if doing so “is likely to render impossible or seriously impair the achievement of the objectives of that processing”. Telling an attacker what attacks you can detect would clearly “seriously impair” our ability to protect systems and data.

A new Commission Decision provides a longer (but still not exhaustive) list of situations when such conflicts might arise: when complying with the exercise of individual data protection rights “would undermine the purpose of providing IT security operations and services, inter alia, by revealing the Commission’s investigative tools, vulnerabilities and methods, or would adversely affect the rights and freedoms and the security of other data subjects”:

• communicate alerts and warnings relating to IT security events and incidents;
• respond to and contain IT security events and incidents;
• facilitate tools and operations through security audits, security assessments and vulnerability management;
• increase the awareness … in the field of cybersecurity;
• monitor, detect and prevent the occurrence of IT security events and incidents;
• review privileged user accounts (Article 2(2)).

Formally, this Decision and its list only apply to the security and incident response activities of the European Commission itself. But it’s still a helpful indication to other CSIRTs – and to regulators – that the importance of these activities for protecting personal data may make it necessary to apply the more general exemptions (such as Article 14(5)) provided in the GDPR.

Over the past decade or more, we’ve developed federated access management as a technical, policy and legal framework to exchange up-to-date information to help current staff and students access the resources they need. Authentication, status and membership information all need to be fresh to be useful and frequent use makes it worth organisations entering into formal federation agreements to ensure that.

But there’s another kind of educational information that’s far longer-lived. Degree certificates are still relevant forty years on; transcripts of course content at least a year or two. These are used much less often, but with a much wider range of organisations. Various technologies have been proposed to store and process these in digital form. So what might their legal and organisational framework look like? Existing processes around paper degree certificates provide an interesting angle.

Issue:

• Institution creates certificate and provides it to graduate. Certificate has various mechanisms to enable validation (often a university crest) and to help detect forgeries.
• Graduate stores certificate in a safe place.

Presentation (perhaps many years later):

• Graduate presents certificate to potential employer as proof of qualification; employer’s grounds for requesting that personal data will usually be that it is “necessary to take steps prior to entering into a contract” (GDPR Art 6(1)(b)).

Assessment. Employer does one or more of the following, to obtain the verified information they need:

• Checks that certificate is not an obvious forgery . This uses characteristics of the certificate plus generic information (for example whether the institution used that title at that date). There is no processing or transfer of personal data.
• Checks that the qualification is relevant to the job applied for. Again, this uses only information on the certificate, any processing personal data should be limited to matching the qualification details against the requirements of the specific job.
• (optional) Verifies with the issuing institution that the information in the certificate is still valid. This does involve a new processing of personal data, by both the employer and the institution: typically the graduate is asked to consent (GDPR Art.6(1)(a)) to this one-off transfer in their application (“you agree that we may check your qualifications with your university”).
• (optional) Requests additional information – topics covered in the course, relevant skills demonstrated, etc. – from the issuing institution. Again, this is a new processing activity based on a one-off consent.

All these steps, except one, either do not involve processing of personal data or have an obvious GDPR provision. So what part of GDPR best fits the first step – the creation and issuing of the certificate?

Consent seems to work at a high level, but thinking of it as “data subject requests copy of own personal data” makes either the Article 15 Subject Access Request (SAR) or the Article 20 Right to Portability  (RtP) a much more precise fit. The right to portability would be best of all – the intention here is, precisely, that the data subject will subsequently “port” the information on their certificate to another data controller – however this formally only covers information the student has provided to the institution. We therefore need to fall back on the more general Article 15 right of any data subject to “obtain … access to data” that is being processed by the controller. In practice, many regulators seem to have treated the Article 20 right as a formatted SAR anyway, and Article 15(3) already requires “provided in a commonly used electronic form”.

So a possible framework for long-lived digital credentials might look like:

• Process 1 (“Issue” above): user requests a digital copy of their qualification: institution provides the data in standard electronic form (e.g. a W3C verifiable credential ) as an Article 15(3) Subject Access Request.
• Process 2 (“Presentation” above): applicant presents qualification data as part of an application, most likely requested as Article 6(1)(b) necessary for contract.
• Process 3a (“Forgery” and “Relevance” above): recipient checks that the offered data is not obviously forged (e.g. that its digital signature is valid ), and is relevant to the role applied for. This may, particularly with a formatted credential whose validity can be checked separately to its content, involve no processing of personal data; or the processing may be necessary for contract or consented.
• Process 3b (“Verification” and “Request” above): recipient may ask the issuing institution to either confirm or supplement the information contained within the certificate. To authorise this one-off transaction the recipient should obtain a recent, specific, consent from the applicant. The institution has a free choice how to respond: for paper certificates, consent is routinely received as a photocopy of a tick on a form, but the digital world may offer more robust processes to check and confirm (for example verified credentials can include expiry dates , when either the holder or issuer must refresh the information and its signature).

This framework, which treats Issue, Presentation and Assessment as individual, free-standing, transactions, is very different to the federated access management model’s ongoing exchange of data backed by a long-term contractual relationship (the “Federation Agreement”, for example for the UK Access Management Federation ). That provides safeguards such that we can take the individual’s requests to access content and services as implicitly directing the necessary release and processing of real-time information, it being in the legitimate interests of institution and service provider to facilitate access to services desired by their members. Requiring explicit consent for each transmission of information would be impossibly onerous for all parties.

However in the qualification context, information flows are much rarer and sparser, occurring between a much wider range of parties. Requiring a pre-existing contract is therefore too heavyweight, and a framework where the individual explicitly requests each transaction is both more appropriate and a much more natural fit for the actual data flows. In both frameworks, the responsibilities of each party (and their limits) are clear and appropriate to the context.

Finally, there is a natural transition between the two models. Where an institution and an employer have a particularly close relationship (for example through student placements) they may well want to enter into a contract to cover that relationship. Such a contract could naturally include moving to the “federated” model with its stronger guarantees of fresh data and safeguards agreed once, by contract, rather than on every individual transaction, by consent.

A colleague spotted an article suggesting, among other things, that Virtual Reality could provide a safe space for students to practice their soft skills . This can, of course, be done by classroom roleplay but the possibility of making mistakes that fellow students will remember could well increase stress. This certainly chimes with feedback I received when suggesting that my team practised giving presentations in what seemed to me the “safe” environment of a company lunchtime chat: “no, we’d far rather have an audience of complete strangers”.

So what about an audience of avatars, whose memories can be wiped at the end of the session? The article suggests that AI can provide feedback on tone, body-language and eye-contact; even that the session could be replayed with the student taking another role and watching an avatar act out their behaviour.

But this gets ethically interesting. This sort of recording and feedback involves what would normally be considered “high-risk” uses of AI, particularly processing of faces and emotions. Conventional wisdom says that if that is to be done at all then there must be a lot of human oversight and involvement. But providing that involvement seems to break the private safe space, which was why we used VR in the first place. I was reminded of school language labs, where it was just me and the non-judgemental tape reels… Until the teacher’s voice suddenly burst in to my headset…

Giving the student the option to invite another human to view the recording seems fine: “I don’t understand the AI’s comments, please help”. But should the teacher listen in? Or intervene? What happens if the student starts to interact in ways that could harm themselves or others? There are also fascinating articles on how our interactions with devices can quickly become uncivil , because “it’s only a robot”. Can the VR system recognise those situations and, if so, what should it do?

Should “What happens in VR, stay in VR”? I don’t know…

Recently I was in a video-conference where Apple’s “smart” assistant kept popping up on the presenter’s shared screen. Another delegate realised this happened whenever the word “theory” was spoken. It’s close…

These events – which I refer to as “false-wakes” – are privacy risk: maybe small, but that depends very much on the nature of the conversations that are going on around them. So it would be good if privacy law helped suppliers to reduce them.

However the recent guidance from European privacy regulators seems to have the opposite effect. They treat each “wake-word” as granting consent for the subsequent processing, which means that even detecting a false-wake involves unlawful processing, as it turns out (in retrospect) that the speaker had no intention of granting consent, so there was no lawful basis for the processing. And it makes it impossible to use the recorded data to tune the system to reduce future false-wakes, because the only possible response to not having a lawful basis is to delete the data immediately (and probably silently, to avoid admitting the law-breaking). So a position has been created where the Lawfulness Principle is actively discouraging compliance with the Accuracy one. Having written about using GDPR as a design tool , I wondered whether I could do better. Is there a way:

• To avoid having to tolerate unlawful processing (even if time-limited);
• To provide a mechanism whereby speakers can let their recordings be used to improve accuracy; and, incidentally
• At least partially, to address the EDPB’s overall concerns about the invisibility of voice-triggered devices?

On closer reading, there’s only one lawful basis that can possibly cover a false-wake spoken by someone who hasn’t previously interacted with the device. Consent or contract might work for the person who installed the system, someone who has enrolled in its voice recognition process, or who intended to issue a command. But those require the speaker (as “data subject”) to have done, known or intended something. For accidental processing, perhaps of a visitor’s voice, the only possible lawful basis is “legitimate interest”.

That immediately triggers the notice obligation in Article 13, to provide information “at the time when personal data are obtained”. That probably shouldn’t mean reading out a complete privacy notice (unlikely to meet the requirement for intelligibility), but the device should draw attention to itself and where that information can be found. Suggestions for the latter have included QR codes or other layered notices .

To meet the storage limitation principle, and give some chance of satisfying the legitimate interests rights-balancing test, the legitimate interest in listening for a wake-word should terminate as soon as possible. However that might still leave enough time for the system to offer the speaker the choice of whether a short sound recording may be used for the new purpose of reducing the likelihood of future false-wakes. That is probably best offered as an opt-in consent dialogue (“Sorry, you woke me by mistake, may I process a five-second recording to work out why?”), where anything other than a clearly-spoken “yes” results in the recording being deleted. A further refinement might be to play the recording, so the speaker knows what will be shared.

So, yes, all three objectives can be achieved. Just pick the right legal basis, and the rest follows 🙂

We can probably agree that “Ethical Artificial Intelligence” is a desirable goal. But getting there can involve daunting leaps over unfamiliar terrain. What do principles like “beneficence” and “non-maleficence” mean in practice? Indeed, what is, and is not, AI?

Working with the British and Irish Law, Education and Technology Association (BILETA) , Jisc’s National Centre for AI has mapped an alternative route, taking smaller steps through more familiar ground, towards that goal. This involves two key insights. First, that the intuitions and practice of the broad education community will usually guide us towards responsible and ethical actions and away from unethical ones. And, second, that discussing more familiar questions can help us discover those intuitions and practices.

Or, to put it another way, unethical actions are usually bad for other reasons, too.

Our “ Pathway Towards Responsible, Ethical AI ” considers four key questions for any proposal that relies on data or algorithms; suggests groups – including students, tutors, minorities, and boards – with whom they can most usefully be discussed; and links to resources that can provide a framework for those discussions:

• “Does this proposal fit our institution’s objectives?” clarifies what we are trying to achieve and why; whether it is likely to work; and how it relates to the institution’s mission;
• “Does using AI fit our institution’s culture?” considers how policy-makers have viewed similar uses of algorithms and data; their compatibility with our local culture; and whether they are likely to be welcomed or resisted;
• “Are we ready?” looks at both institutional readiness – in terms of skills, systems, data and trust – and whether suppliers can provide services appropriate to that level of readiness;
• “Does AI raise specific issues?” considers what the application and context require – in terms of control, explanation, bias and learning – from any technology that may be used.

Finally, for ideas that make it this far, the Pathway reviews how tools from law, technology and ethics can support a successful deployment. It’s only this stage that requires specialist knowledge: the four questions are much more about human and institutional experience.

[Update (Nov’21): I’ve discovered that Patrick Breyer MEP has published a “ parallel text ” of the three current proposals (Commission, Parliament and Council). Not exactly easy reading, but it makes it much easier to see where they are similar, and where there remain significant differences]

[Original (Feb’21) post…]

After four years, and nearly three years after it was meant to be in force, the EU Council of Ministers has finally agreed on a text of the proposed ePrivacy Regulation . This isn’t the end of the process: before it becomes law the Council and European Parliament have to agree on a single text. That may take a while, as the version the Parliament agreed on more than three years ago took the European Commission’s original proposal in a significantly different direction.

One area where there does seem to be agreement is the use of data about communications (and in some cases their content) to protect the security of networks, systems and end-users. Whereas the Commission draft only covered the security of networks, the Parliament’s Amendment 15 copied the text of Recital 49 from the GDPR into Recital 16, recognising the need to protect both the security of networks and of connected devices, and the range of organisations involved. Article 6(1)(b) was amended (Amend.72) to explicitly add “availability, integrity, confidentiality” of networks to the “security” permission, and Article 8(1)(da) added (Amend.90) to permit patches to protect the “security, confidentiality, integrity availability and authenticity of … terminal equipment”. The Council add more threats to the list of examples in Recital 16 – viruses, phishing and spam (as a threat to availability) – and explicitly link security measures to the prevention of personal data breaches. Processing of metadata and content to “detect or prevent security risks or attacks on end-users’ terminal equipment” is added to Art. 6(1)(c) and use of end-device capabilities to Art. 8(1)(da). They also recognise the need for security patches in Recital 21b and Article 8(1)(e).

Much less clear is what will be agreed on processing for other purposes. The Parliament retained the Commission’s closed list of purposes: transmission and security; quality of service, billing, and fraud prevention (these three only being allowed to use traffic data, not content); and where the user has requested a specific service and granted consent to the processing it requires. Where possible, the Parliament tightened these permissions, notably by the requirement in Art 6(2a) (Amend.77) that processing likely to result in a high risk to the rights and freedoms of individuals must be subject to a Data Protection Impact Assessment. The Council, however, has extended the Commission’s list, to include “compatible purpose” processing of both network and terminal information (Rec.17aa/20aa & Art.6c/8(g)), protecting “an interest which is essential for the life” of the user (Rec.17a & Art.6b(1)(d)), and scientific research (Rec.17b & Art.6b(1)(e)&(f)) (Parliament also mention scientific use, but only in relation to analytics: Amend 89).

On cookies and other use of end-device capabilities, the permissions for those necessary to transmit a communication (e.g. load balancers) or to provide a service requested by the user (e.g. shopping carts) are largely carried over from the existing Directive ; there is also general agreement – though variation in detail – on a point first raised by the Article 29 Working Party in 2013 (!), that at least some analytics cookies should be permitted without prior consent (Art 8(1)(d) Amend 89). Otherwise the Parliament and Council positions are very different. Parliament insist on prior consent for all other use of end-device capabilities and (Amend 92) that refusal to give such consent must not result in the user being denied access to any service or function (often referred to as “cookie walls”). Council Recital 20aaaa, however, allows cookie walls so long as the user has a choice between free and (implicitly) pay-for versions of the service. This does not apply where this would “deprive the end-user of a genuine choice”, for example websites operated by public authorities or dominant service providers. But Recital 21aa suggests that some ad-funded services (online newspapers are given as one example) may not need to offer a choice.

These are the main areas I’ve been keeping an eye on, but there are also significant divergences between the Council and Parliament elsewhere. Resolving those seems unlikely to be quick.

Something made me uneasy when a colleague recently referred to “AI bias”. I think that’s because it doesn’t mention the actual source of such bias: humans! AI may expand and expose that bias, but it can’t do that unless we give it the seed. That’s rarely deliberate: we might treat it as a result of “how the world is”. But maybe we should be using “AI bias” less as an unconscious excuse and more as a sign that something about that world is wrong, and needs fixing?

Some alternative terms I find useful when thinking about that:

• Learned bias : even if we could provide AI with a perfect view of the human world, that world contains biases. And the way we choose what view to give the AI may introduce more. A good example of this is the “gendered pronoun” issue: AI that is given a corpus of human text to read will encounter the phrases “she is a CEO”/”he is a nurse” less often than their gender-flipped counterparts. So when translating from one of the many languages that doesn’t have gendered pronouns – such as Turkish , Finnish or Persian – an AI is likely to pick the more common version. Heroic efforts are being made to detect and fix this issue in translation sources and algorithms. But it would be good to try to work on the underlying issue, too.
• Blind-spot bias : AI feeds on the data we give it, but what about the data we can’t give it, because we don’t have it? Are the white spaces on our maps of city air quality because there is no pollution, or because there are fewer smartphone owners there to report it? We need to learn from Abraham Wald’s “returning bombers” paradox and investigate those blind-spots at least as intensely as the data-rich ones. The blind-spot may, itself, be telling us something important.
  • Digital confirmation bias: a stronger subset of blind-spot bias occurs when our use of the AI’s predictions creates a positive feedback loop and increases the disparity between data-rich and data-poor areas. Sending more police into areas where there are more reports of crime is the classic example.
• Design bias : sometimes bias doesn’t come from the data, it comes from oversights or assumptions during the design of the AI or its surrounding processes. We need to try to detect and question these. Do all students actually have a smartphone ? Are skin-tones generally lighter than the background ? I know it’s much easier for me to remember that about 1 in 20 people don’t have “normal” colour vision : those who do can try to keep us in mind, but making your design team as diverse as possible can reduce the amount of “others’ shoes” effort that’s needed.
• Incentive bias : it’s all too easy for a “goal” to slip into “bias”. Social media analysts noticed a long time ago that the goal of “increase attention” was closely linked to the bias that will “ increase outrage ”. Hence the algorithms that curate for the former are likely to promote extreme expressions and views. It’s not yet clear whether a different definition of “attention” can fix this problem, or whether the AI will always (re-)discover the human weakness for strong emotions. Perhaps in these stressful times there might be a market for a social media platform that delivered attention through calming, mindful moments? I’d certainly be willing to give it a try.

The Information Commissioner’s response to proposals for data protection reform has another take on my idea of the law helping us to find sweet spots : those points shouldn’t be seen as “trade-offs”, but as mutually beneficial. As the ICO puts it:

The economic and societal benefits of this digital growth are only possible through earning and maintaining people’s trust and their willing participation in how their data is used. Data-driven innovations rely on people being willing to share their data.

Others have suggested a safety analogy:

Good brakes let you drive faster

That has certainly been our experience at Jisc. We’ve been developing, using and publishing GDPR tools as part of our innovation in use of data since before the GDPR was passed! Not because it’s a legal requirement (which is often arguable) but because it’s a really good way to think through issues and explore concerns with customers and users of our services. And, because we’ve done that thinking and exploration, those stakeholders seem inclined – when we come up with a new idea – to approach it with confidence. They may want to point out issues we haven’t thought of, which is great as we can work together to improve, but we rarely get a reaction of pure suspicion.

When I first suggested using a DPIA to explore and explain our network security services , it felt like radical transparency. Now it seems much more like common sense. You can find the tools we’ve developed along the way at:

• Security Operations Centre DPIA
• Learning Analytics DPIA
• Intelligent Campus DPIA template
• Wellbeing Analytics Code of Practice (including DPIA and Purpose Compatibility templates)

I keep coming back to the idea that Data Protection law (at least as expressed in the GDPR) has two explicit objectives: to “protect natural persons” and to enable “free movement of data”. And those are presented as compatible, not conflicting. In the case of a couple of the Article 6 lawful bases for processing that’s fairly obvious: if I enter into a contract with you then I want you to process the data that’s necessary to deliver that contract; if a life is at risk then society wants the processing of data that’s necessary to save it.

But can we view the other lawful bases, with their associated conditions and safeguards, as guides to finding similar sweet spots? If you want to do this kind of thing, under these conditions and with these safeguards can we (either as individuals whose data are processed, or as members of society whose data might be processed in future if we experience particular situations) reach consensus that the processing benefits both?

If all the conditions of the other four bases are genuinely met – notably fully-informed free consent, laws (whether permissive or mandatory) that include appropriate safeguards, other interests that are both legitimate and not overridden by rights and freedoms – then it seems plausible.

And this is increasingly important, because these win-win situations are stable. Alternatives, where one party wins at the expense of the other, probably aren’t. There are an increasing number of options for those who want to resist, frustrate or corrupt the processing of their data. Long ago I got fed up waiting for a consensus resolution to the targeted advertising debate, so I adjusted my browsers and behaviour to exclude that ecosystem as far as I could. That does make some websites practically unusable, and every now and then a volunteer site reminds me that I am a lousy freeloader depriving them of income.

But that’s the point: the alternative to win-win is lose-lose, where both personal protection and data availability are diminished. If we can use data protection law as a guide to how to avoid those conflicts, it has to be a good thing.

“Consent” is a word with many meanings. In data protection it’s something like “a signal that an individual agrees to data being used”. But in political theory “consent to be governed” is something very different. A panel at the PrivSec Global conference suggested that the latter – also referred to as the “social contract” – might be an interesting lens through which to consider the use of data, algorithms and Artificial Intelligence. The basic idea is that creating a society involves a balance: we give up some individual freedoms (for example to choose which side, and how fast, to drive on the roads; or to take whatever property we choose) in order to create a communal life that works for everyone.

So how does that help in discussing new technology? How can we create technologies that enhance humanity rather than exploiting our weaknesses? First is the idea that a valid social contract must include everyone, it can’t be imposed by those with (current) political, technological or economic power. All views, impacts and situations need to be considered and weighed. Which means we need to make that discussion accessible, particularly to children and the (digitally) vulnerable. That may actually be easier if we debate principles, rather than technological details: “it’s too complicated” and unquestioned following of algorithmic outputs are signs that we’re getting the debate wrong. Complication, algorithms/goals and data sources are choices made by humans: we need to discuss whether and when it’s acceptable to make those choices. Above all, principles (and systems) must be designed for the vulnerable and, maybe, adapted by those with greater autonomy: not the other way around. Tools such as consequence scanning and ethically-inclined design can help us explore possible futures.

To claim this kind of consent, organisations must commit to putting the principles into practice, and their doing so must be monitored and publicly reported on. As in many fields, without “trust but verify” there will be a natural tendency to creep into loopholes. Data Protection Officers may be the first layer in this verification, but their burden of maintaining independence and capability is likely to need external support and reinforcement. And we must beware of confusing adoption with acceptance. Something that is convenient but resented is not part of the social contract and should not be read as such. Popularity creates a particular risk: that widespread reluctant adoption may squeeze out the alternatives that would be a better fit for the social contract. The difficulty of buying paper street maps of major cities (“everyone uses their phone”) was mentioned. Bringing new technologies within the social contract won’t be quick or easy, but doing so should reduce the risk of individual harm or resistance, and of future “techlash” by parts or all of society.

A fascinating panel at the PrivSec Global conference looked at how individual courts and regulators have responded to the Schrems II decision on international transfers of personal data . That decision, and the subsequent guidance from the European Data Protection Board, aimed to establish a consistent regime for transferring personal data from the EEA to external countries. However individual regulators now seem to be applying the case in ways that reflect particular local circumstances, such as the existence of functionally-equivalent alternatives or the sentiment of local populations towards transfers (in particular to the US). That may be good for avoiding a complete breakdown in personal data flows, but such divergence doesn’t help organisations trying to work out what they need to do, either as exporters or importers.

Although Schrems II stressed that exports might be possible, based on an assessment of risk, the EDPB guidance sets a very high standard . In effect it permits only exports for encrypted storage and technically-arcane forms of processing. Exports to most normal data handling services are essentially prohibited. Individual regulators do, however, seem to be returning to the risk-based idea: asking what is the risk to data subjects of transferring that data to that organisation for that purpose. As a per-instance assessment, this is still onerous: perhaps something that large corporate law teams can do, but unlikely to be feasible for a start-up. Even the corporates may be pointing out that this is the third export regime in a decade with, at least in the case of exports to the UK, every possibility of another one within four years. Worryingly, they may be tempted to consider corporate risk – will we get caught and how much will it cost – rather than the risk to data subjects.

This is made worse by the fact that the main risk that Schrems II focused on – compelled access by foreign security services – may be unknowable, certainly to foreign exporters, and quite possibly to local importers as well. Until or unless regulators start providing, at least, baseline per-country or per-sector opinions, the best option seems to be for exporters and importers to collaborate to make a reasoned assessment . This should include not only differences in law (exports to a bank may well have more legal protection than those to a technology company) but also in practice (a dating site may have a different level of law enforcement interest to one dedicated to professional networking). The good news is that, as far as the panel were aware, regulators’ actions have not yet gone beyond warnings and – admittedly short notice – orders to cease transfers. In a world where even regulators seem uncertain, exporters and importers who do their best to assess the risks and document their decisions may hope they will not suffer worse than these.

The Information Commissioner’s new blog post explains how Data Protection law should be seen as a guide to when and how to share information in emergencies , not an obstacle to such sharing. In health emergencies three provisions are most likely to be relevant:

Explicit Consent (GDPR Art.9(2)(a)): where an individual chooses to disclose information, such as a health condition or disability, their university or college can discuss the different ways that information could be used or shared, and let the user choose which of them should be done or allowed.

Vital Interests (GDPR Art.9(2)(c)): where there is an imminent threat to life or serious injury and the individual (for example because they are unconscious) cannot give explicit, informed, consent.

Employment and other laws (GDPR Art.9(2)(b)): allow states to legislate to either allow or require sharing of health and other sensitive data. Recital 52 gives “prevention or control of communicable diseases” as an example of such legislation; Recital 53 “monitoring and alerting purposes”. Such laws must provide “appropriate safeguards for the fundamental rights and interests of [individuals]”. Schedule 1 Part 1 of the UK Data Protection Act 2018 provides a general framework; further details may be contained in emergency legislation.

The Information Commissioner’s Data Sharing Code of Practice includes a section on Data Sharing in an Urgent Situation or in an Emergency .  This stresses that organisations should try to anticipate and plan for such emergencies, but that when an unforeseen or unplanned emergency occurs, “it might be more harmful not to share data than to [proportionately] share it”.

A fascinating discussion at today’s QMUL/SCL/WorldBank event on AI Ethics and Regulations on how we should develop such ethics and regulations. There was general agreement that an ethical approach is essential if any new technology is to be trusted; also, probably, that researchers and developers should lead this through professionalising their practice. First steps are already being taken, with journals requiring that papers consider ethical issues: both those that can be addressed in design and implementation, and those that need to be monitored as the technology or context develops.

However for some fields, applications and contexts, society will not trust AI (and there are already plenty of people ready to lead a backlash against it) unless there are legal sanctions as well as ethical codes. The process of developing those laws must identify and address the concerns of all stakeholders, not just legal or technology experts. That needs a lot of genuine consultation (the multi-stakeholder process of Internet governance was cited with approval), with stakeholders, legislators and regulators all being enabled to make their contributions. Participation must represent all those who may be affected: lack of prior technological or legal knowledge must not be a barrier. Some technologies may be found to be socially unacceptable.

Successful regulation should be a benefit to those who want to enter these markets, by stabilising society’s attitudes to them. This was illustrated by two contrasting “new technologies” of the past: GM crops, and the Warnock Committee on Human Fertilisation and Embryology.

[UPDATE: slides from my TF-CSIRT presentation are now available]

Several years ago I wrote a paper on using the GDPR to decide when the benefits of sharing information among network defenders outweighed the risks . That used the Legitimate Interests balancing test to compare the expected benefits – in improving the security of accounts, systems or globally deployed software – against the risk of sharing personal/pseudonymised data – on a bilateral, community or public basis – that would be needed to deliver those benefits.

That framework has been widely used by incident response and security teams, however it left a couple of loose ends. First, that the framework was motivated by Recital 49 of the GDPR which, back in 2016, only said that such data processing “could” be linked to that legal provision; and second – particularly for the international sharing that is essential to protect the international internet – that GDPR Article 49 requires data exports to serve a “compelling legitimate interest” of the exporter. Information sharing more directly benefits the recipients, who can learn from others’ experience and analysis how to secure their own systems, so we need a bit of logical gymnastics to claim that improving overall security benefits the exporter, too.

I’m delighted to report that my latest paper – “ NISD2: A Common Framework for Information Sharing Among Network Defenders ” – ties up those loose ends.

This is based on Recitals in the European Commission’s draft Network and Information Security Directive (NIS2D) . Published in early 2021, these show how thinking has developed. First, although Recital 69 repeats the GDPR wording linking incident response to the Legitimate Interests basis, the permissive “could” in the GDPR is now a significantly stronger “should” in the NIS2D. The same Recital explicitly describes sharing “to raise awareness in relation to specific cyber threats, exchange of information in the context of vulnerability remediation and coordinated disclosure … voluntary exchange of information on those incidents, cyber threats and vulnerabilities, [Indicators of Compromise], tactics, techniques and procedures, cybersecurity alerts and configuration tools” as a component of the defender toolbox.

Second, the individual harms that motivated incident response in 2016 (“unauthorized access … malicious code distribution and … denial of service attacks”) have been replaced in NIS2D Recital 3 by societal harms (“impede the pursuit of economic activities …, generate financial losses, undermine user confidence and cause major damage to … economy and society”). This seems to invoke a different GDPR export provision, that sharing is “necessary for important reasons of public interest” (Article 49(1)(d)). Not only is this a more natural description of what information sharing actually does, it also removes the possible duty – if relying on a “compelling legitimate interest” – to inform regulators of all transfers.

So now I have a more complete framework, with NIS2D thinking joining up GDPR law:

• When : network defenders should share information when this will serve an “important public interest” (GDPR Art.49(1)(d)) …
• Including those of preventing “financial losses, undermin[ing] user confidence, … caus[ing] major damage to … economy or damage to society” (NIS2D Rec.3) …
• What : the information shared should be “necessary for [that] interest” and “not over-ridden by the rights and freedoms of individuals” (GDPR Art.6(1)(f) in fact, as discussed in my original paper, sharing among defenders normally benefits those rights and freedoms) …
• How : protected by tools such as the Traffic Light Protocol and, perhaps, norms such as those of the UN Global Group of Experts (NIS2D Rec.6).

Thanks to ScriptED for publishing both papers as open access.

The EDPB’s new Guidance on Data Protection issues around Virtual Voice Assistants (Siri, Alexa and friends) makes interesting reading, though – as I predicted a while ago for cookies – they get themselves into legal tangles by assuming “If I need consent for X, might as well get it for Y”.

We’ve been focusing more on text-interface chatbots than voice interfaces, so I did a quick compare and contrast. My conclusion is that voice interfaces do raise novel data protection challenges: text interfaces probably only familiar ones. Parts of the EDPB guidance would become relevant if a chatbot were used to continually monitor what you typed into other applications, to combine its text input with other data such as location or device type, or to provide access to emergency services.

Using the EDPB’s headlines and paragraph numbers…

Transparency . Both kinds of bot must provide accurate notices of processing (62); where the bot is part of a wider set of functions, the bot’s operations should be made clear and not buried in that notice (61); both types must support rights of information, subject access, etc. in accordance with the GDPR (65). In what becomes one common theme of the guidance, these rights are likely to be easier to provide through a (layered) text interface than audio: the gabbled style used to speak the rules on lottery adverts almost certainly wouldn’t pass Article 12’s intelligibility requirement! Accountability returns to this idea, requiring that any transparency messages communicated through voice interfaces should also be provided (to users and regulators) in written form on websites (143).

The other theme is the loose connection between a voice-bot and the humans around it: unlike a screen/keyboard interface you don’t need to physically touch, or even look at, a voice-bot to use it. Reminding humans of the bot’s presence and state is a new challenge (63), as is the likelihood that more than one human will interact through the same interface while needing to be provided with personalised transparency information and rights (64). There are useful reminders (66) that both kinds may have access to incidental information about the surroundings – sounds for an audio interface; location, device type, etc. for text – and the need to design technical, process and legal controls to handle this data (or exclude it) in accordance with the GDPR ( Data Minimisation : 139).

Purpose limitation . Both kinds must meet the usual GDPR requirement to only provide functions that users expect and only perform processing that is necessary for those functions (89). The EDPB also recall (90) the need to provide separate opt-ins for each purpose that is based on consent. Their heavy emphasis on consent may make this more challenging than it needs to be.

Data Retention . Both kinds must minimise their storage of personal data, both in quantity and duration (108, 106). The GDPR normally offers anonymisation as an alternative to deletion, but effectively anonymising a voice recording may be impossible (107). The EDPB’s framing makes misidentified wake-words a particular problem (109). This is unlikely to arise for text-based bots unless they are continually monitoring typing and popping up when they think help is required.

Security . Both kinds of bot need to meet the usual GDPR requirements on security of data (123); if either is used as a way to access emergency services then availability should be a major design focus (124). Bots that provide transactional facilities, or that implement rights such as subject access, must ensure they appropriately authenticate the user before making changes or providing personal information (122). Text chat-bots can do this using a wide range of existing keyboard/screen mechanisms. For voice-bots many of these are unavailable (reading out a password is not secure!) and …

Processing Special Category Data . … voiceprint identification – which is a unique feature of the voice interface – are in the GDPR’s most sensitive category of data (“biometric authentication”). Nonetheless the EDPB do seem attracted to them as a way to address the “multiple-users” issue as well as individual authentication. They point out legal and technical challenges: that voiceprints should be stored and processed on the device, not a central server (133); that additional standards on protecting data must be followed (134); and that individual identification must be sufficiently accurate for all individuals and demographic groups (135). If either type of bot is used to access special category data then GDPR’s Article 9 provisions will, of course, apply.

It’s interesting to see the (UK) ICO’s response to the (EU) consultation on an AI Act ​​​​​​. The EU proposal won’t directly affect us, post-Brexit, but it seems reasonable to assume that where the ICO “supports the proposal”, we’ll see pretty similar policies here. Three of those seem directly relevant to education:

• That remote biometric identification for non-law-enforcement purposes is high risk (para 26). This is unsurprising, given the ICO’s past comments that live face recognition (LFR) represents a step-change in invasiveness . But the EU proposal isn’t limited to LFR in public spaces: any processing of live (or near-live) video to try to identify or classify faces is likely to be covered.
• That managing high-risk deployments needs a “continuous iterative process” (para 19). The EU proposal stated (again not binding on the UK, but may be a strong hint) that high-stakes assessments in education (e.g. course entry/selection) are high-risk places to use AI.
• And (para 21) that there should be a public registry for high-risk AI systems “in particular for systems deployed in the public sector”. I don’t know whether they count education as “public sector” for this purpose – neither of the commonly-cited registers in Amsterdam or Helsinki seem to include educational deployments at the moment – but Helsinki does include a couple of library management applications and three chatbots.

Incidentally, Helsinki’s register may actually go further than the EU proposal. Two of the chatbots are deployed in health contexts, so do have an obvious reason for high-risk categorisation. But I can’t see why other three applications (parking information, book recommendations and shelf management) would meet that threshold. The draft EU Act actually suggests that chatbots would normally qualify as low (but not no)-risk: any register that lists every use of natural language processing is going to need some good navigation tools to find the few high-risk applications in amongst all the mostly-harmless ones.

The ICO’s proposals for international transfers seem closer to the actual findings of the Schrems II case than the EDPB’s effective demand that processing of non-pseudonymised data be kept within Europe. However, as a risk-based scheme, it will require more work from both exporters and importers to demonstrate that transferring doesn’t create significantly greater risk to individuals.

The ICO’s scheme has two components:

• an International Data Transfer Agreement (IDTA), which is a contract between an exporter and an importer (playing the same role as the current Standard Contractual Clauses), and
• an International Transfer Risk Assessment (ITRA), which exporters should use to determine whether the IDTA is an appropriate mechanism for a particular transfer.

The ITRA is the new feature, responding to the Schrems II requirement that exporters consider whether any contract can provide sufficient protection for individuals whose data are exported. Whereas the EDPB focused on the powers and actions of law enforcement and security services and whether those “ go beyond what is necessary and proportionate in a democratic society ”, the ICO takes a broader view.

Its first stage is to consider whether the contract itself is likely to be enforceable: does the receiving jurisdiction recognise international agreements, including those that confer benefits on persons other than the contracting parties (here, data subjects)? If not, are there factors about the specific transfer – for example the behaviour of the importer or the nature of the data being transferred – that make it unlikely that lack of enforceability will actually create a significant risk to data subjects? If not, and judicial enforcement against the importer is likely to be needed to protect data subjects, are there ways to change the nature of the transfer (for example making data pseudonymous) to reduce the risk to an acceptable level?

The second stage is to look at third-party access (including under surveillance laws) using a similar series of tests: is the regulation of such access sufficiently similar to the UK that the transfer does not create significant additional risks? If not, how likely is it that third-party access to the transferred data will occur? If more than minimal, what risk to data subjects does such access present? If more than low, are there additional steps and protections the exporter can apply?

So long as both stages conclude that the additional risk created by transferring from the UK to the overseas jurisdiction is low, the transfer can take place under an IDTA.

For each question, the proposed guidance provides tables of factors that would create, increase or decrease risk. These seem to have been chosen with specific applications in mind: for example employee data is separated into basic low-risk data such as name, job title and contact details; medium-risk non-sensitive records such as CV and payroll history; and high-risk banking details and special category data. This is much more helpful (and realistic) than the EDPB’s bald statement that there are no circumstances in which “transfer to cloud service providers … which require access to data in the clear” or “remote access to data for business purposes” can be adequately protected . However it does still leave a potentially complex assessment for general-purpose cloud services. It’s possible that these will be addressed in the Example blocks (which are blank in the current draft); if not, then at least using an ITRA to explore and document the risks will show that the exporter has done its best to identify these and mitigate any increase caused by the transfer.

The ICO’s consultation is open till October 7th.

The Government’s Online Safety Bill proposes to impose duties on “user-to-user services” to deal with harmful (including both lawful and unlawful) content and to protect free speech while doing so. Unlike most operators of on-line discussion platforms, educational institutions already have legal duties in both areas: through legislation on safeguarding , preventing radicalisation , and free speech . These have been extensively discussed – in Parliament and courts, in committees and among practitioners – to find and implement an appropriate balance. It’s therefore important to work out what impact, if any, the new legislation might have.

Unfortunately this isn’t clear, at least in the most recent text published for pre-legislative scrutiny. The Government’s preparatory report in December 2020 proposed (on p.18) excluding

services managed by educational institutions that are already subject to regulatory or inspection frameworks (or similar processes) that address online harm.

“Educational Platforms” are still mentioned in the current impact assessment (para 111) but not in the draft Bill itself. That does contain an exemption for “public bodies” (Schedule 1 para 6): some activities of educational institutions fall within that term for freedom of information and data protection purposes, but the reference here is to human rights law, whose application to education is less clear.

The impact may also depend on technology. Education rarely involves providing the unstructured public discussion spaces that are the Government’s main concern. More often, discussion starts from content provided by the institution: perhaps a lecture, research idea or discussion question. Participation may be limited to members of the institution, in which case it may be exempt as an “Internal Business Service” (Schedule 1 para 4), but what about discussions shared between multiple institutions? Or public discussion of research, which is something that Governments have been keen to encourage and institutions to facilitate?

The draft Bill also makes a distinction (in Schedule 1 para 5) between comments on the original material (exempt) versus comments on other comments (not exempt). But this seems hard to implement in practice and impossible in technology. If A comments “I found this helpful” and B comments “I agree with A”, is B commenting on the original material or on A’s comment? In technology terms, the platform operator might disable threaded comments. But, as is familiar to anyone who has tried to follow the unthreaded chat alongside a video-conference, the main effect of that is merely to make discussion more confusing to readers and contributers.

There are likely to be more opportunities for clarification as the proposal develops. The current text is a draft, which a Parliamentary committee will read and comment on, probably in the autumn. The Government is then expected to publish a Bill, which will go through the usual Parliamentary process, potentially including amendments. Then, to a greater extent than many legislative proposals, the requirements will be interpreted by a regulator – expected to be Ofcom – in the light of changing circumstances. If educational institutions are brought into scope, this will need to be done carefully and clearly, to avoid creating mismatches with existing practice and law that could benefit harmful content and/or suppress legitimate and necessary free speech.

The ICO’s Age Appropriate Design Code (more familiarly the “Children’s Code”) may have been written before lockdown, but it could provide useful guidance to everyone designing or implementing systems for the post-COVID world. We’re all trying to work out what a “hybrid” world should look like, whether in schools, colleges, universities, workplaces or social spaces. A Code that helps us provide respectful digital systems should be relevant to all these and more.

It’s also worth remembering that, as a statutory Code, the guidance fits within the legal requirements of the GDPR. It doesn’t create new law. It may highlight features that are particularly important when working with children, but only to suggest how to comply with what is the law for users of all ages. And which of us adults wouldn’t welcome digital services that offered clear explanations of what they were doing, had respectful default settings, and didn’t try to push the boundaries of the law?

The ICO’s description of being “‘datafied’ with companies and organisations recording many thousands of data points about [you]. These can range from details about [your] mood and [your] friendships to what time [you] woke up and when [you] went to bed ” is creepy whether you are 8 or 80, not something that suddenly becomes acceptable on a particular birthday.

The Code sets out 15 “ technology-neutral design principles and practical privacy features ”:

• Best interests of the [individual]: which should already be our focus in education;
• Data Protection Impact Assessments (DPIAs): to help understand the risks and how we might mitigate them; we’ve also found them a really useful way to show individuals that we are taking care of them and their data;
• Age-appropriate application: provide services and systems that meet the needs and safeguards of your users (the ICO specifically notes that you can “apply th[is] standard to all your users” if you aren’t sure of their ages or needs);
• Transparency: using clear language – accessible to the likely audience(s) – to explain what we are doing with data (a GDPR requirement in any case);
• Detrimental use of data: the ICO provides an indicative list of these , with industry sector standards to be aware of;
• Policies and community standards: acting in accordance with our statements, and our community’s expectations;
• Default settings: it’s not just children who “will just accept whatever default settings you provide”, that’s why “privacy by default” is a GDPR requirement;
• Data minimisation: collecting and using only the minimum data you need, and for the time you need it; again, a GDPR requirement for everyone;
• Data sharing: arguably a principle that is stricter for adults than for children – there may be lawful reasons to share children’s data with parents & guardians, or in emergencies, but make sure you know what they are and have well-defined processes for them;
• Geolocation: location data has always been regarded as particularly sensitive (for everyone) under ePrivacy laws, but has never made it explicitly into GDPR. This is a reminder of that sensitivity;
• Parental controls: adults might want to control these for themselves, but most people still have things they would prefer not to encounter accidentally;
• Profiling: again, the GDPR warnings about profiling apply to users of all ages. The Code suggests that it should be off by default;
• Nudge techniques, or “exploitation of human psychological bias”: should not be deployed to encourage users to make bad (for them) choices;
• Connected toys and devices: remember that users may forget that it’s not just a toy/smart speaker/fitness device/etc. but also a powerful data collection device;
• Online tools: that make it easy for individuals to exercise their data protection rights.

Which of these would you want to reserve only to children, and not want in services designed for adults too?

An example in the ICO’s Frequently Asked Questions highlights that the principles aren’t bans, but areas to think carefully about. Using geo-location to provide information relevant to an international student’s country is fine, but let them choose whether to give you access to location data, and stop using that access as soon as you have the information you need.

In a few cases it may be reasonable to expect adults to (slightly) better understand the consequences of their choices. But distracted post-lockdown adults will still be grateful for clear explanations and services that just do the right thing.

Jisc’s 2020 Future of Assessment report identifies five desirable features that assessors should design their assessments to deliver: authentic, accessible, appropriately automated, continuous and secure. Those can sometimes seem to conflict, for example if you decide that “secure” assessment requires the student to be online through their exam, then you have an “accessibility” problem for students who may not have the required broadband provision in their best location for taking the test. But that example highlights some differences between the features, which may help us avoid those clashes and produce assessments that are better for everyone .

First is how much control we, as assessors, have over the definition of the terms. “Accessible” is almost entirely defined externally: either by law, for some groups of students, or by circumstance such as broadband disparity. If we want our assignments to be “accessible” then we don’t have much choice what that means. But, at the opposite extreme, if you define “secure” as meaning “conducted within the assessment rules”, then it’s clear that we have a lot of freedom to change those rules and make nearly any kind of assessment “secure”. This needn’t create a free-for-all:  security rules should still offer an appropriate combination of (pre-assessment) prevention, (during assessment) enforcement, and (post-assessment) verification.

For example if you think that collusion among students is a “security” problem, you can re-define the assessment as group, rather than individual, work; if you think using external resources is a “security” problem, then make it an open-book research question, rather than a memory one. In each case, what used to be a “security” problem is now an authentic assessment of a valuable transferrable skill. It may well be that the only non-negotiable aspect of “secure” is “assessment was completed by the student”: but even here there are several possibilities, including checking that the assessment product is consistent with previous work, in quality, style, etc. Sometimes a student will “get it” on the eve of the exam, and their mark jump beyond the expected range: sometimes they may have a bad day and do significantly worse. In both cases we should be investigating gently, rather than jumping to conclusions about what happened.

Which raises another difference between the features: when they have to be present. Accessibility generally needs to be delivered at the time(s) of assessment, though even here there may be some potential for prior or subsequent adjustment. For security the moment of assessment is just one among a host of stages where measures – both preventive and corrective – can be applied. These can and should be designed to work together. Traditional invigilators will generally give warnings and make notes of anything that looks like a breach of the rules: only if a student’s behaviour is disrupting others will they be asked to leave. Combining the notes, output and other relevant information to produce an assessment score is done, later, by markers.

To avoid conflicts between the features, it may be helpful to start by looking at the ones where we have fewest options. Those are probably “accessible” and “authentic”: the latter offers some choices around which skill(s) or scenario(s) we want to be authentic to, but once that decision has been made, the only question is how close to reality we need to get. “Appropriately automated” and “continuous” are likely to be somewhat constrained by external environmental factors: including assessment facilities, technologies and staff workload. But, as well as being a little more flexible than accessible and authentic, these two are actually quite hard to even define until you’ve done those first two. And, as discussed above, “secure” has the widest suite of options, so it ought to be possible to apply it to nearly anything the first four have produced. There may still be some negotiation and adjustment between the five, but getting the right sequence should avoid the sort of painting-into-a-corner situation created by starting with the feature that is actually most flexible.

This morning’s “multiplier event” from the Onward from Learning Analytics (OfLA) project highlighted the importance of human and institutional aspects in a productive LA deployment. They begin at the end – what is the desired outcome of your LA deployment? The answer probably isn’t “a business intelligence report”, and almost certainly not “a dashboard”. Starting from “a one-to-one conversation between a student and their personal tutor” gives a much richer perspective.

That approach makes clear the importance of tutor and student preparedness: are tutors confident of having those conversations, do they know when and how to hand over to others, and what kind of conversations do students find most helpful? The project has developed online tools for the first two of those: to help tutors explore the most appropriate timing, messenger, medium, content and follow-up for their interventions, and to help them do “warm referrals” where recommending the student talks to other experts is seen as supportive, rather than a brush-off.

Striking the right note for an individual student is hard, since it will depend on many factors, some of which are (at least initially) unknown to the tutor. For this reason, however tempting it may be, it’s probably not a good idea to try to shock students into working harder: there are just too many possible reasons why they may not appear to be making the expected progress. Sharing data with the student also needs to be handled with care: among other negative responses, too many graphs may be seen as obfuscating the message, comparison with benchmarks may be demotivating. One productive approach is to present “the system” as an antagonist, and invite the student to collaborate with the tutor in changing behaviour so as to confound its expectations. More generally: data can be part of conversations, but it must not be at their core.

This greater understanding of what comes after the application of learning analytics technology should then inform what comes before . LA purpose(s) that align with the institution’s mission are much more likely to be supported. The answers to “Why do we need an LA platform? Who are we trying to help?” lead naturally to answers to “who needs to access it?”, “what data presentations will be helpful?” and “what data literacy will users need?”. These, in turn, help derive requirements for both systems and data.

In this light, identifying a small set of effective data sources becomes an operational requirement as much as a legal one: too many sources make it hard to explain how students’ difficulties can be addressed. Human understanding is essential: one algorithm identified “enrolment status” as a (statistically) strong indicator of outcome. Well, yes! An interesting idea was to use transparency as an operational requirement/test: if students can’t explain the system to each other then it’s too complex. Reducing the number of sources also reduces the work needed to maintain data quality and ensure that changes in collection or systems don’t disrupt the student support purpose. The project has an excellent info-graphic on these policy issues, which should be available on their website soon.

Finally, any application of learning analytics involves many trade-offs. Early interventions will be less accurate than waiting for more data, but intervening the day before a student obtains a poor grade isn’t helpful. Ease of use tends to increase automation, which reduces both student and staff autonomy. And there is no guaranteed right way to communicate bad news. But a multi-layered approach that covers everything from data to process, presentation and literacy, seems to provide the best opportunity to adapt to circumstances and bring each alert to a satisfactory conclusion.

This morning’s Westminster Forum event on the Future of Artificial Intelligence provided an interesting angle on “Trust in AI”. All speakers agreed that such trust is essential if AI is to achieve acceptance, and that (self-)regulatory frameworks can help to support it. However AI doesn’t stand alone: it depends on technical and organisational foundations. And if those aren’t already trusted it will be much harder – perhaps impossible – to create trust in any AI that is built on them. At the very least, a realistic assessment of how much trust we already have can inform how much of a “trust leap” the introduction of AI might involve.

The first layer is the context within which we work, or propose to act. Are organisations in that field generally trusted to behave responsibly, or are there concerns about hidden agendas and motivations? If you need to establish yourself as the only ethical actor in an unethical field, then do that first before you introduce further technological complexity, which may well be perceived and portrayed as suspicious opaqueness.

Next, since most AI systems will consume data, are our existing practices when handling and using data trusted? If we are seen as behaving responsibly in the ways we have humans collect, process and use data, then carefully introducing AI for the bulk data handling (thereby reducing the amount of access by human eyeballs) could even increase trust. Research by the Open University found that educational institutions generally are trusted by students to use data appropriately and ethically, but there are also stories of students creating their own data segregation , because they were not sufficiently confident of the university’s. We must be careful that introducing AI contributes to the former, rather than the latter.

If this analysis has identified a “trust gap”, that needn’t mean avoiding AI entirely. But while working to strengthen the trust foundations, it will probably be best to stick to low-risk applications of AI, rather than over-loading what trust we have. Interestingly, risk involves the same three factors – technology, data and context. Here, again, the context or purpose and the data we use may well be at least as important than the technology. Natural language processing involves some of the most complex AI algorithms, but many of its useful applications – for example chatbots and subtitling – involve little risk. But an algorithm as simple as a linear regression may be high-risk if used in a context where it influences life-changing decisions, or to process sensitive data.

For the past twenty-five years I’ve tried to avoid saying “no”. Whether in website management, security or law, “have you thought of…?” seems much more fruitful. In the short term it lets us discuss alternatives, in the long term it encourages – or at least doesn’t discourage – the questioner to come back.

So it was depressing this week to see European data protection regulators apparently saying “no” to a wide range of trans-national cloud services and business arrangements . The context of the Schrems II case seems to have led them to focus solely on the risk to data protection from the US Government. Sadly, that’s far from the only threat to our privacy and ability to control what happens to our personal data.

Taking care of personal data is complicated, and needs discussion, rather than over-simple solutions. I’m hoping the UK regulator’s response to Schrems II will let me carry on having those discussions, rather than cut them short.

Heard in a recent AI conversation: “I’m worried about black boxes”. But observation suggests that’s not a hard and fast rule: we’re often entirely happy to stake our lives, and those of others, on systems we don’t understand; and we may worry even about those whose workings are fully public. So what’s going on?

Outside our houses , motor cars are probably the most dangerous thing that most of us interact with on a regular basis. But, unless you’re a specialist engineer, how a modern car works is almost certainly completely opaque. Indeed, the tone of articles when security vulnerabilities are discovered suggests we’d like it to stay that way: too much visibility inside the box may be more alarming than too little.

We may be reassured by the presence of expert examiners, with legal powers. Any car more than three years old shouldn’t be on the UK’s public roads if it hasn’t had an annual inspection . Systemic interference with that process – even on an issue not directly related to safely – did cause major public concern . But even those inspectors don’t routinely carry out “white box” inspections on our behalf: much of what they do is still limited to examining inputs and outputs, not how those are linked inside the system.

And full “white box” transparency probably isn’t satisfactory, either. It may even create what Edwards and Veale refer to as the “ transparency fallacy ”, overloading individuals with information without giving them any meaningful ability to act on it. The notorious trolley problem suggests that even if we could be told, in advance, exactly how a self-driving car would respond in every possible scenario, even non-engineers would want to know why those particular trade-offs and choices were made. Questions like “what’s your business model?”, “what training data, limitations and monitoring have you designed for?” (a recent ACM article pointed out that training on European and American roads may be poor preparation for those in the rest of the world ), even “why replace humans at all?”.

So it seems that the transparency, or otherwise, of the box may be less important than the transparency of the decision-making. Neither “black box” nor “white box” should be a way to escape accountability. Those who choose either model to develop or deploy should expect to have to explain and justify their choices.

I’d been musing on a post on how “Artificial Intelligence” can be an unhelpful metaphor. But the European Parliament’s ThinkTank has written a far better one , so read theirs…

“Algorithms” haven’t had the best press recently. So it’s been fascinating to hear from the ReEnTrust project, which actually started back in 2018, on Rebuilding and Enabling Trust in Algorithms. Their recent presentations have  looked at explanations, but not (mostly) the mathematical ones that are often the focus. Rather than trying to reverse engineer a neural network, they have been exploring the benefits of clear and coherent messaging about why a task was chosen for automation, what the business models and data flows are.

That chimes with an idea I’ve had for a while about “shared interests”. If the organisation using the algorithm shares my discomfort (or worse) when something goes wrong, then I’m much happier to rely on its judgement. If the relationship is adversarial, where the organisation benefits from things that make me uncomfortable, then I’m much more likely to demand detailed explanations, or simply use opt-outs and technology to obstruct data collection or reduce data quality. Sometimes that undoubtedly makes me a free-rider – receiving benefits of data processing without contributing to it – but that’s the fault of the organisation that failed to explain how its limits of acceptability aligned with mine. If you want me to be altruistic, you have to continually earn it.

And that idea of alignment leads on to another idea about how we relate to our own algorithms. If we want others to behave as if we have made a good choice, then we must behave that way ourselves. And that applies even when things go wrong. If our algorithm responds badly to unforeseen circumstances or exposes unpalatable facts then we, who chose it, must own its behaviour and accept the blame.

Or maybe we can do better? Many years ago when I started working in what’s now called “cyber-security” it was really hard to get organisations to talk about their incidents. It was assumed that security should be perfect and that any breach was evidence of failure. The first breach notification laws were explicitly intended to “ name-and-shame “. Now, we’re are a bit more mature: recognising that occasional breaches will happen, and what really matters is rapid detection and effective response. Claims that an organisation or product is unbreachable now cause me to lose trust: even if they have been lucky so far, it suggests that they will be unprepared when something does go wrong. What really builds trust in an organisation’s cyber-security is clear public explanations of what went wrong, what has been done to stop it happening again, and recognition that they aren’t the only (or even main) victim. That might be an interesting model for those working with algorithms, too.

So many “AI ethics frameworks” are crossing my browser nowadays that I’m only really keeping an eye out for things that I’ve not seen before. The Government’s new “ Ethics, Transparency and Accountability Framework for Automated Decision-Making ” has one of those: actively seeking out ways that an AI decision-making system can go wrong.

The terminology makes pretty clear that this is based on how we have been finding security vulnerabilities in software systems for many years: using “red teams” and “bounty schemes”.  But here the aim isn’t to find bugs in software that give access to what should be private data or systems, it’s to find situations where an AI decision-maker or support-system will make wrong, biased, discriminatory or harmful choices.

“Red-teaming” – in the sense of an internal activity to test the limits of systems – isn’t really new for AI. Practices such as ensuring test data sets are comprehensive and include examples that have caused problems in the past should be routine, especially given the availability of tools such as Facebook’s Casual Conversations . And there’s an active field of research exploring how adversarial modifications can make AI vision processors, in particular, mis-classify or mis-read what they see.

But the idea of a bounty scheme does seem relatively new (it turns out it was proposed in a paper in 2020). For software bugs, payments for making helpful reports were introduced into what was already a thriving security researcher scene. Since many security bugs could be exploited to make money, either by the researcher or by someone who paid them for their knowledge, the idea was to create a counter-incentive. If you discover a new bug, report it to the software vendor and help them fix it then you may receive a financial reward without the risk of being involved in criminality. That model has grown to the point where a software producer can subscribe to a bounty-as-a-service platform such as HackerOne ; a few researchers make enough for bounties to be their main source of income, but for most they seem to be a token of appreciation, a way to fund the next round of research, or simply to justify the time taken to make a quality report.

The context for a bias and safety bounty is a bit different. It’s less obvious how a criminal would make money from secret knowledge of an AI bias risk and – as far as I know – there isn’t the same community of hobbyists searching for such problems as in the heyday of bugtraq in the late 1990s. So perhaps the main focus of a bounty scheme is to create that community, with the signal it sends being as important as the payment: we want to know about bias problems, help us find them and we’ll show our gratitude in an appropriate way (which, the history of bug bounties suggests, could include T-shirts or public thanks, as well as money).

One thing that is common to both types of bounty is that the benefits depend heavily on how organisations respond to the reports they receive. It would be nice to think that – twenty years on – we won’t see a return to vendors threatening to sue researchers, and researchers threatening to “go public” with their findings. At the very least, work together to fix the problem that has been identified: ideally, take a step back, work out how the problem got into the system and fix the development process as well.

The European Commission has just published its draft Regulation on Artificial Intelligence (AI). While there’s no obligation for UK law to follow suit, the Regulation provides a helpful guide to risk from different applications of AI, and the sort of controls that might be required.

What “AI” is covered?

According to Article 3(1) [with sub-clauses split out and interpolating Annex I], it’s…

• software…
• developed with one or more of:
  • machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning;
  • logic-and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems;
  • statistical approaches, Bayesian estimation, search and optimization methods…
• that can, for a given set of human objectives…
• generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with.

That’s a huge improvement in precision over ‘definitions’ such as “a standard industry term for a range of technologies”. This one does seem like a reasonable basis for regulation: in particular you can imagine different people reaching the same conclusion on whether a given system was, or was not, “AI”. But it’s pretty broad.

So is it all alike?

No. The draft identifies four kinds of purpose that AI technology might be used for. Some purposes have minimal risk, and are not mentioned further. Some have low risk: here the main requirement is to make humans aware when they are being used. Some have high risk, and carry significant obligations for both suppliers and users. Some are unacceptable, and prohibited.

Low risk AI

According to Article 52, there is some risk whenever AI interacts with a human (e.g. chatbots); is used to recognise emotion or assign categories such as age, hair colour, sex, or ethnic origin; or to produce images, audio, video that might appear authentic or truthful (“deep fakes”). Humans must be informed when any of these are used. The introductory text says this lets people “make meaningful choices or step back”: although there’s no right to “step back” in this regulation, that may well arise from the rules for processing personal or special category data under the GDPR.

High-risk AI

These purposes are listed in Annex III, together with any use of AI as part of a safety mechanism in regulated products. Specific to education, paragraph 3 lists:

• AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and vocational training institutions; [and]
• AI systems intended to be used for the purpose of assessing students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to educational institutions.

These uses are considered high-risk even if AI supports a human decision-maker, thus representing a considerable extension of the GDPR Article 22 provisions on Automated Decision Making.

Other high-risk uses that may be relevant include remote biometric identification and categorisation of natural persons (para 1); recruitment or selection of employment candidates, promotion, termination, task allocation and monitoring of employees (para 4).

Using AI for these purposes may be permitted – subject to prior registration (Art.16) and conformity checks (Art.19) – but there are significant and continuing obligations on both suppliers and users. Suppliers must, for example: continually manage risk from both normal operation and foreseeable misuse (Art.9); comply with requirements on training data (Art.10), technical documentation (Art.11), and provision of logging facilities (Art.12); ensure accuracy, robustness and security, including against feedback loops, data poisoning and adversarial examples (Art.15). They must inform users (i.e. organisations): how to interpret the system’s output and use it appropriately; of situations that may lead to risks to health, safety or fundamental rights; of groups of people who the system is, and is not, designed to be used on; the expected lifetime of the system and ongoing maintenance measures (Art.13). Suppliers and users must work together to: ensure effective human oversight; understand the system’s capacities and limitations; monitor for anomalies, disfunction and unexpected performance (Art.14). Users must keep logs, monitor performance; they must stop using the system and inform the supplier if there is any serious incident or malfunction or if operation presents an unexpected risk (Art.29).

Prohibited AI

Some uses of AI are considered unacceptable (Art.5) because they contravene EU values, such as fundamental rights. These include subliminal manipulation and exploiting individual vulnerabilities so as to distort behaviour and cause harm to them or others; AI-based “social scoring” by public authorities. Real-time remote biometric identification in spaces accessible to the public (which is high-risk in any case) is unacceptable for law enforcement purposes except in certain defined circumstances.

A third way?

Some commentators have seen the draft Regulation as a “third way?” to regulate AI: between free market and complete control. Others have focussed on the bureaucracy required for high-risk applications, or the exemptions for law enforcement. To me, the most interesting thing is how it works together with the GDPR, in particular the Accountability principle. Both require organisations to think carefully about risks to individuals before implementing new uses of data and technology. This AI Regulation actually provides more detailed guidance on those risks. Having heard, just last week, that “most ‘AI ethics’ questions turn out to be ‘data ethics’ ones”, drawing those two strands of closer can only be helpful.

Over the past twenty years, I’ve seen a lot of attempts to start information sharing schemes. And a lot of those have failed, some very slowly, despite huge amounts of effort. I wondered if there pointers that could be used, early on, to try to spot those.

Story

First, what is the story? If you want to receive information you should already have a story about how it will help. But what’s the story for those who will provide the information? What do they get out of it, and why is that something worth doing? The best stories are where sharing will address a common problem, but there are other kinds. Sharing may address a different problem for the two sides, or save money, or just improve their reputation. But there must be a significant overlap between the donor and recipient stories, where sharing looks like a win-win. And reputation matters. A long time ago I was asked if schools could provide the data source for an age verification scheme. Could, maybe: would, no. There was no story in which helping pupils to buy alcohol and tobacco, even if lawfully, looked good.

Scale

Second, scale. It’s rare for an information sharing scheme to be beneficial for the first few participants: bilateral agreements would be simpler if that was all they wanted. But what really matters how many participants you need before you start seeing benefits from choosing to do more than that. If it’s a handful on each side, you can go and make your case individually; if it’s half the community, you need a compelling case so they can decide for themselves to do it, plus excellent instructions so they can join without individual help. If the majority need to be sharing before anyone benefits, be prepared to provide long term support and encouragement. Network effects – when each new participant brings value to everyone – are critical for a scheme to become self-motivating, so you need to know when those can be expected.

Safeguards

Putting together stories and scale. At some point in a growing sharing scheme, some participants will use or provide information in ways that weren’t anticipated. What does that do the stories – both for donors and recipients? Sometimes it will make them better, by decreasing risk and/or increasing value; sometimes it will only affect that participant; but sometimes it will undermine the story for everyone else. How will you assess that, and how will you respond to an undermining action? If necessary, can you exclude – either temporarily or permanently – the misbehaving party? Or will their exclusion (and their likely response to it) do more damage than the misbehaviour? Ironically, successful sharing schemes may run the greatest risks: if participation becomes essential then both misbehaviour and exclusion – if there is no other option – may be highly damaging for everyone.

Thinking about these shouldn’t take long. Indeed that’s much of the point. If you, as the proponent of the scheme, can’t quickly see and explain how it will work for everyone, then you should expect to be spending a long time doing so, perhaps unsuccessfully.

One striking aspect of the new Ethical Framework for AI in Education is how little of it is actually about AI technology. The Framework has nine objectives and 33 criteria: 18 of these apply to the ‘pre-procurement’ stage, and another five to ‘monitoring and evaluation’.
That’s a refreshing change from the usual technology-led discussions in this space: here it’s almost all about the organisation within which the AI will work. Do we understand our goal in choosing to use AI, is there a sound educational basis for that, what changes will this involve for processes and skills, do we understand the risks, how can we detect and change course if it doesn’t work out? And, equally important, does our supplier understand what we are trying to achieve and commit to supporting our choice of goals and assessment of risks?

Even the seven ‘implementation’ criteria are about process: how can AI be used in assessment to demonstrate skills and support well-being; how can we create safe spaces outside continuous assessment; how can AI help us avoid unfavourable outcomes for individuals; how will we help all stakeholders (students and staff) work effectively and ethically with AI; how will we manage the changes that introducing it should bring?

With this comprehensive understanding of the context we want AI to support and enhance, the actual technology choice should be much simpler. Some technologies (maybe even some applications) will be clearly unsuitable: others will be a good, or perfect, fit. Best of all, we’ll be able to provide the most important explanation for trustworthy AI: why we chose to use it.

To improve websites and other online services, measuring how they are used is a key tool. However the law on measuring visitors to websites is a mess. Nine years ago, when reviewing the types of cookies that do not need consent , the Article 29 Working Party of data protection regulators concluded that requiring consent when sites measure their own audiences was a major source of “consent fatigue”. A law to fix this was proposed in 2017 but has been stuck in debate in the European Council of Ministers ever since. While there has recently been some progress , Brexit means there is no guarantee that the UK will follow the result. Meanwhile the entry into force of the GDPR – in both the EU and UK – made that consent requirement even more onerous for both websites and their visitors. Last year the UK’s Information Commissioner said that enforcing this aspect of law was “ unlikely to be prioritised ”, but that could change at any time. If it does, regulators already have automated “ cookie sweep ” tools that would make widespread enforcement straightforward.

The law currently distinguishes between two (in future maybe three) different groups of technologies:

• Multi-site measurement cookies, tracking pixels, fingerprinting, etc . All technologies that store or access data from user devices require prior opt-in consent (unless they are solely for an exempt purpose ) under the ePrivacy Directive. Using such data also requires a justification under the GDPR, and doubts have been cast whether this is possible given the close link between multi-site measurement, user profiling and targeted advertising.
• Single-site measurement cookies, etc . These technologies currently require opt-in consent under the ePrivacy Directive, however the proposed ePrivacy Regulation would make at least some of them exempt from this requirement.
• Logfiles . It is possible to measure website traffic using the logfiles routinely collected by the server. This is outside the scope of the ePrivacy Directive, and may be considered a Legitimate Interest under the GDPR. However, using general logfiles for this purpose may well be more intrusive than a tool that is specifically designed to produce audience statistics, so this approach is likely to require more in the way of safeguards.

It’s worth noting that all audience measurements are inaccurate for both technical and social reasons. Many people use cookie and script blockers that reduce the numbers recorded by those technologies. Changes to the default behaviour of popular browsers can also have a significant impact. On the other hand logfiles contain records of visits by search engines and other web mapping tools, so are likely to over-report. Adding or changing consent banners is also likely to change user behaviour, either reducing the number granting consent, or increasing the number willing to trust that the technology is beneficial.

There’s no obvious “right answer” to how to do audience measurement, either in law or technology. Changing involves a trade-off. Moving to a different technology on your own schedule should allow you to determine the effect of the change in measurement, and how to compare figures from before and after. Waiting until a regulator, legislator, technology firm or public sentiment forces the change may give more consistency in the short term, but a greater risk of an irreconcilable break when the change has to be made.

My Digifest talk yesterday developed a couple of ideas on how we might move Towards Ethical AI, at least as that is defined by the EU High-Level Experts Group .

First is that three of the HLEG’s four Principles, and at least five of their seven Requirements, look strikingly similar to the requirements when processing personal data under the GDPR. That shouldn’t be a surprise: GDPR has long been recognised as more than just a “privacy law”. But it does suggest that applying GDPR concepts and guidance – Accountability/Transparency, Purpose, Fairness/Accuracy/Impact, Minimisation/Security – even when we aren’t processing personal data may help us to be perceived as behaving ethically.

That leaves three areas:

• Respect for Autonomy: which the HLEG explain as making sure that AI augments, complements, and empowers humans. In Q&A I borrowed from Priya Lakhani’s keynote to explain that as recognising that “Artificial Intelligence” and “Human Intelligence” aren’t, and shouldn’t try to be, the same. Autonomy, then, is about finding the proper roles for AI and HI, such that they can complement each other;
• Diversity/non-Discrimination: in its impact on groups and society (GDPR should already have taken care of the impact on individuals) and
• Societal/Environmental Wellbeing. As I realised earlier in the week , a significant amount of these two is actually “ethics”, it’s not specific to AI. Discriminating against groups (or individuals) is likely to be unethical, whether it’s done by a human, an AI or, come to that, Mechanical Turk.

So before getting into questions of “AI ethics” we should probably start by working out whether the actions are considering should be done at all. Here I suggested another quick tool: four questions to help explore new ideas for both feasibility and stakeholder reaction.

Since I make no claim to be an ethicist, I think this is about as far as I can take my journey “towards” Ethical AI. There remain two important questions:

• If we have concluded that a thing should be done, is there an ethical reason to distinguish between it being done by humans or being done by AI? Here there does seem to be a general feeling that high-stakes decisions should be taken by humans: perhaps with AI assistance, but not by machines alone. I was also reminded of an instance where students wanted gentle “you need to do more work” reminders to come from computers and not their tutors, concerned that if tutors knew how much prompting they had needed, they might consciously or sub-consciously reflect this in their marks; and
• What do we do if two ethical principles conflict? It’s easy to imagine (see many of Isaac Asimov’s Laws of Robotics stories!) scenarios where Respect for Autonomy might conflict with Prevention of Harm. And I suspect the same applies to most, or all, of the other pairs.

For those I think you do need an ethicist. Or, at the very least, a representative and thoughtful group of stakeholders.

The good news is that I think we can wait for that. Treating the appearance of either of those questions as a “no”, at least for now, doesn’t seem to limit the potential for using AI in education very much. There are still lots of applications waiting to be discovered, developed or delivered.

One of the trickiest questions I’m being asked at the moment is about “the ethics of Artificial Intelligence”. Not, I think, because it is necessarily a hard question, but because it’s so ill-defined. Indeed a couple of discussions at Digifest yesterday made me wonder whether it’s the simply the wrong question to start with.

First, on “chatbots”. These use AI – in the form of natural language processing – to provide an additional interface between students and digital data sources. Those may be static Frequently Asked Questions (“when is the library open?”), transactions (“renew my library book”) or complex queries across linked data sources (“where is my next lecture?”). Here the role of the AI is to work out what the student wants to do, translate that to the relevant back-end function and translate the result of that function back into natural language. In these sessions, ethics hardly featured: an interesting point was made that a chatbot should not replace skills – such as navigating and selecting from academic literature – that the student should be learning for themselves; and there was a question whether the right answer to a student trying to work at 3am should actually be “get some sleep”.

Second on the use of student data to provide support and guidance. Here the conversation was almost entirely about ethics: are our recommendations biased? when do predictions become pre-judgements? when do personalised nudges become unethical? if a student has chosen the wrong institution, it is ethical to try to keep them on our register, or should we help them find a better option?

What struck me is that none of these ethical questions change significantly if the actions are done by humans rather than AI. Discrimination is unethical, no matter who/what does it. So maybe they aren’t about “ethical AI” at all, but “ethical behaviour”? It may be that some of the behaviours aren’t actually possible without the use of computers to crunch statistics, so here we’re looking at “AI-enabled ethical questions”. Conversely if we make our AI explainable – which will almost always be a practical necessity in education, where we need to understand predictions if we are going to help students beat them – then AI may actually give us a better understanding of human bias: “AI-illuminated ethical questions”, perhaps.

My talk (“Towards Ethical AI”) on Thursday will sketch a map containing three different kinds of purpose: those that are ethical no matter who/what does them; those that are unethical no matter who/what does them; and those where the human/computer choice actually makes an ethical difference. True “ethics of AI” may only arise in that last group, and it’s much the smallest.

Priya Lakhani’s Digifest keynote was titled “How COVID-19 has catalysed edtech adoption” but actually ranged much more widely. What has the pandemic shown us about the role of technology in education and, indeed, how does that relate to education’s role in future society.

One obvious result of the pandemic is that we have (nearly) all got a lot more familiar with using technology. In some cases it has become second nature: we no longer qualify “chat” with the prefix “video-”, for example. Although we are all missing physical human contact, that deep engagement with technology presents an opportunity. In future we should always remember we have a choice of how to conduct meetings, and many other aspects of our lives. We should look out for opportunities for humans and machines to work together, each enhancing the other by adding its unique capabilities. In particular, we should remember that “artificial intelligence” and “human intelligence” are complementary, not competing. Human Intelligence will always be required of teachers, tutors and lecturers: Artificial Intelligence (AI) should help increase the amount of time they have to practise it.

For example AI can personalise and inform students: including taking a cross-curriculum view. It can analyse – not just identify – gaps in student knowledge and skills: is this student struggling in physics because they didn’t grasp a concept in maths? AI may be able to triage and suggest appropriate remedies: is this an individual issue, or a common one that we need to address in teaching materials as well as interventions? But it cannot analyse long-form essays, develop inter-personal skills, comfort a distressed student or celebrate with one who has “got” a tricky concept.

Getting this right isn’t just important for the school/college/university setting: it’s what we need to prepare students for in the workplace, too. Future employers should be looking for adaptive humans, who can continue to make productive use of whatever new technologies may develop. It’s common to talk about “life-long learning”, but maybe our concept starts too late? If you think of key habits and skills as “making things that work and making things work better”, then very young children already do that instinctively: might traditional education even be suppressing those ways of thinking – adapting, visualising, problem finding, problem solving, systems thinking – that we will need?

But – and hence my “nearly” above – we must ensure this opportunity is available to everyone. At the moment, something like 1 in 10 families does not have a laptop, desktop or tablet at all; in many more, learners have to share their devices. Software and connectivity also need to be sufficient for effective learning. And, though we often talk about post-pandemic “blended learning”, we must remember “blended teaching” as well. Tutors need new skills, too. In particular, if they are to work effectively with technology, they need to understand why it makes the suggestions it does, neither blindly accepting, nor blindly rejecting them. Technologists, deployers and managers all need to work together on this.

Ultimately, though, we must recognise that the pandemic has forced many of us to adopt a radically new way of making progress: trying something, failing or succeeding, and learning from both failure and success. Will we continue to do that, or will we revert to the safer ground of inertia, learning little or nothing?

Anyone who works with flows, logs and other sources of information to protect network and information security should already be familiar with Recital 49 of the GDPR, where European legislators explained why that was (subject to a risk-based design) a good thing. Now the European Commission has published its draft of the replacement Network and Information Security Directive (NIS2D), it’s interesting to see how that thinking has been refined. Comparing Recital 69 of NIS2D with Recital 49 of GDPR gives us an update of what, how and why the Commission think we should be doing to defend networks, systems, users and data.

Both start with exactly the same premise:

processing … for the purposes of ensuring network and information security … constitutes a legitimate interest of the data controller concerned

But, while GDPR moves straight to examples of what defenders “could” try to achieve:

preventing unauthorised access … and malicious code distribution and stopping ‘denial of service’ attacks and damage to computer and electronic communication systems

NIS2D starts with a much more complete description of the classical process for protecting systems that we “should” be following:

measures related to the prevention, detection, analysis and response to incidents

and recognises that this cannot be done by individuals or teams working alone:

to raise awareness in relation to specific cyber threats, exchange of information in the context of vulnerability remediation and coordinated disclosure … voluntary exchange of information on those incidents, cyber threats and vulnerabilities, IoCs, tactics, techniques and procedures, cybersecurity alerts and configuration tools

The change from “could” to “should” reflects the move by Data Protection regulators (in their Opinion on Breach Notification , and subsequent fines ) from viewing incident detection and response as something that is permitted to something that is required.

And the explicit recognition of the need for defenders to share information is very welcome, as this was one area where there remained some nervousness about whether GDPR might require European teams to reduce their sharing activities. Here we have a very clear statement to the contrary: information sharing “should” be happening, and the legal framework for it is the same legitimate interests basis as for our internal system defence activities.

Finally, there’s an interesting shift between the two laws in why network and information security matters. GDPR’s examples – unauthorised access, malware and DoS – are incidents that harm individuals. But NIS2D Recital 3 adds a much broader perspective:

cyber incidents can impede the pursuit of economic activities …, generate financial losses, undermine user confidence and cause major damage to … economy and society

I think that may also signpost a clearer legal framework for international sharing than the current patchwork of relevant GDPR measures, but I’m still working on that idea.

Finally note that, although the main focus of the NIS2 Directive is National CSIRTs and Critical Infrastructures, Article 27 is explicit that “entities falling outside the scope” must be included in information sharing. Together with the exact repetition of Recital 49’s motivating sentence, that seems a clear justification for reading Recital 69 back to the full GDPR scope: i.e. – according to the Article 29 Working Party Opinion – all data controllers.

Early in 2021 I was invited to give a one-hour presentation on Data Protection and Incident Response, looking at how the demands of the two fields align and support each other, and how law and guidance have come to recognise that over the past decade or so.

Incident Response and GDPR: slides

Discussion at that event gave me a better way of thinking about information sharing

Information Sharing and NIS2D: slides

If you’d like to know more, look at the incident response tag on this blog, or read my peer-reviewed papers:

• Incident Response: Protecting Individual Rights Under the General Data Protection Regulation
• Processing Data to Protect Data: Resolving the Breach Detection Paradox

• NISD2: A Common Framework for Information Sharing Among Network Defenders

To celebrate my 500th blog post, here’s another sea shanty:

What shall we do with the stolen data?
What shall we do with the stolen data?
What shall we do with the stolen data?
Early in the morning.

Way-hey the fines are rising
Way-hey the fines are rising
Way-hey the fines are rising
Early in the morning.

Got to tell the DPA unless there’s no risk
Got to tell the DPA unless there’s no risk
Got to tell the DPA unless there’s no risk
Seventy-two hours!

Way-hey the fines are rising…

Thoroughly protected using strong encryption
Thoroughly protected using strong encryption
Thoroughly protected using strong encryption
And we have the keys safe!

Way-hey the fines are rising…

Quickly we detected it and mitigated
Quickly we detected it and mitigated
Quickly we detected it and mitigated
Hooray for the C-SIRT!

Way-hey the fines are rising…

Yes we trained the personnel in all departments
Yes we trained the personnel in all departments
Yes we trained the personnel in all departments
And we have the records!

Way-hey the fines are rising…

Technical and org’nisational our measures
Technical and org’nisational our measures
Technical and org’nisational our measures
Starting to feel better!

Way-hey the fines are rising…

May not be required to inform the users
May not be required to inform the users
May not be required to inform the users
Think if it’s good practice?

Way-hey the fines are rising…

Thoroughly investigated, found the root cause
Thoroughly investigated, found the root cause
Thoroughly investigated, found the root cause
No more early mornings!

Way-hey the fines are rising
Way-hey the fines are rising
Way-hey the fines are rising
Learn from this and prosper!!

[ Original by The Irish Rovers ]

A fascinating discussion session with colleagues who worked on Jisc’s “ Future of Assessment ” report. When that was written, in the first months of 2020, its intention was to look at how things might change over the next five years. Little did we know…

When the pandemic hit, suddenly many of things we had expected to happen by 2025 needed to be done by June. It was very quickly apparent that traditional exam halls were not going to be possible for the 2020 cycle, so there was a very rapid pivot to other ways of assessing students’ abilities. And that was amazingly successful. As I commented: the Future came 57 months early!

So now we know that it is possible to do assessments under lockdown conditions, can we use that experience as an opportunity to think what the future of assessment might actually look like? Maybe getting back to normal in terms of travel, meetings and gatherings shouldn’t just mean reverting to the traditional forms of assessment?

We gradually coalesced around five questions:

What are we actually assessing, and is that what we want to assess? Knowledge, skill, ability to work under pressure, ability to write/type intensively for an extended period, short-term memory, long-term memory, digital wealth?

Who/which groups does that put at a disadvantage? And is the disadvantage something – like lack of computers, bandwidth or a quiet place for undisturbed assessment – which can be fixed by providing appropriate resources; or something inherently incompatible between the student and the assessment style?

Can we reduce the stress of being assessed? We should at least be wary of approaches that increase stress.

Can we make our exams more realistic? How many jobs actually require us to sit at a desk for three hours, with no access to external information resources?

What is malpractice? Can we adapt our style of assessment to make that ineffective or meaningless? Maybe a student who can use reference materials quickly to produce a well-argued response within the time-limit is actually demonstrating their knowledge of the subject? Having done both open-book and exam hall assessments during my law degrees, I felt the former tested my subject knowledge, the latter my exam technique. And have the words “digital” and “remote” triggered an excessive concern with on-the-spot “security”, overriding other important concerns? If we take a broader view of the context in which assessment takes place, there may be ways to detect cheating that don’t require us to discard the other priorities for effective assessment.

“Big Data” has – often rightly – had a bad press. Is there a better way to think about it? Starting from potential benefits and discussing how they might be achieved should help us choose the right outcomes to aim for when using data, make it more likely that those aims will be delivered, and build trust and confidence in our approach.

Inspired by Gavin Freeguard’s National Data Strategy Sea-Shanty , and in homage to the shanty-makers (I’ve worked the North Atlantic on small ships), here’s my “Adequacy Shanty”…

Farewell and adieu to you, fair Spanish data,
Farewell and adieu to you data of Spain,
For our UK law may be judg-ed inadequate,
And we may never see your fine data again.

The first of our problems is data retention
Ruled non-compliant in twenty sixteen;
Then a new issue with immigration processing
Oh how can we fathom just what that might mean?

Farewell and adieu…

We’ll rant and we’ll roar about model clauses
Or, where it’s appropriate, write BCRs.
For contracts essential and users consenting
We have to be grateful to our lucky stars

Farewell and adieu…

Perhaps the Commission will rule in our favour ,
And judge our position with some sympathy?
A problem postponed, or a permanent solution?
We wait for the outcome of Schrems number three!

Farewell and adieu…

[Original performed by The Longest Johns]

It seems easy to come up with new ways we might re-use data we already have. But harder to work out, in advance, whether an idea is likely to be perceived as unethical, intrusive, or just creepy. In a recent paper – “ Between the Devil and the Deep Blue Sea (of Data) ” – I explored how simple questions might help us look at ideas from different perspectives and identify the ones most likely to be accepted.

Before starting to implement anything – while the idea is at back-of-an-envelope stage – we can discuss it with stakeholders, to test how they are likely to respond. This should include at least those whose data will go into the proposed process, those who will do the processing, and those who will use the results. Some questions to frame that discussion:

• Will it help? Do we have a process/resources to act on any signals the data may contain?
• Will it work? Now we know what the process is, do the data contain the signals that process needs?
• Will it comfort? Or might it make people nervous, whether about the data we are collecting, how/why we are processing it, and who can see the data or results.
• Will it fly? Are the assumptions we are making valid, and will they stay that way? This may be particularly tricky when the aim of the process is to modify the environment within which the data are collected. Will the process have different effects, either in terms of traditional discrimination, or for groups with different digital footprints, because of different learning styles, different equipment, or different experiences?

If we still have a “warm feeling” about the idea, Data Protection law can provide further sanity checks, even if we don’t think we are processing personal data. For now, we are mostly looking at points to discuss – though the information gathered will be a great help if we do come back later to look formally at legal requirements. But if it’s hard to explain the answers to these high-level questions (based on the Information Commissioner’s Twelve Steps to Prepare for GDPR ), that’s probably a warning sign that the idea itself needs more work:

• Data Protection by Design/Impact assessments . What are the benefits and risks for individuals (not – at least not directly – the organization)? Are there ways we could reduce the risks?
• Information Lifecycle . How do we collect data, use it, do we need to involve others, when and how will we destroy it?
• Breach Notification . How will we know when something has gone wrong and limit the harm? How might our handling and reporting of breaches – to users, regulators or management – increase or decrease trust?
• Legal Basis . This may sound dry, but having a clear legal basis will make your proposal much easier, and more reassuring, to explain. It’s also the gateway to lots of useful guidance. Is it needed
  • to deliver an agreement you’ve made with the individual (e.g. processing bank details to pay salary),
  • to meet a legal requirement (informing the tax office about the salary),
  • to protect life (from an imminent threat),
  • to serve a public interest (something that’s within our lawful remit),
  • to serve a legitimate interest (whether ours, individuals’, third parties’). And are we sure that that benefit doesn’t come at too high a cost to the rights and freedoms of individuals?
  • Or, if you can’t fit it into any of the above, can you meet the high bar for giving students or staff a genuinely free and easily withdrawn choice?
• Privacy Notice . The content of the notice flows fairly automatically from the questions above, but how will you communicate a new use of data that you already have?
• Individual Rights Processes (information, subject access, objection, etc.). As with Privacy Notices, if these look hard to provide, then the proposal may need rethinking to reduce the risk of it being seen as unethical.

Discussing these questions early in the design stage doesn’t just test out whether the idea is a good one. It can also reveal opportunities to make it better. For example if there’s a spread of opinion as to whether it seems creepy, can we make it optional, or add in additional controls for those (users and stakeholders, as well as data subjects) who are uncomfortable with it? If not, maybe that’s another warning sign. If we can, then it’s much easier to do it early than late in the development process.

[Based on a presentation for the NISO Plus conference , February 22-25, 2021]

One thing it seems everyone knows about Europe is that we have a strong privacy law: the General Data Protection Regulation, or GDPR. In this talk I’d like to get you viewing that not just as a law, but as a really useful way to think about designing systems and processes. And maybe challenge a few myths along the way.

Here’s what the GDPR itself says it’s about:

This Regulation lays down rules relating to the protection of natural persons with regard to the processing of personal data and rules relating to the free movement of personal data.

You’ll hear a lot about the “rules relating to the protection of natural persons”, so I’m not going to talk much about that. What I’d like to focus on is the much less referenced “rules relating to the free movement of personal data”. GDPR is explicitly – in its very first Article – about helping the movement and use of data, so long as that’s done in a way that’s safe for individuals.

First Myth

So – my first myth – GDPR isn’t (primarily) about individuals, it’s about the organisations that handle their data. All the GDPR Principles are aimed at them. And those Principles are a really useful guide to designing safe products, services, and other activities. For example:

Accountability requires not only that organisations are compliant, but that they can show they are compliant. So we must think – before we start to use personal data – about the design of our systems and processes, safeguards against error and misuse, how we will operate them safely, and how we will ensure those plans actually happen. A key point is that the focus must be on the individuals and groups whose data we process, not on the organisation. And the GDPR provides a tool – the Data Protection Impact Assessment or DPIA – to guide that thinking. DPIAs are mandatory for large-scale and otherwise high-risk processing, but they are a really useful tool for thinking about smaller activities, too. And once you’ve done a DPIA, why not publish it to show your users and other stakeholders that you are taking care of their interests?

Another principle (both of law and design) is Purpose Limitation . This requires us to think clearly and precisely about why we are collecting and using personal data. Multiple purposes may be OK, but we have to be clear – in our own minds and in our documentation – what those are. “In case it comes in useful” isn’t a convincing purpose either for Regulators or for stakeholders. And, having set our purposes, we must avoid “creep” beyond them.

And once you have identified one or more purposes, you need to ensure that your organisation has a lawful basis for that purpose. Is it something you need to do in order to fulfil an agreement with the individual (for example to pay a salary, or deliver a service they have requested)? Or something you are required to do by law (telling the tax office about the salary)? Or (and we hope not to be in this situation) something that’s needed to save a life or prevent serious injury? Or something that is in the public interest – and where our organisation is best placed to do it – or something that is in the interests of the organisation itself, individuals or third parties it may work with? Each of these has its own conditions that our design must satisfy: in particular for public interest and legitimate interest we must balance our interests with those of the individuals whose data we propose to process. If it’s hard to meet those conditions, then you probably need to rethink either your design, or whether you should be doing this at all.

Second myth

GDPR isn’t about preventing processing, it’s about allowing processing that’s necessary. And “necessary” has a very specific meaning – that there’s no less intrusive way to achieve the purpose. So it forces us to think – again good design practice – about minimisation . How little data does the purpose need, how little processing, how little disclosure (both internally and externally), and how soon can we get rid of it?

GDPR and its guidance recognise lots of technologies as contributing to this: attributes (what someone is – student, staff, guest – is often more useful than who anyway); pseudonyms, which let us recognise a returning user, but not identify them; statistics, where we can achieve our purpose with counts, averages, and so on; roles that allow us to define and enforce policies; and federations, which we’ll come back to later.

Third myth

GDPR isn’t (mostly) about choice, it’s about notice.

With very few exceptions, people must be told the “natural consequences” of the situation they are in, or about to enter. Most of what you must tell them is the product of the thinking in the first two stages: who is processing their data, what processing you are doing (including the legal basis), why (including the purpose(s)), how long this will continue (and what happens to the data when it stops), who else (and where) may be involved, and how to exercise their rights over their data.

Sometimes – but far less often than is claimed – individuals will actually have a free choice whether or not to give you their data. But remember the five legal bases: if you are offering them a service, or required by law to process the data, or saving life, or serving a public or other interest, then their choice probably isn’t free. In those cases, this quote from Guy Singh-Watson is relevant:

Customers expected us to do the right thing on their behalf, not just give them the info to choose for themselves (arguably an abdication of corporate responsibility).

And Guy isn’t a data protection guru – he’s a farmer, who runs an environmentally responsible veg.box scheme. If he knows what corporate responsibility looks like, shouldn’t we try a bit harder?

Most often, I’d suggest, true consent will be appropriate when you’d like an individual to volunteer additional information, to get into a deeper relationship with you. Not to discover whether they want a relationship at all. If you can’t find a basis for that initial relationship among the first five bases, maybe re-think your plans.

So, thinking with GDPR helps us to meet the expectations of our users, customers and wider stakeholders.

• We reduce the flow of information;
• We increase the benefits we deliver from what we have;
• and, by doing that publicly, we can provide a basis for increasing confidence and trust.

How does that work in practice?

Federated Access Management

Let’s look first at how students get access to the content they need for their courses.

Historically, that was a two-party relationship, where the student had to set up a personal account with the content provider, containing lots of personal data. Most of which didn’t actually help the provider either to decide whether the student should have access (because it was self-declared) or to deal with problems if they misbehaved.

Thinking with the GDPR principles – and some smart technologists – we realised that inserting the student’s institution as a trusted third party produced a very different data flow. Now, the student requests access, the provider checks with the designated institution whether they are covered by the licence. The institution uses its existing relationship and data to strongly authenticate the student, associate them with the licence and undertake to deal with any misbehaviour. Everyone benefits under this Federated Access Management model.

Analytics

Or, thinking about Analytics . Institutions do stuff: whether teaching, providing support, or providing facilities. Data trails generated by students and staff in using those facilities can be analysed (as a compatible purpose) to work out how to improve them (an obvious legitimate interest, with the balancing test ensuring it’s done safely).

If additional information from the student would help, we can ask them to provide it, always being aware that they may refuse or lie. And, if there’s an opportunity for individual improvement, as well as system-wide, we can suggest that. Again, the student can refuse to follow the suggestion. Limiting consent to these last two stages means our analytics and improvements can be based on whole-cohort data, not self-selected. Students can be reassured that the institution has weighed the risks and benefits to them, and that their actions in donating data or acting on personalised suggestions are free and fully-informed. Again, everyone benefits.

In a chat at the DataMatters conference I was asked about the ethics of universities and colleges using social media providers to contact students. In breaking down that question, I think it illustrates a continuum: the more we interfere with individuals’ own choices of what and how to use, the more thinking we need to do beforehand to ensure they are protected from the consequences of our actions. So, for example:

• Putting adverts on the social media platforms where we expect to find future students has little effect on their autonomy. Though we do still need to look carefully at the data flows and processes that may result from them reading or responding to our adverts;
• Staff contacting students on social media, or even asking them for their contact details, need to be aware that students may consider those platforms to be personal social spaces, where intrusions from teachers (or parents) are not welcome;
• If we expect, or require, students or staff to sign up to particular social media platforms (or anything else) as part of their learning, teaching, assessment or research, then we bear the greatest responsibility for ensuring our choice of platforms, applications, etc. is appropriate and safe. And, where possible, provide alternatives: some may have deliberately chosen not to use a particular platform for very good reasons.

This feels like an extension/generalisation of the Data Protection rule that as you process more, or more sensitive, data, the level of prior thought (there called “accountability”) must get deeper. Indeed many of the data protection tools – in particular Data Protection Impact Assessments – may well apply to choice of social media platforms as well. So it’s well worth involving your Data Protection team, who can help you use those tools and may be able to suggest alternative approaches and settings.

And, particularly at the upper end of the continuum, we need to review our choices and requirements periodically in the light of changing circumstances. Changes in platforms’ policies and practices may make them less or more appropriate for our, and our students’, purposes.

The European Commission’s proposed update of the Network and Information Security Directive may revive discussions about access to WHOIS data. When a domain name is registered, contact details are typically requested for various purposes, including billing, administrative and technical questions. For most of the history of the DNS this ‘WHOIS’ data – including names, postal and email addresses – was published openly by domain name registries or registrars.

Although its quality is patchy, WHOIS became a frequent source for a much wider range of parties than had originally been envisaged, including law enforcement, incident responders, rightsholders and spammers. European privacy regulators expressed concern about this for most of the 21 ^st century, but it was only when the GDPR introduced the possibility of massive fines that many European registrars and registries changed their behaviour, often by stopping publishing, or even collecting, WHOIS data at all.

Recital 60 of the Commission’s proposal now notes that (I’ve added one set of parentheses for easier reading):

The availability and timely accessibility of these data to public authorities, including competent authorities under Union or national law for the prevention, investigation or prosecution of criminal offences, CERTs, CSIRTs, and (as regards the data of their clients) to providers of electronic communications networks and services and providers of cybersecurity technologies and services acting on behalf of those clients, is essential to prevent and combat Domain Name System abuse, in particular to prevent, detect and respond to cybersecurity incidents.

Recital 62, and Article 23, therefore call on registries and registrars

1. to publish WHOIS data that is not subject to GDPR, and
2. to make data that is subject to GDPR available to “legitimate access seekers”.

That raises a number of questions:

• Which WHOIS data are “not personal data” and required to be published by Art.23(4)? Although it is recommended practice to enter role-based contact details, which will survive the departure of an individual member of staff, that’s very hard for registrars to check or enforce.

And, for data that are personal:

• Which groups are “legitimate access seekers” who must be given access according to Art.23(5)?
• What is the “lawful, and duly justified” basis for that access? The NIS Directive repeatedly stresses that this must be in accordance with GDPR, and does not appear to be creating any new basis for the processing.
• How (technically, since Rec.62 suggests “the use of an interface, portal or other technical tool”) can those “access seekers” be authenticated, authorised and provided with access?

Back in 2018, I sketched out how, if registries and registrars wished to give access to CSIRTs/CERTs, they could use GDPR Recital 49 and Legitimate Interest as a GDPR lawful basis, and how membership organisations such as FIRST and TF-CSIRT might be able to provide the necessary authentication, authorisation and policy infrastructures to ensure this delivers data protection benefits, rather than risks, to domain registrants. Now that such access is being recognised as “essential … to prevent, detect and respond to cybersecurity incidents” (Rec.60) that approach may be worth revisiting.

The draft Directive – partly because it is a Directive – leaves open the question of who might do that. Member States have considerable flexibility whether to add their own detail, or simply pass on the Commission’s wording to registrars and registries within their jurisdiction. But there’s an interesting hint of a more coordinated approach right at the end of Recital 62: “With a view to promoting harmonised practices across the internal market, the Commission may adopt guidelines on such procedures without prejudice to the competences of the European Data Protection Board”.

We’ve all been trained how to spot phishing emails: check the sender address, hover over links to see where they go, etc. But that’s a lot of work and mental effort. And, given that most emails aren’t phish, almost all wasted. So can we do it better?

A fascinating paper by Rick Wash looked at how experts (in this case, university IT staff) do it. Yes, they use all those techniques, but only after something has flipped them into “look carefully” mode for a particular email. Most messages they classify instinctively, and correctly, as not-phish. So what is that something else: the pre-filter that leaves them relaxed about most emails?

It turns out that one academic definition of an expert is someone who is sensitive to the unusual. If that sensitivity is linked to a willingness to change your view of the world – from “this email is fine” to “this email is suspicious” – then that may be exactly the pre-filter we need. What we more often think of as “expertise” – deep understanding of a knowledge domain – may then be useful to assess which anomalies are probably harmless accidents, versus the ones that are likely to be created by a phisher. But it’s that initial sensitivity and willingness to abandon pre-conceptions that are key to optimising our mental workload.

If we can get better at paying attention when our instinct says “that colleague normally uses Slack, not email”, or “my bank tells me when I have a message, it doesn’t send it to me”, or “why did my friend write so formally?”, or “why didn’t the supplier give the order number”, or “why didn’t I hear about this by another route?” then we may be able to save ourselves a lot of conscious effort, without increasing the number of phish we fall victim to. Subsequent conscious inspection may reveal that the unusual feature in fact had a legitimate explanation: if we don’t get a few “false positive” triggers then we should try to increase our sensitivity. But we’ll still need to get into that mindset less often than every time “you have mail”.

One particularly interesting aspect of this is that it suggests that expert phish detectors don’t need any technical knowledge. If you have that then, of course, it means you can do the stage two conscious inspection yourself. But if you can become an expert in what your own email environment feels like, and develop expert-level sensitivity to when something doesn’t fit (maybe practice reviewing consciously why you felt uneasy about a particular mail), then dealing with your inbox should become a lot less stressful for you and for others. That’s something we should all  be able to manage.

During the pandemic, a lot of ideas have come up – not just contact tracing! – where useful information might be derived from location data. It struck me that a selection of those might be an interesting illustration of how intrusiveness isn’t just about the data we use, but what we use it for. Here’s a video…

https://www.youtube.com/watch?v=6HpARfQhW70

If you’d like to know more about “might”: how to assess whether an envelope sketch could actually work, that’s discussed in my new paper:

Between the Devil and the Deep Blue Sea (of data)

And if you’d like to do a more formal assessment of intrusiveness, try our

Intelligent Campus DPIA Toolkit

The latest report on ICO sandbox participation contains a rapid pivot, and some useful discussion of the “public interest” justification for processing. Back in mid-2019, NHS Digital was awarded a sandbox place for a system for recruiting volunteers into clinical trials (the actual conduct of trials is out of scope). A few months into 2020 that, like many of us, pivoted to respond specifically to the COVID-19 pandemic.

A particularly interesting feature of the resulting report is the discussion of lawful basis in paragraph 4.3. Even after NHS Digital had been required by law to set up the system – which might have been expected to trigger an Article 6(1)(c) Legal Obligation – the preferred basis for processing any particular volunteer’s data remains Art.6(1)(e) Public Interest. This provides a useful middle ground between mandatory participation and the hard-to-explain morass created by the different meanings of “consent” in research and data protection law. There’s a hint here of an old, pre-GDPR, framing, that “public interest” was what you used when you chose to help someone who had a legal obligation.

Another suggested benefit of using Public Interest is that, unlike Legal Obligation, it preserves the individual’s right to object to processing. This is certainly what Article 21 of the GDPR says, though the report doesn’t make clear what the effect of such an objection should be. Under Article 6(1)(f) Legitimate Interest, an objection requires the data controller to repeat the rights-balancing exercise, but applying the individual’s specific circumstances, rather than those of data subjects in general. But Article 6(1)(e) doesn’t have an initial rights-balancing test: it presumes that whatever legislator created the law will have taken relevant rights into account. Rather than trying to work out what those were, it might be simpler for a data controller to consider whether they have “compelling legitimate grounds” for continuing (some) processing, and/or need to keep the data in case of legal claims. Or simply treat any objection as a direct opt-out.

Finally, paragraph 4.8 makes an interesting point on describing benefits in privacy notices. Where someone is volunteering to help “the public interest”, it’s useful to break that interest down to different stakeholder groups. This feels right: if I’m being invited to be altruistic then the benefits to identifiable groups such as “frontline NHS staff” or “high-risk patients” may well be more persuasive than broad appeals to “health” or the “NHS”.

Tertiary educational institutions have a very specific role in promoting free speech, whether verbal, in writing or on-line. This is set out in general in the Education (No.2) Act 1986 , with specific limitations – monitored by the sector regulators – to manage the risk of radicalisation in the Counter-Terrorism and Security Act 2015 and, for Further Education, to safeguard students. Where researchers need to access material that would otherwise be unlawful, well-established sector-wide guidance explains how to work with law enforcement authorities to ensure this can be done safely.

The Government’s ongoing review of Online Harms appears to have concluded that this existing, sector-specific, regime should not be disrupted by imposing an overlapping regime designed primarily for social media platforms. According to paragraph 1.6 of the new White Paper , the future Online Harms legislation will have an exemption for:

Online services managed by educational institutions, where those institutions are already subject to sufficient safeguarding duties or expectations . This includes platforms used by teachers, students, parents and alumni to communicate and collaborate. This is to avoid unnecessarily adding to any online safeguarding regulatory or inspection frameworks (or similar processes) already in place.

That legislation is due next year but, to judge from this White Paper, it shouldn’t affect the careful processes by which institutions, sector bodies, regulators and legislators have thought about speech over many years.

[UPDATE] Recordings from the event are now available

David Clark of MIT is one of the best people to take a long view of the Internet: he has been working on it since the 1970s. So his suggestion – in a Weizenbaum Institute Symposium yesterday – that the 2020s may see as dramatic a change in Internet regulation as the 1990s is significant.

Before the 1990s, most Internet development had taken place in the public sector, in research and military organisations. Once the technology had been shown to work, and become usable (to some extent) by private individuals and organisations, government reduced its involvement and commercial organisations were allowed, indeed encouraged, to take more of a role. To facilitate this, regulation of the new medium was deliberately light touch: in both the USA and Europe, existing models of publisher liability were rejected as being too onerous for a developing commercial market.

There’s no question that that created an explosion of new ideas, services and possibilities: for Jisc’s recent 25 ^th anniversary a number of us “silverlocks” reflected on what we were doing with networks in 1994 (I was setting up Cardiff University’s first web server) and our younger interviewers were amazed how primitive it all sounded. But David’s sense is that Governments are now looking, increasingly unhappily, at the consequences of that decision.

Much of that is down to simple economics. The original vision of the Internet was a mesh of cooperating entities, providing distributed services. That’s OK, perhaps, for large research universities who are used to collaborating. But it’s a really hard model to sustain: it’s much easier to build a centralised encyclopedia or discussion group than a distributed one. There’s no need for entities to agree to define (and then, which may be even harder, implement) complex standards and protocols; nor to persuade users that it might be better not to head to the service where all their friends are; nor to explain to regulators why discussing future plans is positive for competition and not the early stages of a cartel. Centralised services are much easier to monetise: the Internet has lots of protocols for moving bytes, very few for moving pennies. Dominant players emerge naturally, it doesn’t need any evil intent.

But once you have dominant players, offering frictionless and essentially free exchange of information, they naturally become a focus for societal problems that are normally constrained by either friction or economics. The list of issues that Governments are being told to “do something” about is increasing: privacy, trust, use of platforms by malicious and adverse interests, societal and economic dependency on large platforms, anonymity, national security, concentration of power, erosion of democracy, taxation… And there are signs that policy-makers in many countries and regions are looking to respond to at least some of those demands.

But in the 1990s the Government stakeholders were relatively few and shared a common approach, and the commercial ones were small and fragmented. Now, by contrast, Governments have very diverse views on Internet regulatory policy: many even hold competing policies (for example on encryption) in different departments. The consistent view is now on the commercial side – at least among the large platforms. It now at least as common for platform choices to change Government policies (notably on COVID contact-tracing apps and link taxes ) as the other way around. As David concluded, we may well be at the start of a change in Internet regulation that is as significant as twenty-five years ago, but significantly slower and messier.

The European Data Protection Board (the gathering of all EU Data Protection Regulators) has now published its initial guidance on transfers out of the EEA following the Schrems II case. This recommends that exporting organisations follow a similar roadmap to the earlier one from the European Data Protection Supervisor (who regulates the EU institutions). In particular, it only applies where personal data physically leave the EEA. But the EDPB takes a significantly harder line than the EDPS where the receiving organisation is subject to the US FISA Section 702, or similar, rules. According to footnote 49 that’s: telecoms carriers, providers of electronic communications service, providers of remote computing service, any other communication service provider with access during transmission or storage. Whereas the EDPS suggested a risk-based approach to those – prioritising large-scale, complex processing chains and sensitive data – the EDPB make no such distinction. If “the power granted to public authorities of the recipient country to access the transferred data goes beyond what is necessary and proportionate in a democratic society”, then the exporter is responsible for ensuring adequate supplementary measures that protect against that power.

Annex 2 contains example scenarios where the EDPB consider that adequate supplementary measures might be possible, as well as those where they can see no such possibility. The former include (with paragraph numbers in parentheses):

• (79) storage of encrypted backups, where encryption and decryption keys remain within the EU;
• (80-83) transfer of effectively pseudonymised data, where keys are kept in the EU and the transferred data (along with any other information available to the receiving state authorities) cannot be used to identify or single out individuals (a high threshold!);
• (84) transit of encrypted data over the internet to a safe third country (though the requirements for “flawless implementation” and “backdoors … ruled out” will be challenging);
• (85) encrypted transfer to an organisation that is protected by law from disclosure powers. In the main text there is a reference to “children’s data”, which seems likely to be a reference to the US FERPA, so also includes at least some US universities;
• (86) split processing by multiple parties in different jurisdictions, such that neither party nor jurisdiction can access actual content.

However the two scenarios where the EDPB cannot envisage any measures giving adequate protection include the most common uses of Standard Contractual Clauses and Binding Corporate Rules, respectively:

• (88-89) “Transfer to cloud services providers or other processors which require access to data in the clear”; and
• (90-91) “Remote access to data for [shared] business purposes”, for example within a corporate group or joint economic activity.

The EDPB does concede that, although the requirement for data exporters to check foreign law and practice applies to all transfers that are protected by contractual safeguards (including Article 46 Model Clauses and Article 47 Binding Corporate Rules), it does not apply to transfers under Article 45 Adequacy Decisions or Article 49 Specific Derogations. The latter include “necessary for the performance of a contract” (often known as the hotel booking derogation), “compelling legitimate interests” (only for non-repetitive and small-scale transfers) and “explicit consent”, each with specific requirements that must be satisfied. For example “explicit consent” (Article 49(a)) can only be used after the data subject has “been informed of the possible risks” of the transfer, and agreed to them. This might work, for example, where a user requests out-of-hours support from a cloud provider (the EDPB explicitly states that non-EEA support access to data in an EEA-hosted cloud is an “export”), if consent took the form of offering the user a choice between immediate support from a jurisdiction without adequate protection or office hours support from within the EEA.

These Article 49 derogations were designed to be used only for occasional, ad hoc, transfers, with Articles 46 and 47 covering regular and larger scale ones. There seems to be a significant risk that the EDPB’s hard line on the routine provisions may have the counter-productive effect of forcing organisations to push the boundaries of the ad hoc ones, as has previously happened with necessary, consent and legitimate interests.

These are draft guidelines, so it’s possible there may be some relaxation in the final version. We also don’t know how individual national regulators will respond . But with the test for non-EEA legal systems being set so high ( higher than several EU member states could attain , according to the European Law Blog) the tendency does seem to be strongly towards “data localisation” where European personal data must remain within Europe.

This raises a particular problem for the UK, post-Brexit. Onward transfers are a particular focus of both the Schrems II judgment and the EDPB guidance. If the UK does not follow EDPB guidance on limiting transfers from the UK out of the EEA, then it increases the likelihood that that guidance will be applied to limit transfers to the UK from the EEA.

Dataguidance is reporting that the German presidency has produced its progress report on the last six months of discussions on the ePrivacy Regulation. Recall that this was supposed to come into force on the same day as the GDPR… And it seems that Member States still haven’t reached agreement on what purposes might justify a service provider processing communications metadata, or using the processing and storage capabilities of end-user devices (including storing and reading cookies).

There is a hint, at least, that the range of options may have narrowed slightly – with “broad support” for the idea that it should be less than simply adopting the Legitimate Interests basis provided for processing under the GDPR. But also a general feeling that the current detailed list of permitted options was “too restrictive towards innovation”. So far, so good…

But then there’s a suggestion from “a number of Member States” that discussions under the Portugese presidency in the first half of 2021 should start, not from the current German proposal, but from the one handed over by the Finns at the end of 2019. So I’m not sure whether we are going forwards or backwards…

It’s still common to hear stories where privacy is supposedly in conflict with other objectives. I’ve been writing for years about how that’s not the case in security or access management . This morning’s ICO webinar on Security and Data Minimisation in Artificial Intelligence came up with a counter-example in that field, too…

You might think that the more training data you give to a Machine Learning algorithm, the better it will get. Humans get better with more practice, after all. But that’s not typically how ML works. For a while, more training datapoints will indeed improve the algorithm’s ability to derive general patterns. But if you give it too much training data, then it will tend to simply memorise datapoints. This is called “ over-fitting ”, where the algorithm gets so good at explaining the points it has already seen that it actually becomes worse at explaining any new ones.

So it turns out that the GDPR’s data minimisation principle – that data shall be adequate, relevant and limited to what is necessary – is the right way to train a Machine Learning algorithm, too!

A recording of the webinar should be in the ICO’s YouTube channel soon.

Two talks at last week’s FIRST conference looked at how Artificial Intelligence might be used in incident response. In both cases, the use of AI improves user privacy directly – by reducing or eliminating the need for human responders to look at user data – and also indirectly, by producing faster detection and mitigation of security/privacy threats.

Both talks stretched my knowledge in fascinating ways, so apologies if anything in the following isn’t correct. I’ll add links to the recordings when they are publicly available…

CK Chen looked at more traditional approaches, giving an excellent walk-through of how to construct a Machine Learning pipeline with the characteristics needed to support human threat hunters. Human threat hunting approaches involve looking at lots of user activity data, and generate lots of false positives. Can Machine Learning do better? As source data, the demonstration pipeline uses Windows process creation and command line events, gathered during (a) normal operation and (b) simulated attacks using APT3 and Metasploit.

• Stage 0: training an algorithm using these two – good and bad – datasets still produces lots of errors. A few processes/commands only appear in a single dataset, but things such as ‘netstat’ and ‘whoami’ are in both. Whether these are good or bad depends on the context. So…
• Stage 1: noting that a human threat hunter will look for significant sequences of events, try grouping parent and child processes into clusters, then get a human to label those clusters as good or bad, and train the algorithm using the clusters. This gives much fewer false positives, but still needs significant effort and data inspection. So…
• Stage 2: noting that attacks are less common than normal behaviour, add an anomaly detection stage to the pipeline. Processes common in the ‘bad’ dataset that appear in anomalous clusters give a strong signal. But that same pattern applies to novel legitimate activity, so this algorithm tends to generate false positives that are hard to explain. So…
• Stage 3: noting that attackers probably have to start from one of a small number of vulnerable processes, look for abnormally densely connected clusters around those processes. Interestingly, this algorithm performs slightly less well, in terms of number of false positives, but it provides a storyline that is much easier for a human to interpret: here’s where they gained initial access, escalated privilege, exploited the vulnerability.

For an AI that is designed to work with a human – protecting the privacy of (most) legitimate activity from human eyeballs, and passing the rest on to a human investigator – easy comprehension is  more relevant (to both privacy and security) than highest numerical performance.

Holly Stewart, Anna Bertinger and Sharada Acharaya from Microsoft looked at approaches that go further, and take the human out of the loop (except when invited in) entirely. When millions of new spam and phishing attacks appear every day, and last no more than an hour, full automation is essential from a security, as well as a privacy, point of view. In each case, the aim is to build AI that can classify previously unseen threats, based on self-reported data. The first approach they described was data obfuscation: eliminating personal data that you don’t need. For example if someone reports malware hiding files in their Favourites directory, that information is sufficiently well structured that you can remove the username from the path, benefitting privacy without affecting either training or detection. Next is “eyes-off training” where human data scientists see the “shape” of submitted data (flows, volumes, etc) but only the AI sees content for both training and detection. This can work well, but raises a problem when investigating false positives and negatives, and understanding what the model is doing, since the investigator can’t look at live content to understand its behaviour. Finally, one of the first practical uses of homomorphic encryption I’ve seen. This is a class of encryption algorithms that preserve arithmetic operations, so adding or multiplying two encrypted values produces the encrypted version of the true sum or product. It turns out that it is (just) possible to write machine learning classifiers within these constraints. So, when inviting users to submit screenshots of phishing pages, features of the pages are extracted and encrypted on the local machine, the encrypted values are submitted to a classifier in the cloud, and the user gets back a “don’t go there” message, fast enough that it should prevent them being tempted. Seriously impressive, both from a privacy and security point of view!

Threat Intelligence is something of a perennial topic at FIRST conferences. Three presentations this year discussed how we can generate and consume information about cyber-threats more effectively.

First Martin Eian from Mnemonic described using (topological) graphs to represent threat information. Objects, such as domain names, IP addresses and malware samples are vertices in the graph. Facts about them are edges. So an edge of type “resolves to” would connect a domain name and an IP address. Databases that use this kind of structure are widely available and make it easy to explore threats by visualisation and pivoting. Access controls can be applied to edges, thus allowing different people and partners to access parts of the graph that are relevant to them. While individual threat reports tend to create islands in the graph, databases can suggest links between those islands, where what may be the same object appears in more than one island. To be most effective this requires those populating the graph to use strict vocabularies: if two things are the same we should endeavour to make them the same in the graph. But where things may be the same (for example multiple names for malware or threat actors) it’s better to keep the different names as different objects, linked by “alias” facts. Choosing a single canonical name turned out to be a bad idea as it makes it impossible to disentangle the graph if you later change your mind. These – in fact all facts – should be labelled with a measure of “confidence”, so we know how much reliance we can place on the conclusions we draw. Changes of mind should also be explicitly recorded: facts should be timestamped and never changed. Instead, new, later, perhaps different confidence, facts should be added to the graph. As well as recording how our thinking developed, the sequence of timestamped facts should itself reveal threats such as fast-flux DNS (a rapidly changing “resolves to” link) and the development of malware families. Such labelling may be more work for the creator of threat intelligence than simply copying machine-generated alerts into a file. But good threat intelligence should aim to be write-once, read-many. As Trey Darley (CERT.be) and Andras Iklody (CIRCL) pointed out, adding context is like putting comments in source code: it’s worth taking the time to help others, not to mention your future self.

Trey and Andras developed these ideas further. Ideally, our threat intelligence should be valuable to many different types of consumer, including Security Operations Centres, Internet Service Providers, incident responders, threat analysts, risk analysts, and decision-makers. The technical information for each group may be similar, but how they use it will be very different and highly dependent on the contextual information that surrounds it. SOCs want to know which alerts they already have, or can deploy, protection against and which novel ones need deeper investigation; ISPs, who control Internet access for thousands of users, cannot afford false positives, so need to know which data points are sufficiently robust to use in blocking rules; Incident Responders want to see how an incident developed, so they can look out for similar signs; Threat Analysts want to understand motivation, modus operandi, attacker infrastructure, and unknown attack vectors; Risk Analysts want to see patterns in attacks, sectors and geography; Decision-Makers want evidence to inform decisions on resource allocation, including which threat information feeds to continue paying for! Use existing protocols to indicate how information may be used (for example the Permissible Action Protocol) and whether it may be shared (Traffic Light Protocol), but be clear whether these apply to the whole report or only parts of it. The aim of sharing should be to help others protect themselves: if you have reports, scripts or configurations that helped you, consider whether you can pass these on, too. Although a lot of the discussion around information sharing has focused on machine-readable information, this highlights the need to connect this to human-readable information, too.

A tool for doing just that was presented in a wonderful – costumed! – talk by the Fujitsu team of Ryusuke Masuoka, Toshitaka Satomi and Koji Yamada. Their S-TIP platform creates a bridge between human and machine worlds, on both the input and output sides. Human sources – blogs, incident reports, social media posts and emails – are scanned for data such as IP addresses and domain names, bitcoin addresses, CVEs, malware hashes and threat group names. These are tagged so they can be linked to machine-readable alerts and Indicators of Compromise. Chatbots within the system can then add richer information and links. Using this combined information, human analysts can quickly determine what (if any) action may need to be taken. This, too, is automated: the system has one-click links to Jira (to create block requests), Slack (to share internally), MISP (for external sharing) etc. In each case the original human-readable context accompanies the machine-readable instructions.

Some security incidents need more than a technical solution. Two talks at this week’s FIRST conference looked at the importance of human factors, in crisis management and vulnerability handling.

Jaco Cloete looked at situations where a cyber-incident can become a business incident, causing reputational damage, social media fallout, loss of market share, regulatory fines, even a liquidity crisis. These need a Cyber Crisis Management Team (CCMT) to coordinate between stakeholder groups and internal teams, the latter including the CSIRT doing the technical investigation. The core of the CCMT should be the CISO, CIO, executive responsible for risk, and executive responsible for the affected business unit. Others who may be brought in – as required by the nature and progress of the incident – include legal counsel, insurance, social media, CSIRT, crisis management experts, contact centre, investor relations and forensics. Where the organisation does not have these functions in-house, it may need to engage external help. The CCMT needs to meet regularly, at least daily, in an appropriately secured and resourced “war room”, not dependent on any infrastructure that may have been compromised. Even organisations whose size does not justify maintaining such a facility permanently should have plans, policies and protocols in place to create and operate one on demand. The decision to invoke these steps (something else to document in the crisis plan) may be based on publicity; sensitive client data; market, industry, regulatory or operational impact. One major job of the CCMT is to facilitate all communications – both external and internal, though in a crisis the difference should not be relied upon. Key points for the communications strategy: regulators and clients/victims first; honesty/trust; show competence; regular updates; social media/microsite; facts, not speculation; right tone (clients are victims, not you – apologise to them); respect confidentiality.

Mark Stanislav looked at another situation where communications are critical: receiving reports of vulnerabilities in products or services. He suggested treating this as a customer support function, rather than a technical one, and using personas to help staff maintain appropriate communications with those reporting vulnerabilities. At a minimum, a persona should suggest the likely motivations, pain points, technical expertise and community visibility of the individual you are talking to. This may be more natural if those are set within an appropriate back-story, name and (stock) image. It occurs to me that a pre-defined persona might also provide a flag when someone is behaving “out of character” and may need particular care. The personas need to be realistic. Specifically for bug reporters, HackerOne have some very useful statistics on why people find bugs – is it a hobby, an educational project, a job, or did they find it accidentally in the course of their work (even I have done this!). This may well give clues about how they may react to both successful and unsuccessful engagements: will the story be told at a major conference, on a popular social media feed, in an essay, or forgotten in mutual embarrassment. But there are also more subtle signals to bear in mind: a professional researcher will expect (and be worth) individual treatment, but for a hobbyist with a fuzzing tool it’s actually more dangerous to leave the standard script and risk being seen as not delivering on what you promise. It may be tempting to respond to all calls with “that’s a really interesting bug”, but don’t do that unless you are in a position to follow through on the implied promise. People who choose to report bugs to those who can fix them, rather than sell them on the black market, are demonstrating good intent. But don’t be complacent. Failing to understand and respond to their reasons for doing so can quickly turn a friend into an enemy.

[Typing that reminded me of another possible source for real researcher personas, and an excellent read: Chris van’t Hof’s Helpful Hackers (e)Book ]

Ben Hawkes, from Google’s Project Zero , gave a fascinating keynote presentation on vulnerability disclosure policies at this week’s FIRST Conference. There is little disagreement about the aim of such policies: to ensure that discovering a vulnerability in software or hardware reduces/minimises the harm the vulnerability subsequently causes. And, to achieve that, there are only really three things a vulnerability discoverer can control: Who to tell, What to tell them, and When. So why have we been debating the answers since the Security Digest and Bugtraq lists of the late 1980s and early 1990s, still without reaching a conclusion?

Mostly it comes down to that word “harm”: how do we measure it, and how do we predict it.

First, different people and different organisations will have different measures of “harm”. Is a vulnerability that allows a million PCs to be used for cryptocurrency mining more or less harmful than one that allows a trade negotiator’s communications to be read by an adversary? Are we concerned about short- or long-term harms? Harms to individuals, organisations or societies? There probably is no single right answer to these questions, in which case the best that organisations processing vulnerabilities can do is to think what their answer is, and document it so others can understand their behaviour.

Where there does seem to be scope for improvement is in our collective ability to assess the likelihood of a particular vulnerability causing harm. This needs an assessment of how attackers who know about the security weakness are likely to use that knowledge: before, during and after our discovery and disclosure process. How many of our vulnerabilities were already known and being silently exploited? If we discover it, how long before an attacker does? How much skill do they need to exploit it? How are they motivated to use it? Will our disclosure significantly change the situation?

Here we are dealing with incomplete information. Attackers have strong incentives to conceal their actions and reasons from us, maybe even to actively mislead. The ones we do find out about are the “failures”: the attackers who got caught. We may be able to learn a little from these, but it’s the successful ones we really need to be able to predict. I was reminded of the “ bullet-hole misconception ”, though here we are observing only the attackers that failed to escape, rather than only the planes that succeeded in doing so.

Statistical efforts to fill in this information gap include FIRST’s Exploit Prediction Scoring System SIG . But Ben suggested another angle on this might be to look at ourselves: ask experts to predict which recent vulnerabilities will cause significant harm, record their predictions, later test those predictions against what actually happens, and then try to understand how the more successful experts do it?

The ICO’s latest notice of a Monetary Penalty Notice , on Ticketmaster, contains unusually detailed guidance on the good practice they expect transactional websites to adopt. Although the particular breach concerned credit card data, this seems likely to apply to any site that takes customer data or that uses third party components. The whole notice is worth reading but, since it’s 73 pages long, here are the key points I spotted.

Simplify payment pages , or any others where you collect customer data. The breach occurred due to the insertion of malicious code via a chatbot. The ICO considers (6.13 – 6.17)  that the risk of  including third party JavaScript code was well known at the time of the breach in February 2018 and (6.18 – 6.20) that pages that accepted credit card details were particularly likely to be targeted. Removing the code from that page would have reduced the attack surface.

Use a layered approach to security, so you are not reliant on any single factor. In particular:

Don’t rely solely on third party certifications (6.22). Ensure contracts for third party code are clear where such products should and should not be used, and make sure security checks occur at a frequency appropriate to the speed of development of the threat (6.22.2).

When using third party written or hosted code, consider the technical measures you might deploy yourself , both in design and operation, in case any security issues arise in that code (6.24). These might include isolating the code from the rest of the page and applying security protections to the communications between the code and the rest of the site. And…

Relevant to both third-party and in-house code. Do your own checks , such as monitoring for unexpected changes (6.12), performing test transactions (6.22.5) and examining network traffic (7.13).

When these checks, or external reports (see 3.3 – 3.26), identify a problem, involve an incident response team promptly , and give them full information. It appears (7.12) that Ticketmaster’s response was delayed by focussing on a particular event and PC operating system when the problem was actually on the website and could have affected all bookings made through it.

The latest reports from the ICO sandbox provide important clarification of how data protection law applies to, and can guide, the application of novel technologies. This post looks at information sharing…

FutureFlow’s Transaction Monitoring and Forensic Analysis Platform lets financial institutions such as banks upload pseudonymised transaction data to a common platform where they, regulators and other agencies can look for patterns across the combined data set to detect and investigate “unusual behaviours and transaction patterns” that may indicate financial crime. This looks a lot like the information sharing platforms used by computer security incident response teams (CSIRTs) so it’s good to see that the sandbox report largely supports the legal model ( explained in detail by the MISP project ) that those have been using.

First, although uploaded data are typically pseudonyms – in FutureFlow’s case through standardised hashing of identifying data, in CSIRTs because most data are associated with pseudonyms such as IP addresses – they should not be treated as anonymous. Pseudonymisation reduces the identifiability of data but these datasets are sufficiently rich that a “motivated intruder” might still be able to identify individuals if they were to gain access. Data Protection law applies.

The first question is therefore which parties are Data Controllers and which Data Processors. Institutions that upload data into the platform (and presumably those that access and download it, if different) are data controllers, since they decide which data to upload and what purposes (within the platform’s technical and policy limits) to use it for. Where, as in in FutureFlow’s case, the platform operator does neither, it is likely to be a Data Processor. This suggests that in federated sharing platforms, where contributors can run their own instance of the platform and link it to others, the contributor function would dominate and those organisations would be Data Controllers. But a platform operator – like FutureFlow – that merely develops algorithms and runs them on others’ data may be a Data Processor.

Interestingly, although GDPR Article 35 only makes Data Protection Impact Assessments (DPIAs) a requirement for Data Controllers, the sandbox report suggests that a DPIA might be a good way for a Data Processor to document the risks it has considered and the security measures it has adopted to manage them (this is the approach that Jisc has taken with its Learning Analytics platform DPIA ).

Using a platform to share personal data requires a legal basis, chosen from GDPR Article 6. Although financial institutions may have a legal obligation to prevent fraud, the FutureFlow report suggests that “necessary for legal obligation” (Article 6(1)(c)) is probably not the best choice. This is because the sharing platform “demonstrates maximum effectiveness when applied to a broad account base, prior to any firm indication that any accounts have been involved in suspicious activity”. Or, as we tend to express it in incident response, to identify malicious anomalies you need to know what normal looks like. At this “pre-suspicion” state, “necessary for a legitimate interest” (Article 6(1)(f)) is more appropriate, and brings the additional reassurance that each Data Controller must ensure its use of data is not just legitimate (for computer and network security this is likely to invoke GDPR Recital 49), but is not over-ridden by the impact on the rights and freedoms of the individuals involved. The platform’s DPIA should be useful in performing this balancing test.

Finally, one difference between FutureFlow and CSIRTs sharing platforms is that FutureFlow only appears to let institutions see their own data, with automated flags added by the platform. Incident response – at least at present – typically relies on more manual investigation, with participants likely to have some access to data uploaded by others. To reassure contributors that this increased risk is mitigated, platforms and the communities that use them may need to supplement technical and operational security measures with policies and/or legal agreements that ensure uploaded data will only be used in ways and for purposes intended. Where the platform offers a DPIA, such measures should be included there.

The latest reports from the ICO sandbox provide important clarification of how data protection law applies to, and can guide, the application of novel technologies. This post looks at machine learning…

Onfido’s engagement looked at how to train and review the performance of machine learning models. In thinking about that I’d concluded that the GDPR provided more useful guidance if you thought of Training and Review as separate processes from actually Operating the model. In Onfido’s case that’s a legal necessity, because its model operates as software-as-a-service, with the customer as data controller and Onfido as data processor. Customers aren’t involved in Training and Review – though obviously they want those to happen – so Onfido must be the data controller for those steps.

When thinking about a single organisation doing Training, Operation and Review, I’d suggested that there should be a primary legal basis for Operation, with the GDPR “research” provisions used as compatible extensions of that to Training and Review. That provides strong safeguards, notably that any impact on individuals must be minimised, justified by the benefits, and that there must be no possibility of using the resulting data to make decisions about individuals. Since Onfido is a data processor for Operation, it needs a primary legal basis for Training and Review: the sandbox report suggests Article 6(1)(f) Legitimate Interests. That provides safeguards that – you guessed – any impact on individuals must be minimised, justified by the benefits, and that (now as part of impact minimisation) there must be no possibility of using the resulting data to make decisions about individuals.

The details of the Onfido service raise a couple of other interesting issues. It supports banks – particularly in COVID times when it may not be possible to go into a branch to open an account – by verifying that a photograph of a face (typically from an ID document presented by the applicant) matches a current selfie – “are these the same person?” – and has not been tampered with. Training and Review are done with pairs of historic images and, perhaps, information about the origin of the ID document, but no information that would identify the individual. It might be argued that this is not personal data at all. But if the same individual were later to apply for another account, then Onfido’s data processor function might handle identifying information about them, bringing the training and review processes within scope of GDPR as well.

If the face pairs might be personal data, then it’s likely that they count as “biometrics”, and so will be classed as Special Category Data under GDPR. But since it has been widely reported that many Machine Learning algorithms perform very differently with faces of different racial types, there is an Article 9 basis that fits snugly and, again, provides strong safeguards: the “substantial public interest” in reducing discrimination.

In summary: a textbook example of how, if you are trying to do the right thing, a detailed study of data protection law will usually be a strong and helpful guide, rather than a barrier.

The European Data Protection Supervisor (EDPS) has responded to the Schrems II judgment with a risk-based roadmap for EU institutions :

• Perform an inventory of all flows of personal data to entities outside the EU;
• Priority for change will be existing transfers with either no legal basis, those based on a derogation, and those to organisations “clearly subject” to the US FISA s702 or EO12333 laws and that involve large-scale or complex or sensitive data/processing;
• A strong precautionary principle should be applied to new contracts, with institutions strongly encouraged not to enter into any agreements that involve transferring data to the US;
• More detailed “Transfer Impact Assessment” questions will follow.

Since the EDPS works closely with national regulators within the European Data Protection Board (confusingly, the EDPB), we may well see those national regulators adopting a similar approach.

This morning’s discussion – jointly hosted by the All-Party Parliamentary Groups on Data Analytics and Health – suggested that if we want uses of health data to be trusted, we need to trust citizens and patients to think more deeply about benefits and risks than media headlines might suggest. The session was inspired by a recent Understanding Patient Data study and report which used citizen juries and a survey to discover public attitudes to data partnerships between the NHS and researchers, charities and businesses.

The results indicate that both citizens and patients are keen to engage, and can quickly develop sophisticated understandings of benefits and risks in data partnerships. If we give them the chance…

The framing of the study must have helped, by asking what would a fair partnership look like? This encouraged contributors to range widely in their interpretations: fair to individual patients whose data may be used; but also fair between the health service and its current and future partners (benefits should not be locked away to commercial advantage), fair to the population (benefits should not be localised, either geographically or by community), fair to past patients (who almost always want their experiences to benefit others), and fair to future patients (who should be able to benefit from the experiences of those who have been treated before).

But opportunities to benefit from partnerships are not limited to patient data: providing health services also generates large amounts of administrative and operational data, and a growing amount from sensors. For example data has already been used to route supplies of blood products, which have a very short shelf-life, to places where they are most likely to be needed.

Since the study was completed just before the onset of the COVID-19 pandemic, panellists considered what had changed. Public awareness of the role of data has definitely been raised, with both good and bad experiences being explored by the media in more detail than might have previously been the case. COVID-focussed research has revealed surprising, and important, gaps in data: 26% of Electronic Health Records (EHRs) omit ethnicity, making them hard to use to determine whether this is a factor in the severity of the disease; long COVID is poorly detected by any quantitative statistics, and EHRs are not a good vehicle for capturing the qualitative data where this form shows up. Responding rapidly to the onset of the pandemic meant that a number of partnerships were set up quickly: as has been suggested for education we should be reviewing the policies, infrastructures and norms that those emergency measures encoded to ensure they are appropriate to maintain public trust in the longer term.

All of which seems to make the study’s recommendations even more relevant. Health is such a sensitive issue that trust will always be fragile and need to be continually re-earned. This requires good governance, accountability and meaningful transparency (see below). There is an expectation that citizens will be involved in decision-making, both about uses of personal data and anonymised, and an appetite to understand how data can be used for public benefit. Since it should also provide a wider perspective to reveal unexpected consequences, early citizen engagement should be a win-win. Even if full public consultation is not possible in a rapidly-changing situation, having citizens/patients as regular members of boards, panels and juries provides many benefits.

Communications are likely to be critical to trust in any partnership, and need to be developed alongside technology and health, perhaps even before. This is a specialist task, particularly given the need to address many audiences, who may want a simple reassurance or a deep dive. We need to be ready to invest in expert support, especially when circumstances demand communication needs be done quickly. We must be open about benefits and risks: citizens have now seen sufficient unsuccessful data projects to presume the worst if the latter are not mentioned, or if information is not available. But we should also celebrate the successful partnerships that have been going on for many years, perhaps not wishing to draw attention to themselves for fear of adverse publicity.

Communication and support is also likely to be needed within the health and research communities: if expectations, safeguards and red lines are not explained and understood then cautious interpretation is likely to be much more of a limit than law, policy or governance actually require. We need to get better at explaining what rules such as the GDPR already allow, why and under what safeguards, rather than letting them be cited as barriers. Here the virus response may provide a model, with simple Information Governance guidance being issued quickly, and cross-skilled teams assembled to work together on technical, legal, governance and communications requirements.

It seems to me that much of this is applicable far beyond the health sector. Especially the closing line: “regulation and governance are not barriers to responsible innovation, they allow it to go faster: like brakes on cars”.

Jisc performs a number of different activities to keep Janet and customer sites secure. Here’s a very short video on how we used a Data Protection Impact Assessment and a Legitimate Interests Assessment to check that those activities do not themselves create disproportionate risks.

You can read the reports:

• Security Operations Centre (SOC) : Data Protection Impact Assessment
• Penetration Testing : Legitimate Interests Assessment

Since it has provided the foundation for most of the work I’ve been doing on data for the past couple of years, I’ve recorded a video explaining our standard model for “analytics”, in both practical and legal terms

If you’d like to know more, a couple of papers set out the theory

• Downstream Consent: a Better Legal Framework for Big Data
• A Data Protection Framework for Learning Analytics

The practice is reflected in Jisc’s Codes of Practice for

• Learning Analytics
• Wellbeing Analytics
• Intelligent Campus

And there’s a paper on the motivation and sources for the Wellbeing Code:

• Developing a Code of Practice for Using Data in Wellbeing Support

As well as in an ongoing series of blog posts, if you follow the topic tags below

After a couple of years when the question of data location had dropped a little down the priority list, two things have pushed it back up again. First, the Schrems II decision of the European Court, which cancelled the US-EU Privacy Shield and added some – but it’s not yet clear how onerous – new duties to those relying on Standard Contractual Clauses (SCCs). And second, the 31 ^st December deadline when the UK will leave the post-Brexit transition arrangements .

For organisations in the UK, those lead to uncertainties in two directions :

• Transfers of personal data from Europe to the UK might have to take place under anything from an adequacy decision to a US-like third country status;
• Transfers from the UK to the US can no longer be done under the Privacy Shield, and the Information Commissioner has yet to provide guidance on what kinds of data may still be transferred, to what organisations, under SCCs, with or without additional technical and US legal protection .

Furthermore, whatever the law or regulators say may be pre-empted by partners or individuals who are reluctant to have their data transferred to what they perceive to be less safe locations.

With so much uncertainty, it’s impossible to pick a single location and be sure it will be the “best” choice. But it may help to look at the things that, for each choice, could cause us to have to invoke “Plan B”:

• UK hosting : if data includes personal data about Europeans, could become problematic if UK and EU laws diverge significantly, since EU individuals, partners and regulators will expect their data to be held according to what might be non-local standards. As many large US companies have found, that can be an uncomfortable situation. If there is no adequacy decision, or if such a decision were to be successfully challenged in court, then transfers of personal data from EU organisations will need to be covered by Standard Contractual Clauses, and it’s not clear whether exporting organisations will be willing (or importing organisations able) to implement the additional checks required by Schrems II; it’s also worth checking how reliable a single-country cloud service is, as these are more likely to be subject to single points of failure, congestion, etc.
• EU/EEA hosting : could be disrupted if the hosting state’s regulator decided to consider the “fetching” of data to the UK as an export, but there has been no sign of this happening in the 25 years since it (arguably) became a theoretical legal possibility under the 1995 DP Directive; no Schrems II issue and, unless the UK Government changes its view that the EU will continue to provide equivalent protection to the UK , unlikely to be any Brexit one either;
• US hosting under SCCs : likely to be disrupted, for an unknown amount of high-risk data, when the ICO publishes guidance on Schrems II checks. Some transfers may be acceptable under an appropriate combination of technical controls (such as encrypted transfer) and type of receiving organisation (which will affect the protection available under US law): others may be judged too high risk to continue. Acceptability to individuals may not be the same as acceptability to regulators.
• US hosting under Privacy Shield : already illegal. Move to one of the above.

To ensure a lively discussion at a recent round-table on AI Ethics participants were asked, provocatively, “was the A Level algorithm fair?”. OK, I can be provoked…

It depends on what you mean by “fair”…

As has been widely discussed , the main objective set for those who designed the algorithm  seems to have been to reproduce the pattern of results that each school obtained in past years. In other words to be “fair” to previous years’ students, who can’t now be compared to a 2020 cohort whose different form of assessment might have resulted in a different pattern of marks.

What does not seem to have been a priority is “fairness” to this year’s students. They were mathematically unable to score better than their predecessors, even if their work might have indicated that they should. The range and distribution of marks within each school had to be the same as before, even if the level of achievement was different.

So, clearly, the definition of “fair” is something we need to discuss and decide on, long before we choose an algorithm, training data, etc.

A related question is whether technology is the “best” way to achieve a purpose. Here, again, thinking about what we mean by “best” can be very informative.

For example, are we trying to do something humans could do, but not at that scale? Or something that humans could do, but at greater cost? Or something that humans can’t do? Or something that humans could do, but not as consistently? As with “fair”, all may be valid choices, but they are likely to have different outcomes. Being clear about that from the start of a development should greatly reduce the risk of misunderstandings, mistakes and miscommunications later on: during development, deployment, operation, and review.

You may have spotted that I said “not as consistently”, rather than “not as fairly”. Creating, and sustaining, an AI that is more “fair” than the context within which it operates is really hard . There are just too many ways that existing unfairness can creep in: even the way we pose the question to be answered may contain implicit assumptions, training sets that do not contain a balanced representation of the population are a well-known issue, but what about unfairness in people’s ability to access the system or to act on its recommendations? That definitely doesn’t mean we should give up on trying to make our systems “fair”: doing so may actually be one of the best ways to highlight those real-world unfairnesses that society needs to address.

On and off, I’ve been researching the legal aspects of incident detection and response for fifteen years, and published more than 25000 words in law journals. So, can that be summarised in less than five minutes? You judge…

And if you’d like to read more, here are the original papers:

• Processing Data to Protect Data: resolving the breach detection paradox (SCRIPTed, 2020)
• Can CSIRTs Lawfully Scan for Vulnerabilities (SCRIPTed, 2014)
• Incident Response: Protecting Individual Rights Under the GDPR (SCRIPTed, 2016)

Here are the slides , and the script .

Alan Shark’s SOCITM ShareNational keynote looked at why regulation is not sufficient to deal with emerging technologies, and the complementary role that needs to be played by ethics.

Although privacy is not the only threat posed by such technologies, it does seem to be the one that has got people interested in the debate, whether over face recognition, tracking by apps, surveillance cameras, biometrics, smart “speakers” (actually, microphones) and deep fakes. It’s no longer just privacy activists who are worried about how much we give away to these and other applications, asking how long this is kept, where it is stored, who has access, who it might be shared with: who does, and who should, decide? The GDPR was called out as a leader in both its regulatory and ethical aspects.

Most technologies have the potential for both socially beneficial and socially harmful uses, so simple technology bans will have unintended effects: an Illinois ban on automated face-recognition makes it illegal to sell a robot dog that can recognise its owner. Whether this is an acceptable loss, or whether the law needs to be more nuanced, is an ethical question. Should we ban uses, rather than technologies? But then, how to define the uses: can regulation deal with the viral spread of conspiracy theories on social media and, if so, what levers should it apply? Again, a difficult, ethical question.

Although “AI” may be seen as the most urgent area to address, we need to break that term down. At least three categories can be identified:

• Systems that analyse large datasets to derive (relatively) static general-purpose insights or capabilities: much of “big data” falls into this category along with things like speech and image recognition;
• Systems that use individual data to inform human decision-makers. Perhaps “Augmented Intelligence” is a better term for these, since humans can – at least in principle – interrogate the system’s suggestion and ignore or override it;
• Systems that make autonomous decisions, learn from what happens, and modify themselves. Here there may be no way to either predict or understand the system’s decisions or – depending on the effects of the decisions that were assigned to it – to override or reverse those consequences.

In the first of those the main focus of ethics is probably on the choice of application and data (the answers may then lead to graduated regulation, as in the GDPR/ePrivacy Directive) and how systems are trained and tested; in the second on ensuring that humans are actually able to make appropriate decisions; in the third ethics may identify domains and kinds of decision where we simply do not want to develop or deploy such tools.

And, though privacy is a good starting point, our ethical thinking needs to range wider, on both the technological and consequence sides. What happens if inaccurate data are fed into new technologies? Are assumptions reliable (e.g. that blockchain is immutable, even in the presence of market dominance)? Uses have been proposed – some already implemented – that could lead to bad decision making; loss of life or destruction of property; loss of justice (even if “perfect” judges were possible, could society cope?); irreversible decisions; distrust of Government and technology. We need to consider those risks as well as the benefits that the technologies might deliver.

For these discussions to take place, ethics needs to be recognised as a complementary tool to regulation. At the moment technology ethics is increasingly being included in HE science curricula; making it part of policy and public administration ones as well might help increase its visibility. Ethical review boards – by government, region or discipline – might examine emerging technologies and suggest an appropriate balance of ethical and regulatory tools for each. Standards, testing processes and certification of ethical engagement (in the whole development process, not just the operation of technology) might inform markets and, where appropriate, legislators.

Allison Gardner’s keynote to the SOCTIM ShareNational conference last week highlighted how using AI responsibly is at least as much about how decisions are made as about the technology itself. Questions of “transparency” often focus on whether the AI is explainable, but how decisions were made – even how a particular problem was identified and chosen as appropriate for an AI solution – need at least as much transparency.

Humans are involved in many decisions: before, during and after the technology is put to work. How the question is framed, which data are made available to answer it, how those data are processed and cleaned, which features are selected, what weightings are given, which algorithms are chosen, what metrics are used to set objectives, how systems are deployed and how their results are evaluated are all human choices that will affect the results. So we need to understand what those decisions were, and why they were made.

Although AI supply chains may be complex – though hardly unique in this – the organisation that decides to deploy AI in a particular situation is responsible for what happens. They should be very wary of “black boxes”, which may conceal too much complexity or too little. A striking example of the latter is an “image analysis” tool that recommended treatment for hip fractures: on investigation it turned out that the patient’s age and whether the image was taken on a portable scanner carried much more weight than any feature of the X-ray. If data analysis finds that those are the key factors, fine, just tell humans that and avoid the expensive tech. Beware of explanations that simply assert “fair” or “accurate”: both have many different mathematical definitions, which are mutually exclusive: a system that is “fair” by one of them is very likely to be “unfair” by another measure. “Accuracy” figures can easily be improved by choosing the most common outcome. Make sure your system implements the one you actually need.

Interfaces are critical: they need to be clear, but avoid encouraging operators to over-rely on them. Finding a way to support operators and supervisors in exercising their professional judgement involves a very narrow line between causing them to doubt that judgement and, on the other hand, dismissing the machine as useless. Interestingly, this is an issue that goes back at least 500 years: the first printed books were quoted as “truth”, even though they were obviously internally inconsistent (for example using the same woodcut illustration for two different medicinal herbs to save money). Interfaces that seem to offer certainty can easily result in a “human in the loophole” situation, where decisions appear to be made by humans, but in practice they always follow the computer. Black boxes that don’t reveal the limits of their own skills are particularly dangerous.

Be realistic about bias. Automated systems will be biased, because they learn from a world that is biased. So make sure you have good processes to detect and correct it when (not if) it does emerge. More broadly, and excitingly: humans, and human-created systems, are biased; machines should be able to explain why they made particular decisions. So if we hold machines to higher standards of fairness, might we be able to use their explanations to learn about our own human biases?

Finally we need to think, and explain, what effects our introduction of AI may have on individuals and across the workforce. If decision-makers come to regard AI as the “expert”, does that limit the incentive, or possibility, to develop their own expertise? Or, perhaps even worse, could encoding current knowledge into an AI even limit the discovery of new knowledge across our domain? We need to find ways to get humans and machines working together – complementing each other – both at the level of individual decisions and in advancing our combined abilities.

Perhaps surprisingly – given that its title was “ Digital ethics ” – last week’s SOCITM panel session spent a lot of  time exploring things that aren’t “digital”. Although the discussion focussed on local government, a lot of the ideas seemed relevant to education, too.

Don’t be solutionist: technology might not be the right option.

When identifying issues, might a survey be more effective than using “big data”? It’s probably is less privacy invasive and – unlike using existing data – you can work to make sure the results represent all of the affected community. Even in universities and colleges, disparate access to – and confidence with – digital devices means that data from digital services will not be representative. Working out how unrepresentative after data are collected may well be more effort than gathering a representative sample in the first place. And engaging with affected people opens up discussion about the actual reasons for behaviour, rather than assuming those can be inferred from observations.

When proposing solutions, again, actively seek out and work with those who will be affected. Start grounded and simple, both in the aims and how they are communicated. Take as much time as is needed to explain and understand: time spent at this stage should be more than recouped later in the development. And don’t be simplistic. Be open about risks, and invite dissenting views. Reduce the number of unanticipated (by you) problems at this stage, not after you have implemented the system. Late-2020 is a good opportunity to escape the idea that technology is, or can be, perfect! Be honest about the objectives – cost reduction or staff redeployment may be OK, but don’t dress them up as something else. Know how progress against those goals will be tested, and what will be done if things don’t work out as hoped.

When implementing solutions (whether digital or not) ensure this is done by a multi-disciplinary team, not just statisticians and technologists. And make sure this is genuine engagement, not just a tickbox. Even if you don’t formally adopt Agile methods, try to test and learn during development, not just after. And capture and share the lessons of failures, not just successes: the former are at least as valuable. Be willing to learn from, and build on, other communities, tools, and resources: starting from scratch shouldn’t be necessary. Every process will need to include some facility for human contact, if only to detect, remedy and fix when “computer says no”. And be open about when it is a computer – Amsterdam’s new “ Algorithm Register ” is an interesting approach, though as it grows I suspect it will need to become more granular to avoid enforcement support systems getting lost in a swarm of chatbots.

Don’t (just) think “digital ethics”: think “ethical process change”.

Thinking through an idea that occurred to me during our SOCITM ShareNational panel on ethical use of data and technology.

What happens if we explicitly think about “our spaces, which people use”, rather than “people that use our spaces”? That may seem like a semantic quibble, but I think it leads in three interesting directions:

• First, space sensors are much simpler than people sensors. Even before lockdown, Jisc colleagues were looking at CO ₂ monitors and low-resolution infrared cameras. Now: might that be enough to work out when a room needs a period of ventilating to reduce the virus load or, longer term, which rooms we need to reduce effective capacity to prevent dangerous (or, in normal times, sleep-inducing) levels of human exhaust building up; might those fuzzy pictures be analysed to work out the gaps between warm bodies (and whether they are stationary or moving) and let us identify places where we need to improve traffic flow (a Microsoft video suggests how this might work)? Simple messages and low data rates might mean we can dispense with power and network cables, and just use battery and (low-power) wireless for sensors that can be quickly deployed to any location where there is a concern;
• Second, space sensors are much less of a privacy threat than people sensors. If we explain them – maybe even show examples of their output – then we should be able to calm fears of surveillance. Similarly with data, if we look for data about spaces, rather than people, we should be able to discard a lot of the identifiers that raise dataveillance concerns;
• And third, building on these two. Thinking about spaces seems to lead naturally to measures that involve improved design: thinking about people may be more likely to tempt us towards measures that involve enforcement.

Spaces are something we all use. Using sensors and data to improve them should benefit everyone. Maybe thinking about spaces can help build a sense of shared community, rather than a divide between campus managers and campus occupants?

When I was invited to join a panel at the SOCITM ShareNational event for local government I presumed my role was to provide a different, external, perspective on “Ethical Use of Emerging Technologies and Data”. So I offered to contribute a five-minute “sparkler” introduction: a bit of illumination, some striking of ideas, maybe a smile. In fact, the conference programme was already buzzing with new thinking (there will be lots more blog posts next week), so I didn’t need to add to that. Here’s what I would have said…

Over the past decade or so my job has been to work with Jisc colleagues to ensure the networking and analytics services they want to provide are “lawful”. More recently, they’ve been asking about “ethical”. But maybe what we actually need is “comforting”. That’s different, because it’s about how others perceive those services, not how we think about them.

We need to talk…

A couple of phrases I don’t find comforting. “For the Greater Good”. So why are you throwing me under a bus? Aren’t there sufficient ways to use technology and data that benefit everyone?

Perhaps more surprisingly. “Individual Control”. Why would that be important if the proposal is a no-brainer? Too often “individual control” is a sign of laziness – we can’t work it out, you do it. We should be working together to find those no-brainers; then use individual control to find and address situations that couldn’t have been foreseen, not as a belated acceptance test to discover we built the wrong thing.

So be careful about digital volunteering. It might help, but it might also amplify digital divides into visibility divides. I was shocked to discover that when I assume “students have smartphones” I’m actually missing 1 in 6 .

An example: I love the idea of an app that uses phone accelerometers to detect and report potholes . So cool! But… What about drivers who don’t have phones? What about cyclists, pedestrians and residents, who may welcome potholes as informal traffic-calming measures? Does that app actually identify desire lines and divert scarce resources to building smooth rat-runs?

I’ve been using four questions to explore new ideas for data and technology.

• Will it help? Do we have a process that would be improved by any signal the data might produce?
• Will it work? Will there be a sufficiently accurate signal, or am I relying on false assumptions about technology or behaviour?
• Will it comfort? Or will I be perceived as Big Brother?
• Will it fly? What are the broader effects on the community and society?

As a colleague observed, after exam results and virus testing, probably more people than ever before now have personal experience of the discomfort caused by inappropriate use of data. If we can use that engagement opportunity to move the discourse from “mutant algorithms” to pride in how our community uses technology then we’ll have salvaged something really valuable from 2020.

[UPDATE 8/1/21: the “four questions”, and how they might be used in practice to assess ideas for data (re)use, became the subject of a peer-reviewed paper – “ Between the Devil and the Deep Blue Sea (of Data) ” – published by the Journal of Law, Technology and Trust]

A panel on Algorithms at the UK IGF asked whether the summer of 2020 was a catastrophe – “mutant algorithm” having entered political discourse – or an opportunity to work with a population that is now much more aware of the personal significance of the debate ? “Transparency” is often cited as a remedy, but we now know that knowing how an algorithm works is far from sufficient: we need to know much more about how and why it came to be doing that job.

Viewing the “algorithm” space as multi-disciplinary, and building on existing work on (open) data, data (science) ethics, and governance/procurement can take us a long way. Indeed, a lot of what is perceived as an “AI Governance/Ethics” is actually data Governance and, as I’ve previously written, is already addressed in tools such as Data Protection Impact Assessments .

On Data , responsible use is the backbone: without trust there will be suspicion, with it there will come licence to innovate. But even perfect data wouldn’t guarantee a perfect outcome: we also need to look at behaviours, understandings, rules. Are we competent users of data, in the sense that we ask the right questions, apply emotional/social/political intelligence and domain knowledge? Are we working within trusted systems – institutions as well as algorithms, trusted as well as trustworthy – does the organisational culture encourage critical opinions, diverse viewpoints? Our processes and tools must be sustainable, in the sense that they respond appropriately to unforeseen situations. It was suggested that the A-level issue was so significant because those exams are so critical to social mobility: a societal issue, not a technological or legal one.

On Ethics , version 3 of the Government’s Data Ethics framework has just been released. This has three Principles – transparency, accountability and fairness – and five detailed Specific Actions – define and understand public benefit and user need, involve diverse expertise, comply with the law, review data quality and limitations, consider wider policy implications. The framework is likely to be used initially as a gateway for individual projects, but is designed to promote organisational change by developing skills, providing feedback and user stories. But, as with data, we need to prepare for imperfection. Algorithms will inevitably reflect existing societal biases: we need to be humble, to accept that, plan to mitigate harms and fix flaws. And be trusted to do so. Part of that is to ask whether data/apps are the right solution for a particular problem: might it be better to just talk?

On Governance inviting help from the public and experts is essential to picking the right problems and solutions, and to building trust in what we do. Here Open Government approaches – deliberation, participation, citizen assemblies, dialogues, citizen participation and feedback – are worth considering. We must achieve shared goals and understanding: applying data or technology to a problem that isn’t agreed or understood is likely to just amplify those disagreements. Definitions are critical: OFQUAL’s algorithm may have been “fair” across cohorts, but amplified social unfairness to individuals. Two practical tools were mentioned: AI Now’s Algorithmic Impact Assessment and AI Global’s AI Design Assistant . Governance must provide effective oversight of the whole life-cycle of a system: information gathering, monitoring/response, review/improvement. When procuring a system or service, ensure that your requirements for values and transparency match what the supplier is offering. Check that their Governance frameworks are compatible with yours. And be clear about the respective responsibilities of supplier and procurer/user.

Finally, a couple of thoughts on Explanation . There’s a useful distinction between “interpretable” systems, where a domain expert can understand what it is doing, versus “explainable” ones, where an individual can ask why it reached that conclusion for them [it occurs to me that I was trying to get at this distinction , without having the terminology, a couple of years ago]. Counter-factuals may be useful for the latter: what would need to change for the result to change? But such a “contrastive” explanation must also be “selective” (a limited number of things) and allow for a “social” dialogue asking what-if questions. And explanation can have a much deeper role: by holding algorithms to a higher standard than human decision-makers, we may learn about our own biases, too.

I’m pleased to announce the publication of our Intelligent Campus Data Protection Impact Assessment Toolkit .

Intelligent Campuses use existing data and new sensors to deliver better places to study, work, live and socialise. But there’s a risk with any use of data or sensors that even the best-intentioned ideas will be misused or misunderstood: as inappropriate, intrusive or even surveillance. Data Protection law suggests – may even require – conducting a Data Protection Impact Assessment (DPIA) as a way to understand risks to individuals, to explore less intrusive ways of achieving the objective (or conclude that it cannot be achieved with acceptable risk), and to implement appropriate safeguards. There are plenty of guides to how to conduct a DPIA, but it may be hard to work out how to apply these to a specific domain, especially when that is as novel as the intelligent campus.

Our toolkit aims to fill that gap, by providing domain-specific help on how to assess intrusiveness, the risks to consider, and the controls and mitigations that might help to reduce those to, and keep them at, an acceptable level. It was inspired by a DPIA template for RFID applications that was approved by European Regulators back in 2011, and informed by lots of presentations and conversations with many people . Thanks for all your inputs!

If you’d like to know more about the background, there’s a peer-reviewed paper – “ See no… Hear no… Track no… Ethics and the Intelligent Campus ”. But the toolkit should contain all you need to use it. I’d very much like to add a collection of experiences, case studies and similar supporting material, so please let me know how you are using it.

A panel on “Building Trust in a Digital Identity” at the UK IGF may have raised more questions than answers, but at least highlighted why doing so is taking so long. Since terminology can be confusing, what was being discussed was how to prove facts about your real-world self to an online service: for example to claim a furlough payment, or to gain access to an age-restricted site.

Why do we want to?

Those two examples immediately throw up the first challenge: why would we want to do this anyway? Digital identities are already used effectively to pay TV licences or tax a car; somewhat less so to pay income tax. But these Government services are things we have to do, not things we choose to. It’s hard to get excited about them, and a successful digital identity (eco)system needs customers to want to use it. Takeup of digital identity in services of choice – mostly in the private sector – has been much slower.

That may be partly a question of what “identity” is actually needed. Government services typically do need to know (at least indirectly) the name and address of “who” they are dealing with. Commercial services may be more interested in “what”: how old or – perhaps an unconscious reference to a Jisc service – whether it has a degree. It’s possible to combine the two functions, but not clear to me how much incentive there has been to do this. And, of course, the more data and functions a service offers, the harder it has to work to maintain trust that those will be protected against misuse. And, although the painful experience of high-quality account linking may be acceptable for a handful of Government services, would someone volunteer to go through that for a dozen or more commercial ones that they may only use occasionally? It would be nice to make the process easier, but that runs the risk of increased fraud.

Who (else) could do it?

Outside Government, the kinds of organisations that know enough about us to make a secure link between real world and online identities may not be the ones we (either as consumers or as service providers) want to rely on. Data brokers and others whose business model relies on data reuse probably aren’t the best foundation for a trusted identity service. Is there a risk that a socially-important function will be cross-subsidised by activities whose lawfulness is questionable; or, that those activities would be legitimised by the parallel use? We need to look very closely at business models and, at least, be prepared to pay enough (either directly, or via increased costs of services) for the identity functions to discourage providers from reusing data in the short term, and to remain in the market (since many service providers are going to rely on them) for the long term. The adtech industry was cited as a warning where both customers and services may be harmed by dependency on a “free” offering.

An opposite problem arises if we rely on device manufacturers to provide identity services. With control over everything from applications down to hardware, these can provide state-of-the-art privacy protection. Indeed one of the complaints about device support for contact tracing apps is that its privacy protection is better than public interests might wish. Here the problem – for both individuals, software developers, and nation states – is control. Who chooses what privacy-protecting identity services exist, and to whom they are made available?

Do we need a market, or a regulated infrastructure?

It’s possible to imagine a world where interoperable, standards-based, identity services compete for business (on functionality, privacy and cost, among other things). But such a market definitely isn’t where we are at present, and it’s not clear that it would ever be stable anyway. Identity services are intermediaries in multi-sided markets, and those gatekeeper functions have a strong economic tendency to tip to a dominant provider (video-conferencing, through which the conference took place, is a rare counter-example!). Should a government be comfortable with a market-dominant Identity service – which, if all goes well, will be a keystone of online society – that may decide to change or withdraw its offering for commercial or geopolitical reasons?

Or should nation states recognise that identity provision is a (inter-)national infrastructure and regulate it as such, for example to mandate access for SMEs? Although this might appear attractive, the history of regulating global tech companies suggests that only the largest states or groupings (e.g. the US or EU) have much chance of enforcing their will. And, while enforcement against economic monopolies has had some success, information monopolies seem much harder to control. Not least because the desired outcomes are much harder to identify and agree.

We must beware of mission creep and perverse incentives. COVID “passports” are a timely example. If employers insist that only proven-healthy workers can return to the office, does this create an incentive to intentionally catch the disease (currently the most likely way to demonstrate immunity)? When I was a child it was normal for parents to try to expose children to mumps, measles and chickenpox, because the likely consequences of infection later in life were much more serious! Or, if a disease turns out to be more prevalent in some communities, does such a requirement create indirect discrimination or reinforce deprivation?

And finally, remember the “three Ds”: devices, documents and disadvantage. If we rely on a portable device to carry our identity, what happens to the 16% of students who do not have a smartphone? Where we need documents to establish even our real-world identity, what happens to people who do not have either a driving licence or a passport (24%, and higher among young people); what alternative proofs might they offer? And how to include those who may be disabled from using these systems, either because of medical or social disadvantage, or for lack of the first two Ds?

Lilian Edwards gave a fascinating keynote at the UK IGF this morning, on Protecting Digital Rights During a Pandemic . Though privacy is the most often discussed right in the context of pandemic response, rights of free speech and free assembly also need to be borne in mind.

Although the impact of national schemes (contact tracing apps, etc.) has been widely discussed, how we respond to the virus in re-opening workplaces may be at least as significant for individual rights: not least because employers can invoke a wider range of permissions from data protection law. Will our response to COVID-19 accelerate an existing trend towards workplace surveillance, through the re-purposing of existing sensors and controls such as CCTV, swipecards and smartphones? Might employers feel justified in compelling employees to run, and provide access to, a location-recording app to gain entry to the office? Or, might some even refuse access to those who have chosen to run a social distancing app? What are the effects on our rights – and those of family and visitors – if employers insist on extending these into our off-work time, or our homeworking space? Or can we use the pandemic as an opportunity to demonstrate good practice in how we choose our purposes and technologies?

Considering university campuses as a possible testbed for developing good practice, it’s clear that there are many options to choose from (all these examples have appeared in the press in recent weeks). Even slight modifications to how we think about purpose could make new technological available, and have significant benefits for rights. For example, rather than individual testing to pick up early signs of an outbreak – which affects privacy and uses testing resources inefficiently – could samples be pooled before analysis? If students are expected to live and learn together, it doesn’t matter which member of a “study bubble” tests positive: all need to isolate anyway. Or could we test wastewater and avoid bodily intrusion entirely? If we think of social distancing as an issue about spaces, rather than about people, then many more sensors become available: blurred CCTV images can be effectively anonymous, but still let us measure the distance between bodies; even atmospheric CO2 level might be used to detect when there are more people in a space than is safe. Now we can avoid discriminating against the 1-in-6 students who does not have a smartphone, leave those who do have one with a free choice how to use and when to carry it, but still inform campus users of spaces and behaviours to avoid.

Adjusting how we think about the problem could be a win-win. If we aim low – in the sense of rights infringement – then we are likely to get greater participation (or less resistance) and end up with more accurate, relevant information on how to stay safe.

Colleagues set me the challenge of saying something about my work in one minute. So here (on YouTube) is a “peacast” – my wife says it’s too small to be a “podcast” – on Brexit and GDPR:

Comments very welcome on the format and, if you like it, suggestions for any other topics I could cover in a similar way. I am also working on a five-minute format, which allows more in-depth exploration!

As part of Jisc’s exploration of Artificial Intelligence, we’ve created a free “mini-MOOC” (mini, because you should be able to complete it in 30 minutes, or longer if you do the additional reading). We’re planning to run it monthly, but you can sign up any time for the next run.

The course, and the format are both new, so we’d really welcome your feedback. There’s a comment form at the end of the course, or you can comment on this blog post.

If you’re interested in Artificial Intelligence, and have half an hour to spare, Jisc’s new “mini-MOOC” is designed for you. We hear a lot about how we should be worried about “AI”, usually illustrated with an image from Terminator. But most of us frequently use AI, without even noticing, to do things that would have seemed impossible just a few years ago. Through videos, articles and exercises, the course explores why that might be. We hope it’ll help you think about your own uses of AI and how you can make them more likely to be perceived as useful rather than creepy. And if you haven’t got a whole half-hour, you can even split the course into four shorter chunks!

This mini-MOOC is open to all learners from Jisc member institutions. The MOOC includes commenting on other students’ posts, so we are granting access in “waves”, so that there is a group passing through together each month. We hope this will encourage you to discuss your ideas. This may mean that there is a short wait between applying to do the course and being granted access. We’ll advertise the next starting date (usually the start of the month), in case you need to plan.

We currently have an enrolment limit for each “wave”. If you’re from an eligible Jisc member institution you will be accepted as soon as we have space; we appreciate your patience in awaiting the confirmation email.

BEREC, the board of European Telecoms Regulators, has just published its updated guidance on enforcing the Network Neutrality Regulation . Jisc has been working with the Forum of Incident Response and Security Teams (FIRST) for nearly five years to ensure that this legislation and guidance didn’t discourage legitimate practices to secure the operation of networks: this new version suggests our advice has been heard and taken.

Article 3(3) of the Regulation permits “blocking, slowing down, altering, restricting, interfering with, degrading or discriminating” only in three specific circumstances, one of which (covered in Article 3(3)(b) is where this is necessary to “preserve the integrity and security of the network, of services provided by that network, and of the terminal equipment of end-users”.

The new guidance (para 83) identifies “typical attacks and threats that will trigger integrity and security measures”:

• flooding network components or terminal equipment with traffic to destabilise them (e.g. Denial of Service attack);
• spoofing IP addresses in order to mimic network devices or allow for unauthorised communication;
• hacking attacks against network components or terminal equipment;
• distribution of malicious software, viruses etc.

Paragraph 84 lists “typical examples of … traffic management measures”:

• blocking of IP addresses, or ranges of them, because they are well-known sources of attacks;
• blocking of IP addresses from which an actual attack is originating;
• blocking of IP addresses/IAS showing suspicious behaviour (e.g. unauthorised communication with network components, address spoofing);
• blocking of IP addresses where there are clear indications that they are part of a bot network;
• blocking of specific port numbers which constitute a threat to security and integrity.

Paragraphs 85 and 86 note that the need to enable for such measures might be identified by the network’s own monitoring, or by reports/complaints from end-users or blocking lists from recognised security organisations.

And, as discussed in the earlier post on the consultation paper , paragraph 85 also explains why permanently configured blocks, such as for BCP-38 spoofing, do not contravene the Regulation’s “only when needed” requirement, as the block only takes effect when an offending packet is seen.

Although the Regulation doesn’t apply to Janet and private networks within our customers, it’s good to have such explicit confirmation of our security practices. If other networks follow them, then there should be less hostile traffic for us all to deal with.

Might some of the problems in applying data protection law to machine learning arise because we’re using too simple a model? Sometimes an over-simplified model can be hard to apply in practice. So here’s a model that’s a bit more complex but, I hope, a lot easier to apply. It’s also a lot more informative, especially about what data we should be using, about what people, and for how long.

This views a machine learning model as having four stages:

• Development, where we decide which features are of interest, which algorithms to use, etc.;
• Learning, where algorithms decide what weights to give to each feature;
• Use, where we apply the model to real data and situations;
• Review, where we consider how well the model worked.

These form a cycle: review may trigger a new round of development, learning, etc. There may also be shorter cycles within the loop – iterating between development and learning when we discover that we chose the wrong features or algorithms, for example – but I haven’t shown those in the diagram.

Each stage has different requirements for the data it uses, so they should be considered separately under data protection law (that law definitely applies if machine is learning about people, though the same model seems to work well even if it isn’t). In particular, this approach suggests there may be a nice balance between the quantity of data required at each stage and the safeguards that can be applied: stages that need more data may also be able to apply more safeguards.

Note that, at a higher level, the ICO has new Guidance on AI and Data Protection , covering Governance, Accountability, Lawfulness, Fairness, Transparency, Security, Data Minimisation and Individual Rights.

Development/Feature Selection

This stage needs a broader selection of data than any of the others, precisely because one of its roles is to identify and eliminate fields and sources that don’t provide sufficient useful information for the model’s purpose. Here, we need a representative set of data, but it may be acceptable for those data to be less than comprehensive, so long as they include the features likely to be relevant, and a representative range of subjects. It should be possible to use pseudonymised data for feature selection, and there should be no need to keep the data after the model has been constructed.

A wide range of GDPR lawful bases could be used for this stage: in education the most likely are probably Necessary for Contract (if the model is part of the contracted service), Necessary for Public Task , Necessary for Legitimate Interests , and Consent (provided we ensure the resulting data are still representative). If the institution already has the required data for some other “necessary for…” purpose then the “statistical” provisions may be worth considering since these provide helpful guidance on how to maintain separation between the purposes and avoid creating risks to individuals.

Learning

This stage uses the reduced set of fields identified in the Development stage, but it is essential that the data in those fields be comprehensive and unbiased. To reduce the risk of learning discovering proxies for protected characteristics (typically, those covered by discrimination law), we may need to include data about those protected characteristics at the learning stage. Again, it should be possible to use pseudonymised data for Learning, and it should not be necessary to keep the data after the model has been constructed.

Here, Consent is less likely to be an appropriate basis, because opt-in data is likely to result in models that recognise those who opt-in. Again, the statistical provisions are informative.

Use

Here we apply the model to real people and situations, usually with the aim of providing them with some kind of personalised response. We probably need all the fields that were identified in the Development stage, though Learning may have identified some that are less important (or are correlated with others) so can be eliminated. The model should, at least, recognise data records that are too sparse to be reliably usable. Here the data does need to allow identification of, or personalisation to, an individual, so pseudonyms are unlikely to work. But allowing individuals to self-select whether, or when, the model is applied to them is much less of a problem. And tight time-limits on how long we keep data should be possible.

Here Consent and Necessary for Contract are the most likely lawful bases; Necessary for Public Task or Legitimate Interests are possible, though we need to consider the risk of applying the model to those who have not actively engaged; statistical, which prohibits individual impact, is not.

Review

Finally the Review stage will need to include historical data – what actually happened – and may need additional information about outcomes that was not relevant to the Use stage. It should be possible, however, to protect this wider range and duration of data by either pseudonymisation or anonymisation.

The review stage is likely to process some data on the basis of Consent (from individuals who are either particularly happy or unhappy with their treatment). However this must be balanced with a representative selection of data from those who do not respond. Sampling should be possible, even desirable, as a safeguard. It may be necessary to use Consent for this as well (for example if the only way to discover outcomes is through self-reporting), but if the institution already has the data needed for Review then Necessary for Public Task or Legitimate Interests may be more appropriate ways to achieve the required representativeness.

Summary

The following table shows how this might work…

We’re delighted to have launched our Wellbeing Analytics Code of Practice, something we’ve been working on in the ICO’s Regulatory Sandbox for almost exactly a year. The resulting Code builds on Jisc’s widely-used Learning Analytics Code of Practice and includes tools for Data Protection Impact Assessment and Purpose Compatibility assessment. We hope it will give students, staff and institutions confidence that data-informed wellbeing support can be provided safely. You can download:

• The Code of Practice
• The ICO’s announcement & report
• Jisc’s announcement

We’d welcome your comments, feedback and, especially, experiences of using the Code.

A couple of new documents provide ideas on how to think about ethics when we deploy Artificial Intelligence.

First is an article by Linda Thornton for EDUCAUSE, on Artificial Intelligence and Ethical Accountability . This looks at who should be thinking ethically, finding responsibilities for programmers, managers, marketers, salespeople and organisations that implement AI. Since this is an EDUCAUSE article, it focuses on Higher Education Institutions in their role as purchasers of AI, and proposes a five-step approach – with lots of references – to selecting and using AI ethically.

Second is the latest document from the European Commission’s High-Level Expert Group on Trustworthy AI (HLEG): the Assessment List for Trustworthy Artificial Intelligence . This has specific questions relating to each of the seven requirements set out in the HLEG’s Ethics Guidelines for Trustworthy AI : Human Agency and Oversight; Technical Robustness and Safety; Privacy and Data Governance; Transparency; Diversity, Non-discrimination and Fairness; Societal and Environmental Well-being; and Accountability.

The authors note that answering these questions should involve discussions among a multi-disciplinary team: given that the questions range from whether the AI is likely to become addictive and  its effect on the environment and “other sentient beings” to technical questions about the security of data and the stability of algorithms in the face of data attacks, those would be fascinating meetings to be involved with.

One oddity is that, whereas I’ve previously noted that GDPR compliance seemed a good (and, if using personal data, essential) starting point for the HLEG requirements , this Assessment list seems to take things the other way around, suggesting that the GDPR is a useful source when completing the assessment. “Protect[ing] personal data relating to individuals in line with GDPR” is mentioned, but only as one of the things that “might” be included in a prior Fundamental Rights Impact Assessment.

That seems to run the risk of missing both useful guidance and legally-required measures. For example there’s no mention under Transparency of the information requirements in GDPR Articles 13 & 14; nor in Accountability of the overlapping GDPR Principle of the same name. Even more fundamentally, there’s no mention of the need to define a legal basis (GDPR Articles 6 & 9) for processing personal data, nor to check purpose compatibility when reusing data. Those may be challenging for some AI systems – though perhaps not as challenging as is sometimes claimed – but that can’t be a reason to ignore them.

Great to have my paper – “ Processing Data to Protect Data: Resolving the Breach Detection Paradox” – published by ScriptEd.

Everything you always wanted to know about logfiles and the GDPR:

1. Why Data Protection requires breach detection;
2. What’s the GDPR “Purpose” of breach detection;
3. What’s “Necessary”, when it comes to breach detection;
4. What Safeguards are required;
5. How Automation helps;
6. What about automated breach prevention;
7. More than compliance.

In other words, the prequel to “ Incident Response: Protecting Individual Rights under the GDPR “, from 2016.

The recent Schrems II decision on Standard Contractual Clauses found that, in some situations, data exporters and importers might need to agree additional measures beyond just relying on SCCs. While we’re waiting for the Information Commissioner and EDPB to give more detailed advice on which situations and which measures, here are some themes I’ve spotted in articles and webinars:

• Those additional measures can’t be contractual in nature, because the problem identified in Schrems II was that the US Government was not bound by any contract that a US organisation might enter into;
• The concern still seems to be with personal data that is physically moved to the US, not merely put within the technical reach of US companies. The laws quoted in paragraphs 60-63 of the judgment as giving the US Government power to access data despite contractual protection appear to be limited to particular physical locations;
• Additional measures could be in the form of laws of the foreign state, since these could bind that state’s Governmental authorities. Interestingly the judgment mentions GDPR Article 45(2)(a), which covers “both general and sectoral” laws as something that should be taken into account in an adequacy decision or, now, what I’m thinking of as an SCCplus discussion. Since the US does have a sectoral privacy law covering educational institutions – the Family Education Rights & Privacy Act (FERPA) – this might be something to discuss with the recipient if you are exporting to a US university or college;
• Additional measures could be technological, if they protect against the foreign Government’s legal powers. Encryption has been mentioned as an option. This clearly works best if the exporting organisation controls the encryption throughout, for example if storing encrypted backups on an IaaS cloud server;
• But, as far as I can see, there’s no requirement for a single measure to cover the whole transfer of data. So, for example, you might argue sufficient protection was provided by a hybrid solution that used encryption to technically protect data as it flowed over public networks into (and back from) an enclave that was protected by law.

I’ll update this post as and when there’s any more detailed guidance from regulators.

[UPDATE: not from Regulators, but Chris Pounder has posted a helpful summary of how we used to think about transfers under the 1998 Act . This may be relevant once again]

[UPDATE 27/7/20: the ICO has now published a statement on the decision ]

On July 16 ^th 2020, the European Court of Justice made its long-awaited decision in the case of Data Protection Commissioner [Ireland] v Facebook Ireland Ltd and Maximillian Schrems , generally known as “Schrems II”. This concerned two of the GDPR’s mechanisms for transferring personal data from the EU to other countries: Standard Contractual Clauses (SCCs) and the US Privacy Shield. This is relevant to organisations in the UK for two reasons:

• SCCs and Privacy Shield were options available – both now and after we leave the EU – for transferring personal data to third countries;
• After we leave the EU, SCCs are one of the options that may be used to receive data from EU countries.

Taking the Court’s summary of findings (see the very last section of the judgment – p61 of the PDF) in reverse order, for clarity:

1. Point 5: Privacy Shield is invalid because it does not protect against US surveillance laws which do not provide safeguards equivalent to the EU requirements for proportionality and judicial redress.
2. Point 4: SCCs are still valid (para 149) and sufficient to ensure the recipient organisation provides adequate protection (para 124), but…
3. Point 3: …since a contract cannot bind the authorities of the receiving state (para 125), National Regulatory Authorities (and data exporters) must also examine the legal environment into which transfers under SCCs are made, and must stop/order suspension of individual transfers where that environment does not offer sufficient protection;
4. Point 2 – In particular (para 129), where data transferred under SCCs may be accessed by the receiving country’s public authorities, the legal framework covering that access must be assessed on the same basis as when the Commission makes (or doesn’t) an adequacy decision (including rule of law, relevant legislation, independent supervisory authorities, international commitments, etc. Para 45(2)) is full list).

I think that means

1. Watch ICO for indications of which transfers from UK to US are likely to raise concerns under SCCs; review yourself any transfers that might need additional safeguards.
2. Post-Brexit, unless the UK receives an adequacy decision, SCC transfers to the UK may be investigated/stopped by any EU regulator if they feel UK law does not provide adequate protection in case of access by authorities. Such decisions must be taken by individual regulators, but may be discussed by the EDPB.
3. SCCs are still valid and worth including (both when exporting and importing personal data) as they continue to ensure the recipient organisation provides adequate protection.
4. But exporters and importers also need to consider whether additional protective measures are needed in case of access by authorities (note, for example, that transfers of student data to US institutions – unlike the transfers within Facebook that were the subject of the case – are likely to be covered by FERPA when they get there). Document these decisions, in case they are challenged. It’s also worth reminding (and expecting to be reminded) of SCC Clause 5(b)’s obligation on importers to inform the exporter if national law has a substantial adverse effect on their ability to comply with the contract. And have a process ready for receiving/sending such notifications.
5. If an exporter or a Regulator decides a data flow isn’t safe, the only option seems to be to change the flow. Para 132 makes clear that a contract can’t fix the problem, because it doesn’t bind the national authorities. But Para 112 considers “all the circumstances of the transfer of personal data in question”: Bird & Bird suggest that encryption, tokenisation and other technical measures may be relevant circumstances .

Finally, my attention has been drawn (thanks!) to the excellent “ EU-US Privacy Shield, Brexit and the Future of Transatlantic Data Flows ”, published by UCL’s European Institute in May – after the Advocate-General’s advice to the ECJ has been published but before the Court ruling. Highly recommended if you want to know more about the legal and practical background, the options available to the Court, and the implications for post-Brexit EU-UK data flows.

[Update: a recording of my talk is now available]

When I submitted my proposal for a talk at the EUNIS 2020 conference , I was planning to talk about the need to work with staff and students to agree why and how to use intelligent campus sensors and data . That was intended to be looking into the future, but in the past three months it has become much more imminent. As campuses re-open after COVID-19 lockdown, data and sensors seem likely to play a significant part.

Most universities seem to be planning to use a hybrid model , where most students and staff are present on campus but use on-line tools for large-scale activities such as lectures that may be unfeasible under social distancing requirements. That combination is likely to give institutions an unprecedented amount of data: they may well have access to data from both the connectivity layer (wifi, logins, etc.) and the application layer (VLEs, video-conferences, etc.). Under normal circumstances they would have application data about remote students and connectivity data from on-campus ones. During the return to campus stage of the virus there may well be additional purposes for which institutions might consider using that information: from planning and managing social distancing to supporting those who may need to self-isolate on campus.

However it is important to remember that both staff and students are likely to be highly stressed during this period. Wikipedia defines “ technostress ” as the “result of altered habits of work and collaboration that are being brought about due to the use of modern information technologies at office and home situations”. That’s exactly what most of us have been experiencing for three months or more. Additional uses of data could easily be perceived as surveillance, rather than support. Even before the virus outbreak, student and staff wellbeing was a concern in many countries. We must be particularly careful that anything we do (or might be perceived as doing) does not make that worse.

In such an environment, it seems particularly important to achieve consensus among all those involved before taking action that, if misinterpreted, might do more harm than good. In particular, we must agree which uses are temporary measures that will be regularly reviewed and terminated when they are no longer required by a medical emergency.

It seems a long time since I wrote about the ePrivacy Regulation . This was supposed to come into force alongside the GDPR, back in May 2018, and provide specific guidance on its application to the communications sector. You may remember it as “Cookie law”, though it was never just that. Unfortunately its scope grew and, for at least the past eighteen months, discussion among Member State Governments has been going around in (sometimes increasing) circles.

Germany took over the EU Presidency last week, and on Monday published what looks (mostly) like a decisive approach to the outstanding issues . They have summarised (nearly all) the issues in three questions to be discussed at a meeting in a couple of weeks’ time (my summary, though theirs aren’t much longer, once you discount the strawman sections of legislation):

1. Particularly in the light of COVID-19, should there be sector specific rules on processing communications metadata for health purposes, or can we just rely on the GDPR “vital interests” justification and safeguards?
2. Should there be sector-specific rules on processing communications data for purposes others than security, service delivery, etc., or can we just rely on the GDPR “legitimate interests” justification and safeguards?
3. Should there be sector-specific rules on accessing terminal devices (e.g. to install security patches), or can we just rely on the GDPR “legitimate interests” justification and safeguards?

So far, so good. But, several presidencies ago, it was noticed that any law on how networks could process communications data and content was likely to have an impact on how networks detect and block illegal content, in particular child abuse imagery. Alarmingly, although the Presidency thinks it can assume that all Member States think those activities should continue, it seems that “elementary questions” on the correct legislative approach to achieving that “remain highly controversial”. So that discussion is postponed to “a later date”.

Finally, note that we are still in the process of European Governments coming to a shared agreement on what the legislation should look like. Once they’ve done that, they still need to negotiate with the European Parliament (which reached its opinion two and a half years, and an election, ago ) to find a mutually agreeable text. Not quite so long ago, the UK Government indicated that, despite Brexit, it was minded to follow the European lead. But when there might be such a lead to consider is still a very open question.

An interesting virtual water-cooler discussion with colleagues who are exploring the potential of AI as a Service. They tested a selection of easily available cloud face-processing systems on a recording of one of our internal Zoom meetings, and were startled by the results.

Face identification wasn’t a surprise: everyone who has changed the background on a conference call has used software to pick out a face from a background. Identifying other objects in the picture, estimating age and gender we were expecting. But the ability to attribute names (by comparing with publicly available photographs) and emotions is much more striking when you see it done to you, rather than just described.

It’s not always accurate, of course. My age varied by 10 years depending on lighting level, and was reduced by 50 when I tried on my COVID mask! The exec who was speaking was mostly “angry” (we think he’d prefer “passionate”) and he might have hoped fewer of us would be “neutral” or “sad” while listening. He did manage to change the emotion assigned to a colleague’s background picture of Mount Rushmore , though!

Face recognition and emotion detection could, of course, be valuable assistive technologies for those who have difficulty doing those things themselves. But they have also been banned by law in some states and contexts. Faces are especially sensitive, and specially protected, parts of our humanity.

So, some questions to consider before using any of these technologies:

• What are we trying to achieve? Can it be done another way? Why aren’t we doing that instead?
• Can we ensure everyone supports this? How will they respond if they don’t?
• What’s the risk, and how will we handle it, when (not if) the results come out wrong or discriminatory?
• Are we setting an example we’d be happy for others to follow? All the technology we used is available for students, staff, friends, colleagues and strangers to apply and share to pictures and videos of us, too.

In the week that would have been their annual conference, EEMA have been hosting a series of fascinating online discussions among experts in the identity world . Today’s featured Steve Purser, Dave Birch and Kim Cameron in a deep discussion about whether we might have been looking at the wrong kind of “identity” all along…

The words “identity” and “identification” seem so close that it’s easy to think that the thing we really need to know is who someone is. And Governments, in particular, have spent a lot of effort in building systems to do that. But adoption of them has been, at best, slow and confined to relatively narrow niches. For many, perhaps most, applications, “who” is actually among the least useful things to know: often it’s not enough, sometimes it’s too much. In financial transactions we actually want to know whether someone is financially reliable; when granting access to data we want to know whether their role and other qualities entitle them to see it; in a bar we want to know whether someone is old enough to buy a drink.

The emergence of the Internet of Things makes this particularly obvious. We may well want to “authenticate” and “authorise” a car, a lightbulb or a fitness app, but this has nothing to do with its passport. On social media we might simply want to know “is the account I’m interacting with a human?”; if we’re taking a report from a whistleblower we want to know “do they work there?” but we must be able to guarantee that we cannot identify them. Technologies exist that can do this, but they’ve not been the focus of mainstream development.

Kim Cameron suggested going back, way back, to the definition of “identity”. According to the Oxford English Dictionary this has three strands: “selfness”/ sameness/individuality (a sense dating back to 1611); “whatness” (also 1611) and “whoness” (which appears in 1922). So far, identity technologies have been focussed on whoness: the thing we need when dealing with Government and colleagues. IoT and most other applications actually want whatness (interestingly this – is someone a student or staff member, are they entitled to a car parking permit – has been of interest in Research and Education). We need to do a lot more development of that and of the selfness that allows us to bring together the multiple strands of our “who” and “what” and control which we use in each context. A bar may get to see confirmation of drinking entitlement and perhaps a photo, as certified by the driving agency, but it does not need to see the rest of the information on our licence.

So, although identity technologies have already contributed a great deal to digital transformation of enterprises and government, there is still a lot to do. Virus lockdown has shown that organisations are at very different stages of that digital transformation. And, whereas previous digital identity transformations have focussed on organisations, this time we really need to consider personal digital transformation, too, and help individuals through it. Many of the problems in our current “identity” space actually derive from the two sides of the transaction being far out of step. Organisations may have transformed to demand digital identity, but the uncoordinated nature of this transformation has overwhelmed the individual, who has to deal with tens, hundreds or thousands of “transformed” entities. If every one presents different identity demands, it is little wonder that individuals cannot cope and either give up on digital services entirely, or use them in the most convenient/least secure way possible.

This morning’s Westminster e-Forum event on regulating Online Harms contained a particularly rich discussion of both the challenges and opportunities of using regulation to reduce the amount of harmful content we see on the Internet. The Government published a white paper in April 2019 and an initial response to comments in February 2020. A full response is expected later this year with legislation to follow in this Parliament.

One of the biggest challenges is the ambition to address both content that is illegal – typically well-defined in laws such as the Protection of Children Act – and content that is lawful but harmful. In the latter category the current pandemic has drawn particular attention to misinformation (accidental) and disinformation (deliberate): apparently a single item of “junk news” can achieve greater reach through social media than individual articles in the mainstream media. The Government’s initial response has recognised that these do need to be treated separately and suggests the approach to lawful but harmful will focus on processes rather than individual items of content.

Among the challenges identified:

• The scope of the regulation is still unclear. Although the Government has stated that “less than 5% of UK businesses” will be covered, it have not provided any suggestion of how the current broad definition – any website allowing user-generated content, comments, or interaction – will be reduced. TechUK noted the very wide range of size and capability of organisations in the UK’s digital sector: regulation must not require measures that only large organisations can deliver. It may help to consider different ways of reducing the visibility of harmful content: preventing, disrupting, de-ranking and de-monetising as well as removing. Appropriate measures of “success” will be needed, which may well vary between types of harm, approaches to reduction and types/scales of regulated entity.
• The range of lawful content to be addressed is not clear, likely to be dynamic and, by its nature, hard to deal with at scale. Targets of hate-speech have changed dramatically over the past three months: could systems and processes (which may need to review millions of items a day) keep up with changes in vocabulary? There may also be a high level of dependency on context: whether a statement is teasing or bullying may well depend on the relationship between the individuals. Victoria Nash from the Oxford Internet Institute suggested we may well need specific legislation on statements by political leaders: should these be judged by the same standards as those applied to citizens? This should definitely not be left to platforms and regulators to decide.
• Although the White Paper mentioned the importance of protecting free (lawful) speech, this has not been supported by the same level of detail as commitments on rapid removal. Internet regulation has a long, and largely undistinguished, history of creating one-sided incentives that make it safer for platforms to remove material rather than justify their decision to continue to publish it. The White Paper set ambitious targets for removal of material within 24 hours, but accurate decisions need to be given as much credit as fast ones. Asgar Koene raised an interesting point that clarity is particularly important for UK legislation since, unlike Germany, we share a language with countries whose approach to this free speech balance may be significantly different. Global platforms cannot simply apply our national law to content in “our” national language.

The sessions also included presentations on some models of regulation that could be more widely adopted.

• The Internet Watch Foundation’s work in using public reports and their own proactive investigations to generate takedown notices, URL lists and hashlists for child abuse images is well-known. The depth of their collaboration with industry perhaps less so. The charity has been industry-funded from its origins twenty years ago. It now works with technical experts from members to understand the complex ways in which material is distributed and how this might best be disrupted; and it holds hackathons where members can explore new ways of using IWF data in their systems.
• The Internet Commission is a newer non-profit organisation that aims to identify and promote best practices in content moderation. These look broadly at the moderation process: reporting, moderation, promotion, notice, appeal, resources, governance. Companies from diverse sectors have volunteered for an independent assessment, involving confidential bilateral review, private benchmarking and information exchange. Some common issues that are already emerging are the relationship between automated and human moderation, and how to provide rights of appeal. The first annual report, due before Christmas, will be interesting both for content and model.
• Finally, since many of the problems are social in origin, we should not ignore the contribution of social solutions. Though attempts at “awareness raising” and “digital literacy” have been common for many years, panellists suggested it might be more fruitful to focus on “digital civility”. Rather than trying to teach readers how to assess the source of material we receive, this might usefully focus on responsible habits in how we should use it. Ofcom’s digital literacy surveys suggest that while a lot of people are concerned about material they encounter, many don’t do anything about it.

I was delighted to be invited to contribute an article to IDPro’s Body of Knowledge for professionals working in the field of digital identity . Mine is (of course) on how the GDPR applies to identity management .

But as well as standards and regulation the collection is steadily expanding to cover things like privacy for consumers, architectures, access control, digital identity, project management and knowledge sharing. If you’re interested in any of those, it’s well worth a look.

WONKHE has published my article on the need to be careful in introducing, and withdrawing, with any post-virus data processing (the absolute sub-head isn’t mine!)

Maintaining trust in university data handling

As data protection regulators keep reminding us , the research and data protection communities mean different things when they talk about “consent”. A couple of recent conversations have made me wonder whether that terminology clash may have another effect: are those putting research into practice missing out on existing guidance that could help with that transition?

In the research world it seems that once you’ve obtained consent, or decided it is not applicable, the only other guidance available is from the field of “ethics” as it applies to the particular research domain. Hence it may also seem natural when reading data protection law to look at the section marked “consent” and then try to resolve all remaining questions using “ethics”. In fact, the rest of the General Data Protection Regulation (GDPR) provides much more concrete guidance on how to do the research-to-practice transition.

First is the way that research is explicitly allowed (subject to appropriate safeguards) to take a broader focus than services: for example contrast “consent to certain areas of scientific research” in Recital 33 with “consent … for one or more specific purposes ” in Article 6(1)(a); or the recognition of research as a broad “compatible purpose” in Article 5(1)(b). This indicates that we should expect to narrow our focus, and set more specific requirements, as we move from research to implementation. Indeed one of the outcomes of research should be precisely to discover which practical activities are likely to be beneficial.

Similarly with data, there should be a narrowing of focus from data that might be needed for research down to the data that the research concludes are actually required to achieve a specific result. If that result is the delivery of a service, then the GDPR’s principles of data and storage minimisation are particularly relevant. These appear in many ethics codes but the GDPR provides better guidance. In particular, the GDPR concept of processing “ necessary for … ” provides a clear test to distinguish information that is essential to service delivery from optional information that we could use if the user chooses to provide it.

Where research results are used to improve how an organisation conducts its activities then two other legal bases are informative, whichever may formally apply. The “ public interest ” basis helps to explore the requirement, common to most ethics codes, for “ lawfulness, fairness and transparency ”. The “ legitimate interest ” basis requires – and provides guidance on – consideration of “fundamental rights and freedoms”, as well as the principles of “ purpose limitation ”, “ accuracy ”, “ integrity and confidentiality ”.

Finally, the principle of “ accountability ” requires that we not only do this thinking, but that we be able to demonstrate that we have done it.

As I’ve written elsewhere , this takes us a long way into what has traditionally been solely a matter for “ethics”. Once we have actually exhausted the guidance that is available through the GDPR, the remaining questions that really do need to be dealt with by ethics may seem less daunting.

Tony Sheehan, of Gartner, observed in this morning’s EUNIS 2020 keynote that Higher Education has changed more in the past three months than in the whole of his previous career. Universities have delivered an “extraordinary achievement” in delivering learning continuity through various intensities of COVID-19 lockdown. Now we’re approaching a recovery stage when we can review – “learning optimisation” – and consider what to do next – “learning transformation”.

This needs to recognise that our current (June 2020) practices almost certainly aren’t sustainable in the long term. Most institutions currently have an imperfect implementation of online learning, which depends on the goodwill of both students and staff. But we’ve also learned a lot about what might be possible in future: again picking up an unofficial conference theme that the virus may well have brought the future of education a lot closer.

So now it’s worth reflecting on which aspects of that change we want to keep; which we want to continue but do differently; and which we want to treat as short-term emergency measures. It’s pretty clear that things won’t go back to the way they were: staff and students who have experienced new possibilities won’t want those to be lost. Equally, trying to sustain everything about our current operations is likely to be too stressful for both staff and students: we need to choose the right things to “return to normal”. The resulting plans need to be flexible. We don’t know either when, or under what constraints, that future can be delivered; we may even have to return to stricter measures during any second wave, though there will be less tolerance of improvisation second time around.

In the time between future vision and past experience we should review:

• Our systems: which worked well in the new environment, and which only coped thanks to staff and student commitment;
• Our skills: what did we already have, how can we make improvisations sustainable, and which might we need to train or recruit;
• Our students and staff: how did they feel, what is sustainable beyond crisis mode, how do we prepare for the new world.

A pair of interesting sessions at today’s EUNIS conference looked at how universities responded to the impact of COVID-19 lockdown on end of year assessment. An audience survey indicated that 60% have changed the form of their assessment, 15% cancelled exams, and 15% adopted some kind of remote proctoring system to allow for traditional-format exams to be taken in uncontrolled locations, such as the home. In countries where travel was possible, but limited, existing arrangements for taking exams in other institutions had become highly relevant.

For remote proctoring (only one option could be chosen), 33% of the audience were concerned about privacy, 31% about equality: mostly because students with less appropriate devices or poorer internet connectivity might be disadvantaged, either in fact, or by increased stress from worrying about it. Cost and other factors worried the remainder. Case studies from the Norwegian and Finnish university systems suggested that the only use of remote video was to verify the identity of the person sitting the exam against their official documents. Technical tools mentioned included kiosk applications that limited what other applications could run during the exam period; blacklists on collaboration tools; plagiarism detection; and noting significant pauses or sudden bursts of typing for later investigation.

The much more popular option was, however, to change the style of assessment, with many people commenting that lockdown had brought forward changes they had hoped to make anyway . These included  moving to open book exams and eliminating multiple-choice questions, which were felt to have the greatest risk of cheating. The limitations of digital exam platforms may even have advantages: if the discipline involves notation that is not easily supported by keyboard and mouse, then get the student to write that part of their answer and submit the picture. This gives not only a check of knowledge, but also of the person doing the handwriting!

Finally, there was a reminder not to aim for perfection. Paper exams have risks, too. And, as one audience member commented, if someone can use reference materials fast enough to pass a time-limited multiple-choice paper, then perhaps they do know the subject quite well anyway!

Alberto Cairo , Knight Chair in Visual Journalism at the School of Communication of the University of Miami, gave a wonderful EUNIS 2020 keynote on Making Good Visualisation Decisions . His argument – Visualisation is like writing: there are basic (grammatical) rules, but also choices, and those should be reasoned. Bad decisions can cause real-world harm. Just three of my highlights from his presentation.

How, even whether, to use visualisation must be appropriate for the context. A cartoon bubble map of Google searches beginning “Why does my dog/cat…?” is great for the cover of a report . A representation of COVID-19 statistics by anyone who does not understand epidemics is very different.

The choice of visualisation style must be led by the message you want to convey, not by incidental features of the data. Just because numbers sum to 100% doesn’t mean they should be displayed as a pie chart; just because a table contains geographic data doesn’t mean they should be shown as a map. Don’t be afraid to explain your visualisation and to highlight key points . This will make it accessible to many people who find visual presentation hard to read and may help increase visual literacy . The Financial Times has an excellent visualisation vocabulary .

And use visualisation yourself to identify, understand and, if necessary, correct outliers. On any statistics of internet use, the state of Kansas is likely to be one of these. Not because of anything to do with the population, but because it’s the geometric centre of the country, and that’s where many geo-location tools show VPN accesses as coming from!

There seems to be a widespread perception that “AI is creepy”. But at the same time as reacting strongly against an app that would check social media posts for signs that we were struggling to cope , we don’t think twice about the grammar checker that continually reads everything we type. I wondered why and if there were any rules of thumb we could use when proposing AI as a helpful assistant, rather than a creepy intruder.

Using AI to process faces provides an informative range of examples. Automated face recognition to find criminals has had a strong negative reaction in the UK and is being banned in an increasing number of other states . But I accepted the same technology letting me board a plane in the US last year; was happy to join the shorter queue for an automated face/passport verification to get airside; and I hadn’t even thought about the face detection involved every time we insert an alternative background into a video call.

This suggests there are two significant factors: whether the AI operates continuously or only at times I choose; and how limited its function seems to be. Ideally that should be a technological limit, though more general technology may be acceptable if there are strong policies and sanctions for misuse. In a video conference I choose whether or not to use face detection, and separating a picture from the background has few other uses. A boarding corridor that identifies passing faces and checks they are on the correct flight is time-limited, and I could opt for a human document check instead. So:

Checking this model with AI that processes words, rather than faces, suggests that it fits there too. Smart “speakers” (actually, it’s the smart microphone function that’s spooky) are always on and potentially unlimited in purpose – the common concern is “what else is it doing?”; grammar checkers are always on, but limited to a single function; voice bots that try to help us through phone menus are limited in time (though frustrating if they don’t offer a human fallback option); asking a translator to work on a text gives us control over both what and when we disclose.

Other factors that seem to increase the sensation of creepiness include:

• Tasks that get too close to our identity as humans: e.g. identifying faces or emotions;
• Decisions we feel should only be made by humans: e.g. on early release from prison , in education course entry or continuation may be one of these;
• Systems that are, or may learn to be , discriminatory (even identifying gender may be problematic);
• Using information the individual might not have realised that we have, or not expected to be used in that way. Here a clear notice may satisfy the law , but may not remove the unease.

Any of these is likely to push our application down and to the right in the matrix, increasing the risk that it will be perceived as creepy. Conversely, the further up and left we can place our application, the more likely it will just be accepted for the contribution it makes to our daily lives.

For an excellent introduction to AI, with no mention of creepiness, see DSTL’s “Biscuit Book” .

I’ll be delivering the opening keynote, “ See No… Hear No… Track No…: Ethics and the Intelligent Campus “, at the EUNIS 2020 (online) conference on Wednesday June 10th at 0815 (UK-time).

Registration for the conference is free , and there are lots of other interesting talks on the programme.

An article, on “The value of e-proctoring as Exams move on-line”/”Technology can reduce exam stress”, was published in University Business (6/5/20) and the Jisc website (13/5/20).

[with thanks to a former university Head of Examinations for input and discussion]

Recent years, and weeks, have seen a move away from the traditional examination context, where candidates gather in large halls to write on paper, to candidates being assessed using computers, in small groups or individual work spaces. In this change, the role of the invigilator (also known as “proctor”) remains important; there may be new opportunities to use technology to assist in the invigilation process. However some of the technological systems being proposed appear to significantly expand, or even to contradict, the role of the invigilator. Such changes require particularly careful scrutiny, as they are likely to change both the purpose and the effect of examinations.

Invigilator’s Role

The role of the invigilator is to help all candidates complete their exams to the best of their abilities, in as supportive an environment as possible. Significant departures from appropriate conduct, either of examinations in general or of the specific paper being assessed, must be recorded and should be corrected where possible. These records will then be used by academic staff to determine how, and in some cases whether, the candidate’s assessment should be marked. An invigilator must only prevent a candidate completing their assessment if this is the only way to prevent disruption to others.

Management of the Examination

The invigilator has three main roles in managing the conduct of the examination:

• Time-keeping : the invigilator will generally provide interval warnings, for example when there is one hour, or ten minutes to go. Each candidate should be able to choose how much more precisely to keep track of time, to minimise their own stress level;
• Faults in papers : if candidates have questions about the examination paper – for example because it appears to have an error or be unclear – the invigilator(s) should be able to refer these promptly to a member of academic staff and then to disseminate their ruling to all candidates in all locations;
• Managing/recording unexpected situations : in a physical exam hall the invigilator is responsible for recording incidents such as fire alarms or water spills and maintaining, so far as is safely possible, compliance with the exam rubric. The impact of such disturbances on individual candidates’ performance is assessed later, by academic staff. In an online context incidents could include, for example, loss of internet connectivity.

Improper conduct that should be detected

The invigilator is expected to detect three main types of improper conduct (note that details of these may vary depending on the exam, for example whether specified external resources are allowed, and whether annotations are permitted):

• Wrong candidate taking exam : the identity of candidates should be checked to confirm that the individual who sits the exam is who they claim to be. If a candidate cannot show a matching identity document they should normally be permitted to sit the exam (in case they have good reason for lack of ID) but the circumstances must be recorded. Similarly, if a candidate who is not on the list wishes to sit a paper, they should normally be allowed to do so, as this may be the result of simple mistakes by student or institution. Both situations can be resolved later.
• Candidate using unauthorised materials : the exam rubric should state what additional resources (if any) the candidate is allowed to consult; use of resources not on the list should be prevented or detected.
• Candidates colluding (either within or outside room): unless the exam rubric permits collaboration, candidates’ attempts to communicate with others should be prevented or detected.

Improper conduct that is not expected to be detected

It is important to remember that there are some types of improper conduct that invigilators are not expected to detect. These include those that would require extreme preparation by candidates, or where detection would require unacceptable levels of intrusion and stress for them. In particular, invigilators are not expected to detect:

• Anything that cannot be detected during the exam session (e.g. that requires a recording of the whole examination process);
• Anything that requires long-term preparation (e.g. candidate substitution after obtaining false ID documents);
• Anything that requires intimate inspection/monitoring/restriction of candidates.

Methods (face-to-face)

During a traditional examination, invigilators use a number of techniques to detect improper conduct:

• At the start of the exam, or at least early in the period, identity documents are checked against the candidate’s appearance (see above for if the candidate does not present an ID); invigilators are not expected to be experts in detecting forgeries;
• Where candidates are allowed to bring materials into the examination, these are checked to ensure they meet requirements (for example on annotations);
• If there is a grace period for latecomers to enter the exam, candidates are prohibited from leaving their desks until after that period;
• If candidates are permitted to leave their desks and return they are supervised, but not intimately, while away;
• Invigilators continually survey the examination hall from a distance; if they see something that, based on their experience, raises concerns then this may trigger…
• Invigilators periodically inspect the desk area: the candidate has warning of their approach.

Thus invigilation consists of a combination of continuous, coarse-grained observation from a distance and occasional close-focus inspection. To reduce stress, the candidate is aware when the latter is taking place: a candidate who believes they may be under continuous close-focus human surveillance or recording is likely to experience considerable stress and will not perform their best.

Methods (remote/digital)

Technology should be used to contribute to the invigilation process, not merely to allow it to be conducted remotely. Simply providing a continuous video-conferencing link that effectively seats the invigilator on the candidate’s desk fails the invigilator, the candidate and the technology: the invigilator has to do at least as much work, the candidate is placed in a more stressful environment, and the technology is badly under-utilised.

Rather than merely a communications medium that removes any distance (even the length of the exam hall) between the invigilator and the candidate, technology should aim to be part of the invigilator’s alerting and record-keeping process: detecting abnormal situations, recording these for later consideration, and drawing the attention of a human invigilator in cases where either correction or further investigation is required. Where possible, technological innovation should support the invigilator’s full role: allowing it to be performed with less stress (for both candidate and invigilator) and more effectively than a physical human presence. For example:

• An e-proctoring system might take a snapshot of the candidate’s work at the point of any alert, break, or interruption. This extension of the face-to-face practice of ruling a line on the script after any fire alarm allows the work to be checked for sudden bursts of “creativity” or correction that might suggest a candidate had been consulting unauthorised materials during the break;
• An e-proctoring system might detect unexpected sounds, such as the turning of pages in a book, record and alert the human invigilator to these;
• An e-proctoring system might detect, record and alert on changes in typing cadence, suggesting that the individual entering the assessment is no longer the intended candidate;
• An e-proctoring system might detect, record and alert on unusual patterns of system or network activity that may suggest the candidate is engaged in unauthorised activity;
• An e-proctoring system might detect, record and alert on environmental changes, such as loss of network connectivity;
• Etc. etc….

[UPDATE: colleagues in Norway have pointed out that many of these ‘in-machine’ mechanisms already exist in systems designed for sitting computerised exams in traditional exam hall environments]

Since digital systems cannot understand the full human context of what may take place during the examination, their most likely role is in the continuous, “distance” monitoring aspects of invigilation. If an e-proctoring system detects behaviour that it does not understand, or believes may be suspicious, it should raise an alert. The circumstances around the alert can then be checked, either by a human invigilator during the exam or, if this is not possible, by a member of academic staff, using the system’s records and their knowledge of the individual candidate, afterwards.

“E-proctoring” systems that do no more than reproduce, badly, the face-to-face invigilation process should be regarded with suspicion.

Other Factors

Finally, remember that the invigilation process is not the only thing that prevents candidates cheating in examinations. Invigilation/proctoring, whether in person or remote, must be part of an assessment system that is designed to be resistant to cheating. Other means to this end include:

• Before assessment: considering alternative ways of assessing or verifying performance; designing assessment so that it is hard for candidates to benefit from collusion.
• During assessment: considering open book exams (so the risk of “unauthorised sources” is designed out) or, conversely, conducting on-line exams using kiosk systems that only allow candidates to view permitted resources.
• After assessment: using a moderation process to identify candidates whose examination performance differs markedly from what was expected from their work in class; also to review invigilators’ records for anything that might significantly affect the performance (either negatively or positively) of an individual or group.

Having acted as programme chair for the FIRST Security and Incident response conference last year, I also got to co-edit the special conference issue of the ACM journal Digital Threats: Research and Practice (DTRAP) . FIRST sponsored the journal, so our issue is open access, available for anyone to read. Topics covered:

• Using power consumption to extract encryption keys from IoT devices;
• Detecting threats in advance by looking at patterns of DNS domain name registration;
• Identifying and classifying malware independent of instruction set;
• Using time-series statistics to raise alerts from many different security-related statistics;
• Re-considering “false positives” as a potential source of valuable information.

I found it a fascinating and mind-stretching collection to read. I hope you will too.

Jisc responded to the Information Commissioner’s consultation on draft guidance on explaining AI. The final guidance was published in May 2020.

We welcome the ICO/Turing Institute’s draft guidance on Explaining AI Decisions , and believe that it could be useful well beyond the narrow question of when and how decisions need to be explained. However, as a regulatory tool we suggest that it needs a clearer, and objective, definition of which systems are, and are not, covered by the term “AI”. We also have some suggestions to improve the usability of the guidance.

We consider the most significant contribution of the document to be its identification and analysis of six different types of explanation: rationale, responsibility, data, fairness, safety and performance, and impact. The guidance also provides helpful clarification that some of these occur before processing takes place and apply to the whole system, while others occur after processing and apply to individual decisions. We believe that this analysis could usefully be applied to most systems involving complex flows or large amounts of data, whether or not they involve “Artificial Intelligence”. Considering rationale, responsibility, data, fairness, safety and performance, and impact throughout the design, development, implementation and operation of large-scale data processing systems should be good practice to improve their safety for operators and individuals alike.

To deliver its full benefit, we therefore consider that the guidance should both make this broader scope explicit and, within it, provide a clear, objective, definition of “AI”. The draft contains only a statement (on Part 1, page 4) that “AI is an umbrella term for a range of technologies and approaches that often attempt to mimic human thought to solve complex tasks. Things that humans have traditionally done by thinking and reasoning are increasingly being done by, or with the help of, AI”. This appears to accept that “AI” is often used purely as a marketing term, and to leave it to individual marketing departments to decide whether their product or service falls within the guidance. Those that do not wish to follow the guidance may simply reduce the prominence of the term “AI” in their marketing materials, or replace it by some other term. Conversely, unrealistic expectations may be raised among data subjects that anything labelled “AI” will provide the explanations described in the guidance, even when these may not be relevant or necessary.

Our principal recommendation is therefore that the ICO/Turing Institute adopt an objective definition that can be applied consistently and objectively to determine which systems are, and are not, “Artificial Intelligence”. We have found the definitions used by DSTL helpful: in particular that AI consists of “Theories and techniques developed to allow computer systems to perform tasks normally requiring human or biological intelligence.”

Alongside that objective definition, we consider that the broad applicability of the guidance would be made clearer by changing the order of the Legal Framework section in Part 1. At present it begins with the narrow set of legally-significant decisions covered by Article 22. The much wider group of data controllers who may be required to provide explanations under Article 5 Fairness might easily conclude that the guidance does not apply to them. To avoid this, we would suggest presenting the Article 5 requirement, then Article 22, then the situations where explanations are good practice.

Part 2 is currently a very long block of text and, as a result, hard to navigate. Some sort of graphical representation would help readers find the sections most relevant to their application. It would also be helpful to provide a graphical indication when discussing types of algorithm that are inherently non-explainable.

Part 2 also contains an important point, which we consider should also be raised in the introductory or management Parts, on the need to train the humans who will be working with AI as an assistant. This involves striking a tricky balance between unquestioning over-reliance on the AI’s recommendations and encouraging humans to substitute their own biased judgments. Both training on when to over-ride the algorithm, and support systems to ensure this facility is not (consciously or unconsciously) misused are likely to be needed.

Part 3, page 17 mentions the Article 21/Article 17 right to object, but without explaining its scope and nature. In our experience the application of this right to AI has been widely misunderstood by both data subjects and data controllers. In the most extreme form of this misunderstanding we have heard model builders assert that it requires them to keep all the personal data used to build a model, in case one person exercises their “right to object” and they are required to rebuild the whole model from scratch omitting that individual’s data. This guidance would be a good place to counter such high-risk practices by providing an authoritative statement of what the right does, and does not, require.

Finally, we welcome the recognition that for some applications of AI, “gaming the algorithm” is a positively desirable feature. Jisc has done considerable work on applying analytics to various fields in education and research. By examining data generated by teaching and research processes we hope not merely to predict likely outcomes, but to identify changes that can result in actual outcomes being better than those predicted. This presents new challenges throughout the lifecycle of such systems: algorithm developers must not just explain why a certain prediction was made, but also what needs to be done to improve it; users and regulators must understand that predictions that turn out to be inaccurate may actually be a sign that that the system is achieving its objective.

One of the key steps in preparing for the General Data Protection Regulation is to know why you are processing each set of personal data, and which of the six legal justifications applies: consent, contract, legal obligation, vital interest, public interest or legitimate interest. The Regulation significantly tightens the rules on when consent can be used, so data controllers may well have to look more closely at the other five. Each justification has different implications for when processing is allowed, the information you provide to data subjects, and the rights they can exercise, so the choice of the most appropriate one for each processing activity is likely to drive much of your subsequent compliance activity.

[Note that, as the Article 29 Working Party noted on page 8 of their Opinion on Consent , a single activity may involve a combination of justifications. For example on an interactive website such as this, the processing of user accounts required to post comments is “necessary for contract”; the processing of logs to detect misuse and security incidents is “necessary for a legitimate interest”; adding your name or a photo to the comment is by “consent”]

The Recitals to the Regulation contain quite a lot of information, and examples, of when each of the justifications is likely to be appropriate. This post attempts to gather that into a summary of the kinds of processing likely to be suited to each justification. In each case I’ve suggested a question as an initial check whether the justification is likely to fit your processing – these are just hints and carry no legal significance, nor are they the only questions you need to consider before making your choice.

Contract : covers processing that is necessary (i.e. there is no less intrusive way to perform the agreement) either for an existing contract to which the data subject is a party, or in preparation to enter into a contract at the data subject’s request. This includes exporting personal data where that is a necessary part of the contract. The EDPB has provided more detailed guidance on this basis . Note that in English law “contracts” are not limited to those on paper, so this justification is also likely to cover less formal agreements between a data subject and a data controller.
Q: “is this required to deliver an agreement with the individual?”

Legal Obligation : covers processing that is necessary to fulfil a legal obligation to which the data controller is subject. The obligation must be set out in EU or national law, must meet an objective of public interest and be proportionate. Examples include obligations in the fields of employment and social security, including those that require processing sensitive (now known as “special category”) personal data.
Q: “am I required to do this by law?”

Vital Interest : covers processing that is necessary to protect a vital interest (something essential to life) of the data subject or a third party. This can include necessary exports of personal data. If the data subject is capable (both in law and in practice) of giving consent to the processing, that justification should be preferred.
Q: “is this processing needed to protect someone’s life?”

Public Interest : covers processing (by both public and private bodies) that is necessary for some public interest. That interest must be set out in EU or national law, and any processing must be proportionate to it. The law may designate a particular data controller to carry out the function (“exercise of official authority”); this justification also covers other organisations that wish to share relevant information with those authorities. Examples include taxation; reporting crimes; humanitarian purposes; preventive or occupational medicine; public health; social care; quality and safety of products, devices and services; election campaigns.
Q: “is this processing needed for some legally-defined public purpose?”

Legitimate Interest : covers processing that is necessary for a legitimate interest of the data controller or a third party, provided that interest is not overridden by the interests and rights of the individual. The Article 29 Working Party provided detailed guidance on how to ensure this balance of interests is met . Processing is more likely to satisfy the balance if it is expected given the nature of the relationship between the individual and the data controller. This justification cannot be used when exercising official authority (use “necessary for public interest” instead). Examples include processing necessary to detect fraud or report criminal activity, to protect network and information security, for internal administration in a corporate group or not-for-profit organisation.
Q: “would this processing surprise or upset the individual, given our relationship?”

Consent : the only justification that does not include the word “necessary”. May therefore be used for data and processing that are not necessary, but should not be used for processing that is (use one of the “necessary” justifications instead). According to the Information Commissioner’s guidance consent is appropriate when the individual, not the organisation, is in control of processing. Individuals must have a genuine choice what (if any) personal data to provide and must be able to change their mind at any time. In particular consent should not be used where there is a clear imbalance in negotiating strength (employers and those exercising official authority are likely to have difficulty obtaining genuine consent), or when “consent” is made a condition of providing some other service (use “necessary for contract” instead).
Q: “is this processing truly optional for both the organisation and the individual?”

I’ve been asked a number of times whether GDPR affects the sharing of information between incident response teams. This slideset from a recent RUGIT Security meeting discusses how GDPR encourages sharing to improve security, and provides a rule of thumb for deciding when the benefit of sharing justifies the data protection risk.

Information Sharing and the GDPR v0-08

I was recently invited by EDUCAUSE to present a webinar on GDPR to their community of mostly North American universities and colleges. The number of participants indicates that European data protection law is a topic of interest. But the most common question was why, as non-EU organisations, they should care about GDPR. So I wrote a blog post, which EDUCAUSE has now published …

Reflecting on the scope chosen by Blackboard for our working group – “ Ethical use of AI in Education ” – it’s worth considering what, if anything, makes education different as a venue for artificial intelligence. Education is, I think, different from commercial businesses because our measure of success should be what pupils/students achieve. Educational institutions should have the same goal as those they teach, unlike commercial settings where success is often a zero-sum game. We should be using AI to achieve value for those who use our services, not from them. Similarly, we should be looking to AI as a way to help tutors do their jobs to the best of their ability. AI is good at large-scale and repetitive tasks – it doesn’t get tired, bored, or grumpy. Well-used AI should help both learners and teachers to concentrate on the things that humans do best.

Clearly there are also risks in using AI in education – there would be little for an ethics working group to discuss if there weren’t! The technology could be deployed for inappropriate purposes or in ways that are unfair to students, tutors, or both. The current stress on using AI only to “prevent failure” feels a bit close to these lines: if we can use AI to help all students and tutors improve then they won’t presume that any notification from the system is bad news. Getting this right is mostly about purposes and processes . However there’s also a risk of AI too closely mimicking human behaviour: poorly-chosen training sets can result in algorithms that reproduce existing human and systemic pre-conceptions; too great a reliance on student feedback could result in algorithms delivering what gives students an easy life, rather than what will help them achieve their potential. An AI that never produces unexpected results is probably worth close examination to see if it has fallen into these traps.

Computers work best when presented with clear binary rules: this course of action is acceptable, that one isn’t. However the rules provided by the legal system rarely provide that. Laws are often vague about where lines are drawn, with legislators happy to leave to courts the question of how to apply them to particular situations. As Kroll et al point out, when laws are implemented in AI systems, those decisions on interpretation will instead be made by programmers – something that we should probably be less comfortable about (p61). Conversely, laws may demand rules that are incomprehensible to an AI system: for example European discrimination law prohibits an AI from setting different insurance premiums for men and women even if those are what the input data demand. Finally, and particularly in education, we may well be asking AI systems to make decisions where society has not yet decided what actions are acceptable: how should we handle data from a student that tells us about their tutor or parent? when is it OK to for charities to target donors based on their likely size of donations ? when should a college to recommend an easier course to a borderline student?

Education Technology have just published an article I wrote (though I didn’t choose the headline!) on how security and incident response fit into the General Data Protection Regulation . It aims to be an easy read: if you want something more challenging follow the “incident response protects privacy” link to get the full legal analysis.

Most of us are familiar with the recorded messages at the start of phone calls that warn “this call may be recorded for compliance and training purposes”. Some may recognise it as meeting the requirement to notify callers under the snappily titled Telecommunications (Lawful Business Practice) (Interception of Communications) Regulations 2000 . But the data protection implications of call recording are perhaps more interesting.

Any conversation involves two people, so organisations need to think of two groups of data subjects before recording calls: staff and callers. For staff, the requirements are set out in Part 3 of the Information Commissioner’s Employment Practices Code :

• There must be a clear purpose to the recording (page 60 mentions examples “to listen to as part of workers training, or simply to have a record to refer to in the event of a customer complaint about a worker”);
• An impact assessment should be carried out, including identifying any possible alternatives;
• Unless an exception applies, staff must be informed of the recording and the reason(s) for it.

From the caller’s side, the organisation needs to think about the legal justification for processing, the rights that callers will have over their personal data, and how long the recording will be kept. A few industries may have a legal obligation to record calls but normally – as the ICO’s examples indicate –this will be done to support a legitimate interest of the organisation. This justification involves three tests: is the purpose of processing legitimate, is the processing necessary to achieve that purpose, and can the risk to the data subject be reduced to a level where it does not override the organisation’s interest in the processing.

For example, identifying areas where helpdesk staff could benefit from training seems to be recognised by the ICO as legitimate, and listening to recordings is likely to identify needs that might not be discovered by other approaches. Reducing risk to callers will require controlling access to recordings, ensuring that those with access only use recordings for the specified purpose, and deleting recordings as soon as they have been checked. To improve service to its customers the organisation should want to do that as soon as possible after the call, even if it weren’t also a requirement under data protection law.

However, using a recording as an example in a training course seems much harder to justify under these criteria. If the caller’s or recipient’s voice is played back there is a risk – which the organisation cannot control – that trainer or trainees will identify them, either during the course or next time the individual calls. The same purpose can be as well, or better, achieved by using an anonymised transcript as an illustration, role-play, or script voiced by someone else. And an anonymised script doesn’t need to be deleted under a retention requirement or disclosed under a subject access request. However the balancing test still needs to be applied to the anonymisation process to protect the individuals’ interests – if they use distinctive phrases or styles of speech then the risk of identification from a transcript may still remain too high for the use to be acceptable.

Further legal and practical details can be found in an article from Wright Hassall

To mark one year to go till the General Data Protection Regulation comes into force, we’ve published an article on “ How Universities and Colleges Should be Preparing for New Data Regulations ” on the Jisc website.

Jisc responded to the DCMS consultation on implementing the Research provisions of the GDPR into UK law .

The exemptions from certain obligations and data subject rights contained in section 33 of the Data Protection Act 1998 have been vital in enabling long-term research studies, including in health and social sciences, while ensuring the protection of individuals whose data may be used in those studies. We consider it important for the continuation of those studies and datasets that the same conditions continue to apply under the General Data Protection Regulation (GDPR). Any requirement on researchers to change their existing practice, or any new right of data subjects to prevent or place conditions on processing, could damage ongoing research and make future work unviable. We therefore encourage the Government to use the derogations in Article 89 to reproduce, so far as possible, the current Data Protection Act s.33 regime.

In particular we welcome the European Data Protection Supervisor’s recent suggestion ( Opinion 6/2017 , p.16) that Member States should use Article 89 to provide “additional, limited and specific exceptions in the ePrivacy Regulation, for example … to allow processing for the purposes of scientific research”. As provider of Janet, the UK’s National Research and Education Network, Jisc both benefits from and supports advanced network research in the UK. We therefore encourage the Government to use the Article 89 derogations to provide a legal regime that supports such research while providing appropriate safeguards for users of electronic communications systems.

We also note that Article 85 requires the Government to provide exemptions or derogations to several sections of the GDPR in order to protect the right to freedom of expression and information for the purposes of academic expression. The dissemination and publication of research represent speech of very high importance to society, which should be awarded an appropriately high level of legal protection. Subject to conditions to protect individuals (again, section 33(1) of the Data Protection Act 1998 provides an appropriate model) we consider that academic expression requires and deserves the same level of protection as currently provided for journalistic expression by section 32 of the Data Protection Act 1998 . We therefore encourage the Government to use its obligation under Article 85 to provide that protection.

Additional Question – cost impact

Probably the greatest cost impact on UK organisations will be if, after leaving the European Union, they are required to include Model Clauses in, or obtain permission from EU regulators for, every contract with an EU counter-party. We therefore consider it essential that the UK seeks and obtains a declaration of adequacy under Article 45(3) of the General Data Protection Regulation , and would be concerned if derogations or other measures in the UK’s implementation risked reducing the likelihood of obtaining such a declaration.

We’ve just responded to the ICO’s request for feedback on Profiling under the General Data Protection Regulation . Thanks to the work we’ve already done on Learning Analytics , we were able to include several examples of good practice in that area, including the Code of Practice we developed with universities and the National Union of Students.

Recently I’ve been doing some work with Niall Sclater on how education organisations might inform students about the use of learning analytics, and when they might seek students’ consent. The resulting blog post is at https://analytics.jiscinvolve.org/wp/2017/02/16/consent-for-learning-analytics-some-practical-guidance-for-institutions/

These are Jisc’s comments on the Article 29 Working Party’s Guidelines on the Right to Data Portability (WP242).

Jisc is the UK’s expert body for digital technology and digital resources in higher education, further education and research. Since its foundation in the early 1990s, Jisc has played a pivotal role in the adoption of information technology by UK universities and colleges, supporting them to improve learning, teaching, the student experience and institutional efficiency, as well as enabling more powerful research.

Our incident response team, Janet CSIRT, frequently helps universities and colleges to deal with the consequences of security breaches of third-party internet services, recently including TalkTalk, LinkedIn and Yahoo!. These databases often contain personal details of students and staff who happen to be customers of such services: once these have been accessed or disclosed by hackers there is little that can be done to remedy the damage to those individuals’ privacy and data protection rights. We are therefore concerned that, by suggesting (contrary to data protection regulators’ existing security recommendations) that all data controllers should make personal data stores, including their customer and employee databases, accessible on-line, the Working Party’s guidance on the GDPR Portability Right will greatly increase the number of such security breaches and the harm they cause.

Since the primary aim of the portability right is now stated (pp4&5) as “to facilitate switching from one service provider to another, … enhancing competition” and “preventing lock-in”, we believe that a technological implementation of the right should only be required of the small minority of data controllers that raise competition concerns. The Working Party’s guidance should explicitly recognise that for many data controllers on-line access to personal data will be an unjustified security risk, and other ways of providing the right will be more appropriate.

Risks of Internet-connected customer/employee databases

Good security practice holds that personal data should not be placed on computers accessible from the Internet unless this is essential. Systems holding such data should normally only be accessible to a limited number of trusted staff. The UK Information Commissioner’s Protecting personal data in online services: learning from others notes the importance of segregating internal and externally-accessible systems using firewalls and de-militarised zones (para 108) and reducing the number of people granted external access to personal data as far as possible (para 47).

Providing “download tools and APIs”, as the Working Party’s guidance recommends for all data controllers (p3), breaks this security model. Such tools will require systems holding relevant personal data to be accessible over the Internet. This will include every customer and employee database since these, being “necessary for a contract”, are subject to the portability right (p7). Furthermore such access must be available to all data subjects, in case they wish to exercise that right: it can no longer be limited to authorised, trusted, staff.

The number of reports of compromises of existing on-line customer databases indicate how hard it is to provide such access in a secure fashion. The businesses suffering these breaches are those – communications providers, social networks, etc. – where providing customers with access to their data is a core business function. These already have a business incentive to spend money, effort and skill on designing and maintaining secure systems and providing customers with appropriate authentication systems, including two-factor authentication. However the majority of data controllers affected by the Working Party’s guidance will have only a regulatory incentive to create their new remote access systems – in business terms these will be a pure cost. It seems very unlikely that these data controllers will achieve or maintain the same level of security. This means the rate of compromise of systems created solely to support the portability right is likely to be much higher. The impact of such breaches may also be greater as, by design, they will give attackers access to all the personal data the organisation holds by consent or contract.

Risk of idle accounts

A common way for online systems to be compromised is through unused accounts. If these are left with default passwords then, as the Information Commissioner notes (para 129-133), they provide easy access to unauthorised attackers. Even if individual passwords are set, accounts that are not used provide attackers with an opportunity to guess passwords with little risk of detection. The current rate of subject access requests suggests that the majority of data subjects will never exercise their portability right, so tools and APIs are likely to provide a rich source of opportunities for such attacks. According to the Information Commissioner (para 44): “If you have services which are publicly accessible and are not being actively used, you are exposing a range of potential attack vectors unnecessarily.”

Risk of user deception

Even if the data controllers’ systems for exercising the right to portability can be kept secure, there is a global criminal industry dedicated to persuading users to disclose their passwords. Successfully phishing a data subject’s portability password may only give access to that individual’s data but, again, the portability right ensures significant harm as all the relevant data held by the data controller will be disclosed.

Even the cost of a phishing campaign may not be necessary, as many passwords chosen by users will be trivial for attackers to guess . Even for the minority of data controllers that are capable of providing a technically secure portability interface, the benefits to attackers seem likely to be far greater than to data subjects.

Recommendation

Creating and maintaining a secure download tool or API will be a challenging software development task that only a small proportion of data controllers will have the skills to achieve.

To avoid creating many opportunities for large-scale breaches of data protection and privacy rights we recommend that:

1. Data controllers should inform data subjects of their portability right;
2. Data controllers should only enable processes or systems to support that right for individual data subjects when an individual, having been securely authenticated, requests it;
3. Only data controllers who already provide secure on-line access to customer data as a business function (e.g. banks, social network providers) should consider providing tools or APIs to exercise the right. All other data controllers should handle portability requests in the same way as their current subject access processes. In most cases this will involve manual processes by trusted staff, not direct access to databases by data subjects.
4. The Working Party’s guidance should highlight the importance of data security, not leave it till the very last page, and emphasise that for most data controllers a manual implementation of the portability right will be the best way to achieve this.

Finally, we note that only a tiny minority of data controllers will raise the competition and lock-in issues that the portability right is intended to address. The European Data Protection Supervisor’s 2014 paper identified concerns only with “free services paid for by personal information. Many of these will already be in the group that could provide technical portability tools as a natural extension of their existing business access to customer data. For all other data controllers, a technical implementation of the portability right will create great risks to the security of personal data with little or no benefit to competition.

At the request of the Research Councils UK e-Infrastructure group, Janet established a working group from 2013-2016 to support those providing and using e-infrastructure services in achieving an approach that both protects services from threats and is usable by practitioners. More detail about the group can be found in the Terms of Reference .

The Working Group published the following papers:

• E-infrastructures: Access and Security (summary paper) (Jan 16)
• Federated Authentication for e-Infrastructures (Sep 14)
• Technical Security for E-infrastructures (Nov 14)
• Authorisation/Group Management for e-Infrastructures (May 15)
• Policies for e-Infrastructures (Jan 16)
• Accounting and e-Infrastructures (working paper) (Nov 16)

Abstract : Reconciling big data techniques with a legal approach relying on prior consent has proved difficult. By definition, when organisations collection personal information for data-led investigations they do not know what the results and impact of their processing will be. This paper suggests how other parts of the current European data protection framework can provide more effective protection for individuals, better guidance for big data users, and more effective tools for regulators. Splitting big data into collection, analysis and intervention stages, and applying appropriate legal provisions to each, could guide the development of privacy-respecting big data techniques that support the needs of individuals, businesses and society.

Published by the Journal of Information Rights, Policy and Practice

DOI: https://doi.org/10.21039/irpandp.v1i1.9