Conscious Risk Framework for Governance, Ethics and Trustworthiness of AI
There are many definitions of AI, and many AI technologies with differing capabilities and uses moving us closer to what is called a “sociotechnical society,” in which people and machines are more integrated. Specifically, Generative AI (GenAI) refers to AI techniques that learn a representation of artifacts from data, and use it to generate brand-new, completely original artifacts that preserve a likeness to original data. These artifacts can serve benign or nefarious purposes. Due to AI hype, organisations are quickly building and deploying solutions without much consideration being given to ethical impacts/aspects. As AI becomes a part of everything (software, hardware, consumer devices) and autonomously communicates with other AI, new policies and governance are emerging to protect consumers, citizens and businesses. Organisations developing AI solutions now need to proactively make sure that the data is correct and integrated properly and that they have the right types of data and datasets.
Managing Risks of AI (or GenAI)
Since GenAI amplifies the human type of intelligence that underlies every human activity, providing superhuman capabilities in a range of activities, it also creates brand new risks and it amplifies many existing risks. And this is only complicated further by the output of GenAI often being unpredictable and unexplainable. Organizations should therefore create their own risk taxonomy specifically for GenAI. Since GenAI risk will be so all-encompassing, it is key that organizations set risk tolerance levels for the various types of GenAI risk. Some risks will be very hard for organizations to meaningfully mitigate.
Even if GenAI is “only” used as intended, it will disrupt multiple industries. For instance, the educational services industry is suffering because of students turning to ChatGPT. Generative AI fuels new types of competition with smaller companies getting superpowers and being able to take on bigger competitors. It will also lead to increased ease of entry for new disruptive startups in many industries. Also since GenAI leads to such productivity improvements, it will be adopted widely — soon perhaps in every Microsoft office product. However, since at least the current generations of GenAI tend to “hallucinate,” make up incorrect information and present it in a convincing way, there is great risk of inaccurate, incorrect AI-informed decision making.
The risks associated with ChatGPT and similar open-access LLM tools can come from different sources. As open-access platforms, LLMs may be used by actors outside of the organization in ways that impact it, as well as within the organization in both governed and ungoverned ways.
Use by Actors Outside the Organization
With the democratization of generative AI technology, organizations face a variety of vulnerabilities from outside the organization. LLMs can be used to create mis- or disinformation, such as counterfeit emails or documents, as well as convincingly spoofing a company’s brand. Organizations could find their brands or IP impersonated for malicious purposes, opening up significant reputational risk. Beyond attempts to sabotage an organization’s reputation, malicious actors could also attempt financial fraud or identity theft. ChatGPT in particular is an open-access platform with wide availability, including to malicious actors. Its text-generation capabilities can be used to increase the volume and sophistication of cyberattacks. Broad spamming as well as targeted spearphishing messages can be made easily using ChatGPT, including convincing efforts at business email compromise.
Ungoverned Use by Employees
ChatGPT can be readily accessed by employees, who could use the tool’s capabilities to generate or edit text and/or lines of code to speed up workflows. The possibility of ungoverned employee use of LLM platforms points to significant information governance risk, as employees may disclose confidential, sensitive or proprietary information when using the tool. Inputs entered by users are incorporated into ChatGPT’s training data and could, therefore, become fodder for future outputs generated by third parties. These issues have already arisen at Amazon, where company lawyers warned employees not to share confidential information with ChatGPT, citing text generated by ChatGPT that “closely” resembled internal company data. Microsoft and PWC have issued similar warnings to employees. Combined with access to ChatGPT by malicious actors, ungoverned use also presents serious privacy risks for any data shared with the model. Training data extraction attacks can be used to prompt the system to generate verbatim records of data points in its training set.
Incorporation of LLMs into Business Processes
Even where LLM tools such as ChatGPT are purposefully adopted and governed by the organization, many risks are still present. One major pitfall of ChatGPT is its potential to generate inaccurate outputs. Sometimes these inaccuracies are the result of “hallucination,” where the model presents false information that seems correct. ChatGPT’s hallucination rate has been estimated at about 20%. Reliance on LLM systems could expose organizations to reputational risk in cases of high-profile errors. Google provides an apt illustration, as a factual error surfaced in its promotional materials announcing the imminent release of its Bard platform, a competitor to ChatGPT using similar technology. Shares of Alphabet fell 8% on the day of the announcement. Errors and hallucinations point to a need for thorough human review of outputs by subject matter experts, potentially undercutting the time-saving advantages of the technology.
Regulation of AI (or GenAI)
Regulatory challenges are set to become more complicated, as the EU’s proposed AI Act works its way through the legislative process. The act is expected to build on General Data Protection Regulation (GDPR) requirements by focusing on privacy and ethics implications of AI systems. The scope of these AI regulations is broader - creating categories of unacceptable use based on new advances in generative AI. Unacceptable uses include “social scoring” or using the technology to rank and assess people based on their trustworthiness or value to society. It will require organizations to regularly assess and report on the role of AI in their decision making. Most organizations will integrate generative AI tools rather than build their own. Here are some tactical tips for safely integrating generative AI in business applications.
Trustworthy AI
Trustworthiness is a prerequisite for people and societies to develop, deploy and use AI systems. Trust in the development, deployment and use of Intelligent systems concerns not only AI’s inherent properties, but also the qualities of the socio-technical processes that are created or managed by AI. A trustworthy approach is key in driving revenue and profitability, by providing the foundation upon which all those affected by AI can trust that their design, development and use are lawful, ethical and robust. In the world of AI, people want to work with organisations they can trust with their personal data and that will use algorithms to help rather than manipulate. If GenAI is to last, the trustworthiness issue must be resolved. Currently, generative AI tools creating images often make mistakes — for example, these tools are typically incapable of creating images that include words or coherent text of any kind. And tools like ChatGPT often “hallucinate,” and come up with incorrect responses. To make their output more trustworthy, these tools will need to make improvements in four key areas:
Focus on transparency - It will be hard for the machine learning to be transparent. Even the engineers creating generative AI models don’t fully know how the results work. But you can be transparent about the origin of the output. Make sure the output is always recognizable as machine generated. This could be done by adding a notification in textual output, a watermark in images and a form of vocal identification in generated voice output.
Mind bias and accuracy - This is increasingly hard as generative AI training algorithms may not be very transparent and explainable. Bias and accuracy can be improved in a different way though — through the use of generative AI tools in the real world. This is called “reinforcement learning from human feedback.” The model improves itself through real-world interaction. However, be very aware of active trolling taking place. Bad actors will try to mess up your generative AI tools. Another way to counteract bias is through feeding the model corrective data. This will be a new skill to develop in the years to come. You could see this as “behavioral therapy” for AI.
Privacy and security - Be very careful in feeding your intellectual property or other types of private data to a public LLM. Third parties may learn and use the data that belongs to you. The interesting thing is that generative AI itself may even be part of the solution. Generative AI is often used to create synthetic data.
Promote tolerance for imperfection -AI will improve, but it will make mistakes. Educate designers and developers on how to responsibly use generative AI tools. Embrace regulation that will help people use generative AI in appropriate ways. One best practice that has emerged is to keep generative AI systems in beta for a very long term — a clear indication that one should expect imperfection.
Data & AI Ethics
AI ethics focuses on the ethical issues raised by the development, deployment and use of AI. AI systems should follow human-centric design principles and leave meaningful opportunity for human choice. This means securing human oversight over AI work processes. The use of AI should never lead to people being deceived or unjustifiably impaired in their freedom of choice. In the world of AI, it is about the organisation and employees who built the AI solution putting themselves in the shoes of the person who will be using the AI solution.
The quality of the data sets used is paramount to the performance of AI systems. Data sets used by AI systems (both for training and operation) may suffer from the inclusion of inadvertent historic bias, incompleteness and bad governance models. Feeding malicious data into an AI system may change its behaviour, particularly with self-learning systems. Processes and data sets used must be tested and documented at each step such as planning, training, testing and deployment. Identifiable and discriminatory bias should be removed in the collection phase where possible. The way in which AI systems are developed (e.g. algorithms’ programming) may also suffer from unfair bias. This could be counteracted by putting in place oversight processes to analyse and address the system’s purpose, constraints, requirements and decisions in a clear and transparent manner.
Explicability is crucial for building and maintaining users’ trust in AI systems. This means that processes need to be transparent, the capabilities and purpose of AI systems openly communicated, and decisions – to the extent possible – explainable to those directly and indirectly affected. Consumers and businesses will require transparency of AI solutions as a “right to explanation” in how an AI-derived decision was made and proof that it was made in an unbiased way.
Velocity and Materiality of AI Risks
Research and development with AI and GenAI models is ongoing, and we’re likely to see significant advancements and improvements in the capabilities of these models in the future. Nonetheless, given the velocity and materiality of ChatGPT and LLM risks, there is an urgent need to identify associated risks, map controls and provide assurance guidance. Following are some of tactical tips for developing risk frameworks and overseeing the execution of AI audits:
Assess how the organization monitors for unauthorized software use.
Assess whether the organization is developing guidelines to protect the company’s intellectual property.
Evaluate the use of LLMs within third-party IT platforms.
Review information security policies to be sure they explicitly preclude unauthorized software use and ensure effective training exists to make employees and third parties aware of this prohibition.
Create an AI auditing framework to provide assurance over LLM use that identifies risks and related controls associated with LLM operations, security and trustworthiness.
Collaborate with the business, particularly second-line functions such as legal and compliance, to ensure policies and frameworks address all relevant areas of the business and can be easily implemented throughout the business.
Consult with leaders throughout the organization as well as IT operations to conduct an inventory of LLM use and associated risks and controls.
Highlight the risks of LLMs and their impact on strategic priorities in board and steering committee discussions.