# data[].attributes.attachment_link.provider_name
Carnegie Endowment for International Peace

# data[].attributes.attachment_link.readable_text
Introduction

Significant technological advances are being made across a range of fields, including information communications technology (ICT); artificial intelligence (AI), particularly in terms of machine learning and robotics; nanotechnology; space technology; biotechnology; and quantum computing to name but a few. These breakthroughs are expected to be highly disruptive and bring about major transformative shifts in how societies function.

The technological advances in question are driven by a digital revolution that commenced more than four decades ago. These innovations are centered on the gathering, processing, and analyzing of enormous reams of data emerging from the information sciences with implications for countless areas of research and development. These advances promise significant social and economic benefits, increased efficiency, and enhanced productivity across a host of sectors.

Technology's Disruptive Potential

But there are mounting concerns that these technologies and how they are used will pose serious challenges, including labor force dislocations and other market disruptions, exacerbated inequalities, and new risks to public safety and national security. The technologies are mostly dual use, in that they can be used as much to serve malicious or lethal purposes as they can be harnessed to enhance social and economic development, rendering efforts to manage them much more complex. ¹ Relatively easy to access and use, most of them are inherently vulnerable to exploitation and disruption from both near and far.

In parallel, geopolitical tensions around the world are growing, and most countries increasingly view these technologies as central to national security. The potential for misuse is significant. And greater economic integration and connectivity mean that the effects and consequences of technological advances are far less localized than before and are liable to spread to countries and industries worldwide.

Technological innovation is largely taking place beyond the purview of governments. In many cases, the rate of innovation is outpacing states' ability to keep abreast of the latest developments and their potential societal impacts. And even if one or a handful of national governments devise policies for managing these effects, the global reach of many emerging technologies and their impacts requires new approaches to multilateral governance that are much more difficult to agree on. A greater number and variety of actors must be involved to initiate, shape, and implement both technical and normative solutions.

Yet, like governments, many of these other actors do not have (or simply do not invest in) the means to consider the broader, cross-border societal implications of their investments, research, and innovations. When they do so, identifying the most relevant or effective policy-oriented or normative-focused platforms to discuss these implications can be challenging, not least because existing platforms sometimes do not consider the variety of actors implicated, the cross-border reach of the technologies in question, and the different value and political systems at play. This is particularly the case when technologies are designed for profit-making alone and their trajectories are entirely dependent on market forces.

Today's technological advances are deemed disruptive not only in market terms but also in the sense that they are "provok[ing] disruptions of legal and regulatory orders" and have the potential to "disturb the deep values upon which the legitimacy of existing social orders rests and on which accepted legal and regulatory frameworks draw." ²

Complex, dynamic frameworks already govern some fields of technology. For instance, cyberspace is governed by an amalgamation of existing international law; as well as an ever-growing, complicated array of political agreements; technical standards and protocols; trade, business, and human rights standards; research principles; national regulations; and self-regulation in the private sector. These legal and normative touchstones are underpinned by existing norms, values, and principles. They span different policy areas and issues, enlist a dizzying host of actors, and straddle numerous international regimes. What is more, they are complemented by a growing body of confidence- and capacity-building measures and efforts aimed at enhancing security and resilience. ³ Yet important elements of this regime remain contested, even as new risks and vulnerabilities-such as those related to the Internet of Things, AI, and other technologies-emerge and need to be managed.

Looming Governance Dilemmas

The World Economic Forum's (WEF) founder, Klaus Schwab, has described technological advances (for better or worse) as a "revolution" due to their "velocity, scope, and systems impact." ⁴ In discussing what he has dubbed a "Fourth Industrial Revolution," Schwab emphasized a number of emerging policy and normative issues. Similarly, a 2019 paper prepared for the WEF stressed the need to "transform traditional governance structures and policy-making models" and adapt more "agile" methods of governance. ⁵

But responding to the attendant risks and challenges will not require just exploring new governance structures, tools, and processes. This task calls for a deeper understanding of the social, cultural, economic, and geopolitical contexts in which policy, norms, and regulations are crafted as well as a firmer grasp of the overarching questions of power and conflict that shape humanity's relationships with technology. Moreover, significant public engagement is sorely needed, since governments and companies cannot-and should not-expect to resolve these dilemmas alone.

Some of the key challenges include:

Identifying key principles and values: Relevant actors must articulate a vision of the principles and values-such as equity, equality, inclusivity, responsibility, transparency, and accountability - that might be negatively impacted by certain technological advances and how those values might best be protected. This task will require new thinking on how to ensure certain technologies (like predictive algorithms, biotechnology, or technologies allowing space resource exploitation) do not exacerbate existing socioeconomic inequalities, or how best to respond to new encroachments on personal privacy involving the body and the brain. Certain forms of research (like some aspects of biological engineering) may need to be restricted.

Determining appropriate principles and values requires more than a shared understanding of how technological innovations are evolving, how they have been designed, and how they might be used. Such normative reflection also calls for a clear sense of the different values and principles held by various communities worldwide as well as the means and actors by which such values and principles should be protected. It also necessitates going beyond the mere articulation and proliferation of principles to the practical application and acknowledgment of associated challenges and limitations.

Engaging stakeholders effectively: Legislators, regulators, private business leaders, researchers, and civic actors need to be ready to respond more responsibly and effectively to the effects and consequences of technological advances and the potential societal risks they pose. Ongoing experiments with new rules, principles, and protocols to deal with concrete relevant policy issues are certainly noteworthy, one example being the WEF's "agile" governance initiative. ⁶ Yet implementing these responses at scale and applying them to a host of cross-border socioeconomic and security challenges will be a complex challenge. This task likely will require a blend of approaches, increased spending in independent public research, and the involvement of many actors other than just states. To this end, it is vital that relevant actors delineate domestic and international responsibilities to determine which technology-related challenges should be addressed at home and which problems require cross-border coordination and cooperation.

Broadening existing platforms of multilateral engagement: Clarifying how various actors (and not just states) can responsibly contribute to the work of multilateral mechanisms focused on how existing international law or political norms apply to state uses of certain technologies is imperative. These platforms include the various United Nations (UN) Groups of Governmental Experts (GGEs), as well as other specialized working groups concerned with matters of technology (including ICT, machine learning, autonomous weapons, biotech, and space technology) and international security. Such efforts can also advance ongoing discussions about how to ensure that multilateral mechanisms are agile enough to determine when new global norms or rules (binding and/or nonbinding ones) are required to manage technology-driven challenges and risks and constrain certain applications of these technologies or specific behaviors by states and other actors.

Craft suitable regulations: Fresh approaches to policy and regulation are also needed. For instance, advances in ICT, machine learning, biotechnology, and the convergence of these technologies are already driving multifaceted discussions on the focus and rationale of relevant policies and the most suitable type of regulation (precautionary, preventive, reactive, or a combination of all three). Questions also abound around whether to opt for hard regulation, soft policy initiatives (such as guidelines, certification procedures, and labeling schemes) that stop short of binding regulations, or the current trend of self-governance measures by private entities. As noted above, a key question is how to ensure these regulatory solutions are fit for purpose, and whether they should be coordinated nationally or internationally. Given the cross-border effects of the technologies at play as well as the growing convergences between them, uncoordinated national rules and policies will likely be ineffective.

Enhance transparency, oversight, and accountability: Finally, new policy and regulatory approaches will require greater investment in transparency, oversight, and accountability mechanisms. This will necessitate agreeing on the nature of national regulatory oversight bodies and determining whether they should be public, private, or of a mixed composition. This task should also entail ensuring that technology companies and organizations accept greater scrutiny. For example, tech companies should heighten internal monitoring and external reporting of their self-regulatory initiatives, provide appropriately insulated, publicly funded researchers with safe access to their data, and, above all, ensure that accountability covers all aspects of the supply chain and that both the direct and indirect costs (such as labor and environmental costs) of the technologies in question are clearly understood. ⁷ Agreeing on what is ethically acceptable in terms of industry's role in funding and participating in oversight mechanisms (such as ethics councils and advisory boards) is an equally important lever for injecting legitimacy into some of these processes. Such scrutiny would also help identify remaining gaps and engage more actors to help gauge if and how a certain technology or its application should be regulated.

Challenges to Effective Governance and Coordination

The international environment is hardly conducive to discussions of how best to coordinate responses to the complex, cross-border dilemmas emerging around new technologies. In some corners, existing multilateral platforms are increasingly perceived as unsuitable for resolving these challenges. The international community is notoriously slow at adopting new rules and institutions to deal with new challenges, and the quandaries posed by questions of national sovereignty and democratic legitimacy are persisting. In contrast, corporate actors appear to be racing ahead, intent on shaping the "science, morality and laws" of new technologies such as AI, with limited public debate underpinning or guiding their efforts. ⁸ Many of these same companies and the technologies they produce or exploit are increasingly viewed as instruments of state power, a fact that only adds to these sovereignty and legitimacy-related questions.

Meanwhile, growing strategic competition between the world's leading powers, especially in high-tech sectors, does not bode well for multilateral efforts to respond cooperatively and effectively. Such a competitive landscape is contributing to regulatory fragmentation and will likely delay much needed normative and regulatory action. This potential impasse places strains on existing efforts and could further delay the attainment of pressing social and economic objectives such as the 2030 UN Sustainable Development Goals, which are already under stress. Moreover, the resulting trust deficit between countries poses a significant threat to international peace and security, one that existing political institutions are not necessarily prepared to handle.

Throughout history, new challenges (including those relating to technology and governance) have generally opened new opportunities and channels for cooperation. Today is no different, although the challenges at hand are highly complex and are emerging at a time of systemic political change and a rising sense of conflict and crisis. More meaningful dialogue and cooperation-however difficult-on how technological developments are affecting societies and the uses and applications of technology generating the most disruption and contestation are urgently required. Such an approach would likely afford greater legitimacy to emergent governance efforts, while also tethering them to the common good. As one expert noted in the context of the WEF's Fourth Industrial Revolution initiative:
The moment you start saying that technology is the most powerful force today, then what you have done is argue that people have a right to shape how technologies are developed and used. In practice, this means consciously trying to shape technological trajectories, it means setting research agendas, it is to direct foreseeable technologies, to articulate uses that might benefit the majority and not the few; placing technology inside of democratic politics.⁹
Developed on the basis of interviews with experts and extensive research, this analysis assesses four areas of technology around which governance dilemmas are evident and discusses the emergent responses. These four areas include ICT, AI, biotechnology, and space technology. The decision to focus on these fields was informed by their disruptive character, the potential national and international security risks associated with their dual-use character, and the growing degree of state competition emerging around them. Importantly, this focus is also informed by the fact that these same issues are underscoring the urgent need for greater cooperation and strengthened governance frameworks to manage the associated risks. Greater understanding of these efforts can help inform how relevant actors approach current-and prepare for new-technology-related governance challenges.

Information and Communications Technology

While it is not a new area of technology, there is no doubt today that continuous advances in networked computing and other aspects of ICT are converging with advances in other technological fields, greatly increasing human dependence on these digital tools. ¹⁰ Governments around the world are establishing new institutions, identifying the policy implications of this growing digital dependence, and developing integrated frameworks for whole-of-government approaches to manage the resulting economic and societal transformations. Despite the associated benefits, humanity's growing dependency on ICT continues to present significant risks. Cybersecurity and the stability of ICT systems more generally have become top policy priorities.

The Current Normative Landscape

The "regime complex" governing ICT stems from multiple sources: the existing body of international law, political agreements, and voluntary norms; an ever-growing body of technical protocols and standards on internet governance managed by technical groups and research communities; national regulations; self-regulation by the private sector; and other forms of nonbinding soft law. ¹¹ These norms span different topical areas, straddle numerous international regimes, and require the engagement of multiple actors.

These norms and governance mechanisms aim to strengthen cybersecurity in national and global terms; enhance international security and stability more broadly as well as general human well-being; and promote responsible behavior by drawing on existing rules, values, and principles. For instance, human rights, privacy norms, and the freedom of opinion and expression have been bolstered by a norm upholding the right to privacy in the digital age and the confirmation by the UN Human Rights Council that human rights apply online as they do offline. ¹²

Many states, however, do not appear to be upholding these norms, and beyond basic privacy and human rights questions, there are increasing concerns surrounding the human costs of cyber operations, notably operations that affect healthcare and industrial systems or those that can generate systemic effects. ¹³ Where cyber crime is concerned, progress is equally slow. States have not managed to agree on an international framework to counter cyber crime, although many states are leaning on the existing Council of Europe (Budapest) Convention and bilateral treaties as they adopt national cyber crime legislation and cooperate on this issue.

As for international peace and security, the work of a series of UN GGEs (five to date) has reaffirmed the applicability of international law, including the UN Charter, to cyberspace and has recommended a number of voluntary, nonbinding political norms aimed at encouraging states to use ICT responsibly. ¹⁴ The norms seek to promote restraint, best practices, and other forms of positive behavior.

Specifically, many of the norms draw from existing principles of international law and address several facets of responsible use of ICT by states, including the importance of recognizing the challenges of attribution, preventing the use of a state's territory for the commission of internationally wrongful acts, not conducting or knowingly allowing activity that damages critical infrastructure and essential services to the public, safeguarding national computer emergency response teams (CERTs) and their systems from malicious cyber activities, responding to requests for assistance by other states, and reporting and sharing information on vulnerabilities. The norms also include ensuring the integrity of supply chains, upholding human rights online and privacy rights in the digital age, and enhancing cooperation and information sharing with regard to terrorist and other criminal cases that involved cyber operations. ¹⁵ Subsequently, the UN General Assembly recommended that states "be guided in their use of ICT" by the 2015 GGE report. ¹⁶

Several international, regional, and other specialized bodies have since endorsed the GGE recommendations. Nonetheless, efforts to advance this work stalled when a new GGE (established in 2016) failed to produce a consensus report mainly due to disagreements on international law and the future work of the group. As the topic has become more politicized, in December 2018, the UN General Assembly's First Committee agreed to establish two new parallel processes: an Open-Ended Working Group involving the entire General Assembly and a new twenty-five-member GGE. ¹⁷

Other intergovernmental organizations complement these UN-led efforts. For example, regional organizations such as the Association of Southeast Asian Nations (ASEAN), the European Union (EU), and the Organization of American States (OAS) all have endorsed the GGE norms. So, too, have the Group of 20 (G20) and international financial institutions. Additionally, the G20 has developed guidance on strengthening the cyber resilience of the financial system and has sought to foster a norm aimed at protecting the integrity of financial data. ¹⁸ Some specialized organizations, such as the International Atomic Energy Agency, have been actively developing capacity-building tools, guidance, and standards including, for instance, resources for protecting the computer and information systems of nuclear infrastructure. ¹⁹

In national terms, states are under increasing pressure to ensure that government agencies, cybersecurity firms, and researchers discover and disclose cyber vulnerabilities in a more timely fashion and prevent these vulnerabilities from being illicitly traded or otherwise misused. In some states, these efforts are evolving into vulnerability equities processes (VEPs) or coordinated vulnerability disclosure mechanisms. While a principal aim is to strengthen transparency and oversight of government use of discovered zero-day vulnerabilities, there are concerns that such processes are bureaucratically complex and expensive and that they might remove pressure on companies to produce more secure products and services. Moreover, explicit processes for managing vulnerabilities might be seen as legitimizing government hacking. ²⁰ Yet, realistically, governments will unlikely eschew all use of vulnerabilities, so imposing greater due diligence, transparency, and oversight in this domain would be more beneficial than not doing so.

On a related note, another important factor is designing more secure ICT products and systems so states do not use vulnerabilities in the technology products and services that underpin people's daily lives against citizens and other states. The costs to the global economy are certainly significant, and there are growing concerns about the potential human costs. ²¹ The principle of security by design (see below) has been gaining currency among many engineers and entrepreneurs.

The Cybersecurity Act follows on the heels of the EU's General Data Protection Regulation (GDPR), which seeks to bolster EU citizens' data privacy and harmonize data privacy laws across Europe. The 2016 EU Directive on Security of Network and Information Systems is the first piece of legislation on cybersecurity that the EU has adopted. ²⁵ In the United States, there is increasing pressure on companies to prioritize consumer protection and citizen safety, as well as to introduce "proactive responsibility and accountability into the marketplace," including through product liability. Such an approach might be particularly useful when security flaws are easily prevented "by commonly accepted good engineering principles." ²⁶

Meanwhile, technical issues related to internet infrastructure remain largely within the purview of the so-called I* organizations, which include the Regional Internet Registries, the Internet Corporation for Assigned Names and Numbers, the Internet Engineering Task Force, the Internet Architecture Board, the Internet Society, and the World Wide Web Consortium, as well as the regional associations of country code domain name registries. ²⁷ Initiatives such as the Internet Governance Forum promote multi-stakeholder policy dialogue on internet-related issues, while intergovernmental bodies (including the World Summit on the Information Society, the International Telecommunication Union, and the European Telecommunications Standards Institute) deal with some policy aspects of internet governance. And a number of UN departments and agencies provide governments with capacity building as well as technical, legislative, and other forms of support, as do national technical bodies such as CERTs and computer security incident response teams (CSIRTs), in line with a provision (Action Line 5) of the World Summit on the Information Society's Tunis Agenda. ²⁸

Industry actors are also active in multiple ways. While some of these efforts may be seen as an attempt to forestall regulation, many aim to pressure industry actors or states to commit to behaving more responsibly. For instance, in 2018, Siemens launched a Charter of Trust for a Secure Digital World at the Munich Security Conference, a document that outlined principles that the initial signatories (eight companies and the Munich Security Conference) believe to be essential for establishing trust between political actors, business partners, and customers as the world becomes more dependent on digital technology. ³² The number of charter signatories has since grown to sixteen. ³³

Microsoft, too, has promoted norms of responsible behavior for both state and industry actors, and the company has reportedly responded more positively than other corporate peers in terms of complying with new regulations such as the EU's GDPR. ³⁴ The firm has also raised the idea of a "Digital Geneva Convention," a binding instrument that would protect users from malicious state activity. ³⁵ Along with several other industry leaders, Microsoft has also announced the Cybersecurity Tech Accord, which advocates for increased investment in (and heightened responsibility for) cybersecurity by leading industry actors.

What Lies Ahead?

Despite this progress, significant governance challenges remain for cyberspace. Efforts to not only protect data, privacy, and human rights online but also attend to national and international security concerns are improving in some cases. For instance, according to one assessment, the EU's GDPR provides much stricter guidelines and "strict security standards for collecting, managing, and processing personal data." But the instrument does provide exemptions for data controllers or processors when it comes to "national defense, criminal investigations, and safeguarding the general public)." ⁴² Progress remains much more limited or has even regressed in other countries and regions.

On cyber crime, despite concerns about the growing scale, economic and societal costs, and other risks of online criminal activity, states have not been able to (and likely will not) agree on a common framework for dealing with cyber crime or other malicious online activity that imperils users and hampers economic growth and development. This state of affairs is unlikely to change, given that some states continue to insist on the need for a common framework, while others remain wedded to the expansion of the existing Budapest Convention.

There are other challenges too. Progress remains slow in terms of achieving public and private sector commitments to bridge existing technological divides and move the digital transformation agenda forward. Inequalities within and between states (and cities) are growing even as technological advances continue to be made. This situation may make it even more challenging to meet the UN Sustainable Development Goals. ⁴³

Some countries will further challenge modalities of internet governance, particularly states that view greater state involvement in internet governance as crucial to national security. These divergences over how the internet should be governed continue to foment tensions among states and other stakeholders. ⁴⁴ Meanwhile, several countries have announced they will seek to build their own national alternatives to the global internet, possibly further fracturing an (already fractured) world wide web, though some observers have questioned the feasibility of such alternatives. ⁴⁵

As for international security, while tensions between countries continue to fester around normative restraints on state behavior, two new multilateral processes will commence this year through an Open-Ended Working Group (September 2019-July 2020) and a new GGE (December 2019-May 2021). ⁴⁶ Many observers view the two processes as conflicting and competing initiatives (given that the former was proposed by Russia and the latter by the United States). Some parties also view them as outdated (since the main actors in both mechanisms are states, although, importantly, both have included consultative mechanisms to engage with other actors such as regional organizations, private companies, academia, and civil society). But there is a signaled interest in ensuring the processes are both complementary and constructive.

That said, there are concerns, for instance, that GGE discussions on how international law applies to cyberspace might once again hit a wall. As noted, disagreements on international law-related issues were partly what impeded the last GGE from producing a consensus report. Since then, some states have decided to publicly share their views on how international law applies, although, to date, only a few states-Australia, France, the United Kingdom (UK), and the United States-have done so. ⁴⁷ Nonetheless, it is hoped that the two new processes will make further contributions to the discussion, as per their mandates. ⁴⁸ Importantly, the resolution establishing the GGE also suggests that the report include an annex in which participating governmental experts can include national views on how international law applies to the use of ICT by states. ⁴⁹

Meanwhile, some countries-and some nonstate actors-appear to remain committed to a binding international treaty. Yet the likelihood of such a treaty is perhaps slim, not least because the key actors crucial to any agreement view cyberspace and cybersecurity in very different strategic terms; at present, there appear to be limited incentives to agree on a new regime.

Beyond binding international law, the coming twenty-four months will serve as an important window into progress that has been made toward socializing and institutionalizing the political, nonbinding norms and confidence-building measures that past GGEs recommended. The two new UN processes and their associated consultative mechanisms will serve as important platforms for sharing national experiences and lessons on how states are implementing the norms and confidence-building measures nationally and through regional organizations, and how other actors actively contribute to this end. ⁵⁰

Other groupings will likely be proactive. For example, the Group of Seven (G7) members recently committed via the Dinard Declaration to sharing lessons and experiences on said norms, and it is likely they will channel those lessons and experiences into the multilateral process. ⁵¹ Similarly, multi-stakeholder groups such as the Global Commission on the Stability of Cyberspace or industry initiatives such as the Tech Accord will surely present lessons and experiences from the norms they too have been advocating.

How countries hold each other accountable for violating norms is just as important. Indeed, some states' persistent misuse (and potentially lethal use) of ICT is driving a dangerous security dilemma involving tit-for-tat activities that have significant escalatory potential. Beyond the fact that such activities raise serious questions about the rule of law, most related crisis-management or confidence-building mechanisms would likely prove ineffective in the event of escalation if there are no real channels of diplomatic dialogue between key states. Such channels are largely nonexistent at present. In this respect, more political and financial investment in operationalizing existing commitments to confidence building, track 1 or 1.5 dialogues, and other cooperative measures is imperative.

The growing number of initiatives aimed at fostering greater cybersecurity and stability do not (and perhaps cannot) deal with some of the structural issues driving insecurity and instability. This is particularly the case with respect to ICT products and services, which remain highly vulnerable to exploitation by actors with malicious intent. ⁵² Greater, and more participatory, dialogue on the nature of global ICT market trends and the structural levers for making ICT products and services more safe and secure is urgently required and should not be inhibited by the growing (and valuable) focus on VEPs and other similar measures.

Finally, existing threats and vulnerabilities will surely be compounded by new problems. This means that conceptions of security will need to be reconsidered over time and that existing normative and governance frameworks will likely need to be adapted. For instance, new threats and vulnerabilities related to the Internet of Things are emerging: as the lines between human agency and "smart agent-like devices" become increasingly blurred, the safety and security of related services and devices remain serious problems. ⁵³ Likewise, new threats are also developing in relation to critical systems dependent on AI (such as the growing number of sectors and industries reliant on cloud computing), critical satellite systems, and information and decisionmaking processes, which are increasingly susceptible to manipulation for political and strategic effect. Heightened strategic competition and deteriorating trust between states further compounds these challenges. More than ever, countries need to invest in diplomacy to foster greater dialogue, cooperation, and coordination on the ICT-related issues that pose the greatest risks to society.

Artificial Intelligence

Although AI research has existed for more than five decades, interest in the topic has intensified over the past few years. This highly complex field emerged from the discipline of computer science.

The classic definition of AI dates back to 1955, when John McCarthy and his fellow researchers characterized artificial intelligence as "making a machine behave in ways that would be called intelligent if a human were so behaving." ⁵⁴ Noting that a similar counterfactual understanding of AI underpins the earlier Turing test, Florini and Cowls conceptualize AI as "a growing resource of interactive, autonomous, and often self-learning agency (in the machine learning sense . . . ), that can deal with tasks that would otherwise require human intelligence and intervention to be performed successfully." Succinctly put, AI can be viewed as a " reservoir of smart agency on tap."⁵⁵

AI encompasses numerous subdisciplines including natural language processing, machine inference, statistical machine learning, and robotics. ⁵⁶ Certain subdisciplines such as deep machine learning and machine inference are often seen as points along a continuum on which progressively fewer human beings are required in complex decisionmaking. ⁵⁷ Some observers believe this will eventually lead to artificial general intelligence or superintelligence that either achieves or surpasses human intelligence. ⁵⁸ Yet it is fiercely debated whether AI will ever actually achieve or exceed such cognition and abstract decisionmaking capabilities. ⁵⁹

Nonetheless, advances in the various AI subfields are expected to bring about great economic and social benefits. Communications, healthcare, disease control, education, agriculture, transportation (autonomous vehicles), space exploration, science, and entertainment are just a few of the areas already benefiting from breakthroughs in AI. Yet the risks inherent in the ways these technologies are researched, designed, and developed and how they might be used can just as easily undermine these benefits.

The immediate risks and challenges include the expansion of existing cybersecurity threats and vulnerabilities into increasingly critical AI-dependent systems (like cloud computing); unintended or intended consequences as AI converges with other technologies including in the biotech and nuclear domains; algorithmic discrimination and biases; weak transparency and accountability in AI decisionmaking processes; overly narrow ways of conceptualizing ethical problems; and limited investment in safety research and protocols. Meanwhile, policymakers are fixated on predictions about how automation will transform industries, the labor force, and existing forms of social and economic organization. Predictions that automation and advanced machine learning may exacerbate economic inequalities in particular have stoked anxiety. Several studies on subjects like the future of work, the future of food, and even the future of humanity seek to allay these concerns, while also highlighting and forecasting risk. ⁶⁰

Different AI applications and models derive (or will derive) much of their power from large quantities of collected, stored, and processed online data. Concerns over data protection, privacy, and other principles and values such as equity and equality, autonomy, transparency, accountability, and due process are growing. The dual-use nature of AI applications also makes it difficult to constrain their development and regulate their use. Moreover, recently world leaders including Chinese President Xi Jinping, Russian President Vladimir Putin, and U.S. President Donald Trump have made public declarations painting AI in terms of national power projection, a trend that suggests the development and use of such technologies will be complicated by growing strategic competition (in geopolitical, military, economic, and normative terms). ⁶¹ Moreover, some countries' desire to use AI as a "critical enabler and force multiplier for capabilities across all aspects of military power" is becoming increasingly evident. ⁶²

In short, further advances in AI likely will significantly alter the contours of economics, sociopolitical life, geopolitical competition, and conflict. According to some observers, the technology may even pose existential risks. Yet there is still time to think seriously about AI and develop stronger responses to the challenges ahead.

The Current Policy and Normative Landscape

Several classes of actors are playing important roles in advancing discussions about some of these issues, including nongovernmental organizations (NGOs) and nonprofits, commercial actors, governments, and multilateral forums. These efforts are resulting in the proliferation of new rules, values, principles, tenets, standards, and declarations, making it increasingly difficult to make sense of where these various standards overlap, create unnecessary duplications, or cause confusion and ambiguity. ⁶³

Nonprofits and multi-stakeholder initiatives: A number of initiatives involving leading scientists, technologists, researchers, civil society organizations, and investors in the fields of AI and robotics are promoting important principles to help mitigate risks and spread the benefits of AI.

Examples include the Asilomar Principles, developed in 2017 to "underpin the future development of AI and ensure shared opportunities and benefits of AI for all." ⁶⁴ The twenty-three principles cover research goals, funding for safety-related research, questions of human control and responsibility, privacy, shared benefits and prosperity, efforts to ensure that AI is not used to fuel an arms race or develop lethal autonomous weapons and other military capabilities, and other values and risks. This principle on military applications and the central question of human control are integral to a campaign spearheaded by an international coalition of NGOs-the Campaign to Stop Killer Robots. ⁶⁵ The principles and the question of meaningful human control are also central to the recent Lethal Autonomous Weapons Pledge signed by key companies, organizations, and individuals, ⁶⁶ and the 2015 Open Letter on AI and Autonomous Weapons signed by a host of AI and robotics researchers. ⁶⁷

Meanwhile, AI Now, a research institute working out of New York University, has produced a number of in-depth reports on various governance challenges relating to AI. This work includes promotion of principles such as fairness, accountability, and transparency across all aspects of the production and life cycles of AI systems and an assessment of (accountability for) both the direct and indirect effects of these systems. On a related note, the institute has also published what it calls the "Algorithmic Accountability Policy Toolkit." ⁶⁸ More recently, it examined the principle proliferation trend and proposed that existing efforts should be unified under a common framework (discussed below).

For its part, the Institute of Electrical and Electronics Engineers (IEEE) continues to update its 2016 "global treatise" on principles and recommendations for designing AI systems in ethical ways that promote human well-being; this document was developed from more than 1,000 crowd-sourced contributions from across business, academic, and policy circles. ⁶⁹ In 2019, the IEEE launched an effort to move beyond the articulation of principles. The Principles Into Practice initiative, also founded on crowd-sourced contributions, is centered on validating the existing principles. ⁷⁰ The results of this effort will likely be monitored closely.

Meanwhile, the Partnership on AI to Benefit People and Society brings together a growing number of research and civil society groups as well as major AI companies to promote similar principles. The group's aim is to "study and formulate best practices on AI technologies, advance the public's understanding of AI, and serve as an open platform for discussion and engagement about AI and its influences." ⁷¹

Commercial actors: Some companies are gradually realizing the need to create new governance structures and "clearly define how AI progress and implementation are managed" before they are compelled to do so by governments. ⁷² For instance, Open AI, a firm that participates in the Partnership on AI, has developed its own charter outlining the principles underpinning its work and overall strategy, a document crafted in consultation with other actors. ⁷³ Anchored in a pledge to act in "the best interests of humanity throughout [artificial general intelligence] development," the principles commit the company to ensuring: broadly distributed benefits; long-term safety; technical leadership (to effectively address AI's impact on society), and cooperation with other research and policy institutions worldwide to tackle the global challenges associated with artificial general intelligence and provide related public goods. ⁷⁴

Technology companies are increasingly propelled by the growing threat of regulation and are seeking to ensure that whatever regulation comes about is favorable to their interests and does not stifle innovation. At the same time, their actions are also driven by a deep understanding of the potential risks of how AI technologies might be used. For instance, in response to the company's ethics being called into question by its own staff over a military contract, Google adopted seven principles to guide its AI work. ⁷⁵ These include developing AI that is socially beneficial, avoids creating or reinforcing unfair bias, is built and tested for safety, is accountable to people, incorporates privacy design principles, upholds high standards of scientific excellence, and is made available for uses that accord with these principles. Other companies have made similar announcements, like Microsoft's "AI Principles," ⁷⁶ or Accenture's broader "Ethical Framework for Responsible AI and Robotics." ⁷⁷ Others such as Salesforce have established dedicated research streams on AI ethics and values, specific personnel positions such as "chief ethical and humane use officer," and other entities like ethics advisory councils. ⁷⁸

Governments: Political actors increasingly view AI as a strategic technology with immense promise in the areas of economics, politics, and national defense. According to a recent report, at least twenty-seven national governments have articulated an interest in "encouraging and managing the development of AI technologies" and are organizing their efforts through national policies, strategies, and doctrine. ⁷⁹ They are also working to mitigate the societal risks posed by increased automation and digitization. Certainly, there are mounting concerns about robots taking over a broad range of blue- and white-collar jobs and the associated impact these job losses would have on societies around the world. Yet some experts suggest that the speed of such automation could be blunted. They have stated that although the uptake of robotics will certainly influence service sectors such as construction, healthcare, and business, in other sectors it will more likely lead to job transformation rather than replacement. ⁸⁰ Consequently, governments are increasingly urged to invest in social safety nets and human capital, so that workers can build the skills required for a transforming labor market. ⁸¹

AI subfields such as machine learning, deep learning, and robotics have important competitive advantages for national security and defense. Over the past few years, China, Russia, the UK, and the United States, in particular, have been engaged in an intense competition to dominate the field. Each of these countries are spending huge amounts of resources on innovation; in many instances, they have developed or are developing close ties with global technology leaders in AI for social policy as well as defense R&D, not least because private sector investment in AI continues to significantly outpace that of governments. ⁸² Growing interest in the military applications of AI is already evident in discussions on topics like how machine learning can enable "high degrees of automation in labour-intensive activities such as satellite imagery analysis and cyber defense." ⁸³

But the military aspects are broader. Take, for example, the United States, where Congress is grappling with issues ranging from balancing public and commercial funding for AI development, articulating congressional and executive oversight roles and responsibilities for AI development, reconciling AI-related R&D and autonomous systems with ethical considerations, identifying the legislative or regulatory changes needed for the integration of military AI applications, and figuring out how it can help manage the AI competition globally.

The national security risks AI is expected to pose include AI-related strategic competition and related security dilemmas; cybersecurity risks to critical AI-dependent systems; consequential convergences with dual-use technologies in biotech, the nuclear domain, and other fields; and the overall unpredictability surrounding AI algorithms and their vulnerability to bias, theft, and manipulation. One particularly vexing potential tool of manipulation is deepfakes, which are "realistic photo, audio, and video forgeries" that could be weaponized for "information operations." ⁸⁴

Some researchers are helping deepen humanity's understanding of these AI-related security risks. A recent comparative study of national AI security policies and strategies across ten countries by the Center on Long-Term Cybersecurity points to highly diverging approaches to AI security. ⁸⁵ Unless reconciled, these differences could, over the long term, present serious risks to both national and international security.

Multilateral Forums: AI is increasingly appearing on the agendas of various UN norm-shaping bodies and other specialized agencies (including the UN Conference on Disarmament; the UN Educational, Scientific, and Cultural Organization [UNESCO]; the International Labor Organization; and the World Bank) as well as other international and regional organizations (such as the EU, the Organization for Economic Cooperation and Development [OECD], and the Council of Europe).

These various bodies are focusing on matters like the economic consequences of labor displacement as well as safety, ethical, and security risks, including those associated with AI-augmented surveillance and data privacy. In addition, a number of these organizations-including some UN entities-are engaging with academics and industry actors to determine how AI solutions might best be marshalled to help address global humanitarian and development challenges and meet the UN Sustainable Development Goals and the Paris Agreement for Climate Change. In some instances, notably in the field of ethics, individual governments or regional groups have established new institutions, such as working groups or centers on AI and ethics, as well as ministerial or ambassadorial positions.

In regional terms, the EU has ratcheted up its engagement on AI. For instance, the European Group on Ethics in Science and New Technologies has intensified its calls for an "internationally recognized ethical framework for the design, production, use and governance of AI, robots and autonomous systems," requiring "a collective, wide-ranging and inclusive process." ⁸⁶ In April 2018, the EU published a Declaration on AI Cooperation, ⁸⁷ which was soon followed by the EU Communication on Artificial Intelligence. ⁸⁸ The communication establishes a European Initiative on AI, which aims to boost Europe's technological and industrial capacity in AI and its uptake, tackle its socioeconomic effects, and ensure suitable legal and ethical standards are in place. This framework is based on EU values and aligned with the Charter of Fundamental Rights of the EU, including "forthcoming guidance on existing product liability rules, a detailed analysis of emerging challenges and cooperation with stakeholders, through a European AI Alliance, for the development of AI ethics guidelines." ⁸⁹ Such guidelines, in turn, would build on the work of the European Group on Ethics in Science and New Technologies.

Soon thereafter, the EU established a High-Level Expert Group on Artificial Intelligence (AI HLEG). Composed of members of industry, academia, and civil society, it will support the implementation of the European Initiative on Artificial Intelligence. The AI HLEG will make recommendations on future AI policy developments; offer guidance on related ethical, legal, and societal issues; and serve as the steering group for the European AI Alliance 's work. It will interact with other initiatives, help stimulate a multi-stakeholder dialogue, gather participants' views, and reflect those views in its analysis and reports. How the EU intends to scale these initiatives and guidelines internationally remains unclear.

There is a growing perception that military applications of AI require normative action and that the responsible and ethical use of AI systems in the military should be prioritized. ⁹³ One of the most prominent examples is lethal autonomous weapon systems. Using sensor suites and advanced machine learning algorithms, these weapons can identify a target, make an engagement decision, and guide a weapon to destroy the target, independent of human intervention or control. ⁹⁴ These weapons systems have received significant attention since 2016, when parties to the UN Convention on Certain Conventional Weapons (CCW) established a GGE to study the legal, normative, ethical, technological, and military dimensions of emerging technologies and how they apply to lethal autonomous weapons systems. Unlike GGEs established by the UN General Assembly's First Committee on Disarmament and International Security, the CCW GGE allows for the participation of the research community, civil society, and the private sector. Many of these nongovernmental actors participate in the Campaign to Stop Killer Robots.

In 2017, the CCW GGE agreed on several issues for study, including how best to characterize the systems under consideration so as to promote a common understanding of concepts and characteristics relevant to the objectives and purposes of the CCW. ⁹⁵ The GGE also agreed to consider the human element in the use of lethal force; contemplate aspects of human-machine interaction in the development, deployment, and use of emerging technologies related to lethal autonomous weapons systems; and review potential military applications of related technologies. Importantly, the group was asked to submit options for addressing the humanitarian and international security challenges posed by technologies related to lethal autonomous weapons in the spirit of the convention. ⁹⁶

Since 2017, twenty-eight countries have joined the call for a ban on fully autonomous weapons. ⁹⁷ At the meeting of the first session of the CCW GGE in 2018, eighty-five states "publicly elaborated their views on lethal autonomous weapons systems in a multilateral forum," with some pushing for the development of a legally binding instrument on fully autonomous weapons. ⁹⁸ The UN secretary general has also voiced support for some form of normative restraint on the use of such technologies, ⁹⁹ although he recognizes that states have differing positions on the issue. For instance, in his Agenda for Disarmament, he stated that while "a growing number of States, including some with advanced military capabilities, have called for a preventative prohibition on lethal autonomous weapon systems," other states believe that "existing international humanitarian law is sufficient to address the risks." ¹⁰⁰ Importantly, however, he notes that regardless of which position individual countries take, there is agreement between all parties that, "at minimum, human oversight over the use of force is necessary." ¹⁰¹

What Lies Ahead?

Unknowns about how the potential benefits and risks of AI will play out in the coming decades will force policymakers to confront a series of interconnected governance problems. Significant coordination and cooperation between public and private actors will be necessary. And internationally speaking, it is imperative to establish participatory or cooperative mechanisms to understand the evolving implications of AI and how best to manage them, while navigating the uncertainties and risks of future developments.

These mechanisms will need to be informed by broader public discussion and engagement on a range of issues. First, much wider public dialogue is required on the ethical values and principles put at risk by the development of complex AI models and systems as well as the nature and direction of change they are engendering. This dialogue should involve exchanges and dialogue on the emerging body of principles and values that public and private entities are (sometimes jointly) promoting and conversations about more transparently managing the associated cross-border risks.

The current trend, true of governmental and nongovernmental bodies alike, toward developing principles and guidance for AI (the principle proliferation discussed above) suggests there may be much duplication where the principles overlap and significant potential for "confusion or ambiguity where they differ." ¹⁰⁵ Moreover, the majority of these principle-driven initiatives tend to be emerging in advanced Western economies, in turn suggesting a degree of fragmentation that may be difficult to reconcile in the medium to long term if alternative voices are not brought into the discussion. This trend has profound implications for ongoing efforts to achieve a more equitable distribution of AI-related benefits. So-called fairness norms advocate for making coordinated AI development paths in specific fields or sectors provide reasonable compensation or benefit sharing to those exposed to risk.

Yet there is hope. For instance, based on comparative analysis it conducted of current, high-profile efforts to promote ethical AI, the AI Now Institute has recommended an overarching framework consisting of five central tenets to overcome such duplication, confusion, and ambiguity. Drawn largely from the field of bioethics, these framing principles include beneficence, nonmaleficence, autonomy, and justice as well as one new principle-explicability, which incorporates both the sense of intelligibility (how a specific AI system works) and the ethical sense of accountability (how to track who ultimately holds responsibility for the effects of an AI system). ¹⁰⁶ This overarching framework could be an important foundation for cross-regional dialogue aimed at ensuring more ethical AI that benefits societies across the globe.

The nature and composition of oversight bodies is equally problematic. For instance, while some companies have established ethics boards, they are not always open or transparent about their membership or operations. For instance, DeepMind, a company purchased by the Alphabet Group in 2014, has created DeepMind Ethics and Society, a research unit focused on exploring "the key ethical challenges facing the field of AI." ¹¹⁰ The group pledges that its work should be "governed by a set of principles that seek to guarantee the rigor, transparency and social accountability of this work," including by helping technologists put ethics into practice and by helping society anticipate and manage the impacts of AI in a manner that benefits all. ¹¹¹ At the same time, however, the company has been criticized for the lack of transparency around its AI Ethics Board. ¹¹² This board was written into a contract called the Ethics and Safety Review Agreement when the Alphabet Group acquired the company. The agreement essentially puts control of DeepMind's "most valuable and potentially most dangerous technology"-its core artificial general intelligence technology-under the governance of the ethics board, whose actual composition has never been made public. ¹¹³

This is undoubtedly an important move. However, the board's operating principles will likely raise additional questions as to how boards like these-diverse and professional as they are-can provide legitimate guidance and oversight for products and services that have profound implications for a diverse range of communities if the public itself is not engaged in the process. In the case of Axon, the connection with broader policing policy and democratic oversight appears to be lacking. That being said, the establishment of the independent board and the additional questions it has raised is still an interesting step that companies championing AI ethics might learn from as this complex and competitive field moves forward.

Third, it is necessary to have a deeper discussion on how best to balance investment in the research and design of safety protocols, equity, transparency, and accountable decisionmaking with investment in technological advances. To date, most investment has gone toward the latter. Distilling and sharing lessons from sector-specific AI and automation piloting, sandboxing efforts, and initial regulatory steps (like those under way in the EU) can advance this discourse. In addition, such an approach can help engender more informed conversation on the nature and scope of the regulatory, legal, and oversight frameworks that might be required for particular AI systems in the future.

Fourth, while all states have the right to shape their own AI national security policies and strategies as they deem fit, greater convergence around ultimate goals is required to safeguard against broader systemic risk. As research into AI-related national security policy and strategy development deepens, further thought might be afforded to the recommendations put forward in the recent report by the Center on Long-Term Cybersecurity. Particularly notable are those recommendations related to promoting early coordination around identified common interests; encouraging public investment in identifying and sharing best practices; understanding the risks of prioritizing one subset of issues over another and the opportunities and challenges that may be ignored; ensuring greater accountability of the technology sector; and pressing for greater inclusivity in the shaping of government policy and strategy. ¹¹⁷

Fifth, public discourse on AI must tackle the issue of how international norms, rules, and principles can mitigate (or even prevent) novel ways of misusing AI and maintain peace and stability as the global balance of power shifts. In the short term, legal and ethical questions related to whether weapons systems should be permitted to make life and death decisions (that is, the question of meaningful human control, or explicability) will likely remain central and differentiated from questions pertaining to the application of international humanitarian law or other rules for other national security uses of AI systems and capabilities. Yet efforts should be made to ensure that these important discussions on lethal autonomous weapons systems do not drown out consideration of pressing normative issues relating to AI and national security. ¹¹⁸

To this end, it may be necessary to establish more concrete and integrated mechanisms for navigating the uncertainties of future developments in AI and autonomous systems and for managing related risks, including those resulting from the growing relevance of AI to strategic competition. Other issues related to the growing militarization of AI will likely appear on the multilateral security agenda in the coming years. ¹¹⁹ Initiatives such as the UN Institute for Disarmament Research's Trust But Verif AI Project can bring fresh thinking to debates surrounding AI and international peace and security. ¹²⁰ This specific project aims to bring together AI technologists, arms control practitioners, and other experts in regulatory and technology control policy to consider how AI arms control might be feasible.

Biotechnology

Significant advances are also being made in the field of biotechnology, which the UN defines as "any technological application that uses biological systems, living organisms, or derivatives thereof, to make or modify products or processes for specific use." ¹²¹ In recent years, the field's principal breakthroughs have occurred in genome editing, gene drives, and synthetic biology. Current trends suggest there will be a profusion of biotechnology products that are significantly different in type, scope, and complexity than previous ones. Lower barriers to access have opened the field to more and more actors.

Technological advances are spurring rapid growth in the bioeconomy. According to the Nuclear Threat Initiative's (NTI) biosecurity program, the global bioeconomy in 2014 accounted for a significant portion of world trade and offered societal benefits in the areas of energy, food production, healthcare, and other sectors vital to sustainable development. For instance, India's bioeconomy was already valued at more than $4.3 billion in 2012. ¹²² Meanwhile, the EU's bioeconomy was reported to already be generating a turnover of more than 2.3 trillion euros (or about $2.6 trillion) as well as some 18 million jobs in 2015. ¹²³

Public awareness about biotechnology-related risks and challenges is growing in many countries. The debacle involving Chinese researcher He Jiankui's claim to have altered the genes of twin girls to prevent them from contracting human immunodeficiency virus (HIV) is a case in point, not just because of the skepticism surrounding his claims but also because of the ethical questions his research raises, including about what it means to be human. ¹²⁷ Meanwhile, national regulatory agencies are struggling to keep pace with the rate and scope of change, and it is becoming increasingly challenging to balance the competing interests of industry, academia, the rest of society, and the state. Internationally, rapid advances in biotechnology and persisting divides between states have heightened the potential consequences of a biological attack, placing new pressures on existing regimes such as the Biological Weapons Convention (BWC). As the world becomes more dependent on biological systems, there is a greater need to strengthen existing governance systems, including norms and oversight mechanisms, and to identify new ones where appropriate.

The Current Normative Landscape

Biotechnology offers humans a vastly improved understanding of the basic building blocks of life. The Human Genome Project finished mapping the human genome in early 2000, identifying every one of the roughly 20,000-25,000 genes in human deoxyribonucleic acid (DNA). ¹²⁸ Since then, researchers have made significant advances in gene editing and gene drive research. Genome editing involves the precise cutting and pasting of DNA segments by specialized proteins. The method that has captured the most attention is called clustered regularly interspaced short palindromic repeats (or CRISPR) and CRISPR associated protein 9 (Cas9). This simple, versatile, and cheap technology was initially discovered in the late 1980s, and more recently researchers have learned how to use CRISPR and Cas9 for gene editing. ¹²⁹

CRISPR is now being tested for various uses such as treating some genetic diseases, growing climate-resilient crops, combating vector-borne diseases like malaria, designing food and drugs, and even saving endangered species from extinction. The technology also carries significant potential for what are termed autologous treatments, which involve teaching cells to "fight disease in one's own body, delete hereditary diseases for one's self and one's own offspring, and shape new generations of organisms as genetic modifications are passed to future generations." ¹³⁰ Few other new technologies offer more hope for alleviating disease. At the same time, gene editing could bring about unpredictable permanent genetic changes, some of which could harm humankind and the biosphere.

Several advances in the field raise new ethical and normative questions and risks that require new solutions, particularly in four key areas, some of which overlap:

Ecological risk: Gene alterations can generate unpredictable and undesirable consequences in the modified species, as well as in other species, and can even give rise to new and unknown animal and human diseases. Researchers are investigating how gene drive systems can be used to eradicate diseases; the most advanced of these projects seeks to wipe out malaria-carrying mosquitoes. And some scientists had hoped to use a CRISPR gene drive to prevent the extinction of wildlife species threatened by invasive organisms. The thinking went that modifying an invasive organism's DNA to reduce its ability to reproduce and then releasing that organism back into the wild would cause the fertility-reducing gene to spread through the invasive population, eradicating the pests. But a recent study by some of the same scientists found that altered genes might actually spread to places where that same organism is not invasive at all, but a well-established part of the local ecosystem. ¹³¹ Such unintended consequences would pose unacceptable risks.

Such projects demonstrate the need to first develop safer versions of this powerful technology and raise important issues related to standards and regulation. The consequences of using such technologies are not contained within national borders, and these effects will likely require some form of transnational mechanism that considers the views of all stakeholders. This calls for a more harmonized approach to biotech policy, especially since rules and practices of gene editing vary significantly from country to country. Meanwhile, all stakeholders, including those funding and supporting research, need to ensure adherence to existing guidance such as the World Health Organization's (WHO) 2014 set of guidelines for testing genetically modified mosquitoes. ¹³² According to some experts, this push to shore up standards might require the establishment of some form of accountability mechanism that can, at a minimum, oversee adherence to this guidance and identify possible obstacles to implementation. ¹³³

Another ecological risk is the potential for biological attacks against food and water resources, which, as discussed further below, would cause economic damage as well as a "loss of confidence in the food supply, and possible loss of life." ¹³⁴ For example, a recent study determined that scientists' deepening understanding of the genomes of plants and animals will make it theoretically possible to target vulnerabilities with greater precision or to create new varieties of organisms with potentially harmful properties. The authors posit that "if plant or animal pathogens [a]re engineered to spread widely in the world in crops or herds, respectively, the result could be widespread and lasting food shortages." ¹³⁵

Other health and safety risks: Incidents like laboratory accidents or research aimed at rendering deadly viruses more contagious or pathogenic pose dangers as well. Relatedly, the publication of such techniques and research outcomes could educate and empower malevolent actors. Concerns of this kind arose from research published by two scientists in 2012 in prominent journals ( Nature and Science). Both had succeeded in genetically engineering strains of the avian flu virus (H5N1).¹³⁶ The controversial decision to publish their research findings stemmed from intense public concerns that the virus could leak out of a laboratory in the event of an accident. More seriously, the research could be leveraged by terrorists to create a biological weapon and unleash a devastating pandemic.

National guidance, regulations, and codes of conduct by the research community are helping address some of these challenges. ¹³⁷ Globally, the International Standards Organization is developing new standards to harmonize attempts to help laboratories manage biological risk, drawing from EU and WHO efforts to establish and implement effective biorisk management systems. ¹³⁸ But there are still important gaps. As biotechnology becomes cheaper and more readily available, these softer forms of regulation will likely be insufficient for managing the associated risks.

Questions of consent and human-induced permanent genetic changes also raise new quandaries. For example, fears about modifying fetuses or otherwise enhancing subjects' aesthetic performance or other attributes drive concerns about genetic discrimination, human cloning, and eugenics. National regulations and practices on this aspect of biotechnology greatly vary from state to state. For example, many countries cite ethical considerations to either prohibit or otherwise circumscribe certain human CRISPR trials to treat or prevent disease or disability in germline cells (as opposed to somatic cells). ¹³⁹

Rule of law risks: Law enforcement agencies' growing use of data-sensitive biotechnologies-such as fingerprinting, systematic collection of human DNA samples, and the introduction of biometric identifiers and intelligent implants-all pose important societal risks, including to core principles of the rule of law and human rights. For instance, in the United States, law enforcement agencies have shown a mounting interest in accessing privately run genetic genealogy banks after using DNA information from a genealogy website to identify a suspect in the infamous Golden State Killer case. ¹⁴³ The fact that the owners of some of these websites have provided full access to law enforcement, unbeknownst to their users, has raised serious questions about privacy, consent, and due process. ¹⁴⁴

International security risks: Concerns are also mounting about the dual-use nature of technological advances in the life sciences and their potential for malicious use. The diversity of possible applications of genome engineering and gene drive technology (including the production of bioweapons) and the potential impacts are unpredictable. While such biotech applications would likely still require the resources of a state or large enterprise, the technological barriers to acquire, develop, and use biological agents as weapons have lowered significantly in recent years. The risk of biological warfare, biocrimes, and bioterrorism are particularly grave.

Apropos of biological warfare, the BWC is a legally binding treaty that outlaws the use of biological weapons and codifies a strong, age-old, and cross-cultural norm against the use of disease as a weapon. But the BWC is institutionally weak and lacks verification, monitoring, and enforcement provisions. Furthermore, the convention may be unable to keep pace with rapid scientific and technological developments.

Notably, because biological science is inherently dual use-the same tools and technologies that can be used to benefit society can also be used to cause deliberate harm-the key difference is user intent. While there are no concrete examples of novel developments inconsistent with the aims of the BWC, advances in biotechnology do have security-related implications. Such advances make the development and production of biological weapons easier and more accessible, even if such research endeavors are largely pursued for entirely legitimate purposes (such as general scientific discovery, public health, and food production) and therefore cannot, in and of themselves, be prohibited.

It is unclear whether or not the erosion of technical and access barriers might lead to the more widespread acquisition and use of biological weapons. While scientific and technical advances may continue to put pressure on the norm against using disease as a weapon, they also can serve as a deterrent. For example, the creation of versatile diagnostic and therapeutic platforms could enable a rapid response to any engineered threat, deterring nefarious actors from considering biology as a potential tool of large-scale terror, as discussed above. ¹⁴⁵ Yet, without accompanying advances to monitor compliance and differentiate between offensive and defensive (or peaceful) intent, advances in biotechnology could make it harder to identify violations of the BWC.

The convention's eighth review conference was held in November 2016 and ended without an agreement on any significant steps to enhance the treaty's effectiveness, including the creation of a regular advising mechanism to assist in monitoring and assessing relevant scientific developments. At the same time, several regional workshops, funded by the EU and the United States, are under way, as are workshops aimed at making the BWC's post-review conference discussions on science and technology more global and diverse. These workshops hopefully will help provide a baseline for future review and assessment.

Beyond the risks of full-on biological warfare, there are other international security risks. One such risk is the growing convergence of biotech and AI, notably in the area of synthetic biology. While AI-powered bioengineering is mostly used for benign and beneficial purposes at present, a U.S. National Academy of Sciences report cautions that these technologies "expand the risks of bioweapons because new or more virulent pathogens could be created from scratch." ¹⁴⁶

Despite a certain degree of hype, the use of biotechnology to commit terrorist attacks or other criminal acts remains limited. Recent research points out that there has only been one documented death attributable to biocrimes (although that is debatable), ¹⁴⁷ a number of bioterrorism incidents with no directly associated deaths, ¹⁴⁸ and five deaths from a documented lone-wolf terrorist incident in the United States. ¹⁴⁹ Nonetheless, as the technological barriers to harnessing biological agents as weapons have fallen, such risks increase. The growing availability of new biotech tools could make it possible for terrorist groups to create and modify pathogens or otherwise misuse biological materials and expertise. ¹⁵⁰ Groups such as al-Qaeda and the self-proclaimed Islamic State have reportedly called on "scientists, doctors, and engineers to join their cause, which includes the use of specialized skills to inflict harm." ¹⁵¹

What Lies Ahead?

The current framework for governing biotech-related ethical and normative challenges and for managing biological risk is fragmented. Responsibilities are dispersed across different intergovernmental agencies and national regulatory and oversight bodies. ¹⁵² For decades, the WHO has helped shape norms relating to biotechnology, healthcare, and bioethics. The UN and other international and regional bodies have long played a role in shaping principles, standards, and norms in some of these areas with mixed results. For its part, UNESCO has also been involved in bioethics since the 1970s. Its work revolves around the sociocultural, legal, and ethical implications of advances in the life sciences (including biotechnology), and it has established bodies such as the International Bioethics Committee to study such matters.

The UN is also responsible for enforcing the ban enshrined in the BWC (which entered into force in 1975), the first multilateral disarmament treaty banning the development, production, and stockpiling of a specific category of weapons of mass destruction. Through a number of resolutions, the UN Security Council, too, has engaged on certain biotech-related issues, notably in relation to strengthening member states' capacities to prevent biocrime and bioterrorism. These include UN Security Council Resolution 2325 (2016), which calls on all states to strengthen national antiproliferation regimes in the course of implementing UN Security Council Resolution 1540 (2004), which in turn seeks to keep nonstate actors from acquiring nuclear, biological, and chemical weapons of mass destruction and to submit timely reports on their efforts to do so.

Through its Group on Ethics and Science in New Technologies (EGE), the EU, too, is increasing its attention to biotechnology and its normative implications. After an earlier statement on gene editing, the EGE is preparing an opinion on gene editing, due to be completed in the summer of 2019. The opinion will focus on the "bigger picture of this issue," which crosses existing ethical divides, as well as "specific aspects of concern" such as "gene editing applied to animals" and "gene editing in the context of biodiversity and ecosystems." ¹⁵³ International forums like the WEF are also seeking to shape new related governance solutions. For instance, in addition to its work on the Fourth Industrial Revolution, the forum has established a Global Future Council on Biotechnology to consider ethical and safety issues emerging from biotechnology-related discoveries, applications, and policy issues. ¹⁵⁴

In national terms, rules and practices relating to certain biotech applications (like gene editing) continue to differ widely from country to country. Policymakers around the world are struggling to keep pace with the latest developments and adapt domestic institutions to manage the risks associated with the emergence of future advances and products. Countries have not yet aligned their respective efforts, although some are advocating for more global responses, including consensus on guiding ethical principles, through bodies such as the WHO. ¹⁵⁵ In this respect, the WHO's decision to establish an Expert Advisory Committee to "examine the scientific, ethical, social and legal challenges associated with human gene editing" with the aim of advising and producing recommendations on "appropriate governance mechanisms for human gene editing" may be a step in the right direction. ¹⁵⁶

In recent discussions at the UN General Assembly, the WEF, and the Munich Security Council, key stakeholders have been urged to find creative ways to reduce biological risks. In response, the NTI program on biosecurity, the Wellcome Trust, and the WEF are pushing for the establishment of new global norms by leaders in the virology and synthetic biology communities, as well as models for adopting such norms. ¹⁶⁰ To this end, in 2018, these three organizations launched a joint initiative aimed at bringing together leaders and experts in genomics, virology, synthetic biology, bioethics, security, insurance, and scientific publishing. Following an initial roundtable, the NTI aims to convene a multi-stakeholder council to spur the adoption of new global norms and technical approaches that enhance biosecurity innovation and reduce risk. ¹⁶¹

Undoubtedly, given the sheer number of ethical and normative issues at hand, governments and other key actors will need to ensure a more aligned, cooperative, and participatory policy environment. Countries need to make progress and identify cases in which international cooperation and coordination could help resolve persisting or emerging challenges. Doing so would engage more stakeholders, increase efficiency, raise overall safety and standards, and reduce risk. This approach would also promote greater transparency and accountability, conferring much-needed legitimacy on policymaking, norm-shaping, and regulatory processes. And it will be necessary to identify whether current platforms adequately serve this purpose or whether new ones are needed.

Space Technology

Historically, outer space has been shaped by strategic and resource competition between major powers. ¹⁶² Today, overlapping advances in technology are enabling new forms of commercial and military activity in space. With more tangible threats, lower access barriers, and a growing number of stakeholders, there is an urgent need for states and other actors to engage in more dialogue and cooperation to determine how relevant governance frameworks can best be strengthened.

Developments in Space Technology

Although outer space was originally dominated by states, it has become a theater for economic growth, innovation, and private commercial activity. Over the past decade alone, the international commercial space sector has diversified significantly. According to the OECD, in 2013 outer space generated some $250 billion in revenue, including such sectors as the space manufacturing supply chain, satellite services, and consumer services such as satellite television and global positioning system (GPS) devices. ¹⁶³

Innovation has continued to transform a variety of space-related economic sectors, including the satellite industry. Large and small companies alike as well as universities are increasingly using powerful miniature satellites originally developed for costly scientific research purposes in low Earth orbit. These small satellites are often developed with off-the-shelf commercial components. Amateurs, sometimes funded through crowd-sourcing campaigns, also use them. Beyond that, technological innovation is also poised to open and/or expand new sectors such as commercial space launches and space tourism, space manufacturing, and resource recovery (asteroid mining). In some countries, privately backed investments in rocket launches, rocket recovery, and the delivery of payloads to space stations, for instance, are already producing breakthroughs and delivering noteworthy efficiency compared to government-funded competitors.

In some cases, actors are innovating on the back of existing space technology. For instance, the use of space-based remote sensors is transforming how a diverse range of fields operates, including humanitarian assistance, crisis management, and agriculture. Social media and technology companies are assessing how to use satellites in low Earth orbit to expand access to the internet (and their own products and services) in remote areas. These technological advances are expected to provide new opportunities for economic growth, development, and the involvement of new actors. In 2014, the OECD inferred that "countries with long-established space programmes face growing challenges as lower costs and technological advances draw more countries and companies into the sector and give rise to a burgeoning commercial space industry." ¹⁶⁴ By 2017, the total value of space-related industries was estimated at $350 billion and forecast to grow to about $2.7 trillion by 2040, according to Bank of America, a trend rendered possible by "new drivers," such as "reusable launch by SpaceX, the growth of both private ownership in the market, investment and financing by more than 80 countries and the falling launch costs from vehicles by the likes of Rocket Lab and Vector." ¹⁶⁵

At the same time, increased activity in outer space presents new risks, such as the possibility of collisions and the hazardous debris that could result from such collisions. And the fact that states increasingly view the space economy as a new source of power is driving strategic competition at a time of heightened geopolitical tensions. This will likely influence related normative discussions. Furthermore, despite the growing number of actors engaging in commercial space activity, these talks tend to take place in a handful of wealthy nations, meaning that the number of potential participants and beneficiaries of the space-based economy remains low. This reality exacerbates concerns about technology-related divides and questions of equitable access to the benefits of economic activity in space.

Beyond commercial activities, over the past few decades, there has been growing interest in the use of space as a medium to support tactical military operations and other existing uses related to strategic defense. Both existing and emerging space powers are investing more in military space systems for communications, navigation, and reconnaissance purposes, so as to ensure the operability of a range of capabilities, including drones and precision weapons. Such systems can be used to detect potential targets, attack adversaries, and otherwise harm the interests of others. Many states are also investing in technologies that can disrupt or destroy these same space-based capabilities.

In some cases, PNT assurance is being discussed in conjunction with possible terrestrial ASAT components-such as the development of new space-based shields with the added protection of new ground-based military hardware-and other capabilities for disrupting satellites such as communications and GPS jamming. According to the UN Institute of Disarmament Research, the Chinese, Russian, and U.S. militaries are "developing capabilities that can target space assets, including missile defense interceptors and maneuvering satellites." ¹⁶⁹ These leading militaries are also investing in technologies and techniques designed to counter their rivals' space-enabled intelligence, surveillance and reconnaissance, and precision strike activities.

Undoubtedly, using these capabilities against an adversary's space assets in a conflict or even preemptively could have disastrous consequences for the Earth's orbit and the planet itself. In addition, the development and testing of ASAT capabilities is also worsening the risks posed by hazardous space debris. For instance, debris from China's 2007 direct ascent ASAT test is still in orbit (and will likely remain there for decades). ¹⁷⁰ Although India's more recent March 2019 test was conducted in low orbit (reportedly allowing the debris to disperse over time), this test still posed safety risks, including to the International Space Station. ¹⁷¹

Beyond the aforementioned security concerns, vulnerabilities in satellite and spacecraft computer systems could be potential targets for cyber attacks by states, their proxies, or terrorist groups. And there is mounting concern that state actors could wield offensive cyber capabilities or operations as part of their anti-satellite toolboxes. For instance, an attack on GPS satellites would be felt across a wide range of economic and social sectors, potentially affecting the delivery of essential public services such as power supplies, banking services, and credit card services, all of which rely on GPS. ¹⁷²

The Current Normative Landscape

Technological developments enabling increased commercial and military space activities are reanimating old questions about the nature of human activity in outer space. A set of new questions is also coming to the forefront, answers to which may require adapting the international normative framework governing outer space activities and revising relevant national policies and regulations.

These worries are not new. The Sputnik satellite launch in 1957 and the ensuing space race between the United States and the Soviet Union drove initial concerns that outer space would be used for nonpeaceful purposes, namely for the placement and deployment of nuclear weapons and the deployment of military satellites that could enable or warn of nuclear attacks. Policymakers also worried about commercial satellite applications, particularly those for monitoring, remote sensing, and broadcasting. These technologies could directly capture information from inside a state or beam information into a state. At this time, only the world's two major powers possessed the financial and technological wherewithal necessary to explore and exploit outer space.

As decolonization occurred, a debate unfolded over the growing military uses of outer space. The debate involved sovereignty-related principles such as prior consent and jurisdiction as well as newer principles such as equitable access to the benefits derived from commercial satellite use. The latter resulted in the establishment of the UN Committee on the Peaceful Uses of Outer Space (COPUOS) in 1959, as well as the current body of international space law and related nonbinding measures and guidance. These various instruments have shaped national space policies and regulations ever since.

The 1967 Outer Space Treaty is the central instrument in this body of law and remains the principal framework governing activities in outer space today. The treaty's principles were negotiated to advance international peace and security. It was not intended to regulate human spaceflight and exploration. Rather, it granted rights for the exploration and use of outer space, provided those activities are "carried out for the benefit and in the interests of all countries." ¹⁷³ The principles were designed to ensure that "Outer Space, the Moon and other celestial bodies" remained an arena for peaceful activity and that no single entity or nation could claim sovereignty over outer space. ¹⁷⁴ Throughout the Cold War, adherence to the treaty as well as the major powers' doctrines of deterrence helped ensure that military activity in outer space did not lead to escalated tensions.

Yet concerns about military activity in outer space have persisted. Since the early 1980s, several attempts have been made to restrict certain activities. Numerous proposals have been made at the UN General Assembly or the Conference on Disarmament for the "prevention of an arms race in outer space." ¹⁷⁵ These proposals include a nonbinding UN General Assembly resolution called the Prevention of an Arms Race in Outer Space (PAROS), which has been passed every year since 1982. The resolution "reaffirms the fundamental principles of the 1967 Outer Space Treaty" and "advocates for a ban on the weaponization of space." ¹⁷⁶ It also advocates for "further measures to prevent an arms race in outer space" and "calls on the Conference on Disarmament to establish an ad-hoc committee on issues relating to PAROS."

Other proposals include Chinese- and Russian-backed draft instruments aimed at preventing the placement of weapons in outer space and prohibiting the use of anti-satellite weapons and a related 2008 treaty proposal. In 2014, Moscow and Beijing submitted a new version of a draft proposal called the Treaty on the Prevention of the Placement of Weapons in Outer Space, and Russia proposed a new UN General Assembly resolution containing a pledge against the placement of weapons in outer space. For various legal, technical, and political reasons-including concerns about verifiability or a lack of equitability-these proposals have not advanced. The EU's alternative efforts to promote a nonbinding International Code of Conduct for Outer Space Activities outside the purview of the UN and the Conference on Disarmament have not gained traction either. While there was likely limited disagreement on the substance of the proposed code of conduct, the process of institutionalizing it was mired in problems from the outset.

In 2011, the UN General Assembly established a GGE to study transparency and confidence-building measures related to outer space activities. The group was tasked with developing recommendations on rules of conduct and measures aimed at expanding the transparency of outer space activities and space programs. It produced a consensus report in 2013, welcomed by the UN General Assembly in Resolution 68/50 and subsequent resolutions that encouraged member states to review and implement the proposed measures. The UN General Assembly also referred the report for consideration to the Committee on the Peaceful Uses of Outer Space, the Disarmament Commission, and the Conference on Disarmament. But progress on implementing the GGE recommendations has been slow, and distrust has continued to prevail. To date, only one UN member, the United States, has submitted a report to the General Assembly outlining how it is implementing the recommendations of the 2013 GGE report. ¹⁷⁷

There was some hope that progress could be achieved. For instance, the Conference on Disarmament (finally) established four subsidiary bodies in February 2018 to advance its work. One of these subsidiary bodies was to focus on the "prevention of an arms race in outer space." ¹⁷⁸ Yet, in 2019, the Conference on Disarmament failed to reestablish the four bodies. For its part, the Disarmament Commission has held informal discussions over the past two years on the practical implementation of transparency and confidence-building measures. Member states have agreed to take up the issue in the commission's 2018-2020 three-year cycle, although it is unclear what this will mean in practical terms.

On a slightly more positive note, COPUOS has continued its complementary work on substantive and legal issues relating to peaceful uses of outer space. Building on the 2007 Space Debris Mitigation Guidelines, it is working to develop guidelines aimed at increasing the odds that outer space activities are sustainable over the long term. These more recent efforts play an important confidence-building role as well.

In addition, it appears as if public pressure is effectively generating some good practices regarding ASAT testing and the imperative not to create debris. For instance, the public outcry that followed China's 2007 ASAT test appears to have convinced Beijing that this capability must test in a controlled manner, without blowing up objects and scattering debris in space. But this does not mean that the country will not use alternative ASAT capabilities. According to a recent report, China has conducted numerous tests of technologies "for close approach and rendezvous in both low-earth orbit (LEO) and geosynchronous orbit (GEO) that could lead to a co-orbital ASAT capability." ¹⁷⁹ There is not enough evidence to conclude whether Beijing will gain this operational capability in the near future. The same report notes that Russia, the United States, and numerous other countries also have developed-or are developing-novel ASAT and other counterspace capabilities in five categories: direct ascent, co-orbital, electronic warfare, directed energy, and cyber.

What Lies Ahead?

Looking to the future, a number of technology-related issues will likely remain or appear on the international space agenda. The space environment is radically different now than it was when the existing normative framework was adopted. Space law will likely need to be revisited to account for new forms of commercial exploration, resolve issues relating to states' obligations in terms of international cooperation and coordination, and ensure equitable future economic access in accordance with the Outer Space Treaty.

When the treaty was being negotiated, there were already attempts to restrict space exploration to government enterprises, largely in reaction to the U.S. private sector's dominance of the commercial satellite industry. The resulting compromise laid out in Article VI of the treaty permits private activities in outer space, making national governments legally responsible for the actions of their nationals. Yet today new commercial activities are raising new questions. The first missions to probe the moon for resources are already scheduled to launch in 2020. In accordance with their obligations under international law to "authorise and continually supervise" such activity conducted by their nationals, several countries-including Luxemburg, the United Arab Emirates, and the United States-have already adopted new legislation concerning resource extraction rights in space, while also loosening regulations to enable commercial activity. ¹⁸⁰ Other states-including France, Germany, Japan, and the UK-will soon follow suit.

But it is likely that this dash to exploit space resources will foment new tensions. Resource recovery in space will likely be scrutinized on the basis of sovereignty. Article I of the treaty states that "outer space, including the moon and other celestial bodies, shall be free for exploration and use by all States without discrimination of any kind, on a basis of equality and in accordance with international law, and there shall be free access to all areas of celestial bodies." But the treaty's subsequent article states clearly that space resources are "not subject to national appropriation." Some experts interpret this latter provision as a prohibition on the extraction of space resources and have criticized recent U.S. legislation for entitling U.S. companies engaging in commercial recovery of space resources "to possess, own, transport, use and sell the space resources obtained." ¹⁸¹ Furthermore, countries with limited capabilities will likely decry unequal and inequitable access to space resources. It may well be that the emerging body of national rules and regulations will eventually need to align with international views on the activities in question in the form of an international agreement on managing commercial space exploration and utilization.

An important step forward was the creation in 2017 of the Hague International Space Resources Governance Working Group. In the absence of a clear framework to govern the extraction of space resources, this multi-stakeholder group has been studying existing concepts "to ensure that they meet existing treaty obligations regarding on-orbit operations and space resource rights." ¹⁸² The group's overall aim is to complement other global, regional, and national efforts and prepare a foundation for a new international regulatory framework for space resource-related activities. Reaching consensus on what is and is not permitted under the terms of the treaty will undoubtedly be challenging. At the same time, the number of actors intent on leveraging space-to foster commercial activity and spur economic growth, for instance-may actually incentivize dialogue and agreement on common principles. So, too, should other more global objectives such as the UN Sustainable Development Goals.

On questions relating to military uses of outer space, it is likely that the debate over a space-based arms race will continue and remain divisive. In 2017, the UN General Assembly established a new GGE under the Conference on Disarmament with the specific mandate to "consider and make recommendations on substantial elements of an international legally binding instrument on the prevention of an arms race in outer space, including, inter alia, on the prevention of the placement of weapons in outer space." ¹⁸³ There had been serious doubts that a new GGE could achieve much in the current environment, as all major powers and a growing number of smaller states are developing counterspace capabilities. Two sessions of the GGE were held, one in 2018 and another in 2019. However, the group failed to produce a consensus report.

Again, the potential for economic growth and development derived from commercial space activity could serve as an important incentive for such efforts. Nonetheless, in each of these areas, maintaining dialogue on the core principles and norms that should guide activity in outer space- particularly those that restrain destabilizing behaviors and ensure equitable access to potential benefits-will be imperative as will ensuring that the private sector, industry associations, and other relevant actors in these government-led processes and mechanisms are given greater opportunities to voice their views.

Conclusion

The array of new technologies emerging on the world stage, the new threats they can pose, and the associated governance dilemmas highlight a set of common themes. First, an explosive growth of technological innovation is outstripping the capacity (or willingness) of technology creators, private investors, national governments, and the existing multilateral system to understand, monitor, and effectively govern the attendant effects and consequences.

Second, these technologies offer great benefits to society while, at the same time, imposing grave risks ranging from malicious use to exacerbating socioeconomic inequalities. Despite the range of initiatives under way and broad acknowledgment of the governance challenges at hand, it is alarming to see how lackluster the responses have been to date. Without the necessary political will and concerted cooperation across actors and regions, it will be impossible to ensure that technologies are used responsibly; their benefits are shared equitably; and the serious ethical, social, economic, personal safety, environmental, and national security risks are managed deftly.

Third, this absence of political will and cooperation is driven, increasingly, by structural dynamics that undermine efforts to reach consensus. Technology is a deeply embedded facet of geopolitical competition, particularly among the great powers, and such innovation is central to both spurring economic growth and gaining a military edge. As a result, few powerful states have been willing to restrict their pursuit of perceived technological advantages. The effects of this security dilemma are growing more acute as economic, political, and military competition among China, Russia, the United States, and a growing number of middle powers increases. This competitive dynamic is already evident in the areas of ICT and outer space, and similar trends are emerging in relation to AI, biotechnology, and other areas not discussed in this paper such as quantum computing and nanotechnology. States (including the great powers) have overcome heated competition and heightened tensions in the past. They should commit to doing the same now.

Fourth, the economic model undergirding global technology markets, driven by the need for companies to reach economies of scale in terms of how many users they have, has led many companies to focus on offering products and services first and fixing resulting problems later. This mentality has left software, hardware, and supply chains vulnerable to disruption, manipulation, and capture by governments, criminal groups, and other malicious actors around the world. Again, this state of affairs is evident in the field of ICT, and similar behavior will likely surface in other technological areas as well. Notably, these fields call on a wide array of (sometimes antagonistic) actors to coordinate and reach agreement, including individual states, the companies driving innovation, technologists, civil society, international organizations, and academic institutions.

While these dynamics offer an unfavorable outlook for cooperation, there is, in parallel, a wide range of efforts already under way to deal with emerging challenges and risks. Some of these actions are being spearheaded by governments, while others are the product of a host of actors from across an equally wide array of disciplines and sectors. These efforts-complex, fragmented, and wide-ranging (and often nonproductive) as they may be-need to be acknowledged, strengthened, and scaled. Lessons-both positive and negative-must be shared across regions, as these experiences might inform more efficacious approaches or models for effective governance and multiactor cooperative arrangements.

The UN, too, should be able to support its member states and all other stakeholders by facilitating dialogue and cooperation on the policy and normative challenges stemming from technological advances. Through a number of initiatives, including his Agenda for Disarmament, his Strategy on New Technologies, and the establishment of a High-Level Panel on Digital Cooperation, the UN secretary general has acknowledged this. ¹⁸⁷ The initiatives signal a recognition that these advances are emerging at a critical moment in international relations, a time when consensus is becoming increasingly difficult to achieve and during which efforts to manage and govern new risks and challenges are becoming increasingly complicated. Like other leaders, he understands that supporting ongoing efforts and facilitating dialogue and cooperation will be difficult, yet he also grasps that the UN has a responsibility to safeguard its principles and purposes. The body's member states (and their national governments) should do the same, as the world's dependence on cutting-edge technology continues to deepen.

About the Author

Camino Kavanagh is a visiting fellow in the Department of War Studies at King's College London. She also works as an independent consultant, with current contracts including a senior advisory role with the UN Department of Political Affairs' Policy and Mediation Division on a project relating to conflict prevention and emerging technologies. Kavanagh served as rapporteur/consultant of the 2016-2017 United Nations Group of Governmental Experts on Developments in the Field of Information and Telecommunications in the Context of International Security. She is involved in a number of policy and research initiatives on ICT and emerging technologies as they relate to conflict and international security. Kavanagh received her PhD from the Department of War Studies at King's College London in 2016.

Acknowledgments

This publication could not have been produced without the valuable contributions of many individuals. First, the author would like to thank the Carnegie Endowment for International Peace for agreeing to publish it, particularly George Perkovich, Kate Charlet, Tim Maurer, Charlotte Stanton, and Ryan DeVries for reviewing and providing such valuable feedback. The author also offers thanks to the United Nations Foundation for funding the initial research that led to this publication and especially to Kaysie Brown and Megan Roberts for their constructive feedback. She also wishes to express her gratitude to Marc Jacquand and Fabrizio Hochschild of the Executive Office of the UN Secretary General for the opportunity to work on the Secretary-General's Strategy on New Technologies in 2018 and develop some of that work further through this publication.

Finally, the author is grateful to a number of experts who shared their insights on the governance and normative dilemmas emerging in their respective fields. This list includes Kerstin Vignard and Daniel Porras at the UN Institute for Disarmament Research, Gillian Goh at the UN Office of Disarmament Affairs, Piers Millet and Carrick Flynn at the Future of Humanity Institute, Beth Cameron and Jacob Jordan at the Nuclear Threat Initiative, Michael Page at OpenAI, Madeline Carr at University College London, Paul Cornish at LSE IDEAS, and Nigel Inkster at the International Institute for Strategic Studies.

Notes

¹ Camino Kavanagh and Paul Cornish, "Preventive Diplomacy, ICT and Inter-State Conflict: A Review of Current Practice with Observations," Swiss Federal Department of Foreign Affairs, (forthcoming 2019).

² Roger Brownsword, Eloise Scotford, and Karen Yeung, (eds.), Oxford Handbook of Law, Regulation and Technology (Oxford: Oxford Handbooks Online, 2017).

³ Harvard professor Joseph S. Nye Jr. refers to this framework as "the global regime complex governing global cyber activities." See Joseph S. Nye, Jr., "The Regime Complex for Governing Global Cyber Activities," Global Commission on Internet Governance, May 2014, https://www.cigionline.org/sites/default/files/gcig_paper_no1.pdf.

⁴ World Economic Forum, "The Fourth Industrial Revolution: What It Means, How to Respond," January 14, 2016, https://www.weforum.org/agenda/2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/.

⁵ World Economic Forum, "Agile Governance: Reimagining Policy Making in the Fourth Industrial Revolution," January 2018, http://www3.weforum.org/docs/WEF_Agile_Governance_Reimagining_Policy-making_4IR_report.pdf.

⁶ World Economic Forum, "The Fourth Industrial Revolution: What It Means, How to Respond."

⁷ Yochai Benkler, "Don't Let Industry Write the Rules for AI," Nature, May 2019; Vol. 569 (7755):161, https://www.nature.com/magazine-assets/d41586-019-01413-1/d41586-019-01413-1.pdf; and Luciano Floridi and Josh Cowls, "A Unified Framework of Five Principles for AI in Society," Harvard Data Science Review, June 23, 2019, https://doi.org/10.1162/99608f92.8cd550d1.

⁸ Benkler, "Don't Let Industry Write the Rules for AI."

⁹ Anne Marie Engtoft Larsen, "Regulation for the Fourth Industrial Revolution," WEF Podcast Episode 5, [interview portion with Rob Sparrow of Monash University], January 30 2018, https://www.weforum.org/agenda/2018/01/podcast-regulation-for-the-fourth-industrial-revolution/.

¹⁰ Joseph S. Nye Jr., "The Regime Complex for Governing Global Cyber Activities."

¹³ Laurent Gisel and Lukasz Olejnik, "The Potential Human Cost of Cyber Operations: Starting the Conversation," November 14, 2018, International Committee of the Red Cross Humanitarian Law and Policy (blog), https://blogs.icrc.org/law-and-policy/2018/11/14/potential-human-cost-cyber-operations/.

¹⁴ UN Office of Disarmament Affairs, "Developments in the Field of Information and Telecommunications in the Context of International Security," https://www.un.org/disarmament/ict-security/; UN General Assembly, "Group of Governmental Experts on Developments in the Field of Information and Telecommunications in the Context of International Security," Document A/65/201, July 30, 2010, https://undocs.org/A/65/201; UN General Assembly, "Group of Governmental Experts on Developments in the Field of Information and Telecommunications in the Context of International Security," Document A/68/98*, June 24, 2013, http://www.unidir.org/files/medias/pdfs/developments-in-the-field-of-information-and-telecommunications-in-the-context-of-international-security-2012-2013-a-68-98-eng-0-518.pdf; and UN General Assembly, "Group of Governmental Experts on Developments in the Field of Information and Telecommunications in the Context of International Security," Document A/70/174, July 22, 2015, https://undocs.org/A/70/174. On the applicability of international law, see the international law sections in UN General Assembly Document A/68/98* and UN General Assembly Document A/70/174. For an analysis of the outcome of these reports, see Camino Kavanagh, "The United Nations, Cybersecurity and International Peace and Security: Responding to Complexity in the 21st Century," United Nations Institute for Disarmament Research (UNIDIR), 2017, http://www.unidir.org/files/publications/pdfs/the-united-nations-cyberspace-and-international-peace-and-security-en-691.pdf.

¹⁵ See Section III of UN General Assembly Document A/70/174, 7.

¹⁶ UN General Assembly, "Developments in the Field of Information and Telecommunications in the Context of International Security," Resolution 70/237, December 2015, 3, https://undocs.org/en/A/RES/70/237.

²⁰ On VEPs, their genesis, and their character, see Katherine Charlet, Sasha Romanosky, and Bert Thompson, "It's Time for the International Community to Get Serious about Vulnerability Equities," Lawfare, November 15, 2017, https://www.lawfareblog.com/its-time-international-community-get-serious-about-vulnerability-equities; and Sven Herpig and Ari Schwartz, "The Future of Vulnerability Equities Processes Around the World," Lawfare, January 4, 2019, https://www.lawfareblog.com/future-vulnerabilities-equities-processes-around-world.

²¹ Laurent Gisel and Lukasz Olejnik, "The Potential Human Cost of Cyber Operations: ICRC Expert Meeting 14-16 November 2018 - Geneva," International Commitee of the Red Cross, https://www.icrc.org/en/document/potential-human-cost-cyber-operations. and the UN Department of Political and Peacebuilding Affairs and Humanitarian Dialogue Center, "Digital Technologies and Mediation in Armed Conflict," March 2019, https://peacemaker.un.org/sites/peacemaker.un.org/files/DigitalToolkitReport.pdf. The digital version of the toolkit is available here: https://peacemaker.un.org/digitaltoolkit.

²² Digital Europe, "DIGITALEUROPE's Position Paper on the European Commission's Proposal for a European Framework for Cybersecurity Certification Scheme for ICT Products and Services," December 15, 2017, https://www.digitaleurope.org/resources/digitaleuropes-position-paper-on-the-european-commissions-proposal-for-a-european-framework-for-cybersecurity-certification-scheme-for-ict-products-and-services/.

²³ Eileen Donahue, Melissa Hathaway, James A. Lewis, Joseph S. Nye Jr., Eneken Tikk, and Paul Twomey, "Getting Beyond Norms: New Approaches to International Cyber Security Challenge," Center for International Governance Innovation, September 7, 2017, 9, https://www.cigionline.org/sites/default/files/documents/Getting%20Beyond%20Norms.pdf.

²⁴ Karen Yeung, "Regulation for the Fourth Industrial Revolution," WEF Podcast Episode 5, [interview portion with Karen Yeung, formerly of King's College London], January 30 2018, https://www.weforum.org/agenda/2018/01/podcast-regulation-for-the-fourth-industrial-revolution/".

²⁷ See the following explanation of the I* organizations: "I* Organizations," Asia-Pacific Network Information Center, https://www.apnic.net/community/ecosystem/iorgs/.

²⁸ "Call to Protect the Public Core of the Internet," Global Commission on the Stability of Cyberspace, November 2017, https://cyberstability.org/wp-content/uploads/2018/07/call-to-protect-the-public-core-of-the-internet.pdf.

²⁹ Kofi Annan Commission on Elections and Democracy in the Digital Age, "About the Commission," https://www.kofiannanfoundation.org/our-work/kofi-annan-commission/.

³⁰ "Norm Package Singapore," Global Commission on the Stability of Cyberspace, November 2018, https://cyberstability.org/wp-content/uploads/2018/11/GCSC-Singapore-Norm-Package-3MB.pdf.

³¹ "One Internet," Center for International Governance Innovation and Chatham House, 2016, Available at: https://www.cigionline.org/sites/default/files/gcig_final_report_-_with_cover.pdf.

³² See Siemens, "The Charter of Trust Takes a Major Step Forward to Advance Cybersecurity," February 15, 2019, https://www.siemens.com/press/en/feature/2018/corporate/2018-02-cybersecurity.php.

³³ Siemens, "Presentation on the Siemens Charter of Trust," March 2019, https://assets.new.siemens.com/siemens/assets/public.1552654550.55badda4-4340-46d3-b359-f570e7d1f4c2.charter-of-trust-presentation-en.pdf.

³⁵ Microsoft, "A Digital Convention to Protect Cyberspace," Microsoft Policy Papers, https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RW67QH.

³⁶ Tom Burt, "Announcing the Defending Democracy Program," Microsoft, April 13, 2018, https://blogs.microsoft.com/on-the-issues/2018/04/13/announcing-the-defending-democracy-program/.

³⁸ Microsoft, "Digital Peace Now," https://digitalpeace.microsoft.com.

³⁹ "Cybersecurity: Paris Call for Trust and Security in Cyberspace," France Diplomatie, November 12, 2018, https://www.diplomatie.gouv.fr/en/french-foreign-policy/digital-diplomacy/france-and-cyber-security/article/cybersecurity-paris-call-of-12-november-2018-for-trust-and-security-in.

⁴⁴ Deborah Brown, "What to Expect at the 2018 ITU Plenipotentiary Conference and What It Means for the Internet," Council on Foreign Relations, March 12, 2018, https://www.cfr.org/blog/what-expect-2018-itu-plenipotentiary-conference-and-what-it-means-internet.

⁴⁶ UN Office for Disarmament Affairs, "Submission of the Report of the Secretary-General on Resolution 73/27 on Developments in the Field of Information and Telecommunications in the Context of International Security and Resolution 73/266 on Advancing Responsible State Behaviour in Cyberspace in the Context of International Security," UN Document ODA/2019-00116/ICTS, February 6, 2019, https://s3.amazonaws.com/unoda-web/wp-content/uploads/2019/02/19-074nve.pdf.

⁴⁷ Brian Egan [former legal adviser to the U.S. Department of State], "Remarks on Stability and International Law in Cyberspace," U.S. Department of State, November 10 2016, https://2009-2017.state.gov/s/l/releases/remarks/264303.htm.

⁵⁰ UN Office for Disarmament Affairs, "Submission of the Report of the Secretary-General on Resolution 73/27 on Developments in the Field of Information and Telecommunications in the Context of International Security and Resolution 73/266 on Advancing Responsible State Behaviour in Cyberspace in the Context of International Security."

⁵¹ G7, "Dinard Declaration on the Cyber Norm Initiative," G7 Foreign Ministers Meeting, Dinard - Saint-Malo, April 6-7, 2019, https://www.elysee.fr/admin/upload/default/0001/04/d37b5326306c7513b58c79d26938f678d95cb2ff.pdf.

⁵² Camino Kavanagh, "Stemming the Exploitation of ICT Threats and Vulnerabilities: An Overview of Current Trends, Enabling Dynamics and Private Sector Responses," UNIDIR, June 25 2019, http://www.unidir.org/files/publications/pdfs/stemming-the-exploitation-of-ict-threats-and-vulnerabilities-en-805.pdf.

⁵⁴ John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude E. Shannon, " Proposal for the Dartmouth Summer Research Project on Artificial Intelligence," AI Magazine 27, no. 4 ( 2006): 12, https://aaai.org/ojs/index.php/aimagazine/issue/view/165. (This piece was originally written in August 1955 but was published in this journal issue.

⁵⁵ Floridi and Cowls, "A Unified Framework of Five Principles for AI in Society."

⁵⁶ Rodney Brooks, "The Origins of Artificial Intelligence," FoR&AI, April 27, 2018, https://rodneybrooks.com/forai-the-origins-of-artificial-intelligence/; John Mallery, "Intelligent Computation in National Defense Applications: Artificial Intelligence or Magic?," the 2018 Roundtable on Military Cyber Stability, Framing presentation in the session entitled "Applications of New Technologies in National Defense," Washington, DC, July 17, 2018; and Rodney Brooks, "The Seven Deadly Sins of AI Predictions," MIT Technology Review, October 6, 2017, https://www.technologyreview.com/s/609048/the-seven-deadly-sins-of-ai-predictions/.

⁵⁸ See, for instance, Max Tegmark, Life 3.0: Being Human in the Age of Artificial Intelligence, (New York: Knopf, 2017); and Nick Bockstrom, Superintelligence: Paths, Dangers, Strategies, (Oxford: Oxford University Press, 2014).

⁵⁹ "The Weaponization of Increasingly Autonomous Technologies: no. 8 Artificial Intelligence (A Primer for CCW Delegates)," UNIDIR, 2018, http://www.unidir.ch/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-artificial-intelligence-en-700.pdf.

⁶⁰ See the International Labor Organization's Future of Work research series and its Global Commission on the Future of Work, the OECD's Future of Work Initiative, and the WEF's Preparing for the Future of Work Initiative. On the future of food, see Lassonde School of Engineering, "Can AI Help Feed the World? The Future of Food is Here," Medium, April 16, 2018, https://medium.com/lassondeschool/can-ai-help-feed-the-world-the-future-of-food-is-here-429e7c10290b; and Emiko Terazono, "Future of Food: Inside Agritech's Silicon Valley," Financial Times, October 15, 2018, https://www.ft.com/content/199cae4c-cbc6-11e8-b276-b9069bde0956. On the future of humanity, see research by the Future of Humanity Institute and the Future of Life Institute as well as Michio Kako, The Future of Humanity: Our Destiny in the Universe, (New York: Penguin Books, 2019).

⁶¹ James Vincent, "Putin Says the Nation That Leads in AI 'Will be the Ruler of the World,'" Verge, September 4, 2017, https://www.theverge.com/2017/9/4/16251226/russia-ai-putin-rule-the-world; Zhou Xin and Choi Chi-yuk, "Develop and Control: Xi Jinping Urges China to Use Artificial Intelligence in Race for Tech Future," South China Morning Post, October 31, 2018, https://www.scmp.com/economy/china-economy/article/2171102/develop-and-control-xi-jinping-urges-china-use-artificial; and White House Office of Science and Technology Policy, "Accelerating America's Leadership in Artificial Intelligence," February 11, 2019, https://www.whitehouse.gov/articles/accelerating-americas-leadership-in-artificial-intelligence/.

⁶³ Floridi and Cowls, "A Unified Framework of Five Principles for AI in Society."

⁶⁴ Future of Life Institute, "Asimolar AI Principles," https://futureoflife.org/ai-principles/.

⁶⁵ Campaign to Stop Killer Robots, "About Us," https://www.stopkillerrobots.org/about/.

⁶⁶ Future of Life Institute, "Lethal Autonomous Weapons Pledge," https://futureoflife.org/lethal-autonomous-weapons-pledge/?cn-reloaded=1.

⁶⁷ Future of Life Institute, "Autonomous Weapons: an Open Letter From AI and Robotics Researchers," https://futureoflife.org/open-letter-autonomous-weapons/.

⁶⁹ Institute of Electrical and Electronics Engineers (IEEE), " Ethically Aligned Design: A Vision for Prioritizing Human Well-Being with Autonomous and Intelligent Systems," first edition, 2016, https://ethicsinaction.ieee.org/#read; and IEEE, "The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems," https://standards.ieee.org/industry-connections/ec/autonomous-systems.html.

⁷¹ Partnership on AI, "Tenets," https://www.partnershiponai.org/tenets/.

⁷² Salil Sethi, "7 Predictions for Artificial Intelligence in 2019," OpenSource.com, February 5, 2019, https://opensource.com/article/19/2/predictions-artificial-intelligence.

⁷³Open AI, "Open AI Charter," https://blog.openai.com/openai-charter/.

⁷⁵ Google, "AI at Google: Our Principles," https://blog.google/technology/ai/ai-principles/.

⁷⁶ Microsoft, "Microsoft AI Principles," https://www.microsoft.com/en-us/ai/our-approach-to-ai.

⁷⁷ Accenture, "Responsible AI and Robotics: An Ethical Framework," https://www.accenture.com/gb-en/company-responsible-ai-robotics.

⁷⁹ Jessica Cussins Newman, "Toward AI Security: Global Aspirations for a More Resilient Future," University of California, Berkeley, Center for Long-Term Cybersecurity, February 2019, https://cltc.berkeley.edu/wp-content/uploads/2019/02/Toward_AI_Security.pdf.

⁸¹ World Bank, "The Changing Nature of Work."

⁸² Daniel Araya, "Artificial Intelligence and the End of Government," Forbes, January 4, 2019, https://www.forbes.com/sites/danielaraya/2019/01/04/artificial-intelligence-and-the-end-of-government/#27e90bc3719b; and UK Department for Business, Energy, and Industrial Strategy and Department for Digital, Culture, Media and Sport, "AI Sector Deal," updated May 2019, https://www.gov.uk/government/publications/artificial-intelligence-sector-deal/ai-sector-deal.

⁸³ Greg Allen and Taniel Chan, "Artificial Intelligence and National Security," Harvard Kennedy School Belfer Center for Science and International Affairs, July 2017, https://www.belfercenter.org/sites/default/files/files/publication/AI%20NatSec%20-%20final.pdf.

⁸⁴ Kelley M. Sayler, "Artificial Intelligence and National Security," Congressional Research Service, Updated January 2019, 11, https://fas.org/sgp/crs/natsec/R45178.pdf.

⁸⁵ Cussins Newman, "Toward AI Security."

⁸⁶ European Group on Ethics in Science and New Technologies, "Statement on Artificial Intelligence, Robotics and 'Autonomous' Systems," European Commission Directorate-General for Research and Innovation, March 9, 2018, https://ec.europa.eu/research/ege/pdf/ege_ai_statement_2018.pdf.

⁸⁷ "EU Declaration of Cooperation on Artificial Intelligence," European Commission, April 10, 2018, https://ec.europa.eu/jrc/communities/en/node/1286/document/eu-declaration-cooperation-artificial-intelligence.

⁸⁸ European Commission, "Communication on Cooperation on Artificial Intelligence for Europe," (COM(2018) 237 final), April 25, 2018, https://ec.europa.eu/digital-single-market/en/news/communication-artificial-intelligence-europe.

⁹⁰ Organization for Economic Co-operation and Development (OECD), "OECD Principles on AI," May 2019, https://www.oecd.org/going-digital/ai/principles/. The OECD boasts that these "are the first such principles signed up to by governments" and that, in addition to OECD members, a number of other countries-including Argentina, Brazil, Colombia, Costa Rica, Peru, and Romania-have signed up.

⁹¹ G20, "G20 Ministerial Statement on Trade and Digital Economy," June 2019, https://g20trade-digital.go.jp/dl/Ministerial_Statement_on_Trade_and_Digital_Economy.pdf.

⁹² Council of Europe Committee of Ministers, "Technological Convergence, Artificial Intelligence and Human Rights: Recommendation 2102," 2017; Council of Europe Committee of Ministers, "Technological Convergence, Artificial Intelligence and Human Rights: Reply to Recommendation," October 19, 2017; Council of Europe Commissioner for Human Rights "Unboxing Artificial Intelligence: 10 Steps to Protect Human Rights," May 2019; and Council of Europe Committee of Ministers, " Declaration by the Committee of Ministers on the Manipulative Capabilities of Algorithmic Processes" February 13, 2019. All three of these documents available on the Council of Europe's webpage on artificial Intelligence. See Council of Europe, "Council of Europe and Artificial Intelligence," https://www.coe.int/en/web/artificial-intelligence/home.

⁹³ Cussins Newman, "Toward AI Security."

⁹⁵ For more details, see Group of Governmental Experts of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, " Report of the 2017 Group of Governmental Experts on Lethal Autonomous Weapons Systems (LAWS)," (UN Document CCW/GGE.1/2017/3), December 22, 2017, https://undocs.org/CCW/GGE.1/2017/3.

⁹⁷ The twenty-eight countries are: Algeria, Argentina, Austria, Bolivia, Brazil, Chile, China, Colombia, Costa Rica, Cuba, Djibouti, Ecuador, Egypt, El Salvador, Ghana, Guatemala, the Holy See, Iraq, Mexico, Morocco, Nicaragua, Pakistan, Panama, Peru, the State of Palestine, Uganda, Venezuela, and Zimbabwe. See Future of Life Institute, "Lethal Autonomous Weapons Pledge," https://futureoflife.org/lethal-autonomous-weapons-pledge/.

⁹⁹ Campaign to Stop Killer Robots, "UN Head Calls for a Ban," November 12, 2018, https://www.stopkillerrobots.org/2018/11/unban/.

¹⁰¹ For differing reasons, four of the five permanent members of the UN Security Council have reportedly rejected moving to negotiate new international law on autonomous weapons.

¹⁰³ Group of Governmental Experts of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, "Report of the 2018 session of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems."

¹⁰⁵ Floridi and Cowls, "A Unified Framework of Five Principles for AI in Society."

¹⁰⁷ Tristan Greene, "Google's Principles for Developing AI Aren't Good Enough," Next Web, June 8, 2018, https://thenextweb.com/artificial-intelligence/2018/06/08/googles-ai-principles-7-shades-of-gray-area/.

¹⁰⁹ Natasha Lomas, "DeepMind Now Has an AI Ethics Research Unit. We Have a Few Questions for It . . . ," TechCrunch, October 4, 2017, https://techcrunch.com/2017/10/04/deepmind-now-has-an-ai-ethics-research-unit-we-have-a-few-questions-for-it/?guccounter=1.

¹¹⁰ DeepMind, "DeepMind Ethics and Society: About Us," https://deepmind.com/applied/deepmind-ethics-society/.

¹¹¹ DeepMind, "Principles," https://deepmind.com/applied/deepmind-ethics-society/principles/.

¹¹⁴ Axon, "Axon AI and Policing Technology Ethics Board," https://es.axon.com/info/ai-ethics.

¹¹⁶ Axon, "Axon AI and Policing Technology Ethics Board."

¹¹⁷ Cussins Newman, "Toward AI Security," 4.

¹¹⁸ See, for example, UNIDIR, "The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values," 2015, http://www.unidir.org/files/publications/pdfs/considering-ethics-and-social-values-en-624.pdf; UNIDIR, "The Weaponization of Increasingly Autonomous Technologies: Considering How Meaningful Human Control Might Move the Discussion Forward," 2014, https://www.files.ethz.ch/isn/185834/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf; and IISS, The Military Balance 2018, Chapter 1, Part Two "Big Data, Artificial Intelligence and Defence."

¹²⁰ See UNIDIR, "Trust but Verif AI: Options for Regulating Defence and Security AI Technologies," http://www.unidir.org/programmes/security-and-technology/trust-but-verif-ai-options-for-regulating-defence-and-security-ai-technologies.

¹²¹ United Nations, "Convention on Biological Diversity," Article 2: Use of Terms, https://www.cbd.int/convention/articles/default.shtml?a=cbd-02.

¹²² Author interview with NTI, July 2018. For the India figure, see Burrill Media, Biotechnology Industry Organization, and the Association of Biotechnology Led Enterprises, "Accelerating Growth: Forging India's Bioeconomy," 2014, https://www.bio.org/sites/default/files/files/Burrill_AcceleratingGrowth_India-6-9-final.pdf.

¹²³ European Commission's Knowledge Centre for Bioeconomy, "EU Bioeconomy," September 1, 2018, https://ec.europa.eu/knowledge4policy/sites/know4pol/files/2018_09_01_bioeconomi_infographic_update_en.pdf. The infographic is based on the following peer-reviewed publication: Tévécia Ronzon and Robert M'Barek, "Socioeconomic Indicators to Monitor the EU's Bioeconomy in Transition," Sustainability 10 no. 6, (May 26, 2018), https://www.mdpi.com/2071-1050/10/6/1745.

¹²⁴ James C. Greenwood, "Biotech in China," Biotechnology Innovation Organization, January 2013, https://www.bio.org/sites/default/files/files/Biotechnology-Industry-Pg62-64.pdf.

¹²⁵ Author interview with NTI, July 2018; European Commission, "What Is the Bioeconomy?," May 9, 2018, https://ec.europa.eu/research/bioeconomy/index.cfm; [Obama] White House, "National Bioeconomy Blueprint," April 2012, https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/national_bioeconomy_blueprint_april_2012.pdf; Beate El-Chichakli et al., "Policy: Five Cornerstones of a Global Bioeconomy," Nature 535, no. 7611 (2016): 221-222, https://www.nature.com/news/policy-five-cornerstones-of-a-global-bioeconomy-1.20228; and Beth Baker, "Synthetic Biology and the Marketplace: Building the New Bioeconomy," BioScience 67, no. 10 (2017): 877-883, https://doi.org/10.1093/biosci/bix101.

¹²⁶ Arjun Bhutkar, "Synthetic Biology: Navigating the Challenges Ahead," Journal of Biolaw and Business 8 no. 2, (2005):19-29.

¹²⁸ U.S. National Library of Medicine, "Genetics Home Reference: Your Guide to Understanding Genetic Conditions," June 25, 2019, https://ghr.nlm.nih.gov/primer/basics/gene.

¹²⁹ See Martin Jinek et al., "A Programmable Dual-RNA-Guided DNA Endonuclease in Adaptive Bacterial Immunity," Science, 337, no. 6096, (August 17, 2012): 816-821, https://science.sciencemag.org/content/337/6096/816; and Le Cong et al., "Multiplex Genome Engineering Using CRISPR/Cas Systems," Science, 339, no. 6121 (February 15, 2013), 819-823, https://science.sciencemag.org/content/339/6121/819.

¹³² World Health Organization, "The Guidance Framework for Testing Genetically Modified Mosquitoes," 2014, http://www.who.int/tdr/publications/year/2014/guide-fmrk-gm-mosquit/en/.

¹³³ Author interview with NTI, June 2018.

¹³⁴ U.S. National Academies and the Department of Homeland Security, "Biological Attack, Human Pathogens, Biotoxins and Agricultural Threats: A Fact Sheet," 2004, https://www.dhs.gov/sites/default/files/publications/prep_biological_fact_sheet.pdf

¹³⁶ Maski Imai et al., "Experimental Adaptation of an Influenza H5 HA Confers Respiratory Droplet Transmission to a Reassortant H5 HA/H1N1 Virus in Ferrets," Nature, 486, (June 21, 2012), 420-428, https://www.nature.com/articles/nature10831; and Colin A. Russell et al., "The Potential for Respiratory Droplet-Transmissible A/H5N1 Influenza Virus to Evolve in a Mammalian Host," Science, 336 (June 22, 2012): 1541-1547, https://science.sciencemag.org/content/336/6088/1541.

¹³⁸ Ibid. Also see: European Committee for Standardization, "Laboratory Risk Management," September 2011, http://www.uab.cat/doc/CWA15793_2011.

¹³⁹ National Academy of Sciences and National Academy of Medicine, "Human Genome Editing: Science, Ethics and Governance," 2017, https://www.nap.edu/catalog/24623/human-genome-editing-science-ethics-and-governance. On the highly controversial practice of germline editing, see Joel Achenbach, "Ethicists Advise Caution in Applying CRISPR Gene Editing to Humans," Washington Post, February 14, 2017, https://www.washingtonpost.com/news/speaking-of-science/wp/2017/02/14/ethicists-advise-caution-in-applying-crispr-gene-editing-to-humans/?utm_term=.dfd0a40218a2.

¹⁴³ According to the New York Times, Joseph DeAngelo was identified and later charged with twenty-six counts of murder and kidnapping "after a genealogist helped investigators in California identify a third cousin of Mr. DeAngelo's through GED match and other genealogical records." See Elizabeth Joh, "Want to See My Genes? Get a Warrant," New York Times, June 11, 2019, https://www.nytimes.com/2019/06/11/opinion/police-dna-warrant.html.

¹⁴⁷ According to an article in Clinical Microbiology and Infection, the murder of Hungarian dissident Georgi Markov in London in 1978 with a ricin pellet that was injected into the victim with an umbrella "could be considered an act of biocrime." See Hugo Jan Jansen et al., "Biological Warfare, Terrorism and Biocrime," Clinical Microbiology and Infection, 20 no. 6, (June 2014): 488-496, https://reader.elsevier.com/reader/sd/pii/S1198743X14641732?token=AFE37540897C1FA821BF43B528E3FE75A1DCEB1FDE9FE237962B44A475988F1FC8433AA1E092F2437BD416B51AC884A9.

¹⁴⁹ Federal Bureau of Investigation, "Amerithrax or Anthrax Investigation," https://www.fbi.gov/history/famous-cases/amerithrax-or-anthrax-investigation.

¹⁵⁰ Interviews with Beth Cameron and Jacob Jordan, NTI, July 2018.

¹⁵¹Gryphon Scientific, "Risk and Benefit Analysis for Gain of Function Research: Final Report," 2016, 177-179, http://gryphonsci.wpengine.com/wp-content/uploads/2018/12/Risk-and-Benefit-Analysis-of-Gain-of-Function-Research-Final-Report-1.pdf.

¹⁵² Author interview with Richard Lennane, Geneva Disarmament Platform, January 2018.

¹⁵³ The request for the opinion was made in a letter from European Commissioner for Research, Innovation and Science Carlos Moedas in July 2018. See Carlos Moedas, "Untitled Letter," European Commission, July 10, 2018, https://ec.europa.eu/info/sites/info/files/research_and_innovation/ege/letter_chair_of_the_ege_group.pdf.

¹⁵⁶ See World Health Organization Advisory Committee on Human Genome Editing, "Global Health Ethics: Human Genome Editing," December 14, 2018, https://www.who.int/ethics/topics/human-genome-editing/en/.

¹⁵⁸ Author interviews with Beth Cameron and Jacob Jordan, NTI, May 2018.

¹⁶⁰ Author interview with Beth Cameron, NTI, May 2018.

¹⁶² Benjamin Bahney and Joshua Pearl, "Why Creating a Space Force Changes Nothing: Space Has Been Militarised From the Start," Foreign Affairs, March 26, 2019, https://www.foreignaffairs.com/articles/space/2019-03-26/why-creating-space-force-changes-nothing; and Camino Kavanagh, "Information Technology and the State: The Long View," PhD Thesis, King's College London Department of War Studies, unpublished, 2016.

¹⁶³ OECD, "The Space Economy at a Glance," 2014, 3, https://www.oecd.org/sti/futures/space-economy-at-a-glance-2014-highlights.pdf.

¹⁶⁵ Michael Sheetz, "The Space Industry Will Be Worth Nearly $3 Trillion in 30 Years, Bank of America Predicts," CNBC, October 31 2017, https://www.cnbc.com/2017/10/31/the-space-industry-will-be-worth-nearly-3-trillion-in-30-years-bank-of-america-predicts.html.

¹⁶⁶ Victoria Samson, "U.S. Assessments and Evolving Perceptions of Space Threats and Capabilities," Secure World Foundation, December 12, 2018, https://swfound.org/media/206305/us-assessments-and-evolving-perceptions-of-space-threats-and-capabilities.pdf.

¹⁶⁸ Bahney and Pearl, "Why Creating a Space Force Changes Nothing: Space Has Been Militarised From the Start."

¹⁶⁹ Daniel Porras, "Towards ASAT Test Guidelines," UNIDIR, 2018, 2, http://www.unidir.org/files/publications/pdfs/-en-703.pdf.

¹⁷⁰ Brian Weeden, "2007 Chinese Anti-Satellite Test Fact Sheet," Secure World Foundation, updated November 23, 2010, https://swfound.org/media/9550/chinese_asat_fact_sheet_updated_2012.pdf.

¹⁷² For instance, the Atlantic describes how, in 2015, a barely noticeable error disrupted "GPS-dependent timing equipment around the world for around 12 hours." Although there were redundancies built into the systems, in different parts of the United States and Canada "police, fire, and EMS radio equipment stopped functioning." In several locations, BBC digital radio could not be accessed. Moreover, the article reports that the glitch was also detected in electrical power grids. Dan Glass, "What Happens If GPS Fails," Atlantic, June 13, 2016, https://www.theatlantic.com/technology/archive/2016/06/what-happens-if-gps-fails/486824/.

¹⁷³ UN Office for Outer Space Affairs, "Treaty on Principles Governing the Activities of States in the Exploration and Use of Outer Space, Including the Moon and Other Celestial Bodies," December 19, 1966, http://www.unoosa.org/pdf/gares/ARES_21_2222E.pdf.

¹⁷⁶ Federation of American Scientists, "Prevention of an Arms Race in Outer Space: General Provisions," 2013, https://fas.org/programs/ssp/nukes/ArmsControl_NEW/nonproliferation/NFZ/NP-NFZ-PAROS.html.

¹⁷⁷ UN Conference on Disarmament, "Note Verbale Dated 29 August 2016 From the Delegation of the United States of America Addressed to the Secretary-General of the Conference on Disarmament Transmitting the Submission of the United States to the Conference on Disarmament: "Implementing the Recommendations of the Report (A/68/189*) of the Group of Governmental Experts on Transparency and Confidence-Building Measures in Outer Space Activities to Enhance Stability in Outer Space," September 16, 2016, https://undocs.org/pdf?symbol=en/CD/2078.

¹⁷⁸ International Panel on Fissile Materials (IPFM), "Conference on Disarmament Fails to Re-establish Subsidiary Bodies," IPFM blog, March 14, 2019, http://fissilematerials.org/blog/2019/03/conference_on_disarmament.html.

¹⁷⁹ Brian Weeden and Victoria Samson (eds.), "Global Counterspace Capabilities: An Open Source Assessment," Secure World Foundation, April 2018, https://swfound.org/media/206118/swf_global_counterspace_april2018.pdf.

¹⁸⁰ See Article VI of the Treaty on Principles Governing the Activities of States in the Exploration and Use of Outer Space, Including the Moon and Other Celestial Bodies.

¹⁸¹ In 2015, former president Barack Obama signed the Space Resource Exploration and Utilization Act, which is part of the Commercial Space Launch Competitiveness Act (H.R. 2262). See U.S. Congress, "U.S. Commercial Space Launch Competitiveness Act [H.R. 2262]," November 25, 2015, https://www.congress.gov/bill/114th-congress/house-bill/2262.

¹⁸² University of Leiden, "The Hague International Space Resources Governance Working Group," https://www.universiteitleiden.nl/en/law/institute-of-public-law/institute-for-air-space-law/the-hague-space-resources-governance-working-group.

¹⁸³ United Nations, "Further Practical Measures for the Prevention of an Arms Race in Outer Space," UN Document A/RES/72/250, January 12, 2018, https://undocs.org/en/A/RES/72/250.

¹⁸⁵ Porras, "Towards ASAT Test Guidelines."

¹⁸⁷[xvi] UN Office of Disarmament Affairs, Securing Our Common Future: An Agenda for Disarmament; United Nations, "UN Secretary-General's Strategy on New Technologies," September 2018, https://www.un.org/en/newtechnologies/; and United Nations, "Secretary-General Launches High-Level Panel on Digital Cooperation," July 12, 2018, https://www.un.org/en/digital-cooperation-panel/. Please also see the panel's final report: United Nations, "The Age of Digital Interdependence," June 2019, https://www.un.org/en/pdfs/HLP%20on%20Digital%20Cooperation%20Report%20Executive%20Summary%20-%20ENG.pdf.

# data[].attributes.attachment_link.title
New Tech, New Threats, and New Governance Challenges: An Opportunity to Craft Smarter Responses?

# data[].attributes.attachment_link.description
Significant technological advances are being made across a range of fields, including information communications technology (ICT); artificial intelligence (AI), particularly in terms of machine learning and robotics; nanotechnology; space technology; biotechnology; and quantum computing to name but a few. These breakthroughs are expected to be highly disruptive and bring about major transformative shifts in how societies function.

# data[].attributes.attachment_link.title
‘Beauty and the Beast’ how emerging technology and Industry 5.0 will allow us to be more human

# data[].attributes.attachment_link.readable_text
Philosopher Nick Bostrom believes it's entirely possible that artificial intelligence (AI) could lead to the extinction of Homo sapiens. In his 2014 bestseller Superintelligence: Paths, Dangers, Strategies, Bostrom paints a dark scenario in which researchers create a machine capable of steadily improving itself. At some point, it learns to make money from online transactions and begins purchasing goods and services in the real world. Using mail-ordered DNA, it builds simple nanosystems that in turn create more complex systems, giving it ever more power to shape the world.

Now suppose the AI suspects that humans might interfere with its plans, writes Bostrom, who's at the University of Oxford in the United Kingdom. It could decide to build tiny weapons and distribute them around the world covertly. "At a pre-set time, nanofactories producing nerve gas or target-seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe."

For Bostrom and a number of other scientists and philosophers, such scenarios are more than science fiction. They're studying which technological advances pose "existential risks" that could wipe out humanity or at least end civilization as we know it-and what could be done to stop them. "Think of what we're trying to do as providing a scientific red team for the things that could threaten our species," says philosopher Huw Price, who heads the Centre for the Study of Existential Risk (CSER) here at the University of Cambridge.

The idea of science eliminating the human race can be traced all the way back to Frankenstein. In Mary Shelley's novel, the monster gets angry at his creator, Victor Frankenstein, for having spurned him. He kills Frankenstein's little brother William, but then offers the doctor a deal: Make a female companion for me and we will leave you in peace and go to South America to live out our days. Frankenstein starts working on the bride, but realizes that the couple might reproduce and outcompete humans: "A race of devils would be propagated upon the earth who might make the very existence of the species of man a condition precarious and full of terror." He destroys the half-finished female, reigniting the creature's wrath and bringing about his own demise.

"I think Frankenstein illustrates the point beautifully," says physicist Max Tegmark of the Massachusetts Institute of Technology (MIT) in Cambridge, a board member of CSER and a co-founder of a similar think tank, the Future of Life Institute (FLI), near MIT. "We humans gradually develop ever-more-powerful technology, and the more powerful the tech becomes, the more careful we have to be, so we don't screw up with it."

//www.sciencemag.org/sites/default/files/frankeneye-graphic.jpg

The study of existential risks is still a tiny field, with at most a few dozen people at three centers. Not everyone is convinced it's a serious academic discipline. Most civilization-ending scenarios-which include humanmade pathogens, armies of nanobots, or even the idea that our world is a simulation that might be switched off-are wildly unlikely, says Joyce Tait, who studies regulatory issues in the life sciences at the Innogen Institute in Edinburgh. The only true existential threat, she says, is a familiar one: a global nuclear war. Otherwise, "There is nothing on the horizon."

Harvard University psychologist Steven Pinker calls existential risks a "useless category" and warns that "Frankensteinian fantasies" could distract from real, solvable threats such as climate change and nuclear war. "Sowing fear about hypothetical disasters, far from safeguarding the future of humanity, can endanger it," he writes in his upcoming book Enlightenment Now: The Case for Reason, Science, Humanism, and Progress.

But advocates predict the field will only get more important as scientific and technological progress accelerates. As Bostrom pointed out in one paper, much more research has been done on dung beetles or Star Trek than on the risks of human extinction. "There is a very good case for saying that science has basically ignored" the issue, Price says.

Humanity has always faced the possibility of an untimely end. Another asteroid of the size that ended the dinosaurs' reign could hit Earth; a volcanic cataclysm could darken the skies for years and starve us all.

But existential risks arising from scientific advances were literally fiction until 16 July 1945, when the first atomic bomb was detonated. Based on some back-of-the-envelope calculations, physicist Edward Teller had concluded that the explosion might set off a global chain reaction, "igniting" the atmosphere. "Although we now know that such an outcome was physically impossible, it qualifies as an existential risk that was present at the time," Bostrom writes. Within 2 decades a real existential risk emerged, from growing stockpiles of the new weapons. Physicists had finally assembled Frankenstein's bride.

Other scientific disciplines may soon pose similar threats. "In this century we will introduce entirely new kinds of phenomena, give ourselves new kinds of powers to reshape the world," Bostrom says. Biotechnology is cheaper and easier to handle than nuclear technology has ever been. Nanotechnology is making rapid strides. And at a 2011 meeting in Copenhagen, Estonian computer programmer and Skype co-developer Jaan Tallinn told Price about his deep fears about AI during a shared taxi ride. "I'd never met anyone at that point who took that as seriously as Jaan," says Price, who was about to start working at the University of Cambridge.

Price introduced Tallinn to astronomer Martin Rees, a former president of the Royal Society, who had long warned that as science progresses, it will increasingly place the power to destroy civilization in the hands of individuals. The trio decided to launch CSER, the second such center after Bostrom's Future of Humanity Institute in Oxford, which he launched in 2005. CSER's name was "a deliberate attempt to push the idea of existential risk more towards the mainstream," Price says. "We were aware that people think of these issues as a little bit flaky."

CSER has recruited some big-name supporters: The scientific advisory board includes physicist Stephen Hawking, Harvard biologist George Church, global health leader Peter Piot, and tech entrepreneur Elon Musk. In a sign of just how small the field still is, Tallinn also co-founded FLI in 2014, and Church, Musk, Hawking, Bostrom, and Rees all serve on its scientific advisory board. (Actor Morgan Freeman, who has literally played God, is also an FLI adviser.)

Most of CSER's money comes from foundations and individuals, including Tallinn, who donated about $8 million to existential risk researchers in 2017. CSER's academic output has been "ephemeral" so far, Tallinn concedes. But the center was set up as "a sort of training ground for existential risk research," he says, with academics from elsewhere coming to visit and then "infecting" their own institutions with ideas.

The dozen people working at CSER itself-little more than a large room in an out-of-the-way building near the university's occupational health service-organize talks, convene scientists to discuss future developments, and publish on topics from regulation of synthetic biology to ecological tipping points. A lot of their time is spent pondering end-of-the-world scenarios and potential safeguards.

Church says a "crunch," in which a large part of the world population dies, is more likely than a complete wipe-out. "You don't have to turn the entire planet into atoms," he says. Disrupting electrical grids and other services on a huge scale or releasing a deadly pathogen could create chaos, topple governments, and send humanity into a downward spiral. "You end up with a medieval level of culture," Church says. "To me that is the end of humanity."

Existential risks stemming from the life sciences are perhaps easiest to imagine. Pathogens have proved capable of killing off entire species, such as the frogs that have fallen victim to the amphibian fungus Batrachochytrium dendrobatidis. And four influenza pandemics have swept the world in the past century, including one that killed up to 50 million people in 1918 and 1919. Researchers are already engineering pathogens that in principle could be even more dangerous. Worries about studies that made the H5N1 bird flu strain more easily transmissible between mammals led the United States to halt such research until late last year. Terrorists or rogue states could use labmade agents as a weapon, or an engineered plague could be released accidentally.

Rees has publicly wagered that by 2020, "bioterror or bioerror will lead to 1 million casualties in a single event." Harvard microbiologist Marc Lipsitch has calculated that the likelihood of a labmade flu virus leading to an accidental pandemic is between one in 1000 and one in 10,000 per year of research in one laboratory; Ron Fouchier of Erasmus MC in Rotterdam, the Netherlands, one of the researchers involved in the H5N1 studies, has dismissed that estimate, saying the real risk is more like one in 33 billion per year and lab.

One measure against "bioerror" might be to make researchers who carry out risky experiments buy insurance; that would require an independent assessment of the risk and would force researchers to face up to it, Lipsitch says. Still, the most important countermeasure is to strengthen the world's capacity to contain an outbreak early on, he adds, for instance with vaccines. "For biological risks, short of a really massive, coordinated, parallel attack around the world, the only way we are going to get to a really catastrophic scenario is by failing to control a smaller scenario," he says.

Viruses are unlikely to kill every last human, Bostrom says; for him and others, it is AI that poses truly existential threats. Most scenarios center on machines out-smarting humans, a feat called "super-intelligence." If such AI were ever achieved and it acquired a will of its own, it might turn malevolent and actively seek to destroy humans, like HAL, the computer that goes rogue aboard a spaceship in Stanley Kubrick's film 2001: A Space Odyssey.

Max Tegmark, Massachusetts Institute of Technology

Most AI experts worry less about machines rising up to overthrow their creators, however, than about them making a fatal mistake. To Tallinn, the most plausible way in which AI could end humanity is if it simply pursued its goals and, along the way, heedlessly created an environment fatal to humans. "Imagine a situation where the temperature rises by 100° or is lowered by 100°. We'd go extinct in a matter of minutes," Tallinn says. Tegmark agrees: "The real problem with AI is not malice, it's incompetence," he says.

A current-day analogy is the 2015 tragedy in which a suicidal Germanwings pilot told his plane's computer to descend to an altitude of 100 meters while flying over the French Alps. The machine complied, killing all 150 on board, even though it had GPS and a topographic map. "It did not have a clue about even the simplest human goal," Tegmark says. To avoid such calamities, scientists are trying to figure out how to teach AI human values and make sure they stick, a problem called "value alignment." "There might be fewer than 20 people who work full time on technical AI safety research," Bostrom says. "A few more talented people might substantially increase the rate of progress."

Critics say these efforts are unlikely to be useful, because future threats are inherently unpredictable. Predictions were a problem in every "foresight exercise" Tait has taken part in, she says. "We're just not good at it." Even if you foresee a risk, economic, political, and societal circumstances will all affect how it plays out. "Unless you know not only what is going to happen, but how it is going to happen, the information is not much use in terms of doing something about it," Tait says.

Pinker thinks the scenarios reveal more about human obsessions than real risks. We are drawn to prospects "that are highly improbable while having big impacts on our fitness, such as illicit sex, violent death, and Walter-Mittyish feats of glory," he writes. "Apocalyptic storylines are undoubtedly gripping-they are a supernormal stimulus for our morbid obsessions." Sure, he says, one can imagine a malevolent, powerful AI that people can no longer control. "The way to deal with this threat is straightforward: Don't build one."

Tallinn argues it's better to be safe than sorry. A 2017 survey showed that 34% of AI experts believed the risks associated with their work are an important problem; 5% said they are "one of the most important problems." "Imagine you're on a plane, and 40% of experts think that there is a bomb on this plane," Tallinn says. "You're not going to wait for the remaining experts to be convinced."

Price says that critics who accuse him and his colleagues of indulging in science fiction are not entirely wrong: Producing doomsday scenarios is not that different from what Shelley did. "The first step is to imagine that range of possibilities, and at that point, the kind of imagination that is used in science fiction and other forms of literature and film is likely to be extremely important," he says.

Scientists have an obligation to be involved, says Tegmark, because the risks are unlike any the world has faced before. Every time new technologies emerged in the past, he points out, humanity waited until their risks were apparent before learning to curtail them. Fire killed people and destroyed cities, so humans invented fire extinguishers and flame retardants. With automobiles came traffic deaths-and then seat belts and airbags. "Humanity's strategy is to learn from mistakes," Tegmark says. "When the end of the world is at stake, that is a terrible strategy."

# data[].attributes.attachment_link.title
Could science destroy the world? These scholars want to save us from a modern-day Frankenstein

# data[].attributes.attachment_link.description
Philosopher Nick Bostrom believes it's entirely possible that artificial intelligence (AI) could lead to the extinction of Homo sapiens. In his 2014 bestseller Superintelligence: Paths, Dangers, Strategies, Bostrom paints a dark scenario in which researchers create a machine capable of steadily improving itself.

# data[].attributes.attachment_link.description
Reprint: R0309B For most of the last 50 years, technology knew its place. Yes, we all spent a lot of time with it, but even five years ago, few people would seriously claim that technology had taken over their lives. It’s very different today. Technology is not only ubiquitous but has become highly intrusive as well. On the Internet, people invent imaginary identities in virtual chat rooms, playing out the lives they wish they really lived. Children are growing up with interactive toy animals that respond to them like real pets. Indeed, some critics claim that technology has not just entered our private lives but started to define them. If we want to be sure we’ll like who we’ve become in 50 years, we need to take a closer look at the psychological effects of current and future technologies. The smartest people in technology have already started. Universities like MIT and Caltech have been pouring millions of dollars into researching what happens when technology and humanity meet. To learn more about this research, HBR senior editor Diane L. Coutu spoke with one of the field’s most distinguished scholars: Sherry Turkle, MIT’s Abby Rockefeller Mauzé Professor in the Program in Science, Technology, and Society and the author of Life on the Screen, which explores how the Internet is changing the way we define ourselves. In a conversation with Coutu, Turkle discusses the psychological dynamics that can develop between people and their high-tech toys, describes ways in which machines might substitute for managers, and explains how technology is redefining what it means to be human. She warns that relatively small differences in technology design can have disproportionate effects on how humans relate to technology, to one another, and to themselves.

# data[].attributes.attachment_link.provider_name
Pew Research Center: Internet, Science & Tech

# data[].attributes.attachment_link.readable_text
Digital life is augmenting human capacities and disrupting eons-old human activities. Code-driven systems have spread to more than half of the world's inhabitants in ambient information and connectivity, offering previously unimagined opportunities and unprecedented threats. As emerging algorithm-driven artificial intelligence (AI) continues to spread, will people be better off than they are today?

Some 979 technology pioneers, innovators, developers, business and policy leaders, researchers and activists answered this question in a canvassing of experts conducted in the summer of 2018.

The experts predicted networked artificial intelligence will amplify human effectiveness but also threaten human autonomy, agency and capabilities. They spoke of the wide-ranging possibilities; that computers might match or even exceed human intelligence and capabilities on tasks such as complex decision-making, reasoning and learning, sophisticated analytics and pattern recognition, visual acuity, speech recognition and language translation. They said "smart" systems in communities, in vehicles, in buildings and utilities, on farms and in business processes will save time, money and lives and offer opportunities for individuals to enjoy a more-customized future.

Many focused their optimistic remarks on health care and the many possible applications of AI in diagnosing and treating patients or helping senior citizens live fuller and healthier lives. They were also enthusiastic about AI's role in contributing to broad public-health programs built around massive amounts of data that may be captured in the coming years about everything from personal genomes to nutrition. Additionally, a number of these experts predicted that AI would abet long-anticipated changes in formal and informal education systems.

Yet, most experts, regardless of whether they are optimistic or not, expressed concerns about the long-term impact of these new tools on the essential elements of being human. All respondents in this non-scientific canvassing were asked to elaborate on why they felt AI would leave people better off or not. Many shared deep worries, and many also suggested pathways toward solutions. The main themes they sounded about threats and remedies are outlined in the accompanying table.

[chart id="21972″]

Specifically, participants were asked to consider the following:
"Please think forward Our question: By 2030, do you think it is most likely that advancing AI and related technology systems will enhance human capacities and empower them? That is, most of the time, will most people be better off than they are today? Or is it most likely that advancing AI and related technology systems will lessen human autonomy and agency to such an extent that most people will not be better off than the way things are today?" to the year 2030. Analysts expect that people will become even more dependent on networked artificial intelligence (AI) in complex digital systems. Some say we will continue on the historic arc of augmenting our lives with mostly positive results as we widely implement these networked tools. Some say our increasing dependence on these AI and related systems is likely to lead to widespread difficulties.
Overall, and despite the downsides they fear, 63% of respondents in this canvassing said they are hopeful that most individuals will be mostly better off in 2030, and 37% said people will not be better off.

A number of the thought leaders who participated in this canvassing said humans' expanding reliance on technological systems will only go well if close attention is paid to how these tools, platforms and networks are engineered, distributed and updated. Some of the powerful, overarching answers included those from:

Sonia Katyal, co-director of the Berkeley Center for Law and Technology and a member of the inaugural U.S. Commerce Department Digital Economy Board of Advisors, predicted, "In 2030, the greatest set of questions will involve how perceptions of AI and their application will influence the trajectory of civil rights in the future. Questions about privacy, speech, the right of assembly and technological construction of personhood will all re-emerge in this new AI context, throwing into question our deepest-held beliefs about equality and opportunity for all. Who will benefit and who will be disadvantaged in this new world depends on how broadly we analyze these questions today, for the future."

We need to work aggressively to make sure technology matches our values. Erik Brynjolfsson

Erik Brynjolfsson, director of the MIT Initiative on the Digital Economy and author of "Machine, Platform, Crowd: Harnessing Our Digital Future," said, "AI and related technologies have already achieved superhuman performance in many areas, and there is little doubt that their capabilities will improve, probably very significantly, by 2030. ... I think it is more likely than not that we will use this power to make the world a better place. For instance, we can virtually eliminate global poverty, massively reduce disease and provide better education to almost everyone on the planet. That said, AI and ML [machine learning] can also be used to increasingly concentrate wealth and power, leaving many people behind, and to create even more horrifying weapons. Neither outcome is inevitable, so the right question is not 'What will happen?' but 'What will we choose to do?' We need to work aggressively to make sure technology matches our values. This can and must be done at all levels, from government, to business, to academia, and to individual choices."

Bryan Johnson, founder and CEO of Kernel, a leading developer of advanced neural interfaces, and OS Fund, a venture capital firm, said, "I strongly believe the answer depends on whether we can shift our economic systems toward prioritizing radical human improvement and staunching the trend toward human irrelevance in the face of AI. I don't mean just jobs; I mean true, existential irrelevance, which is the end result of not prioritizing human well-being and cognition."

Marina Gorbis, executive director of the Institute for the Future, said, "Without significant changes in our political economy and data governance regimes [AI] is likely to create greater economic inequalities, more surveillance and more programmed and non-human-centric interactions. Every time we program our environments, we end up programming ourselves and our interactions. Humans have to become more standardized, removing serendipity and ambiguity from our interactions. And this ambiguity and complexity is what is the essence of being human."

Judith Donath, author of "The Social Machine, Designs for Living Online" and faculty fellow at Harvard University's Berkman Klein Center for Internet & Society, commented, "By 2030, most social situations will be facilitated by bots - intelligent-seeming programs that interact with us in human-like ways. At home, parents will engage skilled bots to help kids with homework and catalyze dinner conversations. At work, bots will run meetings. A bot confidant will be considered essential for psychological well-being, and we'll increasingly turn to such companions for advice ranging from what to wear to whom to marry. We humans care deeply about how others see us - and the others whose approval we seek will increasingly be artificial. By then, the difference between humans and bots will have blurred considerably. Via screen and projection, the voice, appearance and behaviors of bots will be indistinguishable from those of humans, and even physical robots, though obviously non-human, will be so convincingly sincere that our impression of them as thinking, feeling beings, on par with or superior to ourselves, will be unshaken. Adding to the ambiguity, our own communication will be heavily augmented: Programs will compose many of our messages and our online/AR appearance will [be] computationally crafted. (Raw, unaided human speech and demeanor will seem embarrassingly clunky, slow and unsophisticated.) Aided by their access to vast troves of data about each of us, bots will far surpass humans in their ability to attract and persuade us. Able to mimic emotion expertly, they'll never be overcome by feelings: If they blurt something out in anger, it will be because that behavior was calculated to be the most efficacious way of advancing whatever goals they had 'in mind.' But what are those goals? Artificially intelligent companions will cultivate the impression that social goals similar to our own motivate them - to be held in good regard, whether as a beloved friend, an admired boss, etc. But their real collaboration will be with the humans and institutions that control them. Like their forebears today, these will be sellers of goods who employ them to stimulate consumption and politicians who commission them to sway opinions."

Andrew McLaughlin, executive director of the Center for Innovative Thinking at Yale University, previously deputy chief technology officer of the United States for President Barack Obama and global public policy lead for Google, wrote, "2030 is not far in the future. My sense is that innovations like the internet and networked AI have massive short-term benefits, along with long-term negatives that can take decades to be recognizable. AI will drive a vast range of efficiency optimizations but also enable hidden discrimination and arbitrary penalization of individuals in areas like insurance, job seeking and performance assessment."

Michael M. Roberts, first president and CEO of the Internet Corporation for Assigned Names and Numbers (ICANN) and Internet Hall of Fame member, wrote, "The range of opportunities for intelligent agents to augment human intelligence is still virtually unlimited. The major issue is that the more convenient an agent is, the more it needs to know about you - preferences, timing, capacities, etc. - which creates a tradeoff of more help requires more intrusion. This is not a black-and-white issue - the shades of gray and associated remedies will be argued endlessly. The record to date is that convenience overwhelms privacy. I suspect that will continue."

danah boyd, a principal researcher for Microsoft and founder and president of the Data & Society Research Institute, said, "AI is a tool that will be used by humans for all sorts of purposes, including in the pursuit of power. There will be abuses of power that involve AI, just as there will be advances in science and humanitarian efforts that also involve AI. Unfortunately, there are certain trend lines that are likely to create massive instability. Take, for example, climate change and climate migration. This will further destabilize Europe and the U.S., and I expect that, in panic, we will see AI be used in harmful ways in light of other geopolitical crises."

Amy Webb, founder of the Future Today Institute and professor of strategic foresight at New York University, commented, "The social safety net structures currently in place in the U.S. and in many other countries around the world weren't designed for our transition to AI. The transition through AI will last the next 50 years or more. As we move farther into this third era of computing, and as every single industry becomes more deeply entrenched with AI systems, we will need new hybrid-skilled knowledge workers who can operate in jobs that have never needed to exist before. We'll need farmers who know how to work with big data sets. Oncologists trained as robotocists. Biologists trained as electrical engineers. We won't need to prepare our workforce just once, with a few changes to the curriculum. As AI matures, we will need a responsive workforce, capable of adapting to new processes, systems and tools every few years. The need for these fields will arise faster than our labor departments, schools and universities are acknowledging. It's easy to look back on history through the lens of present - and to overlook the social unrest caused by widespread technological unemployment. We need to address a difficult truth that few are willing to utter aloud: AI will eventually cause a large number of people to be permanently out of work. Just as generations before witnessed sweeping changes during and in the aftermath of the Industrial Revolution, the rapid pace of technology will likely mean that Baby Boomers and the oldest members of Gen X - especially those whose jobs can be replicated by robots - won't be able to retrain for other kinds of work without a significant investment of time and effort."

Barry Chudakov, founder and principal of Sertain Research, commented, "By 2030 the human-machine/AI collaboration will be a necessary tool to manage and counter the effects of multiple simultaneous accelerations: broad technology advancement, globalization, climate change and attendant global migrations. In the past, human societies managed change through gut and intuition, but as Eric Teller, CEO of Google X, has said, 'Our societal structures are failing to keep pace with the rate of change.' To keep pace with that change and to manage a growing list of 'wicked problems' by 2030, AI - or using Joi Ito's phrase, extended intelligence - will value and revalue virtually every area of human behavior and interaction. AI and advancing technologies will change our response framework and time frames (which in turn, changes our sense of time). Where once social interaction happened in places - work, school, church, family environments - social interactions will increasingly happen in continuous, simultaneous time. If we are fortunate, we will follow the 23 Asilomar AI Principles outlined by the Future of Life Institute and will work toward 'not undirected intelligence but beneficial intelligence.' Akin to nuclear deterrence stemming from mutually assured destruction, AI and related technology systems constitute a force for a moral renaissance. We must embrace that moral renaissance, or we will face moral conundrums that could bring about human demise. ... My greatest hope for human-machine/AI collaboration constitutes a moral and ethical renaissance - we adopt a moonshot mentality and lock arms to prepare for the accelerations coming at us. My greatest fear is that we adopt the logic of our emerging technologies - instant response, isolation behind screens, endless comparison of self-worth, fake self-presentation - without thinking or responding smartly."

John C. Havens, executive director of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems and the Council on Extended Intelligence, wrote, "Now, in 2018, a majority of people around the world can't access their data, so any 'human-AI augmentation' discussions ignore the critical context of who actually controls people's information and identity. Soon it will be extremely difficult to identify any autonomous or intelligent systems whose algorithms don't interact with human data in one form or another."

At stake is nothing less than what sort of society we want to live in and how we experience our humanity. Batya Friedman

Batya Friedman, a human-computer interaction professor at the University of Washington's Information School, wrote, "Our scientific and technological capacities have and will continue to far surpass our moral ones - that is our ability to use wisely and humanely the knowledge and tools that we develop. ... Automated warfare - when autonomous weapons kill human beings without human engagement - can lead to a lack of responsibility for taking the enemy's life or even knowledge that an enemy's life has been taken. At stake is nothing less than what sort of society we want to live in and how we experience our humanity."

Greg Shannon, chief scientist for the CERT Division at Carnegie Mellon University, said, "Better/worse will appear 4:1 with the long-term ratio 2:1. AI will do well for repetitive work where 'close' will be good enough and humans dislike the work. ... Life will definitely be better as AI extends lifetimes, from health apps that intelligently 'nudge' us to health, to warnings about impending heart/stroke events, to automated health care for the underserved (remote) and those who need extended care (elder care). As to liberty, there are clear risks. AI affects agency by creating entities with meaningful intellectual capabilities for monitoring, enforcing and even punishing individuals. Those who know how to use it will have immense potential power over those who don't/can't. Future happiness is really unclear. Some will cede their agency to AI in games, work and community, much like the opioid crisis steals agency today. On the other hand, many will be freed from mundane, unengaging tasks/jobs. If elements of community happiness are part of AI objective functions, then AI could catalyze an explosion of happiness."

Kostas Alexandridis, author of "Exploring Complex Dynamics in Multi-agent-based Intelligent Systems," predicted, "Many of our day-to-day decisions will be automated with minimal intervention by the end-user. Autonomy and/or independence will be sacrificed and replaced by convenience. Newer generations of citizens will become more and more dependent on networked AI structures and processes. There are challenges that need to be addressed in terms of critical thinking and heterogeneity. Networked interdependence will, more likely than not, increase our vulnerability to cyberattacks. There is also a real likelihood that there will exist sharper divisions between digital 'haves' and 'have-nots,' as well as among technologically dependent digital infrastructures. Finally, there is the question of the new 'commanding heights' of the digital network infrastructure's ownership and control."

Oscar Gandy, emeritus professor of communication at the University of Pennsylvania, responded, "We already face an ungranted assumption when we are asked to imagine human-machine 'collaboration.' Interaction is a bit different, but still tainted by the grant of a form of identity - maybe even personhood - to machines that we will use to make our way through all sorts of opportunities and challenges. The problems we will face in the future are quite similar to the problems we currently face when we rely upon 'others' (including technological systems, devices and networks) to acquire things we value and avoid those other things (that we might, or might not be aware of)."

James Scofield O'Rourke, a professor of management at the University of Notre Dame, said, "Technology has, throughout recorded history, been a largely neutral concept. The question of its value has always been dependent on its application. For what purpose will AI and other technological advances be used? Everything from gunpowder to internal combustion engines to nuclear fission has been applied in both helpful and destructive ways. Assuming we can contain or control AI (and not the other way around), the answer to whether we'll be better off depends entirely on us (or our progeny). 'The fault, dear Brutus, is not in our stars, but in ourselves, that we are underlings.'"

Simon Biggs, a professor of interdisciplinary arts at the University of Edinburgh, said, "AI will function to augment human capabilities. The problem is not with AI but with humans. As a species we are aggressive, competitive and lazy. We are also empathic, community minded and (sometimes) self-sacrificing. We have many other attributes. These will all be amplified. Given historical precedent, one would have to assume it will be our worst qualities that are augmented. My expectation is that in 2030 AI will be in routine use to fight wars and kill people, far more effectively than we can currently kill. As societies we will be less affected by this as we currently are, as we will not be doing the fighting and killing ourselves. Our capacity to modify our behaviour, subject to empathy and an associated ethical framework, will be reduced by the disassociation between our agency and the act of killing. We cannot expect our AI systems to be ethical on our behalf - they won't be, as they will be designed to kill efficiently, not thoughtfully. My other primary concern is to do with surveillance and control. The advent of China's Social Credit System (SCS) is an indicator of what it likely to come. We will exist within an SCS as AI constructs hybrid instances of ourselves that may or may not resemble who we are. But our rights and affordances as individuals will be determined by the SCS. This is the Orwellian nightmare realised."

Mark Surman, executive director of the Mozilla Foundation, responded, "AI will continue to concentrate power and wealth in the hands of a few big monopolies based on the U.S. and China. Most people - and parts of the world - will be worse off."

William Uricchio, media scholar and professor of comparative media studies at MIT, commented, "AI and its related applications face three problems: development at the speed of Moore's Law, development in the hands of a technological and economic elite, and development without benefit of an informed or engaged public. The public is reduced to a collective of consumers awaiting the next technology. Whose notion of 'progress' will prevail? We have ample evidence of AI being used to drive profits, regardless of implications for long-held values; to enhance governmental control and even score citizens' 'social credit' without input from citizens themselves. Like technologies before it, AI is agnostic. Its deployment rests in the hands of society. But absent an AI-literate public, the decision of how best to deploy AI will fall to special interests. Will this mean equitable deployment, the amelioration of social injustice and AI in the public service? Because the answer to this question is social rather than technological, I'm pessimistic. The fix? We need to develop an AI-literate public, which means focused attention in the educational sector and in public-facing media. We need to assure diversity in the development of AI technologies. And until the public, its elected representatives and their legal and regulatory regimes can get up to speed with these fast-moving developments we need to exercise caution and oversight in AI's development."

The remainder of this report is divided into three sections that draw from hundreds of additional respondents' hopeful and critical observations: 1) concerns about human-AI evolution, 2) suggested solutions to address AI's impact, and 3) expectations of what life will be like in 2030, including respondents' positive outlooks on the quality of life and the future of work, health care and education. Some responses are lightly edited for style.

# data[].attributes.attachment_link.title
Artificial Intelligence and the Future of Humans

# data[].attributes.attachment_link.description
Experts say the rise of artificial intelligence will make most people better off over the next decade, but many have concerns about how advances in AI will affect what it means to be human, to be productive and to exercise free will.

# data[].attributes.headline
Technology pros and cons: is tech good for society? - netivistnetivist.org › debate › pros-and-cons-of-technology

# data[].attributes.subject
Technology pros and cons: is tech good for society? - netivistnetivist.org › debate › pros-and-cons-of-technology

# data[].attributes.attachment_link.readable_text
Find out what are the main the pros and cons of technology. We discuss the history, importance and role of technology, as well as its impact on society. Is technological progress good or is it becoming a threat to us? Vote in our poll and debate (see below)
Is technology good for society? A brief history

The impact of technology on society is undeniable. Technology and science have played a central role in human history and help shape entire civilizations. Technological progress was key for the emergence and downfall of empires. The development of hunting and farming tools allowed our ancestors to dominate other hominid species. The invention of the wheel and writing, as well as the introduction of metal tools and weapons were other landmarks in the history of technology. Many successive civilizations have contributed to the world's advancement. Often the development of technology also helped these societies to dominate militarily, politically, and economically their neighbors, as well as increase the welfare of their citizens.

The Egyptians invented many farming, medical and construction technologies. The Mesopotamians are credited for introducing irrigation and drainage systems, as well as sophisticated mud-brick and stone architecture techniques. Greeks were responsible for many inventions, such as the watermill, and the improvement on many existing technologies. Still today Greek mathematicians, engeneers and philosophers are recognised as fundamental to the history of human thought and technology. The Romans brought technology to a new level, and their monumental amphitheatres, aqueducts, bridges, harbours, dams and public baths help them dominate the Western world for centuries. Ancient Indian civilizations are credited for developing good understanding of seafaring, sanitation and hydrological technologies. Chinese discoveries include paper, matches, the cross-bow, seismological detectors, the wheelbarrow, the suspension bridge and the compass, among others.

During the Middle Ages architecture, navigation, papermaking and military technologies were developed. The Arab conquest of the Iberian Peninsula helped to introduce Europe to many technological advances developed in the east. The Renaissance and the Age of Explorations also demonstrated how innovation shaped societies. Research and inventions were put into practice. The use of artillery, new cranes and medical techniques marked a beginning of a scientific revolution. The Portuguese and Spanish discoveries, were enabled by technological progress but also help connecting different civilization which accelerated the spread of innovation. The industrial revolution brought the steam engine and developments in mining, metalurgy, manufacturing and transport. Since the 19th century, science and technology have evolved even faster. The 20th century brought the expansion of electrification and communication technologies, mass industrial production, electronic computing, nuclear technology and space exploration among others. It also demonstrated the devastating power of some of the technologies developed by humans. The weapons developed during that century, including the weapons of mass destruction, caused the deaths of millions.

The 21st century seems to have accelerated even faster these processes and intensified the impact of technology on society. Technology colleges such as MIT and Stanford have help accelerate scientific discovery. Genetic engineering, nanotechnology, 3D printing, wireless powered devices, augmented reality, articifical intelligence, drones, quantum computers and superconductivity are among the many new technologies we are witnessing today. But what come next? Can technology continue to help our lives or is it becoming a real threat to us? Can we keep scientific progress under control or will technology control our lives? Let's discuss the advantages and disadvantages or technology.

Pros and cons of technology
Pros of technological progress

Thanks to technological progress humans live longer and much more comfortable lives. The medical advancements have helped us develop vaccines and treatment for diseases which were previously lethal. Giving birth is not as dangerous as it used to be. Technological progress has allowed develop new techniques for diagnosis and mitigation of diseases and other conditions. Scientific research has improved our understanding of nutrition and contributed to healthier lifestyles.

Technology has allowed humans to travel faster and trade goods all over the world. Crossing the Atlantic was historically an adventure that would take weeks. Now in just a few hours you can travel from New York to London or Paris. Thanks to technology we can have holidays in remote locations and capture these moments through pictures. We can now buy and consume goods produced far away. In the same supermarket you can find French cheeses, South African wines, Spanish olive oils, Brazilian coffees, and Japanese fish.

Without technology we would still be nomads hunting and gathering fruits and vegetables. Industrial societies heavily rely on technological progress. We can feed a fast growing world population thanks to the continuous innovation in production techniques. New inventions help foster the production, storage, treatment and transportation of goods. Societies which invest in research and development have a competitive edge and thrive. The people in technologically advanced societies live more comfortable lives.

Genetically modified foods (GMO) may help fight hunger and ensure that world population continues to be fed. Genetic modification techniques contribute to produce more food and to maintain agricultural production at affordable prices.

The Internet, computers and mobile phones illustrate the role of technology in improving society. Efficiency has skyrocketed thanks to these inventions. Our work and social lives have been transformed. People can now work from home and collaborate with teams located in other towns, countries or even continents. We can keep a fluid communication and relationship with friends and family living abroad. News of events cross the globe in seconds. Social netwoks such as facebook and twitter are extremely useful. Thanks to technology grandparents get to see and chat with their grandchildren much more often. People today get to meet others sharing similar hobbies or interests.

Thanks to new technologies, alternative forms of entertainment and art have developed during the last century. Photography, radio, movies, television shows, music and video games occupy a central spot in people's lives. There are new forms of entertainment at our doorstep, such as virtual reality. Additionally, IT is facilitating the work of creators and help increase the quality of entertainment.

The importance of technology in the delopment of renewable energies is evident. Without technological progress it would be difficult to envisage a green future in which the problem of climate change could be kept under control. Scientific advancements are making electric cars more affordable and enhancing the effectiveness and efficiency of solar and eolic energy, as well as that hydropower.

Cons and risks of technology

Technology is altering our lifestyle and will alter the cognitive and social development of current and future generations. There are many different ways in which the evolution of technology and society are connected. Technological innovation has changed our lives. Computers, smartphones and the internet have strongly affected how we interact with other people. Many claim that they are dehumanizing our lives and making us more solitary people. Technology may be also facilitating cultural colonialism and reducing diversity. Today, children play less with other children and spend much of their time watching videos in their tablets and playing video games. People are doing less physical exercise than their ancestors. We are becoming increasingly detached from nature and attached to technological gadgets.

Human cloning technology is a reality and in addition to some obvious advantages, human cloning brings some risks. For instance it could create worrisome divides in society between those genetically divided to be smarter or physically more attractive and the rest. Human cloning will be difficult to regulate and will bring concerns regarding its interference with nature and religious beliefs.

Weaponization of viruses. For instance, viruses such as Ebola or AIDS could be transformed into a virus that could be transmitted through the air. This could endanger or even cause the extinction of the the human race. Lifeforms can be created through genetic manipulation. With techonological progress the techniques to create or manipulate lifeforms will be increasingly accessible to us. Potentially even high school children will be able to create life in science experiments. Genomes of infectious diseases will be available to download from the internet. Terrorists, psychopaths and other criminals will have the capacity to use genetic manipulation to harm or threaten others.

Similarly, scientific experimentation might create enormous dangers for society. Risky experiments may go wrong. Researchers are currently mutating microorganism in order to find cures to diseases. By accident these diseases could escape the laboratory and spread. Experiments with particle accelerators, such as the Large Hadron Collider, entail some serious risks. Some scientists even claimed that humans could create a black hole that could destroy Earth.

Enrichment of uranium is becoming an increasingly cheaper process. Traditionally the infrastructure required to produce nuclear power and build nuclear bombs was extremely expensive. Thanks to technological progress and the use of laser beams to separate U-235 and U-238, in the not so distant future, people might be able to enrich uranium home. Nuclear weapons in the hands of the wrong organizations or individuals could create enormous unrest in the world population and a serious security threat.

Technological progress is to be blamed for the negative effects of global warming and climate change. The role of technology fostering economic progress is difficult to deny. However, at the same time the generation of enegy necessary to the increased production and transportation of goods, for instance through combustion engines and thermoelectric generators, has produced an increased in the emission and concentration of greenhouse gases in the atmosphere.

Technology can be also used as an undesirable tool of control. For instance, scientists are working to develop brain scanning machines which could allow read a person's thoughts. This would have great benefits as could allow people with disabilities or people having suffered brain damage to communicate. Moreover if through a magnetic resoncance we could enquire criminals and terrorist we could also prevent harm for society. Similarly, thanks to advanced IT, it is possible to analyse the communications of millions of people and identify potential crimes and wrongdoers. Unfortunately, there is a very real possibility that technology will end up used by some governments in a draconian way. A dystopian future where people are constantly inquisitorially surveilled by a Big Brother as in Orwell's science fiction classic 1984.

What do you think, is technology good or bad? Is technological progress out of control? Will the rest of the 21st century see the importance of technology reduced?

Technology pros and cons: is technology good or a threat for society? Vote and share your views and predictions on the impact of technology on society

If you change your mind, you can change your vote simply by clicking on another option.

# data[].attributes.attachment_link.title
Technology pros and cons: is tech good for society?

# data[].attributes.attachment_link.description
Check out the pros and cons of technology. We discuss the history, importance and role of technology, as well as its impact on society. Is technology good?

# data[].attributes.headline
Is Technology Moving Too Fast? - The Long Nowhttps://longnow.org › essays › technology-moving-too-...

# data[].attributes.subject
Is Technology Moving Too Fast? - The Long Nowhttps://longnow.org › essays › technology-moving-too-...

# data[].attributes.attachment_link.readable_text
Is Technology Moving Too Fast?

Published on Monday, June 19, 02000 * 21 years, 4 months ago
Written by Stewart Brand for Time

The newest technologies--computers, genetic engineering and the emerging field of nanotech--differ from the technologies that preceded them in a fundamental way. The telephone, the automobile, television and jet air travel accelerated for a while, transforming society along the way, but then settled into a manageable rate of change. Each was eventually rewarded more for staying the same than for radically transforming itself--a stable, predictable, reliable condition known as "lock-in."

Computers, biotechnology and nanotech don't work that way. They are self-accelerating; that is, the products of their own processes enable them to develop ever more rapidly. New computer chips are immediately put to use developing the next generation of more powerful ones; this is the inexorable acceleration expressed as Moore's law. The same dynamic drives biotech and nanotech--even more so because all these technologies tend to accelerate one another. Computers are rapidly mapping the DNA in the human genome, and now DNA is being explored as a medium for computation. When nanobots are finally perfected, you can be sure that one of the first things they will do is make new and better nanobots.

Technologies with this property of perpetual self-accelerated development--sometimes termed "autocatalysis"--create conditions that are unstable, unpredictable and unreliable. And since these particular autocatalytic technologies drive whole sectors of society, there is a risk that civilization itself may become unstable, unpredictable and unreliable.

Perhaps what civilization needs is a NOT-SO-FAST button. Proponents of technological determinism make a strong case for letting self-accelerating technologies follow their own life cycle. Rapid development in computer technology, they point out, has spun off robotics and the Internet--to the great benefit of industry and human communications. Besides, it isn't so easy for a free society to put the brakes on technology. Even if one country decided to forgo the next technological revolution, another country would gladly take it up.

There are scenarios, however, in which technology may brake itself. In the aging population of the developed world, many people are already tired of trying to keep up with the latest cool new tech. Youth-driven tech acceleration could be interpreted as simple youthful folly--shortsighted, disruptive, faddish. The market for change could dry up, and lock-in might again become the norm. Stress and fatigue make powerful decelerators.

So do religious and cultural factors. Radical new technologies are often seen as moral threats by conservative religious groups or as economic and cultural threats by political groups. Powerful single-issue voting blocs like the antiabortionists could arise. Or terrorists like Theodore Kaczynski.

Change that is too rapid can be deeply divisive; if only an elite can keep up, the rest of us will grow increasingly mystified about how the world works. We can understand natural biology, subtle as it is, because it holds still. But how will we ever be able to understand quantum computing or nanotechnology if its subtlety keeps accelerating away from us?

Constant technological revolution makes planning difficult, and a society that stops planning for the future is likely to become a brittle society. It could experience violent economic swings. It could trip into wars fought with vicious new weapons. Its pervasive new technologies could fail in massive or horrible ways. Or persistent, nagging small failures could sap the whole enterprise.

With so many powerful forces in play, technology could hyperaccelerate to the stars with stunning rapidity, or it could stall completely. My expectation is that it will do both, with various technologies proceeding at various rates. The new technologies may be self-accelerating, but they are not self-determining. They are the result of ever renegotiated agreement with society. Because they are so potent, their paths may undergo wild oscillations, but I think the trend will be toward the dynamic middle: much slower than the optimists expect, much faster than the pessimists think humanity can bear.

First published in Time.

# data[].attributes.attachment_link.title
Is Technology Moving Too Fast? - The Long Now

# data[].attributes.attachment_link.description
Published on Monday, June 19, 02000 * 21 years, 4 months ago Written by Stewart Brand for Time The newest technologies--computers, genetic engineering and the emerging field of nanotech--differ from the technologies that preceded them in a fundamental way.